Hi,
And thanks for the welcome!
If you are travelling from the A5, you come up woodford rd, then under the railway bridge, straight on up and past "clock hill". then you come to the "green" with the Stag & Pheasant on LHS and Michael Drayton flats on RHS (Drayton Court).
Carry on past the Society of Friends chapel, (on LHS) as you go up the road you come to some 1950's flats? that are on the left hand side, directly oppposite is a field, this is where the castle is, It is set back from the road quite a way, and if you blink then you will miss it.....lol.
I do hope that helps, I did try to put in a more accurate placement on your "map" as the one that is currently there is way off the mark.
Hopefully, cathidaw will be about soon and be in touch.
Julie