I don't want, however, to inform anyone around about that cache - I want to keep it secret only for 'chosen'.
Although you don't want to, I'd strongly suggest rethinking this - does it really matter if a few neighbours know, and it stops an embarrassing incident or two when the police are called out? When I placed a cache in a (public) residential area overlooked by a lot of houses, I was asked to make sure I'd let the neighbours know for this reason.
This doubles in importance when it involves entering a garden, since if by chance geocachers stray into a neighbours garden, you want to make sure that if they're confronted and talk about Geocaching, your neighbours know not to worry and can politely ask them to move on.
Regardless though, you need to make sure that geocachers know as much as possible that it's the right garden they're entering, and not someone else's - the last thing you want is them stumbling into a neighbouring garden. Personally I'd get around this by a (coded) sign on the door, and corresponding clue - "XYFH5" written on the door for instance, and then "Come through XYFH5" as the clue. (Sure, it doesn't have to be that simple, you could make it part of a puzzle - but the important thing is that it has to be as unambiguous as possible!)