What to sleep in?
Any building will be inviting vandals, so I would just level out a part of the plot with easy access for your car or a camper van.
Instead of having a building you can bring a tent or a camper van, RV or a trailer that is ready to camp in.
In (parts of) Europe it is also quite common to see a small trailer that holds a tent, which you fold out into a roomy tent big enough to sleep 4 or even 6 people. The beds are part of the structure and there is some space for the essentials.
A base for your nights!
That flat area can be just dirt if the dirt in your plot is going to work for that, it can also be tiles or concrete slabs you put in, cut wood or if allowed in the area, a slap of concrete you poor on the plot.
Or you can make and maintain a lawn to camp on or around if you are willing to put in the time for mowing it and the tent (or vehicle) will not be on it enough to do damage.
Of all above the concrete slap will have the most impact but is also the best collecting all that drips from the vehicle so you can collect that and take it off site if needed.
If you are in an area where forest fires are common or very dangerous, I would not make a fire pit but use a fire container which you can put in your car when you leave, either a BBQ or a fire cage.
How to make the going out effort free?
If you want to stay in a tent or several small tents, you can have all of the gear packed in a trailer or a couple of duffel bags which you can put in the back of your car.
A trailer, camping van or RV are always ready to go.
And the fold in trailer tent is basically ready to go, although you might prefer not to store the extra's inside long term.
Drive on or not?
The less easy the access to the plot, the less likely someone will come and make use of it, (either as you would or as a vandal) but the less easy it will be for you to arrive and leave. So it will be a trade off.
Consider parking a bit farther out and walk your gear in over a narrow path compared to driving onto the site.
If the site is a bit farther from an access road you may have to walk your gear in. In which case a small cart with wheels suitable for the ground might be a very valued part of your kit.
If you are just camping there once in a long while, you can dig a latrine or bring out all human waste as well as all your (kitchen)waste.
But if you do go there often, you might want to consider a 'loo' structure where you can easily collect the human waste to take off, (which can be a tent like structure) or build a system that does not need the taking away of waste, compost loo's come to mind but are likely not the only suitable structure.
Using a camping vehicle you might have a waste water system doing it for you.
Waste water that is true water might be allowed to get into the ground or through the ground into a natural stream in/near the plot, but only if it is clean of chemicals and disease. A shower (behind a screen for privacy) can be rigged with a bucket or watering can, over a small tiled area for foot safety, as long as no unwanted chemicals are used.