Quality of sleep can depend on a multitude of factors. These are some of my experiences:
Quality and appropriateness of gear
Most people sleep poorly if they get too cold or too hot. Having a quality sleeping bag that is appropriate for your climate and time of year will definitely increase the likelihood that you will get quality sleep. Material matters when it comes to sleeping bags. Down will regulate temperature better, but is useless when it gets wet. Synthetic materials are poor at regulating temperature, but keep you warm even when they get wet. Don't buy cotton.
Having a sleeping pad will contribute to a good night's rest. The ground acts as a heat sink, especially when your sleeping spot is on sand or rock. Forrest duff will act as an insulator, and I will go lighter on the sleeping pad if I know in advance that I will be staying below the timber line. When it comes to sleeping pads, you have a choice between air mattresses and foam pads. The former can be a bit noisier and obviously can be susceptible to leaks, but, in general, is lighter and more comfortable. The latter is cheaper, and can withstand a beating.
Some people sleep best in a tent. Some people sleep best under the open sky, or as close to it as possible. In the summer, I often just bring a tarp in case it rains, since I have a hard time getting up in time if I sleep inside a tent. Tents (or insect nets) are a major plus when setting up camp next to mosquito-infested waters.
Maintenance of gear
If you get wet you will sleep poorly. Make sure you keep your sleeping gear dry. I keep my sleeping bag in a dry-bag to prevent it from getting wet.
For a couple of years, I have led wilderness trips for children. I have observed a correlation between children who spend time outdoors on a regular basis and getting better night's rest. I suspect this might have to do with being in an unfamiliar environment. On longer trips, most children were sleeping well at the end, probably due to exhaustion and a new-found familiarity with the outdoors.
The human factor
I find that the single most annoying thing for myself when trying to fall asleep in the outdoors are inconsiderate individuals in close proximity. European campgrounds can be very dense, especially during the peak season, and often, at least, one group of individuals does not respect the quiet hours. For that reason, I avoid proximity to others whenever possible. I have a pair of noise-cancelling headphones that I use (they don't have to be plugged into anything) when I find myself in a situation like that. For me, it also helps if I can be next to a source of white-noise, like a creek. When going backpacking, especially in areas where you have some choice when it comes to campsite selection, the "human factor" can often be completely avoided. I have been on many trips in the mountains of California's Sierra Nevada where I had the nights completely to myself.
Many people sleep poorly at elevation. What the elevation is that makes people sleep poorly varies heavily from person to person. In my experience, the slower you gain elevation, the better you will acclimatized, and your sleep quality improves.
I find that heavy rain does not distract me unless I am worried that my stuff could get wet. It helps with "the human factor" as well since people generally won't stay up as late in poor weather. It also provides white noise, which helps me sleep.
The elements pretty much tie back to the appropriateness and quality of gear. If I trust that my gear will withstand the situation, I normally sleep soundly.
The sun can be a factor as well. If you would like to sleep in during the summer months, make sure that you set up in a shady spot. There is nothing worse that waking up in your sleeping bag inside a tent that has been baking for 30 minutes.
I have observed (on myself and others) that often the first night can be rough, even with good gear and a lot of experience.
Your general disposition. If you have trouble sleeping at home, you will probably find the same to be true in the outdoors. If your sleeplessness is due to stress, though, a couple of nights in the outdoors may help wonders.