I'm sceptical that it would be good, or efficient on packing.
I have slept under cover (bus shelters, on wooden benches) in cycling gear suitable for the conditions, even for sitting outside to eat and drink, plus a fleece jacket and fleece hat.
It wasn't very successful. I woke up really cold after a few minutes, and had to get moving again to warm up, which wasn't quick. I normally run quite hot including when I sleep, though less so after exertion.
I'm not saying it's not possible, but you'll need a lot more clothing than you think, and while you can carry a down jacket in little space, your lower half will struggle (down salopettes would be your best best but then the bulk and weight would be comparable to a down sleeping bag, and the price higher).
All over base layers are enough to get up briefly when it's really quite cold - a few degrees above freezing and all I reckon you need is enough for decency* or to get around an organised campsite with toilet facilities if you're getting back into your sleeping bag soon. Even the latter is far less than you'd need to sleep all night.
*this could mean anything from nothing at all (e.g. wild camping solo or in a couple), or covering top and bottom halves (campsites, popular wild camping spots, remote wild camping with mixed gender friends), and it can vary within a trip.