Mesh lined running shorts. Roughly equivalent to boxers in coverage. I routinely use these on canoe trips and hiking trips under my wind pants. Morning starts with longs, and as the day warms up, they come off as the day warms up. I often put longs back on immediately on hitting camp for bug protection.
That said: From years of outdoor activities, I'm used to being wet.
If you get ones that are 100% nylon/polyester they dry very fast. Blotting as much water as possible with a towel then 10-15 minutes of lounging about nearby will leave you dry enough to put your over clothes on.
The difficulty is the differential temperature. If you have been in hot water, you are likely flushed, certainly damp, and acclimated in part to the heat. Coming out into cold will result in lots of heat loss initially. You may not want even damp shorts on.
Changing in the wild
Comment on the question said that changing in unknown mixed company isn't viable. Consider:
Put on a wind parka, or any other garment that comes down to mid thigh. Fasten the bottom two snaps, withdraw your arms, and drop your underwear and put on your swimwear.
Any place this popular likely has some form of bathroom facility. Change in the outhouse.
Walk a hundred meters into the bush, put a large tree between you and the group and change there.