(Use case: A hike to heated waters--hiking with only swimwear for lower attire is not an option.)
I find most swimwear is not exactly suitable for wearing under clothing, speedos would work but I'm hoping for some middle ground--snug fitting but not quite so skimpy.
Mesh lined running shorts. Roughly equivalent to boxers in coverage. I routinely use these on canoe trips and hiking trips under my wind pants. Morning starts with longs, and as the day warms up, they come off as the day warms up. I often put longs back on immediately on hitting camp for bug protection.
That said: From years of outdoor activities, I'm used to being wet.
If you get ones that are 100% nylon/polyester they dry very fast. Blotting as much water as possible with a towel then 10-15 minutes of lounging about nearby will leave you dry enough to put your over clothes on.
The difficulty is the differential temperature. If you have been in hot water, you are likely flushed, certainly damp, and acclimated in part to the heat. Coming out into cold will result in lots of heat loss initially. You may not want even damp shorts on.
Comment on the question said that changing in unknown mixed company isn't viable. Consider:
Put on a wind parka, or any other garment that comes down to mid thigh. Fasten the bottom two snaps, withdraw your arms, and drop your underwear and put on your swimwear.
Any place this popular likely has some form of bathroom facility. Change in the outhouse.
Walk a hundred meters into the bush, put a large tree between you and the group and change there.
I regularly cycle to the pool with limited time, and have found that the best option is trunks, but they don't have to be as skimpy as we tend to think of as speedos. Mine (similar to these but a little longer) are like close fitting short shorts, and feel much less exposed than some styles.
Unpadded cycling shorts would also be good. Unpadded because cheap foam padding holds water and good padding is the wrong shape for walking.
I've found the proper term for what I had in mind; they're fairly popular at my pool: Triathlon shorts (or tri shorts). The linked article says they're not for chlorinated pools, but many brands claim at least some chlorine resistance. Triathlon kit can be expensive, but for mainly swimming a cheap brand would do.
Even the fastest-drying shorts will not get completely dry in a reasonable amount of time. If the weather is cold enough, wearing damp shorts for the return hike might be dangerous.
If that's the case, I suggest using a rain kilt for coverage while changing out of the wet swim trunks. Rain kilts are lightweight (2-3 oz), inexpensive ($10-20), small enough to fit in a cargo pocket, and provide more complete coverage than a towel. Be sure to get a colored kilt, as the uncolored ones can be see-through.
You can hedge your bets by wearing swim shorts to the pool, and carrying a rain kilt and change of underwear just in case (or plan on going commando for the hike home).
