I regularly cycle to the pool with limited time, and have found that the best option is trunks, but they don't have to be as skimpy as we tend to think of as speedos. Mine (similar to [these](https://www.wiggle.co.uk/speedo-classic-aquashort-1/) but a little longer) are like close fitting short shorts, and feel much less exposed than some styles.

*Unpadded* cycling shorts would also be good. Unpadded because cheap foam padding holds water and good padding is the wrong shape for walking. 

I've found the proper term for what I had in mind; they're fairly popular at my pool: [***Triathlon shorts***](https://completetri.com/triathlon-shorts/) (or *tri shorts*).  The linked article says they're not for chlorinated pools, but many brands claim at least some chlorine resistance.  Triathlon kit can be expensive, but for mainly swimming a cheap brand would do.