Lots of swim/board shorts come with key keepers in the pockets. Chances are you may even have one in the swimwear you own. It's simply a length of elastic cord stitched into your pocket. If you don't have one, it can easily be added. You attach your keys to the loop using a girth hitch:

[![enter image description here][1]][1]

  [1]: https://i.stack.imgur.com/9a453.jpg