Cotton kills - but only when not in water. Whilst in water, cotton will not have any thermal effect. However, the material itself can easily wear you down through drag. You might be able to swim short distances, but longer ones - depending on your fitness - can slow you down considerably through drag. Elite swimmers will wear suits with soft and flowing curves, and no extraneous material. Some even go so far as to shave their body to reduce drag their hair might have. I don't know if that's effective or not, but when races are won by a thousandths of a second, perhaps it's worth it.
Clothes that are worn are not made with elite swimming in mind, and so any material - cotton or not - which doesn't easily allow water to pass through it or by it will tend to drag the swimmer, leading to early fatigue. If one were to wear tight-fitting clothes, which allows water to pass around it more easily, that could reduce that lag; but if the clothes were too tight, that can be restrictive on the muscles needed for motion or breathing.
In addition, cotton does bind to water, and holds tight to it. This is the reason we advocate against wearing cotton in cold weather, because water that gets into the fabric will have a tendency to cling to it, making the wearer colder. In the water, again, there isn't much of a thermal effect because water is in contact with the body regardless of the cotton; but the fact remains that cotton binds with water. When you are swimming, you're dragging all that water along creating additional drag. So, wearing a loose-fitting rayon shirt, for example, will have a tendency to drag the swimmer. But a similarly-fitting cotton shirt will have more of a drag effect because it holds onto the water.
Try swimming with a jacket on, and you'll feel the extra weight you're lugging around when you swim. This is a reason we call for removal of as much clothing as possible when there is a need to swim some distance. If you tread or are using a life-saving device, the clothing won't matter - and will come in handy once rescued.