Basically, the law on river swimming in England and Wales is a mess. Below the tidal limit swimming is unambiguously legal, provided you access the water from public land. In fresh water, the true answer is that nobody knows.
This is from a submission by the Outdoor Swimming Society to the National Assembly of Wales, and it's about as authoritative as you're going to get:
The laws governing rights of access to (and restrictions on the use of) inland water are immensely complex and piecemeal, being derived from a combination of very old case law, miscellaneous statutory provisions and local byelaws.
Perhaps more importantly, there is a great deal of uncertainty about
what the law actually permits, even where legal advice is obtained.
Much of the case law is contradictory and unclear as to its scope. There is uncertainty as to whether laws relating to commercial uses apply to recreational uses, and whether laws relating to certain types of access to inland water (for example, rights of navigation) extend to other types of access (for example, swimming).
See also Douglas Caffyn, The Right of Navigation on Nontidal
Rivers and the Common Law, 2004, which comes to a similar conclusion.
In practical terms it seems to boil down to local use and practice. Land owners can allow permissive access at their discretion. On the Dart, where I swim, the situation is good and most of the best spots are accessible.
As for conflict with anglers, we should surely apply a bit of courtesy and commonsense. They have often paid for their fishing, and can be an important contributor to the local economy. As wild swimming groups are pressing for improvements to the law, it would seem best not to antagonise this large and well-organised lobby. So whenever possible it makes sense to avoid the times and places where angling is popular.
The solution, of course, would be legislation similar to the Land Reform
(Scotland) Act 2003, which liberalised access to wild land and water in Scotland. In general, all parties agree that it has been a striking success.