It is possible to catch fish with a hook and dental floss, and the flies can just be stored stuck in the brim of a hat or very small tin and then one improvises a rod from a stick.
The advantage of taking dental floss is that it could be used for flossing in addition to catching fish but how does it compare strength wise?
When it comes to survival gear in wilderness scenarios, one needs to consider A) the utility of the equipment, and B) how difficult it is to find or craft the equipment from materials in the field. Using this framework to consider dental floss and fishing line as survival gear, the main uses that are difficult to craft in the field are as general-use low-impact rope, or as fishing line. In both cases tensile strength is important, and resistance to decay is another factor. Lastly, storage / weight is an important consideration.
I presume fishing line easily surpasses floss in tensile strength, but it may not be as easy to store loads of it. I don't have data on this though, perhaps a better answer will come along with some. Anyway, given my experiences with bushcraft and minimalist camping, I would opt for fishing line as a more reliable substitute for rope than floss, and as a more adequate...fishing line...than floss. Think of what you need rope for: lashings of all kinds, providing a supporting surface to hang things from or rest things upon, stitching things in a pitch. In all those cases, I'd rather be working with fishing line than floss, as fishing line feels sturdier and less prone to fraying or decay. Then comes the use of line for actual fishing: safe to say fishing line would work better.
As for dental hygiene, items which serve this function are relatively easy to find or craft in forests (yellow birch twigs is the local example which comes to mind), and in any case this is a low priority need in a survival situation. So, floss doesn't score much points in this comparison.
