You most definitely want to be selective when selecting rope for a sailboat's running rigging. I'll group sailboats into two categories: racing and non-racing. Racing sailboat is simply any boat where it participates in a race, not any particular construction.
For racing sailboats, you will want to stick as close to the manufacturer's specified lines, and your class rules likely also require it. The details of that are beyond the scope of this answer.
For all other sailboats, what is most important is the rope material. For size, within a millimeter or so is usually fine, though you'll want to err on too small rather than too large so that the rope doesn't bind in through any blocks or winches. For dinghies and small boats, control lines are usually sized large to be easy to grip and hold on to, and as a result don't come anywhere close to their yield strength, so don't worry if you're a tiny bit smaller than the stock line.
For material, your hardware store options were nylon and polypropylene. Nylon is a no-go for running rigging. It is stretchy and loses strength when wet.
Polypropylene is acceptable but isn't the best. It's not very durable and will have to get replaced more often and you'll have to inspect it more regularly.
The materials that remain are basically polyester and a bunch of newish high-tech fibers. Unless you're in a world-class race, there really is no point in getting the other fibers unless you want to throw more money into the water than you would be normally.
Polyester is what you'll want. Virtually all sailboats not owned by millionaires are equipped with polyester running rigging. For a sunfish, pretty much any double-braided polyester line in the right size will do the job just fine.
This page goes into additional detail. (West marine is a boating retailer, but I have no association with them)
So in short: go to a specialty boating retailer.