There are many reasons a trail takes the particular path it does. Probably the most common is historical, as you mention.
Probably the least common is deliberate design. Most trails have evolved from usage patterns before anyone went out to deliberately make a trail.
Trails that are deliberately designed as trails are the results of lots of tradeoffs. There are many criteria that go into deciding where exactly to route a trail. Some are:
- Access to points of interest.
- Limit the slope. People generally don't like steep trails. They are also more difficult to maintain.
If you are designing a handicapped-accessible trail, then you have to be careful about maximum grade (among many other things).
- Ease of cutting the trail.
- Ease of maintaining the trail.
- Trying not to create erosion. You really don't want a trail going straight up a slope, even if the magnitude of the slope is acceptable. Use will make a rut. Then water will run down the rut making a bigger rut. In the end you created a new streambed, not a trail. Then people walk along the sides, making a even bigger mess.
- To avoid particularly sensitive ecological areas.
- To keep things like nearby houses out of sight of the trail users.
- Because conservation restrictions, deed restrictions, requirements of the land donor, or various other legal restrictions don't give you much choice.
- You (or the organization you are designing the trail for) only owns a narrow corridor.
- To avoid certain terrain, like rock jumbles, soggy areas, etc.
Added in response to new question
US national park ... How would I go about routing a public-use trail
Hacking your own trails in a National Park is a really bad idea, not to mention a federal offense that will have consequences.
If you really think there should be a new trail somewhere in a National Park, you very humbly propose it to park management. Most likely they won't be interested. If you do actually convince them, it will be a process. They will have standards that new trails must adhere to, various policies and procedures that must be followed, etc. They will certainly want to choose the route. You might be able to help or have influence over that process, probably depending on what they think of you. They will most likely want to do the construction with their own trail crews. Volunteers they feel comfortable with may be able to contribute, but don't take that for granted.