What would be some of the most difficult obstacles to overcome in starting such a business?
Frankly, none in particular - by which I mean nothing is really standing out from the rest. Everything will be new to you, from legalities to general know-how (routes etc.) to finding a steady supply of customers, how to handle it when you are ill, and so on.
Or in other words: the first step will be the most difficult one. Go and find out which prerequisites you need to offer your service (do you need to file something with your local city or something like that, do you need to acquire some kind of certificate), fulfill those, and then just start doing it. See how the competition is doing it (but don't bother worrying too much about them, try to have a healthy, good-natured relationship). Whenever you have no customers lined up on a weekend, work on other aspects (map out new routes, work on your website, work on local/regional advertisement and so on).
Start slow (not too many events too soon), easy (trips that are trivial for yourself) and maybe do a few private trips for friends&family first to get some free experience.
As you only want to do it on weekends, I assume you are keeping your regular job, so there should be little risk involved. Obviously, find out what risks are there regarding liabilities (in case of accidents, errors on your part, bad weather, illness and so on), but that should be quite manageable; more of a chore than a difficult obstacle.