In boating, I know that having the right of way is much different than driving, what are the rules and how does this work?
To get the complete answer, you would need to study up on your Navigation Rules, International and Inland, also known as NAVRULES and as the 1972 COLREGS (Collision Regulations). This book is the size of a small novel, and has the force of a treaty within those nations that have agreed to it. NAVRULES describes things beyond just giving way, like maintaining safe speed and lookouts, lighting and sounds, and so on.
The answer already given by @renesis covers Rules 13, 14, and 15, regarding overtaking, meeting, and crossing situations respectively, and the actions which shall be taken by the Give-Way (Rule 16) and Stand-On (Rule 17) vessel.
I wish to add Rule 18, regarding responsibility between vessels, often called the order of precedence, and given the mnemonic "New Reels Catch Fish So Purchase Some":
- N: Not Under Command- a vessel which due to circumstances beyond its control cannot maneuver (i.e. mechanical issues)
- R: Restricted Ability To Maneuver - a vessel which due to its work (i.e. dredging or scuba support) cannot maneuver easily
- C: Constrained by Draft - a vessel which must remain in the deeper part of a channel and cannot maneuver into shallower areas
- F: Fishing - a vessel engaged in commercial fishing or trawling operations
- S: Sailing - a vessel propelled solely by the wind
- P: Power Driven - a vessel propelled by a mechanical device
- S: Sea Plane - an aircraft in the act of landing on or taking off from water
In short, vessels lower on the list give way to vessels higher on the list. Note that most "boats" in the sense of typical pleasure craft would be defined as power driven for purposes of Rule 18.
A quick google search turned up a reasonable answer.
The following looks like good general set of rules, but it may vary depending on where you are, which you didn't say.
Sailboats under sail power only are always the stand-on vessels in crossing and meeting situations, so look out for them when you’re under power. Also, commercial vessels restricted by their draft or by fishing gear, such as nets or trawls, hold privilege over all recreational vessels, including sailboats.
Passing a boat
Your vessel: If you’re following another vessel in a river, narrow canal, or marked channel, you’re the give-way vessel, meaning you have the greater burden of responsibility should anything go wrong when you try to pass. Your vessel, in this case, is also called the burdened vessel.
The other vessel: The vessel you want to pass is the stand-on vessel. It’s privileged and the skipper can deny you passage if she thinks it’s unsafe (or doesn’t like the color of your paint).
Asking permission to pass: You sound two short blasts from your horn, signifying you’d like to pass the skipper on his port (left) side.
Receiving permission to pass: He signals back with two short blasts to say “Okay!”
Permission denied: She blasts the horn five times, signifying there’s danger involved in such a maneuver. If she doesn’t respond at all, consider it five short blasts and don’t attempt to pass.
Your vessel: You’re on a crossing course with another vessel that could result in a collision if neither boat changes course or speed.
The other vessel’s on the right: It’s the privileged or stand-on vessel and you must let it pass in front of you.
The other vessel is on the left: You’re the privileged or stand-on vessel and you must pass in front of the other vessel.
Meeting a boat head-on
Your vessel: You’re meeting another vessel head-on.
Both vessels: You should both steer to the right to such a degree that each can see the other’s intentions to pass safely portside to portside (left to left, for the landlubber).
When all else fails When it seems like no one but you knows or follows the rules, the rules say you must give way to avoid a collision. If you exercise stand-on privilege and an accident results, you’ll be held at least partially responsible.
The short version is that whoever can most easily get out of the way, does.
Others have answered with the legal, official rules, and if you are at all serious about boating, you need to learn these. But if you are out boating and need to figure out what to do quickly, this rule of thumb usually works.
The other thing to remember is that if you are a driver of a small recreational boat, you are always going to be able to get out of the way more easily than a working boat, a bigger vessel or a sailboat, so don't get in their way in the first place.