People perish from falling from 0 meters (just standing, you faint and the back of the head hits the concrete)*. Others can jump from roofs and continue running (parkour guys).
Therefore, I think this is a question of confidence. Cars are among the biggest killers, but we do not generally wear protective equipment when crossing the street. We know that if we are careful, the danger it tolerable and not worth putting on knee pads and spine support. But a drunk man, or a reckless kid, at the middle of the street, are in danger.
As a child I have climbed up to about 10 meters up trees - for fun and to shake down walnuts. The latter includes jumping onto the branch one is standing, or reaching far with a staff in one hand, to hit the nuts. All the time being confident that this is no riskier than crossing the street.
So, as with other dangerous sports, I would advise not to overcome one's fear. If you are afraid of doing something - then maybe it is objectively dangerous. Go train on lower difficulty activities until the fear goes away on it's own. Then it is safe to, very carefully, thy it.
For example have a beer while sitting on a low branch and chatting with friends. Climb a tree and climb down at another spot. Find a low branch, but one you cannot reach, and jump, grabbing a hand hold, then walk with hands, to see how long can you hang should the need arise (and discover is it difficult to hang on a thick branch). When you are comfortable with a drink in hand on the low branch, you will be comfortable when using both hands on a higher spot.
Lastly, I don't think this is a good question. My answer is subjective and there are no others as of now. The comments under the question doesn't even begin to address safety equipment. It's kind of "Should I have unprotected sex, if I am confident the other person is healthy?" "Should I quit my job if I'm unsure if I'll find as good one?" "Is a bicycle without brakes safe?" (IMHO it's absolutely not, but many people ride them fixies).
* - can't find a reference for that, but on page 17 here is documented a fatality from less than 6 feet.