The forces are really trying to pull the ends of the bow forward. The bow string is not elastic and so does not store energy. The end of the bow may snap back and hit you uncomfortably, but it's not going to whip back at amazing speeds. The arm will basically just let go, and the stem and other arm will get flung by the stores energy. However. You should have a good grip on the stem of the bow, so although it may be surprising, it shouldn't go too far as the bow isn't very heavy and thus you don't have a lot of inertia to overcome.
I think a bowstring snapping would be more dangerous because it would be like a whip, and may hit you in a very inconvenient place such as your face and eye, and may try to rip skin of your fingers as it's ripped through them.