Just to make sure it doesn't completely contradict my experience, I have made a quick google search and came up with exactly one incident between a human and an ibex - a dog was involved.
In my experience there are two types of encounters: Chance encounters of "really" wild ibex and encounters with ibexes accustomed to humans.
The accustomed kind is typical in highly frequented areas and is highly intimidating: These animals are huge and if you have ever seen two males go at each other, you will have a healthy share of fear mixed into the awe when seeing one up close. I never had any of these act aggressively, but as with any wild animal accustomed to humans trouble is possible. As long as they aren't immediately close (>20m) I keep an I on them, but proceed as usual. On the occasion it decides to approach, I avoid any small movements or loud noises. Not that this is necessarily a problem, I have seen a group of tourist scream from enthusiasm when an ibex was approx 30m away and it didn't care in the slightest.
I can't imagine the "really" wild kind to be a problem at all in terms of aggressive actions. I say imagine as I am not a biologist having studied their nature, but have a limited set of experiences. In all of them, they either run away or peculiarly move away but keep relatively close (seeing distance) and seem to observe you. I never felt threatened, but there is a real danger in this situation: Rockfall. Whenever there are ibexes around, be extremely wary of falling rocks. They are sure footed, but they go anywhere including sections with loose rock. So when one is directly above you, get out of the vertical line if possible and if not, be attentive to take shelter (move close to the wall) in case rock is coming towards you.
In your case, I would have made a break too: Enjoy a midday brake with a good view of a magnificent creature. If you needed to go on, I would have done so slowly, obviously monitoring how they react.