To appease the (correctly) security minded: Just don't forget the second helmet ;)
This depends on the specific situation in the crag:
Is there potential rockfall from above the routes?
If there is nothing above the routes (cliffs, flat above, ...), then that's not a problem. If there is steepish terrain above, possibly with a path or animals up there, then it is definitely a risk.
Are the routes themselves crumbly?
Is the belayer protected somehow?
E.g. below an overhang or protruding (massive) rock feature.
Top-roping? Given in your case.
Belayer has lots of freedom to move, e.g. can get away from the wall and/or line vertically down from the climber.
In many scenarios for single-pitch routes it is generally a judgment call whether to wear a helmet or not. Again, it is always recommended to do so, but it's not like suicide if you don't, if the factors above are positive and the crag is very used. Having said that, I still didn't answer the question.
In general the belayer is "more important". Imagine a "mild" accident of a rock causing a bleeding wound and concussion: The climber will fall, can then be lowered and treated (even if unconscious, that's possible, while shit). The belayer will probably let go of the rope and cause the climber to fall -> you have a concussed belayer and a likely severely injured climber.
When in doubt, don't climb without the helmet - if you still climb, give preference to the belayer.