I don't think the specifics of this question can be meaningfully answered without knowing more. However, the general reasoning behind whom/when to call if people overshoot their expected arrival time can be analyzed and is valuable because it can be a matter of life and death.
(Note: my answer isn't about the plan you should have had, it's about what to do with what you do know at the trailhead)
what was the nature of the hike and dangers along the way, what was their experience level and how much variability could you expect from them? All very fit or all/some beginning hikers? People with known medical challenges?
how long was the hike expected to be and what did the trail reviews quote as expected times?
How many hikers were there? 2 could be killed/injured by a bear, out of 5
someone would have made it out to raise the alarm.
Was 10:45 really unreasonable/how long would the hike be? i.e. being 5 hour late on a 12-hour-best-expectations on easy terrain is not good. But it should be a call to emergency services much earlier on a 2 hr hike on known-dangerous mountain terrain. Or in the case of someone on the sea, say kayaking. Hours can very much matter, but that depends on the circumstances.
Weather. We're in summer right now. But in mid-winter or during a heat wave, I'd err on the side of caution.
Was any/none of the route likely to have cell reception? Did anyone carry a PLB?
Light conditions vs lateness. 10:45 is about 90 minutes past sunset now, with deep dark happening earlier in forests and valleys. Little moon. In winter, it means they've been putzing around for 6 hours without light. Did they have flashlights? Is the trail forgiving for hiking in low light conditions? A casual beachside with a full moon is very different from hiking down technical mountain terrain without moon under forest cover.
Last, but quite important. How frequented is this trail, how easy is it to get lost from it? If lots of people go up and down the trail - you've seen them coming out - and if there is no easy way to stray, chances are that if your friends had been injured someone would already have raised the alarm.
My sense? If you can't find any good explanations for them being late and 5 hours seems dangerous, call 911 (emergency service number for US and Canada, others will vary) and work from there. If in doubt, you can still call 9 11, they'll put you in touch with your local Search and Rescue team, who will then walk you through the decisions. For example, I've called 911 to report large debris on TransCanada Highway #1. I didn't expect they would solve the problem, only that they would route me to the appropriate highway maintenance service to report something that could have caused an accident.
Before you call, figure out the 30 second explanation for why you are calling them, where you are, where your friends are and who might be able to help. That minimizes the load on the hotline's availability for other emergencies. If in reasonable doubt, calling early - but with clear information - also allows you to shunt off the decision making to more qualified people if you are just providing pickup services to friends without being all that knowledgeable about the back country. In that case, it's really your friends who need to explain themselves to emergency services, not you.
If you are over-worrying, but are acting in good faith, you will not be penalized (I know our local SAR team has been extremely resistant to calls to bill rescuees because they figure it will make people call too late).
I can't help but think you all didn't seem to have a very good plan though it I also confess I might not either - you've made me think with your question.
I would also ask some hard questions of your friends on why they were so late - they really put you on the spot to make an emergency call or not and if that was just to enjoy the smell of flowers, not acceptable.
p.s. also, specific to your circumstances, I note that you are also asking questions about SPOT beacons. Keep in mind that you can always loan a PLB to friends if they go on a hike and lack one. Especially if you are the pickup. As long as you get it back by your next hike that has little downside.
p.p.s Specifically to answer this question's situation from additional info/comments:
hike was way harder : if you had gotten patched through to rangers/park services they'd quickly do the math from your expected time, the particular trail at hand, season/conditions and your buddies' experience/fitness level. Conversely, if there had been real risk they'd have known it.
If you had cell reception, you could have checked expected trail completion time. Or you could have called someone with better internet to check that out.
I'd have called round 10:30-11. I don't think emergency services would get upset- too many people die outdoors in BC/WA for that. Why call? More than overdue time, the night has now fallen, your friends are not experienced and they have no lights. Safest, in this season, would be to stop somewhere and wait out sunrise - 6 hrs away. But people may very well try to walk out and get lost or injured. Unprepared hikers being stuck overnight in good weather should be considered a small-e emergency: CALL RESCUE - they'll talk you down if you're overreacting. Bad weather? Big-E Emergency.
p.p.p.s. 911 vs park ranger?
Quoting my Pacific Crest Trail link above, it has this to say:
Search and rescue (SAR) is generally organized through the county sheriff. Figure out what county your loved one is in or was last known to be in and call that sheriff. National Parks are the exception. They often have their own SAR teams. Again, if it’s an emergency, calling 911 is often the right course of action. 911 Dispatchers can pass you to the right people.
Thing to keep in mind is that PCT is a 4000 km long trail. Calling your local 9-1-1 because someone is overdue on somewhere far away on PCT isn't optimal, since local 9-1-1 will not immediately know whom to contact at PCT for that segment. That's why they suggest contacting rangers if possible. Local area and you don't know the contact info? That's what 9-1-1 is for. Can't figure it out? 9-1-1 again.