Short answer: As soon as it's determined that continuing could run the risk of someone getting into trouble or sustaining an injury.
We had to abandon a cave-surveying expedition this past summer because of heavy snow. The expedition spanned two weekends, on the first weekend I was worried I was going to get sun burned on the approach to the high-altitude cave. Exactly 7 days later we started the approach again in the rain, by the time we got above treeline it started to turn into snow, by the time we got within a 1/2 hour of the cave entrance we were pushing snow halfway up to our knees.
I wasn't the trip leader on this expedition, but when the mountains had suddenly disappeared into a whiteout of heavy snow fall I told the trip leader, "I think someone needs to make a call here..."
These were the concerns I brought up:
Several people in the group we not dressed for the weather. Several
were not wearing waterproof footwear or clothing and were already
soaked and cold. Not exactly how you want to start the day inside a
cold damp cave.
The snow had already accumulated to mid-knee. There was no telling
how deep was it going to be when we resurfaced after spending the
better part of the day underground.
The terrain was treacherous enough when dry, but the wet snow cover
presented an increased danger of slipping, and a level of difficulty
for planning foot falls on the loose talus underneath, especially
while carrying a heavy bag. It also made it more difficult to
navigate the terrain, and presented a greater risk of someone taking a tumble down the
mountain.
All things considered, it was determined that continuing ran a potential risk of hypothermia or injury to one or more members in the group. We turned around at that point and began a slow, wet and slippery descent.
The circumstances are not going to be the same for every situation. You need to consider size, strength, and ability of your group, as well as your level of preparedness. There were a number of people well prepared for the weather on my trip who could have continued, in fact there were three who did continue, set up camp and waited out the storm inside the cave, but they were the only ones who packed in sleeping bags and extra gear.
Ultimately, the decision should rest with the group leader, who has hopefully studied the conditions, is aware of the abilities of the group, and is prepared and trained to respond to any number of emergency scenarios. Sometimes it's wise to make the call even if there's a level of uncertainty that it's necessary to do so. You'll rarely regret playing it safe.