As others commented, it depends. But unless you are at camp 4 on Everest or in a similar extreme situation, it won't actually matter much.
If you do the math or let online calculators do it for you (i.e. here) on how much energy it takes to warm water, it's not actually that much. If you go to the extreme and say you down 4l of 1°C water and need to heat it to 36°C body temp, it takes about 140kCal or about half a chocolate bar. Of course, the body will burn a bit more energy to heat up the water but keep in mind, this is an extreme example.
Personal experience: 10 day hike, 20km a day, nights in a tent, temperature between -10°C at night and about 5°C during the day. Had water in a pack in my bag a quite a bit of the tube stuck outside so I was basically always drinking ice cold water. Of course the hot coffee in the morning was more enjoyable than always drinking ice cold water. But I never once felt cold because of cold water or burning more calories even though I severely underestimated how much food I needed (basically had a 800-1000kCal deficit for 10 days). Quite the opposite actualy, as soon as you get moving, you get quite hot and the body will have temperature to spare to heat the water.
So, I'd say no, not dangerous if you otherwise keep well fed, rested, dry and warm (clothes, sleeping bag).