I can actually confirm this from personal experience that when I was 120 pounds heavier, I was way less sensitive to wind-chill and frostbites. I lost last 60 pounds during the winter and I did, in real time, notice how I had to start using thicker glove combos, despite same wind-chill as earlier. You may think that a tiny layer of fat is not going to do much difference, but I can assure you that the difference is very real and very painful.
And contrary to popular misconception, frostbite is unrelated to core heat. You can be sweating in your core, yet you will have stage 1 (or 2) frostbite on your hands. Happens to me all the time.
The reason is that it's very easy to have multiple layers for your core (thermal shirts, long-sleeve shirts, sweater vests, puffer vest and finally a parka). But even with largest gloves, you will be hard-pressed to be able to have 4 layers of gloves, unless your hands are particularly tiny.
Of course, it also depends on the wind-chill. There is a tremendous difference between -40 'F air (zero wind-chill) and 0 'F air (-20 wind-chill). The second one, despite being 20 degrees warmer is way more painful on your hands (particularly if they had too many frostbites) than -40 'F air. May not make much sense on paper, but you sure can feel the difference out there instantly :-)
Probably the single most important factor is that you may have had one too many frostbites (even though it was just reversible stage 1-2). Then your fingers will be extremely sensitive (effectively compounding the absence of fat), while other people walking with you in your group who didn't experience many frostbites, won't notice anything at all.