I think part of what is missing in your question and the answer by flawr is core body temperature.
When you first get cold your body automatically shunts blood away from the surface of your skin to decrease heat loss. As you get colder your core body temperature falls (you could think of it like the cold sinking in, but that is not literally true).
When you stand next to the fire, your outer skin starts to warm, but your core temperature does not respond as quickly. Most of your nerves for temperature are in your skin, not in your muscles and insides. When your skin gets to a certain temperature, your body responds by pushing more blood to the surface, trying to keep you from overheating.
You step away from the fire and your skin is still warm and flushed with blood, you will likely have increased heat loss until your blood vessels in your skin shunt the blood back to your core.
In much the same way that a fire in a fireplace (as opposed to an airtight stove) can result in a net heat loss to the house, a short heating next to a hot fire on a cold day can result in a net heat loss to your body. There are of course many variables, but if you are feeling colder, you probably are.
Per request in comments I looked for research specific to the scenario I described. I have included several references to the basic principals of heat exchange related to blood at the skin level. I did not find anything specific to net heat loss OR gain from a short warming near the fire. I have experienced this perceived colder after a brief warming, and while I believe my hypothesis above is realistic, it seems to be untested under scientific conditions.