No, center of gravity is not a issue between a tall and short person. That is because the rest of the body scales too. The center of gravity relative to the body height stays the same.
Imagine a stool. If you make the legs longer but keep the same footprint, then it would be easier to topple. However, if you enlarged the footprint the same amount as the height, then it would not be easier to topple. It's not just the height of the center of gravity that matters, but that height relative to the footprint or base size. A taller person doesn't just have a higher center of gravity, but also a longer stride, and the ability to extend the feet farther out from the point directly below the center of mass. The two effects cancel. A taller person is not less stable than a shorter one.
However, not all things scale linearly. All else being equal, smaller animals use more quiescent power than larger ones relative to their weight. For example, for a mouse it's no big deal to run up a wall. The mouse uses enough power just to stay alive that the additional power of running up a wall is not that much.
For a typical human, though, the extra power of hiking up a steep hill is significant. Let's do the math of a example to see what the numbers look like.
Let's say the human weighs 200 pounds, which is about 900 newtons. Typical power just sitting around at a comfortable temperature is about 50 W. As a sanity check on that, let's consider the often-quoted 2000 cal/day a human needs to eat to balance energy used. That's actually 2000 kcal in physics terms, which is 8.368 MJ. Over 24 hours, that comes out to 97 W. Considering that's a whole day average, about half that just sitting around seems plausible.
So far we have 50 W quiescent power, and 100 W average daily power. Now let's look at what it takes to hike up a hill. Let's say that means going up a 15% grade at a 30 minute per mile pace, just to pick something. That means for every mile horizontally, the hiker went up 792 feet, which is 241 meters. Times the 900 N weight, that is 217 kJ in potential energy. Divided by the 30 minutes it took to expend that energy yields 121 W.
Note that this is the actual output of the body. The body is not perfectly efficient in producing that 121 W. I don't know what the efficiency is, but that 121 W output must mean at least 150 W expenditure, probably more like 200 W. Of course the body still has to keep itself alive, so the total energy expenditure is 250 W.
So, hiking up this example hill means the body is using up 5 times the power compared to resting. It should be no surprise that feels like a lot of work.
The difference for a smaller animal is that the resting power per unit mass is higher. The physics of going up a hill is still the same, although the speed should also be scaled to the body size to be fair. Both these effects combine to make the difference between sitting around and going up hill much less.