I would say there is no point in walking briskly. With a heavy backpack, it's a no-no for me.
I have observed and have struggled with the same problem when I started off towards some serious trekking with genuinely elevated/steep climbs with a heavy haversack on my back. The sack that I usually carry weighs about 18 kg.
People who are advising you to walk slowly are not precisely correct either. I believe when it's about trekking, the term slow has lot of variants.
For example, what is slow for me, would be extremely slow or probably out of rhythm for you. The point is, everybody has his/her own rhythm of walking (especially while ascending). So, if you ever feel like you are unusually panting for breath, it means that either you are carrying more weight than you normally do, or you are walking faster than you normally do (i.e. walking out of the rhythm that suits you, or you have developed over the years).
I have seen people who don't walk briskly, but they go miles without stopping. Now that's rhythm! And on the contrary, I have also seen people who literally run through an ascend and then rest for quite some time. That is exertion IMHO.
It's an art to maintain the rhythm and keep yourself fit to walk the entire day. Walking in rhythm (neither too fast, nor too slow) over ascends and meadows and plateaus, I have trekked for 22 hrs, without resting for more than 20 min at each break, with around 2 hrs in resting altogether.
When trekking within a group:
It is important for a group to be in pack during the entire trek/hike. We simply can't expect every other guy/girl to walk at your pace. There will be rolling stones (I meant, people who walk a little slower than what suits your walking style).
In that matter, you shouldn't just go ahead, find a shade/tree to rest by and wait for them to come: this isn't the way to trek within a group IMHO. I have observed this humongous amount of times that people who have developed the habit (ability is not the word) to walk briskly often do the same thing.
Rather, if I am a guy who can walk at a milder pace, I can accompany almost every single fellow in the group, often shifting my gears just a bit to get along with the guy/girl marching ahead of the group and ask him/her to slow down if he/she is going too fast, and without much of rest needed, can simply walk a little slow to get to the guy/girl trailing behind the group, having tough time and try to make him/her feel better by deviating his/her mind from the exhaustion that he/she is feeling, help him/her boost up his/her morale to carry on a little further.
To achieve this, I need to have a rhythm that is neither too fast, nor too slow. Someone who walks briskly/faster won't be able to do that task over the longer formats of the treks.