Answering the specific question: why does a shorter but deeper dive leave a lower residual nitrogen limit than a longer shallower dive.
Common gas absorption decompression models such as Buhlmann work on the principle of several gas "compartments" which absorb inert gasses (e.g. nitrogen, helium) at different rates and these represent the speed that gas is absorbed in different tissues. In the case of Buhlmann there's 16 compartments from fast to slow (roughly but not exactly equating to blood absorbing faster than, say, bone).
When you do a deep bounce dive, the fast compartments will on-gas very quickly, but the slower compartments won't. The deeper the dive, the greater the pressure, so there's a greater inert gas pressure which will "fill" the compartments. i.e. the fast compartments "fill" faster.
As the recreational dive planner is for "No Decompression", you cannot allow any of the compartments to "fill", otherwise you'll get gas bubbles when you ascend or will have to hold a decompression stop. With the PADI RDP and all other minimal/no decompression planners, the fast compartments will control the bottom time as they'll have too much inert gas to release during the 9 metre/minute ascent (safety stops aren't included in this calculation).
With a shallower dive, the fast compartments still 'fill' but not as much as with a deep dive as the pressure is lower. In this case they can off-gas during the standard ascent. However there's been a longer bottom time in which case the slower compartments will control the bottom time.
Your surface interval residual nitrogen (inert gas) timings are controlled by the slower compartments. In a shallower dive these are off-gassing on the surface but more slowly than the fast compartments, hence the greater residual nitrogen group.
A deep dive in comparison never allowed the slower compartments to fill, so there's only a small amount of residual nitrogen to off-gas.
Technical divers regularly dive well beyond the limits of the RDP, for example 30 minutes at 40 metres. They're constantly on-gassing during the bottom phase of the dive and need to come up and stop for long durations according to the plan - over 60 minutes of decompression is required if diving to 40 metres on air, during which the diver cannot ascend to the surface without serious risk of DCS!
This can be considerably reduced if nitrox (e.g. 28%) is used for the bottom phase and oxygen is used for decompression (in this example reducing the decompression obligation to less than 20 minutes).
Thus technical divers need to be able to plan the dive (which will include consideration for equipment failures, narcosis, gas selection and requirements), have the right equipment (multiple cylinders containing adequate gas for the plan), and have the skills to dive that plan (e.g. following the plan and holding a decompression stop)
If you're interested in this topic, it's taught to all technical divers by all agencies and is explained very well in books such as
Deco for Divers (Powell, Mark)