The term 'rifle' comes from the fact that the barrel of a rifle has spiral grooves (called rifling) cut into the inside. When the bullet is fired it is forced into these grooves which in turn cause it to spin as it travels down the barrel. This spin around its long axis creates a gyroscopic effect which stabilises it in flight dramatically improving its accuracy and the consistency of its trajectory.
This axial stabilization also means that the bullet can be made with a more aerodynamically efficient shape (typically a cone or cylinder) as it is much less prone to tumbling end over end in flight.
A consequence of this is that a rifle bullet is a tight fit in the barrel and so very little of the gas pressure generated by the propellant can get past it compared to a shotgun and so typically a rifle requires a rather heavier barrel than a shotgun as it must contain greater gas pressures even though a shotgun shell may have a greater mass of propellant.
A shotgun is designed to produce a spread of shot over short to moderate ranges which makes it easier to hit small fast moving targets, such as small game and especially birds in flight. the limitation is that its stopping power and accuracy decrease quickly with range.
In a military context shotguns have the advantage that they have high stopping power at very short ranges, can hit a target without careful aim* and have a very versatile selection of rounds for different purposes. They are also particular useful for breeching unarmored doors either with buckshot, solid slugs or specialist breeching rounds there are also specialist rounds for firing gas and pyrotechnic munitions through light doors and walls as well as for shattering windows (relatively) safely.
*In response to comments I will clarify this point by adding that shotguns are well suited to quickly acquiring and hitting fast moving targets of opportunity both because of their spread pattern and general handling characteristics. Here 'careful aim' was perhaps the wrong choice of words but the point is that they work well for reflexive type shooting styles both in the context of game shooting (eg birds flying from cover) and certain military contexts. This is especially the case over moderate ranges in close terrain (eg dense forest or outdoor urban terrain) shotguns may be a very effective weapon as an initial counter to ambushes or unexpected contact with the enemy at short range.
Solid slugs in shotguns are normally primarily intended as anti-material rounds for use against light vehicles etc at fairly close range but they can also be used to improve effective range and accuracy against personnel.
Pump action shotguns also have the advantage that a single round can be loaded and fired very quickly from an unloaded weapon. For all of these reasons shotguns are often used as primary or additional weapons by at least one member of units specializing in close quarters, urban or hostage rescue.
In contrast rifles are accurate and effective at much greater ranges. Although shot rounds for rifles do exist they are pretty rare.