There are a few things that could be contributing to the presence of bees and/or wasps and things that mimic them as they all have different behaviours and different means to get rid of/deter. Note that if you are allergic to bees, you probably aren't allergic to wasps as they are completely unrelated. The answer also depends on where in the world you are.
First off - are you able to tell which is which?
Bees are fuzzy and typically yellow/black in less obvious stripes with not so brilliant colors, and a not-so-obvious waist. There are only a couple of types of bee (honey and bumble) in most Western urban areas. Bumblebees are large and quite obvious, slow flying, very hairy and (mostly) harmless. Honey bees fly faster and are the typical bee shape and appearance.
Wasps tend to have very defined stripes, bright colors a very narrow waist (like an ant, which they are related to) and are largely hairless. Wasps come in a range of different species, with different "risk" profiles. The classic yellow and black wasp is could be a hornet or a Vespula species, such as the German, common wasp, or yellow jacket. Other colored wasps could be paper wasps (tend to have long hanging legs when flying), and a bunch of other species, which are less common or wide-spread.
There are also mimics - these are completely harmless and just look like bees/wasps to scare off predators. They are actually a type of hoverfly and will not harm you. They are easiest to tell when not flying - their wings will stick out to the side and the head is a fly head, quite different to a bee.
There is a comparison table here, with good images of each and some basic information
For all 3 types of insect, it is best just to let them do their thing, if you can not avoid them. Waving arms around does not scare them off, and may provoke defensive behaviour in wasps.
Bees and bumblebees are typically harmless unless provoked (i.e. crushed) or you are near their hive. If you are near the hive (less than say 30 m/100 ft) then occasionally you might get chased by a guard bee, but this is relatively rare. A bee might fly by close to you if you are wearing bright colors, particularly shades of blue, or have a sweet scented perfume. They will not typically hang around and search you intensively as they are primarily looking for nectar.
Bee hives can be difficult to locate - they are often in hollow trunks of trees or inside walls, rarely outside, if you are seeing a lot of bees, you are probably near the hive or a reliable food source (flowers).
Wasps on the other hand are searching for prey and nectar. They have issues with personal space though and will fly around you at close range, even crawling into openings in clothing. Most wasps are carnivorous and will take any insects they can find. They will hover around and search quite extensively from a fairly short range, and occasionally land on you. Do not swat at a wasp - they will sting. They are also attracted to bright colors and sweet scents.
Wasp nests are varied according to the type of wasp - they can be underground, in hollow trees, in walls or roof-spaces, under the eaves of buildings, hanging from branches and can be easy to identify if you can see it. I've seen them as small as a golf-ball (paper wasp nest starting out) and heard of them as big as a car (german wasp). If you can identify the nest, this will tell you about the type of wasp and your relative risk. German wasps and hornets tend to be very aggressive, especially in defense of the nest, while paper wasps are less so and will only sting if feeling trapped.
There is no one-stop-shop for avoiding all of these different species. The general advice is to stay away from hives/nests, stay away from food sources (and water too - they have to drink also), avoid wearing bright colors, avoid sweet scents and having potential food sources around, avoid sudden movements when they fly by (hard with anxiety, I know). One solution might be to get a small portable tent with a sealable door. Another might be to wear long-sleeved clothing and a hat with a veil.
If you are concerned about allergic consequences - consult your doctor or allergist and get tested and carry the appropriate medication for your condition.