If your water source is clean and assuming you are a relatively clean person in a fairly clean environment (not working in sanitation), You can probably get away with never cleaning it and not get sick. Tap water usually has some chlorine in it to keep basic bacteria and virus levels low. So refilling it with this water will keep the general levels low inside the bottle.
Contamination will primarily come from you -- your hands and mouth. Your body is constantly fighting off bacteria and viruses and the levels you transfer to the bottle will be quite low and easy for your body to kill. However if bacteria left unexposed to the cleansing tap water (like trapped under a piece of food or debris) they can multiply. In these cases the most common bacterias will alter the flavor of the water, however your body will easily fight them off.
Unless you are sick or very unclean (wiping the lip of the bottle with dirty hands), the amount of harmful animals transfered to the bottle will be very small.
Allowing the bottle surfaces to completely dry will kill a large range of bacteria, and rinsing/washing occasionally will also remove debris where they can breed.
An occasional rinse with vinegar will help destroy both viruses and bacteria on many of the hidden surfaces that the chlorinated tap water isn't strong enough to penetrate completely. Vinegar is also non-toxic to drink.
For example I live in the tropics where bacteria and viruses thrive in the heat and humidity. In our case the water is chlorinated, but the water is filtered through a charcoal filter before being placed into the bottle to reduce the chlorine levels (the bottle itself does not get the benefit of the chlorine to clean it). With daily use (and drying, but not washing) it can take several months before a water bottle starts to taste off and a vinegar rinse removes it. We also store water in water bottles that we do not put in our mouths, but pour directly from them. In this case, they are on their second year of never being washed and are fine.