I've been using the same two pair of swimshorts for years now. They dry quick, they're lightweight, they're cheap. Mine are similar to these (with pockets, but not lots), but I got mine at Target.
The only downside is that sometimes the cheap thin string tie bites into my waist on really long summer hikes.