I would recommend taking a small container of rubbing alcohol and some cotton balls, as described for cleaning yourself here. Plain old rubbing alcohol kills everything, and is not sticky like an alcohol-based hand sanitizer. A bottle like this one ought to do you pretty well. It should even fit in your pocket so you don't have to go fishing around through your gear to find it when you need it.
If you want something that's pre-made, I've found a couple of threads on hiking forums that recommend carrying around a small container of baby wipes to clean your hands before handling contacts. I would probably shy away from these due to the residue they tend to leave behind (not to mention the smell!).
Also, as @Russell suggested, you'll probably want to have a small bottle of contact solution to rinse your hands off with afterward-- just in case. A travel size bottle might be nice for that.