No, human smell is only potentially harmful to small fawns, a doe may abandon her infant fawn if it smells of human. At least this is what we're told as kids when we're hiking a trail and encounter a baby fawn. Their defence mechanism is to lay perfectly still, and they will, even if you were to sit down and pet them, they won't twitch. When the mother returns, she may not recognize her fawn if it smells different, and the fawn will be in trouble if mommy takes off and they're too small to keep up, in more trouble still if the mother full out rejects the fawn.
You aren't doing any harm by walking on game trails.
Related: How can I recognize a game trail?