It's not horrible deeds that make people distrust Hibbert.
It's his nervous laughter that makes them uneasy.
He can't help it. People make him self conscious.
They stare when he eats his fish heads, even when he's careful not to make slurping noises when he eats the eyeballs.
They whisper about him when he's pulling snails from their shells, even when he made sure the slimy creatures are unconscious.
They move away from him despite his efforts to keep his shawls and cloak clean from the guts and gore that sometimes stick to the fabric.
He doesn't mind so much. It relieves him of the pressure to make small talk. The few times people engage him in conversation, he is taken by surprise and looks at them, wide-eyed, with a twitchy smile and a shrug, and a breathy chuckle.