He's a xenophobe for the same reason people are racist: it's easier.
If you have a bad experience with another ethnicity, it is extremely easy to paint that entire race as a-holes. You do not have to do any critical thinking and you can move on.
On the other hand, understanding that every individual is different just makes life harder. What? You mean I actually have to get to know people before judging them? Argh, but that takes so much effort and I would actually have to...try!
In the 40k galaxy, it's simply easier to be a xenophobe. Of course there are alien races humanity could work with; ones where it would be mutually beneficial to ally and battle Chaos together. But there are countless more that would sooner grind humanity beneath their boots/tentacles/whatever.
As the Great Crusade happened, did the Imperium have the time and resources to figure out which species was which? In my opinion, no, they didn't. Harsh and brutal but that's the galaxy they lived in.