Xenophobia, while illogical, is a natural response. We naturally fear what we don't understand as a survival instinct. Fear triggers the fight/flight response, something built into our genes. The psychology of human fear is quite intriguing. No amount of reason or intelligence can overcome it because it is one of the few purely animalistic traits we have left.
At the time of the Great Crusade, the Eldar were mostly benign to humanity.(Most were too self-absorbed to care about humans.) But they could not be trusted. They were not intentially deceitful but often did the opposite of what they said. This could be due to the complexity of their language. But that's just more for humans to not understand, thus fear. And the Orks are certainly something to fear.
"Fetch me another plaything. This one seems to have broken." -Urien Rakarth
Space Marine- "This planet is ours, witch."
Eldar- "No ... This planet is theirs ..."