Wait, do we mean armies that are difficult to play against or irritating to play against?
For me, Dark Eldar are thge single hardest army for my Templars to beat, since the few people who play them usually know what they're doing, the irritating netlisting wannabe-powergamer jerks hear from their associates that they're crap and don't play them, then tell their associates that they're crap, thus assuring that only the dedicated or talented play them. That, and they are, unitwise, a brutal match for my
Templars, with the speed to avoid combat, and the uns to kill more marinea than I can afford.
The most irritating army for me to play against, though, is Chaos, since all but three of the Chaos players I've played more than a few times and the aforementioned annoying people, most likely helped along by the enormous amount of netlists for Chaos armies, and the fact that it's rather easier to build a potent force with the Chaos codex than quite a few others. Though, I don't doubt the balance of power has shifted since I stopped playing.