Special Characters have, in both 40k and fantasy, always been the sort of thing not meant to be brought in every single game. Recently, Games Workshop has changed that approach with their whole "key" characters thing, and making named characters just a part of the core army list. My understanding, from interviews with games designers, is that the new approach of putting named characters in the main army lists is to pander to the majority of people who show up at Games Workshop stores, and from what they've gathered, seem unable to play a game without a named character.
I've never gotten the impression that characters (of the non-named persuasion) were particularly broken. You need characters for their leadership, and you need them to provide defense against casters in the form of the ubiquotous scroll caddy. Sure, some characters are perfectly capable of fighting units on their own, but they tend to be more expensive than the units they're fighting, so that seems reasonable. For example, my Chaos Lord on a Juggernaut is tooled out to take on an entire unit all by himself. However, he's 375 points. He should be able to take on a 20-strong Core or Special unit for the most part. Conversely, though, a single character taking on a unit by himself is rather dangerous. Your average brick of 20 models is going to have a static combat of 5 against a single character (3 ranks, banner, numbers), so if the character doesn't kill five models, he's going to lose combat.
Ultimately, fighter characters are there to bolster the battle line by killing two or three more enemies in a combat and provide superior leadership to the unit's that they're leading. When you consider both sides are likely to have characters like that, it seems very fair to me.
ASK ME HOW TO GET 20% OFF GAMES WORKSHOP PRODUCTS!