That would be pretty frustrating to encounter. Wanting your lord to get glory but being denied it. I understand your opponents fears as Chaos lords are scary bastards and one of if not THE best combat characters in the game. But there are a lot of builds that can take him down easy enough. I have a dwarf friend who has a hardcore wound sponge lord that soaks up everything I throw at him and then swings back with a great weapon and owns me. But if they refuse you its all the better because they then go to the back and cant own your unit leaving you more points in combat res.
If you are the last man standing, you aren't fighting hard enough.