Not if he refuses the challenge, furthermore the unit will lose any benefit for having him until the end of that combat after which if he survives he will go back to the position he was in before the challenge. If you have a champion in the unit as well as the lord you can accept the challenge with him instead in which case the rest of the unit will still benefit from the lord.