Logically yes. Becuase it is the horses that fear it. From a fluff perspective, instintcually. Just because the armor the Riders wear makes them immune to flaming doesnt mean the horses dont fear it.
..how can you..?I guarantee that the GW ruling (Always being RAW not RAI) would be that the unit is immune to fire and therefore immune to the fear that fire causes.