Yeah, I'm starting to realise this as well which is really disappointing.
You're only just starting to realize that "Forget the promise of progress and understanding, for in the grim darkness of the far future there is only war. There is no peace amongst the stars, only an eternity of carnage and slaughter, and the laughter of thirsting gods." is not the tagline to a hopeful story of success and goodness? 40k has never tried to hide its darkness, it wears it very proudly, I don't know how you could've been expecting anything else.
indeed, you correctly state "humanity", for me that is on both sides of the fence, not just one, if you remember the conversation Talos had with Abaddon in soul hunter, that was very refreshing for me, from that discussion it seemed on both sides of the fence humanity is losing the fight, this is not the perception I have as of late, it seems there is an overcompensation for the failbaddon perception.
There is no humanity on the other side of the fence. Abaddon isn't human and his victories, and those of Chaos and in general, are just as much defeats for humanity as the military losses the Imperium suffers, if not more so.
The problem I have is that there is no balance on the IOM. I would like to see some more interesting initiatives from the IOM side towards chaos, not just sent the GK and the IG.
Then look at the Gaunts Ghosts or Macharius novels. Cain also regularly triumphs over the forces of evil. Further most Space Marine novels end with them winning the battle, even if Abaddon still manages to advance his ultimate goal. The problem your having is that M999.41 is the eve of the 13th Black Crusade and is therefore the Imperium's darkest hour, they can't be winning in their darkest hour.
That is why the Tau are my favorite faction. Screw this "everything just gets worse and worse" stuff.
You haven't been reading the most recent Tau fluff have you ;) The more exposure the Tau get to the rest of the galaxy the darker they become. Because that's what 40k is. Its dark and its grim and its ultimately hopeless. Its not the bright shining promose of Star Trek, that's why i love it. That's what makes the IoM heroic, their only redeeming feature in my mind, their completely suicidal refusal to give up, even though they recognize how ultimately useless it all is.