The Star Trek ships have an edge in range and speed (not to mention being able to FTL without going through hell) and would probably be able to outrun/outmaneuver most, if not all, imperial ships. However, their ground combat abilities are sorely lacking; one could easily picture a single squad of space marines working through the entire crew of the Enterprise without any serious difficulty. So for the ST side to emerge victorious would rely on them adopting an extremely paranoid and conservative attitude, relying on long range stand-off combat tactics and hit-and-run raids. However, the 40k ships are noted for being extremely resilient; there's only so much damage that can be done to them at long range with ST:TNG weaponry.
I think the real problem is that the crews of the various ST ships do not have sufficiently developed paranoia; entire series of episodes begin with essentially the following dialog:
"Captain, we've found an unidentified object. Its huge!"
"Well, lets get close, drop shields, and scan it careful like."
"Sir, something's happening...."
<CUE PROTAGONIST OF THE WEEK COMPROMISING SHIP VIA PROXIMITY>
The first time a Trek ship makes that error against a ship with a space marine contingent is the last one.