Remove SQL_CALC_FOUND_ROWS from queries

Hi,
We have a site, that we want to optimise as well as possible (400.000+ posts), and it seems like the same queries without SQL_CALC_FOUND_ROWS can execute 100x faster than the ones with it, but all the queries we see have this SQL_CALC_FOUND_ROWS added to it. I know it’s needed for pagination, but do we really need it for all the other queries? I can’t seem to find a way in the query loop to disable it.
Ben