Server latency puzzle

Server latency puzzle

I have a very strange issue on Centos 5.8/apache 2.2.22/php 5.3.10 box with about 50 vhosts. Most pages are served in < 2 seconds and all sites seem to work fine with webkit browsers (Chrome, Safari). However, pages for some virtual hosts loaded by IE or Firefox have recently started to take 12 seconds to load. More interesting is that these pages when fetched using wget on the server seem to take the same 12 seconds to load – the time seems to be spent between the receipt of the request and the start of the response. This seems to happen only on php OR ssi pages but NOT all php or ssi pages and only on 10 of the 50 vhosts. None of this behavior is observed on a staging server running the same sites on apache 2.2.19.

The only recent updates to the config have been yum updates (including a reboot to pick up the latest kernel). Php diagnostics on a few of the 12-second pages indicates that the pages are generated in less than a few 100 ms – so where are the other 11.+ seconds spent? The system is not using swap space and there is nothing in either the apache or vhost log files.

Anyone have any ideas on what is happening here?

For lack of any better idea, I removed mod_deflate but not using gzip compression seemed to have no effect on the response delay. I also tried explicitly flushing a couple of php pages – also no effect.

My only other ideas are to see if I can come up with the mix of page elements that triggers the problem and/or see if I can figure out how to capture the problem in a strace dump (not so nice on a production server).

Neither prospect looks like much fun 🙁

Comments are closed.