[linuxperf] High load under Apache1.3.3/mod_perl 1.16/Linux 2.2.7 SMP

["Mike Whitaker" <mike@altrion.org>]

Posted to both lists since it's a common configuration for the mod_perl 
folks, and I haven't seen the linuxperf guys look at this specific setup.

Our typical webserver is a dual PII450 with 1G, and split httpd's, typically
300 static to serve the pages and proxy to 80-100 dynamic to serve the
mod_perl adverts. Unneeded modules are diabled and hostname lookups turned
off, as any sensible person would.

I'm aware we're a couple of version behind on Apache and mod_perl - this is
partly because I'm a lazy whatnot, and using the versions identical to those
packaged with Debian 2.1 make for installing servers much faster (especially
by trainee idiots 4000 miles away).

There's typically between one and three mod_perl hits/page on top of the
usual dozen or so inline images, and the adserver is not *that* efficient
since it has to figure out the originating *country* of the request (it
cheats a bit with an IP adress block lookup before it resorts to DNS, but
even so, it is quite compute intensive)

The kernel (2.2.7) has MAX_TASKS upped to 4090, and the
unlock_kernel/lock_kernel around csum_and_copy_from_user() in tcp_do_sendmsg
that Andi Kleen suggested.

Performance is .. interesting. Load on the machine fluctuates between 10 and
120, while the user CPU goes from 15% (80% idle) to 180% (0% idle, machine
*crawling*), about once every minute and a half. vmstat shows the number of
processes in a run state to range from 0 (when load is low) to 30-40, and
the static servers manage a mighty 60-70 peak hits/sec. Without the dynamic
httpd's everything *flies*, but our advertsisers and business guys moan like
crazy (*slightly forced grin*).

Does anyone have a feel for what is going on here, or where I should look to
try and cure this?

strace on the master httpd shows it seems to spend a *lot* of its time
throwing out SIGALRMS (which I think with my sketchy understanding of Apache
is not abnormal), a goodly few SIGUSR1's and large chunks (10-15) of
consecutive fork()'s. I assume this is to try and keep the number of httpd's
between MinSpareServers and MaxSpareServers - does this mean that setting
these two values close together (as some suggest) will cause a lot of fork()
and kill()'ing needlessly?
--
Mike Whitaker /  Tel: +44-1733-766619 | Email: mike@cricket.org
CricInfo Ltd /   Fax: +44-1733-346287 | Emergency: mike@sms.cricket.org
Technical   / Mobile: +44-7971-977375 | Mobile: mike@mobile.cricket.org
Manager    / GSM Fax: +44-7970-219651 | Web: http://www.cricket.org/

-
Linuxperf:    Working list for the Linux Performance tuning site
Archive:      http://mail.nl.linux.org/lists/
Web site:     http://www.nl.linux.org/linuxperf/