Re: [linuxperf] High load under Apache1.3.3/mod_perl 1.16/Linux 2.2.7 SMP

["Mike Whitaker" <mike@altrion.org>]

>> Our typical webserver is a dual PII450 with 1G, and split httpd's, typically
>> 300 static to serve the pages and proxy to 80-100 dynamic to serve the
>> mod_perl adverts.
>
> Gee, how do you do that?  (Total Apache newbie here.)

see http://perl.apache.org/guide/

(take a bow, Stas)

>> There's typically between one and three mod_perl hits/page on top of the
>> usual dozen or so inline images, and the adserver is not *that* efficient
>> since it has to figure out the originating *country* of the request (it
>> cheats a bit with an IP adress block lookup before it resorts to DNS, but
>> even so, it is quite compute intensive)
>
> How often does it have to block to look up the country?  That IP address
> lookup usually accesses a table in RAM, I hope.

Basically on server child startup it grabs the lookup table into RAM (what
the hey, it's only 32K of data) and merrily binary chop its way around each
time it needs an answer (and as an extra optimisation caches answers for the
life of the child). If the table entry is 'search me, guv', it *then* tries
a reverse DNS lookup, and again caches the result. We're working on reducing
the 'search me, guv' entries to a minimum.

>> The kernel (2.2.7) has MAX_TASKS upped to 4090, and the
>> unlock_kernel/lock_kernel around csum_and_copy_from_user() in tcp_do_sendmsg
>> that Andi Kleen suggested.
>>
>> Performance is .. interesting. Load on the machine fluctuates between 10 and
>> 120, while the user CPU goes from 15% (80% idle) to 180% (0% idle, machine
>> *crawling*), about once every minute and a half. vmstat shows the number of
>> processes in a run state to range from 0 (when load is low) to 30-40, and
>> the static servers manage a mighty 60-70 peak hits/sec. Without the dynamic
>> httpd's everything *flies*...
>
> 40 processes in run state makes me wonder if you might not be suffering
> from
> the thundering herd problem.  Try compiling Apache to block in accept()
> rather than flock(), and use a wake-one scheduler patch (e.g. the one by
> Andrea mentioned at http://www.kegel.com/mindcraft_redux.html#wakeone )

Time to get my fingers dirty with 2.2.8, huh? *grin*

>> Does anyone have a feel for what is going on here, or where I should look to
>> try and cure this?

> This sounds like a post that might interest the linux-kernel list.
> Give the scheduler patch a try, then post your new situation to
> linux-kernel, too, if it's still poor.

That was the next solution I was looking at: thanks for the confirmation.
--
Mike Whitaker /  Tel: +44-1733-766619 | Email: mike@cricket.org
CricInfo Ltd /   Fax: +44-1733-346287 | Emergency: mike@sms.cricket.org
Technical   / Mobile: +44-7971-977375 | Mobile: mike@mobile.cricket.org
Manager    / GSM Fax: +44-7970-219651 | Web: http://www.cricket.org/

-
Linuxperf:    Working list for the Linux Performance tuning site
Archive:      http://mail.nl.linux.org/lists/
Web site:     http://www.nl.linux.org/linuxperf/