Subject: Re: [vserver] Exploding Load in v2.3
From: Claus Herwig <lists@checon.de>
Date: Fri, 22 Jan 2010 05:32:57 +0100

> 
> I would be very interested in seeing the events and actual activity of
> the oomkiller because a co-worker and I have been chasing something
> very similar.
> 

This is getting interesting.

At the moment I try to hunt down some kind of memory leak myself. The
problem started about two weeks ago on one of my vserver guests. Some
days later I got almost the same symptoms with a second vserver guest on
another host system.

Both host systems are AMD Opterons, kernel x86_64 2.6.20.18 with
vserver-patch 2.2.0.3. Yes, I know this is not really bleeding edge, but
it was working w/o any problem until now. Software besides kernel is
plain debian lenny on host and guests, security updates applied.

My first impression of the symptoms was almost identical to what
"cryptronic" described: Memory and load on the host are normal for days.
Then suddenly memory usage is exploding, kernel starts swapping. Then
swap is full and system load starts exploding (mainly because of kswapd
going mad). I got a load of 600+ within minutes and wasn't able to do
anything besides a restart.

To investigate this (on one machine with about a dozen guests) I set rss
limits and the VIRT_MEM flag for the guest. After that I started seeing
that one guest (in contrast to all others) has a steadily growing RSS
usage. This guest is a typical lamp server: apache2, mysql5, lots of
wordpress blogs and an openx installation.

I restarted this guest several times and found the following: RSS usage
is constant after startup. Then after some (varying) time I get this in
apache error log:
> Inconsistency detected by ld.so: dl-open.c: 260: dl_open_worker:
> Assertion `_dl_debug_initialize (0, args->nsid)->r_state ==
> RT_CONSISTENT' failed!
After this error RSS usage steadily grows until the RSS hardlimit is hit
and I got lots of
(12)Cannot allocate memory: fork: Unable to fork new process
in apache error log.


I admit, it's quite possible, that my problem is totally unrelated to
yours (no 2.3 vserver patch involved in my setup).

But on the other hand I think perhaps you're looking in the wrong place
and linux-vserver isn't the culprit in your case. Perhaps we should
start to search for some shiny new apache bug in the latest debian
apache security update?


Any comments and ideas appreciated, of course...
  Claus


-- 
CHECON   EDV-Consulting und Redaktion
         Claus Herwig * Barer Straße 70 * 80799 München
         +49 89 27826981 * Fax 27826982 * c.herwig@checon.de