> > I would be very interested in seeing the events and actual activity of > the oomkiller because a co-worker and I have been chasing something > very similar. > This is getting interesting. At the moment I try to hunt down some kind of memory leak myself. The problem started about two weeks ago on one of my vserver guests. Some days later I got almost the same symptoms with a second vserver guest on another host system. Both host systems are AMD Opterons, kernel x86_64 2.6.20.18 with vserver-patch 2.2.0.3. Yes, I know this is not really bleeding edge, but it was working w/o any problem until now. Software besides kernel is plain debian lenny on host and guests, security updates applied. My first impression of the symptoms was almost identical to what "cryptronic" described: Memory and load on the host are normal for days. Then suddenly memory usage is exploding, kernel starts swapping. Then swap is full and system load starts exploding (mainly because of kswapd going mad). I got a load of 600+ within minutes and wasn't able to do anything besides a restart. To investigate this (on one machine with about a dozen guests) I set rss limits and the VIRT_MEM flag for the guest. After that I started seeing that one guest (in contrast to all others) has a steadily growing RSS usage. This guest is a typical lamp server: apache2, mysql5, lots of wordpress blogs and an openx installation. I restarted this guest several times and found the following: RSS usage is constant after startup. Then after some (varying) time I get this in apache error log: > Inconsistency detected by ld.so: dl-open.c: 260: dl_open_worker: > Assertion `_dl_debug_initialize (0, args->nsid)->r_state == > RT_CONSISTENT' failed! After this error RSS usage steadily grows until the RSS hardlimit is hit and I got lots of (12)Cannot allocate memory: fork: Unable to fork new process in apache error log. I admit, it's quite possible, that my problem is totally unrelated to yours (no 2.3 vserver patch involved in my setup). But on the other hand I think perhaps you're looking in the wrong place and linux-vserver isn't the culprit in your case. Perhaps we should start to search for some shiny new apache bug in the latest debian apache security update? Any comments and ideas appreciated, of course... Claus -- CHECON EDV-Consulting und Redaktion Claus Herwig * Barer Straße 70 * 80799 München +49 89 27826981 * Fax 27826982 * c.herwig@checon.de