-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 01/22/10 05:32, Claus Herwig wrote: >> >> I would be very interested in seeing the events and actual activity of >> the oomkiller because a co-worker and I have been chasing something >> very similar. >> > > This is getting interesting. > > At the moment I try to hunt down some kind of memory leak myself. The > problem started about two weeks ago on one of my vserver guests. Some > days later I got almost the same symptoms with a second vserver guest on > another host system. > > Both host systems are AMD Opterons, kernel x86_64 2.6.20.18 with > vserver-patch 2.2.0.3. Yes, I know this is not really bleeding edge, but > it was working w/o any problem until now. Software besides kernel is > plain debian lenny on host and guests, security updates applied. > > My first impression of the symptoms was almost identical to what > "cryptronic" described: Memory and load on the host are normal for days. > Then suddenly memory usage is exploding, kernel starts swapping. Then > swap is full and system load starts exploding (mainly because of kswapd > going mad). I got a load of 600+ within minutes and wasn't able to do > anything besides a restart. > > To investigate this (on one machine with about a dozen guests) I set rss > limits and the VIRT_MEM flag for the guest. After that I started seeing > that one guest (in contrast to all others) has a steadily growing RSS > usage. This guest is a typical lamp server: apache2, mysql5, lots of > wordpress blogs and an openx installation. > > I restarted this guest several times and found the following: RSS usage > is constant after startup. Then after some (varying) time I get this in > apache error log: >> Inconsistency detected by ld.so: dl-open.c: 260: dl_open_worker: >> Assertion `_dl_debug_initialize (0, args->nsid)->r_state == >> RT_CONSISTENT' failed! > After this error RSS usage steadily grows until the RSS hardlimit is hit > and I got lots of > (12)Cannot allocate memory: fork: Unable to fork new process > in apache error log. > > > I admit, it's quite possible, that my problem is totally unrelated to > yours (no 2.3 vserver patch involved in my setup). > > But on the other hand I think perhaps you're looking in the wrong place > and linux-vserver isn't the culprit in your case. Perhaps we should > start to search for some shiny new apache bug in the latest debian > apache security update? We have the exploding load problem now for more then a year in v2.3. Until now we thought this is a hardware problem, but now there are 2 more hardware types so we wrote to the list. But it could be. We have also debian lenny guests on our systems but not only. Because i cannot say which apache process of which vserver caused the exploding load i coulnd not say whether this is a apache bug oder uv/lv bug. > > > Any comments and ideas appreciated, of course... > Claus > > -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux) iEYEARECAAYFAktZceYACgkQOBdlVlcPuhz4iACgv235IFINzyFnYvbwsxBk6QBf +BMAoLPo+h3OhMw9+GisHG83b4t431oa =TR3b -----END PGP SIGNATURE-----