Subject: Re: [vserver] opteron server dies with vserver patch.
From:Paweł Sikora <pluto@pld-linux.org>
Date: Mon, 8 Aug 2011 21:45:38 +0200

On Monday 08 of August 2011 20:29:37 Herbert Poetzl wrote:

> could you upload those sections (without line break) and
> also mark the <xx> bytes in the dumps, I'm having a hard
> time to determine them from the screenshot ...
> (seems to be cut off on the right side)

i'll try to catch these stack traces again tomorrow
and adjust monitor to read <xx> markers...

> btw, how long does it take till those traces show up

it varies... from few minutes to few hours.

> and in what way do they affect the system? (total crash,
> single cpu blocked, zombie, nothing)

afaics this partial lock occurs mostly on heavy parallel i/o connected with high cpu
load.

most common scenario:
- 16 farm slaves running on 16 cpu cores cleanup local working directories on /dev/md1
(raid-0, 4 disks).
- 16 farm slaves decompress from common nfs share a .7z installer and install it into
local dir.
- after this i/o peak farm slaves start 100% cpu utilization with minimal i/o (storing
results).
- lock (maybe related to i/o sync? machine has 64GB and caches i/o in buffers until
slave cpu processing).

after lock:
- the ssh is out, but machine responds to ping.
- ipmi console (serial-via-bios-over-lan) is out.
- (weird!) the sysrq on real console handles (only?) reboot sequence (e.g. cannot trace/terminate
tasks).
- hdd leds aren't blinking and cpus stop processing (fans stop flushing hot air from
rack).

so probably it stucks in some kind of i/o deadlock.