On Mon, Aug 08, 2011 at 09:45:38PM +0200, Pawe?? Sikora wrote: > On Monday 08 of August 2011 20:29:37 Herbert Poetzl wrote: >> could you upload those sections (without line break) and >> also mark the <xx> bytes in the dumps, I'm having a hard >> time to determine them from the screenshot ... >> (seems to be cut off on the right side) > i'll try to catch these stack traces again tomorrow > and adjust monitor to read <xx> markers... >> btw, how long does it take till those traces show up > it varies... from few minutes to few hours. well, that's actually a good thing, i.e. we can easily enable more debugging info/output and analyze the actual problem >> and in what way do they affect the system? (total crash, >> single cpu blocked, zombie, nothing) > afaics this partial lock occurs mostly on heavy parallel i/o > connected with high cpu load. > most common scenario: > - 16 farm slaves running on 16 cpu cores cleanup local working > directories on /dev/md1 (raid-0, 4 disks). > - 16 farm slaves decompress from common nfs share a .7z > installer and install it into local dir. > - after this i/o peak farm slaves start 100% cpu utilization > with minimal i/o (storing results). > - lock (maybe related to i/o sync? machine has 64GB and caches > i/o in buffers until slave cpu processing). > after lock: > - the ssh is out, but machine responds to ping. > - ipmi console (serial-via-bios-over-lan) is out. > - (weird!) the sysrq on real console handles (only?) > reboot sequence (e.g. cannot trace/terminate tasks). try with something like 'echo 9 >/proc/sysrq-trigger' and 'echo l >/proc/sysrq-trigger' before anything happens (that should also test that the ipmi console is working) once the problem arises, use 'echo l ..' and 'echo w ..' or send the equivalent (magic sysrq l/w) via console HTH, Herbert > - hdd leds aren't blinking and cpus stop processing > (fans stop flushing hot air from rack). > so probably it stucks in some kind of i/o deadlock.