Hi Herbert, >> [1462583.323969] Call Trace: >> [1462583.329191] [<ffffffff8105cfbc>] try_to_wake_up+0x3c/0x410 >> [1462583.340642] [<ffffffff8105d3e5>] wake_up_process+0x15/0x20 >> [1462583.352091] [<ffffffff812710e0>] freeary+0x1e0/0x260 >> [1462583.362503] [<ffffffff812721b1>] T.623+0x71/0xf0 >> [1462583.372223] [<ffffffff81169685>] ? vfs_write+0x125/0x190 >> [1462583.383326] [<ffffffff81272299>] sys_semctl+0x69/0xa0 >> [1462583.393911] [<ffffffff8100bf82>] system_call_fastpath+0x16/0x1b >> Result of "addr2line -e vmlinux $ADDRESS": >> ffffffff8105cfbc: >> sched.c:0 >> ffffffff8105d3e5: >> ??:0 >> ffffffff812710e0: >> sem.c:0 >> ffffffff812721b1: >> sem.c:0 >> ffffffff81169685: >> ??:0 >> ffffffff81272299: >> ??:0 >> ffffffff8100bf82: >> signal.c:0 > your kernel (or at least the vmlinux) file seems to be > missing the necessary debug information (which isn't > that unusual, as most folks compile the kernel without, > which of course makes debugging the kernel a little > tricker :) If it helps, I can build one with debug enabled. You must me only give a hint which config options I have to enable. >> And the same with RIP: >> [1462583.444912] RIP [<ffffffff8104ecaa>] task_rq_lock+0x4a/0xa0 >> [1462583.456545] RSP<ffff88041e259dc8> >> [1462583.464160] ---[ end trace e26d734810b28493 ]--- >> Result: >> ffffffff8104ecaa: >> sched.c:0 >>> might be fixed in .41 or might just be less likely to >>> happen, after all, 16 days is not something you consider >>> easily reproduceable ... >> Not shure, but my other server with .41 runs since 38 days >> without problems and with a apache installed. > it might be interesting to find what exactly triggers > this and maybe test with some more extreme apache > setups, like for example having a lot more threads or > handling high load (web test or similar) On my systems this happens only on webservers under heavy load (around 2360 virtualhosts enabled). I have only 2 servers with heavy apache load. Other webservers only with less than 25 processes runnig aren't affected and they are working well. >> I think you are right. >> I have running more than 10 servers with kernel 2.6.28.[4-7] > I presume this should be 2.6.38.x ... Right, was a typo. thanks, Urban