-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 01/21/10 19:43, Kyle Bader wrote: >> this is the context when every thing is fine, because we until this thread >> we didn't realize that the server swaps. Because shortly before the crash >> our monitoring reported 0 bytes swap used. And at exploding time there >> were 4019584k of swap in use, so why can this happens when ech vserver >> has a maximum of 3 GB rss settings? In the dmesh output there was only on >> process in oom killer. Is there maybe a bug in oomkiller allocating all >> memroy and swap on the host? > > Do you have kernel logs from when oomkiller was triggered? > > I would be very interested in seeing the events and actual activity of > the oomkiller because a co-worker and I have been chasing something > very similar. > Today i found in dmesg another oomkiller action: Jan 22 10:26:40 host kernel: [313237.887684] apache2 invoked oom-killer: gfp_mask=0x0, order=0, oomkilladj=0 Jan 22 10:26:40 host kernel: [313237.887688] apache2 cpuset=/ mems_allowed=0-1 Jan 22 10:26:40 host kernel: [313237.887691] Pid: 15934, comm: apache2 Not tainted 2.6.31.7-vs2.3.0.36.27 #1 Jan 22 10:26:40 host kernel: [313237.887693] Call Trace: Jan 22 10:26:40 host kernel: [313237.887702] [<ffffffff810ae4e4>] ? oom_kill_process+0x8e/0x265 Jan 22 10:26:40 host kernel: [313237.887705] [<ffffffff810ae9bf>] ? select_bad_process+0xd2/0x12f Jan 22 10:26:40 host kernel: [313237.887708] [<ffffffff810aea9d>] ? __out_of_memory+0x81/0x8e Jan 22 10:26:40 host kernel: [313237.887710] [<ffffffff810aec80>] ? pagefault_out_of_memory+0x64/0x8c Jan 22 10:26:40 host kernel: [313237.887715] [<ffffffff8102b0ac>] ? mm_fault_error+0x39/0xdf Jan 22 10:26:40 host kernel: [313237.887720] [<ffffffff812c14af>] ? thread_return+0x3e/0xc7 Jan 22 10:26:40 host kernel: [313237.887723] [<ffffffff8102b3b1>] ? do_page_fault+0x25f/0x27b Jan 22 10:26:40 host kernel: [313237.887727] [<ffffffff812c2915>] ? page_fault+0x25/0x30 Jan 22 10:26:40 host kernel: [313237.887729] Mem-Info: Jan 22 10:26:40 host kernel: [313237.887730] Node 0 Normal per-cpu: Jan 22 10:26:40 host kernel: [313237.887733] CPU 0: hi: 186, btch: 31 usd: 107 Jan 22 10:26:40 host kernel: [313237.887735] CPU 1: hi: 186, btch: 31 usd: 0 Jan 22 10:26:40 host kernel: [313237.887736] CPU 2: hi: 186, btch: 31 usd: 120 Jan 22 10:26:40 host kernel: [313237.887738] CPU 3: hi: 186, btch: 31 usd: 0 Jan 22 10:26:40 host kernel: [313237.887740] CPU 4: hi: 186, btch: 31 usd: 28 Jan 22 10:26:40 host kernel: [313237.887742] CPU 5: hi: 186, btch: 31 usd: 0 Jan 22 10:26:40 host kernel: [313237.887743] CPU 6: hi: 186, btch: 31 usd: 51 Jan 22 10:26:40 host kernel: [313237.887745] CPU 7: hi: 186, btch: 31 usd: 158 Jan 22 10:26:40 host kernel: [313237.887747] CPU 8: hi: 186, btch: 31 usd: 43 Jan 22 10:26:40 host kernel: [313237.887748] CPU 9: hi: 186, btch: 31 usd: 102 Jan 22 10:26:40 host kernel: [313237.887750] CPU 10: hi: 186, btch: 31 usd: 32 Jan 22 10:26:40 host kernel: [313237.887752] CPU 11: hi: 186, btch: 31 usd: 0 Jan 22 10:26:40 host kernel: [313237.887753] CPU 12: hi: 186, btch: 31 usd: 93 Jan 22 10:26:40 host kernel: [313237.887755] CPU 13: hi: 186, btch: 31 usd: 0 Jan 22 10:26:40 host kernel: [313237.887757] CPU 14: hi: 186, btch: 31 usd: 65 Jan 22 10:26:40 host kernel: [313237.887758] CPU 15: hi: 186, btch: 31 usd: 156 Jan 22 10:26:40 host kernel: [313237.887760] Node 1 DMA per-cpu: Jan 22 10:26:40 host kernel: [313237.887762] CPU 0: hi: 0, btch: 1 usd: 0 Jan 22 10:26:40 host kernel: [313237.887764] CPU 1: hi: 0, btch: 1 usd: 0 Jan 22 10:26:40 host kernel: [313237.887766] CPU 2: hi: 0, btch: 1 usd: 0 Jan 22 10:26:40 host kernel: [313237.887767] CPU 3: hi: 0, btch: 1 usd: 0 Jan 22 10:26:40 host kernel: [313237.887769] CPU 4: hi: 0, btch: 1 usd: 0 Jan 22 10:26:40 host kernel: [313237.887770] CPU 5: hi: 0, btch: 1 usd: 0 Jan 22 10:26:40 host kernel: [313237.887772] CPU 6: hi: 0, btch: 1 usd: 0 Jan 22 10:26:40 host kernel: [313237.887774] CPU 7: hi: 0, btch: 1 usd: 0 Jan 22 10:26:40 host kernel: [313237.887775] CPU 8: hi: 0, btch: 1 usd: 0 Jan 22 10:26:40 host kernel: [313237.887777] CPU 9: hi: 0, btch: 1 usd: 0 Jan 22 10:26:40 host kernel: [313237.887779] CPU 10: hi: 0, btch: 1 usd: 0 Jan 22 10:26:40 host kernel: [313237.887780] CPU 11: hi: 0, btch: 1 usd: 0 Jan 22 10:26:40 host kernel: [313237.887782] CPU 12: hi: 0, btch: 1 usd: 0 Jan 22 10:26:40 host kernel: [313237.887783] CPU 13: hi: 0, btch: 1 usd: 0 Jan 22 10:26:40 host kernel: [313237.887785] CPU 14: hi: 0, btch: 1 usd: 0 Jan 22 10:26:40 host kernel: [313237.887787] CPU 15: hi: 0, btch: 1 usd: 0 Jan 22 10:26:40 host kernel: [313237.887788] Node 1 DMA32 per-cpu: Jan 22 10:26:40 host kernel: [313237.887790] CPU 0: hi: 186, btch: 31 usd: 6 Jan 22 10:26:40 host kernel: [313237.887792] CPU 1: hi: 186, btch: 31 usd: 178 Jan 22 10:26:40 host kernel: [313237.887793] CPU 2: hi: 186, btch: 31 usd: 0 Jan 22 10:26:40 host kernel: [313237.887795] CPU 3: hi: 186, btch: 31 usd: 176 Jan 22 10:26:40 host kernel: [313237.887797] CPU 4: hi: 186, btch: 31 usd: 0 Jan 22 10:26:40 host kernel: [313237.887799] CPU 5: hi: 186, btch: 31 usd: 44 Jan 22 10:26:40 host kernel: [313237.887800] CPU 6: hi: 186, btch: 31 usd: 0 Jan 22 10:26:40 host kernel: [313237.887802] CPU 7: hi: 186, btch: 31 usd: 159 Jan 22 10:26:40 host kernel: [313237.887804] CPU 8: hi: 186, btch: 31 usd: 0 Jan 22 10:26:40 host kernel: [313237.887805] CPU 9: hi: 186, btch: 31 usd: 169 Jan 22 10:26:40 host kernel: [313237.887807] CPU 10: hi: 186, btch: 31 usd: 0 Jan 22 10:26:40 host kernel: [313237.887809] CPU 11: hi: 186, btch: 31 usd: 149 Jan 22 10:26:40 host kernel: [313237.887810] CPU 12: hi: 186, btch: 31 usd: 0 Jan 22 10:26:40 host kernel: [313237.887812] CPU 13: hi: 186, btch: 31 usd: 169 Jan 22 10:26:40 host kernel: [313237.887814] CPU 14: hi: 186, btch: 31 usd: 0 Jan 22 10:26:40 host kernel: [313237.887815] CPU 15: hi: 186, btch: 31 usd: 158 Jan 22 10:26:40 host kernel: [313237.887817] Node 1 Normal per-cpu: Jan 22 10:26:40 host kernel: [313237.887819] CPU 0: hi: 186, btch: 31 usd: 165 Jan 22 10:26:40 host kernel: [313237.887820] CPU 1: hi: 186, btch: 31 usd: 14 Jan 22 10:26:40 host kernel: [313237.887822] CPU 2: hi: 186, btch: 31 usd: 0 Jan 22 10:26:40 host kernel: [313237.887824] CPU 3: hi: 186, btch: 31 usd: 25 Jan 22 10:26:40 host kernel: [313237.887825] CPU 4: hi: 186, btch: 31 usd: 62 Jan 22 10:26:40 host kernel: [313237.887827] CPU 5: hi: 186, btch: 31 usd: 134 Jan 22 10:26:40 host kernel: [313237.887829] CPU 6: hi: 186, btch: 31 usd: 130 Jan 22 10:26:40 host kernel: [313237.887830] CPU 7: hi: 186, btch: 31 usd: 71 Jan 22 10:26:40 host kernel: [313237.887832] CPU 8: hi: 186, btch: 31 usd: 67 Jan 22 10:26:40 host kernel: [313237.887834] CPU 9: hi: 186, btch: 31 usd: 156 Jan 22 10:26:40 host kernel: [313237.887835] CPU 10: hi: 186, btch: 31 usd: 5 Jan 22 10:26:40 host kernel: [313237.887837] CPU 11: hi: 186, btch: 31 usd: 39 Jan 22 10:26:40 host kernel: [313237.887838] CPU 12: hi: 186, btch: 31 usd: 72 Jan 22 10:26:40 host kernel: [313237.887840] CPU 13: hi: 186, btch: 31 usd: 40 Jan 22 10:26:40 host kernel: [313237.887842] CPU 14: hi: 186, btch: 31 usd: 34 Jan 22 10:26:40 host kernel: [313237.887843] CPU 15: hi: 186, btch: 31 usd: 170 Jan 22 10:26:40 host kernel: [313237.887847] Active_anon:3619871 active_file:5826474 inactive_anon:232634 Jan 22 10:26:40 host kernel: [313237.887848] inactive_file:7193754 unevictable:2086 dirty:5508 writeback:0 unstable:0 Jan 22 10:26:40 host kernel: [313237.887849] free:127275 slab:1380361 mapped:408195 pagetables:95354 bounce:0 Jan 22 10:26:40 host kernel: [313237.887851] Node 0 Normal free:80724kB min:17260kB low:21572kB high:25888kB active_anon:8610340kB inactive_anon:487940kB active_file:10031720kB inactive_file:14376252kB unevictable:6308kB present:37232640kB pages_scanned:0 all_unreclaimable? no Jan 22 10:26:40 host kernel: [313237.887857] lowmem_reserve[]: 0 0 0 0 Jan 22 10:26:40 host kernel: [313237.887860] Node 1 DMA free:15648kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB present:15084kB pages_scanned:0 all_unreclaimable? yes Jan 22 10:26:40 host kernel: [313237.887864] lowmem_reserve[]: 0 3246 36324 36324 Jan 22 10:26:40 host kernel: [313237.887867] Node 1 DMA32 free:224184kB min:1540kB low:1924kB high:2308kB active_anon:229184kB inactive_anon:101872kB active_file:757164kB inactive_file:1256576kB unevictable:1204kB present:3324740kB pages_scanned:0 all_unreclaimable? no Jan 22 10:26:40 host kernel: [313237.887872] lowmem_reserve[]: 0 0 33077 33077 Jan 22 10:26:40 host kernel: [313237.887875] Node 1 Normal free:188544kB min:15700kB low:19624kB high:23548kB active_anon:5639960kB inactive_anon:340724kB active_file:12517012kB inactive_file:13142188kB unevictable:832kB present:33871360kB pages_scanned:0 all_unreclaimable? no Jan 22 10:26:40 host kernel: [313237.887880] lowmem_reserve[]: 0 0 0 0 Jan 22 10:26:40 host kernel: [313237.887883] Node 0 Normal: 8306*4kB 4309*8kB 431*16kB 36*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 79840kB Jan 22 10:26:40 host kernel: [313237.887890] Node 1 DMA: 0*4kB 2*8kB 1*16kB 2*32kB 1*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15648kB Jan 22 10:26:40 host kernel: [313237.887896] Node 1 DMA32: 15048*4kB 16319*8kB 1804*16kB 15*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 224184kB Jan 22 10:26:40 host kernel: [313237.887903] Node 1 Normal: 31271*4kB 6963*8kB 205*16kB 12*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 188548kB Jan 22 10:26:40 host kernel: [313237.887910] 13180489 total pagecache pages Jan 22 10:26:40 host kernel: [313237.887911] 43164 pages in swap cache Jan 22 10:26:40 host kernel: [313237.887913] Swap cache stats: add 510763, delete 467599, find 16375590/16392114 Jan 22 10:26:40 host kernel: [313237.887915] Free swap = 139269360kB Jan 22 10:26:40 host kernel: [313237.887916] Total swap = 140624968kB Jan 22 10:26:40 host kernel: [313238.176346] 18874368 pages RAM Jan 22 10:26:40 host kernel: [313238.176348] 280153 pages reserved Jan 22 10:26:40 host kernel: [313238.176350] 8324253 pages shared Jan 22 10:26:40 host kernel: [313238.176351] 11735562 pages non-shared Jan 22 10:26:40 host kernel: [313238.176354] Out of memory: kill process apache2(23962:#9046) score 201733 or a child Jan 22 10:26:40 host kernel: [313238.192825] Killed process apache2(24250:#9046) but after that: # vps aux | grep 15934 www-data 15934 9046 guest 0.0 0.0 277148 58968 ? S Jan21 0:00 /usr/sbin/apache2 -k start the process is still found. Maybe this belongs to that problem? # cat /proc/virtual/9046/limit Limit current min/max soft/hard hits PROC: 29 0/ 45 -1/ - -1 0 VM: 789300 0/ 956157 -1/ - -1 0 VML: 0 0/ 0 -1/ - -1 0 RSS: 136595 0/ 153600 51200/ 153600 7 ANON: 69651 0/ 77840 -1/ - -1 0 RMAP: 66944 0/ 80463 -1/ - -1 0 FILES: 389 0/ 611 -1/ - -1 0 OFD: 381 0/ 424 -1/ - -1 0 LOCKS: 4 0/ 9 -1/ - -1 0 SOCK: 11 0/ 14 -1/ - -1 0 MSGQ: 0 0/ 0 -1/ - -1 0 SHM: 0 0/ 0 -1/ - -1 0 SEMA: 2 0/ 2 -1/ - -1 0 SEMS: 2 0/ 2 -1/ - -1 0 DENT: 25806 0/ 25823 -1/ - -1 0 best regars Oliver -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux) iEYEARECAAYFAktZcLoACgkQOBdlVlcPuhxTpQCgvXq314KWIB55qaRxgrEhOgoE uYoAnRleme2XzUxDyUcZoY4FEkWDCgEo =5pw6 -----END PGP SIGNATURE-----