On Mon, Apr 12, 2010 at 09:19:57PM +0200, Cryptronic wrote: > Hi, > the cgroup thing is a really nice function, but I experienced that if > one cgroup gets into an oom situation the whole host system is nearly > unusable. load raises to 300+ only because of the oom killer. > Can the oom killer be configuered to kill the next best process > instead of using the whole oom killer featureset. yes, there is a sysfs attribute called oom_kill_allocating_task which should do what you want, i.e. kill the task currently allocating memory (i.e. the one active when the issue arises) > This problem makes nearly impossible to use memcg, because all guests > on the host do not react while the cg is in oom. > Is there maybe a possible tuning? tunings on the memcg subsystem should be done on mainline, we are just providing the subsystem for Linux-VServer use HTH, Herbert > Values: > > # head /etc/vservers/testcg/cgroup/memory.* > ==> /etc/vservers/testcg/cgroup/memory.limit_in_bytes <== > 536870912 > > ==> /etc/vservers/testcg/cgroup/memory.memsw.limit_in_bytes <== > 1610612736 > > > example dmest output after oom occured: > > [253637.756988] php5-cgi invoked oom-killer: gfp_mask=0xd0, order=0, > oom_adj=0 > [253637.757020] php5-cgi cpuset=testcg mems_allowed=0 > [253637.757049] Pid: 13023, comm: php5-cgi Tainted: G W > 2.6.33.2-vs2.3.0.36.30.4 #1 > [253637.757097] Call Trace: > [253637.757123] [<ffffffff810b385a>] ? dump_header+0x64/0x184 > [253637.757153] [<ffffffff811e4f71>] ? ___ratelimit+0xc5/0xe0 > [253637.757182] [<ffffffff810b39b9>] ? oom_kill_process+0x3f/0x134 > [253637.757211] [<ffffffff810b41a4>] ? mem_cgroup_out_of_memory+0x67/0x7f > [253637.757242] [<ffffffff810e75d0>] ? > res_counter_check_under_limit+0x2b/0x33 > [253637.757274] [<ffffffff810e8317>] ? > __mem_cgroup_try_charge+0x1c4/0x269 > [253637.757327] [<ffffffff810e96d2>] ? > mem_cgroup_try_charge_swapin+0x5b/0xb5 > [253637.757360] [<ffffffff810c9efd>] ? handle_mm_fault+0x5bb/0x8c3 > [253637.757392] [<ffffffff810cfbb9>] ? mmap_region+0x3e8/0x51d > [253637.757424] [<ffffffff810248a7>] ? do_page_fault+0x25f/0x27b > [253637.757457] [<ffffffff81359805>] ? page_fault+0x25/0x30 > [253637.757486] Task in /testcg killed as a result of limit of /testcg > [253637.757537] memory: usage 524164kB, limit 524288kB, failcnt 2682457 > [253637.757566] memory+swap: usage 1565848kB, limit 1572864kB, failcnt > 3718537 > [253637.757596] Mem-Info: > [253637.757617] Node 0 DMA per-cpu: > [253637.757643] CPU 0: hi: 0, btch: 1 usd: 0 > [253637.757669] CPU 1: hi: 0, btch: 1 usd: 0 > [253637.757695] CPU 2: hi: 0, btch: 1 usd: 0 > [253637.757722] CPU 3: hi: 0, btch: 1 usd: 0 > [253637.757748] CPU 4: hi: 0, btch: 1 usd: 0 > [253637.757774] CPU 5: hi: 0, btch: 1 usd: 0 > [253637.757800] CPU 6: hi: 0, btch: 1 usd: 0 > [253637.757826] CPU 7: hi: 0, btch: 1 usd: 0 > [253637.757852] Node 0 DMA32 per-cpu: > [253637.757878] CPU 0: hi: 186, btch: 31 usd: 171 > [253637.757905] CPU 1: hi: 186, btch: 31 usd: 170 > [253637.757931] CPU 2: hi: 186, btch: 31 usd: 57 > [253637.757957] CPU 3: hi: 186, btch: 31 usd: 179 > [253637.757983] CPU 4: hi: 186, btch: 31 usd: 170 > [253637.758009] CPU 5: hi: 186, btch: 31 usd: 166 > [253637.758036] CPU 6: hi: 186, btch: 31 usd: 176 > [253637.758062] CPU 7: hi: 186, btch: 31 usd: 166 > [253637.758088] Node 0 Normal per-cpu: > [253637.758114] CPU 0: hi: 186, btch: 31 usd: 181 > [253637.758140] CPU 1: hi: 186, btch: 31 usd: 153 > [253637.758166] CPU 2: hi: 186, btch: 31 usd: 106 > [253637.758192] CPU 3: hi: 186, btch: 31 usd: 107 > [253637.758219] CPU 4: hi: 186, btch: 31 usd: 163 > [253637.758245] CPU 5: hi: 186, btch: 31 usd: 173 > [253637.758271] CPU 6: hi: 186, btch: 31 usd: 141 > [253637.758297] CPU 7: hi: 186, btch: 31 usd: 116 > [253637.758325] active_anon:455567 inactive_anon:345966 isolated_anon:0 > [253637.758326] active_file:636964 inactive_file:860936 isolated_file:0 > [253637.758327] unevictable:0 dirty:69 writeback:39265 unstable:0 > [253637.758328] free:285568 slab_reclaimable:334023 > slab_unreclaimable:86402 > [253637.758329] mapped:125147 shmem:25748 pagetables:44064 bounce:0 > [253637.758478] Node 0 DMA free:15828kB min:16kB low:20kB high:24kB > active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB > unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15244kB > mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB > slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB > pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB > pages_scanned:0 all_unreclaimable? yes > [253637.758646] lowmem_reserve[]: 0 2991 12081 12081 > [253637.758686] Node 0 DMA32 free:298360kB min:3480kB low:4348kB > high:5220kB active_anon:378100kB inactive_anon:371588kB > active_file:197516kB inactive_file:1184520kB unevictable:0kB > isolated(anon):0kB isolated(file):0kB present:3063520kB mlocked:0kB > dirty:16kB writeback:60916kB mapped:15892kB shmem:17240kB > slab_reclaimable:353484kB slab_unreclaimable:53416kB > kernel_stack:3976kB pagetables:24644kB unstable:0kB bounce:0kB > writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > [253637.758859] lowmem_reserve[]: 0 0 9090 9090 > [253637.758895] Node 0 Normal free:829944kB min:10576kB low:13220kB > high:15864kB active_anon:1444168kB inactive_anon:1010528kB > active_file:2350340kB inactive_file:2259224kB unevictable:0kB > isolated(anon):0kB isolated(file):0kB present:9308160kB mlocked:0kB > dirty:260kB writeback:96144kB mapped:484696kB shmem:85752kB > slab_reclaimable:982608kB slab_unreclaimable:292192kB > kernel_stack:15384kB pagetables:151612kB unstable:0kB bounce:0kB > writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > [253637.759086] lowmem_reserve[]: 0 0 0 0 > [253637.759123] Node 0 DMA: 3*4kB 3*8kB 3*16kB 2*32kB 1*64kB 0*128kB > 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15828kB > [253637.759205] Node 0 DMA32: 18652*4kB 4161*8kB 1880*16kB 1885*32kB > 877*64kB 233*128kB 29*256kB 3*512kB 0*1024kB 1*2048kB 1*4096kB = 299352kB > [253637.759291] Node 0 Normal: 153155*4kB 26609*8kB 92*16kB 0*32kB > 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 831060kB > [253637.759375] 1789422 total pagecache pages > [253637.759399] 265776 pages in swap cache > [253637.759424] Swap cache stats: add 17070423, delete 16804666, find > 11968919/13733735 > [253637.759469] Free swap = 21619328kB > [253637.759493] Total swap = 23446776kB > [253637.804817] 3145712 pages RAM > [253637.804842] 64004 pages reserved > [253637.804865] 1465339 pages shared > [253637.804889] 2061427 pages non-shared > [253637.804913] ------------[ cut here ]------------ > [253637.804944] WARNING: at mm/oom_kill.c:411 __oom_kill_task+0x69/0x10b() > [253637.804974] Hardware name: MS-7522 > [253637.804997] Modules linked in: cls_u32 sch_htb ifb xt_tcpudp > xt_state ipt_REJECT ipt_LOG xt_limit iptable_filter xt_DSCP xt_recent > nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack_irc nf_conntrack_ipv4 > nf_conntrack nf_defrag_ipv4 ip_tables x_tables nfs lockd nfs_acl > auth_rpcgss sunrpc pcspkr evdev tpm_tis tpm tpm_bios i2c_i801 i2c_core > wmi processor button dm_mirror dm_region_hash dm_log dm_snapshot > dm_mod raid1 md_mod sd_mod crc_t10dif ahci libata r8169 mii ehci_hcd > uhci_hcd usbcore nls_base thermal fan thermal_sys [last unloaded: > scsi_wait_scan] > [253637.805342] Pid: 13145, comm: php5-cgi Tainted: G W > 2.6.33.2-vs2.3.0.36.30.4 #1 > [253637.805390] Call Trace: > [253637.805415] [<ffffffff810b3629>] ? __oom_kill_task+0x69/0x10b > [253637.805445] [<ffffffff810b3629>] ? __oom_kill_task+0x69/0x10b > [253637.805476] [<ffffffff8103f4cb>] ? warn_slowpath_common+0x77/0xa3 > [253637.805507] [<ffffffff810b3629>] ? __oom_kill_task+0x69/0x10b > [253637.805536] [<ffffffff810b3a50>] ? oom_kill_process+0xd6/0x134 > [253637.805566] [<ffffffff810b41a4>] ? mem_cgroup_out_of_memory+0x67/0x7f > [253637.805598] [<ffffffff810e75d0>] ? > res_counter_check_under_limit+0x2b/0x33 > [253637.805631] [<ffffffff810e8317>] ? > __mem_cgroup_try_charge+0x1c4/0x269 > [253637.805662] [<ffffffff810e8f90>] ? mem_cgroup_charge_common+0x4d/0x7a > [253637.805695] [<ffffffff810c79e1>] ? __do_fault+0x101/0x457 > [253637.805731] [<ffffffff810c9cfd>] ? handle_mm_fault+0x3bb/0x8c3 > [253637.805764] [<ffffffff810248a7>] ? do_page_fault+0x25f/0x27b > [253637.805796] [<ffffffff81359805>] ? page_fault+0x25/0x30 > [253637.805825] ---[ end trace 2de4787caa53e878 ]--- > [253637.805852] tried to kill an mm-less task 8389 (php5-cgi)! > [253637.841020] 3145712 pages RAM > [253637.841046] 64004 pages reserved > [253637.841099] 1465351 pages shared > [253637.841123] 2061037 pages non-shared > [253637.841150] Memory cgroup out of memory: kill process > php5-cgi(8270:#7041) score 27164 or a child > [253637.841200] Killed process php5-cgi(8270:#7041) vsz:108656kB, > anon-rss:2872kB, file-rss:5912kB > > another question: > > Does vhashify also safe memory when using cgroups? > > best regards & thanks