Subject: Re: [vserver] Exploding Load in v2.3
From: Cryptronic <mail@cryptronic.de>
Date: Mon, 18 Jan 2010 22:35:38 +0100
Mon, 18 Jan 2010 22:35:38 +0100
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello Nik,

yes indeed i have 74 GB of RAM in that machine. VServers 50+.

I use a shared partition with ext3 or ext4.

remote syslog is not enabled yet, will do that in the next days. And
in syslog i could not find anything.

In messages i found the following which must be appeared in between
the load explosion, but not at the beginning:

before:
Jan 18 21:00:02 host kernel: [20304486.773634] vxW:
[»landscape-sysin«,28639:#9113|9113|9113] did lookup hidden
proc:ffff8812034377f8[#0,4026531914] »/proc/acpi«.

then:
Jan 18 21:11:45 host kernel: [20304486.368940] apache2 invoked
oom-killer: gfp_mask=0x0, order=0,
oomkilladj=0                                                                       
                            
 
 
 

Jan 18 21:11:45 host kernel: [20304486.368944] apache2 cpuset=/
mems_allowed=0-1                                                                   
                                                              
 
 
 

Jan 18 21:11:45 host kernel: [20304486.368947] Pid: 5090, comm:
apache2 Not tainted 2.6.31.7-vs2.3.0.36.27
#1                                                                                 
              
 
 
 

Jan 18 21:11:45 host kernel: [20304486.368949] Call
Trace:                                                                             
                                                                          
 
 
 

Jan 18 21:11:45 host kernel: [20304486.368959]  [<ffffffff810ae4e4>] ?
oom_kill_process+0x8e/0x265                                                        
                                                       
 
 
 

Jan 18 21:11:45 host kernel: [20304486.368962]  [<ffffffff810ae9bf>] ?
select_bad_process+0xd2/0x12f                                                      
                                                       
 
 
 

Jan 18 21:11:45 host kernel: [20304486.368965]  [<ffffffff810aea9d>] ?
__out_of_memory+0x81/0x8e                                                          
                                                       
 
 
 

Jan 18 21:11:45 host kernel: [20304486.368967]  [<ffffffff810aec80>] ?
pagefault_out_of_memory+0x64/0x8c                                                  
                                                       
 
 
 

Jan 18 21:11:45 host kernel: [20304486.368972]  [<ffffffff8102b0ac>] ?
mm_fault_error+0x39/0xdf                                                           
                                                       
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369034]  [<ffffffff8105085a>] ?
do_sigaction+0x86/0x169                                                            
                                                       
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369036]  [<ffffffff8102b3b1>] ?
do_page_fault+0x25f/0x27b                                                          
                                                       
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369041]  [<ffffffff812c2915>] ?
page_fault+0x25/0x30                                                               
                                                       
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369043]
Mem-Info:                                                                          
                                                                               
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369045] Node 0 Normal
per-cpu:                                                                           
                                                                 
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369047] CPU    0: hi:  186,
btch:  31 usd:
75                                                                                 
                                           
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369049] CPU    1: hi:  186,
btch:  31 usd:
14                                                                                 
                                           
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369051] CPU    2: hi:  186,
btch:  31 usd:
169                                                                                
                                            
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369053] CPU    3: hi:  186,
btch:  31 usd:  
0                                                                                  
                                          
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369055] CPU    4: hi:  186,
btch:  31 usd:
52                                                                                 
                                           
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369056] CPU    5: hi:  186,
btch:  31 usd:  
0                                                                                  
                                          
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369058] CPU    6: hi:  186,
btch:  31 usd:
119                                                                                
                                            
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369060] CPU    7: hi:  186,
btch:  31 usd:  
1                                                                                  
                                          
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369061] CPU    8: hi:  186,
btch:  31 usd:
50                                                                                 
                                           
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369063] CPU    9: hi:  186,
btch:  31 usd:
159                                                                                
                                            
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369065] CPU   10: hi:  186,
btch:  31 usd:
85                                                                                 
                                           
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369067] CPU   11: hi:  186,
btch:  31 usd:
184                                                                                
                                            
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369068] CPU   12: hi:  186,
btch:  31 usd:
145                                                                                
                                            
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369070] CPU   13: hi:  186,
btch:  31 usd:
156                                                                                
                                            
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369071] CPU   14: hi:  186,
btch:  31 usd:
74                                                                                 
                                           
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369073] CPU   15: hi:  186,
btch:  31 usd:
183                                                                                
                                            
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369075] Node 1 DMA
per-cpu:                                                                           
                                                                    
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369077] CPU    0: hi:    0,
btch:   1 usd:  
0                                                                                  
                                          
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369079] CPU    1: hi:    0,
btch:   1 usd:  
0                                                                                  
                                          
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369080] CPU    2: hi:    0,
btch:   1 usd:  
0                                                                                  
                                          
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369085] CPU    3: hi:    0,
btch:   1 usd:  
0                                                                                  
                                          
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369086] CPU    4: hi:    0,
btch:   1 usd:  
0                                                                                  
                                          
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369088] CPU    5: hi:    0,
btch:   1 usd:  
0                                                                                  
                                          
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369089] CPU    6: hi:    0,
btch:   1 usd:  
0                                                                                  
                                          
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369091] CPU    7: hi:    0,
btch:   1 usd:  
0                                                                                  
                                          
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369092] CPU    8: hi:    0,
btch:   1 usd:  
0                                                                                  
                                          
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369094] CPU    9: hi:    0,
btch:   1 usd:  
0                                                                                  
                                          
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369095] CPU   10: hi:    0,
btch:   1 usd:  
0                                                                                  
                                          
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369097] CPU   11: hi:    0,
btch:   1 usd:  
0                                                                                  
                                          
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369099] CPU   12: hi:    0,
btch:   1 usd:  
0                                                                                  
                                          
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369100] CPU   13: hi:    0,
btch:   1 usd:  
0                                                                                  
                                          
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369102] CPU   14: hi:    0,
btch:   1 usd:  
0                                                                                  
                                          
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369103] CPU   15: hi:    0,
btch:   1 usd:  
0                                                                                  
                                          
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369104] Node 1 DMA32
per-cpu:                                                                           
                                                                  
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369106] CPU    0: hi:  186,
btch:  31 usd:
165                                                                                
                                            
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369108] CPU    1: hi:  186,
btch:  31 usd:
156                                                                                
                                            
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369110] CPU    2: hi:  186,
btch:  31 usd:
157                                                                                
                                            
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369111] CPU    3: hi:  186,
btch:  31 usd:
174                                                                                
                                            
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369113] CPU    4: hi:  186,
btch:  31 usd:
147                                                                                
                                            
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369114] CPU    5: hi:  186,
btch:  31 usd:
178                                                                                
                                            
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369116] CPU    6: hi:  186,
btch:  31 usd:
57                                                                                 
                                           
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369118] CPU    7: hi:  186,
btch:  31 usd:
180                                                                                
                                            
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369119] CPU    8: hi:  186,
btch:  31 usd:
50                                                                                 
                                           
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369121] CPU    9: hi:  186,
btch:  31 usd:
175                                                                                
                                            
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369122] CPU   10: hi:  186,
btch:  31 usd:
94                                                                                 
                                           
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369124] CPU   11: hi:  186,
btch:  31 usd:
157                                                                                
                                            
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369125] CPU   12: hi:  186,
btch:  31 usd:
17                                                                                 
                                           
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369127] CPU   13: hi:  186,
btch:  31 usd:
182                                                                                
                                            
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369129] CPU   14: hi:  186,
btch:  31 usd:
113                                                                                
                                            
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369130] CPU   15: hi:  186,
btch:  31 usd:
161                                                                                
                                            
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369131] Node 1 Normal
per-cpu:                                                                           
                                                                 
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369133] CPU    0: hi:  186,
btch:  31 usd:
164                                                                                
                                            
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369135] CPU    1: hi:  186,
btch:  31 usd:
172                                                                                
                                            
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369136] CPU    2: hi:  186,
btch:  31 usd:
165                                                                                
                                            
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369138] CPU    3: hi:  186,
btch:  31 usd:
15                                                                                 
                                           
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369139] CPU    4: hi:  186,
btch:  31 usd:
173                                                                                
                                            
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369141] CPU    5: hi:  186,
btch:  31 usd:
60                                                                                 
                                           
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369142] CPU    6: hi:  186,
btch:  31 usd:
183                                                                                
                                            
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369144] CPU    7: hi:  186,
btch:  31 usd:
15                                                                                 
                                           
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369146] CPU    8: hi:  186,
btch:  31 usd:
166                                                                                
                                            
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369147] CPU    9: hi:  186,
btch:  31 usd:
37                                                                                 
                                           
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369149] CPU   10: hi:  186,
btch:  31 usd:
179                                                                                
                                            
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369150] CPU   11: hi:  186,
btch:  31 usd:
103                                                                                
                                            
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369152] CPU   12: hi:  186,
btch:  31 usd:
156                                                                                
                                            
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369153] CPU   13: hi:  186,
btch:  31 usd:
21                                                                                 
                                           
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369155] CPU   14: hi:  186,
btch:  31 usd:
169                                                                                
                                            
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369156] CPU   15: hi:  186,
btch:  31 usd:
40                                                                                 
                                           
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369160] Active_anon:3764691
active_file:3669111
inactive_anon:287488                                                               
                                       
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369161]  inactive_file:3477160
unevictable:2098 dirty:483 writeback:0
unstable:0                                                                         
                 
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369162]  free:139331
slab:6942059 mapped:402654 pagetables:170875
bounce:0                                                                           
                     
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369164] Node 0 Normal
free:42472kB min:17260kB low:21572kB high:25888kB
active_anon:7050920kB inactive_anon:479312kB active_file:7335212kB
inactive_file:6451136kB unevictable:6276k
B present:37232640kB pages_scanned:0 all_unreclaimable?
no                                                                                 
                                                                        
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369169] lowmem_reserve[]: 0 0 0
0                                                                                  
                                                         
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369171] Node 1 DMA free:15648kB
min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB
active_file:0kB inactive_file:0kB unevictable:0kB present:15084kB
pages_scanned:0 all_unr
eclaimable?
no                                                                                 
                                                                                   
                                
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369176] lowmem_reserve[]: 0
3246 36324
36324                                                                              
                                                  
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369178] Node 1 DMA32
free:136268kB min:1540kB low:1924kB high:2308kB active_anon:513180kB
inactive_anon:135772kB active_file:469384kB inactive_file:497316kB
unevictable:0kB present
:3324740kB pages_scanned:0 all_unreclaimable?
no                                                                                 
                                                                                  
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369183] lowmem_reserve[]: 0 0
33077
33077                                                                              
                                                     
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369186] Node 1 Normal
free:362936kB min:15700kB low:19624kB high:23548kB
active_anon:7494664kB inactive_anon:534868kB active_file:6871848kB
inactive_file:6960188kB unevictable:2116
kB present:33871360kB pages_scanned:0 all_unreclaimable?
no                                                                                 
                                                                       
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369191] lowmem_reserve[]: 0 0 0
0                                                                                  
                                                         
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369193] Node 0 Normal: 7756*4kB
917*8kB 1*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB
0*4096kB =
42376kB                                                          
Jan 18 21:11:45 host kernel: [20304486.369200] Node 1 DMA: 0*4kB 2*8kB
1*16kB 2*32kB 1*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB
3*4096kB =
15648kB                                                                  
Jan 18 21:11:45 host kernel: [20304486.369265] Node 1 DMA32: 16357*4kB
8349*8kB 8*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB
1*2048kB 0*4096kB =
136380kB                                                        
Jan 18 21:11:45 host kernel: [20304486.369272] Node 1 Normal:
88607*4kB 567*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB
0*1024kB 0*2048kB 1*4096kB =
363060kB                                                        
Jan 18 21:11:45 host kernel: [20304486.369279] 7328314 total pagecache
pages                                                                              
                                                         
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369280] 76871 pages in swap
cache                                                                              
                                                             
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369282] Swap cache stats: add
5351866, delete 5274995, find
197725297/198123099                                                                
                             
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369284] Free swap  =
136605644kB                                                                        
                                                                    
 
 
 

Jan 18 21:11:45 host kernel: [20304486.369285] Total swap =
140624968kB                                                                        
                                                                    
 
 
 

Jan 18 21:11:46 host kernel: [20304486.781057] 18874368 pages
RAM                                                                                
                                                                  
 
 
 

Jan 18 21:11:46 host kernel: [20304486.781060] 280153 pages
reserved                                                                           
                                                                    
 
 
 

Jan 18 21:11:46 host kernel: [20304486.781061] 6318038 pages
shared                                                                             
                                                                   
 
 
 

Jan 18 21:11:46 host kernel: [20304486.781063] 15270861 pages
non-shared                                                                         
                                                                  
 
 
 


after that: reset happend.
                                                                                   
                                                      
 
 
 



I could not get any guest process causing that issue. In vtop no
abnormal cpu heavy process could be found.

Best regards.

On 01/18/10 22:15, Nikolay Kichukov wrote:
> Hello cryptonic.
>
> Sorry to hear that. Are you really having 74Gigs of RAM on that
> machine? How many virtual machines are there? What file system do
> you use for the guest partitions? Do you have a certain
> guest/process causing this load as it seems CPU on the host is not
> so busy.
>
> Do you have remote syslog enabled? Any clues in the host
> /var/log/messages or /var/log/syslog?
>
> Cheers, -Nik
>
> Cryptronic wrote: Hi list,
>
> we had problems on different kernel versions since vserver patch
> 2.3.
>
> We had the problem, that load runs in a view seconds into great
> dimensions:
>
> top - 21:11:53 up 23 days,  5:55,  2 users,  load average:
> 1092.35, 530.76, 212.88 Tasks: 5614 total,   3 running, 5531
> sleeping,   0 stopped,   0 zombie Cpu(s):  3.6%us,  8.2%sy,
> 0.0%ni, 83.2%id,  4.6%wa,  0.0%hi, 0.3%si,  0.0%st Mem:  74376860k
> total, 73832132k used,   544728k free,  4835704k buffers Swap:
> 140624968k total,  4019584k used, 136605384k free, 24147056k
> cached
>
> As you can see "wa" is not very high.
>
> At a load of 500 Apache and Co stop working and delivering
> webpages.
>
> Load before  it explodes is around 3 - 5.
>
> On shell in host system you can work as at load levels like 0.50,
> but vserver operations aren't possible: vserver XXX stop does not
> take any affect.
>
> dmesg is also empty.
>
> We're using the following versions:
>
> # vserver-info Versions: Kernel: 2.6.31.7-vs2.3.0.36.27 VS-API:
> 0x00020305 util-vserver: 0.30.216-pre2864; Nov 29 2009, 00:39:14
>
> Features: CC: gcc, gcc (Debian 4.3.4-6) 4.3.4 CXX: g++, g++ (Debian
> 4.3.4-6) 4.3.4 CPPFLAGS: '' CFLAGS: '-Wall -g  -O2 -std=c99 -Wall
> -pedantic -W -funit-at-a-time' CXXFLAGS: '-g -O2 -ansi -Wall
> -pedantic -W -fmessage-length=0 -funit-at-a-time' build/host:
> x86_64-pc-linux-gnu/x86_64-pc-linux-gnu Use dietlibc: yes Build C++
> programs: yes Build C99 programs: yes Available APIs:
> v13,net,v21,v22,v23,netv2 ext2fs Source: e2fsprogs syscall(2)
> invocation: alternative vserver(2) syscall#: 236/glibc crypto api:
> nss python bindings: no use library versioning: yes
>
> Paths: prefix: /usr sysconf-Directory: /etc cfg-Directory:
> /etc/vservers initrd-Directory: $(sysconfdir)/init.d
> pkgstate-Directory: /var/run/vservers vserver-Rootdir:
> /var/lib/vservers
>
> At Kernel 2.6.29.6-vs2.3.0.36.14 this also happend. The hardware
> is also not the thing, because this happens on different
> architectures.
>
> To get the server back online, you have to reset the hardware,
> reboot and co not doing anything.
>
> Best regards
>>

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (GNU/Linux)

iEYEARECAAYFAktU1CoACgkQOBdlVlcPuhxdwQCgyR4o0ZEYcfW5TUmlyxYrzzYz
25wAoJT6mReugXt1iYV/ud3VFtI/rYoF
=TZyO
-----END PGP SIGNATURE-----



-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello Nik,

yes indeed i have 74 GB of RAM in that machine. VServers 50+.

I use a shared partition with ext3 or ext4.

remote syslog is not enabled yet, will do that in the next days. And
in syslog i could not find anything.

In messages i found the following which must be appeared in between
the load explosion, but not at the beginning:

before:
Jan 18 21:00:02 host kernel: [20304486.773634] vxW:
[»landscape-sysin«,28639:#9113|9113|9113] did lookup hidden
proc:ffff8812034377f8[#0,4026531914] »/proc/acpi«.

then:
Jan 18 21:11:45 host kernel: [20304486.368940] apache2 invoked
oom-killer: gfp_mask=0x0, order=0,
oomkilladj=0                                                                                                      
 
 

Jan 18 21:11:45 host kernel: [20304486.368944] apache2 cpuset=/
mems_allowed=0-1                                                                                                                                    
 
 

Jan 18 21:11:45 host kernel: [20304486.368947] Pid: 5090, comm:
apache2 Not tainted 2.6.31.7-vs2.3.0.36.27
#1                                                                                                  
 
 

Jan 18 21:11:45 host kernel: [20304486.368949] Call
Trace:                                                                                                                                                          
 
 

Jan 18 21:11:45 host kernel: [20304486.368959]  [<ffffffff810ae4e4>] ?
oom_kill_process+0x8e/0x265                                                                                                                  
 
 

Jan 18 21:11:45 host kernel: [20304486.368962]  [<ffffffff810ae9bf>] ?
select_bad_process+0xd2/0x12f                                                                                                                
 
 

Jan 18 21:11:45 host kernel: [20304486.368965]  [<ffffffff810aea9d>] ?
__out_of_memory+0x81/0x8e                                                                                                                    
 
 

Jan 18 21:11:45 host kernel: [20304486.368967]  [<ffffffff810aec80>] ?
pagefault_out_of_memory+0x64/0x8c                                                                                                            
 
 

Jan 18 21:11:45 host kernel: [20304486.368972]  [<ffffffff8102b0ac>] ?
mm_fault_error+0x39/0xdf                                                                                                                     
 
 

Jan 18 21:11:45 host kernel: [20304486.369034]  [<ffffffff8105085a>] ?
do_sigaction+0x86/0x169                                                                                                                      
 
 

Jan 18 21:11:45 host kernel: [20304486.369036]  [<ffffffff8102b3b1>] ?
do_page_fault+0x25f/0x27b                                                                                                                    
 
 

Jan 18 21:11:45 host kernel: [20304486.369041]  [<ffffffff812c2915>] ?
page_fault+0x25/0x30                                                                                                                         
 
 

Jan 18 21:11:45 host kernel: [20304486.369043]
Mem-Info:                                                                                                                                                            
 
 

Jan 18 21:11:45 host kernel: [20304486.369045] Node 0 Normal
per-cpu:                                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369047] CPU    0: hi:  186,
btch:  31 usd:
75                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369049] CPU    1: hi:  186,
btch:  31 usd:
14                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369051] CPU    2: hi:  186,
btch:  31 usd:
169                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369053] CPU    3: hi:  186,
btch:  31 usd:  
0                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369055] CPU    4: hi:  186,
btch:  31 usd:
52                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369056] CPU    5: hi:  186,
btch:  31 usd:  
0                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369058] CPU    6: hi:  186,
btch:  31 usd:
119                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369060] CPU    7: hi:  186,
btch:  31 usd:  
1                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369061] CPU    8: hi:  186,
btch:  31 usd:
50                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369063] CPU    9: hi:  186,
btch:  31 usd:
159                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369065] CPU   10: hi:  186,
btch:  31 usd:
85                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369067] CPU   11: hi:  186,
btch:  31 usd:
184                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369068] CPU   12: hi:  186,
btch:  31 usd:
145                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369070] CPU   13: hi:  186,
btch:  31 usd:
156                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369071] CPU   14: hi:  186,
btch:  31 usd:
74                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369073] CPU   15: hi:  186,
btch:  31 usd:
183                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369075] Node 1 DMA
per-cpu:                                                                                                                                                  
 
 

Jan 18 21:11:45 host kernel: [20304486.369077] CPU    0: hi:    0,
btch:   1 usd:  
0                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369079] CPU    1: hi:    0,
btch:   1 usd:  
0                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369080] CPU    2: hi:    0,
btch:   1 usd:  
0                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369085] CPU    3: hi:    0,
btch:   1 usd:  
0                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369086] CPU    4: hi:    0,
btch:   1 usd:  
0                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369088] CPU    5: hi:    0,
btch:   1 usd:  
0                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369089] CPU    6: hi:    0,
btch:   1 usd:  
0                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369091] CPU    7: hi:    0,
btch:   1 usd:  
0                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369092] CPU    8: hi:    0,
btch:   1 usd:  
0                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369094] CPU    9: hi:    0,
btch:   1 usd:  
0                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369095] CPU   10: hi:    0,
btch:   1 usd:  
0                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369097] CPU   11: hi:    0,
btch:   1 usd:  
0                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369099] CPU   12: hi:    0,
btch:   1 usd:  
0                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369100] CPU   13: hi:    0,
btch:   1 usd:  
0                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369102] CPU   14: hi:    0,
btch:   1 usd:  
0                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369103] CPU   15: hi:    0,
btch:   1 usd:  
0                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369104] Node 1 DMA32
per-cpu:                                                                                                                                                
 
 

Jan 18 21:11:45 host kernel: [20304486.369106] CPU    0: hi:  186,
btch:  31 usd:
165                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369108] CPU    1: hi:  186,
btch:  31 usd:
156                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369110] CPU    2: hi:  186,
btch:  31 usd:
157                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369111] CPU    3: hi:  186,
btch:  31 usd:
174                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369113] CPU    4: hi:  186,
btch:  31 usd:
147                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369114] CPU    5: hi:  186,
btch:  31 usd:
178                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369116] CPU    6: hi:  186,
btch:  31 usd:
57                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369118] CPU    7: hi:  186,
btch:  31 usd:
180                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369119] CPU    8: hi:  186,
btch:  31 usd:
50                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369121] CPU    9: hi:  186,
btch:  31 usd:
175                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369122] CPU   10: hi:  186,
btch:  31 usd:
94                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369124] CPU   11: hi:  186,
btch:  31 usd:
157                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369125] CPU   12: hi:  186,
btch:  31 usd:
17                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369127] CPU   13: hi:  186,
btch:  31 usd:
182                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369129] CPU   14: hi:  186,
btch:  31 usd:
113                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369130] CPU   15: hi:  186,
btch:  31 usd:
161                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369131] Node 1 Normal
per-cpu:                                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369133] CPU    0: hi:  186,
btch:  31 usd:
164                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369135] CPU    1: hi:  186,
btch:  31 usd:
172                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369136] CPU    2: hi:  186,
btch:  31 usd:
165                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369138] CPU    3: hi:  186,
btch:  31 usd:
15                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369139] CPU    4: hi:  186,
btch:  31 usd:
173                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369141] CPU    5: hi:  186,
btch:  31 usd:
60                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369142] CPU    6: hi:  186,
btch:  31 usd:
183                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369144] CPU    7: hi:  186,
btch:  31 usd:
15                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369146] CPU    8: hi:  186,
btch:  31 usd:
166                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369147] CPU    9: hi:  186,
btch:  31 usd:
37                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369149] CPU   10: hi:  186,
btch:  31 usd:
179                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369150] CPU   11: hi:  186,
btch:  31 usd:
103                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369152] CPU   12: hi:  186,
btch:  31 usd:
156                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369153] CPU   13: hi:  186,
btch:  31 usd:
21                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369155] CPU   14: hi:  186,
btch:  31 usd:
169                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369156] CPU   15: hi:  186,
btch:  31 usd:
40                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369160] Active_anon:3764691
active_file:3669111
inactive_anon:287488                                                                                                         
 
 

Jan 18 21:11:45 host kernel: [20304486.369161]  inactive_file:3477160
unevictable:2098 dirty:483 writeback:0
unstable:0                                                                                             
 
 

Jan 18 21:11:45 host kernel: [20304486.369162]  free:139331
slab:6942059 mapped:402654 pagetables:170875
bounce:0                                                                                                   
 
 

Jan 18 21:11:45 host kernel: [20304486.369164] Node 0 Normal
free:42472kB min:17260kB low:21572kB high:25888kB
active_anon:7050920kB inactive_anon:479312kB active_file:7335212kB
inactive_file:6451136kB unevictable:6276k
B present:37232640kB pages_scanned:0 all_unreclaimable?
no                                                                                                                                                            
 
 

Jan 18 21:11:45 host kernel: [20304486.369169] lowmem_reserve[]: 0 0 0
0                                                                                                                                              
 
 

Jan 18 21:11:45 host kernel: [20304486.369171] Node 1 DMA free:15648kB
min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB
active_file:0kB inactive_file:0kB unevictable:0kB present:15084kB
pages_scanned:0 all_unr
eclaimable?
no                                                                                                                                                                                                        
 
 

Jan 18 21:11:45 host kernel: [20304486.369176] lowmem_reserve[]: 0
3246 36324
36324                                                                                                                                   
 
 

Jan 18 21:11:45 host kernel: [20304486.369178] Node 1 DMA32
free:136268kB min:1540kB low:1924kB high:2308kB active_anon:513180kB
inactive_anon:135772kB active_file:469384kB inactive_file:497316kB
unevictable:0kB present
:3324740kB pages_scanned:0 all_unreclaimable?
no                                                                                                                                                                      
 
 

Jan 18 21:11:45 host kernel: [20304486.369183] lowmem_reserve[]: 0 0
33077
33077                                                                                                                                      
 
 

Jan 18 21:11:45 host kernel: [20304486.369186] Node 1 Normal
free:362936kB min:15700kB low:19624kB high:23548kB
active_anon:7494664kB inactive_anon:534868kB active_file:6871848kB
inactive_file:6960188kB unevictable:2116
kB present:33871360kB pages_scanned:0 all_unreclaimable?
no                                                                                                                                                           
 
 

Jan 18 21:11:45 host kernel: [20304486.369191] lowmem_reserve[]: 0 0 0
0                                                                                                                                              
 
 

Jan 18 21:11:45 host kernel: [20304486.369193] Node 0 Normal: 7756*4kB
917*8kB 1*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB
0*4096kB =
42376kB                                                          
Jan 18 21:11:45 host kernel: [20304486.369200] Node 1 DMA: 0*4kB 2*8kB
1*16kB 2*32kB 1*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB
3*4096kB =
15648kB                                                                  
Jan 18 21:11:45 host kernel: [20304486.369265] Node 1 DMA32: 16357*4kB
8349*8kB 8*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB
1*2048kB 0*4096kB =
136380kB                                                        
Jan 18 21:11:45 host kernel: [20304486.369272] Node 1 Normal:
88607*4kB 567*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB
0*1024kB 0*2048kB 1*4096kB =
363060kB                                                        
Jan 18 21:11:45 host kernel: [20304486.369279] 7328314 total pagecache
pages                                                                                                                                          
 
 

Jan 18 21:11:45 host kernel: [20304486.369280] 76871 pages in swap
cache                                                                                                                                              
 
 

Jan 18 21:11:45 host kernel: [20304486.369282] Swap cache stats: add
5351866, delete 5274995, find
197725297/198123099                                                                                                
 
 

Jan 18 21:11:45 host kernel: [20304486.369284] Free swap  =
136605644kB                                                                                                                                               
 
 

Jan 18 21:11:45 host kernel: [20304486.369285] Total swap =
140624968kB                                                                                                                                               
 
 

Jan 18 21:11:46 host kernel: [20304486.781057] 18874368 pages
RAM                                                                                                                                                     
 
 

Jan 18 21:11:46 host kernel: [20304486.781060] 280153 pages
reserved                                                                                                                                                  
 
 

Jan 18 21:11:46 host kernel: [20304486.781061] 6318038 pages
shared                                                                                                                                                   
 
 

Jan 18 21:11:46 host kernel: [20304486.781063] 15270861 pages
non-shared                                                                                                                                              
 
 


after that: reset happend.
                                                                                                                                            
 
 



I could not get any guest process causing that issue. In vtop no
abnormal cpu heavy process could be found.

Best regards.

On 01/18/10 22:15, Nikolay Kichukov wrote:
> Hello cryptonic.
>
> Sorry to hear that. Are you really having 74Gigs of RAM on that
> machine? How many virtual machines are there? What file system do
> you use for the guest partitions? Do you have a certain
> guest/process causing this load as it seems CPU on the host is not
> so busy.
>
> Do you have remote syslog enabled? Any clues in the host
> /var/log/messages or /var/log/syslog?
>
> Cheers, -Nik
>
> Cryptronic wrote: Hi list,
>
> we had problems on different kernel versions since vserver patch
> 2.3.
>
> We had the problem, that load runs in a view seconds into great
> dimensions:
>
> top - 21:11:53 up 23 days,  5:55,  2 users,  load average:
> 1092.35, 530.76, 212.88 Tasks: 5614 total,   3 running, 5531
> sleeping,   0 stopped,   0 zombie Cpu(s):  3.6%us,  8.2%sy,
> 0.0%ni, 83.2%id,  4.6%wa,  0.0%hi, 0.3%si,  0.0%st Mem:  74376860k
> total, 73832132k used,   544728k free,  4835704k buffers Swap:
> 140624968k total,  4019584k used, 136605384k free, 24147056k
> cached
>
> As you can see "wa" is not very high.
>
> At a load of 500 Apache and Co stop working and delivering
> webpages.
>
> Load before  it explodes is around 3 - 5.
>
> On shell in host system you can work as at load levels like 0.50,
> but vserver operations aren't possible: vserver XXX stop does not
> take any affect.
>
> dmesg is also empty.
>
> We're using the following versions:
>
> # vserver-info Versions: Kernel: 2.6.31.7-vs2.3.0.36.27 VS-API:
> 0x00020305 util-vserver: 0.30.216-pre2864; Nov 29 2009, 00:39:14
>
> Features: CC: gcc, gcc (Debian 4.3.4-6) 4.3.4 CXX: g++, g++ (Debian
> 4.3.4-6) 4.3.4 CPPFLAGS: '' CFLAGS: '-Wall -g  -O2 -std=c99 -Wall
> -pedantic -W -funit-at-a-time' CXXFLAGS: '-g -O2 -ansi -Wall
> -pedantic -W -fmessage-length=0 -funit-at-a-time' build/host:
> x86_64-pc-linux-gnu/x86_64-pc-linux-gnu Use dietlibc: yes Build C++
> programs: yes Build C99 programs: yes Available APIs:
> v13,net,v21,v22,v23,netv2 ext2fs Source: e2fsprogs syscall(2)
> invocation: alternative vserver(2) syscall#: 236/glibc crypto api:
> nss python bindings: no use library versioning: yes
>
> Paths: prefix: /usr sysconf-Directory: /etc cfg-Directory:
> /etc/vservers initrd-Directory: $(sysconfdir)/init.d
> pkgstate-Directory: /var/run/vservers vserver-Rootdir:
> /var/lib/vservers
>
> At Kernel 2.6.29.6-vs2.3.0.36.14 this also happend. The hardware
> is also not the thing, because this happens on different
> architectures.
>
> To get the server back online, you have to reset the hardware,
> reboot and co not doing anything.
>
> Best regards
>>


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (GNU/Linux)

iEYEARECAAYFAktU1CoACgkQOBdlVlcPuhxdwQCgyR4o0ZEYcfW5TUmlyxYrzzYz
25wAoJT6mReugXt1iYV/ud3VFtI/rYoF
=TZyO
-----END PGP SIGNATURE-----