Subject: Kernel oops with NULL pointer dereference when using cgroup hardlimiting
From: Cryptronic <mail@cryptronic.de>
Date: Sun, 07 Feb 2010 16:24:03 +0100

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

i have running a server with a 2.6.31.11-vs2.3.0.36.28 kernel.

When using cgoups as descriped in
http://linux-vserver.org/util-vserver:Cgroups#cgroup_and_CFS_based_CPU_hard_limiting_that_replaces_sched_hard
after a view minutes i get a kernel null pointer:

[  300.166119] BUG: unable to handle kernel NULL pointer dereference
at
0000000000000038                                                                   
                                              
 
 
[  300.174083] IP: [<ffffffff8103668b>]
set_next_entity+0xa/0x56                                                           
                                                                                   
       
 
 
[  300.179918] PGD
0                                                                                  
                                                                                   
                            
 
 
[  300.182018] Oops: 0000 [#1]
SMP                                                                                
                                                                                   
                
 
 
[  300.185383] last sysfs file:
/sys/devices/virtual/net/lo/operstate                                              
                                                                                   
               
 
 
[  300.191590] CPU
14                                                                                 
                                                                                   
                            
 
 
[  300.193775] Modules linked in: ifb xt_tcpudp xt_state ipt_REJECT
ipt_LOG xt_limit iptable_filter xt_DSCP xt_recent nf_nat_ftp nf_nat
nf_conntrack_ftp nf_conntrack_irc nf_conntrack_ipv4 nf_conntrack
nf_defrag_ipv4 ip]
[  300.252780] Pid: 65, comm: events/14 Not tainted
2.6.31.11-vs2.3.0.36.28-beng
#1                                                                                 
                      
 
 
[  300.261525] RIP: 0010:[<ffffffff8103668b>]  [<ffffffff8103668b>]
set_next_entity+0xa/0x56                                                           
                                                               
 
 
[  300.269807] RSP: 0018:ffff881205e63d50  EFLAGS:
00010046                                                                           
                                                                                
 
 
[  300.275160] RAX: 0000000000000000 RBX: ffff8811f5054680 RCX:
0000000000000000                                                                   
                                                                   
 
 
[  300.282333] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
ffff8811f5054680                                                                   
                                                                   
 
 
[  300.289506] RBP: 0000000000000000 R08: 0000000000000000 R09:
0000000000000000                                                                   
                                                                   
 
 
[  300.296681] R10: 119026601aff243a R11: ffffc90001c10010 R12:
ffff8811f5054680                                                                   
                                                                   
 
 
[  300.303853] R13: ffffc90001c15708 R14: 0000000000000000 R15:
ffffffff810d8248                                                                   
                                                                   
 
 
[  300.311026] FS:  0000000000000000(0000) GS:ffffc90001c00000(0000)
knlGS:0000000000000000                                                             
                                                              
 
 
[  300.319166] CS:  0010 DS: 0018 ES: 0018 CR0:
000000008005003b                                                                   
                                                                                   
 
 
[  300.324952] CR2: 0000000000000038 CR3: 0000000001001000 CR4:
00000000000006e0                                                                   
                                                                   
 
 
[  300.332126] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000                                                                   
                                                                   
 
 
[  300.339300] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400                                                                   
                                                                   
 
 
[  300.346472] Process events/14 (pid: 65, threadinfo
ffff881205e62000, task
ffff881205e61520)                                                                  
                                                      
 
 
[  300.354871]
Stack:                                                                             
                                                                                   
                                
 
 
[  300.356931]  ffff8811f5054680 0000000000000000 ffffc90001c12d00
ffffffff81037474                                                                   
                                                                
 
 
[  300.364347] <0> ffffffff812d29d0 ffffc90001c12d00 0000000000000000
ffffffff81033138                                                                   
                                                             
 
 
[  300.372314] <0> ffffc90001c12d00 ffffc90001c12d00 000000000000000e
ffffffff812c13d4                                                                   
                                                             
 
 
[  300.380521] Call
Trace:                                                                             
                                                                                   
                           
 
 
[  300.383020]  [<ffffffff81037474>] ?
pick_next_task_fair+0x85/0xa8                                                      
                                                                                   
        
 
 
[  300.389156]  [<ffffffff81033138>] ?
pick_next_task+0x31/0x40                                                           
                                                                                   
        
 
 
[  300.394861]  [<ffffffff812c13d4>] ?
schedule+0x58a/0x767                                                               
                                                                                   
        
 
 
[  300.400218]  [<ffffffff8104f5da>] ?
lock_timer_base+0x26/0x4c                                                          
                                                                                   
        
 
 
[  300.406006]  [<ffffffff8104effa>] ?
__round_jiffies_relative+0x12/0x17                                                 
                                                                                   
        
 
 
[  300.412577]  [<ffffffff810d8248>] ?
cache_reap+0x0/0x20f                                                               
                                                                                   
        
 
 
[  300.417931]  [<ffffffff8105634a>] ?
worker_thread+0xbb/0x20f                                                           
                                                                                   
        
 
 
[  300.423634]  [<ffffffff8105a456>] ?
autoremove_wake_function+0x0/0x2e                                                  
                                                                                   
        
 
 
[  300.430121]  [<ffffffff8105628f>] ?
worker_thread+0x0/0x20f                                                            
                                                                                   
        
 
 
[  300.435735]  [<ffffffff8105a10c>] ?
kthread+0x8b/0x93                                                                  
                                                                                   
        
 
 
[  300.440831]  [<ffffffff8100cb2a>] ?
child_rip+0xa/0x20                                                                 
                                                                                   
        
 
 
[  300.446012]  [<ffffffff8105a081>] ?
kthread+0x0/0x93                                                                   
                                                                                   
        
 
 
[  300.451019]  [<ffffffff8100cb20>] ? child_rip+0x0/0x20
[  300.456198] Code: 00 00 48 89 78 08 48 89 3a 48 89 93 e0 00 00 00
48 8b 53 08 48 8d 42 10 f0 80 4a 10 08 59 5e 5b c3 41 54 49 89 fc 55
48 89 f5 53 <83> 7e 38 00 74 24 48 8d 5e 10 48 39 5f 30 75 0d 48 89 df
e8 d4
[  300.478538] RIP  [<ffffffff8103668b>] set_next_entity+0xa/0x56
[  300.484463]  RSP <ffff881205e63d50>
[  300.487997] CR2: 0000000000000038
[  300.491359] ---[ end trace 7151f959ecbc401e ]---
[  302.149450] BUG: unable to handle kernel NULL pointer dereference
at 0000000000000038
[  302.157439] IP: [<ffffffff8103668b>] set_next_entity+0xa/0x56
[  302.163284] PGD 0
[  302.165400] Oops: 0000 [#2] SMP
[  302.168781] last sysfs file: /sys/devices/virtual/net/lo/operstate
[  302.174999] CPU 10
[  302.177202] Modules linked in: ifb xt_tcpudp xt_state ipt_REJECT
ipt_LOG xt_limit iptable_filter xt_DSCP xt_recent nf_nat_ftp nf_nat
nf_conntrack_ftp nf_conntrack_irc nf_conntrack_ipv4 nf_conntrack
nf_defrag_ipv4 ip]
[  302.236280] Pid: 61, comm: events/10 Tainted: G      D   
2.6.31.11-vs2.3.0.36.28-beng #1
[  302.245805] RIP: 0010:[<ffffffff8103668b>]  [<ffffffff8103668b>]
set_next_entity+0xa/0x56
[  302.254084] RSP: 0018:ffff881205e57d50  EFLAGS: 00010046
[  302.259436] RAX: 0000000000000000 RBX: ffff8811f5054180 RCX:
0000000000000000
[  302.266609] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
ffff8811f5054180
[  302.273782] RBP: 0000000000000000 R08: 0000000000000000 R09:
0000000000000000
[  302.280954] R10: 11902660923a39f2 R11: ffffc90001410010 R12:
ffff8811f5054180
[  302.288126] R13: ffffc90001415708 R14: ffff881205e554e0 R15:
ffffffff810be9bc
[  302.295299] FS:  0000000000000000(0000) GS:ffffc90001400000(0000)
knlGS:0000000000000000
[  302.303438] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[  302.309227] CR2: 0000000000000038 CR3: 0000000001001000 CR4:
00000000000006e0
[  302.316402] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[  302.323575] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[  302.330749] Process events/10 (pid: 61, threadinfo
ffff881205e56000, task ffff881205e554e0)
[  302.339148] Stack:
[  302.341208]  ffff8811f5054180 0000000000000000 ffffc90001412d00
ffffffff81037474
[  302.348626] <0> ffffc90001412d00 ffffc90001412d00 ffffc9000140ff70
ffffffff8103312b
[  302.356594] <0> ffffc90001412d00 ffffc90001412d00 000000000000000a
ffffffff812c13d4
[  302.364803] Call Trace:
[  302.367300]  [<ffffffff81037474>] ? pick_next_task_fair+0x85/0xa8
[  302.373433]  [<ffffffff8103312b>] ? pick_next_task+0x24/0x40
[  302.379134]  [<ffffffff812c13d4>] ? schedule+0x58a/0x767
[  302.384490]  [<ffffffff8104f5da>] ? lock_timer_base+0x26/0x4c
[  302.390279]  [<ffffffff810be9bc>] ? vmstat_update+0x0/0x3a
[  302.395805]  [<ffffffff810be9bc>] ? vmstat_update+0x0/0x3a
[  302.401332]  [<ffffffff8105634a>] ? worker_thread+0xbb/0x20f
[  302.407033]  [<ffffffff8105a456>] ? autoremove_wake_function+0x0/0x2e
[  302.413514]  [<ffffffff8105628f>] ? worker_thread+0x0/0x20f
[  302.419128]  [<ffffffff8105a10c>] ? kthread+0x8b/0x93
[  302.424223]  [<ffffffff8100cb2a>] ? child_rip+0xa/0x20
[  302.429404]  [<ffffffff8105a081>] ? kthread+0x0/0x93
[  302.434410]  [<ffffffff8100cb20>] ? child_rip+0x0/0x20
[  302.439590] Code: 00 00 48 89 78 08 48 89 3a 48 89 93 e0 00 00 00
48 8b 53 08 48 8d 42 10 f0 80 4a 10 08 59 5e 5b c3 41 54 49 89 fc 55
48 89 f5 53 <83> 7e 38 00 74 24 48 8d 5e 10 48 39 5f 30 75 0d 48 89 df
e8 d4
[  302.461949] RIP  [<ffffffff8103668b>] set_next_entity+0xa/0x56
[  302.467875]  RSP <ffff881205e57d50>
[  302.471408] CR2: 0000000000000038
[  302.474768] ---[ end trace 7151f959ecbc401f ]---



Without cgroups feature all works well.

vserver-info
Versions:
                   Kernel:vserver-info
Versions:
                   Kernel: 2.6.31.11-vs2.3.0.36.28-beng
                   VS-API: 0x00020305
             util-vserver: 0.30.216-pre2864; Nov 29 2009, 00:39:14

Features:
                       CC: gcc, gcc (Debian 4.3.4-6) 4.3.4
                      CXX: g++, g++ (Debian 4.3.4-6) 4.3.4
                 CPPFLAGS: ''
                   CFLAGS: '-Wall -g  -O2 -std=c99 -Wall -pedantic -W
- -funit-at-a-time'
                 CXXFLAGS: '-g -O2 -ansi -Wall -pedantic -W
- -fmessage-length=0 -funit-at-a-time'
               build/host: x86_64-pc-linux-gnu/x86_64-pc-linux-gnu
             Use dietlibc: yes
       Build C++ programs: yes
       Build C99 programs: yes
           Available APIs: v13,net,v21,v22,v23,netv2
            ext2fs Source: e2fsprogs
    syscall(2) invocation: alternative
      vserver(2) syscall#: 236/glibc
               crypto api: nss
          python bindings: no
   use library versioning: yes

Paths:
                   prefix: /usr
        sysconf-Directory: /etc
            cfg-Directory: /etc/vservers
         initrd-Directory: $(sysconfdir)/init.d
       pkgstate-Directory: /var/run/vservers
          vserver-Rootdir: /var/lib/vservers

This also happens with 2.6.31.7-vs2.3.0.36.27. Maybe an upstream problem
or vserver related?

Best regards

Oliver
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (GNU/Linux)

iEYEARECAAYFAktu2xMACgkQOBdlVlcPuhzQBQCgovcNHLlXkisBujQEpoiaZrIC
ZPkAoMFbIapePBZUx1vJiyuAwlfJt+6u
=WaNU
-----END PGP SIGNATURE-----