Subject: Re: [vserver] Kernel oops with NULL pointer dereference when using cgroup hardlimiting
From: Herbert Poetzl <herbert@13thfloor.at>
Date: Sun, 7 Feb 2010 18:58:54 +0100

On Sun, Feb 07, 2010 at 04:24:03PM +0100, Cryptronic wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1

> Hi,

> i have running a server with a 2.6.31.11-vs2.3.0.36.28 kernel.

> When using cgoups as descriped in
> http://linux-vserver.org/util-vserver:Cgroups#cgroup_and_CFS_based_CPU_hard_limiting_that_replaces_sched_hard
> after a view minutes i get a kernel null pointer:

could you test again with the current upstream kernel
and patch? 
(http://vserver.13thfloor.at/Experimental/patch-2.6.31.12-vs2.3.0.36.28.diff)

and please enable DEBUG_INFO in your kernel config

TIA,
Herbert


> [  300.166119] BUG: unable to handle kernel NULL pointer dereference
> at
> 0000000000000038                                                                 
                                                
>  
>  
> [  300.174083] IP: [<ffffffff8103668b>]
> set_next_entity+0xa/0x56                                                         
                                                                                   
         
>  
>  
> [  300.179918] PGD
> 0                                                                                
                                                                                   
                              
>  
>  
> [  300.182018] Oops: 0000 [#1]
> SMP                                                                              
                                                                                   
                  
>  
>  
> [  300.185383] last sysfs file:
> /sys/devices/virtual/net/lo/operstate                                            
                                                                                   
                 
>  
>  
> [  300.191590] CPU
> 14                                                                               
                                                                                   
                              
>  
>  
> [  300.193775] Modules linked in: ifb xt_tcpudp xt_state ipt_REJECT
> ipt_LOG xt_limit iptable_filter xt_DSCP xt_recent nf_nat_ftp nf_nat
> nf_conntrack_ftp nf_conntrack_irc nf_conntrack_ipv4 nf_conntrack
> nf_defrag_ipv4 ip]
> [  300.252780] Pid: 65, comm: events/14 Not tainted
> 2.6.31.11-vs2.3.0.36.28-beng
> #1                                                                               
                        
>  
>  
> [  300.261525] RIP: 0010:[<ffffffff8103668b>]  [<ffffffff8103668b>]
> set_next_entity+0xa/0x56                                                         
                                                                 
>  
>  
> [  300.269807] RSP: 0018:ffff881205e63d50  EFLAGS:
> 00010046                                                                         
                                                                                  
>  
>  
> [  300.275160] RAX: 0000000000000000 RBX: ffff8811f5054680 RCX:
> 0000000000000000                                                                 
                                                                     
>  
>  
> [  300.282333] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
> ffff8811f5054680                                                                 
                                                                     
>  
>  
> [  300.289506] RBP: 0000000000000000 R08: 0000000000000000 R09:
> 0000000000000000                                                                 
                                                                     
>  
>  
> [  300.296681] R10: 119026601aff243a R11: ffffc90001c10010 R12:
> ffff8811f5054680                                                                 
                                                                     
>  
>  
> [  300.303853] R13: ffffc90001c15708 R14: 0000000000000000 R15:
> ffffffff810d8248                                                                 
                                                                     
>  
>  
> [  300.311026] FS:  0000000000000000(0000) GS:ffffc90001c00000(0000)
> knlGS:0000000000000000                                                           
                                                                
>  
>  
> [  300.319166] CS:  0010 DS: 0018 ES: 0018 CR0:
> 000000008005003b                                                                 
                                                                                   
 
>  
>  
> [  300.324952] CR2: 0000000000000038 CR3: 0000000001001000 CR4:
> 00000000000006e0                                                                 
                                                                     
>  
>  
> [  300.332126] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000                                                                 
                                                                     
>  
>  
> [  300.339300] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400                                                                 
                                                                     
>  
>  
> [  300.346472] Process events/14 (pid: 65, threadinfo
> ffff881205e62000, task
> ffff881205e61520)                                                                
                                                        
>  
>  
> [  300.354871]
> Stack:                                                                           
                                                                                   
                                  
>  
>  
> [  300.356931]  ffff8811f5054680 0000000000000000 ffffc90001c12d00
> ffffffff81037474                                                                 
                                                                  
>  
>  
> [  300.364347] <0> ffffffff812d29d0 ffffc90001c12d00 0000000000000000
> ffffffff81033138                                                                 
                                                               
>  
>  
> [  300.372314] <0> ffffc90001c12d00 ffffc90001c12d00 000000000000000e
> ffffffff812c13d4                                                                 
                                                               
>  
>  
> [  300.380521] Call
> Trace:                                                                           
                                                                                   
                             
>  
>  
> [  300.383020]  [<ffffffff81037474>] ?
> pick_next_task_fair+0x85/0xa8                                                    
                                                                                   
          
>  
>  
> [  300.389156]  [<ffffffff81033138>] ?
> pick_next_task+0x31/0x40                                                         
                                                                                   
          
>  
>  
> [  300.394861]  [<ffffffff812c13d4>] ?
> schedule+0x58a/0x767                                                             
                                                                                   
          
>  
>  
> [  300.400218]  [<ffffffff8104f5da>] ?
> lock_timer_base+0x26/0x4c                                                        
                                                                                   
          
>  
>  
> [  300.406006]  [<ffffffff8104effa>] ?
> __round_jiffies_relative+0x12/0x17                                               
                                                                                   
          
>  
>  
> [  300.412577]  [<ffffffff810d8248>] ?
> cache_reap+0x0/0x20f                                                             
                                                                                   
          
>  
>  
> [  300.417931]  [<ffffffff8105634a>] ?
> worker_thread+0xbb/0x20f                                                         
                                                                                   
          
>  
>  
> [  300.423634]  [<ffffffff8105a456>] ?
> autoremove_wake_function+0x0/0x2e                                                
                                                                                   
          
>  
>  
> [  300.430121]  [<ffffffff8105628f>] ?
> worker_thread+0x0/0x20f                                                          
                                                                                   
          
>  
>  
> [  300.435735]  [<ffffffff8105a10c>] ?
> kthread+0x8b/0x93                                                                
                                                                                   
          
>  
>  
> [  300.440831]  [<ffffffff8100cb2a>] ?
> child_rip+0xa/0x20                                                               
                                                                                   
          
>  
>  
> [  300.446012]  [<ffffffff8105a081>] ?
> kthread+0x0/0x93                                                                 
                                                                                   
          
>  
>  
> [  300.451019]  [<ffffffff8100cb20>] ? child_rip+0x0/0x20
> [  300.456198] Code: 00 00 48 89 78 08 48 89 3a 48 89 93 e0 00 00 00
> 48 8b 53 08 48 8d 42 10 f0 80 4a 10 08 59 5e 5b c3 41 54 49 89 fc 55
> 48 89 f5 53 <83> 7e 38 00 74 24 48 8d 5e 10 48 39 5f 30 75 0d 48 89 df
> e8 d4
> [  300.478538] RIP  [<ffffffff8103668b>] set_next_entity+0xa/0x56
> [  300.484463]  RSP <ffff881205e63d50>
> [  300.487997] CR2: 0000000000000038
> [  300.491359] ---[ end trace 7151f959ecbc401e ]---
> [  302.149450] BUG: unable to handle kernel NULL pointer dereference
> at 0000000000000038
> [  302.157439] IP: [<ffffffff8103668b>] set_next_entity+0xa/0x56
> [  302.163284] PGD 0
> [  302.165400] Oops: 0000 [#2] SMP
> [  302.168781] last sysfs file: /sys/devices/virtual/net/lo/operstate
> [  302.174999] CPU 10
> [  302.177202] Modules linked in: ifb xt_tcpudp xt_state ipt_REJECT
> ipt_LOG xt_limit iptable_filter xt_DSCP xt_recent nf_nat_ftp nf_nat
> nf_conntrack_ftp nf_conntrack_irc nf_conntrack_ipv4 nf_conntrack
> nf_defrag_ipv4 ip]
> [  302.236280] Pid: 61, comm: events/10 Tainted: G      D   
> 2.6.31.11-vs2.3.0.36.28-beng #1
> [  302.245805] RIP: 0010:[<ffffffff8103668b>]  [<ffffffff8103668b>]
> set_next_entity+0xa/0x56
> [  302.254084] RSP: 0018:ffff881205e57d50  EFLAGS: 00010046
> [  302.259436] RAX: 0000000000000000 RBX: ffff8811f5054180 RCX:
> 0000000000000000
> [  302.266609] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
> ffff8811f5054180
> [  302.273782] RBP: 0000000000000000 R08: 0000000000000000 R09:
> 0000000000000000
> [  302.280954] R10: 11902660923a39f2 R11: ffffc90001410010 R12:
> ffff8811f5054180
> [  302.288126] R13: ffffc90001415708 R14: ffff881205e554e0 R15:
> ffffffff810be9bc
> [  302.295299] FS:  0000000000000000(0000) GS:ffffc90001400000(0000)
> knlGS:0000000000000000
> [  302.303438] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> [  302.309227] CR2: 0000000000000038 CR3: 0000000001001000 CR4:
> 00000000000006e0
> [  302.316402] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [  302.323575] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [  302.330749] Process events/10 (pid: 61, threadinfo
> ffff881205e56000, task ffff881205e554e0)
> [  302.339148] Stack:
> [  302.341208]  ffff8811f5054180 0000000000000000 ffffc90001412d00
> ffffffff81037474
> [  302.348626] <0> ffffc90001412d00 ffffc90001412d00 ffffc9000140ff70
> ffffffff8103312b
> [  302.356594] <0> ffffc90001412d00 ffffc90001412d00 000000000000000a
> ffffffff812c13d4
> [  302.364803] Call Trace:
> [  302.367300]  [<ffffffff81037474>] ? pick_next_task_fair+0x85/0xa8
> [  302.373433]  [<ffffffff8103312b>] ? pick_next_task+0x24/0x40
> [  302.379134]  [<ffffffff812c13d4>] ? schedule+0x58a/0x767
> [  302.384490]  [<ffffffff8104f5da>] ? lock_timer_base+0x26/0x4c
> [  302.390279]  [<ffffffff810be9bc>] ? vmstat_update+0x0/0x3a
> [  302.395805]  [<ffffffff810be9bc>] ? vmstat_update+0x0/0x3a
> [  302.401332]  [<ffffffff8105634a>] ? worker_thread+0xbb/0x20f
> [  302.407033]  [<ffffffff8105a456>] ? autoremove_wake_function+0x0/0x2e
> [  302.413514]  [<ffffffff8105628f>] ? worker_thread+0x0/0x20f
> [  302.419128]  [<ffffffff8105a10c>] ? kthread+0x8b/0x93
> [  302.424223]  [<ffffffff8100cb2a>] ? child_rip+0xa/0x20
> [  302.429404]  [<ffffffff8105a081>] ? kthread+0x0/0x93
> [  302.434410]  [<ffffffff8100cb20>] ? child_rip+0x0/0x20
> [  302.439590] Code: 00 00 48 89 78 08 48 89 3a 48 89 93 e0 00 00 00
> 48 8b 53 08 48 8d 42 10 f0 80 4a 10 08 59 5e 5b c3 41 54 49 89 fc 55
> 48 89 f5 53 <83> 7e 38 00 74 24 48 8d 5e 10 48 39 5f 30 75 0d 48 89 df
> e8 d4
> [  302.461949] RIP  [<ffffffff8103668b>] set_next_entity+0xa/0x56
> [  302.467875]  RSP <ffff881205e57d50>
> [  302.471408] CR2: 0000000000000038
> [  302.474768] ---[ end trace 7151f959ecbc401f ]---
> 
> 
> 
> Without cgroups feature all works well.
> 
> vserver-info
> Versions:
>                    Kernel:vserver-info
> Versions:
>                    Kernel: 2.6.31.11-vs2.3.0.36.28-beng
>                    VS-API: 0x00020305
>              util-vserver: 0.30.216-pre2864; Nov 29 2009, 00:39:14
> 
> Features:
>                        CC: gcc, gcc (Debian 4.3.4-6) 4.3.4
>                       CXX: g++, g++ (Debian 4.3.4-6) 4.3.4
>                  CPPFLAGS: ''
>                    CFLAGS: '-Wall -g  -O2 -std=c99 -Wall -pedantic -W
> - -funit-at-a-time'
>                  CXXFLAGS: '-g -O2 -ansi -Wall -pedantic -W
> - -fmessage-length=0 -funit-at-a-time'
>                build/host: x86_64-pc-linux-gnu/x86_64-pc-linux-gnu
>              Use dietlibc: yes
>        Build C++ programs: yes
>        Build C99 programs: yes
>            Available APIs: v13,net,v21,v22,v23,netv2
>             ext2fs Source: e2fsprogs
>     syscall(2) invocation: alternative
>       vserver(2) syscall#: 236/glibc
>                crypto api: nss
>           python bindings: no
>    use library versioning: yes
> 
> Paths:
>                    prefix: /usr
>         sysconf-Directory: /etc
>             cfg-Directory: /etc/vservers
>          initrd-Directory: $(sysconfdir)/init.d
>        pkgstate-Directory: /var/run/vservers
>           vserver-Rootdir: /var/lib/vservers
> 
> This also happens with 2.6.31.7-vs2.3.0.36.27. Maybe an upstream problem
> or vserver related?
> 
> Best regards
> 
> Oliver
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.14 (GNU/Linux)
> 
> iEYEARECAAYFAktu2xMACgkQOBdlVlcPuhzQBQCgovcNHLlXkisBujQEpoiaZrIC
> ZPkAoMFbIapePBZUx1vJiyuAwlfJt+6u
> =WaNU
> -----END PGP SIGNATURE-----