Subject: Kernel > 3.0 and rcu_sched stalls
From: cryptronic <mail@cryptronic.de>
Date: Fri, 27 Apr 2012 18:15:44 +0200
Fri, 27 Apr 2012 18:15:44 +0200
Hi list,

we have currently problems with kernels greater 3.0.

The situation is the following.

Suddenly all we can not log in via ssh or attached serial console. The
server is still replying on pings but a ssh connect times out.

On serial console we got lines like the following:
INFO: rcu_sched detected stalls on CPUs/tasks: { x,y,z} (detected by a,
t=b jiffies)

Now we got the first time a complete stack trace and now I want to ask,
whether this is known to others or if there is a way to debug such
things. Because i found such issues with all kernels > 3.0 with the
vserver patch i talk first of all to the list.

Kernelversion: various > 3.0.18 but in this case: 3.3.3-vs2.3.3.3
util-vserver: 0.30.216-pre3034

any more information required?


[18052.397180] BUG: Bad page map in process apache2 
pte:8000000f9a57a044 pmd:1133813067
[18052.412862] addr:00007f615ee01000 vm_flags:00100073
anon_vma:ffff880f4588e760 mapping:          (null) index:7f615ee01
[18052.434232] Pid: 18850, comm: apache2 Tainted: G      D     
3.3.3-vs2.3.3.3 #1
[18052.449585] Call Trace:
[18052.454476]  [<ffffffff810c3424>] ? print_bad_pte+0x1f5/0x20d
[18052.465944]  [<ffffffff810c553a>] ? unmap_vmas+0x61f/0x839
[18052.476896]  [<ffffffff810b5a17>] ? pagevec_lru_move_fn+0xb0/0xc6
[18052.489054]  [<ffffffff810c959c>] ? exit_mmap+0x99/0xfb
[18052.499481]  [<ffffffff8103f785>] ? mmput+0x4c/0xec
[18052.509214]  [<ffffffff810435cf>] ? exit_mm+0x103/0x10e
[18052.519855]  [<ffffffff8105bcf1>] ? hrtimer_try_to_cancel+0x6b/0x74
[18052.532364]  [<ffffffff81044e8a>] ? do_exit+0x25b/0x7ee
[18052.542789]  [<ffffffff810ca5f0>] ? do_munmap+0x2c9/0x2eb
[18052.553558]  [<ffffffff81045497>] ? do_group_exit+0x7a/0xa4
[18052.564672]  [<ffffffff810454d3>] ? sys_exit_group+0x12/0x16
[18052.575966]  [<ffffffff814d3379>] ? system_call_fastpath+0x16/0x1b
[18140.774496] INFO: rcu_sched detected stall on CPU 8 (t=780128 jiffies)
[18140.776488] INFO: rcu_sched detected stalls on CPUs/tasks: { 8}
(detected by 7, t=780130 jiffies)
[18140.776495] Pid: 3465, comm: apache2 Tainted: G    B D     
3.3.3-vs2.3.3.3 #1
[18140.776497] Call Trace:
[18140.776499] <IRQ>  [<ffffffff8109d4c0>] ? __rcu_pending+0x21a/0x336
[18140.776509]  [<ffffffff8109d66e>] ? rcu_check_callbacks+0x92/0xc2
[18140.776513]  [<ffffffff8104bf96>] ? update_process_times+0x31/0x63
[18140.776517]  [<ffffffff81079ecc>] ? tick_sched_timer+0x65/0x8b
[18140.776522]  [<ffffffff8105b58b>] ? __run_hrtimer+0x75/0xe9
[18140.776525]  [<ffffffff8105b838>] ? hrtimer_interrupt+0xd8/0x1a6
[18140.776530]  [<ffffffff81018f92>] ? smp_apic_timer_interrupt+0x80/0x93
[18140.776535]  [<ffffffff814d3e1e>] ? apic_timer_interrupt+0x6e/0x80
[18140.776536] <EOI>  [<ffffffff810e7c28>] ? css_tryget+0x1e/0x38
[18140.776544]  [<ffffffff810eb3a0>] ? __mem_cgroup_try_charge+0x187/0x4c4
[18140.776548]  [<ffffffff81101792>] ? __d_lookup_rcu+0x37/0x114
[18140.776552]  [<ffffffff810ebb6d>] ? mem_cgroup_charge_common+0x80/0xae
[18140.776556]  [<ffffffff810dac01>] ? alloc_pages_vma+0xf0/0x11d
[18140.776560]  [<ffffffff810c6ac1>] ? handle_pte_fault+0x1d4/0x95c
[18140.776564]  [<ffffffff810247f7>] ? do_page_fault+0x2c0/0x2e5
[18140.776567]  [<ffffffff810dfabb>] ? virt_to_slab+0x9/0x3c
[18140.776570]  [<ffffffff810dffa1>] ? __cache_free+0x7d/0x1d3
[18140.776572]  [<ffffffff814d2e05>] ? page_fault+0x25/0x30
[18141.039094] Pid: 3754, comm: pop3-login Tainted: G    B D     
3.3.3-vs2.3.3.3 #1
[18141.054545] Call Trace:
[18141.059420] <IRQ>  [<ffffffff8109d328>] ? __rcu_pending+0x82/0x336
[18141.071935]  [<ffffffff8109d66e>] ? rcu_check_callbacks+0x92/0xc2
[18141.084084]  [<ffffffff8104bf96>] ? update_process_times+0x31/0x63
[18141.096408]  [<ffffffff81079ecc>] ? tick_sched_timer+0x65/0x8b
[18141.108039]  [<ffffffff8105b58b>] ? __run_hrtimer+0x75/0xe9
[18141.119147]  [<ffffffff8105b838>] ? hrtimer_interrupt+0xd8/0x1a6
[18141.131127]  [<ffffffff81018f92>] ? smp_apic_timer_interrupt+0x80/0x93
[18141.144144]  [<ffffffff814d3e1e>] ? apic_timer_interrupt+0x6e/0x80
[18141.156465] <EOI>  [<ffffffff810e7c1f>] ? css_tryget+0x15/0x38
[18141.168282]  [<ffffffff810eb3a0>] ? __mem_cgroup_try_charge+0x187/0x4c4
[18141.181469]  [<ffffffff81001703>] ? __switch_to+0x238/0x35f
[18141.192577]  [<ffffffff810ebb6d>] ? mem_cgroup_charge_common+0x80/0xae
[18141.205601]  [<ffffffff810dac01>] ? alloc_pages_vma+0xf0/0x11d
[18141.217230]  [<ffffffff810c6ac1>] ? handle_pte_fault+0x1d4/0x95c
[18141.229205]  [<ffffffff8105bcf1>] ? hrtimer_try_to_cancel+0x6b/0x74
[18141.241703]  [<ffffffff8111f190>] ? ep_read_events_proc+0x6c/0x6c
[18141.253853]  [<ffffffff810247f7>] ? do_page_fault+0x2c0/0x2e5
[18141.265309]  [<ffffffff810637e4>] ? try_to_wake_up+0x19b/0x19b
[18141.276938]  [<ffffffff81008855>] ? read_tsc+0x5/0x16
[18141.287010]  [<ffffffff81073cf2>] ? timekeeping_get_ns+0xd/0x2a
[18141.298812]  [<ffffffff8107494f>] ? getnstimeofday+0x4d/0x80
[18141.310094]  [<ffffffff814d2e05>] ? page_fault+0x25/0x30





Hi list,

we have currently problems with kernels greater 3.0.

The situation is the following.

Suddenly all we can not log in via ssh or attached serial console. The server is still replying on pings but a ssh connect times out.

On serial console we got lines like the following:
INFO: rcu_sched detected stalls on CPUs/tasks: { x,y,z} (detected by a, t=b jiffies)

Now we got the first time a complete stack trace and now I want to ask, whether this is known to others or if there is a way to debug such things. Because i found such issues with all kernels > 3.0 with the vserver patch i talk first of all to the list.

Kernelversion: various > 3.0.18 but in this case: 3.3.3-vs2.3.3.3
util-vserver: 0.30.216-pre3034

any more information required?


[18052.397180] BUG: Bad page map in process apache2  pte:8000000f9a57a044 pmd:1133813067
[18052.412862] addr:00007f615ee01000 vm_flags:00100073 anon_vma:ffff880f4588e760 mapping:          (null) index:7f615ee01
[18052.434232] Pid: 18850, comm: apache2 Tainted: G      D      3.3.3-vs2.3.3.3 #1
[18052.449585] Call Trace:
[18052.454476]  [<ffffffff810c3424>] ? print_bad_pte+0x1f5/0x20d
[18052.465944]  [<ffffffff810c553a>] ? unmap_vmas+0x61f/0x839
[18052.476896]  [<ffffffff810b5a17>] ? pagevec_lru_move_fn+0xb0/0xc6
[18052.489054]  [<ffffffff810c959c>] ? exit_mmap+0x99/0xfb
[18052.499481]  [<ffffffff8103f785>] ? mmput+0x4c/0xec
[18052.509214]  [<ffffffff810435cf>] ? exit_mm+0x103/0x10e
[18052.519855]  [<ffffffff8105bcf1>] ? hrtimer_try_to_cancel+0x6b/0x74
[18052.532364]  [<ffffffff81044e8a>] ? do_exit+0x25b/0x7ee
[18052.542789]  [<ffffffff810ca5f0>] ? do_munmap+0x2c9/0x2eb
[18052.553558]  [<ffffffff81045497>] ? do_group_exit+0x7a/0xa4
[18052.564672]  [<ffffffff810454d3>] ? sys_exit_group+0x12/0x16
[18052.575966]  [<ffffffff814d3379>] ? system_call_fastpath+0x16/0x1b
[18140.774496] INFO: rcu_sched detected stall on CPU 8 (t=780128 jiffies)
[18140.776488] INFO: rcu_sched detected stalls on CPUs/tasks: { 8} (detected by 7, t=780130 jiffies)
[18140.776495] Pid: 3465, comm: apache2 Tainted: G    B D      3.3.3-vs2.3.3.3 #1
[18140.776497] Call Trace:
[18140.776499] <IRQ>  [<ffffffff8109d4c0>] ? __rcu_pending+0x21a/0x336
[18140.776509]  [<ffffffff8109d66e>] ? rcu_check_callbacks+0x92/0xc2
[18140.776513]  [<ffffffff8104bf96>] ? update_process_times+0x31/0x63
[18140.776517]  [<ffffffff81079ecc>] ? tick_sched_timer+0x65/0x8b
[18140.776522]  [<ffffffff8105b58b>] ? __run_hrtimer+0x75/0xe9
[18140.776525]  [<ffffffff8105b838>] ? hrtimer_interrupt+0xd8/0x1a6
[18140.776530]  [<ffffffff81018f92>] ? smp_apic_timer_interrupt+0x80/0x93
[18140.776535]  [<ffffffff814d3e1e>] ? apic_timer_interrupt+0x6e/0x80
[18140.776536] <EOI>  [<ffffffff810e7c28>] ? css_tryget+0x1e/0x38
[18140.776544]  [<ffffffff810eb3a0>] ? __mem_cgroup_try_charge+0x187/0x4c4
[18140.776548]  [<ffffffff81101792>] ? __d_lookup_rcu+0x37/0x114
[18140.776552]  [<ffffffff810ebb6d>] ? mem_cgroup_charge_common+0x80/0xae
[18140.776556]  [<ffffffff810dac01>] ? alloc_pages_vma+0xf0/0x11d
[18140.776560]  [<ffffffff810c6ac1>] ? handle_pte_fault+0x1d4/0x95c
[18140.776564]  [<ffffffff810247f7>] ? do_page_fault+0x2c0/0x2e5
[18140.776567]  [<ffffffff810dfabb>] ? virt_to_slab+0x9/0x3c
[18140.776570]  [<ffffffff810dffa1>] ? __cache_free+0x7d/0x1d3
[18140.776572]  [<ffffffff814d2e05>] ? page_fault+0x25/0x30
[18141.039094] Pid: 3754, comm: pop3-login Tainted: G    B D      3.3.3-vs2.3.3.3 #1
[18141.054545] Call Trace:
[18141.059420] <IRQ>  [<ffffffff8109d328>] ? __rcu_pending+0x82/0x336
[18141.071935]  [<ffffffff8109d66e>] ? rcu_check_callbacks+0x92/0xc2
[18141.084084]  [<ffffffff8104bf96>] ? update_process_times+0x31/0x63
[18141.096408]  [<ffffffff81079ecc>] ? tick_sched_timer+0x65/0x8b
[18141.108039]  [<ffffffff8105b58b>] ? __run_hrtimer+0x75/0xe9
[18141.119147]  [<ffffffff8105b838>] ? hrtimer_interrupt+0xd8/0x1a6
[18141.131127]  [<ffffffff81018f92>] ? smp_apic_timer_interrupt+0x80/0x93
[18141.144144]  [<ffffffff814d3e1e>] ? apic_timer_interrupt+0x6e/0x80
[18141.156465] <EOI>  [<ffffffff810e7c1f>] ? css_tryget+0x15/0x38
[18141.168282]  [<ffffffff810eb3a0>] ? __mem_cgroup_try_charge+0x187/0x4c4
[18141.181469]  [<ffffffff81001703>] ? __switch_to+0x238/0x35f
[18141.192577]  [<ffffffff810ebb6d>] ? mem_cgroup_charge_common+0x80/0xae
[18141.205601]  [<ffffffff810dac01>] ? alloc_pages_vma+0xf0/0x11d
[18141.217230]  [<ffffffff810c6ac1>] ? handle_pte_fault+0x1d4/0x95c
[18141.229205]  [<ffffffff8105bcf1>] ? hrtimer_try_to_cancel+0x6b/0x74
[18141.241703]  [<ffffffff8111f190>] ? ep_read_events_proc+0x6c/0x6c
[18141.253853]  [<ffffffff810247f7>] ? do_page_fault+0x2c0/0x2e5
[18141.265309]  [<ffffffff810637e4>] ? try_to_wake_up+0x19b/0x19b
[18141.276938]  [<ffffffff81008855>] ? read_tsc+0x5/0x16
[18141.287010]  [<ffffffff81073cf2>] ? timekeeping_get_ns+0xd/0x2a
[18141.298812]  [<ffffffff8107494f>] ? getnstimeofday+0x4d/0x80
[18141.310094]  [<ffffffff814d2e05>] ? page_fault+0x25/0x30