Subject: Re: [vserver] Kernel > 3.0 and rcu_sched stalls
From: Herbert Poetzl <herbert@13thfloor.at>
Date: Fri, 27 Apr 2012 18:40:50 +0200

On Fri, Apr 27, 2012 at 06:15:44PM +0200, cryptronic wrote:
> Hi list,

> we have currently problems with kernels greater 3.0.

> The situation is the following.

> Suddenly all we can not log in via ssh or attached serial
> console. The server is still replying on pings but a ssh
> connect times out.

ping is often handled in the interrupt or at least at a
very low level, so it doesn't really need much kernel
intervention to work

> On serial console we got lines like the following:
> INFO: rcu_sched detected stalls on CPUs/tasks: { x,y,z} (detected by a,
> t=b jiffies)

this is just the result of something going wrong in the
kernel and thus a CPU getting stuck and/or being too busy.
but it's good to hear that you are using a serial console,
which is the proper tool to capture kernel traces

> Now we got the first time a complete stack trace and now I want to ask,
> whether this is known to others or if there is a way to debug such
> things. Because i found such issues with all kernels > 3.0 with the
> vserver patch i talk first of all to the list.

from the first glance I'd suspect the hardware, specifically
the memory or the memory controller to have some issues, if
that can be ruled out (diversive hardware fails with the same
or at least similar traces), I'd look for problems with the
memory cgroup

> Kernelversion: various > 3.0.18 but in this case: 3.3.3-vs2.3.3.3
> util-vserver: 0.30.216-pre3034

> any more information required?

yes, this is not the first kernel oops/bug, please search for
the other(s) and provide those as well (from the same system)

thanks,
Herbert

> [18052.397180] BUG: Bad page map in process apache2 
> pte:8000000f9a57a044 pmd:1133813067
> [18052.412862] addr:00007f615ee01000 vm_flags:00100073
> anon_vma:ffff880f4588e760 mapping:          (null) index:7f615ee01
> [18052.434232] Pid: 18850, comm: apache2 Tainted: G      D     
> 3.3.3-vs2.3.3.3 #1
> [18052.449585] Call Trace:
> [18052.454476]  [<ffffffff810c3424>] ? print_bad_pte+0x1f5/0x20d
> [18052.465944]  [<ffffffff810c553a>] ? unmap_vmas+0x61f/0x839
> [18052.476896]  [<ffffffff810b5a17>] ? pagevec_lru_move_fn+0xb0/0xc6
> [18052.489054]  [<ffffffff810c959c>] ? exit_mmap+0x99/0xfb
> [18052.499481]  [<ffffffff8103f785>] ? mmput+0x4c/0xec
> [18052.509214]  [<ffffffff810435cf>] ? exit_mm+0x103/0x10e
> [18052.519855]  [<ffffffff8105bcf1>] ? hrtimer_try_to_cancel+0x6b/0x74
> [18052.532364]  [<ffffffff81044e8a>] ? do_exit+0x25b/0x7ee
> [18052.542789]  [<ffffffff810ca5f0>] ? do_munmap+0x2c9/0x2eb
> [18052.553558]  [<ffffffff81045497>] ? do_group_exit+0x7a/0xa4
> [18052.564672]  [<ffffffff810454d3>] ? sys_exit_group+0x12/0x16
> [18052.575966]  [<ffffffff814d3379>] ? system_call_fastpath+0x16/0x1b
> [18140.774496] INFO: rcu_sched detected stall on CPU 8 (t=780128 jiffies)
> [18140.776488] INFO: rcu_sched detected stalls on CPUs/tasks: { 8}
> (detected by 7, t=780130 jiffies)
> [18140.776495] Pid: 3465, comm: apache2 Tainted: G    B D     
> 3.3.3-vs2.3.3.3 #1
> [18140.776497] Call Trace:
> [18140.776499] <IRQ>  [<ffffffff8109d4c0>] ? __rcu_pending+0x21a/0x336
> [18140.776509]  [<ffffffff8109d66e>] ? rcu_check_callbacks+0x92/0xc2
> [18140.776513]  [<ffffffff8104bf96>] ? update_process_times+0x31/0x63
> [18140.776517]  [<ffffffff81079ecc>] ? tick_sched_timer+0x65/0x8b
> [18140.776522]  [<ffffffff8105b58b>] ? __run_hrtimer+0x75/0xe9
> [18140.776525]  [<ffffffff8105b838>] ? hrtimer_interrupt+0xd8/0x1a6
> [18140.776530]  [<ffffffff81018f92>] ? smp_apic_timer_interrupt+0x80/0x93
> [18140.776535]  [<ffffffff814d3e1e>] ? apic_timer_interrupt+0x6e/0x80
> [18140.776536] <EOI>  [<ffffffff810e7c28>] ? css_tryget+0x1e/0x38
> [18140.776544]  [<ffffffff810eb3a0>] ? __mem_cgroup_try_charge+0x187/0x4c4
> [18140.776548]  [<ffffffff81101792>] ? __d_lookup_rcu+0x37/0x114
> [18140.776552]  [<ffffffff810ebb6d>] ? mem_cgroup_charge_common+0x80/0xae
> [18140.776556]  [<ffffffff810dac01>] ? alloc_pages_vma+0xf0/0x11d
> [18140.776560]  [<ffffffff810c6ac1>] ? handle_pte_fault+0x1d4/0x95c
> [18140.776564]  [<ffffffff810247f7>] ? do_page_fault+0x2c0/0x2e5
> [18140.776567]  [<ffffffff810dfabb>] ? virt_to_slab+0x9/0x3c
> [18140.776570]  [<ffffffff810dffa1>] ? __cache_free+0x7d/0x1d3
> [18140.776572]  [<ffffffff814d2e05>] ? page_fault+0x25/0x30
> [18141.039094] Pid: 3754, comm: pop3-login Tainted: G    B D     
> 3.3.3-vs2.3.3.3 #1
> [18141.054545] Call Trace:
> [18141.059420] <IRQ>  [<ffffffff8109d328>] ? __rcu_pending+0x82/0x336
> [18141.071935]  [<ffffffff8109d66e>] ? rcu_check_callbacks+0x92/0xc2
> [18141.084084]  [<ffffffff8104bf96>] ? update_process_times+0x31/0x63
> [18141.096408]  [<ffffffff81079ecc>] ? tick_sched_timer+0x65/0x8b
> [18141.108039]  [<ffffffff8105b58b>] ? __run_hrtimer+0x75/0xe9
> [18141.119147]  [<ffffffff8105b838>] ? hrtimer_interrupt+0xd8/0x1a6
> [18141.131127]  [<ffffffff81018f92>] ? smp_apic_timer_interrupt+0x80/0x93
> [18141.144144]  [<ffffffff814d3e1e>] ? apic_timer_interrupt+0x6e/0x80
> [18141.156465] <EOI>  [<ffffffff810e7c1f>] ? css_tryget+0x15/0x38
> [18141.168282]  [<ffffffff810eb3a0>] ? __mem_cgroup_try_charge+0x187/0x4c4
> [18141.181469]  [<ffffffff81001703>] ? __switch_to+0x238/0x35f
> [18141.192577]  [<ffffffff810ebb6d>] ? mem_cgroup_charge_common+0x80/0xae
> [18141.205601]  [<ffffffff810dac01>] ? alloc_pages_vma+0xf0/0x11d
> [18141.217230]  [<ffffffff810c6ac1>] ? handle_pte_fault+0x1d4/0x95c
> [18141.229205]  [<ffffffff8105bcf1>] ? hrtimer_try_to_cancel+0x6b/0x74
> [18141.241703]  [<ffffffff8111f190>] ? ep_read_events_proc+0x6c/0x6c
> [18141.253853]  [<ffffffff810247f7>] ? do_page_fault+0x2c0/0x2e5
> [18141.265309]  [<ffffffff810637e4>] ? try_to_wake_up+0x19b/0x19b
> [18141.276938]  [<ffffffff81008855>] ? read_tsc+0x5/0x16
> [18141.287010]  [<ffffffff81073cf2>] ? timekeeping_get_ns+0xd/0x2a
> [18141.298812]  [<ffffffff8107494f>] ? getnstimeofday+0x4d/0x80
> [18141.310094]  [<ffffffff814d2e05>] ? page_fault+0x25/0x30