Subject: Re: [vserver] patch preventing "recursive" spinlocks
From: Art -kwaak- van Breemen <ard@telegraafnet.nl>
Date: Thu, 2 May 2013 18:19:36 +0200
Thu, 2 May 2013 18:19:36 +0200
Hi,

On Mon, Apr 29, 2013 at 06:00:36PM +0200, Art -kwaak- van Breemen wrote:
> [15:18] <ard> We can only replicate(!) this behaviour on servers with a very specific
kind of load...
> [15:38] <ard> maybe I also sould say that that's in kernel/vserver/inet.c
> [15:40] <ard> I think the problem is that the item gets locked, then an interrupt
comes in, which does a tcp_v4_rcv, and that
>           wants to lock the same structure.

Attached the promissed dumps....


root@hattie:~# BUG: spinlock recursion on CPU#0, nginx/5087
 lock: 0xffff88031dcc2db0, .magic: dead4ead, .owner: nginx/5087, .owner_cpu: 0
Pid: 5087, comm: nginx Not tainted 3.7.10-vs2.3.5.6-d64-i7 #1
Call Trace:
 <IRQ>  [<ffffffff8130a7c5>] ? do_raw_spin_lock+0x125/0x140
 [<ffffffff815e5a69>] ? _raw_spin_lock_irqsave+0x9/0x10
 [<ffffffff815325c5>] ? __inet_lookup_listener+0x185/0x520
 [<ffffffff8154e994>] ? tcp_v4_rcv+0x294/0x990
 [<ffffffff815291f4>] ? ip_local_deliver_finish+0xd4/0x290
 [<ffffffff814f9712>] ? __netif_receive_skb+0x222/0x7e0
 [<ffffffff8155f2fc>] ? inet_gro_receive+0x18c/0x1c0
 [<ffffffff814f9e54>] ? netif_receive_skb+0x24/0x80
 [<ffffffff814fac60>] ? napi_skb_finish+0x50/0x70
 [<ffffffffa0035b89>] ? bnx2_poll_work+0x849/0x1430 [bnx2]
 [<ffffffffa00367ac>] ? bnx2_poll_msix+0x3c/0xc0 [bnx2]
 [<ffffffff814fa32f>] ? net_rx_action+0x8f/0x180
 [<ffffffff81081d6f>] ? __do_softirq+0xbf/0x1f0
 [<ffffffff815e760c>] ? call_softirq+0x1c/0x30
 [<ffffffff81045a5d>] ? do_softirq+0x4d/0x80
 [<ffffffff81081fde>] ? irq_exit+0x8e/0xb0
 [<ffffffff810456db>] ? do_IRQ+0x5b/0xd0
 [<ffffffff815e5d6a>] ? common_interrupt+0x6a/0x6a
 <EOI>  [<ffffffff810c5680>] ? ip_v4_find_src+0x390/0x3f0
 [<ffffffff810c5621>] ? ip_v4_find_src+0x331/0x3f0
 [<ffffffff8154c1c2>] ? tcp_v4_connect+0x142/0x4f0
 [<ffffffff8156004f>] ? __inet_stream_connect+0x1ff/0x340
 [<ffffffff815495a0>] ? tcp_delack_timer_handler+0x1e0/0x1e0
 [<ffffffff81534c46>] ? inet_csk_init_xmit_timers+0x76/0xb0
 [<ffffffff811bc30a>] ? ep_call_nested.constprop.10+0xea/0x100
 [<ffffffff815601d0>] ? inet_stream_connect+0x40/0x70
 [<ffffffff814e5e4e>] ? sys_connect+0x6e/0xb0
 [<ffffffff811bc040>] ? ep_create_wakeup_source+0x80/0x80
 [<ffffffff815e6496>] ? system_call_fastpath+0x1a/0x1f
------------[ cut here ]------------
WARNING: at kernel/watchdog.c:245 watchdog_overflow_callback+0x9a/0xd0()
Hardware name: PowerEdge M610
Watchdog detected hard LOCKUP on cpu 0
Modules linked in: dm_mod 8021q bonding e1000 crc32c_intel ghash_clmulni_intel aesni_intel
ablk_helper cryptd lrw aes_x86_64 xts gf128mul bnx2 acpi_power_meter dcdbas hed evdev
Pid: 5087, comm: nginx Not tainted 3.7.10-vs2.3.5.6-d64-i7 #1
Call Trace:
 <NMI>  [<ffffffff810efc00>] ? touch_nmi_watchdog+0x60/0x80
 [<ffffffff81079949>] ? warn_slowpath_common+0x79/0xc0
 [<ffffffff81079a45>] ? warn_slowpath_fmt+0x45/0x50
 [<ffffffff810efc20>] ? touch_nmi_watchdog+0x80/0x80
 [<ffffffff810efcba>] ? watchdog_overflow_callback+0x9a/0xd0
 [<ffffffff8111b30d>] ? __perf_event_overflow+0x9d/0x240
 [<ffffffff811182d5>] ? perf_event_update_userpage+0x25/0x120
 [<ffffffff810591e3>] ? intel_pmu_handle_irq+0x193/0x2e0
 [<ffffffff810474c9>] ? nmi_handle.isra.0+0x49/0x70
 [<ffffffff810475d7>] ? do_nmi+0xe7/0x360
 [<ffffffff815e6266>] ? end_repeat_nmi+0x1e/0x2e
 [<ffffffff81303a31>] ? delay_tsc+0x11/0x50
 [<ffffffff81303a31>] ? delay_tsc+0x11/0x50
 [<ffffffff81303a31>] ? delay_tsc+0x11/0x50
 <<EOE>>  <IRQ>  [<ffffffff8130a78d>] ? do_raw_spin_lock+0xed/0x140
 [<ffffffff815e5a69>] ? _raw_spin_lock_irqsave+0x9/0x10
 [<ffffffff815325c5>] ? __inet_lookup_listener+0x185/0x520
 [<ffffffff8154e994>] ? tcp_v4_rcv+0x294/0x990
 [<ffffffff815291f4>] ? ip_local_deliver_finish+0xd4/0x290
 [<ffffffff814f9712>] ? __netif_receive_skb+0x222/0x7e0
 [<ffffffff8155f2fc>] ? inet_gro_receive+0x18c/0x1c0
 [<ffffffff814f9e54>] ? netif_receive_skb+0x24/0x80
 [<ffffffff814fac60>] ? napi_skb_finish+0x50/0x70
 [<ffffffffa0035b89>] ? bnx2_poll_work+0x849/0x1430 [bnx2]
 [<ffffffffa00367ac>] ? bnx2_poll_msix+0x3c/0xc0 [bnx2]
 [<ffffffff814fa32f>] ? net_rx_action+0x8f/0x180
 [<ffffffff81081d6f>] ? __do_softirq+0xbf/0x1f0
 [<ffffffff815e760c>] ? call_softirq+0x1c/0x30
 [<ffffffff81045a5d>] ? do_softirq+0x4d/0x80
 [<ffffffff81081fde>] ? irq_exit+0x8e/0xb0
 [<ffffffff810456db>] ? do_IRQ+0x5b/0xd0
 [<ffffffff815e5d6a>] ? common_interrupt+0x6a/0x6a
 <EOI>  [<ffffffff810c5680>] ? ip_v4_find_src+0x390/0x3f0
 [<ffffffff810c5621>] ? ip_v4_find_src+0x331/0x3f0
 [<ffffffff8154c1c2>] ? tcp_v4_connect+0x142/0x4f0
 [<ffffffff8156004f>] ? __inet_stream_connect+0x1ff/0x340
 [<ffffffff815495a0>] ? tcp_delack_timer_handler+0x1e0/0x1e0
 [<ffffffff81534c46>] ? inet_csk_init_xmit_timers+0x76/0xb0
 [<ffffffff811bc30a>] ? ep_call_nested.constprop.10+0xea/0x100
 [<ffffffff815601d0>] ? inet_stream_connect+0x40/0x70
 [<ffffffff814e5e4e>] ? sys_connect+0x6e/0xb0
 [<ffffffff811bc040>] ? ep_create_wakeup_source+0x80/0x80
 [<ffffffff815e6496>] ? system_call_fastpath+0x1a/0x1f
---[ end trace d9f7240909cad808 ]---


------------[ cut here ]------------
WARNING: at kernel/watchdog.c:245 watchdog_overflow_callback+0x9a/0xd0()
Hardware name: PowerEdge M610
Watchdog detected hard LOCKUP on cpu 14
Modules linked in: 8021q bonding crc32c_intel ghash_clmulni_intel aesni_intel ablk_helper
cryptd lrw aes_x86_64 xts gf128mul bnx2 dcdbas hed acpi_power_meter evdev
Pid: 0, comm: swapper/14 Not tainted 3.7.10-vs2.3.5.6-d64-i7 #1
Call Trace:
 <NMI>  [<ffffffff810efc00>] ? touch_nmi_watchdog+0x60/0x80
 [<ffffffff81079949>] ? warn_slowpath_common+0x79/0xc0
 [<ffffffff81079a45>] ? warn_slowpath_fmt+0x45/0x50
 [<ffffffff810efc20>] ? touch_nmi_watchdog+0x80/0x80
 [<ffffffff810efcba>] ? watchdog_overflow_callback+0x9a/0xd0
 [<ffffffff8111b30d>] ? __perf_event_overflow+0x9d/0x240
 [<ffffffff811182d5>] ? perf_event_update_userpage+0x25/0x120
 [<ffffffff810591e3>] ? intel_pmu_handle_irq+0x193/0x2e0
 [<ffffffff810474c9>] ? nmi_handle.isra.0+0x49/0x70
 [<ffffffff810475d7>] ? do_nmi+0xe7/0x360
 [<ffffffff815e6266>] ? end_repeat_nmi+0x1e/0x2e
 [<ffffffff81303a31>] ? delay_tsc+0x11/0x50
 [<ffffffff81303a31>] ? delay_tsc+0x11/0x50
 [<ffffffff81303a31>] ? delay_tsc+0x11/0x50
 <<EOE>>  <IRQ>  [<ffffffff8130a78d>] ? do_raw_spin_lock+0xed/0x140
 [<ffffffff815e5a69>] ? _raw_spin_lock_irqsave+0x9/0x10
 [<ffffffff815325c5>] ? __inet_lookup_listener+0x185/0x520
 [<ffffffff8154e994>] ? tcp_v4_rcv+0x294/0x990
 [<ffffffff815291f4>] ? ip_local_deliver_finish+0xd4/0x290
 [<ffffffff814f9712>] ? __netif_receive_skb+0x222/0x7e0
 [<ffffffff8155f2fc>] ? inet_gro_receive+0x18c/0x1c0
 [<ffffffff814f9e54>] ? netif_receive_skb+0x24/0x80
 [<ffffffff814fac60>] ? napi_skb_finish+0x50/0x70
 [<ffffffffa0057b89>] ? bnx2_poll_work+0x849/0x1430 [bnx2]
 [<ffffffff810c966b>] ? ktime_get+0x4b/0xe0
 [<ffffffff8130013a>] ? timerqueue_del+0x2a/0x80
 [<ffffffffa00587ac>] ? bnx2_poll_msix+0x3c/0xc0 [bnx2]
 [<ffffffff814fa32f>] ? net_rx_action+0x8f/0x180
 [<ffffffff81081d6f>] ? __do_softirq+0xbf/0x1f0
 [<ffffffff815e760c>] ? call_softirq+0x1c/0x30
 [<ffffffff81045a5d>] ? do_softirq+0x4d/0x80
 [<ffffffff81081fde>] ? irq_exit+0x8e/0xb0
 [<ffffffff810456db>] ? do_IRQ+0x5b/0xd0
 [<ffffffff815e5d6a>] ? common_interrupt+0x6a/0x6a
 <EOI>  [<ffffffff8132e17f>] ? intel_idle+0xef/0x150
 [<ffffffff8132e15a>] ? intel_idle+0xca/0x150
 [<ffffffff814c405e>] ? cpuidle_idle_call+0xae/0x250
 [<ffffffff8104ce1f>] ? cpu_idle+0x5f/0xd0
---[ end trace 53996d982d5b59b7 ]---
------------[ cut here ]------------
WARNING: at kernel/watchdog.c:245 watchdog_overflow_callback+0x9a/0xd0()
Hardware name: PowerEdge M610
Watchdog detected hard LOCKUP on cpu 8
Modules linked in: 8021q bonding crc32c_intel ghash_clmulni_intel aesni_intel ablk_helper
cryptd lrw aes_x86_64 xts gf128mul bnx2 dcdbas hed acpi_power_meter evdev
Pid: 0, comm: swapper/8 Tainted: G        W    3.7.10-vs2.3.5.6-d64-i7 #1
Call Trace:
 <NMI>  [<ffffffff810efc00>] ? touch_nmi_watchdog+0x60/0x80
 [<ffffffff81079949>] ? warn_slowpath_common+0x79/0xc0
 [<ffffffff81079a45>] ? warn_slowpath_fmt+0x45/0x50
 [<ffffffff810efc20>] ? touch_nmi_watchdog+0x80/0x80
 [<ffffffff810efcba>] ? watchdog_overflow_callback+0x9a/0xd0
 [<ffffffff8111b30d>] ? __perf_event_overflow+0x9d/0x240
 [<ffffffff811182d5>] ? perf_event_update_userpage+0x25/0x120
 [<ffffffff810591e3>] ? intel_pmu_handle_irq+0x193/0x2e0
 [<ffffffff810474c9>] ? nmi_handle.isra.0+0x49/0x70
 [<ffffffff810475d7>] ? do_nmi+0xe7/0x360
 [<ffffffff815e6266>] ? end_repeat_nmi+0x1e/0x2e
 [<ffffffff81303a4f>] ? delay_tsc+0x2f/0x50
 [<ffffffff81303a4f>] ? delay_tsc+0x2f/0x50
 [<ffffffff81303a4f>] ? delay_tsc+0x2f/0x50
 <<EOE>>  <IRQ>  [<ffffffff8130a78d>] ? do_raw_spin_lock+0xed/0x140
 [<ffffffff815e5a69>] ? _raw_spin_lock_irqsave+0x9/0x10
 [<ffffffff815325c5>] ? __inet_lookup_listener+0x185/0x520
 [<ffffffff8154e994>] ? tcp_v4_rcv+0x294/0x990
 [<ffffffff815291f4>] ? ip_local_deliver_finish+0xd4/0x290
 [<ffffffff814f9712>] ? __netif_receive_skb+0x222/0x7e0
 [<ffffffff8155f2fc>] ? inet_gro_receive+0x18c/0x1c0
 [<ffffffff814f9e54>] ? netif_receive_skb+0x24/0x80
 [<ffffffff814fac60>] ? napi_skb_finish+0x50/0x70
 [<ffffffffa0057b89>] ? bnx2_poll_work+0x849/0x1430 [bnx2]
 [<ffffffff810c966b>] ? ktime_get+0x4b/0xe0
 [<ffffffffa00587ac>] ? bnx2_poll_msix+0x3c/0xc0 [bnx2]
 [<ffffffff814fa32f>] ? net_rx_action+0x8f/0x180
 [<ffffffff81081d6f>] ? __do_softirq+0xbf/0x1f0
 [<ffffffff815e760c>] ? call_softirq+0x1c/0x30
 [<ffffffff81045a5d>] ? do_softirq+0x4d/0x80
 [<ffffffff81081fde>] ? irq_exit+0x8e/0xb0
 [<ffffffff810456db>] ? do_IRQ+0x5b/0xd0
 [<ffffffff815e5d6a>] ? common_interrupt+0x6a/0x6a
 <EOI>  [<ffffffff8132e17f>] ? intel_idle+0xef/0x150
 [<ffffffff8132e15a>] ? intel_idle+0xca/0x150
 [<ffffffff814c405e>] ? cpuidle_idle_call+0xae/0x250
 [<ffffffff8104ce1f>] ? cpu_idle+0x5f/0xd0
---[ end trace 53996d982d5b59b8 ]---