Subject: Kernel 3.18.7 hangs when inserting netconsle module on DELL M620 VRTX Blade
From: Urban Loesch <bind@enas.net>
Date: Tue, 07 Apr 2015 14:31:51 +0200

Hi,

I'have installed a new DELL VRTX M620 Blade with self compiled kernel 3.18.7 and the
recent linux-vserver patch.

After system startup I tried to activate the kernel netconsole with remote logging enabled.

I executed the following command and the shell i issued it becomes unresponsive and
it hangs.
#  modprobe netconsole netconsole="@/eth0,514@10.1.10.197/00:10:db:fc:60:0c"

The system load increases slowly and the CPU #3 uses 100% of soft irq. Only a soft reset
witohut loading the netconsole
module solves the issue.

I found the following error in the kernel log:

...
Apr  7 12:33:23 server2 kernel: [  180.582285] ------------[ cut here ]------------
Apr  7 12:33:23 server2 kernel: [  180.582292] WARNING: CPU: 3 PID: 2978 at kernel/softirq.c:147
__local_bh_enable_ip+0x72/0xa0()
Apr  7 12:33:23 server2 kernel: [  180.582303] CPU: 3 PID: 2978 Comm: modprobe Not tainted
3.18.7-vs2.3.7.4-em64t-efigpt #2
Apr  7 12:33:23 server2 kernel: [  180.582304] Hardware name: Dell Inc. PowerEdge M620/0NJVT7,
BIOS 2.4.3 07/02/2014
Apr  7 12:33:23 server2 kernel: [  180.582306]  0000000000000009 ffff881fc4a239e8 ffffffff817438ac
000000001b361b36
Apr  7 12:33:23 server2 kernel: [  180.582307]  0000000000000000 ffff881fc4a23a28 ffffffff81051f8c
ffffffff81f49040
Apr  7 12:33:23 server2 kernel: [  180.582308]  0000000000000200 ffff881fcf0f0dd4 ffff881fcf0f0d58
0000000000000000
Apr  7 12:33:23 server2 kernel: [  180.582308] Call Trace:
Apr  7 12:33:23 server2 kernel: [  180.582313]  [<ffffffff817438ac>] dump_stack+0x46/0x58
Apr  7 12:33:23 server2 kernel: [  180.582315]  [<ffffffff81051f8c>] warn_slowpath_common+0x8c/0xc0
Apr  7 12:33:23 server2 kernel: [  180.582316]  [<ffffffff81051fda>] warn_slowpath_null+0x1a/0x20
Apr  7 12:33:23 server2 kernel: [  180.582318]  [<ffffffff81055fa2>] __local_bh_enable_ip+0x72/0xa0
Apr  7 12:33:23 server2 kernel: [  180.582322]  [<ffffffff8174992b>] _raw_spin_unlock_bh+0x1b/0x20
Apr  7 12:33:23 server2 kernel: [  180.582335]  [<ffffffffa00b7f33>] bnx2x_poll+0x83/0x3c0
[bnx2x]
Apr  7 12:33:23 server2 kernel: [  180.582338]  [<ffffffff81667360>] netpoll_poll_dev+0x110/0x1b0
Apr  7 12:33:23 server2 kernel: [  180.582340]  [<ffffffff81667567>] netpoll_send_skb_on_dev+0x167/0x240
Apr  7 12:33:23 server2 kernel: [  180.582341]  [<ffffffff81667912>] netpoll_send_udp+0x2d2/0x400
Apr  7 12:33:23 server2 kernel: [  180.582343]  [<ffffffffa017985f>] write_msg+0xcf/0x110
[netconsole]
Apr  7 12:33:23 server2 kernel: [  180.582347]  [<ffffffff8109e22b>] call_console_drivers.constprop.27+0x9b/0x100
Apr  7 12:33:23 server2 kernel: [  180.582348]  [<ffffffff8109f28a>] console_unlock+0x3ba/0x440
Apr  7 12:33:23 server2 kernel: [  180.582350]  [<ffffffff810a062a>] register_console+0x29a/0x360
Apr  7 12:33:23 server2 kernel: [  180.582351]  [<ffffffffa0190000>] ? 0xffffffffa0190000
Apr  7 12:33:23 server2 kernel: [  180.582353]  [<ffffffffa01901c5>] init_netconsole+0x1c5/0x1000
[netconsole]
Apr  7 12:33:23 server2 kernel: [  180.582356]  [<ffffffff810002dc>] do_one_initcall+0x8c/0x1c0
Apr  7 12:33:23 server2 kernel: [  180.582359]  [<ffffffff81180ca2>] ? __vunmap+0xc2/0x110
Apr  7 12:33:23 server2 kernel: [  180.582361]  [<ffffffff810d7e7d>] load_module+0x1dbd/0x25b0
Apr  7 12:33:23 server2 kernel: [  180.582363]  [<ffffffff810d4660>] ? show_initstate+0x60/0x60
Apr  7 12:33:23 server2 kernel: [  180.582365]  [<ffffffff8174ba1f>] ? page_fault+0x1f/0x30
Apr  7 12:33:23 server2 kernel: [  180.582366]  [<ffffffff810d870a>] SyS_init_module+0x9a/0xc0
Apr  7 12:33:23 server2 kernel: [  180.582368]  [<ffffffff8174a112>] system_call_fastpath+0x12/0x17
Apr  7 12:33:23 server2 kernel: [  180.582369] ---[ end trace f22e6c017282076b ]---
...

The system runs with 256GB RAM:
#  free -m
             total       used       free     shared    buffers     cached
Mem:        257918       1834     256084          0         19         44
-/+ buffers/cache:       1770     256148
Swap:         7627          0       7627

And has 2 six-core cpu's:
#  lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                24
On-line CPU(s) list:   0-23
Thread(s) per core:    2
Core(s) per socket:    6
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 62
Stepping:              4
CPU MHz:               2599.966
BogoMIPS:              5200.39
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              15360K
NUMA node0 CPU(s):     0,2,4,6,8,10,12,14,16,18,20,22
NUMA node1 CPU(s):     1,3,5,7,9,11,13,15,17,19,21,23


Currently there is running a memtest to be sure that there is no ram issue. But it takes
a while....

A strange thing: the second M620 blade which is installed on the VRTX has no problems
with
inserting the netconsole module. At least until today.

Have you an idea how I can solve this?

Many thanks
Urban Loesch