Subject: Kernel bug that crashes system on 3.0.24-vs2.3.2.3-beng?
From: Jean Weisbuch <jean@phpnet.org>
Date: Thu, 03 May 2012 04:08:26 +0200

I had kernel panic on different servers under 3.0.24-vs2.3.2.3-beng 
lately, these seems to be network related.

There is a recurrent message on the kern.log on these before the crashes.

Here is what i got on one of the servers :
Apr 29 21:42:33 localhost kernel: [823558.569637] ------------[ cut here 
]------------
Apr 29 21:42:33 localhost kernel: [823558.569647] WARNING: at 
net/ipv4/tcp_input.c:3375 tcp_ack+0x8c4/0x19b8()
Apr 29 21:42:33 localhost kernel: [823558.569656] Hardware name: X9SCL/X9SCM
Apr 29 21:42:33 localhost kernel: [823558.569657] Modules linked in: 
nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_tcpudp 
iptable_filter ip_tables x_tables nfs lockd auth_rpcgss nfs_acl sunrpc 
acpi_cpufreq mperf cpufreq_conservative cpufreq_userspace 
cpufreq_powersave cpufreq_stats ext3 jbd coretemp ipmi_si 
ipmi_msghandler loop snd_pcm snd_timer snd soundcore intel_agp tpm_tis 
tpm snd_page_alloc i2c_i801 i2c_core button joydev ghes processor 
intel_gtt tpm_bios hed pcspkr evdev thermal_sys psmouse serio_raw ext4 
mbcache jbd2 crc16 usbhid hid sd_mod crc_t10dif ahci libahci libata 
scsi_mod ehci_hcd usbcore e1000e [last unloaded: scsi_wait_scan]
Apr 29 21:42:33 localhost kernel: [823558.569686] Pid: 20900, comm: 
nodeusage.sh Tainted: G        W   3.0.24-vs2.3.2.3-beng #1
Apr 29 21:42:33 localhost kernel: [823558.569687] Call Trace:
Apr 29 21:42:33 localhost kernel: [823558.569688] <IRQ> 
[<ffffffff8103f4ca>] ? warn_slowpath_common+0x78/0x8c
Apr 29 21:42:33 localhost kernel: [823558.569694]  [<ffffffff812b1133>] 
? tcp_ack+0x8c4/0x19b8
Apr 29 21:42:33 localhost kernel: [823558.569697]  [<ffffffff812b22f4>] 
? tcp_rcv_established+0xcd/0x686
Apr 29 21:42:33 localhost kernel: [823558.569699]  [<ffffffff812b96ce>] 
? tcp_v4_do_rcv+0x1a9/0x3b4
Apr 29 21:42:33 localhost kernel: [823558.569701]  [<ffffffff812bab29>] 
? T.1236+0x37/0x69
Apr 29 21:42:33 localhost kernel: [823558.569703]  [<ffffffff812bb445>] 
? tcp_v4_rcv+0x449/0x6e4
Apr 29 21:42:33 localhost kernel: [823558.569705]  [<ffffffff8129e6cf>] 
? ip_local_deliver_finish+0x13d/0x1aa
Apr 29 21:42:33 localhost kernel: [823558.569708]  [<ffffffff81278556>] 
? __netif_receive_skb+0x466/0x499
Apr 29 21:42:33 localhost kernel: [823558.569710]  [<ffffffff812787c6>] 
? netif_receive_skb+0x77/0x7d
Apr 29 21:42:33 localhost kernel: [823558.569712]  [<ffffffff81278d2d>] 
? napi_gro_receive+0x1f/0x2d
Apr 29 21:42:33 localhost kernel: [823558.569714]  [<ffffffff8127889b>] 
? napi_skb_finish+0x1c/0x31
Apr 29 21:42:33 localhost kernel: [823558.569720]  [<ffffffffa0010941>] 
? e1000_clean_rx_irq+0x1f6/0x28c [e1000e]
Apr 29 21:42:33 localhost kernel: [823558.569724]  [<ffffffffa0010dde>] 
? e1000_clean+0x71/0x229 [e1000e]
Apr 29 21:42:33 localhost kernel: [823558.569726]  [<ffffffff81278e5a>] 
? net_rx_action+0xa8/0x206
Apr 29 21:42:33 localhost kernel: [823558.569728]  [<ffffffff81044c1b>] 
? __do_softirq+0xc4/0x1a0
Apr 29 21:42:33 localhost kernel: [823558.569731]  [<ffffffff81092690>] 
? handle_irq_event_percpu+0x166/0x184
Apr 29 21:42:33 localhost kernel: [823558.569734]  [<ffffffff8133655c>] 
? call_softirq+0x1c/0x30
Apr 29 21:42:33 localhost kernel: [823558.569736]  [<ffffffff8100408b>] 
? do_softirq+0x3f/0x79
Apr 29 21:42:33 localhost kernel: [823558.569738]  [<ffffffff810449eb>] 
? irq_exit+0x44/0xb5
Apr 29 21:42:33 localhost kernel: [823558.569740]  [<ffffffff810039d5>] 
? do_IRQ+0x94/0xaa
Apr 29 21:42:33 localhost kernel: [823558.569742]  [<ffffffff81334c93>] 
? common_interrupt+0x13/0x13
Apr 29 21:42:33 localhost kernel: [823558.569743] <EOI> 
[<ffffffff810ea2b7>] ? kmem_cache_free+0x27/0x62
Apr 29 21:42:33 localhost kernel: [823558.569749]  [<ffffffff810d5ebe>] 
? remove_vma+0x64/0x6c
Apr 29 21:42:33 localhost kernel: [823558.569751]  [<ffffffff810d5fa0>] 
? exit_mmap+0xda/0xfb
Apr 29 21:42:33 localhost kernel: [823558.569753]  [<ffffffff8103d496>] 
? mmput+0x36/0xd5
Apr 29 21:42:33 localhost kernel: [823558.569756]  [<ffffffff810fde0b>] 
? flush_old_exec+0x52b/0x60d
Apr 29 21:42:33 localhost kernel: [823558.569759]  [<ffffffff81135830>] 
? load_elf_binary+0x2af/0x1733
Apr 29 21:42:33 localhost kernel: [823558.569762]  [<ffffffff810fd3f2>] 
? get_arg_page+0x4b/0xd9
Apr 29 21:42:33 localhost kernel: [823558.569764]  [<ffffffff810fd760>] 
? copy_strings+0x1fe/0x245
Apr 29 21:42:33 localhost kernel: [823558.569766]  [<ffffffff810feafd>] 
? search_binary_handler+0xa0/0x243
Apr 29 21:42:33 localhost kernel: [823558.569767]  [<ffffffff810fee36>] 
? do_execve_common+0x196/0x2ca
Apr 29 21:42:33 localhost kernel: [823558.569771]  [<ffffffff8102a991>] 
? sys32_execve+0x3f/0x54
Apr 29 21:42:33 localhost kernel: [823558.569773]  [<ffffffff81336c05>] 
? ia32_ptregs_common+0x25/0x4b
Apr 29 21:42:33 localhost kernel: [823558.569775] ---[ end trace 
1cf81a8e50efa328 ]---

And on another :
May  1 12:19:23 localhost kernel: [95833.147842] ------------[ cut here 
]------------
May  1 12:19:23 localhost kernel: [95833.147844] WARNING: at 
net/ipv4/tcp_input.c:2999 tcp_ack+0xe42/0x19b8()
May  1 12:19:23 localhost kernel: [95833.147845] Hardware name: X8STi
May  1 12:19:23 localhost kernel: [95833.147846] Modules linked in: 
acpi_cpufreq mperf cpufreq_userspace cpufreq_stats cpufreq_conservative 
cpufreq_powersave bonding ipt_LOG xt_limit xt_multiport iptable_filter 
ip_tables x_tables ext3 jbd w83627ehf hwmon_vid coretemp ipmi_si 
ipmi_msghandler loop snd_pcm snd_timer snd soundcore i2c_i801 i2c_core 
tpm_tis tpm ioatdma tpm_bios snd_page_alloc i7core_edac evdev dca 
edac_core button processor thermal_sys psmouse pcspkr serio_raw ext4 
mbcache jbd2 crc16 dm_mod raid1 raid0 md_mod sd_mod crc_t10dif usbhid 
hid uhci_hcd ahci libahci libata scsi_mod ehci_hcd usbcore e1000e [last 
unloaded: scsi_wait_scan]
May  1 12:19:23 localhost kernel: [95833.147866] Pid: 1916, comm: 
php5-cgi Tainted: G        W   3.0.24-vs2.3.2.3-beng #1
May  1 12:19:23 localhost kernel: [95833.147867] Call Trace:
May  1 12:19:23 localhost kernel: [95833.147868] <IRQ> 
[<ffffffff8103f4ca>] ? warn_slowpath_common+0x78/0x8c
May  1 12:19:23 localhost kernel: [95833.147871]  [<ffffffff812b16b1>] ? 
tcp_ack+0xe42/0x19b8
May  1 12:19:23 localhost kernel: [95833.147873]  [<ffffffff812b02d4>] ? 
tcp_validate_incoming+0x68/0x255
May  1 12:19:23 localhost kernel: [95833.147876]  [<ffffffff812b27dc>] ? 
tcp_rcv_established+0x5b5/0x686
May  1 12:19:23 localhost kernel: [95833.147878]  [<ffffffff812b96ce>] ? 
tcp_v4_do_rcv+0x1a9/0x3b4
May  1 12:19:23 localhost kernel: [95833.147880]  [<ffffffff812bab29>] ? 
T.1236+0x37/0x69
May  1 12:19:23 localhost kernel: [95833.147881]  [<ffffffff812bb445>] ? 
tcp_v4_rcv+0x449/0x6e4
May  1 12:19:23 localhost kernel: [95833.147884]  [<ffffffffa02a6a19>] ? 
bond_alb_initialize+0x159/0x159 [bonding]
May  1 12:19:23 localhost kernel: [95833.147886]  [<ffffffff8129e6cf>] ? 
ip_local_deliver_finish+0x13d/0x1aa
May  1 12:19:23 localhost kernel: [95833.147888]  [<ffffffff81278556>] ? 
__netif_receive_skb+0x466/0x499
May  1 12:19:23 localhost kernel: [95833.147890]  [<ffffffff812787c6>] ? 
netif_receive_skb+0x77/0x7d
May  1 12:19:23 localhost kernel: [95833.147892]  [<ffffffff81278d2d>] ? 
napi_gro_receive+0x1f/0x2d
May  1 12:19:23 localhost kernel: [95833.147894]  [<ffffffff8127889b>] ? 
napi_skb_finish+0x1c/0x31
May  1 12:19:23 localhost kernel: [95833.147898]  [<ffffffffa0010941>] ? 
e1000_clean_rx_irq+0x1f6/0x28c [e1000e]
May  1 12:19:23 localhost kernel: [95833.147902]  [<ffffffffa0010dde>] ? 
e1000_clean+0x71/0x229 [e1000e]
May  1 12:19:23 localhost kernel: [95833.147904]  [<ffffffff81278e5a>] ? 
net_rx_action+0xa8/0x206
May  1 12:19:23 localhost kernel: [95833.147906]  [<ffffffff81044c1b>] ? 
__do_softirq+0xc4/0x1a0
May  1 12:19:23 localhost kernel: [95833.147908]  [<ffffffff81092690>] ? 
handle_irq_event_percpu+0x166/0x184
May  1 12:19:23 localhost kernel: [95833.147910]  [<ffffffff81044c94>] ? 
__do_softirq+0x13d/0x1a0
May  1 12:19:23 localhost kernel: [95833.147912]  [<ffffffff8133655c>] ? 
call_softirq+0x1c/0x30
May  1 12:19:23 localhost kernel: [95833.147913]  [<ffffffff8100408b>] ? 
do_softirq+0x3f/0x79
May  1 12:19:23 localhost kernel: [95833.147915]  [<ffffffff810449eb>] ? 
irq_exit+0x44/0xb5
May  1 12:19:23 localhost kernel: [95833.147917]  [<ffffffff810039d5>] ? 
do_IRQ+0x94/0xaa
May  1 12:19:23 localhost kernel: [95833.147919]  [<ffffffff81334c93>] ? 
common_interrupt+0x13/0x13
May  1 12:19:23 localhost kernel: [95833.147920] <EOI> 
[<ffffffff81336a83>] ? ia32_do_call+0x13/0x13
May  1 12:19:23 localhost kernel: [95833.147922] ---[ end trace 
43efaa2da94bb3b9 ]---

It seems to be this bug : http://patchwork.ozlabs.org/patch/144844/

I installed linux-image-3.2.15-vs2.3.2.8-beng on the servers that had 
this kernel hoping it wont happen again on these.

Greetings.