Subject: Suddenly loosing network-interfaces on some hosts
From: Michael Hoffrath <m.hoffrath@clano-it.com>
Date: Tue, 23 Nov 2010 12:47:05 +0100

 Tue, 23 Nov 2010 12:47:05 +0100
Hello,

Summary: Guests suddenly loosing Network-Interfaces.

Description:
we've noticed a problem on 2 of our hosts, not sure if this could be related to linux-vserver.

The System hosting between 5-15 Linux-Vserver running with kernel 2.6.26-2-vserver-amd64.

Both Systems suddenly through out this error (kernel.log):
Nov 22 11:41:54 kernel: [63954.956587] ------------[ cut here ]------------
Nov 22 11:41:54 kernel: [63954.956587] WARNING: at net/core/dst.c:265 dst release+0x23/0x2c()
Nov 22 11:41:54 kernel: [63954.956587] Modules linked in: xt multiport iptable filter
ip tables x tables binfmt misc loop snd pcm snd timer snd soundcore snd page alloc serio
raw psmouse button pcspkr evdev dcdbas ext3 jbd mbcache dm mirror dm log dm snapshot
dm mod raid1 md mod sg sr mod cdrom sd mod ide pci generic ide core ata generic ata
piix libata scsi mod dock ehci hcd uhci hcd tg3 thermal processor fan thermal sys [last
unloaded: scsi wait scan]
Nov 22 11:41:54 kernel: [63954.956587] Pid: 16786, comm: java Tainted: G        W 2.6.26-2-vserver-amd64
#1
Nov 22 11:41:54 kernel: [63954.956587] 
Nov 22 11:41:54 kernel: [63954.956587] Call Trace:
Nov 22 11:41:54 kernel: [63954.956587]  [<ffffffff80234e68>] warn on slowpath+0x51/0x7a
Nov 22 11:41:54 kernel: [63954.956587]  [<ffffffff803e65c0>]   ip route output key+0x1cf/0x8f8
Nov 22 11:41:54 kernel: [63954.956587]  [<ffffffff803e6d02>] ip route output flow+0x19/0x1de
Nov 22 11:41:54 kernel: [63954.956587]  [<ffffffff803ce4c4>] dst release+0x23/0x2c
Nov 22 11:41:54 kernel: [63954.956587]  [<ffffffff804096a0>] udp sendmsg+0x4fc/0x5d3
Nov 22 11:41:54 kernel: [63954.956587]  [<ffffffff803bf69d>] sock sendmsg+0xed/0x1da
Nov 22 11:41:54 kernel: [63954.956587]  [<ffffffff80246f91>] autoremove wake function+0x0/0x2e
Nov 22 11:41:54 kernel: [63954.956587]  [<ffffffff80259c6f>] futex wake+0x74/0xfa
Nov 22 11:41:54 kernel: [63954.956587]  [<ffffffff8025a940>] do futex+0xa6/0x844
Nov 22 11:41:54 kernel: [63954.956587]  [<ffffffff803c0272>] sys sendto+0xf3/0x127
Nov 22 11:41:54 kernel: [63954.956587]  [<ffffffff80212507>] read tsc+0x9/0x20
Nov 22 11:41:54 kernel: [63954.956587]  [<ffffffff80254d7a>] getnstimeofday+0x39/0x98
Nov 22 11:41:54 kernel: [63954.956587]  [<ffffffff80249b32>] ktime get ts+0x22/0x4b
Nov 22 11:41:54 kernel: [63954.956587]  [<ffffffff8020beda>] system call after swapgs+0x8a/0x8f
Nov 22 11:41:54 kernel: [63954.956587] 
Nov 22 11:41:54 kernel: [63954.956587] ---[ end trace 9e9501ed44896e2c ]---

And some of the Guests lost their network-interface. After a restart of the guest the
network interface comes back online, but this problem occurred several times.

Here are some more information:

sh testme.sh 
Linux-VServer Test [V0.17] Copyright (C) 2003-2006 H.Poetzl
chcontext is working.
chbind is working.
Linux 2.6.26-2-vserver-amd64 #1 SMP Thu Sep 16 16:20:47 UTC 2010 x86 64
Ea 0.30.216 236/glibc (DSa) <v13,net,v21,v22,v23,netv2>
VCI: 0002:0303 236 13000ff1 (KtTbsPHIiW)
---
[000]# succeeded.
[001]# succeeded.
[011]# succeeded.
[031]# succeeded.
[101]# succeeded.
[102]# succeeded.
[201]# succeeded.
[202]# succeeded.

./testfs.sh -t -x -y -z -D /dev/loop0 -M /mnt
Linux-VServer FS Test [V0.23] Copyright (C) 2005-2009 H.Poetzl
Linux 2.6.26-2-vserver-amd64 x86 64/0.30.216
VCI:  0002:0303 236 13000ff1 (ID24)
---
testing ext2 filesystem ...
[000]. 

[001]. [002]. (ext2 format)
tag related tests ...
[011]. [012]. [014]. [015]. [019]. 
[020]. [021]. [022]. [023]. [024]. [025]. [026]. [027]. [028]. 
[033]. [034]. [035]. [037]. [045]. [047]. 
xattr related tests ...
[101]. [103]. [104]. [106]. [107]. [109]. 
[112]. [113]. [114]. [115]. [116]* [118]. [119]. 
[122]. [123]. [124]. [125]. [127]. [128]. [129]. 
[131]. [132]. [133]. [134]. [135]. [138]. [139]. 
[148]. [149]. 
disk limit related tests ...
[201]. [202]. [203]. [204]. [205]. [206]. [207]. [208]. 
[211]. [212]. [213]. [222]. [223]. [231]. [232]. [233]. [239]. 
[999]. 
---
testing ext3 filesystem ...
[000]. [001]. [002]. (ext3 format)
tag related tests ...
[011]. [012]. [014]. [015]. [019]. 
[020]. [021]. [022]. [023]. [024]. [025]. [026]. [027]. [028]. 
[033]. [034]. [035]. [037]. [045]. [047]. 
xattr related tests ...
[101]. [103]. [104]. [106]. [107]. [109]. 
[112]. [113]. [114]. [115]. [116]* [118]. [119]. 
[122]. [123]. [124]. [125]. [127]. [128]. [129]. 
[131]. [132]. [133]. [134]. [135]. [138]. [139]. 
[148]. [149]. 
disk limit related tests ...
[201]. [202]. [203]. [204]. [205]. [206]. [207]. [208]. 
[211]. [212]. [213]. [222]. [223]. [231]. [232]. [233]. [239]. 
[999]. 
---
testing ext4 filesystem ...
[000]. [001]* 
---
testing xfs filesystem ...
[000]* (xfs format failed)
---
testing reiser filesystem ...
[000]* (reiserfs format failed)
---
testing jfs filesystem ...
[000]* (jfs format failed)

Kernel:
Linux version 2.6.26-2-vserver-amd64 (Debian 2.6.26-25lenny1) (dannf@debian.org) (gcc
version 4.1.3 20080704 (prerelease) (Debian 4.1.2-25)) #1 SMP Thu Sep 16 16:20:47 UTC
2010

Processor:
processor       : 3
vendor id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Xeon(R) CPU           X3330  @ 2.66GHz
stepping        : 10
cpu MHz         : 2666.859
cache size      : 3072 KB
physical id     : 0
siblings        : 4
core id         : 3
cpu cores       : 4
apicid          : 3
initial apicid  : 3
fpu             : yes
fpu exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat
pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant tsc arch
perfmon pebs bts rep good pni monitor ds cpl vmx smx est tm2 ssse3 cx16 xtpr sse4 1
lahf lm
bogomips        : 5333.77
clflush size    : 64
cache alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

Modules:
ext2 68880 0 - Live 0xffffffffa01e6000
xt multiport 7424 1 - Live 0xffffffffa01e3000
iptable filter 7424 1 - Live 0xffffffffa01e0000
ip tables 21520 1 iptable filter, Live 0xffffffffa01d9000
x tables 25224 2 xt multiport,ip tables, Live 0xffffffffa0195000
binfmt misc 13580 1 - Live 0xffffffffa0190000
loop 19596 1 - Live 0xffffffffa018a000
snd pcm 81800 0 - Live 0xffffffffa01c4000
snd timer 25744 1 snd pcm, Live 0xffffffffa01bc000
snd 63688 2 snd pcm,snd timer, Live 0xffffffffa01ab000
soundcore 12064 1 snd, Live 0xffffffffa01a7000
snd page alloc 13072 1 snd pcm, Live 0xffffffffa01a2000
serio raw 9988 0 - Live 0xffffffffa019e000
psmouse 42268 0 - Live 0xffffffffa017e000
button 11680 0 - Live 0xffffffffa017a000
pcspkr 7040 0 - Live 0xffffffffa0177000
evdev 14208 0 - Live 0xffffffffa0172000
dcdbas 11952 0 - Live 0xffffffffa016e000
ext3 127632 28 - Live 0xffffffffa014d000
jbd 51240 1 ext3, Live 0xffffffffa013f000
mbcache 12804 2 ext2,ext3, Live 0xffffffffa013a000
dm mirror 20608 0 - Live 0xffffffffa0133000
dm log 13956 1 dm mirror, Live 0xffffffffa012e000
dm snapshot 19400 0 - Live 0xffffffffa0128000
dm mod 59376 56 dm mirror,dm log,dm snapshot, Live 0xffffffffa0118000
raid1 24192 4 - Live 0xffffffffa0111000
md mod 80292 5 raid1, Live 0xffffffffa00a6000
sg 36448 0 - Live 0xffffffffa0107000
sr mod 19652 0 - Live 0xffffffffa0101000
cdrom 37928 1 sr mod, Live 0xffffffffa00f6000
sd mod 29376 10 - Live 0xffffffffa00ed000
ide pci generic 9220 0 [permanent], Live 0xffffffffa00e9000
ide core 128284 1 ide pci generic, Live 0xffffffffa00c8000
ata generic 10116 0 - Live 0xffffffffa00c4000
ata piix 22916 8 - Live 0xffffffffa00bd000
libata 165600 2 ata generic,ata piix, Live 0xffffffffa007c000
scsi mod 161016 4 sg,sr mod,sd mod,libata, Live 0xffffffffa0053000
dock 14112 1 libata, Live 0xffffffffa004e000
ehci hcd 36108 0 - Live 0xffffffffa0042000
uhci hcd 25760 0 - Live 0xffffffffa0038000
tg3 97156 0 - Live 0xffffffffa001d000
thermal 22688 0 - Live 0xffffffffa0016000
processor 42304 1 thermal, Live 0xffffffffa000a000
fan 9352 0 - Live 0xffffffffa0006000
thermal sys 17728 3 thermal,processor,fan, Live 0xffffffffa0000000

Kind Regards,