Subject: Re: [vserver] Suddenly loosing network-interfaces on some hosts
From: Michael Hoffrath <m.hoffrath@clano-it.com>
Date: Tue, 23 Nov 2010 14:51:39 +0100

 Tue, 23 Nov 2010 14:51:39 +0100

> On Tue, Nov 23, 2010 at 12:47:05PM +0100, Michael Hoffrath wrote:
>> Hello,
> 
>> Summary: Guests suddenly loosing Network-Interfaces.
> 
>> Description:
>> we've noticed a problem on 2 of our hosts, not sure if this could be
>> related to linux-vserver.
> 
>> The System hosting between 5-15 Linux-Vserver running with kernel
>> 2.6.26-2-vserver-amd64.
> 
> known broken kernel (see wiki) don't use it if you don't
> want to live with the known and unknown problems

We will upgrade the machines and keep them in view, thank you.

> 
>> Both Systems suddenly through out this error (kernel.log):
>> Nov 22 11:41:54 kernel: [63954.956587] ------------[ cut here ]------------
>> Nov 22 11:41:54 kernel: [63954.956587] WARNING: at net/core/dst.c:265 dst release+0x23/0x2c()
>> Nov 22 11:41:54 kernel: [63954.956587] Modules linked in: xt multiport iptable filter
ip tables x tables binfmt misc loop snd pcm snd timer snd soundcore snd page alloc serio
raw psmouse button pcspkr evdev dcdbas ext3 jbd mbcache dm mirror dm log dm snapshot
dm mod raid1 md mod sg sr mod cdrom sd mod ide pci generic ide core ata generic ata
piix libata scsi mod dock ehci hcd uhci hcd tg3 thermal processor fan thermal sys [last
unloaded: scsi wait scan]
>> Nov 22 11:41:54 kernel: [63954.956587] Pid: 16786, comm: java Tainted: G        W
2.6.26-2-vserver-amd64 #1
>> Nov 22 11:41:54 kernel: [63954.956587] 
>> Nov 22 11:41:54 kernel: [63954.956587] Call Trace:
>> Nov 22 11:41:54 kernel: [63954.956587]  [<ffffffff80234e68>] warn on slowpath+0x51/0x7a
>> Nov 22 11:41:54 kernel: [63954.956587]  [<ffffffff803e65c0>]   ip route output key+0x1cf/0x8f8
>> Nov 22 11:41:54 kernel: [63954.956587]  [<ffffffff803e6d02>] ip route output flow+0x19/0x1de
>> Nov 22 11:41:54 kernel: [63954.956587]  [<ffffffff803ce4c4>] dst release+0x23/0x2c
>> Nov 22 11:41:54 kernel: [63954.956587]  [<ffffffff804096a0>] udp sendmsg+0x4fc/0x5d3
>> Nov 22 11:41:54 kernel: [63954.956587]  [<ffffffff803bf69d>] sock sendmsg+0xed/0x1da
>> Nov 22 11:41:54 kernel: [63954.956587]  [<ffffffff80246f91>] autoremove wake function+0x0/0x2e
>> Nov 22 11:41:54 kernel: [63954.956587]  [<ffffffff80259c6f>] futex wake+0x74/0xfa
>> Nov 22 11:41:54 kernel: [63954.956587]  [<ffffffff8025a940>] do futex+0xa6/0x844
>> Nov 22 11:41:54 kernel: [63954.956587]  [<ffffffff803c0272>] sys sendto+0xf3/0x127
>> Nov 22 11:41:54 kernel: [63954.956587]  [<ffffffff80212507>] read tsc+0x9/0x20
>> Nov 22 11:41:54 kernel: [63954.956587]  [<ffffffff80254d7a>] getnstimeofday+0x39/0x98
>> Nov 22 11:41:54 kernel: [63954.956587]  [<ffffffff80249b32>] ktime get ts+0x22/0x4b
>> Nov 22 11:41:54 kernel: [63954.956587]  [<ffffffff8020beda>] system call after swapgs+0x8a/0x8f
>> Nov 22 11:41:54 kernel: [63954.956587] 
>> Nov 22 11:41:54 kernel: [63954.956587] ---[ end trace 9e9501ed44896e2c ]---
> 
> looks unrelated at first glance, but might fall into the
> unknown brokenness category ...
> 
>> And some of the Guests lost their network-interface. After a restart
>> of the guest the network interface comes back online, but this problem
>> occurred several times.
> 
> more likely that you have a primary/secondaries issue 
> here ... 
> 
>> Here are some more information:
>> 
>> sh testme.sh 
>> Linux-VServer Test [V0.17] Copyright (C) 2003-2006 H.Poetzl
>> chcontext is working.
>> chbind is working.
>> Linux 2.6.26-2-vserver-amd64 #1 SMP Thu Sep 16 16:20:47 UTC 2010 x86 64
>> Ea 0.30.216 236/glibc (DSa) <v13,net,v21,v22,v23,netv2>
>     ~~~~~~~~
> there is no 0.30.216 release (yet)

Thats the unmodified output of this script!
> 
> best luck,
> Herbert
> 
>> VCI: 0002:0303 236 13000ff1 (KtTbsPHIiW)
>> ---
>> [000]# succeeded.
>> [001]# succeeded.
>> [011]# succeeded.
>> [031]# succeeded.
>> [101]# succeeded.
>> [102]# succeeded.
>> [201]# succeeded.
>> [202]# succeeded.
>> 
>> ./testfs.sh -t -x -y -z -D /dev/loop0 -M /mnt
>> Linux-VServer FS Test [V0.23] Copyright (C) 2005-2009 H.Poetzl
>> Linux 2.6.26-2-vserver-amd64 x86 64/0.30.216
>> VCI:  0002:0303 236 13000ff1 (ID24)
>> ---
>> testing ext2 filesystem ...
>> [000]. 
>> 
>> [001]. [002]. (ext2 format)
>> tag related tests ...
>> [011]. [012]. [014]. [015]. [019]. 
>> [020]. [021]. [022]. [023]. [024]. [025]. [026]. [027]. [028]. 
>> [033]. [034]. [035]. [037]. [045]. [047]. 
>> xattr related tests ...
>> [101]. [103]. [104]. [106]. [107]. [109]. 
>> [112]. [113]. [114]. [115]. [116]* [118]. [119]. 
>> [122]. [123]. [124]. [125]. [127]. [128]. [129]. 
>> [131]. [132]. [133]. [134]. [135]. [138]. [139]. 
>> [148]. [149]. 
>> disk limit related tests ...
>> [201]. [202]. [203]. [204]. [205]. [206]. [207]. [208]. 
>> [211]. [212]. [213]. [222]. [223]. [231]. [232]. [233]. [239]. 
>> [999]. 
>> ---
>> testing ext3 filesystem ...
>> [000]. [001]. [002]. (ext3 format)
>> tag related tests ...
>> [011]. [012]. [014]. [015]. [019]. 
>> [020]. [021]. [022]. [023]. [024]. [025]. [026]. [027]. [028]. 
>> [033]. [034]. [035]. [037]. [045]. [047]. 
>> xattr related tests ...
>> [101]. [103]. [104]. [106]. [107]. [109]. 
>> [112]. [113]. [114]. [115]. [116]* [118]. [119]. 
>> [122]. [123]. [124]. [125]. [127]. [128]. [129]. 
>> [131]. [132]. [133]. [134]. [135]. [138]. [139]. 
>> [148]. [149]. 
>> disk limit related tests ...
>> [201]. [202]. [203]. [204]. [205]. [206]. [207]. [208]. 
>> [211]. [212]. [213]. [222]. [223]. [231]. [232]. [233]. [239]. 
>> [999]. 
>> ---
>> testing ext4 filesystem ...
>> [000]. [001]* 
>> ---
>> testing xfs filesystem ...
>> [000]* (xfs format failed)
>> ---
>> testing reiser filesystem ...
>> [000]* (reiserfs format failed)
>> ---
>> testing jfs filesystem ...
>> [000]* (jfs format failed)
>> 
>> Kernel:
>> Linux version 2.6.26-2-vserver-amd64 (Debian 2.6.26-25lenny1) (dannf@debian.org)
(gcc version 4.1.3 20080704 (prerelease) (Debian 4.1.2-25)) #1 SMP Thu Sep 16 16:20:47
UTC 2010
>> 
>> Processor:
>> processor       : 3
>> vendor id       : GenuineIntel
>> cpu family      : 6
>> model           : 23
>> model name      : Intel(R) Xeon(R) CPU           X3330  @ 2.66GHz
>> stepping        : 10
>> cpu MHz         : 2666.859
>> cache size      : 3072 KB
>> physical id     : 0
>> siblings        : 4
>> core id         : 3
>> cpu cores       : 4
>> apicid          : 3
>> initial apicid  : 3
>> fpu             : yes
>> fpu exception   : yes
>> cpuid level     : 13
>> wp              : yes
>> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat
pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant tsc arch
perfmon pebs bts rep good pni monitor ds cpl vmx smx est tm2 ssse3 cx16 xtpr sse4 1
lahf lm
>> bogomips        : 5333.77
>> clflush size    : 64
>> cache alignment : 64
>> address sizes   : 36 bits physical, 48 bits virtual
>> power management:
>> 
>> Modules:
>> ext2 68880 0 - Live 0xffffffffa01e6000
>> xt multiport 7424 1 - Live 0xffffffffa01e3000
>> iptable filter 7424 1 - Live 0xffffffffa01e0000
>> ip tables 21520 1 iptable filter, Live 0xffffffffa01d9000
>> x tables 25224 2 xt multiport,ip tables, Live 0xffffffffa0195000
>> binfmt misc 13580 1 - Live 0xffffffffa0190000
>> loop 19596 1 - Live 0xffffffffa018a000
>> snd pcm 81800 0 - Live 0xffffffffa01c4000
>> snd timer 25744 1 snd pcm, Live 0xffffffffa01bc000
>> snd 63688 2 snd pcm,snd timer, Live 0xffffffffa01ab000
>> soundcore 12064 1 snd, Live 0xffffffffa01a7000
>> snd page alloc 13072 1 snd pcm, Live 0xffffffffa01a2000
>> serio raw 9988 0 - Live 0xffffffffa019e000
>> psmouse 42268 0 - Live 0xffffffffa017e000
>> button 11680 0 - Live 0xffffffffa017a000
>> pcspkr 7040 0 - Live 0xffffffffa0177000
>> evdev 14208 0 - Live 0xffffffffa0172000
>> dcdbas 11952 0 - Live 0xffffffffa016e000
>> ext3 127632 28 - Live 0xffffffffa014d000
>> jbd 51240 1 ext3, Live 0xffffffffa013f000
>> mbcache 12804 2 ext2,ext3, Live 0xffffffffa013a000
>> dm mirror 20608 0 - Live 0xffffffffa0133000
>> dm log 13956 1 dm mirror, Live 0xffffffffa012e000
>> dm snapshot 19400 0 - Live 0xffffffffa0128000
>> dm mod 59376 56 dm mirror,dm log,dm snapshot, Live 0xffffffffa0118000
>> raid1 24192 4 - Live 0xffffffffa0111000
>> md mod 80292 5 raid1, Live 0xffffffffa00a6000
>> sg 36448 0 - Live 0xffffffffa0107000
>> sr mod 19652 0 - Live 0xffffffffa0101000
>> cdrom 37928 1 sr mod, Live 0xffffffffa00f6000
>> sd mod 29376 10 - Live 0xffffffffa00ed000
>> ide pci generic 9220 0 [permanent], Live 0xffffffffa00e9000
>> ide core 128284 1 ide pci generic, Live 0xffffffffa00c8000
>> ata generic 10116 0 - Live 0xffffffffa00c4000
>> ata piix 22916 8 - Live 0xffffffffa00bd000
>> libata 165600 2 ata generic,ata piix, Live 0xffffffffa007c000
>> scsi mod 161016 4 sg,sr mod,sd mod,libata, Live 0xffffffffa0053000
>> dock 14112 1 libata, Live 0xffffffffa004e000
>> ehci hcd 36108 0 - Live 0xffffffffa0042000
>> uhci hcd 25760 0 - Live 0xffffffffa0038000
>> tg3 97156 0 - Live 0xffffffffa001d000
>> thermal 22688 0 - Live 0xffffffffa0016000
>> processor 42304 1 thermal, Live 0xffffffffa000a000
>> fan 9352 0 - Live 0xffffffffa0006000
>> thermal sys 17728 3 thermal,processor,fan, Live 0xffffffffa0000000
>> 
>> Kind Regards,
>> Michael