Subject: Re: [vserver] opteron server dies with vserver patch.
From: Herbert Poetzl <herbert@13thfloor.at>
Date: Mon, 8 Aug 2011 16:07:54 +0200

On Mon, Aug 08, 2011 at 01:24:48PM +0200, Pawel Sikora wrote:
> Hi,

> i have few dual opteron-6128 machines which are part of eda
> software auto-test farm. these machines don't host any vserver
> guests but installed kernel has vserver-patch applied by
> vendor. on vanilla kernel 2.6.38 (and newer) with vserver
> patch applied machines lock in some way after few hours of
> quite heavy computing + i/o (avg. load ~16). on 2.6.37.y with
> vserver everything works fine and on pure vanilla kernels also
> everything works fine.

> on the ipmi console i've logged only kernel flood:
> https://lkml.org/lkml/2011/5/23/398 this crash is reproducible
> on 2.6.38.y/2.6.39.y/3.0.y with the latest vserver patches.

the lkml posting says vserver/grsec, now does this mean
 o you are testing with mainline + Linux-VServer patch
 o you are testing with mainline + grsec patch
 o you are testing with a combined patch?

> # cat /proc/version
> Linux version 3.0.0-vs2.3.1-pre8-dirty (pawels@hal) (gcc version 4.5.3 20110515 (release)
(PLD-Linux) ) #2 SMP Fri Aug 5 09:48:25 CEST 2011

what does -dirty mean here?

thanks,
Herbert

> # cat /proc/cpuinfo

> processor       : 0...15
> vendor_id       : AuthenticAMD
> cpu family      : 16
> model           : 9
> model name      : AMD Opteron(tm) Processor 6128
> stepping        : 1
> cpu MHz         : 2000.149
> cache size      : 512 KB
> physical id     : 1
> siblings        : 8
> core id         : 3
> cpu cores       : 8
> apicid          : 39
> initial apicid  : 23
> fpu             : yes
> fpu_exception   : yes
> cpuid level     : 5
> wp              : yes
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
> mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx
> mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc
> rep_good nopl nonstop_tsc extd_apicid amd_dcm pni monitor cx16
> popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a
> misalignsse 3dnowprefetch osvw ibs skinit wdt nodeid_msr npt
> lbrv svm_lock nrip_save pausefilter

> bogomips        : 4001.84
> TLB size        : 1024 4K pages
> clflush size    : 64
> cache_alignment : 64
> address sizes   : 48 bits physical, 48 bits virtual
> power management: ts ttp tm stc 100mhzsteps hwpstate

> # cat /proc/modules 
> ext2 72951 0 - Live 0xffffffffa04d8000
> loop 27624 1 - Live 0xffffffffa04ec000
> nfs 312073 1 - Live 0xffffffffa048a000
> fscache 46134 1 nfs, Live 0xffffffffa0476000
> binfmt_misc 17456 1 - Live 0xffffffffa03fa000
> nfsd 310366 13 - Live 0xffffffffa0429000
> lockd 84640 2 nfs,nfsd, Live 0xffffffffa0413000
> nfs_acl 12741 2 nfs,nfsd, Live 0xffffffffa03f5000
> auth_rpcgss 47785 2 nfs,nfsd, Live 0xffffffffa0406000
> sunrpc 216671 22 nfs,nfsd,lockd,nfs_acl,auth_rpcgss, Live 0xffffffffa03bf000
> exportfs 12851 1 nfsd, Live 0xffffffffa03a6000
> ipmi_si 52811 0 - Live 0xffffffffa03b1000
> ipmi_devintf 17522 0 - Live 0xffffffffa03ab000
> ipmi_msghandler 43683 2 ipmi_si,ipmi_devintf, Live 0xffffffffa039a000
> sch_sfq 17212 2 - Live 0xffffffffa038d000
> iptable_nat 13182 1 - Live 0xffffffffa0395000
> nf_nat 24588 1 iptable_nat, Live 0xffffffffa037e000
> nf_conntrack_ipv4 19478 3 iptable_nat,nf_nat, Live 0xffffffffa0387000
> nf_conntrack 76827 3 iptable_nat,nf_nat,nf_conntrack_ipv4, Live 0xffffffffa036a000
> nf_defrag_ipv4 12673 1 nf_conntrack_ipv4, Live 0xffffffffa0365000
> iptable_filter 12810 0 - Live 0xffffffffa0360000
> xt_TCPMSS 12715 2 - Live 0xffffffffa02f0000
> xt_tcpudp 12603 2 - Live 0xffffffffa02e2000
> iptable_mangle 12695 1 - Live 0xffffffffa02dd000
> ip_tables 26705 3 iptable_nat,iptable_filter,iptable_mangle, Live 0xffffffffa02d1000
> ip6table_filter 12815 0 - Live 0xffffffffa0217000
> ip6_tables 26996 1 ip6table_filter, Live 0xffffffffa027a000
> x_tables 27634 8 iptable_nat,iptable_filter,xt_TCPMSS,xt_tcpudp,iptable_mangle,ip_tables,ip6table_filter,ip6_tables,
Live 0xffffffffa0262000
> ext4 403044 1 - Live 0xffffffffa02fc000
> jbd2 84403 1 ext4, Live 0xffffffffa02bb000
> crc16 12559 1 ext4, Live 0xffffffffa0212000
> raid0 17106 1 - Live 0xffffffffa0245000
> dm_mod 75794 0 - Live 0xffffffffa0296000
> autofs4 28311 11 - Live 0xffffffffa01e6000
> dummy 12955 0 - Live 0xffffffffa01a0000
> usbhid 46436 0 - Live 0xffffffffa02ae000
> hid 93560 1 usbhid, Live 0xffffffffa01fa000
> ide_cd_mod 35415 0 - Live 0xffffffffa01dc000
> cdrom 41847 1 ide_cd_mod, Live 0xffffffffa01d0000
> ata_generic 12899 0 - Live 0xffffffffa019b000
> pata_acpi 13027 0 - Live 0xffffffffa0186000
> pata_atiixp 13165 0 - Live 0xffffffffa015d000
> sp5100_tco 13697 0 - Live 0xffffffffa014f000
> ide_pci_generic 12866 0 - Live 0xffffffffa014a000
> ohci_hcd 31523 0 - Live 0xffffffffa017d000
> psmouse 68641 0 - Live 0xffffffffa0284000
> ghes 13625 0 - Live 0xffffffffa01cb000
> ssb 56258 1 ohci_hcd, Live 0xffffffffa026b000
> mmc_core 85395 1 ssb, Live 0xffffffffa024c000
> igb 115008 0 - Live 0xffffffffa0227000
> i2c_piix4 13167 0 - Live 0xffffffffa0158000
> amd64_edac_mod 28622 0 - Live 0xffffffffa01f2000
> pcmcia 43683 1 ssb, Live 0xffffffffa0171000
> dca 14562 1 igb, Live 0xffffffffa00de000
> pcspkr 12702 0 - Live 0xffffffffa00e3000
> i2c_core 28557 1 i2c_piix4, Live 0xffffffffa0142000
> hed 13102 1 ghes, Live 0xffffffffa01a9000
> edac_core 51596 6 amd64_edac_mod, Live 0xffffffffa0134000
> button 13611 0 - Live 0xffffffffa011c000
> pcmcia_core 21861 1 pcmcia, Live 0xffffffffa02f5000
> atiixp 12917 0 - Live 0xffffffffa02eb000
> processor 30161 0 - Live 0xffffffffa021e000
> ide_core 108026 3 ide_cd_mod,ide_pci_generic,atiixp, Live 0xffffffffa01af000
> serio_raw 13166 0 - Live 0xffffffffa012f000
> ehci_hcd 44738 0 - Live 0xffffffffa018f000
> edac_mce_amd 22882 1 amd64_edac_mod, Live 0xffffffffa016a000
> k10temp 13119 0 - Live 0xffffffffa0162000
> hwmon 12886 1 k10temp, Live 0xffffffffa0117000
> sg 36330 0 - Live 0xffffffffa0125000
> usbcore 159860 4 usbhid,ohci_hcd,ehci_hcd, Live 0xffffffffa00ee000
> evdev 17753 0 - Live 0xffffffffa00e8000
> sd_mod 44764 16 - Live 0xffffffffa00d2000
> crc_t10dif 12483 1 sd_mod, Live 0xffffffffa0083000
> raid1 30545 1 - Live 0xffffffffa00ac000
> md_mod 106323 4 raid0,raid1, Live 0xffffffffa00b5000
> ext3 138195 1 - Live 0xffffffffa0089000
> jbd 55191 1 ext3, Live 0xffffffffa0074000
> mbcache 14317 3 ext2,ext4,ext3, Live 0xffffffffa0031000
> ahci 25854 12 - Live 0xffffffffa006c000
> libahci 25686 1 ahci, Live 0xffffffffa0029000
> libata 197408 5 ata_generic,pata_acpi,pata_atiixp,ahci,libahci, Live 0xffffffffa003a000
> scsi_mod 161130 3 sg,sd_mod,libata, Live 0xffffffffa0000000

> # df -h
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/md0              7.6G  3.6G  3.7G  50% /
> /dev/md1              1.9T  122G  1.8T   7% /home/atest
> none                   32G     0   32G   0% /dev/shm
> nexus:/R0             1.8T  531G  1.2T  31% /remote/nexus/R0

> # cat /proc/mdstat
> Personalities : [raid1] [raid0]
> md1 : active raid0 sda3[0] sdd3[3] sdc3[2] sdb3[1]
>       2047993856 blocks super 1.2 512k chunks

> md0 : active raid1 sda1[0] sdd1[3] sdc1[2] sdb1[1]
>       8000256 blocks [4/4] [UUUU]

> # /bin/bash testme.sh
> Linux-VServer Test [V0.17] Copyright (C) 2003-2006 H.Poetzl
> utility 'vserver' could not be found.

> # ./testfs.sh -t -x -y -z -D /dev/loop0 -M /mnt
> Linux-VServer FS Test [V0.23] Copyright (C) 2005-2009 H.Poetzl
> Linux 3.0.0-vs2.3.1-pre8-dirty x86_64/
> VCI:  0002:0308 236 13011f11 (ID24)
> ---
> testing ext2 filesystem ...
> [000]. [001]. [002]. (ext2 format)
> tag related tests ...
> [011]. [012]* [014]* [015]* [019]. 
> [020]* [021]. [022]* [023]. [024]* [025]* [026]* [027]* [028]* 
> [033]* [034]* [035]* [037]* [045]* [047]* 
> xattr related tests ...
> [101]. [103]* [104]* [106]* [107]* [109]
> [112]* [113]* [114]. [115]* [116]* [118]* [119]*
> [122]* [123]* [124]* [125]. [127]* [128]* [129]*
> [131]* [132]. [133]* [134]* [135]* [138]. [139].
> [148]. [149].
> disk limit related tests ...
> [201]* [202]* [203]. [204]* [205]. [206]. [207]. [208].
> [211]* [212]* [213]. [222]. [223]. [231]. [232]. [233]. [239].
> [999].
> ---
> testing ext3 filesystem ...
> [000]. [001]. [002]. (ext3 format)
> tag related tests ...
> [011]. [012]* [014]* [015]* [019].
> [020]* [021]. [022]* [023]. [024]* [025]* [026]* [027]* [028]*
> [033]* [034]* [035]* [037]* [045]* [047]*
> xattr related tests ...
> [101]. [103]* [104]* [106]* [107]* [109]*
> [112]* [113]* [114]. [115]* [116]* [118]* [119]*
> [122]* [123]* [124]* [125]. [127]* [128]* [129]*
> [131]* [132]. [133]* [134]* [135]* [138]. [139].
> [148]. [149].
> disk limit related tests ...
> [201]* [202]* [203]. [204]* [205]. [206]. [207]. [208].
> [211]* [212]* [213]. [222]. [223]. [231]. [232]. [233]. [239].
> [999].
> ---
> testing ext4 filesystem ...
> [000]. [001]. [002]. (ext4 format)
> tag related tests ...
> [011]. [012]* [014]* [015]* [019].
> [020]* [021]. [022]* [023]. [024]* [025]* [026]* [027]* [028]*
> [033]* [034]* [035]* [037]* [045]* [047]*
> xattr related tests ...
> [101]. [103]* [104]* [106]* [107]* [109]*
> [112]* [113]* [114]. [115]* [116]* [118]* [119]*
> [122]* [123]* [124]* [125]. [127]* [128]* [129]*
> [131]* [132]. [133]* [134]* [135]* [138]. [139].
> [148]. [149].
> disk limit related tests ...
> [201]* [202]* [203]. [204]* [205]. [206]. [207]. [208].
> [211]* [212]* [213]. [222]. [223]. [231]. [232]. [233]. [239].
> [999].
> ---
> testing xfs filesystem ...
> [000]* (xfs format failed)
> ---
> testing reiser filesystem ...
> [000]* (reiserfs format failed)
> ---
> testing jfs filesystem ...
> [000]* (jfs format failed)