Subject: Re: [vserver] opteron server dies with vserver patch.
From: Herbert Poetzl <herbert@13thfloor.at>
Date: Mon, 8 Aug 2011 16:28:20 +0200

On Mon, Aug 08, 2011 at 04:07:54PM +0200, Herbert Poetzl wrote:
> On Mon, Aug 08, 2011 at 01:24:48PM +0200, Pawel Sikora wrote:
>> Hi,

>> i have few dual opteron-6128 machines which are part of eda
>> software auto-test farm. these machines don't host any vserver
>> guests but installed kernel has vserver-patch applied by
>> vendor. on vanilla kernel 2.6.38 (and newer) with vserver
>> patch applied machines lock in some way after few hours of
>> quite heavy computing + i/o (avg. load ~16). on 2.6.37.y with
>> vserver everything works fine and on pure vanilla kernels also
>> everything works fine.

>> on the ipmi console i've logged only kernel flood:
>> https://lkml.org/lkml/2011/5/23/398 this crash is reproducible
>> on 2.6.38.y/2.6.39.y/3.0.y with the latest vserver patches.

> the lkml posting says vserver/grsec, now does this mean
>  o you are testing with mainline + Linux-VServer patch
>  o you are testing with mainline + grsec patch
>  o you are testing with a combined patch?

ah, I forgot, yes there is a way (not the easiest though)
to decode the output your ipmi console recorded:

 - extract the byte sequence listed as Code:
 - search for that sequence in your vmlinux
 - add the offset of <xx> relative to the first byte (44)
 - lookup the resulting address with addr2line or
   compare it with objdump -t vmlinux

note that the traces listed in the lkml posting show at
least 3 different locations where something happened and
I really wonder if those aren't three different perfectly
fine stack traces which just get mangled by your ipmi
setup ... maybe try to explicitely trigger a cpu/kernel
trace with /proc/magic-sysrq and see if it logs properly

best,
Herbert

>> # cat /proc/version
>> Linux version 3.0.0-vs2.3.1-pre8-dirty (pawels@hal) (gcc version 4.5.3 20110515 (release)
(PLD-Linux) ) #2 SMP Fri Aug 5 09:48:25 CEST 2011

> what does -dirty mean here?

> thanks,
> Herbert

>> # cat /proc/cpuinfo

>> processor       : 0...15
>> vendor_id       : AuthenticAMD
>> cpu family      : 16
>> model           : 9
>> model name      : AMD Opteron(tm) Processor 6128
>> stepping        : 1
>> cpu MHz         : 2000.149
>> cache size      : 512 KB
>> physical id     : 1
>> siblings        : 8
>> core id         : 3
>> cpu cores       : 8
>> apicid          : 39
>> initial apicid  : 23
>> fpu             : yes
>> fpu_exception   : yes
>> cpuid level     : 5
>> wp              : yes
>> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
>> mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx
>> mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc
>> rep_good nopl nonstop_tsc extd_apicid amd_dcm pni monitor cx16
>> popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a
>> misalignsse 3dnowprefetch osvw ibs skinit wdt nodeid_msr npt
>> lbrv svm_lock nrip_save pausefilter

>> bogomips        : 4001.84
>> TLB size        : 1024 4K pages
>> clflush size    : 64
>> cache_alignment : 64
>> address sizes   : 48 bits physical, 48 bits virtual
>> power management: ts ttp tm stc 100mhzsteps hwpstate

>> # cat /proc/modules 
>> ext2 72951 0 - Live 0xffffffffa04d8000
>> loop 27624 1 - Live 0xffffffffa04ec000
>> nfs 312073 1 - Live 0xffffffffa048a000
>> fscache 46134 1 nfs, Live 0xffffffffa0476000
>> binfmt_misc 17456 1 - Live 0xffffffffa03fa000
>> nfsd 310366 13 - Live 0xffffffffa0429000
>> lockd 84640 2 nfs,nfsd, Live 0xffffffffa0413000
>> nfs_acl 12741 2 nfs,nfsd, Live 0xffffffffa03f5000
>> auth_rpcgss 47785 2 nfs,nfsd, Live 0xffffffffa0406000
>> sunrpc 216671 22 nfs,nfsd,lockd,nfs_acl,auth_rpcgss, Live 0xffffffffa03bf000
>> exportfs 12851 1 nfsd, Live 0xffffffffa03a6000
>> ipmi_si 52811 0 - Live 0xffffffffa03b1000
>> ipmi_devintf 17522 0 - Live 0xffffffffa03ab000
>> ipmi_msghandler 43683 2 ipmi_si,ipmi_devintf, Live 0xffffffffa039a000
>> sch_sfq 17212 2 - Live 0xffffffffa038d000
>> iptable_nat 13182 1 - Live 0xffffffffa0395000
>> nf_nat 24588 1 iptable_nat, Live 0xffffffffa037e000
>> nf_conntrack_ipv4 19478 3 iptable_nat,nf_nat, Live 0xffffffffa0387000
>> nf_conntrack 76827 3 iptable_nat,nf_nat,nf_conntrack_ipv4, Live 0xffffffffa036a000
>> nf_defrag_ipv4 12673 1 nf_conntrack_ipv4, Live 0xffffffffa0365000
>> iptable_filter 12810 0 - Live 0xffffffffa0360000
>> xt_TCPMSS 12715 2 - Live 0xffffffffa02f0000
>> xt_tcpudp 12603 2 - Live 0xffffffffa02e2000
>> iptable_mangle 12695 1 - Live 0xffffffffa02dd000
>> ip_tables 26705 3 iptable_nat,iptable_filter,iptable_mangle, Live 0xffffffffa02d1000
>> ip6table_filter 12815 0 - Live 0xffffffffa0217000
>> ip6_tables 26996 1 ip6table_filter, Live 0xffffffffa027a000
>> x_tables 27634 8 iptable_nat,iptable_filter,xt_TCPMSS,xt_tcpudp,iptable_mangle,ip_tables,ip6table_filter,ip6_tables,
Live 0xffffffffa0262000
>> ext4 403044 1 - Live 0xffffffffa02fc000
>> jbd2 84403 1 ext4, Live 0xffffffffa02bb000
>> crc16 12559 1 ext4, Live 0xffffffffa0212000
>> raid0 17106 1 - Live 0xffffffffa0245000
>> dm_mod 75794 0 - Live 0xffffffffa0296000
>> autofs4 28311 11 - Live 0xffffffffa01e6000
>> dummy 12955 0 - Live 0xffffffffa01a0000
>> usbhid 46436 0 - Live 0xffffffffa02ae000
>> hid 93560 1 usbhid, Live 0xffffffffa01fa000
>> ide_cd_mod 35415 0 - Live 0xffffffffa01dc000
>> cdrom 41847 1 ide_cd_mod, Live 0xffffffffa01d0000
>> ata_generic 12899 0 - Live 0xffffffffa019b000
>> pata_acpi 13027 0 - Live 0xffffffffa0186000
>> pata_atiixp 13165 0 - Live 0xffffffffa015d000
>> sp5100_tco 13697 0 - Live 0xffffffffa014f000
>> ide_pci_generic 12866 0 - Live 0xffffffffa014a000
>> ohci_hcd 31523 0 - Live 0xffffffffa017d000
>> psmouse 68641 0 - Live 0xffffffffa0284000
>> ghes 13625 0 - Live 0xffffffffa01cb000
>> ssb 56258 1 ohci_hcd, Live 0xffffffffa026b000
>> mmc_core 85395 1 ssb, Live 0xffffffffa024c000
>> igb 115008 0 - Live 0xffffffffa0227000
>> i2c_piix4 13167 0 - Live 0xffffffffa0158000
>> amd64_edac_mod 28622 0 - Live 0xffffffffa01f2000
>> pcmcia 43683 1 ssb, Live 0xffffffffa0171000
>> dca 14562 1 igb, Live 0xffffffffa00de000
>> pcspkr 12702 0 - Live 0xffffffffa00e3000
>> i2c_core 28557 1 i2c_piix4, Live 0xffffffffa0142000
>> hed 13102 1 ghes, Live 0xffffffffa01a9000
>> edac_core 51596 6 amd64_edac_mod, Live 0xffffffffa0134000
>> button 13611 0 - Live 0xffffffffa011c000
>> pcmcia_core 21861 1 pcmcia, Live 0xffffffffa02f5000
>> atiixp 12917 0 - Live 0xffffffffa02eb000
>> processor 30161 0 - Live 0xffffffffa021e000
>> ide_core 108026 3 ide_cd_mod,ide_pci_generic,atiixp, Live 0xffffffffa01af000
>> serio_raw 13166 0 - Live 0xffffffffa012f000
>> ehci_hcd 44738 0 - Live 0xffffffffa018f000
>> edac_mce_amd 22882 1 amd64_edac_mod, Live 0xffffffffa016a000
>> k10temp 13119 0 - Live 0xffffffffa0162000
>> hwmon 12886 1 k10temp, Live 0xffffffffa0117000
>> sg 36330 0 - Live 0xffffffffa0125000
>> usbcore 159860 4 usbhid,ohci_hcd,ehci_hcd, Live 0xffffffffa00ee000
>> evdev 17753 0 - Live 0xffffffffa00e8000
>> sd_mod 44764 16 - Live 0xffffffffa00d2000
>> crc_t10dif 12483 1 sd_mod, Live 0xffffffffa0083000
>> raid1 30545 1 - Live 0xffffffffa00ac000
>> md_mod 106323 4 raid0,raid1, Live 0xffffffffa00b5000
>> ext3 138195 1 - Live 0xffffffffa0089000
>> jbd 55191 1 ext3, Live 0xffffffffa0074000
>> mbcache 14317 3 ext2,ext4,ext3, Live 0xffffffffa0031000
>> ahci 25854 12 - Live 0xffffffffa006c000
>> libahci 25686 1 ahci, Live 0xffffffffa0029000
>> libata 197408 5 ata_generic,pata_acpi,pata_atiixp,ahci,libahci, Live 0xffffffffa003a000
>> scsi_mod 161130 3 sg,sd_mod,libata, Live 0xffffffffa0000000

>> # df -h
>> Filesystem            Size  Used Avail Use% Mounted on
>> /dev/md0              7.6G  3.6G  3.7G  50% /
>> /dev/md1              1.9T  122G  1.8T   7% /home/atest
>> none                   32G     0   32G   0% /dev/shm
>> nexus:/R0             1.8T  531G  1.2T  31% /remote/nexus/R0

>> # cat /proc/mdstat
>> Personalities : [raid1] [raid0]
>> md1 : active raid0 sda3[0] sdd3[3] sdc3[2] sdb3[1]
>>       2047993856 blocks super 1.2 512k chunks

>> md0 : active raid1 sda1[0] sdd1[3] sdc1[2] sdb1[1]
>>       8000256 blocks [4/4] [UUUU]

>> # /bin/bash testme.sh
>> Linux-VServer Test [V0.17] Copyright (C) 2003-2006 H.Poetzl
>> utility 'vserver' could not be found.

>> # ./testfs.sh -t -x -y -z -D /dev/loop0 -M /mnt
>> Linux-VServer FS Test [V0.23] Copyright (C) 2005-2009 H.Poetzl
>> Linux 3.0.0-vs2.3.1-pre8-dirty x86_64/
>> VCI:  0002:0308 236 13011f11 (ID24)
>> ---
>> testing ext2 filesystem ...
>> [000]. [001]. [002]. (ext2 format)
>> tag related tests ...
>> [011]. [012]* [014]* [015]* [019]. 
>> [020]* [021]. [022]* [023]. [024]* [025]* [026]* [027]* [028]* 
>> [033]* [034]* [035]* [037]* [045]* [047]* 
>> xattr related tests ...
>> [101]. [103]* [104]* [106]* [107]* [109]
>> [112]* [113]* [114]. [115]* [116]* [118]* [119]*
>> [122]* [123]* [124]* [125]. [127]* [128]* [129]*
>> [131]* [132]. [133]* [134]* [135]* [138]. [139].
>> [148]. [149].
>> disk limit related tests ...
>> [201]* [202]* [203]. [204]* [205]. [206]. [207]. [208].
>> [211]* [212]* [213]. [222]. [223]. [231]. [232]. [233]. [239].
>> [999].
>> ---
>> testing ext3 filesystem ...
>> [000]. [001]. [002]. (ext3 format)
>> tag related tests ...
>> [011]. [012]* [014]* [015]* [019].
>> [020]* [021]. [022]* [023]. [024]* [025]* [026]* [027]* [028]*
>> [033]* [034]* [035]* [037]* [045]* [047]*
>> xattr related tests ...
>> [101]. [103]* [104]* [106]* [107]* [109]*
>> [112]* [113]* [114]. [115]* [116]* [118]* [119]*
>> [122]* [123]* [124]* [125]. [127]* [128]* [129]*
>> [131]* [132]. [133]* [134]* [135]* [138]. [139].
>> [148]. [149].
>> disk limit related tests ...
>> [201]* [202]* [203]. [204]* [205]. [206]. [207]. [208].
>> [211]* [212]* [213]. [222]. [223]. [231]. [232]. [233]. [239].
>> [999].
>> ---
>> testing ext4 filesystem ...
>> [000]. [001]. [002]. (ext4 format)
>> tag related tests ...
>> [011]. [012]* [014]* [015]* [019].
>> [020]* [021]. [022]* [023]. [024]* [025]* [026]* [027]* [028]*
>> [033]* [034]* [035]* [037]* [045]* [047]*
>> xattr related tests ...
>> [101]. [103]* [104]* [106]* [107]* [109]*
>> [112]* [113]* [114]. [115]* [116]* [118]* [119]*
>> [122]* [123]* [124]* [125]. [127]* [128]* [129]*
>> [131]* [132]. [133]* [134]* [135]* [138]. [139].
>> [148]. [149].
>> disk limit related tests ...
>> [201]* [202]* [203]. [204]* [205]. [206]. [207]. [208].
>> [211]* [212]* [213]. [222]. [223]. [231]. [232]. [233]. [239].
>> [999].
>> ---
>> testing xfs filesystem ...
>> [000]* (xfs format failed)
>> ---
>> testing reiser filesystem ...
>> [000]* (reiserfs format failed)
>> ---
>> testing jfs filesystem ...
>> [000]* (jfs format failed)