On Mon, Aug 08, 2011 at 04:07:54PM +0200, Herbert Poetzl wrote: > On Mon, Aug 08, 2011 at 01:24:48PM +0200, Pawel Sikora wrote: >> Hi, >> i have few dual opteron-6128 machines which are part of eda >> software auto-test farm. these machines don't host any vserver >> guests but installed kernel has vserver-patch applied by >> vendor. on vanilla kernel 2.6.38 (and newer) with vserver >> patch applied machines lock in some way after few hours of >> quite heavy computing + i/o (avg. load ~16). on 2.6.37.y with >> vserver everything works fine and on pure vanilla kernels also >> everything works fine. >> on the ipmi console i've logged only kernel flood: >> https://lkml.org/lkml/2011/5/23/398 this crash is reproducible >> on 2.6.38.y/2.6.39.y/3.0.y with the latest vserver patches. > the lkml posting says vserver/grsec, now does this mean > o you are testing with mainline + Linux-VServer patch > o you are testing with mainline + grsec patch > o you are testing with a combined patch? ah, I forgot, yes there is a way (not the easiest though) to decode the output your ipmi console recorded: - extract the byte sequence listed as Code: - search for that sequence in your vmlinux - add the offset of <xx> relative to the first byte (44) - lookup the resulting address with addr2line or compare it with objdump -t vmlinux note that the traces listed in the lkml posting show at least 3 different locations where something happened and I really wonder if those aren't three different perfectly fine stack traces which just get mangled by your ipmi setup ... maybe try to explicitely trigger a cpu/kernel trace with /proc/magic-sysrq and see if it logs properly best, Herbert >> # cat /proc/version >> Linux version 3.0.0-vs2.3.1-pre8-dirty (pawels@hal) (gcc version 4.5.3 20110515 (release) (PLD-Linux) ) #2 SMP Fri Aug 5 09:48:25 CEST 2011 > what does -dirty mean here? > thanks, > Herbert >> # cat /proc/cpuinfo >> processor : 0...15 >> vendor_id : AuthenticAMD >> cpu family : 16 >> model : 9 >> model name : AMD Opteron(tm) Processor 6128 >> stepping : 1 >> cpu MHz : 2000.149 >> cache size : 512 KB >> physical id : 1 >> siblings : 8 >> core id : 3 >> cpu cores : 8 >> apicid : 39 >> initial apicid : 23 >> fpu : yes >> fpu_exception : yes >> cpuid level : 5 >> wp : yes >> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge >> mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx >> mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc >> rep_good nopl nonstop_tsc extd_apicid amd_dcm pni monitor cx16 >> popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a >> misalignsse 3dnowprefetch osvw ibs skinit wdt nodeid_msr npt >> lbrv svm_lock nrip_save pausefilter >> bogomips : 4001.84 >> TLB size : 1024 4K pages >> clflush size : 64 >> cache_alignment : 64 >> address sizes : 48 bits physical, 48 bits virtual >> power management: ts ttp tm stc 100mhzsteps hwpstate >> # cat /proc/modules >> ext2 72951 0 - Live 0xffffffffa04d8000 >> loop 27624 1 - Live 0xffffffffa04ec000 >> nfs 312073 1 - Live 0xffffffffa048a000 >> fscache 46134 1 nfs, Live 0xffffffffa0476000 >> binfmt_misc 17456 1 - Live 0xffffffffa03fa000 >> nfsd 310366 13 - Live 0xffffffffa0429000 >> lockd 84640 2 nfs,nfsd, Live 0xffffffffa0413000 >> nfs_acl 12741 2 nfs,nfsd, Live 0xffffffffa03f5000 >> auth_rpcgss 47785 2 nfs,nfsd, Live 0xffffffffa0406000 >> sunrpc 216671 22 nfs,nfsd,lockd,nfs_acl,auth_rpcgss, Live 0xffffffffa03bf000 >> exportfs 12851 1 nfsd, Live 0xffffffffa03a6000 >> ipmi_si 52811 0 - Live 0xffffffffa03b1000 >> ipmi_devintf 17522 0 - Live 0xffffffffa03ab000 >> ipmi_msghandler 43683 2 ipmi_si,ipmi_devintf, Live 0xffffffffa039a000 >> sch_sfq 17212 2 - Live 0xffffffffa038d000 >> iptable_nat 13182 1 - Live 0xffffffffa0395000 >> nf_nat 24588 1 iptable_nat, Live 0xffffffffa037e000 >> nf_conntrack_ipv4 19478 3 iptable_nat,nf_nat, Live 0xffffffffa0387000 >> nf_conntrack 76827 3 iptable_nat,nf_nat,nf_conntrack_ipv4, Live 0xffffffffa036a000 >> nf_defrag_ipv4 12673 1 nf_conntrack_ipv4, Live 0xffffffffa0365000 >> iptable_filter 12810 0 - Live 0xffffffffa0360000 >> xt_TCPMSS 12715 2 - Live 0xffffffffa02f0000 >> xt_tcpudp 12603 2 - Live 0xffffffffa02e2000 >> iptable_mangle 12695 1 - Live 0xffffffffa02dd000 >> ip_tables 26705 3 iptable_nat,iptable_filter,iptable_mangle, Live 0xffffffffa02d1000 >> ip6table_filter 12815 0 - Live 0xffffffffa0217000 >> ip6_tables 26996 1 ip6table_filter, Live 0xffffffffa027a000 >> x_tables 27634 8 iptable_nat,iptable_filter,xt_TCPMSS,xt_tcpudp,iptable_mangle,ip_tables,ip6table_filter,ip6_tables, Live 0xffffffffa0262000 >> ext4 403044 1 - Live 0xffffffffa02fc000 >> jbd2 84403 1 ext4, Live 0xffffffffa02bb000 >> crc16 12559 1 ext4, Live 0xffffffffa0212000 >> raid0 17106 1 - Live 0xffffffffa0245000 >> dm_mod 75794 0 - Live 0xffffffffa0296000 >> autofs4 28311 11 - Live 0xffffffffa01e6000 >> dummy 12955 0 - Live 0xffffffffa01a0000 >> usbhid 46436 0 - Live 0xffffffffa02ae000 >> hid 93560 1 usbhid, Live 0xffffffffa01fa000 >> ide_cd_mod 35415 0 - Live 0xffffffffa01dc000 >> cdrom 41847 1 ide_cd_mod, Live 0xffffffffa01d0000 >> ata_generic 12899 0 - Live 0xffffffffa019b000 >> pata_acpi 13027 0 - Live 0xffffffffa0186000 >> pata_atiixp 13165 0 - Live 0xffffffffa015d000 >> sp5100_tco 13697 0 - Live 0xffffffffa014f000 >> ide_pci_generic 12866 0 - Live 0xffffffffa014a000 >> ohci_hcd 31523 0 - Live 0xffffffffa017d000 >> psmouse 68641 0 - Live 0xffffffffa0284000 >> ghes 13625 0 - Live 0xffffffffa01cb000 >> ssb 56258 1 ohci_hcd, Live 0xffffffffa026b000 >> mmc_core 85395 1 ssb, Live 0xffffffffa024c000 >> igb 115008 0 - Live 0xffffffffa0227000 >> i2c_piix4 13167 0 - Live 0xffffffffa0158000 >> amd64_edac_mod 28622 0 - Live 0xffffffffa01f2000 >> pcmcia 43683 1 ssb, Live 0xffffffffa0171000 >> dca 14562 1 igb, Live 0xffffffffa00de000 >> pcspkr 12702 0 - Live 0xffffffffa00e3000 >> i2c_core 28557 1 i2c_piix4, Live 0xffffffffa0142000 >> hed 13102 1 ghes, Live 0xffffffffa01a9000 >> edac_core 51596 6 amd64_edac_mod, Live 0xffffffffa0134000 >> button 13611 0 - Live 0xffffffffa011c000 >> pcmcia_core 21861 1 pcmcia, Live 0xffffffffa02f5000 >> atiixp 12917 0 - Live 0xffffffffa02eb000 >> processor 30161 0 - Live 0xffffffffa021e000 >> ide_core 108026 3 ide_cd_mod,ide_pci_generic,atiixp, Live 0xffffffffa01af000 >> serio_raw 13166 0 - Live 0xffffffffa012f000 >> ehci_hcd 44738 0 - Live 0xffffffffa018f000 >> edac_mce_amd 22882 1 amd64_edac_mod, Live 0xffffffffa016a000 >> k10temp 13119 0 - Live 0xffffffffa0162000 >> hwmon 12886 1 k10temp, Live 0xffffffffa0117000 >> sg 36330 0 - Live 0xffffffffa0125000 >> usbcore 159860 4 usbhid,ohci_hcd,ehci_hcd, Live 0xffffffffa00ee000 >> evdev 17753 0 - Live 0xffffffffa00e8000 >> sd_mod 44764 16 - Live 0xffffffffa00d2000 >> crc_t10dif 12483 1 sd_mod, Live 0xffffffffa0083000 >> raid1 30545 1 - Live 0xffffffffa00ac000 >> md_mod 106323 4 raid0,raid1, Live 0xffffffffa00b5000 >> ext3 138195 1 - Live 0xffffffffa0089000 >> jbd 55191 1 ext3, Live 0xffffffffa0074000 >> mbcache 14317 3 ext2,ext4,ext3, Live 0xffffffffa0031000 >> ahci 25854 12 - Live 0xffffffffa006c000 >> libahci 25686 1 ahci, Live 0xffffffffa0029000 >> libata 197408 5 ata_generic,pata_acpi,pata_atiixp,ahci,libahci, Live 0xffffffffa003a000 >> scsi_mod 161130 3 sg,sd_mod,libata, Live 0xffffffffa0000000 >> # df -h >> Filesystem Size Used Avail Use% Mounted on >> /dev/md0 7.6G 3.6G 3.7G 50% / >> /dev/md1 1.9T 122G 1.8T 7% /home/atest >> none 32G 0 32G 0% /dev/shm >> nexus:/R0 1.8T 531G 1.2T 31% /remote/nexus/R0 >> # cat /proc/mdstat >> Personalities : [raid1] [raid0] >> md1 : active raid0 sda3[0] sdd3[3] sdc3[2] sdb3[1] >> 2047993856 blocks super 1.2 512k chunks >> md0 : active raid1 sda1[0] sdd1[3] sdc1[2] sdb1[1] >> 8000256 blocks [4/4] [UUUU] >> # /bin/bash testme.sh >> Linux-VServer Test [V0.17] Copyright (C) 2003-2006 H.Poetzl >> utility 'vserver' could not be found. >> # ./testfs.sh -t -x -y -z -D /dev/loop0 -M /mnt >> Linux-VServer FS Test [V0.23] Copyright (C) 2005-2009 H.Poetzl >> Linux 3.0.0-vs2.3.1-pre8-dirty x86_64/ >> VCI: 0002:0308 236 13011f11 (ID24) >> --- >> testing ext2 filesystem ... >> [000]. [001]. [002]. (ext2 format) >> tag related tests ... >> [011]. [012]* [014]* [015]* [019]. >> [020]* [021]. [022]* [023]. [024]* [025]* [026]* [027]* [028]* >> [033]* [034]* [035]* [037]* [045]* [047]* >> xattr related tests ... >> [101]. [103]* [104]* [106]* [107]* [109] >> [112]* [113]* [114]. [115]* [116]* [118]* [119]* >> [122]* [123]* [124]* [125]. [127]* [128]* [129]* >> [131]* [132]. [133]* [134]* [135]* [138]. [139]. >> [148]. [149]. >> disk limit related tests ... >> [201]* [202]* [203]. [204]* [205]. [206]. [207]. [208]. >> [211]* [212]* [213]. [222]. [223]. [231]. [232]. [233]. [239]. >> [999]. >> --- >> testing ext3 filesystem ... >> [000]. [001]. [002]. (ext3 format) >> tag related tests ... >> [011]. [012]* [014]* [015]* [019]. >> [020]* [021]. [022]* [023]. [024]* [025]* [026]* [027]* [028]* >> [033]* [034]* [035]* [037]* [045]* [047]* >> xattr related tests ... >> [101]. [103]* [104]* [106]* [107]* [109]* >> [112]* [113]* [114]. [115]* [116]* [118]* [119]* >> [122]* [123]* [124]* [125]. [127]* [128]* [129]* >> [131]* [132]. [133]* [134]* [135]* [138]. [139]. >> [148]. [149]. >> disk limit related tests ... >> [201]* [202]* [203]. [204]* [205]. [206]. [207]. [208]. >> [211]* [212]* [213]. [222]. [223]. [231]. [232]. [233]. [239]. >> [999]. >> --- >> testing ext4 filesystem ... >> [000]. [001]. [002]. (ext4 format) >> tag related tests ... >> [011]. [012]* [014]* [015]* [019]. >> [020]* [021]. [022]* [023]. [024]* [025]* [026]* [027]* [028]* >> [033]* [034]* [035]* [037]* [045]* [047]* >> xattr related tests ... >> [101]. [103]* [104]* [106]* [107]* [109]* >> [112]* [113]* [114]. [115]* [116]* [118]* [119]* >> [122]* [123]* [124]* [125]. [127]* [128]* [129]* >> [131]* [132]. [133]* [134]* [135]* [138]. [139]. >> [148]. [149]. >> disk limit related tests ... >> [201]* [202]* [203]. [204]* [205]. [206]. [207]. [208]. >> [211]* [212]* [213]. [222]. [223]. [231]. [232]. [233]. [239]. >> [999]. >> --- >> testing xfs filesystem ... >> [000]* (xfs format failed) >> --- >> testing reiser filesystem ... >> [000]* (reiserfs format failed) >> --- >> testing jfs filesystem ... >> [000]* (jfs format failed)