On Tue, Jul 19, 2011 at 11:12:17AM +0200, Urban Loesch wrote: > Hi, > I'm new to the list. > We are using linux vserver since 2005 and until the last month it works > very well for us. > About 1 month ago we upgraded our self compiled 64bit kernel from Debian > lenny "2.6.32.22-vs2.3.0.36.29.6" to squeeze > "2.6.38.6-vs2.3.0.37-rc15" because of the problem listed here: I'd suggest to update to 2.6.38.8-vs2.3.0.37-rc17 and see if the issue remains ... best, Herbert > https://bugzilla.kernel.org/show_bug.cgi?id=16991 > http://lkml.org/lkml/2011/1/5/376 > Now about 30 day later all of our host servers are running fine except they > have a running vserver with apache2 installed. > Sometimes it happes that the vserver totally hangs. It's not possible to > enter via "vserver SERVER enter", no > it's possible to restart the vserver or the apache in it. The only thing > which gets back to work is a hard reset of the host server via > sysrq-trigger. > Before this happens /var/log/syslog on the host shows me the following > error: > ... > Jul 18 15:45:26 vhost01 kernel: [4638798.444832] general protection fault: > 0000 [#1] SMP > Jul 18 15:45:26 vhost01 kernel: [4638798.455138] last sysfs file: > /sys/devices/pci0000:00/0000:00:1c.0/0000:03:00.0/host2/scsi_host/host2/proc_name > Jul 18 15:45:26 vhost01 kernel: [4638798.475515] CPU 4 > Jul 18 15:45:26 vhost01 kernel: [4638798.479512] Modules linked in: > netconsole configfs drbd lru_cache sch_hfsc ip6_queue act_police cls_flow > cls_fw cls_u32 sch_htb sch_ingress sch_sfq xt_realm iptable_raw ip6t_LOG > xt_connlimit ip6table_ > raw ipt_ULOG ipt_REJECT ipt_REDIRECT ipt_NETMAP ipt_MASQUERADE xt_comment > ipt_ECN ipt_ecn ipt_CLUSTERIP ip6t_REJECT ipt_ah ipt_addrtype nf_nat_tftp > nf_nat_snmp_basic nf_nat_sip nf_nat_pptp xt_recent nf_nat_proto_gre > nf_nat_irc nf_nat_h32 > 3 nf_nat_ftp nf_nat_amanda ip6table_mangle xt_NFLOG nfnetlink_log > nf_conntrack_ipv6 nf_conntrack_tftp nf_conntrack_sip nf_conntrack_sane > nf_conntrack_proto_udplite nf_conntrack_proto_sctp nf_conntrack_pptp > nf_conntrack_proto_gre nf_connt > rack_netlink nf_conntrack_netbios_ns nf_conntrack_irc nf_conntrack_h323 > nf_conntrack_ftp ts_kmp nf_conntrack_amanda xt_TPROXY nf_tproxy_core > nf_defrag_ipv6 xt_time xt_TCPMSS xt_tcpmss xt_sctp xt_policy xt_pkttype > xt_physdev xt_owner xt_N > FQUEUE xt_multiport xt_mark xt_mac xt_limit xt_length xt_iprange xt_helper > xt_ha > Jul 18 15:45:26 vhost01 kernel: shlimit xt_DSCP xt_dscp xt_dccp ipt_LOG > xt_connmark xt_CLASSIFY xt_tcpudp xt_conntrack xt_state iptable_nat nf_nat > nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack iptable_mangle nfnetlink > iptable_filter ip_t > ables ip6table_filter ip6_tables x_tables ipmi_devintf ipmi_si > ipmi_msghandler psmouse i7core_edac edac_core tpm_tis tpm pcspkr tpm_bios > processor serio_raw power_meter dcdbas button ses enclosure bnx2 igb > megaraid_sas dca [last unloaded > : scsi_wait_scan] > Jul 18 15:45:26 vhost01 kernel: [4638798.734507] > Jul 18 15:45:26 vhost01 kernel: [4638798.737861] Pid: 646, comm: apache2 > Not tainted 2.6.38.6-vs2.3.0.37-rc15-rol-em64t #7 Dell Inc. PowerEdge > R610/086HF8 > Jul 18 15:45:26 vhost01 kernel: [4638798.759501] RIP: > 0010:[<ffffffff8103695a>] [<ffffffff8103695a>] task_rq_lock+0x4a/0xa0 > Jul 18 15:45:26 vhost01 kernel: [4638798.775920] RSP: 0018:ffff88081da17dc8 > EFLAGS: 00010082 > Jul 18 15:45:26 vhost01 kernel: [4638798.786892] RAX: 9066669066666605 RBX: > 0000000000011d00 RCX: ffffffff814f3720 > Jul 18 15:45:26 vhost01 kernel: [4638798.801550] RDX: 0000000000000282 RSI: > ffff88081da17e20 RDI: 00007f8f2f645410 > Jul 18 15:45:26 vhost01 kernel: [4638798.816205] RBP: ffff88081da17de8 R08: > 0000000000000000 R09: 0000000000000001 > Jul 18 15:45:26 vhost01 kernel: [4638798.830858] R10: 0000000000002b28 R11: > 0000000000000400 R12: 00007f8f2f645410 > Jul 18 15:45:26 vhost01 kernel: [4638798.845520] R13: ffff88081da17e20 R14: > 0000000000011d00 R15: 0000000000000000 > Jul 18 15:45:26 vhost01 kernel: [4638798.860188] FS: > 00007f8f308736d0(0000) GS:ffff88083fc40000(0000) knlGS:0000000000000000 > Jul 18 15:45:26 vhost01 kernel: [4638798.876748] CS: 0010 DS: 0000 ES: > 0000 CR0: 0000000080050033 > Jul 18 15:45:26 vhost01 kernel: [4638798.888581] CR2: 0000000006b65290 CR3: > 000000079dd22000 CR4: 00000000000006e0 > Jul 18 15:45:26 vhost01 kernel: [4638798.903236] DR0: 0000000000000000 DR1: > 0000000000000000 DR2: 0000000000000000 > Jul 18 15:45:26 vhost01 kernel: [4638798.917890] DR3: 0000000000000000 DR6: > 00000000ffff0ff0 DR7: 0000000000000400 > Jul 18 15:45:26 vhost01 kernel: [4638798.932545] Process apache2 (pid: 646, > threadinfo ffff88081da16000, task ffff880441ae3e70) > Jul 18 15:45:26 vhost01 kernel: [4638798.949457] Stack: > Jul 18 15:45:26 vhost01 kernel: [4638798.953847] 00007f8f2f645410 > ffff8807f9221eb8 000000000000000f 0000000000000000 > Jul 18 15:45:26 vhost01 kernel: [4638798.969073] ffff88081da17e58 > ffffffff81040bd6 ffff88081da17e48 ffffffff81269f47 > Jul 18 15:45:26 vhost01 kernel: [4638798.984290] 0000000000000000 > 0000000000000003 ffff88081da17e28 0000000000000282 > Jul 18 15:45:26 vhost01 kernel: [4638798.999509] Call Trace: > Jul 18 15:45:26 vhost01 kernel: [4638799.004772] [<ffffffff81040bd6>] > try_to_wake_up+0x36/0x310 > Jul 18 15:45:26 vhost01 kernel: [4638799.016262] [<ffffffff81269f47>] ? > idr_remove+0x187/0x1f0 > Jul 18 15:45:26 vhost01 kernel: [4638799.027576] [<ffffffff81040f05>] > wake_up_process+0x15/0x20 > Jul 18 15:45:26 vhost01 kernel: [4638799.039069] [<ffffffff81231580>] > freeary+0x1e0/0x260 > Jul 18 15:45:26 vhost01 kernel: [4638799.049523] [<ffffffff81232631>] > T.616+0x71/0xf0 > Jul 18 15:45:26 vhost01 kernel: [4638799.059286] [<ffffffff81131de5>] ? > vfs_write+0x125/0x190 > Jul 18 15:45:26 vhost01 kernel: [4638799.070426] [<ffffffff81232719>] > sys_semctl+0x69/0xa0 > Jul 18 15:45:26 vhost01 kernel: [4638799.081050] [<ffffffff81002882>] > system_call_fastpath+0x16/0x1b > Jul 18 15:45:26 vhost01 kernel: [4638799.093402] Code: 00 48 c7 c3 00 1d 01 > 00 49 89 fc 49 89 f5 9c 58 0f 1f 44 00 00 48 89 c2 fa 66 0f 1f 44 00 00 49 > 89 55 00 49 8b 44 24 08 49 89 de <8b> 40 18 4c 03 34 c5 e0 b2 70 81 4c 89 > f7 e8 73 93 > 4a 00 49 8b > Jul 18 15:45:26 vhost01 kernel: [4638799.132205] RIP [<ffffffff8103695a>] > task_rq_lock+0x4a/0xa0 > Jul 18 15:45:26 vhost01 kernel: [4638799.143884] RSP <ffff88081da17dc8> > Jul 18 15:45:26 vhost01 kernel: [4638799.151543] ---[ end trace > f2cfa0bdeab24e4d ]--- > ... > Details of our configuration: > Hardware: > DELL PE610 > 2x Intel(R) Xeon(R) CPU X5550 > 32GB RAM > RAID 10 with 6 disks > Kernel: > # uname -r > 2.6.38.6-vs2.3.0.37-rc15-rol-em64t > # util-vserver: > ii libvserver0 0.30.216-pre2955-1 > dynamic libraries for util-vserver > ii util-vserver 0.30.216-pre2955-1 > utilities for managing Linux-VServer guests > ii util-vserver-core 0.30.216-pre2955-1 > core utilities of util-vserver > ii util-vserver-sysv 0.30.216-pre2955-1 > initscripts for util-vserver > # vserver-info > Versions: > Kernel: 2.6.38.6-vs2.3.0.37-rc15-rol-em64t > VS-API: 0x00020308 > VCI: 0x0000000013001f11 > util-vserver: 0.30.216-pre2955; Mar 26 2011, 10:28:24 > Features: > CC: gcc, gcc (Debian 4.4.5-8) 4.4.5 > CXX: g++, g++ (Debian 4.4.5-8) 4.4.5 > CPPFLAGS: '' > CFLAGS: '-g -O2 -std=c99 -Wall -pedantic -W > -funit-at-a-time' > CXXFLAGS: '-g -O2 -ansi -Wall -pedantic -W > -fmessage-length=0 -funit-at-a-time' > build/host: x86_64-pc-linux-gnu/x86_64-pc-linux-gnu > Use dietlibc: yes > Build C++ programs: yes > Build C99 programs: yes > Available APIs: compat,v11,fscompat,v13,net,v21,v22,v23,netv2 > ext2fs Source: e2fsprogs > syscall(2) invocation: alternative > vserver(2) syscall#: 236/glibc > crypto api: nss > python bindings: yes > use library versioning: yes > Paths: > prefix: /usr > sysconf-Directory: /etc > cfg-Directory: /etc/vservers > initrd-Directory: /etc/init.d > pkgstate-Directory: /var/run/vservers > vserver-Rootdir: /vservers > Assumed 'SYSINFO' as no other option given; try '--help' for more > information. > # ./testme.sh > Linux-VServer Test [V0.17] Copyright (C) 2003-2006 H.Poetzl > chcontext is working. > chbind is working. > Linux 2.6.38.6-vs2.3.0.37-rc15-rol-em64t #7 SMP Mon May 16 19:31:48 CEST > 2011 x86_64 > Ea 0.30.216 236/glibc (DSa) <compat,v11,fscompat,v13,net,v21,v22,v23,netv2> > VCI: 0002:0308 236 13001f11 (TbsPW) > --- > [000]# succeeded. > [001]# succeeded. > [011]# succeeded. > [031]# succeeded. > [101]# succeeded. > [102]# succeeded. > [201]# succeeded. > [202]# succeeded. > # vserver-stat > CTX PROC VSZ RSS userTIME sysTIME UPTIME NAME > 72 923 62.1G 8.3G 4h49m33 31m14s10 11h05m48 VSERVERNAME > Only 1 vserver is running on that machine, but I saw the error on an other > server with the same configuration. After the downgrade of the second > server to kernel "2.6.32.41-vs2.3.0.36.29.7" the problem seems to be > solved. No problems about 20 days. > Have you any idea how I could solve this, or what could it cause? > If you need more information please let me know. > Thanks for your help > Urban Loesch