Subject: Re: [vserver] Linux vServer: general protection fault with apache2 and kernel 2.6.38.6
From: Urban Loesch <bind@enas.net>
Date: Tue, 19 Jul 2011 18:33:40 +0200

Hi Herbert,

 > I'd suggest to update to 2.6.38.8-vs2.3.0.37-rc17 and
 > see if the issue remains ...
 >

thanks for the response, I installed 2.6.38.8-vs2.3.0.37-rc17.

#  uname -r
2.6.38.8-vs2.3.0.37-rc17-rol-em64t

Testme work's
# ./testme.sh
Linux-VServer Test [V0.17] Copyright (C) 2003-2006 H.Poetzl
chcontext is working.
chbind is working.
Linux 2.6.38.8-vs2.3.0.37-rc17-rol-em64t #1 SMP Mon Jun 27 09:29:14 CEST 2011 x86_64
Ea 0.30.216 236/glibc (DSa) <compat,v11,fscompat,v13,net,v21,v22,v23,netv2>
VCI: 0002:0308 236 13001f11 (TbsPW)
---
[000]# succeeded.
[001]# succeeded.
[011]# succeeded.
[031]# succeeded.
[101]# succeeded.
[102]# succeeded.
[201]# succeeded.
[202]# succeeded.


I will switch back the vserver tonight from my backup server. If the error appears again
I
let you know.

 > best,
 > Herbert

Many thanks
Urban


Herbert Poetzl wrote:
> On Tue, Jul 19, 2011 at 11:12:17AM +0200, Urban Loesch wrote:
>> Hi,
> 
>> I'm new to the list.
> 
>> We are using linux vserver since 2005 and until the last month it works 
>> very well for us.
>> About 1 month ago we upgraded our self compiled 64bit kernel from Debian 
>> lenny "2.6.32.22-vs2.3.0.36.29.6" to squeeze
>> "2.6.38.6-vs2.3.0.37-rc15" because of the problem listed here:
> 
> 
>> https://bugzilla.kernel.org/show_bug.cgi?id=16991
>> http://lkml.org/lkml/2011/1/5/376
> 
>> Now about 30 day later all of our host servers are running fine except they 
>> have a running vserver with apache2 installed.
>> Sometimes it happes that the vserver totally hangs. It's not possible to 
>> enter via "vserver SERVER enter", no
>> it's possible to restart the vserver or the apache in it. The only thing 
>> which gets back to work is a hard reset of the host server via 
>> sysrq-trigger.
> 
> 
>> Before this happens /var/log/syslog on the host shows me the following 
>> error:
> 
>> ...
>> Jul 18 15:45:26 vhost01 kernel: [4638798.444832] general protection fault: 
>> 0000 [#1] SMP
>> Jul 18 15:45:26 vhost01 kernel: [4638798.455138] last sysfs file: 
>> /sys/devices/pci0000:00/0000:00:1c.0/0000:03:00.0/host2/scsi_host/host2/proc_name
>> Jul 18 15:45:26 vhost01 kernel: [4638798.475515] CPU 4
>> Jul 18 15:45:26 vhost01 kernel: [4638798.479512] Modules linked in: 
>> netconsole configfs drbd lru_cache sch_hfsc ip6_queue act_police cls_flow 
>> cls_fw cls_u32 sch_htb sch_ingress sch_sfq xt_realm iptable_raw ip6t_LOG 
>> xt_connlimit ip6table_
>> raw ipt_ULOG ipt_REJECT ipt_REDIRECT ipt_NETMAP ipt_MASQUERADE xt_comment 
>> ipt_ECN ipt_ecn ipt_CLUSTERIP ip6t_REJECT ipt_ah ipt_addrtype nf_nat_tftp 
>> nf_nat_snmp_basic nf_nat_sip nf_nat_pptp xt_recent nf_nat_proto_gre 
>> nf_nat_irc nf_nat_h32
>> 3 nf_nat_ftp nf_nat_amanda ip6table_mangle xt_NFLOG nfnetlink_log 
>> nf_conntrack_ipv6 nf_conntrack_tftp nf_conntrack_sip nf_conntrack_sane 
>> nf_conntrack_proto_udplite nf_conntrack_proto_sctp nf_conntrack_pptp 
>> nf_conntrack_proto_gre nf_connt
>> rack_netlink nf_conntrack_netbios_ns nf_conntrack_irc nf_conntrack_h323 
>> nf_conntrack_ftp ts_kmp nf_conntrack_amanda xt_TPROXY nf_tproxy_core 
>> nf_defrag_ipv6 xt_time xt_TCPMSS xt_tcpmss xt_sctp xt_policy xt_pkttype 
>> xt_physdev xt_owner xt_N
>> FQUEUE xt_multiport xt_mark xt_mac xt_limit xt_length xt_iprange xt_helper 
>> xt_ha
>> Jul 18 15:45:26 vhost01 kernel: shlimit xt_DSCP xt_dscp xt_dccp ipt_LOG 
>> xt_connmark xt_CLASSIFY xt_tcpudp xt_conntrack xt_state iptable_nat nf_nat 
>> nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack iptable_mangle nfnetlink 
>> iptable_filter ip_t
>> ables ip6table_filter ip6_tables x_tables ipmi_devintf ipmi_si 
>> ipmi_msghandler psmouse i7core_edac edac_core tpm_tis tpm pcspkr tpm_bios 
>> processor serio_raw power_meter dcdbas button ses enclosure bnx2 igb 
>> megaraid_sas dca [last unloaded
>> : scsi_wait_scan]
>> Jul 18 15:45:26 vhost01 kernel: [4638798.734507]
>> Jul 18 15:45:26 vhost01 kernel: [4638798.737861] Pid: 646, comm: apache2 
>> Not tainted 2.6.38.6-vs2.3.0.37-rc15-rol-em64t #7 Dell Inc. PowerEdge 
>> R610/086HF8
>> Jul 18 15:45:26 vhost01 kernel: [4638798.759501] RIP: 
>> 0010:[<ffffffff8103695a>]  [<ffffffff8103695a>] task_rq_lock+0x4a/0xa0
>> Jul 18 15:45:26 vhost01 kernel: [4638798.775920] RSP: 0018:ffff88081da17dc8 
>> EFLAGS: 00010082
>> Jul 18 15:45:26 vhost01 kernel: [4638798.786892] RAX: 9066669066666605 RBX: 
>> 0000000000011d00 RCX: ffffffff814f3720
>> Jul 18 15:45:26 vhost01 kernel: [4638798.801550] RDX: 0000000000000282 RSI: 
>> ffff88081da17e20 RDI: 00007f8f2f645410
>> Jul 18 15:45:26 vhost01 kernel: [4638798.816205] RBP: ffff88081da17de8 R08: 
>> 0000000000000000 R09: 0000000000000001
>> Jul 18 15:45:26 vhost01 kernel: [4638798.830858] R10: 0000000000002b28 R11: 
>> 0000000000000400 R12: 00007f8f2f645410
>> Jul 18 15:45:26 vhost01 kernel: [4638798.845520] R13: ffff88081da17e20 R14: 
>> 0000000000011d00 R15: 0000000000000000
>> Jul 18 15:45:26 vhost01 kernel: [4638798.860188] FS:  
>> 00007f8f308736d0(0000) GS:ffff88083fc40000(0000) knlGS:0000000000000000
>> Jul 18 15:45:26 vhost01 kernel: [4638798.876748] CS:  0010 DS: 0000 ES: 
>> 0000 CR0: 0000000080050033
>> Jul 18 15:45:26 vhost01 kernel: [4638798.888581] CR2: 0000000006b65290 CR3: 
>> 000000079dd22000 CR4: 00000000000006e0
>> Jul 18 15:45:26 vhost01 kernel: [4638798.903236] DR0: 0000000000000000 DR1: 
>> 0000000000000000 DR2: 0000000000000000
>> Jul 18 15:45:26 vhost01 kernel: [4638798.917890] DR3: 0000000000000000 DR6: 
>> 00000000ffff0ff0 DR7: 0000000000000400
>> Jul 18 15:45:26 vhost01 kernel: [4638798.932545] Process apache2 (pid: 646, 
>> threadinfo ffff88081da16000, task ffff880441ae3e70)
>> Jul 18 15:45:26 vhost01 kernel: [4638798.949457] Stack:
>> Jul 18 15:45:26 vhost01 kernel: [4638798.953847]  00007f8f2f645410 
>> ffff8807f9221eb8 000000000000000f 0000000000000000
>> Jul 18 15:45:26 vhost01 kernel: [4638798.969073]  ffff88081da17e58 
>> ffffffff81040bd6 ffff88081da17e48 ffffffff81269f47
>> Jul 18 15:45:26 vhost01 kernel: [4638798.984290]  0000000000000000 
>> 0000000000000003 ffff88081da17e28 0000000000000282
>> Jul 18 15:45:26 vhost01 kernel: [4638798.999509] Call Trace:
>> Jul 18 15:45:26 vhost01 kernel: [4638799.004772]  [<ffffffff81040bd6>] 
>> try_to_wake_up+0x36/0x310
>> Jul 18 15:45:26 vhost01 kernel: [4638799.016262]  [<ffffffff81269f47>] ? 
>> idr_remove+0x187/0x1f0
>> Jul 18 15:45:26 vhost01 kernel: [4638799.027576]  [<ffffffff81040f05>] 
>> wake_up_process+0x15/0x20
>> Jul 18 15:45:26 vhost01 kernel: [4638799.039069]  [<ffffffff81231580>] 
>> freeary+0x1e0/0x260
>> Jul 18 15:45:26 vhost01 kernel: [4638799.049523]  [<ffffffff81232631>] 
>> T.616+0x71/0xf0
>> Jul 18 15:45:26 vhost01 kernel: [4638799.059286]  [<ffffffff81131de5>] ? 
>> vfs_write+0x125/0x190
>> Jul 18 15:45:26 vhost01 kernel: [4638799.070426]  [<ffffffff81232719>] 
>> sys_semctl+0x69/0xa0
>> Jul 18 15:45:26 vhost01 kernel: [4638799.081050]  [<ffffffff81002882>] 
>> system_call_fastpath+0x16/0x1b
>> Jul 18 15:45:26 vhost01 kernel: [4638799.093402] Code: 00 48 c7 c3 00 1d 01 
>> 00 49 89 fc 49 89 f5 9c 58 0f 1f 44 00 00 48 89 c2 fa 66 0f 1f 44 00 00 49 
>> 89 55 00 49 8b 44 24 08 49 89 de <8b> 40 18 4c 03 34 c5 e0 b2 70 81 4c 89 
>> f7 e8 73 93
>> 4a 00 49 8b
>> Jul 18 15:45:26 vhost01 kernel: [4638799.132205] RIP  [<ffffffff8103695a>] 
>> task_rq_lock+0x4a/0xa0
>> Jul 18 15:45:26 vhost01 kernel: [4638799.143884]  RSP <ffff88081da17dc8>
>> Jul 18 15:45:26 vhost01 kernel: [4638799.151543] ---[ end trace 
>> f2cfa0bdeab24e4d ]---
>> ...
> 
>> Details of our configuration:
>> Hardware:
>> DELL PE610
>> 2x Intel(R) Xeon(R) CPU X5550
>> 32GB RAM
>> RAID 10 with 6 disks
> 
>> Kernel:
>> #  uname -r
>> 2.6.38.6-vs2.3.0.37-rc15-rol-em64t
> 
>> # util-vserver:
>> ii  libvserver0                                        0.30.216-pre2955-1   
>> dynamic libraries for util-vserver
>> ii  util-vserver                                       0.30.216-pre2955-1   
>> utilities for managing Linux-VServer guests
>> ii  util-vserver-core                                  0.30.216-pre2955-1   
>> core utilities of util-vserver
>> ii  util-vserver-sysv                                  0.30.216-pre2955-1   
>> initscripts for util-vserver
> 
>> #  vserver-info
>> Versions:
>>                    Kernel: 2.6.38.6-vs2.3.0.37-rc15-rol-em64t
>>                    VS-API: 0x00020308
>>                       VCI: 0x0000000013001f11
>>              util-vserver: 0.30.216-pre2955; Mar 26 2011, 10:28:24
> 
>> Features:
>>                CC: gcc, gcc (Debian 4.4.5-8) 4.4.5
>>                       CXX: g++, g++ (Debian 4.4.5-8) 4.4.5
>>                  CPPFLAGS: ''
>>                    CFLAGS: '-g -O2 -std=c99 -Wall -pedantic -W 
>>                    -funit-at-a-time'
>>                  CXXFLAGS: '-g -O2 -ansi -Wall -pedantic -W 
>>                  -fmessage-length=0 -funit-at-a-time'
>>                build/host: x86_64-pc-linux-gnu/x86_64-pc-linux-gnu
>>              Use dietlibc: yes
>>        Build C++ programs: yes
>>        Build C99 programs: yes
>>            Available APIs: compat,v11,fscompat,v13,net,v21,v22,v23,netv2
>>             ext2fs Source: e2fsprogs
>>     syscall(2) invocation: alternative
>>       vserver(2) syscall#: 236/glibc
>>                crypto api: nss
>>           python bindings: yes
>>    use library versioning: yes
> 
>> Paths:
>>                    prefix: /usr
>>         sysconf-Directory: /etc
>>             cfg-Directory: /etc/vservers
>>          initrd-Directory: /etc/init.d
>>        pkgstate-Directory: /var/run/vservers
>>           vserver-Rootdir: /vservers
> 
> 
>> Assumed 'SYSINFO' as no other option given; try '--help' for more 
>> information.
> 
>> #  ./testme.sh
>> Linux-VServer Test [V0.17] Copyright (C) 2003-2006 H.Poetzl
>> chcontext is working.
>> chbind is working.
>> Linux 2.6.38.6-vs2.3.0.37-rc15-rol-em64t #7 SMP Mon May 16 19:31:48 CEST 
>> 2011 x86_64
>> Ea 0.30.216 236/glibc (DSa) <compat,v11,fscompat,v13,net,v21,v22,v23,netv2>
>> VCI: 0002:0308 236 13001f11 (TbsPW)
>> ---
>> [000]# succeeded.
>> [001]# succeeded.
>> [011]# succeeded.
>> [031]# succeeded.
>> [101]# succeeded.
>> [102]# succeeded.
>> [201]# succeeded.
>> [202]# succeeded.
> 
>> # vserver-stat
>> CTX   PROC    VSZ    RSS  userTIME   sysTIME    UPTIME NAME
>> 72     923  62.1G   8.3G   4h49m33  31m14s10  11h05m48 VSERVERNAME
> 
>> Only 1 vserver is running on that machine, but I saw the error on an other 
>> server with the same configuration. After the downgrade of the second 
>> server to kernel "2.6.32.41-vs2.3.0.36.29.7" the problem seems to be 
>> solved. No problems about 20 days.
> 
> 
>> Have you any idea how I could solve this, or what could it cause?
>> If you need more information please let me know.
> 
>> Thanks for your help
>> Urban Loesch
>