Subject: Re: [vserver] Linux vServer: general protection fault with apache2 and kernel 2.6.38.6
From: Herbert Poetzl <herbert@13thfloor.at>
Date: Tue, 19 Jul 2011 16:47:38 +0200

On Tue, Jul 19, 2011 at 11:12:17AM +0200, Urban Loesch wrote:
> Hi,

> I'm new to the list.

> We are using linux vserver since 2005 and until the last month it works 
> very well for us.
> About 1 month ago we upgraded our self compiled 64bit kernel from Debian 
> lenny "2.6.32.22-vs2.3.0.36.29.6" to squeeze
> "2.6.38.6-vs2.3.0.37-rc15" because of the problem listed here:

I'd suggest to update to 2.6.38.8-vs2.3.0.37-rc17 and
see if the issue remains ...

best,
Herbert

> https://bugzilla.kernel.org/show_bug.cgi?id=16991
> http://lkml.org/lkml/2011/1/5/376

> Now about 30 day later all of our host servers are running fine except they 
> have a running vserver with apache2 installed.
> Sometimes it happes that the vserver totally hangs. It's not possible to 
> enter via "vserver SERVER enter", no
> it's possible to restart the vserver or the apache in it. The only thing 
> which gets back to work is a hard reset of the host server via 
> sysrq-trigger.


> Before this happens /var/log/syslog on the host shows me the following 
> error:

> ...
> Jul 18 15:45:26 vhost01 kernel: [4638798.444832] general protection fault: 
> 0000 [#1] SMP
> Jul 18 15:45:26 vhost01 kernel: [4638798.455138] last sysfs file: 
> /sys/devices/pci0000:00/0000:00:1c.0/0000:03:00.0/host2/scsi_host/host2/proc_name
> Jul 18 15:45:26 vhost01 kernel: [4638798.475515] CPU 4
> Jul 18 15:45:26 vhost01 kernel: [4638798.479512] Modules linked in: 
> netconsole configfs drbd lru_cache sch_hfsc ip6_queue act_police cls_flow 
> cls_fw cls_u32 sch_htb sch_ingress sch_sfq xt_realm iptable_raw ip6t_LOG 
> xt_connlimit ip6table_
> raw ipt_ULOG ipt_REJECT ipt_REDIRECT ipt_NETMAP ipt_MASQUERADE xt_comment 
> ipt_ECN ipt_ecn ipt_CLUSTERIP ip6t_REJECT ipt_ah ipt_addrtype nf_nat_tftp 
> nf_nat_snmp_basic nf_nat_sip nf_nat_pptp xt_recent nf_nat_proto_gre 
> nf_nat_irc nf_nat_h32
> 3 nf_nat_ftp nf_nat_amanda ip6table_mangle xt_NFLOG nfnetlink_log 
> nf_conntrack_ipv6 nf_conntrack_tftp nf_conntrack_sip nf_conntrack_sane 
> nf_conntrack_proto_udplite nf_conntrack_proto_sctp nf_conntrack_pptp 
> nf_conntrack_proto_gre nf_connt
> rack_netlink nf_conntrack_netbios_ns nf_conntrack_irc nf_conntrack_h323 
> nf_conntrack_ftp ts_kmp nf_conntrack_amanda xt_TPROXY nf_tproxy_core 
> nf_defrag_ipv6 xt_time xt_TCPMSS xt_tcpmss xt_sctp xt_policy xt_pkttype 
> xt_physdev xt_owner xt_N
> FQUEUE xt_multiport xt_mark xt_mac xt_limit xt_length xt_iprange xt_helper 
> xt_ha
> Jul 18 15:45:26 vhost01 kernel: shlimit xt_DSCP xt_dscp xt_dccp ipt_LOG 
> xt_connmark xt_CLASSIFY xt_tcpudp xt_conntrack xt_state iptable_nat nf_nat 
> nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack iptable_mangle nfnetlink 
> iptable_filter ip_t
> ables ip6table_filter ip6_tables x_tables ipmi_devintf ipmi_si 
> ipmi_msghandler psmouse i7core_edac edac_core tpm_tis tpm pcspkr tpm_bios 
> processor serio_raw power_meter dcdbas button ses enclosure bnx2 igb 
> megaraid_sas dca [last unloaded
> : scsi_wait_scan]
> Jul 18 15:45:26 vhost01 kernel: [4638798.734507]
> Jul 18 15:45:26 vhost01 kernel: [4638798.737861] Pid: 646, comm: apache2 
> Not tainted 2.6.38.6-vs2.3.0.37-rc15-rol-em64t #7 Dell Inc. PowerEdge 
> R610/086HF8
> Jul 18 15:45:26 vhost01 kernel: [4638798.759501] RIP: 
> 0010:[<ffffffff8103695a>]  [<ffffffff8103695a>] task_rq_lock+0x4a/0xa0
> Jul 18 15:45:26 vhost01 kernel: [4638798.775920] RSP: 0018:ffff88081da17dc8 
> EFLAGS: 00010082
> Jul 18 15:45:26 vhost01 kernel: [4638798.786892] RAX: 9066669066666605 RBX: 
> 0000000000011d00 RCX: ffffffff814f3720
> Jul 18 15:45:26 vhost01 kernel: [4638798.801550] RDX: 0000000000000282 RSI: 
> ffff88081da17e20 RDI: 00007f8f2f645410
> Jul 18 15:45:26 vhost01 kernel: [4638798.816205] RBP: ffff88081da17de8 R08: 
> 0000000000000000 R09: 0000000000000001
> Jul 18 15:45:26 vhost01 kernel: [4638798.830858] R10: 0000000000002b28 R11: 
> 0000000000000400 R12: 00007f8f2f645410
> Jul 18 15:45:26 vhost01 kernel: [4638798.845520] R13: ffff88081da17e20 R14: 
> 0000000000011d00 R15: 0000000000000000
> Jul 18 15:45:26 vhost01 kernel: [4638798.860188] FS:  
> 00007f8f308736d0(0000) GS:ffff88083fc40000(0000) knlGS:0000000000000000
> Jul 18 15:45:26 vhost01 kernel: [4638798.876748] CS:  0010 DS: 0000 ES: 
> 0000 CR0: 0000000080050033
> Jul 18 15:45:26 vhost01 kernel: [4638798.888581] CR2: 0000000006b65290 CR3: 
> 000000079dd22000 CR4: 00000000000006e0
> Jul 18 15:45:26 vhost01 kernel: [4638798.903236] DR0: 0000000000000000 DR1: 
> 0000000000000000 DR2: 0000000000000000
> Jul 18 15:45:26 vhost01 kernel: [4638798.917890] DR3: 0000000000000000 DR6: 
> 00000000ffff0ff0 DR7: 0000000000000400
> Jul 18 15:45:26 vhost01 kernel: [4638798.932545] Process apache2 (pid: 646, 
> threadinfo ffff88081da16000, task ffff880441ae3e70)
> Jul 18 15:45:26 vhost01 kernel: [4638798.949457] Stack:
> Jul 18 15:45:26 vhost01 kernel: [4638798.953847]  00007f8f2f645410 
> ffff8807f9221eb8 000000000000000f 0000000000000000
> Jul 18 15:45:26 vhost01 kernel: [4638798.969073]  ffff88081da17e58 
> ffffffff81040bd6 ffff88081da17e48 ffffffff81269f47
> Jul 18 15:45:26 vhost01 kernel: [4638798.984290]  0000000000000000 
> 0000000000000003 ffff88081da17e28 0000000000000282
> Jul 18 15:45:26 vhost01 kernel: [4638798.999509] Call Trace:
> Jul 18 15:45:26 vhost01 kernel: [4638799.004772]  [<ffffffff81040bd6>] 
> try_to_wake_up+0x36/0x310
> Jul 18 15:45:26 vhost01 kernel: [4638799.016262]  [<ffffffff81269f47>] ? 
> idr_remove+0x187/0x1f0
> Jul 18 15:45:26 vhost01 kernel: [4638799.027576]  [<ffffffff81040f05>] 
> wake_up_process+0x15/0x20
> Jul 18 15:45:26 vhost01 kernel: [4638799.039069]  [<ffffffff81231580>] 
> freeary+0x1e0/0x260
> Jul 18 15:45:26 vhost01 kernel: [4638799.049523]  [<ffffffff81232631>] 
> T.616+0x71/0xf0
> Jul 18 15:45:26 vhost01 kernel: [4638799.059286]  [<ffffffff81131de5>] ? 
> vfs_write+0x125/0x190
> Jul 18 15:45:26 vhost01 kernel: [4638799.070426]  [<ffffffff81232719>] 
> sys_semctl+0x69/0xa0
> Jul 18 15:45:26 vhost01 kernel: [4638799.081050]  [<ffffffff81002882>] 
> system_call_fastpath+0x16/0x1b
> Jul 18 15:45:26 vhost01 kernel: [4638799.093402] Code: 00 48 c7 c3 00 1d 01 
> 00 49 89 fc 49 89 f5 9c 58 0f 1f 44 00 00 48 89 c2 fa 66 0f 1f 44 00 00 49 
> 89 55 00 49 8b 44 24 08 49 89 de <8b> 40 18 4c 03 34 c5 e0 b2 70 81 4c 89 
> f7 e8 73 93
> 4a 00 49 8b
> Jul 18 15:45:26 vhost01 kernel: [4638799.132205] RIP  [<ffffffff8103695a>] 
> task_rq_lock+0x4a/0xa0
> Jul 18 15:45:26 vhost01 kernel: [4638799.143884]  RSP <ffff88081da17dc8>
> Jul 18 15:45:26 vhost01 kernel: [4638799.151543] ---[ end trace 
> f2cfa0bdeab24e4d ]---
> ...

> Details of our configuration:
> Hardware:
> DELL PE610
> 2x Intel(R) Xeon(R) CPU X5550
> 32GB RAM
> RAID 10 with 6 disks

> Kernel:
> #  uname -r
> 2.6.38.6-vs2.3.0.37-rc15-rol-em64t

> # util-vserver:
> ii  libvserver0                                        0.30.216-pre2955-1   
> dynamic libraries for util-vserver
> ii  util-vserver                                       0.30.216-pre2955-1   
> utilities for managing Linux-VServer guests
> ii  util-vserver-core                                  0.30.216-pre2955-1   
> core utilities of util-vserver
> ii  util-vserver-sysv                                  0.30.216-pre2955-1   
> initscripts for util-vserver

> #  vserver-info
> Versions:
>                    Kernel: 2.6.38.6-vs2.3.0.37-rc15-rol-em64t
>                    VS-API: 0x00020308
>                       VCI: 0x0000000013001f11
>              util-vserver: 0.30.216-pre2955; Mar 26 2011, 10:28:24

> Features:
>                CC: gcc, gcc (Debian 4.4.5-8) 4.4.5
>                       CXX: g++, g++ (Debian 4.4.5-8) 4.4.5
>                  CPPFLAGS: ''
>                    CFLAGS: '-g -O2 -std=c99 -Wall -pedantic -W 
>                    -funit-at-a-time'
>                  CXXFLAGS: '-g -O2 -ansi -Wall -pedantic -W 
>                  -fmessage-length=0 -funit-at-a-time'
>                build/host: x86_64-pc-linux-gnu/x86_64-pc-linux-gnu
>              Use dietlibc: yes
>        Build C++ programs: yes
>        Build C99 programs: yes
>            Available APIs: compat,v11,fscompat,v13,net,v21,v22,v23,netv2
>             ext2fs Source: e2fsprogs
>     syscall(2) invocation: alternative
>       vserver(2) syscall#: 236/glibc
>                crypto api: nss
>           python bindings: yes
>    use library versioning: yes

> Paths:
>                    prefix: /usr
>         sysconf-Directory: /etc
>             cfg-Directory: /etc/vservers
>          initrd-Directory: /etc/init.d
>        pkgstate-Directory: /var/run/vservers
>           vserver-Rootdir: /vservers


> Assumed 'SYSINFO' as no other option given; try '--help' for more 
> information.

> #  ./testme.sh
> Linux-VServer Test [V0.17] Copyright (C) 2003-2006 H.Poetzl
> chcontext is working.
> chbind is working.
> Linux 2.6.38.6-vs2.3.0.37-rc15-rol-em64t #7 SMP Mon May 16 19:31:48 CEST 
> 2011 x86_64
> Ea 0.30.216 236/glibc (DSa) <compat,v11,fscompat,v13,net,v21,v22,v23,netv2>
> VCI: 0002:0308 236 13001f11 (TbsPW)
> ---
> [000]# succeeded.
> [001]# succeeded.
> [011]# succeeded.
> [031]# succeeded.
> [101]# succeeded.
> [102]# succeeded.
> [201]# succeeded.
> [202]# succeeded.

> # vserver-stat
> CTX   PROC    VSZ    RSS  userTIME   sysTIME    UPTIME NAME
> 72     923  62.1G   8.3G   4h49m33  31m14s10  11h05m48 VSERVERNAME

> Only 1 vserver is running on that machine, but I saw the error on an other 
> server with the same configuration. After the downgrade of the second 
> server to kernel "2.6.32.41-vs2.3.0.36.29.7" the problem seems to be 
> solved. No problems about 20 days.


> Have you any idea how I could solve this, or what could it cause?
> If you need more information please let me know.

> Thanks for your help
> Urban Loesch