Subject: Re: [vserver] /usr/sbin/chbind: line 135: 7645 Segmentation fault
From: Herbert Poetzl <herbert@13thfloor.at>
Date: Sat, 23 May 2009 13:44:43 +0200

On Sat, May 23, 2009 at 10:54:50AM +0200, Jarry wrote:
> Herbert Poetzl wrote:
> >On Mon, May 18, 2009 at 04:54:30PM +0200, Jarry wrote:
> >>Herbert Poetzl wrote:
> >>>On Mon, May 11, 2009 at 07:27:36PM +0200, Jarry wrote:
> >>>>I have a gentoo-guest with just a basic install, the only extra
> >>>>service installed is bind. Mostly it starts and works without any
> >>>>problem, but there is about 10% chance that starting this guest
> >>>>fails with following messages:
> >>>>----------------------------------
> >>>>vserver vs4-dns start
> >>>>/usr/sbin/chbind: line 135:  7645 Segmentation fault
> >>>>"${create_cmd[@]}" "${chain_cmd[@]}" -- "$@"
> >>>>
> >>>>An error occured while executing the vserver startup sequence; when
> >>>>there are no other messages, it is very likely that the init-script
> >>>>(env TERM=xterm /lib/rc/sh/init-vserver.sh default) failed.
> >>>>
> >>>>Common causes are:
> >>>>* /etc/rc.d/rc on Fedora Core 1 and RH9 fails always; the 'apt-rpm' 
> >>>>build
> >>>> method knows how to deal with this, but on existing installations,
> >>>> appending 'true' to this file will help.
> >>>>
> >>>>Failed to start vserver 'vs4-dns'
> >>>>-----------------------------------
> >>>>
> >>>>I did not find any regularity in this. Mostly it works, but in some
> >>>>rare ocasions simply does not. And apart from that, I noticed one
> >>>>strange message while stopping this guest which might be related to
> >>>>the above mentioned error:
> >>>>
> >>>>-----------------------------------
> >>>>vserver vs4-dns stop
> >>>>Device and target are missing; try '--help' for more information
> >>>>-----------------------------------
> >>>>
> >>>>What could be reason for that "device and target missing"?
> >>>>And even more important: why do I get random segmentation faults?
> >>>did you try with a recent kernel and util-vserver snapshot?
> >>I have the last stable vserver-kernel (2.6.22 + 2.2.0.7) and
> >>util-vserver (0.30.215). I do not want to play with exp/dev
> >>releases as this should be a stable server. Though it looks so,
> >>it is not so stable as I'd expect. I still get seg-faults... :-(
> >
> >if you get segfaults with that branch, you definitely have
> >some serious hardware or software issues (e.g. toolchain),
> >please provide more information and a test case which
> >segfaults for you ...
> >
> >information:
> >
> > 'vserver-info - SYSINFO'
> > 'testme.sh-0.17 -v'
> 
> obelix ~ # vserver-info - SYSINFO
> Versions:
>                    Kernel: 2.6.22-vs2.2.0.7-gentoo
>                    VS-API: 0x00020200
>              util-vserver: 0.30.215; Mar 31 2009, 18:06:51
> 
> Features:
>                        CC: x86_64-pc-linux-gnu-gcc, 
> x86_64-pc-linux-gnu-gcc (Gentoo 4.3.2-r3 p1.6, pie-10.1.5) 4.3.2
>                       CXX: x86_64-pc-linux-gnu-g++, 
> x86_64-pc-linux-gnu-g++ (Gentoo 4.3.2-r3 p1.6, pie-10.1.5) 4.3.2
>                  CPPFLAGS: ''
>                    CFLAGS: '-march=athlon64 -O2 -pipe -std=c99 -Wall 
> -pedantic -W -funit-at-a-time'
>                  CXXFLAGS: '-march=athlon64 -O2 -pipe -ansi -Wall 
> -pedantic -W -fmessage-length=0 -funit-at-a-time'
>                build/host: x86_64-pc-linux-gnu/x86_64-pc-linux-gnu
>              Use dietlibc: yes
>        Build C++ programs: yes
>        Build C99 programs: yes
>            Available APIs: v13,net,v21,v22,v23,netv2
>             ext2fs Source: e2fsprogs
>     syscall(2) invocation: alternative
>       vserver(2) syscall#: 236/glibc
>                crypto api: beecrypt
> 
> Paths:
>                    prefix: /usr
>         sysconf-Directory: /etc
>             cfg-Directory: /etc/vservers
>          initrd-Directory: $(sysconfdir)/init.d
>        pkgstate-Directory: /var/run/vservers
>           vserver-Rootdir: /vservers
> 
> obelix ~ # ./testme.sh -v
> Linux-VServer Test [V0.17] Copyright (C) 2003-2006 H.Poetzl
> chcontext is working.
> chbind is working.
> chcontext 0.30.215 -- allocates/enters a security context
> This program is part of util-vserver 0.30.215
> 
> Copyright (C) 2004 Enrico Scholz
> This program is free software; you may redistribute it under the terms of
> the GNU General Public License.  This program has absolutely no warranty.
> Linux 2.6.22-vs2.2.0.7-gentoo #2 SMP Fri Mar 20 18:27:58 GMT 2009 x86_64
> Ea 0.30.215 236/glibc (DSa) <v13,net,v21,v22,v23,netv2>
> VCI: 0002:0200 236 030007f1 (TbsPHIiW)
> (root@obelix)
> (gcc version 4.1.2 (Gentoo 4.1.2 p1.3))
> #2 SMP Fri Mar 20 18:27:58 GMT 2009
> ---
> [000]# chcontext --xid 49151 true && chcontext --xid 45678 true
> [000]# succeeded.
> [001]# chcontext --xid 45678 egrep 'context|VxID' /proc/self/status
> [001]# succeeded.
> [011]# chcontext --secure --xid 45678 mknod /tmp/testme.sh.TNuaZ4/node c 0 0
> [011]# succeeded.
> [031]# chcontext --xid 49151 --hostname zaphod.6990 uname -a | grep -q 
> zaphod.6990
> [031]# succeeded.
> [101]# chbind --nid 49151 --ip 192.168.0.42 true
> [101]# succeeded.
> [102]# chbind --nid 49151 --ip 192.168.0.1/255.255.255.0 --ip 
> 10.0.0.1/24 true
> [102]# succeeded.
> [201]# chcontext --xid 45678 --flag fakeinit bash -c 'test $$ -eq 1'
> [201]# succeeded.
> [202]# chcontext --xid 49151 --flag fakeinit bash -c 'test $$ -eq 1'
> [202]# succeeded.
> obelix ~ #
> 
> -----------------------------------------
> 
> Problem is, I do not have any "test case". I have 6 guests,
> 5 of them start without any problem. And then there is one,
> which might start, or not. If it does not (and there is about
> 10% chance it might happen), I get that segfault.

hmm, so that sounds more like something _inside_ the
guest is seg-faulting, especially as the test scripts
should expose a compilation problem in the tools

so we are left with the following scenarios:

 - broken hardware (can strike anytime)
   run memtest and some cpu stress tests like
   xaos or prime number calculations for some time

 - broken guest environment
   try to debug guest startup 
   check dmesg when it just happened for kernel messages

less likely are now:

 - broken kernel/build toolchain
 - broken/miscompiled util-vserver

> All my guests (and host) are gentoo, updated to the last
> stable versions. And this particular gentoo guest making
> problems is actually just empty gentoo-template, with bind
> (chrooted) installed. Nothing more...

> I wrote a simple script which is started at the end of boot-up,
> checks if there is this guest-name in vserver-stat output,
> if not, starts it again until it is running and puts message
> in counter-file (never had to do it twice consecutively).
> This works but... it is not solution. Just workaround...

of course, and I would strongly advise to investigate
this further, as it might be just the tip of an iceberg
(e.g. if the hardware is flakey)

best,
Herbert

> Jarry
> -- 
> _______________________________________________________________
> This mailbox accepts e-mails only from selected mailing-lists!
> Everything else is considered to be spam and therefore deleted.