Subject: Re: [vserver] debian jessie error on vserver shutdown
From: Daniel Urist <durist@ucar.edu>
Date: Tue, 5 Jan 2016 14:45:56 -0700
Tue, 5 Jan 2016 14:45:56 -0700
Thanks very much for your help.

I finally had a chance to further debug this-- the issue was debian's init
script /etc/init.d/sendsigs, which kills off remaining processes on
shutdown, and apparently doesn't wait very long. Since my vserver is
running a database, I need it go down cleanly.

I also found another minor issue with debian jessie shutdown under sysv-rc;
there are no kill scripts for cron, dbus or ssh since the "Default-Stop"
field in the init script headers for these are empty, so these also don't
go down cleanly. Since I'm running my vservers in a pacemaker/corosync
cluster, I fixed these as well.

I'm definitely not a fan of systemd, but it seems pretty clear that the
days of sysv-rc are numbered. Is there any chance vserver will work with
systemd in the future?

On Mon, Nov 16, 2015 at 11:24 AM, Herbert Poetzl <herbert@13thfloor.at>
wrote:

> On Mon, Nov 16, 2015 at 10:03:57AM -0700, Daniel Urist wrote:
> > I've attached the debug trace.
>
> > The line where the failure occurs is this:
>
> > /usr/share/util-vserver/vserver.stop: line 100: 30935 Killed
> >>    "${IONICE CMD[@]}" "${NICE CMD[@]}" "${NETNS CMD[@]}" "${CHBIND \
> >> CMD[@]}" "$ VSPACE" --enter "$S CONTEXT" "${OPTS VSPACE[@]}"
> >> "${OPTS VSPACE SHARED[@]}" -- "$ VTAG" --migrate "${OPTS VTAG ENTER[@]}"
> >> --silent\
> >>  -- $ VCONTEXT $SILENT OPT --migrate $OPT VCONTEXT CHROOT --xid
> >> "$S CONTEXT" -- "${INITCMD STOP[@]}"
>
> | [info] Will now restart.
> | ifdown: shutdown eth0: Operation not permitted
>
> This suggests that the guest has not been properly
> cleaned up (or some kind of update reverted the
> changes) and it tries to mess with the hardware.
>
> | /usr/share/util-vserver/vserver.stop: line 100: 30935 Killed
>
> I presume the very same scripts kill the parent
> at some point (immediately before a reboot) which
> causes the stop script to exit.
>
> I would say this is harmless, as it is unlikely
> that the runlevel scripts kill off all processes
> before trying to shut down the services, but if
> you want to improve this, you have to dig into
> the rc script  inside  the guest and prevent it
> from killing the process executing the runlevel
> change.
>
> HTH,
> Herbert
>
> > I took a snapshot of the process table on both the host and
> > vserver before shutting it down, and at that time there was no
> > process 30935, so I really don't understand where that's coming
> > from.
>
>
> > On Thu, Nov 12, 2015 at 11:49 PM, Herbert Poetzl <herbert@13thfloor.at>
> > wrote:
>
> >> On Thu, Nov 12, 2015 at 08:04:17AM -0700, Daniel Urist wrote:
> >>> On Wed, Nov 11, 2015 at 4:40 PM, Jean Weisbuch <jean@phpnet.org>
> wrote:
>
> >>>> The MySQL shutdown could be quite long, especially if you have
> >>>> a big buffer pool size or simply if you hit a bug and the
> >>>> process can be neverending.
>
> >>>> Are you sure that its not the case you are hitting and that
> >>>> "vserver stop" would timeout before MySQL would be stopped?
>
> >>> I don't think it's the timeout; shutting down the mysql server
> >>> by hand doesn't take more than a few seconds. I've also tried
> >>> bumping up the sync-timeout to 60 seconds with no difference.
>
> >>>> But if its really the "mysqladmin ping" that hangs its strange
> >>>> indeed, does it seems to hang before it crash or it directly
> >>>> crash when it tries to execute it?
>
> >>> From what I can see, it crashes as soon as it executes
> >>> "mysqladmin ping". Again, this only happens during vserver
> >>> shutdown; I can execute "mysqladmin ping" from within the
> >>> vserver or from the host with no issues while the vserver is
> >>> running.
>
> >> Let's see what a 'vserver --debug <name> stop' produces
>
> >> Thanks,
> >> Herbert
>
> >>>> Le 11/11/2015 23:50, Daniel Urist a écrit :
>
> >>>> I've narrowed down the issue to the init script for the Percona XtraDB
> >>>> mysql server. It seems to be crashing the vserver shutdown when it
> calls
> >>>> "mysqladmin ping" to see whether the database is alive. I can run
> >>>> "mysqladmin ping" successfully from within the vserver, and also from
> >> the
> >>>> host with "vserver exec". I can also successfully run
> "/etc/init.d/mysql
> >>>> stop" from the host via "vserver exec". Is there something special
> that
> >>>> happens during vserver shutdown that would cause this to fail?
>
> >>>> On Mon, Nov 9, 2015 at 11:51 AM, Herbert Poetzl <herbert@13thfloor.at
> >
> >>>> wrote:
>
> >>>>> On Mon, Nov 09, 2015 at 10:49:15AM -0700, Daniel Urist wrote:
> >>>>>> I am in the process of upgrading from debian wheezy to jessie
> >>>>>> for both hosts and guests. All hosts and guests are running
> >>>>>> sysv-rc (no systemd!). I'm getting the following error on
> >>>>>> shutdown with jessie guests running on jessie hosts:
>
> >>>>> Hey Daniel,
>
> >>>>>>  /usr/share/util-vserver/vserver.stop: line 100:  3511 Killed
> >>>>>>    "${IONICE CMD[@]}" "${NICE CMD[@]}" "${NETNS CMD[@]}"
> >>>>> "${CHBIND CMD[@]}"
> >>>>>> "$ VSPACE" --enter "$S CONTEXT" "${OPTS VSPACE[@]}"
> >>>>>> "${OPTS VSPACE SHARED[@]}" -- "$ VTAG" --migrate
> >> "${OPTS VTAG ENTER[@]}"
> >>>>>> --silent -- $ VCONTEXT $SILENT OPT --migrate $OPT VCONTEXT CHROOT
> >> --xid
> >>>>>> "$S CONTEXT" -- "${INITCMD STOP[@]}"
>
> >>>>> This looks like the rc scripts executed to stop the guest
> >>>>> kill the util-vserver stop script at some point.
>
> >>>>>> This doesn't happen with wheezy guests running on jessie hosts,
>
> >>>>> This is kind of suspected, as it is most likely a minor
> >>>>> "problem" with the guest side rc scripts.
>
> >>>>>> or with jessie guests on wheezy hosts.
>
> >>>>> This is very interesting, are you sure that the very
> >>>>> same guest does not exhibit this behaviours regardless
> >>>>> of the host system (given that util-vserver is the same
> >>>>> version).
>
> >>>>>> If I switch from init style sysv to plain, I don't see any
> >>>>>> errors, but I'm assuming that just suppresses the message
> >>>>>> since the vserver shuts down almost instantaneously;
>
> >>>>> An almost instant shutdown is not a bad thing per se, after
> >>>>> all container style virtualization is very fast. This
> >>>>> setup will not call any scripts inside the guest directly,
> >>>>> instead it will signal the init process inside the guest
> >>>>> to shut down. If everything goes as expected, your guest
> >>>>> should simply disappear (no more processes in the guest
> >>>>> context)
>
> >>>>>> I'm concerned that it's not going down cleanly.
>
> >>>>> It might be a good idea to check what scripts are actually
> >>>>> executed inside the guest and supplement them with debug
> >>>>> output (or run them with -x for bash scripts).
>
> >>>>>> The kernel version for jessie is 3.18.21-vs2.3.7.4-beng (from
> >>>>>> repo.psand.net), util-vserver is 0.30.216-pre3120-jessie0.1-1.
>
> >>>>>> Any hints on how to fix or at least debug this?
>
> >>>>> You can also run util-vserver with the --debug option,
> >>>>> which might shed some light on the issue.
>
> >>>>> HTH,
> >>>>> Herbert
>
>
>
>
>
>
>
>
>


Thanks very much for your help.

I finally had a chance to further debug this-- the issue was debian's init script /etc/init.d/sendsigs, which kills off remaining processes on shutdown, and apparently doesn't wait very long. Since my vserver is running a database, I need it go down cleanly.

I also found another minor issue with debian jessie shutdown under sysv-rc; there are no kill scripts for cron, dbus or ssh since the "Default-Stop" field in the init script headers for these are empty, so these also don't go down cleanly. Since I'm running my vservers in a pacemaker/corosync cluster, I fixed these as well.

I'm definitely not a fan of systemd, but it seems pretty clear that the days of sysv-rc are numbered. Is there any chance vserver will work with systemd in the future?

On Mon, Nov 16, 2015 at 11:24 AM, Herbert Poetzl <herbert@13thfloor.at> wrote:
On Mon, Nov 16, 2015 at 10:03:57AM -0700, Daniel Urist wrote:
> I've attached the debug trace.

> The line where the failure occurs is this:

> /usr/share/util-vserver/vserver.stop: line 100: 30935 Killed
>>    "${IONICE CMD[@]}" "${NICE CMD[@]}" "${NETNS CMD[@]}" "${CHBIND \
>> CMD[@]}" "$ VSPACE" --enter "$S CONTEXT" "${OPTS VSPACE[@]}"
>> "${OPTS VSPACE SHARED[@]}" -- "$ VTAG" --migrate "${OPTS VTAG ENTER[@]}"
>> --silent\
>>  -- $ VCONTEXT $SILENT OPT --migrate $OPT VCONTEXT CHROOT --xid
>> "$S CONTEXT" -- "${INITCMD STOP[@]}"

| [info] Will now restart.
| ifdown: shutdown eth0: Operation not permitted

This suggests that the guest has not been properly
cleaned up (or some kind of update reverted the
changes) and it tries to mess with the hardware.

| /usr/share/util-vserver/vserver.stop: line 100: 30935 Killed

I presume the very same scripts kill the parent
at some point (immediately before a reboot) which
causes the stop script to exit.

I would say this is harmless, as it is unlikely
that the runlevel scripts kill off all processes
before trying to shut down the services, but if
you want to improve this, you have to dig into
the rc script inside the guest and prevent it
from killing the process executing the runlevel
change.

HTH,
Herbert

> I took a snapshot of the process table on both the host and
> vserver before shutting it down, and at that time there was no
> process 30935, so I really don't understand where that's coming
> from.


> On Thu, Nov 12, 2015 at 11:49 PM, Herbert Poetzl <herbert@13thfloor.at>
> wrote:

>> On Thu, Nov 12, 2015 at 08:04:17AM -0700, Daniel Urist wrote:
>>> On Wed, Nov 11, 2015 at 4:40 PM, Jean Weisbuch <jean@phpnet.org> wrote:

>>>> The MySQL shutdown could be quite long, especially if you have
>>>> a big buffer pool size or simply if you hit a bug and the
>>>> process can be neverending.

>>>> Are you sure that its not the case you are hitting and that
>>>> "vserver stop" would timeout before MySQL would be stopped?

>>> I don't think it's the timeout; shutting down the mysql server
>>> by hand doesn't take more than a few seconds. I've also tried
>>> bumping up the sync-timeout to 60 seconds with no difference.

>>>> But if its really the "mysqladmin ping" that hangs its strange
>>>> indeed, does it seems to hang before it crash or it directly
>>>> crash when it tries to execute it?

>>> From what I can see, it crashes as soon as it executes
>>> "mysqladmin ping". Again, this only happens during vserver
>>> shutdown; I can execute "mysqladmin ping" from within the
>>> vserver or from the host with no issues while the vserver is
>>> running.

>> Let's see what a 'vserver --debug <name> stop' produces

>> Thanks,
>> Herbert

>>>> Le 11/11/2015 23:50, Daniel Urist a écrit :

>>>> I've narrowed down the issue to the init script for the Percona XtraDB
>>>> mysql server. It seems to be crashing the vserver shutdown when it calls
>>>> "mysqladmin ping" to see whether the database is alive. I can run
>>>> "mysqladmin ping" successfully from within the vserver, and also from
>> the
>>>> host with "vserver exec". I can also successfully run "/etc/init.d/mysql
>>>> stop" from the host via "vserver exec". Is there something special that
>>>> happens during vserver shutdown that would cause this to fail?

>>>> On Mon, Nov 9, 2015 at 11:51 AM, Herbert Poetzl <herbert@13thfloor.at>
>>>> wrote:

>>>>> On Mon, Nov 09, 2015 at 10:49:15AM -0700, Daniel Urist wrote:
>>>>>> I am in the process of upgrading from debian wheezy to jessie
>>>>>> for both hosts and guests. All hosts and guests are running
>>>>>> sysv-rc (no systemd!). I'm getting the following error on
>>>>>> shutdown with jessie guests running on jessie hosts:

>>>>> Hey Daniel,

>>>>>>  /usr/share/util-vserver/vserver.stop: line 100:  3511 Killed
>>>>>>    "${IONICE CMD[@]}" "${NICE CMD[@]}" "${NETNS CMD[@]}"
>>>>> "${CHBIND CMD[@]}"
>>>>>> "$ VSPACE" --enter "$S CONTEXT" "${OPTS VSPACE[@]}"
>>>>>> "${OPTS VSPACE SHARED[@]}" -- "$ VTAG" --migrate
>> "${OPTS VTAG ENTER[@]}"
>>>>>> --silent -- $ VCONTEXT $SILENT OPT --migrate $OPT VCONTEXT CHROOT
>> --xid
>>>>>> "$S CONTEXT" -- "${INITCMD STOP[@]}"

>>>>> This looks like the rc scripts executed to stop the guest
>>>>> kill the util-vserver stop script at some point.

>>>>>> This doesn't happen with wheezy guests running on jessie hosts,

>>>>> This is kind of suspected, as it is most likely a minor
>>>>> "problem" with the guest side rc scripts.

>>>>>> or with jessie guests on wheezy hosts.

>>>>> This is very interesting, are you sure that the very
>>>>> same guest does not exhibit this behaviours regardless
>>>>> of the host system (given that util-vserver is the same
>>>>> version).

>>>>>> If I switch from init style sysv to plain, I don't see any
>>>>>> errors, but I'm assuming that just suppresses the message
>>>>>> since the vserver shuts down almost instantaneously;

>>>>> An almost instant shutdown is not a bad thing per se, after
>>>>> all container style virtualization is very fast. This
>>>>> setup will not call any scripts inside the guest directly,
>>>>> instead it will signal the init process inside the guest
>>>>> to shut down. If everything goes as expected, your guest
>>>>> should simply disappear (no more processes in the guest
>>>>> context)

>>>>>> I'm concerned that it's not going down cleanly.

>>>>> It might be a good idea to check what scripts are actually
>>>>> executed inside the guest and supplement them with debug
>>>>> output (or run them with -x for bash scripts).

>>>>>> The kernel version for jessie is 3.18.21-vs2.3.7.4-beng (from
>>>>>> repo.psand.net), util-vserver is 0.30.216-pre3120-jessie0.1-1.

>>>>>> Any hints on how to fix or at least debug this?

>>>>> You can also run util-vserver with the --debug option,
>>>>> which might shed some light on the issue.

>>>>> HTH,
>>>>> Herbert