Subject: Re: [vserver] debian jessie error on vserver shutdown
From: Daniel Urist <durist@ucar.edu>
Date: Mon, 16 Nov 2015 10:03:57 -0700
Mon, 16 Nov 2015 10:03:57 -0700
I've attached the debug trace.

The line where the failure occurs is this:

/usr/share/util-vserver/vserver.stop: line 100: 30935 Killed
>    "${IONICE CMD[@]}" "${NICE CMD[@]}" "${NETNS CMD[@]}" "${CHBIND \
> CMD[@]}" "$ VSPACE" --enter "$S CONTEXT" "${OPTS VSPACE[@]}"
> "${OPTS VSPACE SHARED[@]}" -- "$ VTAG" --migrate "${OPTS VTAG ENTER[@]}"
> --silent\
>  -- $ VCONTEXT $SILENT OPT --migrate $OPT VCONTEXT CHROOT --xid
> "$S CONTEXT" -- "${INITCMD STOP[@]}"


I took a snapshot of the process table on both the host and vserver before
shutting it down, and at that time there was no process 30935, so I really
don't understand where that's coming from.


On Thu, Nov 12, 2015 at 11:49 PM, Herbert Poetzl <herbert@13thfloor.at>
wrote:

> On Thu, Nov 12, 2015 at 08:04:17AM -0700, Daniel Urist wrote:
> > On Wed, Nov 11, 2015 at 4:40 PM, Jean Weisbuch <jean@phpnet.org> wrote:
>
> >> The MySQL shutdown could be quite long, especially if you have
> >> a big buffer pool size or simply if you hit a bug and the
> >> process can be neverending.
>
> >> Are you sure that its not the case you are hitting and that
> >> "vserver stop" would timeout before MySQL would be stopped?
>
> > I don't think it's the timeout; shutting down the mysql server
> > by hand doesn't take more than a few seconds. I've also tried
> > bumping up the sync-timeout to 60 seconds with no difference.
>
> >> But if its really the "mysqladmin ping" that hangs its strange
> >> indeed, does it seems to hang before it crash or it directly
> >> crash when it tries to execute it?
>
> > From what I can see, it crashes as soon as it executes
> > "mysqladmin ping". Again, this only happens during vserver
> > shutdown; I can execute "mysqladmin ping" from within the
> > vserver or from the host with no issues while the vserver is
> > running.
>
> Let's see what a 'vserver --debug <name> stop' produces
>
> Thanks,
> Herbert
>
> >> Le 11/11/2015 23:50, Daniel Urist a écrit :
>
> >> I've narrowed down the issue to the init script for the Percona XtraDB
> >> mysql server. It seems to be crashing the vserver shutdown when it calls
> >> "mysqladmin ping" to see whether the database is alive. I can run
> >> "mysqladmin ping" successfully from within the vserver, and also from
> the
> >> host with "vserver exec". I can also successfully run "/etc/init.d/mysql
> >> stop" from the host via "vserver exec". Is there something special that
> >> happens during vserver shutdown that would cause this to fail?
>
> >> On Mon, Nov 9, 2015 at 11:51 AM, Herbert Poetzl <herbert@13thfloor.at>
> >> wrote:
>
> >>> On Mon, Nov 09, 2015 at 10:49:15AM -0700, Daniel Urist wrote:
> >>>> I am in the process of upgrading from debian wheezy to jessie
> >>>> for both hosts and guests. All hosts and guests are running
> >>>> sysv-rc (no systemd!). I'm getting the following error on
> >>>> shutdown with jessie guests running on jessie hosts:
>
> >>> Hey Daniel,
>
> >>>>  /usr/share/util-vserver/vserver.stop: line 100:  3511 Killed
> >>>>    "${IONICE CMD[@]}" "${NICE CMD[@]}" "${NETNS CMD[@]}"
> >>> "${CHBIND CMD[@]}"
> >>>> "$ VSPACE" --enter "$S CONTEXT" "${OPTS VSPACE[@]}"
> >>>> "${OPTS VSPACE SHARED[@]}" -- "$ VTAG" --migrate
> "${OPTS VTAG ENTER[@]}"
> >>>> --silent -- $ VCONTEXT $SILENT OPT --migrate $OPT VCONTEXT CHROOT
> --xid
> >>>> "$S CONTEXT" -- "${INITCMD STOP[@]}"
>
> >>> This looks like the rc scripts executed to stop the guest
> >>> kill the util-vserver stop script at some point.
>
> >>>> This doesn't happen with wheezy guests running on jessie hosts,
>
> >>> This is kind of suspected, as it is most likely a minor
> >>> "problem" with the guest side rc scripts.
>
> >>>> or with jessie guests on wheezy hosts.
>
> >>> This is very interesting, are you sure that the very
> >>> same guest does not exhibit this behaviours regardless
> >>> of the host system (given that util-vserver is the same
> >>> version).
>
> >>>> If I switch from init style sysv to plain, I don't see any
> >>>> errors, but I'm assuming that just suppresses the message
> >>>> since the vserver shuts down almost instantaneously;
>
> >>> An almost instant shutdown is not a bad thing per se, after
> >>> all container style virtualization is very fast. This
> >>> setup will not call any scripts inside the guest directly,
> >>> instead it will signal the init process inside the guest
> >>> to shut down. If everything goes as expected, your guest
> >>> should simply disappear (no more processes in the guest
> >>> context)
>
> >>>> I'm concerned that it's not going down cleanly.
>
> >>> It might be a good idea to check what scripts are actually
> >>> executed inside the guest and supplement them with debug
> >>> output (or run them with -x for bash scripts).
>
> >>>> The kernel version for jessie is 3.18.21-vs2.3.7.4-beng (from
> >>>> repo.psand.net), util-vserver is 0.30.216-pre3120-jessie0.1-1.
>
> >>>> Any hints on how to fix or at least debug this?
>
> >>> You can also run util-vserver with the --debug option,
> >>> which might shed some light on the issue.
>
> >>> HTH,
> >>> Herbert
>
>
>
>
>
>


I've attached the debug trace.

The line where the failure occurs is this:

/usr/share/util-vserver/vserver.stop: line 100: 30935 Killed                  "${IONICE CMD[@]}" "${NICE CMD[@]}" "${NETNS CMD[@]}" "${CHBIND \
CMD[@]}" "$ VSPACE" --enter "$S CONTEXT" "${OPTS VSPACE[@]}" "${OPTS VSPACE SHARED[@]}" -- "$ VTAG" --migrate "${OPTS VTAG ENTER[@]}" --silent\
 -- $ VCONTEXT $SILENT OPT --migrate $OPT VCONTEXT CHROOT --xid "$S CONTEXT" -- "${INITCMD STOP[@]}"

I took a snapshot of the process table on both the host and vserver before shutting it down, and at that time there was no process 30935, so I really don't understand where that's coming from.


On Thu, Nov 12, 2015 at 11:49 PM, Herbert Poetzl <herbert@13thfloor.at> wrote:
On Thu, Nov 12, 2015 at 08:04:17AM -0700, Daniel Urist wrote:
> On Wed, Nov 11, 2015 at 4:40 PM, Jean Weisbuch <jean@phpnet.org> wrote:

>> The MySQL shutdown could be quite long, especially if you have
>> a big buffer pool size or simply if you hit a bug and the
>> process can be neverending.

>> Are you sure that its not the case you are hitting and that
>> "vserver stop" would timeout before MySQL would be stopped?

> I don't think it's the timeout; shutting down the mysql server
> by hand doesn't take more than a few seconds. I've also tried
> bumping up the sync-timeout to 60 seconds with no difference.

>> But if its really the "mysqladmin ping" that hangs its strange
>> indeed, does it seems to hang before it crash or it directly
>> crash when it tries to execute it?

> From what I can see, it crashes as soon as it executes
> "mysqladmin ping". Again, this only happens during vserver
> shutdown; I can execute "mysqladmin ping" from within the
> vserver or from the host with no issues while the vserver is
> running.

Let's see what a 'vserver --debug <name> stop' produces

Thanks,
Herbert

>> Le 11/11/2015 23:50, Daniel Urist a écrit :

>> I've narrowed down the issue to the init script for the Percona XtraDB
>> mysql server. It seems to be crashing the vserver shutdown when it calls
>> "mysqladmin ping" to see whether the database is alive. I can run
>> "mysqladmin ping" successfully from within the vserver, and also from the
>> host with "vserver exec". I can also successfully run "/etc/init.d/mysql
>> stop" from the host via "vserver exec". Is there something special that
>> happens during vserver shutdown that would cause this to fail?

>> On Mon, Nov 9, 2015 at 11:51 AM, Herbert Poetzl <herbert@13thfloor.at>
>> wrote:

>>> On Mon, Nov 09, 2015 at 10:49:15AM -0700, Daniel Urist wrote:
>>>> I am in the process of upgrading from debian wheezy to jessie
>>>> for both hosts and guests. All hosts and guests are running
>>>> sysv-rc (no systemd!). I'm getting the following error on
>>>> shutdown with jessie guests running on jessie hosts:

>>> Hey Daniel,

>>>>  /usr/share/util-vserver/vserver.stop: line 100:  3511 Killed
>>>>    "${IONICE CMD[@]}" "${NICE CMD[@]}" "${NETNS CMD[@]}"
>>> "${CHBIND CMD[@]}"
>>>> "$ VSPACE" --enter "$S CONTEXT" "${OPTS VSPACE[@]}"
>>>> "${OPTS VSPACE SHARED[@]}" -- "$ VTAG" --migrate "${OPTS VTAG ENTER[@]}"
>>>> --silent -- $ VCONTEXT $SILENT OPT --migrate $OPT VCONTEXT CHROOT --xid
>>>> "$S CONTEXT" -- "${INITCMD STOP[@]}"

>>> This looks like the rc scripts executed to stop the guest
>>> kill the util-vserver stop script at some point.

>>>> This doesn't happen with wheezy guests running on jessie hosts,

>>> This is kind of suspected, as it is most likely a minor
>>> "problem" with the guest side rc scripts.

>>>> or with jessie guests on wheezy hosts.

>>> This is very interesting, are you sure that the very
>>> same guest does not exhibit this behaviours regardless
>>> of the host system (given that util-vserver is the same
>>> version).

>>>> If I switch from init style sysv to plain, I don't see any
>>>> errors, but I'm assuming that just suppresses the message
>>>> since the vserver shuts down almost instantaneously;

>>> An almost instant shutdown is not a bad thing per se, after
>>> all container style virtualization is very fast. This
>>> setup will not call any scripts inside the guest directly,
>>> instead it will signal the init process inside the guest
>>> to shut down. If everything goes as expected, your guest
>>> should simply disappear (no more processes in the guest
>>> context)

>>>> I'm concerned that it's not going down cleanly.

>>> It might be a good idea to check what scripts are actually
>>> executed inside the guest and supplement them with debug
>>> output (or run them with -x for bash scripts).

>>>> The kernel version for jessie is 3.18.21-vs2.3.7.4-beng (from
>>>> repo.psand.net), util-vserver is 0.30.216-pre3120-jessie0.1-1.

>>>> Any hints on how to fix or at least debug this?

>>> You can also run util-vserver with the --debug option,
>>> which might shed some light on the issue.

>>> HTH,
>>> Herbert







["application/octet-stream" not shown]