Subject: Re: [vserver] debian jessie error on vserver shutdown
From: Herbert Poetzl <herbert@13thfloor.at>
Date: Fri, 13 Nov 2015 07:49:28 +0100

On Thu, Nov 12, 2015 at 08:04:17AM -0700, Daniel Urist wrote:
> On Wed, Nov 11, 2015 at 4:40 PM, Jean Weisbuch <jean@phpnet.org> wrote:

>> The MySQL shutdown could be quite long, especially if you have
>> a big buffer_pool_size or simply if you hit a bug and the
>> process can be neverending.

>> Are you sure that its not the case you are hitting and that
>> "vserver stop" would timeout before MySQL would be stopped?

> I don't think it's the timeout; shutting down the mysql server
> by hand doesn't take more than a few seconds. I've also tried
> bumping up the sync-timeout to 60 seconds with no difference.

>> But if its really the "mysqladmin ping" that hangs its strange
>> indeed, does it seems to hang before it crash or it directly
>> crash when it tries to execute it?

> From what I can see, it crashes as soon as it executes
> "mysqladmin ping". Again, this only happens during vserver
> shutdown; I can execute "mysqladmin ping" from within the
> vserver or from the host with no issues while the vserver is
> running.

Let's see what a 'vserver --debug <name> stop' produces

Thanks,
Herbert

>> Le 11/11/2015 23:50, Daniel Urist a écrit :

>> I've narrowed down the issue to the init script for the Percona XtraDB
>> mysql server. It seems to be crashing the vserver shutdown when it calls
>> "mysqladmin ping" to see whether the database is alive. I can run
>> "mysqladmin ping" successfully from within the vserver, and also from the
>> host with "vserver exec". I can also successfully run "/etc/init.d/mysql
>> stop" from the host via "vserver exec". Is there something special that
>> happens during vserver shutdown that would cause this to fail?

>> On Mon, Nov 9, 2015 at 11:51 AM, Herbert Poetzl <herbert@13thfloor.at>
>> wrote:

>>> On Mon, Nov 09, 2015 at 10:49:15AM -0700, Daniel Urist wrote:
>>>> I am in the process of upgrading from debian wheezy to jessie
>>>> for both hosts and guests. All hosts and guests are running
>>>> sysv-rc (no systemd!). I'm getting the following error on
>>>> shutdown with jessie guests running on jessie hosts:

>>> Hey Daniel,

>>>>  /usr/share/util-vserver/vserver.stop: line 100:  3511 Killed
>>>>    "${IONICE_CMD[@]}" "${NICE_CMD[@]}" "${NETNS_CMD[@]}"
>>> "${CHBIND_CMD[@]}"
>>>> "$_VSPACE" --enter "$S_CONTEXT" "${OPTS_VSPACE[@]}"
>>>> "${OPTS_VSPACE_SHARED[@]}" -- "$_VTAG" --migrate "${OPTS_VTAG_ENTER[@]}"
>>>> --silent -- $_VCONTEXT $SILENT_OPT --migrate $OPT_VCONTEXT_CHROOT --xid
>>>> "$S_CONTEXT" -- "${INITCMD_STOP[@]}"

>>> This looks like the rc scripts executed to stop the guest
>>> kill the util-vserver stop script at some point.

>>>> This doesn't happen with wheezy guests running on jessie hosts,

>>> This is kind of suspected, as it is most likely a minor
>>> "problem" with the guest side rc scripts.

>>>> or with jessie guests on wheezy hosts.

>>> This is very interesting, are you sure that the very
>>> same guest does not exhibit this behaviours regardless
>>> of the host system (given that util-vserver is the same
>>> version).

>>>> If I switch from init style sysv to plain, I don't see any
>>>> errors, but I'm assuming that just suppresses the message
>>>> since the vserver shuts down almost instantaneously;

>>> An almost instant shutdown is not a bad thing per se, after
>>> all container style virtualization is very fast. This
>>> setup will not call any scripts inside the guest directly,
>>> instead it will signal the init process inside the guest
>>> to shut down. If everything goes as expected, your guest
>>> should simply disappear (no more processes in the guest
>>> context)

>>>> I'm concerned that it's not going down cleanly.

>>> It might be a good idea to check what scripts are actually
>>> executed inside the guest and supplement them with debug
>>> output (or run them with -x for bash scripts).

>>>> The kernel version for jessie is 3.18.21-vs2.3.7.4-beng (from
>>>> repo.psand.net), util-vserver is 0.30.216-pre3120-jessie0.1-1.

>>>> Any hints on how to fix or at least debug this?

>>> You can also run util-vserver with the --debug option,
>>> which might shed some light on the issue.

>>> HTH,
>>> Herbert