Subject: Re: [vserver] linux 4.4.113-2.3.9.6 kernel crash during "vserver ... stop"
From: Herbert Poetzl <herbert@13thfloor.at>
Date: Tue, 6 Feb 2018 18:54:09 +0100

On Tue, Feb 06, 2018 at 05:30:37PM +0100, Herbert Poetzl wrote:
> On Tue, Feb 06, 2018 at 05:25:19AM -0500, user3441 wrote:
>> Hello!

>> I have tried updating a 3.2.98-vserver kernel to a 4.4.113
>> vserver kernel. It seems to start up correctly. But when I
>> execute "vserver ... stop" on the command line, the kernel
>> crashes with ~50% probability. This did not happen with the
>> 3.2.98 kernel. When it crashes, this is printed on the screen
>> (the top of the trace has been cut, because it did not fit on
>> the screen):

>> [ 101.390624] [<ffffffff810cb927>] ? vprintk_emit+0x517/0x560
>> [ 101.390788] [<ffffffff810cba8a>] vprintk_default+0x1a/0x20
>> [ 101.390951] [<ffffffff81141fab>] printk+0x43/0x4b
>> [ 101.391113] [<ffffffff8109c939>] __schedule_bug+0x39/0xb0
>> [ 101.391277] [<ffffffff816816b5>] __schedule+0x6d5/0x860
>> [ 101.391433] [<ffffffff81681a7e>] preempt_schedule_irq+0x5e/0xa0
>> [ 101.391610] [<ffffffff8168834e>] retint_kernel+0x1b/0x2d
>> [ 101.391774] [<ffffffff812e567c>] ? dump_stack+0x8d/0xa1
>> [ 101.391937] [<ffffffff8109c95e>] __schedule_bug+0x5e/0xb0
>> [ 101.392102] [<ffffffff816816b5>] __schedule+0x6d5/0x860
>> [ 101.392266] [<ffffffff816816ba>] schedule+0x3a/0x90
>> [ 101.392429] [<ffffffff8107b450>] do_exit+0x7f0/0xc70
>> [ 101.392594] [<ffffffff810c146d>] ? trace_hardirqs_on+0xd/0x10
>> [ 101.392761] [<ffffffff81141488>] ? context_tracking_exit+0x18/0x20
>> [ 101.392925] [<ffffffff8107b95b>] do_group_exit+0x4b/0xc0
>> [ 101.393089] [<ffffffff8107b9df>] SyS_exit_group+0xf/0x10
>> [ 101.393254] [<ffffffff81687748>] entry_SYSCALL_64_fastpath+0x1c/0x91
>> [ 101.393429] Code: e8 ea 60 01 00 85 c0 74 09 80 3d 3a 08 8a 00 00 71 65 49 8b
>> 56 60 48 c7 c6 40 3d 96 81 48 63 cb 48 8b 82 d8 00 00 00 48 03 04 ce <4c> 00 20
>> 48 8b 52 48 48 85 d2 75 e9 e8 03 60 01 00 85 c0 74 0d
>> [ 101.396170] RIP  [<ffffffff810bdf0c>] cpuacct_charge+0x8c/0x1a0
>> [ 101.396367]  RSP <fff88023fc43ad0>
>> [ 101.396520] CR2: 000000000000f0e9
>> [ 101.396673] ---[ end trace 365c9bf61aab792c ]---

>> I have once seen this when connected via ssh:

>> test1:~# reboot

>> The system is going down for reboot NOW!
>> test1:~#
>> Message from syslogd@test1 at Feb  5 13:27:00 ...
>> kernel:[ 2326.768581] Kernel panic - not syncing: corrupted stack end detected inside
scheduler

>> Message from syslogd@test1 at Feb  5 13:27:00 ...
>> kernel:[ 2326.768581]
>> Write failed: Broken pipe
>> $

>> Also, there are these messages when booting (I think they are
>> triggered by vservers starting up):

> [stuff zapped]

>> [   50.082543] BUG: sleeping function called from invalid context at include/linux/vs_context.h:152
>> [   50.082832] in_atomic(): 1, irqs_disabled(): 0, pid: 3500, name: exim4
>> [   50.082994] no locks held by exim4/3500.
>> [   50.083167] Preemption disabled at:[<ffffffff8107b95b>] do_group_exit+0x4b/0xc0
>> [   50.083497]
>> [   50.083657] CPU: 5 PID: 3500 Comm: exim4 Not tainted 4.4.113-vs2.3.9.6 #1
>> [   50.083819] Hardware name: Supermicro X8SIL/X8SIL, BIOS 1.1 05/27/2010
>> [   50.083982]  0000000000000000 ffff880233e47e18 ffffffff812e5657 0000000000000000
>> [   50.084395]  ffff8800babd2200 ffff880233e47e40 ffffffff810a6069 ffffffff8182ef96
>> [   50.084814]  0000000000000098 0000000000000000 ffff880233e47e68 ffffffff810a6194
>> [   50.085253] Call Trace:
>> [   50.085410]  [<ffffffff812e5657>] dump_stack+0x68/0xa1
>> [   50.085576]  [<ffffffff810a6069>] ___might_sleep+0x159/0x240
>> [   50.085736]  [<ffffffff810a6194>] __might_sleep+0x44/0x80
>> [   50.085898]  [<ffffffff810ddca5>] exit_vx_info+0x45/0x60
>> [   50.086058]  [<ffffffff8107b412>] do_exit+0x7b2/0xc70
>> [   50.086245]  [<ffffffff810c146d>] ? trace_hardirqs_on+0xd/0x10
>> [   50.086407]  [<ffffffff81141488>] ? context_tracking_exit+0x18/0x20
>> [   50.086568]  [<ffffffff8107b95b>] do_group_exit+0x4b/0xc0
>> [   50.086727]  [<ffffffff8107b9df>] SyS_exit_group+0xf/0x10
>> [   50.086887]  [<ffffffff81687748>] entry_SYSCALL_64_fastpath+0x1c/0x91

>> Any ideas?

> Can you try this patch and see if it fixes the issue for you?

> http://vserver.13thfloor.at/Experimental/delta-exit-fix01.diff

Actually, better test this one as it is more in line with
the proven code from earlier patches ...

http://vserver.13thfloor.at/Experimental/delta-exit-fix02.diff

Thanks again,
Herbert

> Thanks in advance,
> Herbert