Subject: Re: [vserver] guest inodes full => kernel panic !?!
From: Herbert Poetzl <herbert@13thfloor.at>
Date: Tue, 7 Jul 2009 16:19:38 +0200

On Mon, Jul 06, 2009 at 07:46:07PM +0200, Jarry wrote:
> Hi,
> 
> I just suffered the worst nightmare of any server-admin:
> kernel panic! This is the way it happened:
> 
> I tried to update gcc on all my guests. Because they run gentoo
> (as well as guest), updating takes rather long time (everything
> is compilled from sources). After a couple of hours I found out,
> upgrade process failed on some guests. After a little search
> I discovered (with vdlimit) those guests were short of inodes:
> ...
> inodes_used=99996
> inodes_total=100000
> ...
> 
> I increased limits in /etc/vservers/<vs>/dlimits/root/inodes_total
> and tried to stop & start all vservers:
> /etc/init.d/vserver.default stop
> Stopping default vservers...
> 
> (maybe I should have first stop vserver-guests, then increase
>  inodes_total limits)
> 
> And nothing more. Network connection broken and I could not connect
> to my server any more. So I rushed to server-room, attached screen
> just to see "Kernel panic..."!

well, it would be necessary to know what the kernel
paniced about, i.e. get the full stack trace which
accompanies such a panic (usually) ... you might
gather that one with a serial console ...

> I did hard reset and everything seems to work for now. But I'm very
> surprised, full inodes-limit in vserver-guest could bring the whole
> host-server down! The only strange entry I found in messages is:
> 
> Jul 6 16:50:56 obel vxW: [xid #0] !!! limit: ffff8101373b7078[VM,9] = 33 
> on exit.

that is unusual for the kernel you described below,
maybe you could give the exact details, as there never
was a 2.2.0.7 patch for 2.6.22 :)

> Otherwise no trace. Kernel 2.6.22, vserver 2.2.0.7. All guest and
> host are gentoo, stable and updated.
> 
> Disk space was definitelly not a problem; all guests had ~10GB free.
> I also have "10" in /etc/vservers/<vs>/dlimits/root/reserved ,
> but this is probably related only to disk-space, not inodes number
> (can someone from dev confirm?). If it is so, there should definitelly
> be some reserved-limit for inodes too...

well, the root reserve is something relevant for
users vs root account, it is not relevant for guest
vs host, and while the dlimit will deny guest processes
any creation beyond the limit, it should not cause the
kernel to panic ... unless there is a bug somewhere
(which brings us back to the kernel panic trace)

> I'm not going to test it again, but something like this should
> really *never* happen...

well, kernel panics per se should really *never* happen,
but they do, so is life, all we can do is narrow down
what the actual issue is and make sure that it doesn't
happen _again_ (presumed that it is in Linux-VServer code)

best,
Herbert

> Jarry
> 
> -- 
> _______________________________________________________________
> This mailbox accepts e-mails only from selected mailing-lists!
> Everything else is considered to be spam and therefore deleted.