Subject: Re: [vserver] VServer looks 'limited' in 2.6.38.8
From: Herbert Poetzl <herbert@13thfloor.at>
Date: Tue, 21 Jun 2011 23:45:35 +0200

On Tue, Jun 21, 2011 at 11:27:50PM +0200, Adrian Reyer wrote:
> Hi,

> today I upgraded a Linux-DRBD-VServer-Heartbeat Cluster
> from Debian Lenny via Squeeze to the final kernel
> 2.6.38.8-vs2.3.0.37-rc17 with util-vserver 0.30.216-pre2981.

> This combination only starts maximum 24 of the 78 VServers that
> should be started. They are in 3 groups, each group has its own
> DRBD-device mounted like

> /var/lib/vservers/vservers_v1/vserver1
> /var/lib/vservers/vservers_v2/vserver2
> ...
> /var/lib/vservers/vservers_v2/vserver70
> /var/lib/vservers/vservers_v3/vserver71
> ...
> /var/lib/vservers/vservers_v3/vserver74

> The single VServer on vservers_v1 works fine, 3 of the 4 on
> vservers_v3 fail and only 18 succeed on vservers_v2.

> Startup gives this error about a failed server and to check
> with init if nothing else is printed.

> I tried with sysv and plain init style. No difference.

> I only updated one side of the cluster kernel-wise, the other
> still is debian squeeze 2.6.26-2-vserver-amd64 but uses
> util-vserver 0.30.216-pre2981 already.

I presume you're aware that the 2.6.26 (debian) kernel
is horribly broken and you need to fixup your filesystem
whenever you switch from or to that kernel?

> There all VServers start up just fine.

> If I copy a non-starting VServer to the very same location
> where it would be mounted it starts up fine.

> If I diff the /etc/vserver/vserver56 and /etc/vserver/vserver57, 
> one is starting correctly, the other not, there is no difference.

diff won't fine the differences in the filesystem
attributes, and that's where the 2.6.26er kernel
screws up (among other things)

> The VServers that startup fine have a new inode ctime in /,
> /dev and /etc, those that fail to start still have the same old
> ctime they had on creation, sometime between 2009 and today. It
> looks like some files there get changed, but the contents of
> these files is exactly the same like in a non-starting VServer.

> The 3 groups of VServers have 3 different marks to enable me
> starting them with the util-vserver init.d-stuff, no mater
> if I start the default 6 or 100 in parallel, no matter how
> often I start them, it is always exactly the same VServers that
> actually run. If I stop one and try to start one of the others
> it still fails.

> The side with the older kernel, currently running the VServers:
> sirius-c1:~# cat /proc/virtual/status
> #CTotal:        85
> #CActive:       85
> #NSProxy:       170     1203 85 85 85 85 0

> sirius-c1:~# cat /proc/virtual/info
> VCIVersion:     0002:0303
> VCISyscall:     236
> VCIKernel:      13000ff1

> Linux sirius-c1 2.6.26-2-vserver-amd64 #1 SMP Thu Feb 11 02:19:12 UTC 2010 x86_64
GNU/Linux

> The side with the new kernel, refusing to start all VServers:
> sirius-c2:/etc/vservers# cat /proc/virtual/status
> #CTotal:        0
> #CActive:       0
> #NSProxy:       0       72 0 0 1 0 0
> #InitTask:      2       190 116

> sirius-c2:~# cat /proc/virtual/info
> VCIVersion:     0002:0308
> VCISyscall:     236
> VCIKernel:      13000f11

> Linux sirius-c2 2.6.38.8-vs2.3.0.37-rc17 #2 SMP PREEMPT Tue Jun 21 16:51:31 CEST 2011
x86_64 GNU/Linux

> I am quite lost, what information can I provide to get a hint
> on what's wrong?

please make sure that the filesystem is intact when
switching from 2.6.26 to 2.6.38.x, best procedure is to
backup everything (or at least the attributes) and
restore once the new kernel is up an running

best,
Herbert


> Regards,
> 	Adrian
> -- 
> LiHAS - Adrian Reyer - Hessenwiesenstraße 10 - D-70565 Stuttgart
> Fon: +49 (7 11) 78 28 50 90 - Fax:  +49 (7 11) 78 28 50 91
> Mail: lihas@lihas.de - Web: http://lihas.de
> Linux, Netzwerke, Consulting & Support - USt-ID: DE 227 816 626 Stuttgart