Subject: Re: [vserver] Pros and Cons of 32/64bit guests on 64bit host
From: Ed W <lists@wildgooses.com>
Date: Thu, 21 Aug 2008 10:49:22 +0100
Thu, 21 Aug 2008 10:49:22 +0100
Oliver Welter wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi All,
>
> I am running vserver on 32bit boxes for years now. I am know replacing
> one group of nodes with nice Dual AMD Quad-Cores which run a 64bit
> debian host OS.
>
> As I have to update the guests anyway (I used and will use gentoo for
> them), so I am considering to move them to 64bit, too. The guests are
> several LAMP servers, mysql and postgres and a set of qmail nodes.
>
> The only issue that is an obvious go for 64bit would be memory segments
>   
>> 3GB, but I guess I wont have guests who use it. So, anybody here can
>>     
> give pros and cons for 32/64 bit guests?
>   

I have just done this on my gentoo server...

I had initially used quite a few vservers for each main application, so 
it was quite easy to upgrade them one by one.  First thing I did was to 
get a generic guest install for each starting point, eg one for apache, 
one for nginx and one cleanish build for starting other types of 
servers.  I customise these with my main options and get the make.conf 
setup as I want it.  This makes it dramatically easier to keep all 
vservers the same.  I use a package dir which is shared between all 
machines and this means that when I upgrade them the first takes a bit 
of time compiling and the others then install in just seconds (I 
personally find gentoo with packages to be much faster than a centos 
server I have to update - plus all the time to merge files in /etc with 
centos...)

I setup my vservers with all the data in a separate mount to the main 
OS, eg /vservers/images/server1 and /vserver/shared/server1/ and in the 
shared dir  I might mount /var/www for example for a websever.  This 
allows me to do stuff like copying the main vserver, mounting the data 
from another vserver and quickly test an upgrade (obviously needs care) 
without having to copy all the data for a running OS which can be 
substantial.  In this way I can also run different backup strategies for 
the OS to the data used by a vserver.  It's easy to see if you got it 
right - just do a "du" in the vserver dir and if it's more than a couple 
hundred MB (in my case) then you missed something that needs to go in 
the data partition!  I find "ncdu" very helpful to locate the disk hogs

Problems I have found though are that my PHP vservers take MUCH more 
(virtual) memory under 64bit than on 32bit... I am a bit baffled as to 
why, but it's about 4x the VSS allocation and a smaller multiple of the 
RSS allocation (the RSS bit fits with having data being up to twice the 
size, but the VSS allocation is bothering me)

I found switching to nginx + fastcgi dropped the memory consumption 
quite a bit and gave me more control over the number of running 
processes and this was a good solution in my case (note though that all 
the nginx php examples seem to be flawed and create security holes for 
apps which have any kind of upload directory (ie the old upload 
"nasty.php" and then navigate to it via the app and the server will now 
execute it - instead always specify your exact paths which should be PHP 
executable and avoid upload locations)

However, I noticed no real speed difference having switched to 64bit...  
I am now mostly 64bit, but by splitting out the data and the OS as 
described above, upgrading any vserver is generally very easy because 
you can even test the new install on the live data (carefully) and most 
of my installs all derive from a single part setup image which makes it 
a cinch to rollout new vservers

Also I have a problem with a segfault if I resize my SSH terminal while 
running "vserver xyz enter" and being in the resulting shell.  The 
segfault is somewhere in the vserver client utils and the host server is 
running gentoo hardened + 64bit.  I am hoping someone else will now pipe 
up and say it's not happening to them on 64bit which would suggest that 
it's related to some feature of grsecurity in the host OS.  It's a bit 
of a pain, but it's not yet bothered me enough to actually go and get a 
backtrace and fix it...  (suggestions welcome though!)

I would recommend everyone to go 64bit on the host - it gives you 
maximum options.  As for the guests though I think just pick whatever 
works for you, but my experience is that all the mainstream (server) 
apps are now 64bit safe and you should pick that as your first choice, 
but just upgrade at your leisure (yep I just rsyned my 32bit vservers 
across from the old 32bit host to the new 64bit host and roughly 
speaking they just worked straight away)


Good luck

Ed W


Oliver Welter wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi All,

I am running vserver on 32bit boxes for years now. I am know replacing
one group of nodes with nice Dual AMD Quad-Cores which run a 64bit
debian host OS.

As I have to update the guests anyway (I used and will use gentoo for
them), so I am considering to move them to 64bit, too. The guests are
several LAMP servers, mysql and postgres and a set of qmail nodes.

The only issue that is an obvious go for 64bit would be memory segments
  
3GB, but I guess I wont have guests who use it. So, anybody here can
    
give pros and cons for 32/64 bit guests?
  

I have just done this on my gentoo server...

I had initially used quite a few vservers for each main application, so it was quite easy to upgrade them one by one.  First thing I did was to get a generic guest install for each starting point, eg one for apache, one for nginx and one cleanish build for starting other types of servers.  I customise these with my main options and get the make.conf setup as I want it.  This makes it dramatically easier to keep all vservers the same.  I use a package dir which is shared between all machines and this means that when I upgrade them the first takes a bit of time compiling and the others then install in just seconds (I personally find gentoo with packages to be much faster than a centos server I have to update - plus all the time to merge files in /etc with centos...)

I setup my vservers with all the data in a separate mount to the main OS, eg /vservers/images/server1 and /vserver/shared/server1/ and in the shared dir  I might mount /var/www for example for a websever.  This allows me to do stuff like copying the main vserver, mounting the data from another vserver and quickly test an upgrade (obviously needs care) without having to copy all the data for a running OS which can be substantial.  In this way I can also run different backup strategies for the OS to the data used by a vserver.  It's easy to see if you got it right - just do a "du" in the vserver dir and if it's more than a couple hundred MB (in my case) then you missed something that needs to go in the data partition!  I find "ncdu" very helpful to locate the disk hogs

Problems I have found though are that my PHP vservers take MUCH more (virtual) memory under 64bit than on 32bit... I am a bit baffled as to why, but it's about 4x the VSS allocation and a smaller multiple of the RSS allocation (the RSS bit fits with having data being up to twice the size, but the VSS allocation is bothering me)

I found switching to nginx + fastcgi dropped the memory consumption quite a bit and gave me more control over the number of running processes and this was a good solution in my case (note though that all the nginx php examples seem to be flawed and create security holes for apps which have any kind of upload directory (ie the old upload "nasty.php" and then navigate to it via the app and the server will now execute it - instead always specify your exact paths which should be PHP executable and avoid upload locations)

However, I noticed no real speed difference having switched to 64bit...  I am now mostly 64bit, but by splitting out the data and the OS as described above, upgrading any vserver is generally very easy because you can even test the new install on the live data (carefully) and most of my installs all derive from a single part setup image which makes it a cinch to rollout new vservers

Also I have a problem with a segfault if I resize my SSH terminal while running "vserver xyz enter" and being in the resulting shell.  The segfault is somewhere in the vserver client utils and the host server is running gentoo hardened + 64bit.  I am hoping someone else will now pipe up and say it's not happening to them on 64bit which would suggest that it's related to some feature of grsecurity in the host OS.  It's a bit of a pain, but it's not yet bothered me enough to actually go and get a backtrace and fix it...  (suggestions welcome though!)

I would recommend everyone to go 64bit on the host - it gives you maximum options.  As for the guests though I think just pick whatever works for you, but my experience is that all the mainstream (server) apps are now 64bit safe and you should pick that as your first choice, but just upgrade at your leisure (yep I just rsyned my 32bit vservers across from the old 32bit host to the new 64bit host and roughly speaking they just worked straight away)


Good luck

Ed W