On Tue, 2009-03-31 at 07:01 -0700, Roderick A. Anderson wrote: > John A. Sullivan III wrote: > > On Tue, 2009-03-31 at 08:40 +0000, Christoph Lukas wrote: > >> Hi John, > >> > >> Am Dienstag, den 31.03.2009, 04:08 -0400 schrieb John A. Sullivan III: > >>> On Tue, 2009-03-31 at 06:51 +0000, Christoph Lukas wrote: > >>>> Hi John, > >>>> > >>>>> Hello, all. In our earlier deployments on kernel 2.6.22, we were very > >>>>> happy with the results of vhashify. For some reason, our use in > >>>>> 2.6.27.14 using vserver 2.3.0.36.4 does not seem to be working as well. > >>>>> This is important to us because we are planning roughly 400 nearly > >>>>> identical guests on this one host. > >>>>> > >>>>> We currently have 10 Ubuntu 8.0.4 guests running on our CentOS 5.2 based > >>>>> host. Each one is roughly 2GB in size and were all cloned from the same > >>>>> template. Total storage on the vserver partition is roughly 21GB. The > >>>>> only unusual bit about this installation is there is /vservers/vetc > >>>>> directory which is then mounted via bind to /etc/vservers. This was > >>>>> originally because there was a single encrypted partition mounted via > >>>>> iSCSI holding all the vserver information. > >>>>> > >>>>> We've done: > >>>>> mkdir /etc/vservers/.defaults/apps/vunify/hash /vservers/.hash > >>>>> ln -s /vservers/.hash /etc/vservers/.defaults/apps/vunify/hash/root > >>>>> > >>>>> We noticed there is another > >>>>> link, /etc/vservers/.defaults/apps/vunify/hash/00 > >>>>> and hashify complained of a duplicate "root"directory so we deleted the > >>>>> root symlink. With or without it, we have the same results. > >>>> are there any hardlinks visible inside the /vservers/.hash directory? If > >>>> not the hashify did not work. > >>>> > >>>> You can check which files inside the vservers are not unified by running > >>>> > >>>> find /vserver/<guest> -type f -links 1 > >>>> > >>>> and the unified files by running: > >>>> > >>>> find /vserver/<guest> -type f -links +1 > >>>> > >>>> This should give you a hint if vhashify worked correctly. > >>>> > >>>>> For each vserver guest, we did: > >>>>> mkdir /etc/vservers/<name>/apps/vunify > >>>>> > >>>>> Has something changed with 2.6.27.14? > >>>> AFAIK unification is just done in userspace it should not depend on the > >>>> kernel version. Just the copy-on-write-link-breakage is done in the > >>>> kernel. > >>>> > >>>>> Are these realistic numbers? > >>>> Does not seem realistic to me. > >>>> > >>>>> I > >>>>> would think ten identical systems should yield just slightly more than > >>>>> the space of one system after running hashify. > >>>> I would guess something with your .hash directories is not setup > >>>> correctly and therefore the hashify did nor work as expected. You can > >>>> try to fix this and then just run: > >>>> > >>>> vserver <guest> hashify > >>>> > >>>> on a running guest. I have setup unification successfully here and used > >>>> this two wiki pages as howto: > >>>> > >>>> http://linux-vserver.org/Frequently_Asked_Questions#Unification > >>>> > >>>> http://linux-vserver.org/util-vserver:Vhashify > >>>> > >>>> Hope this helps, > >>>> Christoph > >>>> > >>> Thank you, Christoph. The commands show plenty of both types of files > >>> with the hard links clearly outweighing the regular files. Strange that > >>> I still show 20GB for 10 servers. Take care - John > >> Are you sure that your measurements of disk usage are correct? > >> Do the 20 GB disk usage correspond with the df output on the /vservers > >> partition? > >> > >> du -sch /vservers/* > >> > >> should notice multiple hardlinks and count them only once. > > I have this in my script. > > > if [ $( find /vservers/.hash -type f -links 1 ) ] > then > find /vservers/.hash -type f -links 1 -print0 | xargs -0 /bin/rm > else > echo No dangling links. > fi > > > Would this make a difference? > > > Rod Thanks but I don't think so unless I got it wrong in the script we use: #!/bin/sh c=1 for LINE in $(vserver-stat) do if [ $c -gt 8 -a $((c % 8)) -eq 0 ];then vserver ${LINE} hashify if [ $? -ne 0 ];then logger -t "HASHIFY ERROR " "Could not hashify ${LINE}" fi fi ((c = c + 1)) done find /vservers/.hash -type f -links 1 -print0 | xargs -0 rm -f I've also run hashify manually on each system in case the script was misbehaving. Thanks anyway - John > -- > >> > >> Regards, > >> Christoph > >> > >> > > Yes, the numbers match and they also match the size of the space > > consumed on the thinly provisioned ZFS zvol on which they reside (28.1 > > GB): > > [root@vd01 ~]# du -sch /vservers/* > > 4.4G /vservers/cle > > 2.2G /vservers/gssstation > > 2.2G /vservers/jasstation > > 1.4G /vservers/jintra > > 16K /vservers/lost+found > > 1.4G /vservers/mintra > > 1.4G /vservers/mlapo > > 1.4G /vservers/simple > > 1.4G /vservers/smcc > > 1.4G /vservers/tkee > > 1.4G /vservers/tvan > > 1.4G /vservers/vdb > > 1.9G /vservers/vdb2 > > 1.4G /vservers/vde > > 1.1M /vservers/vetc > > 23G total > > > > I am guessing this may also affect my memory usage. Does VServer use > > the hashify information to determine whether a binary being called into > > memory already exists? Thanks - John > -- John A. Sullivan III Open Source Development Corporation +1 207-985-7880 jsullivan@opensourcedevel.com http://www.spiritualoutreach.com Making Christianity intelligible to secular society