Subject: Re: [vserver] Re: [Freedombox-discuss] [vserver] Re: A software architecture for the FreedomBox
From: Gordan Bobic <gordan@bobich.net>
Date: Thu, 14 Apr 2011 10:36:53 +0100

Eugen Leitl wrote:

>>>>> One of the first things you have to do when building a
>>>>> system like the FreedomBox is figure out the software
>>>>> architecture. I have come up with an interesting
>>>>> architecture that is made of Linux Containers (Virtual
>>>> Using Linux vserver guests for service separation (jails on
>>>> steroids) is an excellent idea, actually.
>>> I find it interesting, but am sceptical: I believe each
>>> jail consume separate memory for their libraries - i.e.
>>> cannot benefit from shared libraries. So I worry about memory
>>> consumption.
>> mainly depends on the guest setup, assuming that the
>> guests use identical libraries and the on disk data
>> is properly unified, then both, the inode cache and
>> the page mappings will use the very same memory
> 
> Vserver unification hasn't been something really solid in the past, often
> brokeni, at least in Debian.

I've only seen one instance of it being broken, and I posted about it on 
the list. It was on ext4 fs back before ext4 was actually deemed stable, 
and all the newer kernels since have worked just fine. I suspect you 
just had a duff kernel. Did you roll it yourself or did you use a 
pre-packaged one?

Here's a link to the thread on the subject from last year:
http://www.paul.sladen.org/vserver/archives/201010/0018.html

> Note that the vserver patch support in Debian will stop with wheezy. LXC
> might be a better choice in the long term.

Frankly, the only advantage LXC has is that it's in the mainline. Having 
argued a case for a hashify feature in it before, my impression that the 
chances of having it in LXC is effectively 0.

Here's a thread where I brought it up on the LXC list a while back:
http://www.mail-archive.com/lxc-users@lists.sourceforge.net/msg00433.html

 From what I can tell, LXC maintainers think file unification is 
something for the file system to take care of and point at things like 
snapshots (e.g. in BTRFS) and deduplication (last time I was involved in 
a discussion on deduping on the BTRFS list I was sufficiently 
disappointed by the wrongness of the proposed designs I just gave up - 
thankfully, we now have a fully working ZFS port for Linux so I never 
have to look at BTRFS again). BTRFS guys neither seemed to comprehend 
the importance of CoW hard-links nor had any interest in implementing 
such a feature.

Here's the relevant thread on the BTRFS list:
http://www.spinics.net/lists/linux-btrfs/msg05224.html

Given the lack of interest and understanding from both the LXC camp and 
the BTRFS camp, I don't actually see any alternative to the efficiency 
of vserver any time soon. There are other bodge solutions that may be 
viable, such as marking all memory for same-page merging using 
LD_PRELOAD intercepts of malloc(), as I mentioned here:
http://www.spinics.net/lists/kvm/msg36193.html
but that would still be relatively expensive - KSM iterates through and 
dedupes all the madvise()-ed memory every few seconds. In comparison, 
the vserver's hashify feature is free in terms of run-time overheads.

Gordan