Subject: Re: [vserver] Copy-on-write Hard Links, Shared Libraries, Prelink and Memory
From: Herbert Poetzl <herbert@13thfloor.at>
Date: Wed, 9 Jun 2010 00:11:18 +0200

On Tue, Jun 08, 2010 at 06:13:50PM +0100, Gordan Bobic wrote:
> I apologize in advance if this is a silly question, but I am 
> not familiar enough with the low level workings of mmap() on 
> Linux to know the answer, so I'll ask.

> I understand that VServer has a feature to de-dupe identical 
> files into copy-on-write marked hard-links. My questions are:

> 1) How does this approach co-exist with prelink (daily cron job 
> on most distributions)? 

when prelink modifies the library, the CoW link will be
broken, and if it runs on every guest, there will be
different library files which cannot be unified again

> This modifies the binaries, and different VMs, unless they are 
> all identical, are likely to end up with files getting unshared 
> very quickly. 

correct ....

> Is the only available solution to un-prelink everything and 
> disable prelink? 

well, you can live with the fact that the libraries are
not unified, the executables will still benefit from
unification as well as data files

> Or is there a way to get both the performance advantage of 
> perlink and the storage space saving (and caching efficiency 
> savings)?

IMHO the performance advantage of prelink is at least
debateable (not just because of the fact that often
relocation is still required) but you have the following
option ...

 - disable prelink inside the guests
 - prelink the 'template' guest when created/updated
 - copy over the prelinked libraries to the guests


> 2) I've been pondering how something like KSM could be used 
> for all memory on a physical host rather than having to 
> patch every package, almost to the point of renaming malloc() 
> and wrapping it so that all malloc()-ed memory gets marked 
> by madvise(). 

I presume with KSM you are referring to the Kernel
Samepage Merging recently merged in mainline, and not
the Kernel Security Modules or other things shortened
to KSM :)

> And that got me pondering how mmap() (from what I can find 
> out, used to access shared libraries), works. 

> I know from past experience that the Linux kernel is smart 
> enough to not double-allocate the memory when mmap()-ing 
> a file on tmpfs. 

that is correct, modern tmpfs does not allocate
inode caches for stored files, and page mappings
work from the same pages (given that they are 
mapable :)

> So, what happens when multiple files that are hard-linked 
> get mmap()-ed? If, say, glibc is merged between two VServer 
> VMs into a single file with two hard-links, will its memory 
> be allocated once per VM that accesses it, or will it all 
> be mapped by the same physical block of the shared memory 
> across all the VMs? 

seems you are already stumbling over your nomenclature
here (multiple files vs single file with two hard links :)
so let's put that right first, and the answer will present
itself:

unix uses inodes to store data (content) and metadata
(directory entries).

files are special directory entries which point to a
specific inode containing the actual data (the content).

two (or many :) hardlinks are entries in directory inodes
pointing to the same data (content), while having different
names (pathes to be precise).

mapping and reading of content is done based on inodes
while lookup and naming is done based on directory entries.

conclusion:

hard links to the same inode will use the very same
inode cache and thus end up providing the same physical
page (once mapped) to all 'users' (with the appropriate
virtual mapping of course). this is what makes unification
a huge benefit not only disk but also memory wise when
done properly :)

> All this assuming there isn't something clever going on 
> (e.g. LD_PRELOAD, of some sort) that somehow marks the VM's 
> individual process memory allocation with madvise() so KSM 
> can operate on it.

KSM is basically a way to find 'shared' data where an
explicit 'sharing' can not be expressed, e.g. with VMs
like KVM (it would be hard to tell the kernel that a
file inside a loopback/partition should be shared between
two different machines, possibly across different loop
devices :)

HTC,
Herbert

> Many thanks in advance.
> Gordan