Subject: Re: [vserver] Copy-on-write Hard Links, Shared Libraries, Prelink and Memory
From: Herbert Poetzl <herbert@13thfloor.at>
Date: Wed, 9 Jun 2010 00:11:59 +0200

On Tue, Jun 08, 2010 at 06:14:43PM +0100, Gordan Bobic wrote:
> I apologize in advance if this is a silly question, but I am not 
> familiar enough with the low level workings of mmap() on Linux to know 
> the answer, so I'll ask.
> 
> I understand that VServer has a feature to de-dupe identical files into 
> copy-on-write marked hard-links. My questions are:
> 
> 1) How does this approach co-exist with prelink (daily cron job on most 
> distributions)? This modifies the binaries, and different VMs, unless 
> they are all identical, are likely to end up with files getting unshared 
> very quickly. Is the only available solution to un-prelink everything 
> and disable prelink? Or is there a way to get both the performance 
> advantage of perlink and the storage space saving (and caching 
> efficiency savings)?
> 
> 2) I've been pondering how something like KSM could be used for all 
> memory on a physical host rather than having to patch every package, 
> almost to the point of renaming malloc() and wrapping it so that all 
> malloc()-ed memory gets marked by madvise(). And that got me pondering 
> how mmap() (from what I can find out, used to access shared libraries), 
> works. I know from past experience that the Linux kernel is smart enough 
> to not double-allocate the memory when mmap()-ing a file on tmpfs. So, 
> what happens when multiple files that are hard-linked get mmap()-ed? If, 
> say, glibc is merged between two VServer VMs into a single file with two 
> hard-links, will it's memory be allocated once per VM that accesses it, 
> or will it all be mapped by the same physical block of the shared memory 
> across all the VMs? All this assuming there isn't something clever going 
> on (e.g. LD_PRELOAD, of some sort) that somehow marks the VM's 
> individual process memory allocation with madvise() so KSM can operate 
> on it.
> 
> Many thanks in advance.
> 
> Gordan

see previous answer :)

best,
Herbert