Subject: Copy-on-write Hard Links, Shared Libraries, Prelink and Memory
From: Gordan Bobic <gordan@bobich.net>
Date: Tue, 08 Jun 2010 18:14:43 +0100

I apologize in advance if this is a silly question, but I am not 
familiar enough with the low level workings of mmap() on Linux to know 
the answer, so I'll ask.

I understand that VServer has a feature to de-dupe identical files into 
copy-on-write marked hard-links. My questions are:

1) How does this approach co-exist with prelink (daily cron job on most 
distributions)? This modifies the binaries, and different VMs, unless 
they are all identical, are likely to end up with files getting unshared 
very quickly. Is the only available solution to un-prelink everything 
and disable prelink? Or is there a way to get both the performance 
advantage of perlink and the storage space saving (and caching 
efficiency savings)?

2) I've been pondering how something like KSM could be used for all 
memory on a physical host rather than having to patch every package, 
almost to the point of renaming malloc() and wrapping it so that all 
malloc()-ed memory gets marked by madvise(). And that got me pondering 
how mmap() (from what I can find out, used to access shared libraries), 
works. I know from past experience that the Linux kernel is smart enough 
to not double-allocate the memory when mmap()-ing a file on tmpfs. So, 
what happens when multiple files that are hard-linked get mmap()-ed? If, 
say, glibc is merged between two VServer VMs into a single file with two 
hard-links, will it's memory be allocated once per VM that accesses it, 
or will it all be mapped by the same physical block of the shared memory 
across all the VMs? All this assuming there isn't something clever going 
on (e.g. LD_PRELOAD, of some sort) that somehow marks the VM's 
individual process memory allocation with madvise() so KSM can operate 
on it.

Many thanks in advance.

Gordan