On Tue, Jun 08, 2010 at 06:13:50PM +0100, Gordan Bobic wrote: > I apologize in advance if this is a silly question, but I am > not familiar enough with the low level workings of mmap() on > Linux to know the answer, so I'll ask. > I understand that VServer has a feature to de-dupe identical > files into copy-on-write marked hard-links. My questions are: > 1) How does this approach co-exist with prelink (daily cron job > on most distributions)? when prelink modifies the library, the CoW link will be broken, and if it runs on every guest, there will be different library files which cannot be unified again > This modifies the binaries, and different VMs, unless they are > all identical, are likely to end up with files getting unshared > very quickly. correct .... > Is the only available solution to un-prelink everything and > disable prelink? well, you can live with the fact that the libraries are not unified, the executables will still benefit from unification as well as data files > Or is there a way to get both the performance advantage of > perlink and the storage space saving (and caching efficiency > savings)? IMHO the performance advantage of prelink is at least debateable (not just because of the fact that often relocation is still required) but you have the following option ... - disable prelink inside the guests - prelink the 'template' guest when created/updated - copy over the prelinked libraries to the guests > 2) I've been pondering how something like KSM could be used > for all memory on a physical host rather than having to > patch every package, almost to the point of renaming malloc() > and wrapping it so that all malloc()-ed memory gets marked > by madvise(). I presume with KSM you are referring to the Kernel Samepage Merging recently merged in mainline, and not the Kernel Security Modules or other things shortened to KSM :) > And that got me pondering how mmap() (from what I can find > out, used to access shared libraries), works. > I know from past experience that the Linux kernel is smart > enough to not double-allocate the memory when mmap()-ing > a file on tmpfs. that is correct, modern tmpfs does not allocate inode caches for stored files, and page mappings work from the same pages (given that they are mapable :) > So, what happens when multiple files that are hard-linked > get mmap()-ed? If, say, glibc is merged between two VServer > VMs into a single file with two hard-links, will its memory > be allocated once per VM that accesses it, or will it all > be mapped by the same physical block of the shared memory > across all the VMs? seems you are already stumbling over your nomenclature here (multiple files vs single file with two hard links :) so let's put that right first, and the answer will present itself: unix uses inodes to store data (content) and metadata (directory entries). files are special directory entries which point to a specific inode containing the actual data (the content). two (or many :) hardlinks are entries in directory inodes pointing to the same data (content), while having different names (pathes to be precise). mapping and reading of content is done based on inodes while lookup and naming is done based on directory entries. conclusion: hard links to the same inode will use the very same inode cache and thus end up providing the same physical page (once mapped) to all 'users' (with the appropriate virtual mapping of course). this is what makes unification a huge benefit not only disk but also memory wise when done properly :) > All this assuming there isn't something clever going on > (e.g. LD_PRELOAD, of some sort) that somehow marks the VM's > individual process memory allocation with madvise() so KSM > can operate on it. KSM is basically a way to find 'shared' data where an explicit 'sharing' can not be expressed, e.g. with VMs like KVM (it would be hard to tell the kernel that a file inside a loopback/partition should be shared between two different machines, possibly across different loop devices :) HTC, Herbert > Many thanks in advance. > Gordan