Subject: Re: [vserver] hashify and memory saving
From: Herbert Poetzl <herbert@13thfloor.at>
Date: Wed, 6 Jan 2016 02:31:11 +0100

On Mon, Jan 04, 2016 at 01:51:17AM +0100, Tor Rune Skoglund wrote:
> Den 01. jan. 2016 22:25, skrev Herbert Poetzl:
>> On Fri, Jan 01, 2016 at 09:37:52PM +0100, Tor Rune Skoglund wrote:
>>> Having been a happy linux-vserver user for more than 10 years
>>> now, it was about time to test the hashify feature. The disk
>>> savings are obvious, and easily measured, but I have been
>>> trying a lot harder to measure any possible run-time memory
>>> savings.

>>> For the testing, I created a simple template LAMP guest, and
>>> a lot of hashified guests cloned from that one. I am unable
>>> to measure noticeably less memory usage when running multiple
>>> hashified guests compared to non-hashified ones using free and
>>> /proc/meminfo/'s MemAvailable entry.
>>> However, this could very well be to shortcomings in my own
>>> understanding how this should work or what to look for.
>>> What should I look for regarding possible memory savings?
>>> Anyone with any pointers?

>> You won't see any memory savings with dynamic memory allocations
>> and you won't get any benefits on read-write mappings either,
>> but you should be able to see a reduction for read only mappings
>> like they happen when using static binaries or read only mapped
>> shared libraries as well as read only memory mapped data files.

> Thank you, Herbert. 

> Although reading a lot lately and trying my best to get a grip
> on how this works, I am still a newbie in this area.

> So please excuse me for continuing to ask possibly stupid
> questions.... ;)

No problem.

> As far as I can tell, all code and libraries are by default PIC
> now on my setup. (Is this a requirement?)

PIC (position independant code) is nice because it doesn't
require patching the code after it was loaded, but PIC is
not a requirement.

> Does your comment above then mean that all read only mappings
> can be shared across guests no matter their setting of the
> execute flag and the MAP_SHARED/MAP_PRIVATE flag?

Files do not get mapped in unix, inodes get mapped into
memory. Unification works by using hard links to share
identical files between guests in a "safe" way.

A mapping which results from simply reading a page from
an inode into memory without further modification will
result in a cache entry (inode -> mapping) which will be
reused when that page from the very same inode is mapped
again (at least that was how it worked in 2.4, but I
doubt that has changed since :)

> (In my test setup, based on greping /proc/*/maps for "r--p"    
> and "r--s", there are very few shared read only mapped files   
> ("r--s") compared to read only private ("r--p"). 

Shared read only vs private read only is only relevant
for pages which can be written by somebody but typically
(as you figured already) code and data from binaries and
libraries are usually not written to, so a read only
mapping basically means "get a reference to that page".

> It seems like almost every binary or .so has a considerable
> read-only private section which then will be part of the
> assumed memory savings.)

> If not, what should I look for --- e.g. using /proc/<pid>/maps,
> pmap or in some other way ?

> How does KSM ( https://en.wikipedia.org/wiki/Kernel_same-page_merging ) 
> play with linux-vserver? If at all?

As far as I know, same page merging is only active on 
pages which are explicitely registered for this service.
e.g. from kvm when allocating virtual machines or similar.

So for Linux-VServer there is no real benefit as the
KSM part won't be active for guests.

Nevertheless, you can runn applications with a preload
library which will advise the pages as candidates for
merging.

> Lastly, I am sorry if I am jumping to wrong conclusions
> somewhere here... Please feel free to brutally educate me. :)

The main problem is finding good test scenarios and 
using proper instrumentation to demonstrate a real
benefit.

Honestly I do not consider the savings worthwile for
the typical guest setups (aka virtual server hosting)
because both, memory and disk space have become very
cheap and the main consumers (like for example java :)
won't benefit from shared pages at all.

That said, in scenarios where you run a rather complex
binary code with a lot of read only data several thousand
times in parallel, the saving on memory might be easily
worthwhile.

Best,
Herbert

> BR,
> Tor Rune Skoglund, trs@swi.no

>> If I would devise a test to show the advantages, I would run a
>> binary which doesn't do many dynamic allocations but uses a lot
>> of code and/or libraries and run it as only process in each guest
>> with a few thousand guests in parallell, once with and without
>> unification in place.
>> Best,
>> Herbert

>>> This is Gentoo, util-vserver 0.30.216_pre3120, kernel Linux amd64
>>> 3.18.7-vs2.3.7.4.
>>> BR, Tor Rune Skoglund
>>> trs@swi.no