Subject: Re: [vserver] vhashified files and breaking the hard link in a vserver
From: Allan Latham <alatham@flexsys-group.de>
Date: Wed, 01 May 2013 17:51:52 +0200

Hi Herbert

Thank you for the quick reply.

Interesting result testing without aufs:

Mounted the ext3 fs in the vserver (--bind -rw) during the initialize
script and it now works even better than expected. The hardlink is made
correctly and is only broken when the file is touched:

Before: ls -li

16428 -rw-r--r-- 8 root root  530 Apr 13 09:37 README.app

After: ln README.app README.ajl; ls li

16428 -rw-r--r-- 9 root root  530 Apr 13 09:37 README.ajl
16428 -rw-r--r-- 9 root root  530 Apr 13 09:37 README.app

After: touch README.app; ls -li

16428 -rw-r--r-- 8 root root  530 Apr 13 09:37 README.ajl
24622 -rw-r--r-- 1 root root  530 May  1 15:40 README.app

Conclusion: aufs feature.

About lvm snapshots: I can achieve the same effect (one vserver can keep
a private rw copy of this data) by using snapshots but the downside is
that a backup of this vserver will include all the data. With aufs the
backup contains only the diff from the (large) fixed data.

I may have to live with the aufs feature. At least I know what the cause
is now!

During this testing I also discovered that the 'mv' command also uses
hardlinks (no surprise - should have guessed that) so it's not a nice
feature and will likely bite more often than just dpkg.

All the best and thanks again

Allan


On 01/05/13 16:18, Herbert Poetzl wrote:
> On Wed, May 01, 2013 at 12:13:51PM +0200, Allan Latham wrote:
> 
>> Hi all
> 
>> I am planning on using vservers for an upcoming project and
>> I have been doing extensive checking that I can achieve my
>> objectives.
> 
>> So far so good - I am very impressed.
> 
>> I have come across something strange (= did not do what I
>> expected).
> 
>> First the scenario:
> 
>> A disc contains an ext3 filesystem and is mounted rw on /X 
>> The top level is:
> 
>> /X/DIR HASH HASHES RO
> 
>> The contents of DIR has been hashified and the hashes are in
>> HASHES.
> 
>> /X/DIR is then bind mounted ro on /X/RO
> 
>> This works 100% as I expect - I can mount /X/RO during the
>> vserver initialize script and I have an ro version of the data
>> for the vserver to use. Good.
> 
>> To give each vserver a rw version of /X/DIR without disturbing
>> all the other vservers I use aufs.
> 
>> The ro branch of aufs is /X/RO and the rw branch is outside the
>> vserver client tree. (i.e. the vserver has no direct access to
>> it).
> 
>> The aufs mount is done in the initialize script at the same
>> point where in the previous test I bind mounted /X/RO.
> 
>> This work 99% as I expect. The vserver can use the filesystem
>> in rw mode and it is independent of other vservers.
> 
>> The one exception is if I try to hardlink a file which has been
>> hashified. I expected the kernel to break the hardlink to the
>> hash and give me a fresh copy of the file and then hardlink to
>> that. Instead I get EPERM.
> 
>> If I touch the file it does in fact break the hardlink to the
>> hash and I can then hardlink the file I have just touched.
> 
>> This is all well and good if I am doing this in a script in
>> the vserver but some programs (Debian's dpkg for example) use
>> hardlinking in their logic.
> 
>> Is this a bug? 
>> Is it an enhancement? 
> 
> Link breaking happens in various places in the kernel
> and it might be simply that this case was never tested
> an so just forgotten. 
> 
> I wouldn't consider it a bug, more a missing feature
> and if we figure out where it happens, it will be quite
> simple to add.
> 
>> Is it because of aufs?
> 
> maybe, but that should be rather simple to test, no?
> i.e. just use ext3 and let us know if the same thing
> happens there ...
> 
>> Is there any reason why the kernel cannot treat the link system
>> call the same way as changing the file metadata by touching the
>> file?
> 
> The VFS part of the Linux Kernel is quite complicated.
> 
>> This is not a major obstacle to further development of my
>> project but it would be nice if I had a solution. 
> 
>> Maybe there is a solution using lvm and snapshots but for 
>> all I know I will hit the same problem there.
> 
> I don't see how LVM and/or snapshots would be related
> at all ....
> 
> best,
> Herbert
> 
>> I have many other tests to do and I can revisit giving guests
>> rw versions of a common filesystem later.
> 
>> Thanks to all involved in vservers.
> 
>> Allan
>