Subject: Re: [vserver] Issue with OverlayFS in 3.18
From: Herbert Poetzl <herbert@13thfloor.at>
Date: Sat, 28 Feb 2015 14:46:51 +0100

On Sat, Feb 28, 2015 at 01:20:47PM +0100, Oliver Welter wrote:
> Am 27.02.2015 um 10:28 schrieb Herbert Poetzl:
>> On Thu, Feb 26, 2015 at 03:42:51PM +0100, Oliver Welter wrote:
>>> Am 22.02.2015 um 21:42 schrieb Corey Wright:
>>>> On Sun, 22 Feb 2015 21:10:45 +0100
>>>> Oliver Welter <mail@oliwel.de> wrote:

>>>>> Am 22.02.2015 um 20:05 schrieb Corey Wright:
>>>>>> On Sun, 22 Feb 2015 17:10:20 +0100
>>>>>> Oliver Welter <mail@oliwel.de> wrote:

>>>>>>> Am 22.02.2015 um 17:01 schrieb Herbert Poetzl:

>>>>>>> --snipped--

>>>>>>>> Documentation says it should be a character device with
>>>>>>>> major/minor 0/0, so allowing that device should do the
>>>>>>>> job, no further modification required.

>>>>>>> Yes that is correct - is there a way to do so with the current kernel
>>>>>>> patches or do we(you) need to modify the patch.

>>>>>> i would guess:

>>>>>> mknod ${WHITEOUT} c 0 0

>>>>>> which seems to be confirmed by the script attached to the
>>>>>> end of this email [1], though i'm not sure where this
>>>>>> whiteout device is suppose to reside (same directory as
>>>>>> mount point for overlay?).

>>>>>> you should be able to mknod the whiteout
>>>>>> device from the host in the vserver (eg "mknod
>>>>>> /vservers/${VNAME}/mnt/whiteout c 0 0") or allow the guest
>>>>>> to create it (using cgroup device whitelist controller or
>>>>>> vdevmap).

>>>>> There is no single whiteout device, overlayfs creates a 0/0
>>>>> item for each deleted item ( using the name of the deleted
>>>>> file )

>>>> yeah, it would have helped if i would have not stopped short
>>>> with the first search hit, but read the kernel documentation
>>>> [1]. :)

>>>> as a simple test, you could try giving the vserver MKNOD
>>>> ("echo MKNOD >> /etc/vservers/${VSERVER}/bcapabilities";
>>>> as compared to the all-powerful CAP_SYS_ADMIN) and if that
>>>> works, then whitelist the char 0:0 device with cgroup device
>>>> whitelisting (eg this paste [2], as i previously referenced on
>>>> irc [3]).

>>>> [1] Documentation/filesystems/overlayfs.txt
>>>> [2] http://pastebin.com/SqCMmkpA
>>>> [3] http://irc.13thfloor.at/LOG/2014-11/LOG_2014-11-28.txt

>>> Sorry for the *bump* but can anybody bring up another idea
>>> / fix ?

>>> Using mknod and/or cgroup whitelist doesnt work (or I did
>>> it wrong)

>> It is important to check if your guest is able to create
>> this specific whiteout node with mknod, if not, then it
>> is most likely a problem of the guest configuration and
>> not a kernel problem.

>> So, first, make sure, that you can create the same device
>> node the filesystem creates for whiteout as root inside
>> the guest.

> echo MKNOD > bcapabilities
> # vserver testing start (WITHOUT overlayfs)
> # vserver testing enter
> $ mknod /home/rnd c 1 8
> $ mknod /home/mark c 0 0

> both work.

> When I use the overlayfs as rootfs, it is not possible to
> create a 0/0 node on top of the overlayfs but that should be
> pretty ok. 

Yes, I would expect that any sane design would prevent
creating the whiteout node manually on the filesystem.

But what about overlayfs itself?

Does it work except for the whiteout?

When you "delete" a file on the filesystem, what
happens and what error codes (if any at all) do
you get?

> Might it be a kind of "routing issue" for the vfs call?

Linux-VServer does not change the vfs behaviour, except
for adding the link breaking and barrier checks, so
the "delete" should end up where it belongs, i.e. in
the overlayfs code.

>> Then make sure that the filesystem is mounted inside the
>> guest namespaces, so that the very same conditions are
>> true for the filesystem.

> I mount the overlayfs from the "prepre-start" script, so I
> assume that this is true. I also tried to create the overlay
> from the root before starteing the guest, result is the same.

Try to mount it from the guest namespace after the guest
was started (on a test mountpoint, with vnamespace/vmount)
and see if that changes anything.

Also verify that overlayfs works fine on the host when
using a Linux-VServer patched kernel.

>> If it still fails, try to get the error message (code)
>> which is returned from the kernel for further analysis.

> Neither dmesg nor syslog report anything, might I add some
> debug flags to the kernel?

We probably have to add some printk to the kernel to
get more information if nothing is logged. Note that
debugging might need to be enabled in the kernel config
and you also want to raise the kernel loglevel to 7.

All the best,
Herbert

> Oli


> -- 
> Protect your environment -  close windows and adopt a penguin!