Subject: Re: [vserver] Issue with OverlayFS in 3.18
From: Corey Wright <undefined@pobox.com>
Date: Sun, 1 Mar 2015 13:57:09 -0600

On Sun, 1 Mar 2015 02:02:21 -0600
Corey Wright <undefined@pobox.com> wrote:

> On Sun, 22 Feb 2015 16:50:00 +0100
> Oliver Welter <mail@oliwel.de> wrote:
> 
> > Am 22.02.2015 um 15:50 schrieb Oliver Welter:
> > > Hi Bertl,
> > >
> > > Am 02.02.2015 um 11:11 schrieb Herbert Poetzl:
> > >> On Mon, Feb 02, 2015 at 08:10:15AM +0100, Oliver Welter wrote:
> > >>> Am 02.02.2015 um 01:58 schrieb Herbert Poetzl:
> > >>>> On Sun, Feb 01, 2015 at 03:28:22PM +0100, Oliver Welter wrote:
> > >>
> > >>>>> Trying the same without starting the vserver, using a simple
> > >>>>> "chroot" works, so it looks like some limits are not working
> > >>>>> correctly.
> > >>
> > >>>> Please try without Linux-VServer isolation but with the same
> > >>>> namespace setup, I don't think that you are hitting a limit
> > >>>> here, it looks more like a namespace problem of overlayfs.
> > >>
> > >>> Can you be a bit more verbose - I never git in touch with the
> > >>> individual components ;)
> > >>
> > >> Either use vnamespace or the unshare tool to create a namespace
> > >> setup identical to your guest and check if the issue remains.
> > >
> > > I am unable to reproduce it with vnamespace or unshare, I used attached
> > > test script and ran it as:
> > >
> > > vnamespace -n /mnt/test.sh
> > > vcontext --create --xid 42 /mnt/test.sh
> > > unshare  -i -m -u test.sh
> > >
> > > with the correct result (file breakme is replaced by a directory on the
> > > common mount).
> > >
> > > When you do the same inside a vserver, it does not work (use overlay to
> > > assemble rootfs of vserver, start vserver, try to change entity which
> > > exists on the lower but not on the upper fs).
> > >
> > I think I got the root cause - overlayfs uses character files to mark 
> > whiteouts and the vserver env seems to refuse access to this whiteout 
> > file. Adding SYS_ADMIN "makes it work", so I guess the right way would 
> > be to skip the access check for overlayfs whiteouts.
> 
> thanks for debugging it to the point of identifying CAP_SYS_ADMIN, which
> was the pertinent piece of information, but the character devices were a
> red herring [1] (partially because of my confusion in how and when they are
> used in overlayfs).
> 
> vattribute --set --xid ${VS_XID} --ccap ^3
> OR
> echo ^3 >>/etc/vservers/${VS_NAME}/ccapabilities 
> 
> (or "fs_trusted", instead of "^3", once daniel merges this [2].)
> 
> DISCLAIMER: this is just a workaround due to linux-vserver *not*
> supporting the use of overlayfs from within a vserver.
> 
> another pertinent piece of information that i identified from my testing
> is that the error happens to directories, but not regular files.  so
> reading through the overlayfs source code [3] i noticed that in
> ovl_create_over_whiteout() a directory is treated differently than
> everything else by calling ovl_set_opaque() on it which (eventually)
> sets an xattr of "trusted.overlay.opaque" on the directory.  of course
> after reading through the source code i realize this is all documented
> [4] (but it's more memorable this way ;).
> 
> before setting the xattr, the kernel checks for the necessary
> capabilities, which is normally just "capable(CAP_SYS_ADMIN)", but
> patched in linux-vserver kernels as "vx_capable(CAP_SYS_ADMIN,
> VXC_FS_TRUSTED)".  this isn't a problem in the non-vserver overlayfs
> use-case because overlayfs raises CAP_SYS_ADMIN on the potentially
> unprivleged process when it's creating an opaque directory (ie a
> directory in the upperdir that overrides a file in the lower dir).  with
> a vserver-patched kernel, a vserver needs both CAP_SYS_ADMIN and
> VXC_FS_TRUSTED to create the "trusted.overlay.opaque" xattr and as
> overlayfs only raises CAP_SYS_ADMIN, and not VXC_FS_TRUSTED, the
> operation fails.
> 
> vx_capable(CAP_SYS_ADMIN, VXC_FS_TRUSTED) =
> 1. capable(CAP_SYS_ADMIN)
> OR
> 2a. cap_raise(current_cap(), CAP_SYS_ADMIN) (which is always true
>     because of overlayfs' cap_raise(CAP_SYS_ADMIN))
> &&
> 2b. vx_ccaps(VXC_FS_TRUSTED))
> 
> to have overlayfs "just work" in a vserver, we either need to create a
> vx_cap_raise() for setting capabilities in the current vserver context,
> just as we have vx_cap_raised() which tests capabilities in the current
> vserver context, which would cause (1) to return true

which is a stupid idea because it would give the entire vserver
SYS_ADMIN, if even for a very short period of time.

> , or we need a
> vx_ccap_raise() for raising context capabilities so (2b) will return
> true.

which would also be problematic as it would give the entire vserver the
FS_TRUSTED context capability for a (short) period of time, though for
processes within the vserver without SYS_ADMIN it wouldn't change
anything.  yes, giving the vserver FS_TRUSTED all the time is definitely
a longer period of time, but at least it's an intentional administrative
action.

so ignore my ideas.

> the trusted.* xattr namespace appears to only be used within the kernel
> by the overlayfs and lustre filesystems, so i believe it's relatively
> safe to give a vserver FS_TRUSTED (especially as CAP_SYS_ADMIN would
> also be needed), though i don't know what in user-land uses the
> trusted.* xattr namespace, specifically from the host, and therefor
> would next expect a guest to be able to set/clear/modify those
> attributes.

i give it a medium-risk, low-severity assessment: the risk of someone
exploiting FS_TRUSTED is medium because they would also need the
CAP_SYS_ADMIN process capability, but the severity is low because once
they were able to make "trusted.*" xattrs, it doesn't affect anything as
nothing appears to use them (based on a survey of the internet),
especially anything that would violate the host-guest separation (unless
you tell me you are using trusted.* xattrs get/set by the host on guest
files to perform filesystem-based intrusion detection ;).

corey
--
undefined@pobox.com

> [1] https://en.wiktionary.org/wiki/red_herring, second definition
> [2] https://github.com/linux-vserver/util-vserver/pull/17
> [3]
> https://git.kernel.org/cgit/linux/kernel/git/mszeredi/vfs.git/commit/?h=overlayfs.v24&id=07d3c379edef433bba39ac14abdbbe1a1f6bb47a
> [4] https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt
> 
> corey
> --
> undefined@pobox.com
> 
> > Oli
> > 
> > 
> > 
> > -- 
> > Protect your environment -  close windows and adopt a penguin!