On Sun, 22 Feb 2015 16:50:00 +0100 Oliver Welter <mail@oliwel.de> wrote: > Am 22.02.2015 um 15:50 schrieb Oliver Welter: > > Hi Bertl, > > > > Am 02.02.2015 um 11:11 schrieb Herbert Poetzl: > >> On Mon, Feb 02, 2015 at 08:10:15AM +0100, Oliver Welter wrote: > >>> Am 02.02.2015 um 01:58 schrieb Herbert Poetzl: > >>>> On Sun, Feb 01, 2015 at 03:28:22PM +0100, Oliver Welter wrote: > >> > >>>>> Trying the same without starting the vserver, using a simple > >>>>> "chroot" works, so it looks like some limits are not working > >>>>> correctly. > >> > >>>> Please try without Linux-VServer isolation but with the same > >>>> namespace setup, I don't think that you are hitting a limit > >>>> here, it looks more like a namespace problem of overlayfs. > >> > >>> Can you be a bit more verbose - I never git in touch with the > >>> individual components ;) > >> > >> Either use vnamespace or the unshare tool to create a namespace > >> setup identical to your guest and check if the issue remains. > > > > I am unable to reproduce it with vnamespace or unshare, I used attached > > test script and ran it as: > > > > vnamespace -n /mnt/test.sh > > vcontext --create --xid 42 /mnt/test.sh > > unshare -i -m -u test.sh > > > > with the correct result (file breakme is replaced by a directory on the > > common mount). > > > > When you do the same inside a vserver, it does not work (use overlay to > > assemble rootfs of vserver, start vserver, try to change entity which > > exists on the lower but not on the upper fs). > > > I think I got the root cause - overlayfs uses character files to mark > whiteouts and the vserver env seems to refuse access to this whiteout > file. Adding SYS_ADMIN "makes it work", so I guess the right way would > be to skip the access check for overlayfs whiteouts. thanks for debugging it to the point of identifying CAP_SYS_ADMIN, which was the pertinent piece of information, but the character devices were a red herring [1] (partially because of my confusion in how and when they are used in overlayfs). vattribute --set --xid ${VS_XID} --ccap ^3 OR echo ^3 >>/etc/vservers/${VS_NAME}/ccapabilities (or "fs_trusted", instead of "^3", once daniel merges this [2].) DISCLAIMER: this is just a workaround due to linux-vserver *not* supporting the use of overlayfs from within a vserver. another pertinent piece of information that i identified from my testing is that the error happens to directories, but not regular files. so reading through the overlayfs source code [3] i noticed that in ovl_create_over_whiteout() a directory is treated differently than everything else by calling ovl_set_opaque() on it which (eventually) sets an xattr of "trusted.overlay.opaque" on the directory. of course after reading through the source code i realize this is all documented [4] (but it's more memorable this way ;). before setting the xattr, the kernel checks for the necessary capabilities, which is normally just "capable(CAP_SYS_ADMIN)", but patched in linux-vserver kernels as "vx_capable(CAP_SYS_ADMIN, VXC_FS_TRUSTED)". this isn't a problem in the non-vserver overlayfs use-case because overlayfs raises CAP_SYS_ADMIN on the potentially unprivleged process when it's creating an opaque directory (ie a directory in the upperdir that overrides a file in the lower dir). with a vserver-patched kernel, a vserver needs both CAP_SYS_ADMIN and VXC_FS_TRUSTED to create the "trusted.overlay.opaque" xattr and as overlayfs only raises CAP_SYS_ADMIN, and not VXC_FS_TRUSTED, the operation fails. vx_capable(CAP_SYS_ADMIN, VXC_FS_TRUSTED) = 1. capable(CAP_SYS_ADMIN) OR 2a. cap_raise(current_cap(), CAP_SYS_ADMIN) (which is always true because of overlayfs' cap_raise(CAP_SYS_ADMIN)) && 2b. vx_ccaps(VXC_FS_TRUSTED)) to have overlayfs "just work" in a vserver, we either need to create a vx_cap_raise() for setting capabilities in the current vserver context, just as we have vx_cap_raised() which tests capabilities in the current vserver context, which would cause (1) to return true, or we need a vx_ccap_raise() for raising context capabilities so (2b) will return true. the trusted.* xattr namespace appears to only be used within the kernel by the overlayfs and lustre filesystems, so i believe it's relatively safe to give a vserver FS_TRUSTED (especially as CAP_SYS_ADMIN would also be needed), though i don't know what in user-land uses the trusted.* xattr namespace, specifically from the host, and therefor would next expect a guest to be able to set/clear/modify those attributes. [1] https://en.wiktionary.org/wiki/red_herring, second definition [2] https://github.com/linux-vserver/util-vserver/pull/17 [3] https://git.kernel.org/cgit/linux/kernel/git/mszeredi/vfs.git/commit/?h=overlayfs.v24&id=07d3c379edef433bba39ac14abdbbe1a1f6bb47a [4] https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt corey -- undefined@pobox.com > Oli > > > > -- > Protect your environment - close windows and adopt a penguin!