On Sun, 1 Mar 2015 02:02:21 -0600 Corey Wright <undefined@pobox.com> wrote: > On Sun, 22 Feb 2015 16:50:00 +0100 > Oliver Welter <mail@oliwel.de> wrote: > > > Am 22.02.2015 um 15:50 schrieb Oliver Welter: > > > Hi Bertl, > > > > > > Am 02.02.2015 um 11:11 schrieb Herbert Poetzl: > > >> On Mon, Feb 02, 2015 at 08:10:15AM +0100, Oliver Welter wrote: > > >>> Am 02.02.2015 um 01:58 schrieb Herbert Poetzl: > > >>>> On Sun, Feb 01, 2015 at 03:28:22PM +0100, Oliver Welter wrote: > > >> > > >>>>> Trying the same without starting the vserver, using a simple > > >>>>> "chroot" works, so it looks like some limits are not working > > >>>>> correctly. > > >> > > >>>> Please try without Linux-VServer isolation but with the same > > >>>> namespace setup, I don't think that you are hitting a limit > > >>>> here, it looks more like a namespace problem of overlayfs. > > >> > > >>> Can you be a bit more verbose - I never git in touch with the > > >>> individual components ;) > > >> > > >> Either use vnamespace or the unshare tool to create a namespace > > >> setup identical to your guest and check if the issue remains. > > > > > > I am unable to reproduce it with vnamespace or unshare, I used attached > > > test script and ran it as: > > > > > > vnamespace -n /mnt/test.sh > > > vcontext --create --xid 42 /mnt/test.sh > > > unshare -i -m -u test.sh > > > > > > with the correct result (file breakme is replaced by a directory on the > > > common mount). > > > > > > When you do the same inside a vserver, it does not work (use overlay to > > > assemble rootfs of vserver, start vserver, try to change entity which > > > exists on the lower but not on the upper fs). > > > > > I think I got the root cause - overlayfs uses character files to mark > > whiteouts and the vserver env seems to refuse access to this whiteout > > file. Adding SYS_ADMIN "makes it work", so I guess the right way would > > be to skip the access check for overlayfs whiteouts. > > thanks for debugging it to the point of identifying CAP_SYS_ADMIN, which > was the pertinent piece of information, but the character devices were a > red herring [1] (partially because of my confusion in how and when they are > used in overlayfs). > > vattribute --set --xid ${VS_XID} --ccap ^3 > OR > echo ^3 >>/etc/vservers/${VS_NAME}/ccapabilities > > (or "fs_trusted", instead of "^3", once daniel merges this [2].) > > DISCLAIMER: this is just a workaround due to linux-vserver *not* > supporting the use of overlayfs from within a vserver. > > another pertinent piece of information that i identified from my testing > is that the error happens to directories, but not regular files. so > reading through the overlayfs source code [3] i noticed that in > ovl_create_over_whiteout() a directory is treated differently than > everything else by calling ovl_set_opaque() on it which (eventually) > sets an xattr of "trusted.overlay.opaque" on the directory. of course > after reading through the source code i realize this is all documented > [4] (but it's more memorable this way ;). > > before setting the xattr, the kernel checks for the necessary > capabilities, which is normally just "capable(CAP_SYS_ADMIN)", but > patched in linux-vserver kernels as "vx_capable(CAP_SYS_ADMIN, > VXC_FS_TRUSTED)". this isn't a problem in the non-vserver overlayfs > use-case because overlayfs raises CAP_SYS_ADMIN on the potentially > unprivleged process when it's creating an opaque directory (ie a > directory in the upperdir that overrides a file in the lower dir). with > a vserver-patched kernel, a vserver needs both CAP_SYS_ADMIN and > VXC_FS_TRUSTED to create the "trusted.overlay.opaque" xattr and as > overlayfs only raises CAP_SYS_ADMIN, and not VXC_FS_TRUSTED, the > operation fails. > > vx_capable(CAP_SYS_ADMIN, VXC_FS_TRUSTED) = > 1. capable(CAP_SYS_ADMIN) > OR > 2a. cap_raise(current_cap(), CAP_SYS_ADMIN) (which is always true > because of overlayfs' cap_raise(CAP_SYS_ADMIN)) > && > 2b. vx_ccaps(VXC_FS_TRUSTED)) > > to have overlayfs "just work" in a vserver, we either need to create a > vx_cap_raise() for setting capabilities in the current vserver context, > just as we have vx_cap_raised() which tests capabilities in the current > vserver context, which would cause (1) to return true which is a stupid idea because it would give the entire vserver SYS_ADMIN, if even for a very short period of time. > , or we need a > vx_ccap_raise() for raising context capabilities so (2b) will return > true. which would also be problematic as it would give the entire vserver the FS_TRUSTED context capability for a (short) period of time, though for processes within the vserver without SYS_ADMIN it wouldn't change anything. yes, giving the vserver FS_TRUSTED all the time is definitely a longer period of time, but at least it's an intentional administrative action. so ignore my ideas. > the trusted.* xattr namespace appears to only be used within the kernel > by the overlayfs and lustre filesystems, so i believe it's relatively > safe to give a vserver FS_TRUSTED (especially as CAP_SYS_ADMIN would > also be needed), though i don't know what in user-land uses the > trusted.* xattr namespace, specifically from the host, and therefor > would next expect a guest to be able to set/clear/modify those > attributes. i give it a medium-risk, low-severity assessment: the risk of someone exploiting FS_TRUSTED is medium because they would also need the CAP_SYS_ADMIN process capability, but the severity is low because once they were able to make "trusted.*" xattrs, it doesn't affect anything as nothing appears to use them (based on a survey of the internet), especially anything that would violate the host-guest separation (unless you tell me you are using trusted.* xattrs get/set by the host on guest files to perform filesystem-based intrusion detection ;). corey -- undefined@pobox.com > [1] https://en.wiktionary.org/wiki/red_herring, second definition > [2] https://github.com/linux-vserver/util-vserver/pull/17 > [3] > https://git.kernel.org/cgit/linux/kernel/git/mszeredi/vfs.git/commit/?h=overlayfs.v24&id=07d3c379edef433bba39ac14abdbbe1a1f6bb47a > [4] https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt > > corey > -- > undefined@pobox.com > > > Oli > > > > > > > > -- > > Protect your environment - close windows and adopt a penguin!