Subject: Re: [vserver] sharing /vserver
From: "Edward Capriolo" <edlinuxguru@gmail.com>
Date: Thu, 11 Sep 2008 10:15:04 -0400

I have found an issue.

It seems that glusterfs and or fuse is forcing a nodev option on my mount point.

cat /etc/mtab
glusterfs /opt/jtg/mnt/shared fuse
rw,nosuid,nodev,allow_other,default_permissions,max_read=1048576 0 0

So /dev can not be on gluster.
I have a way to get 99% of the system files onto gluster. This way
works but i feel like I missed something obvious.

[root@jtg111 ~]# cat /etc/vservers/test01/fstab
none    /proc           proc    defaults                0 0
none    /tmp            tmpfs   size=16m,mode=1777      0 0
none    /dev/pts        devpts  gid=5,mode=620          0 0
/opt/jtg/mnt/shared/vservers/glustertest/       /data   none    bind
/opt/jtg/mnt/shared/vservers/glustertest/bin    /bin    node    bind
/opt/jtg/mnt/shared/vservers/glustertest/usr    /usr    node    bind
/opt/jtg/mnt/shared/vservers/glustertest/var    /var    node    bind
....
This way works. It has two obvious flaws. 1 if someone creates a
directory off of '/' like '/opt', it will be on the local disk, not
the gluster disk. It is a pain to manage one entry per file system off
the root.

So the questions.
1) Can I get gluster to store and serve a device file. ...If so is
that really my vserver problem or am i hunting in the wrong direction?
2) Is there a way to configure fstab to mount / with the exception of
a directory like /dev, that would still have to be local?


On Wed, Sep 10, 2008 at 4:07 PM, Edward Capriolo <edlinuxguru@gmail.com> wrote:
> Ok! the day has finally come that I have two nodes for testing! My
> Goal: share an active/passive guest across two nodes with glusterfs.
> So here is what I have so far.....My first attempt complained about
> flock so I tried the posix-locks feature.
>
> ---jtg-server.conf--
> volume jtg-ds
>        type storage/posix
>        option directory /opt/jtg/shared/jtg-ds
> end-volume
>
> volume jtg-ds-locks
>        type features/posix-locks
> #       option mandatory on
>        subvolumes jtg-ds
> end-volume
>
> volume jtg-ns
>        type storage/posix
>        option directory /opt/jtg/shared/jtg-ns
> end-volume
>
> volume jtg-ns-locks
>        type features/posix-locks
>        subvolumes jtg-ns
> end-volume
>
> volume server
>        type protocol/server
>        option transport-type tcp/server
>        option auth.ip.jtg-ds-locks.allow X.X.X.*
>        option auth.ip.jtg-ns-locks.allow X.X.X.*
>        subvolumes jtg-ds jtg-ds-locks jtg-ns jtg-ns-locks
> end-volume
> --jtg-server.vol
>
> [root@jtg111 conf]# more jtg-client.vol
> volume remote1
>        type protocol/client
>        option transport-type tcp/client
>        option remote-host jtg111.XXXX
>        option remote-subvolume jtg-ds-locks
> end-volume
>
> volume remote2
>        type protocol/client
>        option transport-type tcp/client
>        option remote-host jtg113.XXX
>        option remote-subvolume jtg-ds-locks
> end-volume
>
> volume mirror0
>        type cluster/afr
>        subvolumes remote1 remote2
>        option self-heal on
> end-volume
>
> On both nodes:
> glusterfsd -f /opt/jtg/conf/jtg-server.vol
> glusterfs -L debug -f /opt/jtg/conf/jtg-client.vol /opt/jtg/mnt/shared/
>
> At this point i have an two node system. Read/write from either side.
> All is well. Now my goal is to install a vserver onto the gluster
> volume.
> mkdir /opt/jtg/mnt/shared/etc/
> mkdir /opt/jtg/mnt/shared/vservers/
>
> ---attempt 1----
> vserver glustertest build -m template --hostname
> glustertest.XXXXXX.com --interface eth0:10.0.0.2/24 --rootdir
> /opt/jtg/mnt/shared/vservers/ --confdir
> /opt/jtg/mnt/shared/etc/glustertest -- -d centos5 -t
> /vservers/.templates/test01.tar
>
> vcontext: open("/dev/null"): Permission denied
>
> An error occured while executing the vserver startup sequence; when
> there are no other messages, it is very likely that the init-script
> (bash -c
>        exec  > /dev/null
>        exec 2> /dev/null
>        trap 'kill -s 9 -- -1; exit 0' INT
>        sleep 15
>        kill -s 15 -- -1
>        sleep 1
>        kill -s 9 -- -1) failed.
>
> Common causes are:
> * /etc/rc.d/rc on Fedora Core 1 and RH9 fails always; the 'apt-rpm' build
>  method knows how to deal with this, but on existing installations,
>  appending 'true' to this file will help.
>
>
> Failed to start vserver 'glustertest'
> 'vserver ... suexec' is supported for running vservers only; aborting...
> 'vserver ... suexec' is supported for running vservers only; aborting...
> 'vserver ... suexec' is supported for running vservers only; aborting...
> 'vserver ... suexec' is supported for running vservers only; aborting...
> 'vserver ... suexec' is supported for running vservers only; aborting...
> 'vserver ... suexec' is supported for running vservers only; aborting...
> 'vserver ... suexec' is supported for running vservers only; aborting...
>
> then i put a link in /etc/vservers
> ---end----
>
>
> ---attempt 2---
>  vserver glustertest build -m template --hostname
> glustertest.XXXXX.com --interface eth0:10.0.0.2/24 --rootdir
> /opt/jtg/mnt/shared/vservers/  -- -d centos5 -t
> /vservers/.templates/test01.tar
>
>
> vshelper.init: can not determine xid of vserver 'glustertest';
> returned value was ''
>
> This usually means that you're using an init-less init-style, but the
> guest isn't configured to start any service. Try enabling a service,
> changing the init-style, or making the contexts persistent.
>
> An error occured after executing the vserver startup sequence. This
> means that some processes may exist in the created context and the
> manual execution of
>
>  /usr/sbin/vserver '/etc/vservers/glustertest' stop
>
> is recommended to fix this.
> ------------------------
> So. The question. Should this work? Does this go back to the extended
> attributes we spoke of before. My goal is not to do any unification
> just an install. If anyone has ideas let me know!
>