I have found an issue. It seems that glusterfs and or fuse is forcing a nodev option on my mount point. cat /etc/mtab glusterfs /opt/jtg/mnt/shared fuse rw,nosuid,nodev,allow_other,default_permissions,max_read=1048576 0 0 So /dev can not be on gluster. I have a way to get 99% of the system files onto gluster. This way works but i feel like I missed something obvious. [root@jtg111 ~]# cat /etc/vservers/test01/fstab none /proc proc defaults 0 0 none /tmp tmpfs size=16m,mode=1777 0 0 none /dev/pts devpts gid=5,mode=620 0 0 /opt/jtg/mnt/shared/vservers/glustertest/ /data none bind /opt/jtg/mnt/shared/vservers/glustertest/bin /bin node bind /opt/jtg/mnt/shared/vservers/glustertest/usr /usr node bind /opt/jtg/mnt/shared/vservers/glustertest/var /var node bind .... This way works. It has two obvious flaws. 1 if someone creates a directory off of '/' like '/opt', it will be on the local disk, not the gluster disk. It is a pain to manage one entry per file system off the root. So the questions. 1) Can I get gluster to store and serve a device file. ...If so is that really my vserver problem or am i hunting in the wrong direction? 2) Is there a way to configure fstab to mount / with the exception of a directory like /dev, that would still have to be local? On Wed, Sep 10, 2008 at 4:07 PM, Edward Capriolo <edlinuxguru@gmail.com> wrote: > Ok! the day has finally come that I have two nodes for testing! My > Goal: share an active/passive guest across two nodes with glusterfs. > So here is what I have so far.....My first attempt complained about > flock so I tried the posix-locks feature. > > ---jtg-server.conf-- > volume jtg-ds > type storage/posix > option directory /opt/jtg/shared/jtg-ds > end-volume > > volume jtg-ds-locks > type features/posix-locks > # option mandatory on > subvolumes jtg-ds > end-volume > > volume jtg-ns > type storage/posix > option directory /opt/jtg/shared/jtg-ns > end-volume > > volume jtg-ns-locks > type features/posix-locks > subvolumes jtg-ns > end-volume > > volume server > type protocol/server > option transport-type tcp/server > option auth.ip.jtg-ds-locks.allow X.X.X.* > option auth.ip.jtg-ns-locks.allow X.X.X.* > subvolumes jtg-ds jtg-ds-locks jtg-ns jtg-ns-locks > end-volume > --jtg-server.vol > > [root@jtg111 conf]# more jtg-client.vol > volume remote1 > type protocol/client > option transport-type tcp/client > option remote-host jtg111.XXXX > option remote-subvolume jtg-ds-locks > end-volume > > volume remote2 > type protocol/client > option transport-type tcp/client > option remote-host jtg113.XXX > option remote-subvolume jtg-ds-locks > end-volume > > volume mirror0 > type cluster/afr > subvolumes remote1 remote2 > option self-heal on > end-volume > > On both nodes: > glusterfsd -f /opt/jtg/conf/jtg-server.vol > glusterfs -L debug -f /opt/jtg/conf/jtg-client.vol /opt/jtg/mnt/shared/ > > At this point i have an two node system. Read/write from either side. > All is well. Now my goal is to install a vserver onto the gluster > volume. > mkdir /opt/jtg/mnt/shared/etc/ > mkdir /opt/jtg/mnt/shared/vservers/ > > ---attempt 1---- > vserver glustertest build -m template --hostname > glustertest.XXXXXX.com --interface eth0:10.0.0.2/24 --rootdir > /opt/jtg/mnt/shared/vservers/ --confdir > /opt/jtg/mnt/shared/etc/glustertest -- -d centos5 -t > /vservers/.templates/test01.tar > > vcontext: open("/dev/null"): Permission denied > > An error occured while executing the vserver startup sequence; when > there are no other messages, it is very likely that the init-script > (bash -c > exec > /dev/null > exec 2> /dev/null > trap 'kill -s 9 -- -1; exit 0' INT > sleep 15 > kill -s 15 -- -1 > sleep 1 > kill -s 9 -- -1) failed. > > Common causes are: > * /etc/rc.d/rc on Fedora Core 1 and RH9 fails always; the 'apt-rpm' build > method knows how to deal with this, but on existing installations, > appending 'true' to this file will help. > > > Failed to start vserver 'glustertest' > 'vserver ... suexec' is supported for running vservers only; aborting... > 'vserver ... suexec' is supported for running vservers only; aborting... > 'vserver ... suexec' is supported for running vservers only; aborting... > 'vserver ... suexec' is supported for running vservers only; aborting... > 'vserver ... suexec' is supported for running vservers only; aborting... > 'vserver ... suexec' is supported for running vservers only; aborting... > 'vserver ... suexec' is supported for running vservers only; aborting... > > then i put a link in /etc/vservers > ---end---- > > > ---attempt 2--- > vserver glustertest build -m template --hostname > glustertest.XXXXX.com --interface eth0:10.0.0.2/24 --rootdir > /opt/jtg/mnt/shared/vservers/ -- -d centos5 -t > /vservers/.templates/test01.tar > > > vshelper.init: can not determine xid of vserver 'glustertest'; > returned value was '' > > This usually means that you're using an init-less init-style, but the > guest isn't configured to start any service. Try enabling a service, > changing the init-style, or making the contexts persistent. > > An error occured after executing the vserver startup sequence. This > means that some processes may exist in the created context and the > manual execution of > > /usr/sbin/vserver '/etc/vservers/glustertest' stop > > is recommended to fix this. > ------------------------ > So. The question. Should this work? Does this go back to the extended > attributes we spoke of before. My goal is not to do any unification > just an install. If anyone has ideas let me know! >