Subject: Re: [vserver] sharing /vserver
From: Oliver Welter <mail@oliwel.de>
Date: Thu, 11 Sep 2008 16:53:16 +0200

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Ed,

what I did a long time for similar reasons might help.
I created a small script in prepre-start.d that mounted a tmpfs on /dev
and created the necessary nodes to init the system. Did fine :)

Oliver

Edward Capriolo schrieb:
> I have found an issue.
> 
> It seems that glusterfs and or fuse is forcing a nodev option on my mount point.
> 
> cat /etc/mtab
> glusterfs /opt/jtg/mnt/shared fuse
> rw,nosuid,nodev,allow_other,default_permissions,max_read=1048576 0 0
> 
> So /dev can not be on gluster.
> I have a way to get 99% of the system files onto gluster. This way
> works but i feel like I missed something obvious.
> 
> [root@jtg111 ~]# cat /etc/vservers/test01/fstab
> none    /proc           proc    defaults                0 0
> none    /tmp            tmpfs   size=16m,mode=1777      0 0
> none    /dev/pts        devpts  gid=5,mode=620          0 0
> /opt/jtg/mnt/shared/vservers/glustertest/       /data   none    bind
> /opt/jtg/mnt/shared/vservers/glustertest/bin    /bin    node    bind
> /opt/jtg/mnt/shared/vservers/glustertest/usr    /usr    node    bind
> /opt/jtg/mnt/shared/vservers/glustertest/var    /var    node    bind
> ....
> This way works. It has two obvious flaws. 1 if someone creates a
> directory off of '/' like '/opt', it will be on the local disk, not
> the gluster disk. It is a pain to manage one entry per file system off
> the root.
> 
> So the questions.
> 1) Can I get gluster to store and serve a device file. ...If so is
> that really my vserver problem or am i hunting in the wrong direction?
> 2) Is there a way to configure fstab to mount / with the exception of
> a directory like /dev, that would still have to be local?
> 
> 
> On Wed, Sep 10, 2008 at 4:07 PM, Edward Capriolo <edlinuxguru@gmail.com> wrote:
>> Ok! the day has finally come that I have two nodes for testing! My
>> Goal: share an active/passive guest across two nodes with glusterfs.
>> So here is what I have so far.....My first attempt complained about
>> flock so I tried the posix-locks feature.
>>
>> ---jtg-server.conf--
>> volume jtg-ds
>>        type storage/posix
>>        option directory /opt/jtg/shared/jtg-ds
>> end-volume
>>
>> volume jtg-ds-locks
>>        type features/posix-locks
>> #       option mandatory on
>>        subvolumes jtg-ds
>> end-volume
>>
>> volume jtg-ns
>>        type storage/posix
>>        option directory /opt/jtg/shared/jtg-ns
>> end-volume
>>
>> volume jtg-ns-locks
>>        type features/posix-locks
>>        subvolumes jtg-ns
>> end-volume
>>
>> volume server
>>        type protocol/server
>>        option transport-type tcp/server
>>        option auth.ip.jtg-ds-locks.allow X.X.X.*
>>        option auth.ip.jtg-ns-locks.allow X.X.X.*
>>        subvolumes jtg-ds jtg-ds-locks jtg-ns jtg-ns-locks
>> end-volume
>> --jtg-server.vol
>>
>> [root@jtg111 conf]# more jtg-client.vol
>> volume remote1
>>        type protocol/client
>>        option transport-type tcp/client
>>        option remote-host jtg111.XXXX
>>        option remote-subvolume jtg-ds-locks
>> end-volume
>>
>> volume remote2
>>        type protocol/client
>>        option transport-type tcp/client
>>        option remote-host jtg113.XXX
>>        option remote-subvolume jtg-ds-locks
>> end-volume
>>
>> volume mirror0
>>        type cluster/afr
>>        subvolumes remote1 remote2
>>        option self-heal on
>> end-volume
>>
>> On both nodes:
>> glusterfsd -f /opt/jtg/conf/jtg-server.vol
>> glusterfs -L debug -f /opt/jtg/conf/jtg-client.vol /opt/jtg/mnt/shared/
>>
>> At this point i have an two node system. Read/write from either side.
>> All is well. Now my goal is to install a vserver onto the gluster
>> volume.
>> mkdir /opt/jtg/mnt/shared/etc/
>> mkdir /opt/jtg/mnt/shared/vservers/
>>
>> ---attempt 1----
>> vserver glustertest build -m template --hostname
>> glustertest.XXXXXX.com --interface eth0:10.0.0.2/24 --rootdir
>> /opt/jtg/mnt/shared/vservers/ --confdir
>> /opt/jtg/mnt/shared/etc/glustertest -- -d centos5 -t
>> /vservers/.templates/test01.tar
>>
>> vcontext: open("/dev/null"): Permission denied
>>
>> An error occured while executing the vserver startup sequence; when
>> there are no other messages, it is very likely that the init-script
>> (bash -c
>>        exec  > /dev/null
>>        exec 2> /dev/null
>>        trap 'kill -s 9 -- -1; exit 0' INT
>>        sleep 15
>>        kill -s 15 -- -1
>>        sleep 1
>>        kill -s 9 -- -1) failed.
>>
>> Common causes are:
>> * /etc/rc.d/rc on Fedora Core 1 and RH9 fails always; the 'apt-rpm' build
>>  method knows how to deal with this, but on existing installations,
>>  appending 'true' to this file will help.
>>
>>
>> Failed to start vserver 'glustertest'
>> 'vserver ... suexec' is supported for running vservers only; aborting...
>> 'vserver ... suexec' is supported for running vservers only; aborting...
>> 'vserver ... suexec' is supported for running vservers only; aborting...
>> 'vserver ... suexec' is supported for running vservers only; aborting...
>> 'vserver ... suexec' is supported for running vservers only; aborting...
>> 'vserver ... suexec' is supported for running vservers only; aborting...
>> 'vserver ... suexec' is supported for running vservers only; aborting...
>>
>> then i put a link in /etc/vservers
>> ---end----
>>
>>
>> ---attempt 2---
>>  vserver glustertest build -m template --hostname
>> glustertest.XXXXX.com --interface eth0:10.0.0.2/24 --rootdir
>> /opt/jtg/mnt/shared/vservers/  -- -d centos5 -t
>> /vservers/.templates/test01.tar
>>
>>
>> vshelper.init: can not determine xid of vserver 'glustertest';
>> returned value was ''
>>
>> This usually means that you're using an init-less init-style, but the
>> guest isn't configured to start any service. Try enabling a service,
>> changing the init-style, or making the contexts persistent.
>>
>> An error occured after executing the vserver startup sequence. This
>> means that some processes may exist in the created context and the
>> manual execution of
>>
>>  /usr/sbin/vserver '/etc/vservers/glustertest' stop
>>
>> is recommended to fix this.
>> ------------------------
>> So. The question. Should this work? Does this go back to the extended
>> attributes we spoke of before. My goal is not to do any unification
>> just an install. If anyone has ideas let me know!
>>

- --
Protect your environment -  close windows and adopt a penguin!
PGP-Key: 3B2C 8095 A7DF 8BB5 2CFF  8168 CAB7 B0DD 3985 1721
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIyTDcyrew3TmFFyERAtc0AJ429Uq/3thjvLe5hRnFx0CllOB3GQCdFGgQ
KgbBI3642GfqoSJcRcXPYpg=
=0k0t
-----END PGP SIGNATURE-----