Subject: Design questions for large cluster deployment with many guests and shared iSCSI storage
From: "John A. Sullivan III" <jsullivan@opensourcedevel.com>
Date: Tue, 20 Jan 2009 08:40:18 -0500

Hello, all.  We are in the midst of designing infrastructure for a new
business launch and VServer is an integral part of the design.  In our
naive optimism, we are planning for upwards of ten thousand VServer
guests running on a cluster of VServer hosts.

We have been greatly aided in our hardware design by Pogo Linux
(http://www.pogolinux.com).  I'll shamelessly plug them as they have
been incredibly helpful and competent. They were one of the most
expensive bidders and have been well worth every extra penny.  They have
succeeded in affordably moving us to an iSCSI SAN for our initial
deployment - something we did not plan to do until at least a year from
now - thus some design issues have appeared earlier than expected.

The problem is the use of shared block device storage in a SAN as
opposed to a shared file system in a NAS.  The original NFS design was
for a file system something like this:

/data
    /client1
        /CorporateData
        /ExecutiveData
        /Users
            /Graham
            /John
            /Paul
    /client2
        /HR
        /Users
            /Peter
            /James

A John's guest would have mounted NFS1:/data/client1 to John:/data and
found his home directory at /data/users/John.  Graham's guest would have
mounted NFS1:/data/client1 to Graham:/data and found his home directory
at /data/users/Graham.

I don't believe I can do that safely with a block device.  It would be a
catastrophe of file system corruption as far as I know (and I don't know
very far!).

My thought was to not do the mounts in the guests.  Rather, we would do
the mount in the underlying VServer cluster, i.e., we mount the iSCSI
device containing all the clients' data at /data.  We then do a mount
bind from the underlying VServer host file system to the guests' file
systems: mount --bind /data/client1 /vservers/john/data; mount
--bind /data/client1 /vservers/graham/data.  Would this solve our
problem? Are there any caveats? Is there a better approach? Issues of
security, performance, data consistency? Thanks - John
-- 
John A. Sullivan III
Open Source Development Corporation
+1 207-985-7880
jsullivan@opensourcedevel.com

http://www.spiritualoutreach.com
Making Christianity intelligible to secular society