Subject: Re: [vserver] Poll: High (ish) availability - how are you doing it?
From: Gordan Bobic <gordan@bobich.net>
Date: Thu, 29 Jul 2010 01:08:18 +0100

Ed W wrote:
>  On 28/07/2010 15:16, Edward Capriolo wrote:
>> I also successfully ran a vserver on top of a Mirrored GlusterFS
>> volume with linux-ha failover.( This is essentially User space DRBD.)
> 
> Thanks to everyone who has replied - very interesting.
> 
> Glusterf is the one which most interests me (active-active clustering), 
> but it seems that the public forums are full of people with problems, 
> but then a handful of folks who are very happy.  Obviously you always 
> hear most about the outliers, but it does suggest that there are quite a 
> few subtleties with gluster?  Anyone got anything to share?

It depends on what you do with it and in what environment.

Pros:
- It stores files on the underlying, conventional file system (e.g. 
ext3, xfs, etc.), so data recovery is quite straightforward if things go 
wrong.
- It doesn't require fencing devices.

Cons:
- Without fencing devices, splitbrain is a common occurrence if packets 
start getting dropped.
- Slow (especially access latencies, even worse than conventional 
cluster file systems like GFS or OCFS2).
- There are a few obscure things that are still subtly broken (e.g. 
nvidia driver installer does something weird with sockets that works 
fine on a normal FS but produces huge and corrupted files on GlusterFS).
- Some performance translators break things in unpredictable ways.

> Seems like there is nothing really on the horizon that will do cross 
> datacenter clustering yet...  Sounds like XtreemFS might turn out to 
> work, but it seems a long away from being stable and well tested...

I wouldn't hold my breath. The problem is two-fold:
1) Performance
Latencies between data centers make achieving decent performance 
essentially impossible on a file system that guarantees consistency.

2) Fencing
You need reliable fencing between the nodes to ensure everything works 
as it should. This isn't a cluster/fs issue, it is a WAN networking 
issue. If your WAN is redundant and fault tolerant, it isn't really a 
problem.

One file system that hasn't been mentioned is SeznamFS. Think of it as 
GlusterFS (both SeznamFS and GlusterFS are fuse based, both store files 
on the underlying conventional file system) but applying file 
replication in the same way that MySQL replication works (binary 
logging, master-slave or ring master-master). I doesn't have any POSIX 
locking, and suffers from all the racing conditions that MySQL suffers 
from in a master-master setup (it is conceptually inherent in the 
approach). It may, however, be good enough for what you want.

IIRC the SeznamFS project only provides debian packages. If you need 
spec files for rpm building, I wrote some a while back and will try to 
find them.

Gordan