Subject: Re: [vserver] Very serious hashify mysql/innodb conflict causes major data loss
From: Corey Wright <undefined@pobox.com>
Date: Tue, 9 Jun 2009 08:34:46 -0500

On Tue, 09 Jun 2009 08:56:25 -0400
"John A. Sullivan III" <jsullivan@opensourcedevel.com> wrote:

> On Tue, 2009-06-09 at 14:38 +0200, Daniel Hokka Zakrisson wrote:
> > John Alberts wrote:
> > > I just wanted to mention I had the same issue with mysql/innodb and
> > > hashify a couple of months ago.  As Herbert suggested, just don't
> > > hashify the mysql db directory.  I stopped using vhashify altogether
> > > and this fixed my problems; however, the default when you run vhashify
> > > is that it just does everything.  Unless you know to avoid running it
> > > on the mysql directory, your going to have problems.
> > 
> > No, the default exclude list excludes all of /var. So unless you're
> > storing the databases elsewhere, or use a custom exclude list
> > where /var is not present, they will not get hashified.
> > 
> > Daniel
> Thanks very, very much to all who replied to this thread.  Indeed,
> Zimbra places the database directories by default under /opt/zimbra.  I
> will take a look at how we exclude those directories.  Thanks again -
> John

per-guest customized vhashify exclusions (on debian)
http://linux-vserver.org/alpha+util-vserver
(that's the link i have in my notes from a few years ago and there's
probably something more relevant if you search the wiki)

# base guest-specific exclusions on the default list
cp
-av /usr/lib/util-vserver/defaults/vunify-exclude /etc/vservers/<vserver>/apps/vunify/exclude
# add all files under /usr/src
echo '/usr/src/*' >>/etc/vservers/<vserver>/apps/vunify/exclude
# view test run of vhashify to see what excluded/included
vserver <vserver> hashify -nv | less -S
# hashify
vserver <vserver> hashify

yeah, what i learned the hard way the first time i customized vhashify
exclusions (like you learned the hard way about vhashify in general) is
that the per-guest list *supplements* the default list, it doesn't
*compliment* it.  otherwise you are going to have fun unhashifying your
guest if you just put "/opt" in /etc/vservers/<vserver>/apps/vunify/exclude
(though look at the wiki and mailing list for how to do it as you won't be
the first ;-).

corey
-- 
undefined@pobox.com

> > 
> > > Regards,
> > > John
> > >
> > >
> > > On Tue, Jun 9, 2009 at 7:08 AM, Herbert Poetzl<herbert@13thfloor.at>
> > > wrote:
> > >> On Tue, Jun 09, 2009 at 07:41:28AM -0400, John A. Sullivan III wrote:
> > >>> Hello, all.  We have been struggling for several weeks with an
> > >>> issue of severe data loss on our zimbra email vserver.  We have
> > >>> narrowed it down with considerable certainty to a conflict between
> > >>> mysql/innodb and hashify.  We are running 2.6.28.7-vs2.3.0.36.7,
> > >>> i.e., a custom patched 2.6.28.7 kernel and
> > >>> util-vserver-0.30.216-1.pre2793.el5.centos.  We can reproduce the
> > >>> issue at will.
> > >>
> > >> updating to a recent version is advised ...
> > >>
> > >>> Zimbra is running its own instance of mysql with innodb support
> > >>> version 5.0.67.  Zimbra runs a zimbra database and then many other
> > >>> mysql instances named mboxgroupX where X is a one or two digit
> > >>> number.
> > >>
> > >>> Whenever we run vserver <zimbra server name> hashify, we see odd
> > >>> behaviors:
> > >>>
> > >>>       * The mysql and zimbra database directories are time stamped
> > >>> with the time hashify ran as if mysql was restarted but there is no
> > >>>         record of mysql being restarted.
> > >>>       * MOST IMPORTANTLY, the innodb log stops recording but it
> > >>> appears that mysql is still functioning.  Mail is displayed, events
> > >>>         (e.g., calendar, tasks) are recorded and are fully
> > >>> functional
> > >>
> > >> there is no pint in hashifying mysql databases,
> > >> so I would suggest to exclude those dirs ...
> > >>
> > >>> The disaster occurs when mysql is restarted, e.g., when restarting
> > >>> the zimbra service.  Since the innodb log has recorded no data
> > >>> since hashify ran, mysql thinks the system has crashed and backs
> > >>> out all data since the last hashify! At least, I believe that's
> > >>> what is happening in my MySQL ignorance.
> > >>
> > >> IMHO hashify should not change the time stamps of
> > >> files, but it's okay to update directory timestamps
> > >> if files have been unified there ...
> > >>
> > >>> Obviously, this kind of data loss is intolerable.  I'd rather solve
> > >>> the problem than simply forgo hashify on our zimbra servers.  Any
> > >>> ideas about what could cause this behavior and how to troubleshoot
> > >>> it?
> > >>
> > >> well, my guess would be that mysql either uses
> > >> the timestamps which get changed when an otherwise
> > >> harmless file is unified (in that dir) ... the
> > >> proper solution IMHO is to exclude that dir from
> > >> hashification ...
> > >>
> > >> HTH,
> > >> Herbert
> > >>
> > >>> Thanks
> > >>> - John
> > >>> --
> > >>> John A. Sullivan III
> > >>> Open Source Development Corporation
> > >>> +1 207-985-7880
> > >>> jsullivan@opensourcedevel.com
> > >>>
> > >>> http://www.spiritualoutreach.com
> > >>> Making Christianity intelligible to secular society
> > >>
> > >
> > >
> > >
> > > --
> > > John Alberts
> > >
> > 
> > 
> -- 
> John A. Sullivan III
> Open Source Development Corporation
> +1 207-985-7880
> jsullivan@opensourcedevel.com
> 
> http://www.spiritualoutreach.com
> Making Christianity intelligible to secular society
>