Subject: Re: [vserver] Very serious hashify mysql/innodb conflict causes major data loss
From: "Daniel Hokka Zakrisson" <daniel@hozac.com>
Date: Tue, 9 Jun 2009 14:38:28 +0200 (CEST)

John Alberts wrote:
> I just wanted to mention I had the same issue with mysql/innodb and
> hashify a couple of months ago.  As Herbert suggested, just don't
> hashify the mysql db directory.  I stopped using vhashify altogether
> and this fixed my problems; however, the default when you run vhashify
> is that it just does everything.  Unless you know to avoid running it
> on the mysql directory, your going to have problems.

No, the default exclude list excludes all of /var. So unless you're storing
the databases elsewhere, or use a custom exclude list where /var is not
present, they will not get hashified.

Daniel

> Regards,
> John
>
>
> On Tue, Jun 9, 2009 at 7:08 AM, Herbert Poetzl<herbert@13thfloor.at> wrote:
>> On Tue, Jun 09, 2009 at 07:41:28AM -0400, John A. Sullivan III wrote:
>>> Hello, all.  We have been struggling for several weeks with an issue of
>>> severe data loss on our zimbra email vserver.  We have narrowed it down
>>> with considerable certainty to a conflict between mysql/innodb and
>>> hashify.  We are running 2.6.28.7-vs2.3.0.36.7, i.e., a custom patched
>>> 2.6.28.7 kernel and util-vserver-0.30.216-1.pre2793.el5.centos.  We can
>>> reproduce the issue at will.
>>
>> updating to a recent version is advised ...
>>
>>> Zimbra is running its own instance of mysql with innodb support version
>>> 5.0.67.  Zimbra runs a zimbra database and then many other mysql
>>> instances named mboxgroupX where X is a one or two digit number.
>>
>>> Whenever we run vserver <zimbra server name> hashify, we see odd
>>> behaviors:
>>>
>>>       * The mysql and zimbra database directories are time stamped with
>>>         the time hashify ran as if mysql was restarted but there is no
>>>         record of mysql being restarted.
>>>       * MOST IMPORTANTLY, the innodb log stops recording but it appears
>>>         that mysql is still functioning.  Mail is displayed, events
>>>         (e.g., calendar, tasks) are recorded and are fully functional
>>
>> there is no pint in hashifying mysql databases,
>> so I would suggest to exclude those dirs ...
>>
>>> The disaster occurs when mysql is restarted, e.g., when restarting the
>>> zimbra service.  Since the innodb log has recorded no data since hashify
>>> ran, mysql thinks the system has crashed and backs out all data since
>>> the last hashify! At least, I believe that's what is happening in my
>>> MySQL ignorance.
>>
>> IMHO hashify should not change the time stamps of
>> files, but it's okay to update directory timestamps
>> if files have been unified there ...
>>
>>> Obviously, this kind of data loss is intolerable.  I'd rather solve the
>>> problem than simply forgo hashify on our zimbra servers.  Any ideas
>>> about what could cause this behavior and how to troubleshoot it?
>>
>> well, my guess would be that mysql either uses
>> the timestamps which get changed when an otherwise
>> harmless file is unified (in that dir) ... the
>> proper solution IMHO is to exclude that dir from
>> hashification ...
>>
>> HTH,
>> Herbert
>>
>>> Thanks
>>> - John
>>> --
>>> John A. Sullivan III
>>> Open Source Development Corporation
>>> +1 207-985-7880
>>> jsullivan@opensourcedevel.com
>>>
>>> http://www.spiritualoutreach.com
>>> Making Christianity intelligible to secular society
>>
>
>
>
> --
> John Alberts
>


-- 
Daniel Hokka Zakrisson