Subject: Re: [vserver] Possible Hashify Corruption
From: Gordan Bobic <gordan@bobich.net>
Date: Mon, 18 Oct 2010 22:53:28 +0100

On 10/18/2010 10:42 PM, John A. Sullivan III wrote:
> On Mon, 2010-10-18 at 21:55 +0100, Gordan Bobic wrote:
>> On 10/18/2010 01:53 AM, Herbert Poetzl wrote:
>>
>>> putting superstition aside, can you recreate the issue?
>>> i.e. is there a script or procedure which reliably
>>> produces the 'corruption'?
>>
>> I believe I have just successfully re-created this, and it seems to be
>> specifically related to hashify.
>>
>> I have two VMs, call them vm1 and vm2. Their contents are fine. Run
>> hashify on them and reboot. All the binaries will be broken in the same way.
>>
>> The content of the files seems to be downright random. e.g. in the
>> current instance, bash binary is zero-filled for the first few
>> screenfuls I bothered to look through, and /sbin/init contains what
>> looks like it might be a fragment of a RPM file. It is almost as if
>> hashifying points the blocks somewhere random, most likely unused blocks
>> since I haven't observed any file system level corruption or corruption
>> of files in the host.
>>
>> I'll re-try it again in a bit, but so far it's been consistent.
>>
 >
> Strange - we've been using 2.6.28.7-vs2.3.0.36.7 and
> util-vserver-0.30.216-1.pre2793.el5.centos with ext4 for almost two
> years now - in fact, we really need to upgrade.  We've rebooted the
> hosts many times and have not seen this problem once - John

Yeah, really weird. It's reproducible, too. Files are fine before being 
hashified. It could be a weird bug that only exhibits itself when 
running with ^has_journal and data=writeback. I know I reported a weird 
bug in the ext4 in 2.6.32.x that RHEL6b2 ships with that made it never 
issue TRIM commands when running in data=writeback mode (data=writeback 
is implicit with ^has_journal). So it is plausbible there are other edge 
cases like this triggered by running ext4 without the journal.

I'll try with the latest available kernel and see if that makes the 
problem go away.

Gordan