Subject: Possible Hashify Corruption
From: Gordan Bobic <gordan@bobich.net>
Date: Sun, 17 Oct 2010 00:30:19 +0100

Hi,

I have just has something very, very strange happen with one of my 
vserver hosts. I have been prototyping things on it for a while and was 
about to move it into production. I powered it off and back on, and 
that's where things started to get inexplicably weird.

Long story short - as far as I can tell _every_ single binary file I 
could find in /bin, /sbin, /lib, /usr/lib, etc is corrputed. They don't 
even check out as executables/libraries using the file command.

My first thought was disk corrpution, but fsck found no errors on the 
disks. The hosts files are all fine, too. Even more strangely, all the 
configuration files in /etc in the guests are fine! I have so far not 
found any files there corrupted!

All the files have the correct size and timestamp (compared with the 
host's non-corrupted files).

I am at a complete loss to explain this at the moment. Everything worked 
fine before shutting down the host.

The only thing I can think of at the moment is that hashify could 
somehow have been responsible for this, but that still doesn't explain 
why everything worked fine right up until shutdown.

Can anybody hazard a guess as to what happened here? I'm prepared to 
consider any theory at the moment, no matter how far fetched.

I'm running 2.6.30.10-vs2.3.0.36.14-pre8. The file system is ext4 
without journal and in data=writeback mode.

Has there ever been a report of anything like this happening before?

I am not too concerned about the damage in this particular instance - 
the RPM database lives outside the guests and is thus OK (I checked), so 
re-building the guests will be fairly trivial and all the configuration 
files in /etc seem to be fine. What DOES concern me, however, is the 
possibility of this happening again, so I am rather keen to hear any 
testable hypotheses that might help get to the bottom of this.

TIA

Gordan