Subject: Re: [vserver] Why no hashify on Gentoo?
From: Ed W <lists@wildgooses.com>
Date: Mon, 21 Jul 2008 12:11:21 +0100

Daniel Hokka Zakrisson wrote:
> and rewriting parts of vhashify has been on my TODO for
> quite some time (where part of the goal is to not require package management support).
>   

We all have our points of view, but I have given up using anything BUT 
gentoo for servers - just seems to be too much work to keep RPM based 
distros updated

Anyway, I haven't dived into the hashify code, but I don't quite 
understand why it needs package management support *at all*?  Is it to 
avoid problems with kernels that don't support link breaking?

I still think the link breaking stuff is useful as a standalone kernel 
patch... It could be used by a couple of backup strategies that are 
doing rsync like backups.  Could also be used as a poor mans lvm 
snapshot alternative

The ideal functionality would appear to be that you can constrain a hard 
link to be constained to a "container" directory and it would only break 
links to outside that container, whilst leaving links inside the 
container intact (ie you can keep hardlinks inside a vserver intact).  
This seems slightly different to current functionality, but I guess few 
people will feel strongly enough to want to implement these changes ..

> get-conffiles really should be smarter than that. What data does emerge use to tell
> whether etc-update (or whatever it's called, as I said, I'm not a Gentoo-guy) should
be
> used for updating a file?
>   

Gentoo build scripts (ebuilds) are somewhat similar to RPM src spec 
files.  So you write a mini install script and you get some macros that 
you can use to hit files with and say they should be treated as config 
files.  Additionally there is some intersection with system level config 
files which constrain which config files you actually care about merging. 

I guess the info about which files are "config" files resides in 
/var/lib/portage/config - however, I'm not an expert - the syntax 
appears to be a list of files with an MD5 (looking) hash of each file 
content, eg:

# cat /var/lib/portage/config
/etc/pam.d/groupmems 65e9ebce249a5b9ed021e2790452b9e1
/etc/init.d/consolefont b90630a5bfb2e26f94091efb8cf06b75
/etc/conf.d/hwclock f8dd802b9dd7c553e46703033816ff50
/etc/rc.conf fdef83496b9084ae7e682fb2e719dbdb
...


On my installation I see internal hardlinks for:

- Any web packages installed (hardlinked back to a common webapps 
directory, ie all your common web applications which end up in some 
variation of /var/www/...
- /usr/share/timezone
- small number of bin commands, eg git, sudo


So for my needs (ie gentoo + modern kernel with link breaking), it would 
appear that I could write a vhashify command to simply:
- Exclude /var/db/webapps
- Exclude /usr/share/timezone (optional)
- Exclude anything with existing hardlinks
- Hashify the rest


To be honest the actual algorithm used to hashify seems almost 
unimportant.  It's going to be a variation of the well trodden rsync 
style of backup in many ways (although it could also look a bit like the 
more fashionable git style structure depending on how it was 
implemented).  I would have thought with a bit of ingenuity that 
vhashify script could be turned into a snapshot backup style script 
(especially with optional use of LVM where available might be 
interesting, but I suspect that will mangle the hardlinks and possibly 
interfere with the strategy?)

Any takers to whip something up?

Ed W