Subject: Re: [vserver] Sponsoring VServer getting up to date with mainline
From: Herbert Poetzl <herbert@13thfloor.at>
Date: Sat, 23 Aug 2008 04:07:24 +0200

On Fri, Aug 22, 2008 at 06:15:23PM -0500, Michael S. Zick wrote:
> On Fri August 22 2008 17:52, Ed W wrote:
> > Michael S. Zick wrote:
> > >
> > > The true advantage of a hardware card, is you only read/write
> > > the storage data once over the PCIe link.
> > 
> > I agree in theory, but I have been desperately disappointed with a two 
> > 3Ware cards of various generations.  They struggled to get read or write 
> > performance to match a half that of a single drive in a 4 drive raid5 
> > config.  

that matches my experience with some 3ware cards
where actually JBOD and software raid is the
superior solution, but if you are looking at 
pure read/write throughput, my old card reached
160M/s on two channels easily (sustained transfer)
which is mostly limited by the raid layout and
the 80-100M/s sustained transfer rate of the disks

but what's more important, is that the seek time
is a lot better than on a single disk ...

> It is very easy to build a card in a 16x form factor, perhaps using
> only a single (1x) channel; These cards would be Bragging Rights
> and/or Marketing Rights only cards.

the card in question is 8x, the slot 16x, so I
think that should be fine

> Ah, but building a 16x form factor card with electronics that can run
> that fast - - much harder, also much too expensive.
> 
> > Some of the limitations looked very much like those of the PCI  
> > bus and indeed 3wares benchmarks suggested that you needed faster than 
> > PCI to get >20-40MB/sec out of their cards (ugg).  

> The PCI bus is too slow, even the double width, double speed version.
> Even a 2x card can run into bandwidth limitations.
> Keep in mind, PCI is a half-duplex, shared, bus.

> The 16x and 32x PCIe is driven directly by the bridge chipset, no PCI
> bus involved getting to/from RAM.  At least, not if your pocket book
> is deep enough.

> This is another reason why it is hard to find a motherboard with more
> than one 16x or 32x slot (in transfer width, not just connector size).

one 16x is actually enough, I don't need (and do
not have) a graphics card in that machine
(works headless over serial port and GB network)

> If Herbert wants fast compiles, he should keep his kernel tree in a
> RAM disk on a 32 or 64Gbyte ram machine.

trust me, I investigated this path, but as it is
perfect for compiling one and the same kernel
(2-4GB are usually enough for that), this doesn't
work that well for hard linked kernel trees which
in total cover more than a terrabyte of space
(plus tons of temporary clones)

for repeated builds, the kernel does a good job 
with caching, so that happens basically out of
RAM, and so does the repeated diff between versions

> That would probably cut him down to a single serving of his favorite
> beverage per compile.

> Mike
> > To me this looked  
> > very much like the data was being transferred too many times versus 
> > theory (also I discovered later that nearly all these hardware cards 
> > can't do xor fast enough to keep up with a sensible modern disk, hence 
> > software raid5 can often be faster because you aren't lagging on the xor 
> > engine...)
> > 
> > I think the safe benchmark is all in software and be very cautious of 
> > these hardware SATA cards.  That said, the various scsi cards I have 
> > tried absolutely scream.  Why there are only cr*p sata hardware cards I 
> > just don't understand?
> > 
> > Oh well

anyway, I'm going to replace that card in the near
future, so if somebody wants to help there, let
me know, if not, so be it :)

TIA,
Herbert


> > Ed W