On Sat, Aug 06, 2011 at 04:30:04PM -0400, John A. Sullivan III wrote: > On Sat, 2011-08-06 at 21:40 +0200, Eugen Leitl wrote: >> I've recently figured out how to make low-end hardware (e.g. HP N36L) >> work well as zfs hybrid pools. The system (Nexenta Core + napp-it) >> exports the zfs pools as CIFS, NFS or iSCSI (Comstar). >> 1) is this a good idea? >> 2) any of you are running vserver guests on iSCSI targets? Happy with it? > Yes, we have been using iSCSI to hold vserver guests for a > couple of years now and are generally unhappy with it. Besides > our general distress at Nexenta, there is the constraint of the > Linux file system. > Someone please correct me if I'm wrong because this is a big > problem for us. As far as I know, Linux file system block size > cannot exceed the maximum memory page size and is limited to no > more than 4KB. iSCSI appears to acknowledge every individual > block that is sent. That means the most data one can stream > without an ACK is 4KB. That means the throughput is limited by > the latency of the network rather than the bandwidth. > Nexenta is built on OpenSolaris and has a significantly higher > internal network latency than Linux. It is not unusual for > us to see round trip times from host to Nexenta well upwards > of 100us (micro-seconds). Let's say it was even as good as > 100us. One could send up to 10,000 packets per second * 4KB > = 40MBps maximum throughput for any one iSCSI conversation. > That's pretty lousy disk throughput. well, largely depends on your networking infrastructe, for example with gigabit ethernet, your upper theoretical limit will be 125MB/s for ethernet, now let's take a closer look at the real thing: iSCSI is basically SCSI over network, more precisely over TCP/IP (at least on ethernet :), so we have a packet size of 1500 bytes in IP land (unless you utilize jumbo packets) which will be reduced by the IPv4 header size (min 20 bytes) the TCP header size (min 20 bytes) and the iSCSI PDU header (min 48 bytes), which leaves us with 1412 bytes payload. now, on the other end, ethernet adds an equivalent of 38 bytes (8 bytes preamble, 14 bytes header, 4 bytes trailer and an interframe gap equal to 12 bytes), which means that it takes the time of 1538 bytes to transfer 1412 bytes of payload, which in turn takes 1.538 microseconds to transmit assuming a TCP receive window of 64k (maximum) we end up with 750 kilobit per second at an optimal roundtrip time of 0.1 miliseconds, which means 93MB/s theoretical maximum for iSCSI over TCP/IP on gigabit ethernet. any latency in target or initiator or any disruption of the ethernet connection, any delay from a switch or any other traffic on the network will drastically reduce that to the more realistical 50MB/s ... and don't forget, that's the raw block device speed and does not account for filesystem overhead and possible fragmentation. so I assume if you speak of pretty lousy disk throughput of 40MB/s you are using isolated 10G ethernet on the client (initiator) side and separate 10G connections on the target side together with TCP offloading and a really smart switch all utilizing jumbo frames :) best, Herbert > Other than that, iSCSI is fabulous because it appears as a > local block device. We typically mount a large data volume into > the VServer host and the mount rbind it into the guest file > systems. A magically well working file server without a file > server or the hassles of a network file system. Our single > complaint other than about Nexenta themselves is the latency > constrained throughput. > Any one have a way around that? Thanks - John