What are results with minra=off? Wouldn't that be a better setting for large sequential files?
Yeah, it definitely is.... I'm getting peaks of almost 15MB/secdisk reads with three clients over three 100tx ports reading three different 256MB files. sysstat reports ~55% CPU usage at that point. Reading all files out of cache gets me close to 24MB/sec (at roughly the same CPU usage).
Good. Glad to hear this.
Is the disk bandwidth limited because of parity calculations onreads? I just realized that RAID literature seems to concentrate on disk write performance hits with a parity system, but it would be reasonable that a RAID will calculate and compare parity for reads as well. That would explain the upper limit on sequential read throughput.
Actually, unless we're in degraded mode (meaning a disk has failed and we either have no spare disk to rebuild onto or we're actually in the time window of the process of rebuilding to the hot spare disk) there should be no RAID overhead whatsoever on reads. I'm sure Guy or someone will correct me if I'm wrong here...
I agree, this seems slow. How many disks in that volume/raid group in question? Could that file somehow be very fragmented?
5x9GB drives, tests were done on a mostly empty filesystem with3x256MB and 4x64MB files laid down sequentially (no other write activity in between).
5 data disks or 5 disks including parity (meaning 4 data disks)? Either way, that's well below our sweet spot of 14 9GB disks per raid group. If sequential performance is critical I would obviously consider adding more drives. FYI - I have no idea what our sweet spot is for 18GB drives. Either way. I suspect this is now your bottleneck.
Reading the same file from cache is faster, but still not close to what I'd expect over an otherwise quiet 100 Mbps link.
Can't agree with you here. You're getting roughly 8.4 MB/s over link capable of about 10-12.5 MB/s max. Not optimal, but not bad either...
I'm also quite willing to believe that Solaris 2.6 doesn't havethe fastest NFS client code either... my FreeBSD box accessing the same Netapp over a switched 100t LAN manages to sustain 9200K/sec. ;-)
I always suspect NFS client code <g>. Actually, what are your mount options? They may also provide some clues...
-Val. ============================================== Val Bercovici Office: (613)724-8674 Systems Engineer Pager: (800)566-1751 Network Appliance valb@netapp.com Ottawa, Canada FAST,SIMPLE,RELIABLE ==============================================