Matt Harrington matt@ucsd.EDU writes:

...

I'm evaluating a NAS filer similar to the ones sold by NetApp and Auspex. This particular model is made by Nitech in Irvine, CA. These NFS vendors speak about benchmarks in ops/sec. It's more intuitive for me to think of benchmarks in terms of MB/sec throughputs for a large file write or read.

My quick-and-dirty tests show that I can write at 1.8 MByte/sec over a 100Mbit network using NFS v3. I can FTP at about 7 MByte/sec.

My question is: do these numbers seem reasonable? If not, what aspects of NFS should I tweak?

---matt

THis is an interesting problem for NetApp. I purchased several NetApps for use on Oracle Servers for the Oracle DB. In evaluating all the info I too had problems relating nfs ops/sec to a disk subsystem throughput.

NetApp wanted us to purchase a system on 30 day trial to see if it would do what I needed. I refused and told them that without an up front proof of concept there was no discussion of a trial purchase. So, our local tech rep brought a F6xx model (I forget) that we played with for half a day. With this box and 100BaseT connections to our RS/6k systems I got sequential writes of around 8-9mb/sec. This was much faster than one raid system I have that can only handle sequential writes at around 5mb/second, but not anywhere near the speed of a Hitachi Raid system I tested that could accept sequential writes at 25mb/sec (it had other problem that made it unusable).

So, I've now got 2 F740 systems I'm looking at and so far like a lot. In using a single gigabit connection to a F740 I can easily get sequential writes of around 15mb/sec. At this point the Netapp is running CPU utilization of around +80% utilization. I've seen peaks of close to 18mb/s where the cpu utilization is pegged on 100%.

What I've also been doing is running some standard benchmark tests against the netapp. While this provides very rough info, it's better than nfs ops/sec. Below are some results from a program called stkio ( http://www.stortek.com/StorageTek/software/freeware ):

----------------------------------------------------------- -- 1 process ---------------------------------------------- ----------------------------------------------------------- STKIO Version 1.0 Copyright 1997 StorageTek Corporation

Run for 1 loops to 1 files/volumes number of processes is 1 IO type is seq writes file size is 2000 MB Initial transfer length = 4096 bytes Ending Transfer length will be 32768 bytes I/O processes terminate after 2097152000 bytes are transferred Statistics capture enabled 16.53 MB per second for 4KB (4232 IO's per second). Slowest IO 42429.75ms, Quickest IO 0.07ms, Mean 0.22ms 16.52 MB per second for 8KB (2114 IO's per second). Slowest IO 49188.87ms, Quickest IO 0.12ms, Mean 0.46ms 13.53 MB per second for 16KB (865 IO's per second). Slowest IO 39593.59ms, Quickest IO 0.24ms, Mean 1.13ms 17.48 MB per second for 32KB (559 IO's per second). Slowest IO 47646.34ms, Quickest IO 0.48ms, Mean 1.76ms ----------------------------------------------------------- STKIO Version 1.0 Copyright 1997 StorageTek Corporation

Run for 1 loops to 1 files/volumes number of processes is 1 IO type is seq reads file size is 2000 MB Initial transfer length = 4096 bytes Ending Transfer length will be 32768 bytes I/O processes terminate after 2097152000 bytes are transferred Statistics capture enabled 15.97 MB per second for 4KB (4087 IO's per second). Slowest IO 2316.58ms, Quickest IO 0.03ms, Mean 0.23ms 18.54 MB per second for 8KB (2373 IO's per second). Slowest IO 370.41ms, Quickest IO 0.05ms, Mean 0.40ms 16.96 MB per second for 16KB (1085 IO's per second). Slowest IO 1754.96ms, Quickest IO 0.11ms, Mean 0.90ms 16.68 MB per second for 32KB (533 IO's per second). Slowest IO 416.97ms, Quickest IO 0.28ms, Mean 1.84ms ----------------------------------------------------------- STKIO Version 1.0 Copyright 1997 StorageTek Corporation

Run for 1 loops to 1 files/volumes number of processes is 1 IO type is seq reads and writes file size is 2000 MB Initial transfer length = 4096 bytes Ending Transfer length will be 32768 bytes I/O processes terminate after 2097152000 bytes are transferred Statistics capture enabled 14.10 MB per second for 4KB (3609 IO's per second). Slowest IO 30118.43ms, Quickest IO 0.07ms, Mean 0.26ms 13.60 MB per second for 8KB (1740 IO's per second). Slowest IO 41456.73ms, Quickest IO 0.15ms, Mean 0.56ms 6.21 MB per second for 16KB (397 IO's per second). Slowest IO 2302.37ms, Quickest IO 0.42ms, Mean 2.49ms 6.99 MB per second for 32KB (223 IO's per second). Slowest IO 2177.81ms, Quickest IO 0.91ms, Mean 4.42ms ----------------------------------------------------------- STKIO Version 1.0 Copyright 1997 StorageTek Corporation

Run for 1 loops to 1 files/volumes number of processes is 1 IO type is random writes file size is 2000 MB Initial transfer length = 4096 bytes Ending Transfer length will be 32768 bytes I/O processes terminate after 2097152000 bytes are transferred Statistics capture enabled 1.60 MB per second for 4KB (410 IO's per second). Slowest IO 38207.61ms, Quickest IO 0.03ms, Mean 2.42ms 4.34 MB per second for 8KB (554 IO's per second). Slowest IO 32236.26ms, Quickest IO 0.05ms, Mean 1.79ms 8.53 MB per second for 16KB (546 IO's per second). Slowest IO 39446.70ms, Quickest IO 0.11ms, Mean 1.82ms 11.47 MB per second for 32KB (367 IO's per second). Slowest IO 49895.58ms, Quickest IO 0.21ms, Mean 2.71ms ---------------------------------------------------------- STKIO Version 1.0 Copyright 1997 StorageTek Corporation

Run for 1 loops to 1 files/volumes number of processes is 1 IO type is random reads file size is 2000 MB Initial transfer length = 4096 bytes Ending Transfer length will be 32768 bytes I/O processes terminate after 2097152000 bytes are transferred Statistics capture enabled 2.26 MB per second for 4KB (579 IO's per second). Slowest IO 635.95ms, Quickest IO 0.03ms, Mean 1.71ms 3.25 MB per second for 8KB (416 IO's per second). Slowest IO 553.14ms, Quickest IO 0.04ms, Mean 2.39ms 4.45 MB per second for 16KB (284 IO's per second). Slowest IO 545.42ms, Quickest IO 0.11ms, Mean 3.49ms 5.86 MB per second for 32KB (187 IO's per second). Slowest IO 533.81ms, Quickest IO 0.21ms, Mean 5.31ms ----------------------------------------------------------- STKIO Version 1.0 Copyright 1997 StorageTek Corporation

Run for 1 loops to 1 files/volumes number of processes is 1 IO type is random reads and writes file size is 2000 MB Initial transfer length = 4096 bytes Ending Transfer length will be 32768 bytes I/O processes terminate after 2097152000 bytes are transferred Statistics capture enabled 1.55 MB per second for 4KB (397 IO's per second). Slowest IO 17135.13ms, Quickest IO 0.05ms, Mean 2.50ms 1.33 MB per second for 8KB (170 IO's per second). Slowest IO 17548.93ms, Quickest IO 0.10ms, Mean 5.84ms 1.47 MB per second for 16KB (93 IO's per second). Slowest IO 20197.14ms, Quickest IO 0.23ms, Mean 10.62ms 3.81 MB per second for 32KB (121 IO's per second). Slowest IO 24408.55ms, Quickest IO 0.47ms, Mean 8.18ms ----------------------------------------------------------- = 2 processes =============================================== ----------------------------------------------------------- STKIO Version 1.0 Copyright 1997 StorageTek Corporation

Run for 1 loops to 1 files/volumes number of processes is 2 IO type is seq writes file size is 2000 MB Initial transfer length = 4096 bytes Ending Transfer length will be 32768 bytes I/O processes terminate after 2097152000 bytes are transferred Statistics capture enabled 14.07 MB per second for 4KB (3602 IO's per second). Slowest IO 3640.43ms, Quickest IO 0.04ms, Mean 0.54ms 14.08 MB per second for 8KB (1802 IO's per second). Slowest IO 6971.74ms, Quickest IO 0.06ms, Mean 1.09ms 14.32 MB per second for 16KB (916 IO's per second). Slowest IO 7378.41ms, Quickest IO 0.24ms, Mean 2.16ms 15.34 MB per second for 32KB (490 IO's per second). Slowest IO 14434.84ms, Quickest IO 0.45ms, Mean 4.04ms ----------------------------------------------------------- STKIO Version 1.0 Copyright 1997 StorageTek Corporation

Run for 1 loops to 1 files/volumes number of processes is 2 IO type is seq reads file size is 2000 MB Initial transfer length = 4096 bytes Ending Transfer length will be 32768 bytes I/O processes terminate after 2097152000 bytes are transferred Statistics capture enabled 13.73 MB per second for 4KB (3519 IO's per second). Slowest IO 377.84ms, Quickest IO 0.03ms, Mean 0.55ms 17.47 MB per second for 8KB (2237 IO's per second). Slowest IO 447.69ms, Quickest IO 0.04ms, Mean 0.87ms 14.29 MB per second for 16KB (915 IO's per second). Slowest IO 451.40ms, Quickest IO 0.10ms, Mean 2.15ms 14.06 MB per second for 32KB (450 IO's per second). Slowest IO 418.53ms, Quickest IO 0.21ms, Mean 4.40ms ----------------------------------------------------------- STKIO Version 1.0 Copyright 1997 StorageTek Corporation

Run for 1 loops to 1 files/volumes number of processes is 2 IO type is seq reads and writes file size is 2000 MB Initial transfer length = 4096 bytes Ending Transfer length will be 32768 bytes I/O processes terminate after 2097152000 bytes are transferred Statistics capture enabled 2.00 MB per second for 4KB (512 IO's per second). Slowest IO 1886.46ms, Quickest IO 0.07ms, Mean 3.88ms 2.93 MB per second for 8KB (375 IO's per second). Slowest IO 2520.66ms, Quickest IO 0.13ms, Mean 5.29ms 5.03 MB per second for 16KB (321 IO's per second). Slowest IO 353.10ms, Quickest IO 0.25ms, Mean 6.17ms 5.55 MB per second for 32KB (177 IO's per second). Slowest IO 1252.26ms, Quickest IO 0.59ms, Mean 11.21ms ----------------------------------------------------------- STKIO Version 1.0 Copyright 1997 StorageTek Corporation

Run for 1 loops to 1 files/volumes number of processes is 2 IO type is random writes file size is 2000 MB Initial transfer length = 4096 bytes Ending Transfer length will be 32768 bytes I/O processes terminate after 2097152000 bytes are transferred Statistics capture enabled 1.65 MB per second for 4KB (422 IO's per second). Slowest IO 27679.15ms, Quickest IO 0.05ms, Mean 4.72ms 4.54 MB per second for 8KB (581 IO's per second). Slowest IO 32073.85ms, Quickest IO 0.09ms, Mean 3.42ms 8.55 MB per second for 16KB (547 IO's per second). Slowest IO 37631.32ms, Quickest IO 0.16ms, Mean 3.64ms 11.54 MB per second for 32KB (369 IO's per second). Slowest IO 48069.40ms, Quickest IO 0.33ms, Mean 5.40ms ---------------------------------------------------------- STKIO Version 1.0 Copyright 1997 StorageTek Corporation

Run for 1 loops to 1 files/volumes number of processes is 2 IO type is random reads file size is 2000 MB Initial transfer length = 4096 bytes Ending Transfer length will be 32768 bytes I/O processes terminate after 2097152000 bytes are transferred Statistics capture enabled 5.81 MB per second for 4KB (1488 IO's per second). Slowest IO 648.85ms, Quickest IO 0.03ms, Mean 1.33ms 6.98 MB per second for 8KB (894 IO's per second). Slowest IO 1760.59ms, Quickest IO 0.05ms, Mean 2.22ms 8.78 MB per second for 16KB (561 IO's per second). Slowest IO 610.85ms, Quickest IO 0.10ms, Mean 3.54ms 11.62 MB per second for 32KB (372 IO's per second). Slowest IO 609.94ms, Quickest IO 0.22ms, Mean 5.35ms ----------------------------------------------------------- STKIO Version 1.0 Copyright 1997 StorageTek Corporation

Run for 1 loops to 1 files/volumes number of processes is 2 IO type is random reads and writes file size is 2000 MB Initial transfer length = 4096 bytes Ending Transfer length will be 32768 bytes I/O processes terminate after 2097152000 bytes are transferred Statistics capture enabled

---------------------------------------------------------------------- Richard L. Rhodes e: rhodesr@firstenergycorp.com Ohio Edison Co. p: 330-384-4904 f: 330-384-2514