THis is an interesting problem for NetApp. I purchased several NetApps for use on Oracle Servers for the Oracle DB. In evaluating all the info I too had problems relating nfs ops/sec to a disk subsystem throughput. ---
Thats because you cant. NFS operations are many things, file creation, removal, buffering, data, lookups, info..etc. Writing a 1Gb file wont create nearly the operations that creating 1GB of 10k files.
So, I've now got 2 F740 systems I'm looking at and so far like a lot. In using a single gigabit connection to a F740 I can easily get sequential writes of around 15mb/sec. At this point the Netapp is running CPU utilization of around +80% utilization. I've seen peaks of close to 18mb/s where the cpu utilization is pegged on 100%. ---
This is about where you'll peg out Ive seen. Until the box has some serious CPU dedicated to RAID stripe calculations, and much MUCH more NVRAM..the platform is artificially limited. And if I understand correctly, your NVRAM caching becomes even less effective when you cluster.
These are not BAD points about the filer at all, just places where improvements are taking place.