On Mon, 6 Nov 2000, Brian Tao wrote:
http://www.spec.org/osg/sfs97/results/res2000q3/sfs97-20000905-00200.html
What's special about a "scalable storage cluster", or is that justa new marketing name given to a bunch of independent F840's? A quarter million ops per second... sweet. :)
Clueless customer: "Wow, that sucks compared with EMC!"
EMC could then create their own "scalable storage cluster" by latching 16 Celerra's together to get ~1.6 M ops/s throughput.
http://www.spec.org/osg/sfs97/results/res2000q3/sfs97-20000711-00180.html
Clueless customer: "Wow, that sucks compared with NetApp!"
NetApp and EMC have to stop staging their own arms race with these SPEC numbers. The Celerra was designed to hold 14 datamovers (N+1 failover) and it is fair to test it as such. Filers were designed to be clustered in pairs, and it is fair to test them as such. It is not fair, in my opinion, to test "clusters" by stringing N of each of them together. The limit on scaling in such scenarios then becomes effectively, infinite, which is meaningless.
EMC gained over 100,000 ops/s with their Celerra the hard way: NFS v3 over TCP. NetApp gained their numbers the easy way: NFS v2 over UDP. My plea to NetApp is: please publish NFS v3 over TCP numbers, so we customers can make a fair comparison. You used to do this with the F760s. Or, conversely, to EMC: please publish NFS v2 over UDP. Then we can see who has the bigger di...<nevermind>. [2]
Anyone who takes these benchmark numbers at face value is a fool and deserves to have their money taken away from them by either vendor, for not doing their homework.
My challenge to BOTH vendors is: in addition to "maximum throughput" configurations and numbers, please publish SPEC numbers in REAL-LIFE configurations, configurations that customers actually use. [2,3] (Note the plurality.)
In the meantime, stop the foolishness, boys.
[unreferenced footnote] Sorry, folks, for the rant, but this really pushes my buttons. As some of you may know, The Mathworks now has NetApp filers *and* an EMC Celerra/Symmetrix. We went through all this number bullshit for months with both vendors [1], so I hate to see this foolishness again. When you dive into the numbers, both vendors' CPU units (filer heads or datamovers) are "comparable." Sometimes one is a bit ahead of the other, sometimes vice versa. But, they are approximately the same when you are able to sift through the numbers and compare apples to apples. If there were a clear winner in the strict ops/s game, then everyone would buy from that vendor if all they needed were ops/s. Luckily for us customers, there is healthy competition, which gives us what we need: better performance year after year. And, one more thing: good buying decisions are rarely as one-sided as choosing one vendor for one specification. Look at the whole picture: performance, scalability, reliability, ease of use, service, in-house knowledge/experience, etc.
[1] With the numbers published so far, for F760s, for instance, we can see there is approximately and conservatively a 45% scaling factor between NetApp's NFS v2 over UDP versus their NFS v3 over TCP numbers. Assuming that ratio approximately translates to the F840s (ONTAP and WAFL are the same for 5.3.x, for instance), this magic 16-node filer cluster has 250,000 ops/s NFS v2 over TCP, which I figure is about 112,500 ops/s NFS v3 over TCP. That's about the same as EMC's Celerra with 14 datamovers. 14 datamovers or 16 filer heads...about the same performance within 10-15%.
[2] And, to preempt the inevitable questions, yes, the Celerra is not running in mirrored mode like most people would, and yes, it has more than one Symmetrix behind it. And, yes, the NetApp disables snapshots, and, yes, the filer has checksum blocks off, and yes, they minimize read-ahead (default values are all the opposite). The devil is in the footnotes...
[3] EMC: How about a mirrored configuration on one Symmetrix with, say, 8 datamovers, one being an active failover? NetApp: How about an out-of-the-box default clustered pair configuration? To both: How about NFS v3 over TCP (hard) and NFS v2 over UDP (easy), to see the *range* of your respective boxes?
Until next time...
The Mathworks, Inc. 508-647-7000 x7792 3 Apple Hill Drive, Natick, MA 01760-2098 508-647-7001 FAX tmerrill@mathworks.com http://www.mathworks.com ---