Marc --

Given that no one else has offered you opinions (none that I've seen on this distribution list, anyway), I thought I ought to comment. I'll try not to exhibit too much of a NetApp vendor bias.

Apologies if this is off the charter -- let me know (gently, please!) and I'll desist -- but I'm evaluating dedicated NFS servers and have narrowed things down to Network Appliance (F220, qty 2) versus Falcon Systems (FastfilePro 7000, qty 1). I must admit that I'm leaning toward the 7000 due to its superior expandability and greater administrative flexibility (multiple filesystems and RAID sets, ability to run old drives via JBOD on the SCSI port). ...

As I'm sure you are aware, NetApp has made design choices towards simplification. The goal is not to offer "administrative flexibility" but instead to provide a dedicated function appliance where admin is streamlined to a minimal set of tasks.

For example, unlike a mundane file server that requires the administrator(s) to load balance among multiple file systems, and reallocate storage between them, a NetApp filer has a single file system that can be *software*-partitioned among subtrees such that space allocation can be managed on-the-fly by simply editing the tree quotas in /etc/quotas, and where no load-balancing whatsoever need be attempted. You can export the single file system as if it were multiple file systems by exporting subtrees with various permissions and options specified in the /etc/exports file.

... Also, the 7000 tests about even with the F540 in various magazine reviews, and NetApp's own LADDIS testing shows the F220 to be only half as fast as the F540 (is this true in the real world?).

Well, in at least one magazine review I think it was UNIX REVIEW, as I recall) a benchmark called "bigdisk" was used, which showed Falcon's performance as comparable to a NetApp filer. Other than that I don't recall other reviews where Falcon matched NetApp. Anyway, in that article, bigdisk exercised the Falcon using NFSv2 over UDP, because at that time Falcon's software did not support NFSv3 nor NFS over TCP. NetApp was tested with NFSv3 over TCP. Running NFS over TCP is known to hurt performance compared to UDP, but TCP is provided because it is better for WANs and other "lossy" networks where packet loss can cause heavy retrans overhead. Anyway, we ran bigdisk internally, using NFSv2 over UDP, and our performance was significantly better than what that magazine reported for Falcon.

On the broader topic of benchmarking, the industry-standard benchmark of NFS file server performance is SFS (also known as "LADDIS") from SPEC. SPEC has a peer review process whereby official results are submitted, and if accepted by SPEC, can be published in the SPEC Newsletter and on the SPEC website -- (see http://www.specbench.org/osg/sfs93/results/results.html). You will find Falcon and NetApp SFS benchmark results at that site (and in the hardcopy Newsletter, if you have access to it). NetApp has also published SFS results which have been accepted by SPEC, but which have been inexplicably delayed in appearing on the SPEC website. SPEC assures me that they will appear very soon, probably by early next week. In the meanwhile, you can see our latest results for our F210, F230, F520, and F630 filers at http://www.netapp.com/technology/performance.html.

Falcon has published four results via SPEC, but none of them are annotated as being for a model "7000", so I'm not sure what that machine's SFS benchmark results might be. Also, in 3 out of those 4 tested configurations, Falcon used a SSD (SolidState Disk) to significantly accelerate their performance. If the proposed Falcon configuration you are considering does not include that SSD, then you should ask to have that added if you want to optimize that system's performance. On the 4th Falcon configuration for which they have published SFS results, instead of SSD, they combined a BBU (Battery Backup Unit) with a Mylex disk controller's RAM cache to accelerate Write performance. If you are considering that approach, instead of using SSD, be sure not to omit the BBU, because otherwise pending Writes (asynchronously acknowledged to the clients) could be lost in the event of a power interruption. (NetApp uses battery-backed NVRAM ((Non-Volatile RAM)) to capture and preserve updates to the file system, evenin the event of an interruption in power.

The SPEC results will show you that our filers outperform Falcon's servers in terms of average response time. This means we provide faster replies to clients. Also, Falcon has only published results for one configuration where parity RAID protection was enabled. That happens to be for a model 9000 system with 62 disks, with three RAM-caching Mylex controllers, and an SSD. Even so, two F220s together will present more total ops/s (1213 ops/s each, or 2426 together) than the large-config Falcon 9000 (2392 ops/s). Each F220 needed only 14 disks to achieve this performance, so even two of them together were exercising less than half the spindles than the high-end Falcon. And with our latest software release, and the newer F230 filer, the performance has improved both in terms of faster response time (4.5ms for 1221 ops/s, compared to the F220's 9.4 ms for 1231 ops/s) and greater max throughput (1610 ops/s).

But this is benchmarking numerology, to some extent. You are wise to ask about realworld performance. We have consistently found that NetApp filers deliver better performance for actual application environments than indicated by the SFS benchmark because the benchmark has a "worst-case" workload, compared to most application requirements. Also, unlike other vendors (including Falcon) where the SFS benchmark load is *perfectly* load-balanced across many file systems, NetApp is benchmarking with a single file system. Realworld applications depend on the performance of the individual file system where the target data resides. It is rare to find an individual application that is actively exercising data stored in multiple file systems.

I've written a paper which goes into greater detail on this topic: "Interpreting SPEC SFS ('LADDIS') Benchmark Results" (http://www.netapp.com/technology/level3/3010.html). It doesn't specifically mention Falcon, but I think you will find it useful in the analysis of your requirements and the comparison of NetApp filer performance versus benchmark results for multiple-file system configurations such as Falcon's.

On the other hand, NetApp tells me that the highly random nature of my traffic -- typical ISP work including email, USENET news, and webservers -- should point me toward multiple servers to maximize main memory cache hits. They say that the main memory cache on the 7000 will become worthless as I expand storage and that performance will drop off. I don't have a good feel for the importance of main memory cache here -- the 7000 uses hardware RAID controllers with onboard cache, so it seems like it would be less important to the Falcon product than it is to the NetApp line. But I must admit that NetApp's larger installed base -- 1000's versus 100's of systems -- makes it a safer choice on the surface.

On the above topic, you should check out Karl Swartz's paper: "The Brave Little Toaster Meets Usenet" (a USENIX LISA paper -- see http://www.netapp.com/technology/level3/brave.html).

In general, for large data sets and the highly randomized loads seen by growing ISPs, it is futile to try to solve your I/O problems with larger caches. Let's say you have 20 GB of data that is being accessed by a large population of diverse users, such that there is relatively little likelihood of data re-use from the cache. If even only 25% of that 20-GB is exercised, then you would need 4 GB RAM, and you'd merely fill it with data retrieved only to never need it again, as the next 4-GB of different data flows through the cache over the next equivalent time interval.

So what you really need to focus on is disk subsystem performance. And that's where we excel. Note that we have published all our benchmark results with relatively small disk complements, giving us the best per-disk (and per-file-system) performance results in the industry.

Anyone care to share his/her thoughts on this topic?

Well, I hope this helped. I tried to provide as much relevant information as I could, with references, and to avoid spouting unsubstantiated opinions.

Good luck in your exploration of alternatives!

-- Andy

Andy Watson Director, Technical Marketing watson@netapp.com Network Appliance +1 408 367 3220 voice 2770 San Tomas Expressway +1 408 367 3151 fax Santa Clara, CA 95051 http://www.netapp.com/

"It's really stupid to be an intellectual when you're young. You should be an intellectual when you're a hundred years old and can't feel anything anymore." -- a character in Bruce Sterling's novel, HOLY FIRE