Brian, your comparison was very interesting to read. We're currently working on choosing a storage consolidation, and the main contenders are a Filer and a general-purpose fileserver with a direct-attached raid.
Depends on the application, of course, but I've been surprisedmany times in the past when I thought for sure the Netapp would not be able to keep up. I have a 4x450-MHz E420R with a VxVM RAID-0 device, spread over 16 50GB 7200 rpm drives on two U2SCSI buses. The server also has a Gigabit Ethernet connection to an F740 with one shelf of 36GB 10000 rpm drives (5 data, 1 parity, 1 spare). The local filesystem is vxfs, mounted with delaylog and the largest allowable log area.
If I'm reading this correctly, you're using all software raid for the DAS side, correct? I would be very interested in seeing a similar comparison with a hardware raid controller, as this would offload all the raid overhead from the host cpus to a subsystem better optimized to handle it. This would also unfortunately introduce a lot more variables of the sort Alan Yoder pointed out, as I'm sure the performance under load of the various available raid controllers is far from uniform.
I re-ran some of the same tests with a Veritas RAID-5 volume (tobe fair to the Netapp), but I stopped after the first couple. There is no contest at that point. Veritas software RAID-5 is dog-slow (I think I saw bursts of 8MB/sec sequential writes). Turn on a Veritas snapshot, and writes to the snapped filesystem go even further into the toilet.
This is probably a product of the raid overhead all being handled by a relatively ponderous general-purpose computer, and the lack of a safe way to do write caching.
jm