Hi Bryan !
Please take into account these issues, which favor the appliance solution over the Linux box:
1. Proven stability of filesystem - NetApp's WAFL has real-life proven stability, which is consistent thru shutdown and powerdowns - even though ext3 is using journaling, WAFL's Consistency Points using the NVRAM should cover stability much better (and faster). 2. Crash recovery - filer's fsck (wack) is much much faster than other vendors' solutions - ~600GB/hr and more. 3. Ease of use - adding capacity is that easy - no extra config. Also - you're wasting time on choosing the right CPU arch. and motherboard bus chipsets - leave that to NetApp's engineering experts, and you can save time (and money, and get anyway better performance). 4. Extra NetApp-specific "bonus-features" - snapshot can lower your restore needs dramatically, and can help for some recovery cases. 5. Enterprise-level features which you can leverage in the future (which Linux doesn't have YET, not that I've seen, others - correct me if I'm wrong) - SnapMirror/SnapRestore, which complement the disaster-recovery issues. 6. Future growth - one filer can grow (now) to 1.4TB, which I don't think many Linux storage solution go that high, and NetApp will upgrade to 3TB and higher for one filer very soon. 7. Performance - CIFS performance should be much higher, NFS also much higher. 8. NFS/CIFS securuty - Linux can only share to PCs with Samba, which is fine, but only a user-level application (slower that Data ONTAP's kernel level implementation), and also the filesystem on Linux will still be Unix-based, and will not be able to maintain NT ACLs etc.
I hope it sums it up somehow, Eyal.
--- Bryan -TheBS- Smith thebs@theseus.com wrote:
Linux-NFS list recommended I post here -- RE: NetApp versus Linux
All --
The Linux-NFS list recommended I post here. I am looking at either a $35K NetApp F720 or a $18K Linux (w/Ext3+NFS3 patches) solution. I'd greatly appreciate any input anyone has.
My boss favors the NetApp solution on paper (never used one himself, nor does he do any admin-level stuff). But IMHO, unless we spend $70K to buy two (2) NetApp F720s and take advantage of the NetApp's superior failover/load-balancing capability, I'm not sure it's the right move. We still have to have a Linux server around for CVS (and NIS?) anyway, which would put a NetApp at the mercy of the Linux box anyway (any way to get CVS on NetApp?).
I break down my limited knowledge below. I currently have a 100GB Linux 2.2 server (Ext2+NFS2) that I didn't setup (but have heavily reconfigured because the consulting firm didn't know jack crap, but some things like using 1KB block sizes in the /home filesystem that take forever to fsck is something I can't change right now).
We're looking at going to 250GB (possibly 500-1000GB in the next 6 months). Most of our clients are Solaris 2.6 (about a dozen serious production boxes with dual to 6-way processor, 1 to 6GB of RAM, etc...). We are starting to add more and Linux workstations (both dedicated and dual-booting with NT).
Stability:
+++ NetApp (F720) Boots in under 2 minutes due to the use of NVRAM even after an improper shutdown. Buying two (2) units is an extremely viable failover/load-balancing solution. [ Unfortunately, my boss doesn't want to spend more than $30K).
--- Linux (Ext2) Although our Linux systems have averaged >100 days uptime, a routine power outtage without proper shutdown, unable to umount NFS clients, etc... causes an fsck. We figure 30 minutes per 100GB for Ext2 fscks on 8-16KB blocks (currently takes 90 minutes of 1KB blocks, something the consultants before me did -- dumb!). Never had a problem with lost data though.
--- Linux (ReiserFS) Meta-data journaling, but has NFS locking issues (except for a SuSE patch to the old knfsd?). Eliminated from consideration
??? Linux (Ext3) Full data journaling means sync I/O??? What about current stability (I'm using Trond+Higgen 0.21.3 on non-production systems with Solaris 2.6 clients and I'm happy)? Locking with Ext3 (never used Ext3, but many others are now)? I know VA Linux now has Linux 2.2 kernels with Ext3 and NFS3 that are supposively quite stable.
Performance:
??? NetApp (F720 + 1000Base-SX NIC) While the NVRAM in the NetApp allows almost instant sync acknoledgement to clients, the F720 only comes with 1/4th the NVRAM of the F740/760 (8MB versus 32MB). The result is that the F720 can only handle 1/3-1/4th the NFS requests and throughput of the F740/760. I have seen good marks on the later, but cannot find much on the former which makes me wonder if we will run into a performance issue? Transmeta also mentioned that they have had performance issues with the 700-series on high-loads because the on-baord CPU, a single Alpha (usually ~600MHz) cannot even keep up with the number of XORs for just the RAID-4 volume. Of course I only have a dozen NFS clients and Transmeta has many more. Hard drives are of older, 36GB, 7200rpm, Ultra2 (aka Ultra80) SCSI type.
+++ Linux (dual-cpu/memory ServerSet IIIHE chipset + dual-733MHz Pentium III + 233MHz StrongARM-powered, 4-channel, Ultra160 RAID controller on its own PCI 64-bit channel + 1000Base-SX NIC card on it's own 66MHz PCI 64-bit channel) These ServerSet III-powered mainboards seem to best both the older 4-way i450HX EDO mainboards, and the 2-way i840 in just about every memory benchmark. Going to use 2GB of RAM (which is 8x as much as in the F720), and can expand to 8GB later. The 3-independent PCI channels (of which, 2 are 66MHz PCI 64-bit) allows me to put the disk controller and network controller on their own, 64-bit channels. They Mylex 233MHz StrongARM-powered RAID controllers are the fastest you can get for just about anything (including Linux, especially when compared to vendors who still use a 100MHz i960 ASIC) and I'm going to pair it with 73GB, 10000rpm, Ultra160 disks with two disks per channel spread over all 4 channels. On the software side, I've had no issues running async with Linux. I'm more worried about Ext3's full-data journaling being the slouch here. Also wondering how long it takes, exactly, to recover after an improper shutdown with full-data journaling Ext3 (I'm assuming <5 minutes for 250GB?).
[ Further insight wanted: Just wondering if the F720 will be a slouch compared to this? Or will the NVRAM really kick Ext3's butt, even if there is only 8MB of it? ]
Other Considerations:
NetApp (Failover Option) Features "ready-to-go" fail-over option via proprietary gigabit links between units. Another unit can even use the disk controller and disks on a failed unit CPU unit. Very nice. Also lowers the admin requirements (like having to keep kernel/apps up-to-date, etc...). Someone on the Linux-NFS list actually mentioned administration wasn't so easy with NetApp (no web-based admin???) and would rather have Linux (I personally couldn't believe this, but she was a seasoned Linux admin like myself).
Linux (Software Support) I still need a Linux box to server as the CVS server, Mail server (including various Sendmail/Mail-based programs like AV scanner, HP OpenMail, Majordomo, GNATS, etc...), NIS server (or does NetApp have NIS server capabilities?), etc... This makes me wonder if I really will "save on TOC" with the NetApp since my Linux admin outside of this has only been about 40-80 hours over the past year (compiling new kernels, installing new NFS, etc...).
Cost:
--- NetApp F720 (7 x 36GB disk = 252GB, 216GB usable RAID-4, 180GB if using hot spare) $35K for the configuration (F720 = 600MHz? Alpha, 256MB RAM, 8MB NVRAM) with both SMB and NFS plus the 1000Base-SX upgrade.
+++ Linux (5 x 73GB disk = 365GB, 292GB usable RAID-5, 219GB if using hot spare) $18K for the configuration as above (dual-733MHz, 2GB RAM, RAID controller, 1000Base-SX NIC, etc...) with SAF-TE rackmount disk chassis, ATX rackmount chassis, cabling and redundant power on both the SAF-TE and ATX case.
Thanx in advance ...
-- Bryan "TheBS" Smith "Lead Computer Geek" Theseus Logic, Inc.
-- Bryan "TheBS" Smith CONTACT INFO
Chat: thebs413 @ AOL/MSN/Yahoo (see http://Everybuddy.com) Email: mailto:thebs@theseus.com,b.j.smith@ieee.org Home: http://www.SmithConcepts.com
===== Yours, Eyal Traitel eTraitel@yahoo.com, Home: 972-3-5290415 (Tel Aviv) *** eTraitel - it's the new eBuzzword ! ***
__________________________________________________ Do You Yahoo!? Get Yahoo! Mail � Free email you can access from anywhere! http://mail.yahoo.com/
On Thu, 20 Jul 2000, Eyal Traitel wrote:
- Proven stability of filesystem - NetApp's WAFL has real-life proven
stability, which is consistent thru shutdown and powerdowns - even though ext3 is using journaling, WAFL's Consistency Points using the NVRAM should cover stability much better (and faster).
I understand that. I'm just getting back what I hear from VA Linux developers. Ext3 does full-data journaling, but NVRAM is still much, much faster.
My main problem is that a lot of people like the F740/760, but the F720 only has 1/4th the NVRAM. If I could afford a F740, I'd say F740 for sure. But since I can only get a F720 (and have to argue for one), the Linux option still comes up from a cost perspective.
- Crash recovery - filer's fsck (wack) is much much faster than
other vendors' solutions - ~600GB/hr and more.
Yes, the filer doesn't have to do a fsck because of the NVRAM. I find that Linux Ext2 (with large block sizes) actually does a real, full fsck faster than many other filesystems (especially versus NTFS -- ouch!). But without meta-data journaling, Linux takes forever when you "just want to get the system back up."
- Ease of use - adding capacity is that easy - no extra config.
The Mylex eXtremeRAIDs are very nice, and can resize the volumes on the fly. But resizing Ext2/3 is a pain, I know, because it is a traditional filesystem with pre-laid Inodes. I like the NetApp way with Inode creation on the fly.
Also - you're wasting time on choosing the right CPU arch. and motherboard bus chipsets - leave that to NetApp's engineering experts, and you can save time (and money, and get anyway better performance).
Multiple PCI channel and memory on x86 is getting mighty cheap. In fact, some people have commented that the Alpha in the NetApp gets saturated with duties under high-load conditions.
- Extra NetApp-specific "bonus-features" - snapshot can lower your
restore needs dramatically, and can help for some recovery cases.
Yes, this is a huge bonus in the NetApp department.
- Enterprise-level features which you can leverage in the future
(which Linux doesn't have YET, not that I've seen, others - correct me if I'm wrong) - SnapMirror/SnapRestore, which complement the disaster-recovery issues.
For $5K I can get backup features that are similar, but not as intuitive as the NetApp I admit.
- Future growth - one filer can grow (now) to 1.4TB, which I
don't think many Linux storage solution go that high, and NetApp will upgrade to 3TB and higher for one filer very soon.
I can easily go that high with that Mylex controller (or multiple controllers). One thing I do like about the NetApp that most RAID controllers don't seem to have is RAID-4 support.
- Performance - CIFS performance should be much higher, NFS also much
higher.
I could care less about CIFS performance (which is 10% of my performance), NFS is what I care about.
- NFS/CIFS securuty - Linux can only share to PCs with Samba, which is
fine, but only a user-level application (slower that Data ONTAP's kernel level implementation),
Microsoft makes the same argument against Samba as well, of course Microsoft's CIFS crashes every couple weeks. As long as OnTap doesn't crash, then that's interesting to note.
and also the filesystem on Linux will still be Unix-based, and will not be able to maintain NT ACLs etc.
So I can assign multiple users/groups to a file in NetApp's CIFS support? Does it support all NT permissions? I can modify NT ACLs with Samba, but only for the user/group/other (Samba maps NT ACLs to UNIX equivalents).
I hope it sums it up somehow,
Yes, thanx for info on CIFS ACLs.
-- TheBS
-- Bryan "TheBS" Smith CONTACT INFO *********************************************************** Chat: thebs413 @ AOL/MSN/Yahoo (see http://Everybuddy.com) Email: mailto:thebs@theseus.com,b.j.smith@ieee.org Home: http://www.SmithConcepts.com
The Mylex eXtremeRAIDs are very nice, and can resize the volumes on the fly. But resizing Ext2/3 is a pain, I know, because it is a traditional filesystem with pre-laid Inodes. I like the NetApp way with Inode creation on the fly.
Actually, it doesn't do it on the fly per se, it can just put file anywhere. You still have to monitor inode usage and increase the number of files when needed (just a simple command).
Multiple PCI channel and memory on x86 is getting mighty cheap. In fact, some people have commented that the Alpha in the NetApp gets saturated with duties under high-load conditions.
This is just a red herring. What doesn't get saturated under high-load conditions?? That's what high-load means!!
It doesn't matter how many PCI channels you have; if you overload your Linux server it'll get slow too. If they are overloading their Netapp, they need a faster Netapp (or another one).
and also the filesystem on Linux will still be Unix-based, and will not be able to maintain NT ACLs etc.
So I can assign multiple users/groups to a file in NetApp's CIFS support?
Yes.
Does it support all NT permissions?
Yes.
I can modify NT ACLs with Samba, but only for the user/group/other (Samba maps NT ACLs to UNIX equivalents).
Netapp also provides mapping capabilities.
Bruce
Bruce Sterling Woodcock wrote:
Actually, it doesn't do it on the fly per se, it can just put file anywhere. You still have to monitor inode usage and increase the number of files when needed (just a simple command).
I'm talking about the RAID volume, not the Linux filesystem. Many RAID controllers actually cannot resize their own volumes. Or were you talking about the NetApp?
This is just a red herring. What doesn't get saturated under high-load conditions?? That's what high-load means!!
;->
It doesn't matter how many PCI channels you have; if you overload your Linux server it'll get slow too. If they are overloading their Netapp, they need a faster Netapp (or another one).
Well, putting a $100 mainboard into a $20K server like the consultants before me did was just moronic. Whenever Linux flushes it's 1GB of memory, it just kills the PCI bus on which both the network and disk controller are on.
TheBS> So I can assign multiple users/groups to a file in TheBS> NetApp's CIFS support?
Yes.
TheBS> Does it support all NT permissions?
Yes.
Cool! I take it I have to have a full PDC to do that, eh?
I know there is an on-going effort to get Linux to do ACLs. At the same time, the Samba guys are trying to map Windows ACLs to those UNIX platforms that have their own ACL support.
Netapp also provides mapping capabilities.
Again, very cool!
-- TheBS
From: "Bryan J. Smith" b.j.smith@ieee.org
I'm talking about the RAID volume, not the Linux filesystem. Many RAID controllers actually cannot resize their own volumes. Or were you talking about the NetApp?
I was talking about Netapp. It's no big deal, just something to be aware of; if you start to run low on inodes you can just make more.
TheBS> So I can assign multiple users/groups to a file in TheBS> NetApp's CIFS support?
Yes.
TheBS> Does it support all NT permissions?
Yes.
Cool! I take it I have to have a full PDC to do that, eh?
Right. Well, at least to do some of it. I'm not sure how the filer handles a simple Workgroup case.
Bruce
Eyal Traitel etraitel@yahoo.com writes: [...]
- Crash recovery - filer's fsck (wack) is much much faster than other
vendors' solutions - >~600GB/hr and more.
The point about "wack" is that you will probably never have to run it. The checkpoints keep the filing system self-consistent, even if you have the unusual experience of losing the NVRAM contents after a power failure.
This is probably just as well, as my memories of "wack" are that it was infuriatingly slow! But that was several years ago (FASware 2.x) and on an FAServer 450. I think I came to the conclusion that the main problem was that it was reading the directories in a random order rather than optimising head movement.
"wack" isn't in the regular filer command set. These days it is hidden behind "rc_toggle_basic"; if I remember correctly, in 1995 it was "22/7" that was the magic word one had to use!
Chris Thompson University of Cambridge Computing Service, Email: cet1@ucs.cam.ac.uk New Museums Site, Cambridge CB2 3QG, Phone: +44 1223 334715 United Kingdom.
-
Bruce Sterling Woodcock -
Bryan -TheBS- Smith -
Bryan J. Smith -
Chris Thompson -
Eyal Traitel