Homestead.com has 8 F760 filers. We have just installed a Celerra with 7 datamovers attached to a Symmetrix 3930 (3 raw terabytes, 1 TB effective). While both are NAS solutions, the approaches are very different. How they are evaluated must be done from the perspective of the specific environment they are used in. In ours, both perform the same duty - we have roughly 70 HTTP servers (MS IIS 4.0) on Compaq Proliant servers (NT 4.0 SP6a), a few Linux and FreeBSD servers and some non-HTTP servers using either CIFS or NFS. The mix is about 65-70% CIFS and the rest NFS. Each "client" server uses UNC pathing to find the specific volume\directory\file, or to create the same.
Performance - too early to compare, but initial results show parity of response. I have no hard data other than the individual read/write tests we have done that show little significant variance.
Administration - big difference. The netapp is very easy to setup and install and monitor. the combined Celerra/Symmetrix environment is not. Let's just say that I am not the most technical guy in the world and I setup 2 F760's, cross-connected for Snapmirror, with all the RAID definitions, etc., by myself in one afternoon. I would not even know where to begin doing this in the EMC environment. On the other had, for the price difference, I expect EMC to get that done and get it done correctly.
Support - here again the 2 approaches differ greatly. The EMC platform comes with premium service - you get no choice, you only get top-of-line, phone-home, on site in 2 hours. netapp, even though we have premium support, does not have the same level of system-to-support automatic ties (autosupport really is not the same). being located very close to netapp has mitigated any issue this might cause and we have nothing but great support from them.
Why did Homestead.com choose to buy EMC at this point? The main reason is that Network Appliance's next generation is not yet ready, so the ability to scale based on current products was not optimal for us. Also, we wanted a RAID 1+ solution - WAFL is RAID 4 (and very good, just not as tolerant of disk failures). Lastly, I could use parts of the Symmetrix for direct attach (SAN) while having the other for NAS.
Sam Schorr Homestead.com ph: (650) 549-3152 fax: (650) 364-7329 sschorr@homestead-inc.com
Homestead: It's a new world. Make your place.
-----Original Message----- From: Gilles TOURPE [mailto:gilles@bnpcn.com] Sent: Monday, April 03, 2000 5:28 AM To: toasters@mathworks. com Subject: EMC Celerra vs NetApp Filer
Hello,
Did anybody get in the same site,
an EMC Celerra system and a NETAPP F7xx ( at least 740 ) Filer ?
I would be interrested in Performances, administration, supports issues ...
I'm very curious about the Celerra solution. On the paper it looks smart and really really NAS-oriented ( quite funny for SAN-guys :-) ). Some tests we did we pretty concluant also ( and complex also ... ).
Any thinking, remarks ????
Thanx
Gilles
------------------------------------------------------------ Gilles TOURPE - System and Network Administrator APS BNP ARBITRAGE - Salle des marches ACTIONS - PARIS - France Phone : (33)-1-40-14-24-03 e-mail : gilles@bnpcn.com
Why did Homestead.com choose to buy EMC at this point? The main reason is that Network Appliance's next generation is not yet ready, so the ability
to
scale based on current products was not optimal for us.
What do you mean by this? Not enough total storage? Just buy a second Netapp, since they are as easy as you note to set up and administer, this shouldn't be prohibitive. Plus, if you are simply thinking about raw storage capacity, this won't help you; the Celerra doesn't have the CPU to support an infinite amount of disk, and you'll have to get another one just like you would have a Netapp.
Also, we wanted a RAID 1+ solution - WAFL is RAID 4 (and very good, just not as tolerant of disk failures).
Hmm. Maybe I'm confused, but isn't RAID 1+0 *less* tolerant of disk failures, not more? Both can tolerate a single disk failure. RAID 1+0 has essentially twice as many disks, so the chance of a single disk failure is twice as high. Once that happens, the chance of a second disk failure in the same array is about the same was the Netapp. (That's assuming one RAID group).
And what's the price difference for a Celerra, and a Symmetrix, with *3 times* as much disk as you need for the Netapp model? Can't you buy 2 Netapps for that price?
Bruce
On Mon, 3 Apr 2000, Bruce Sterling Woodcock wrote:
What do you mean by this? Not enough total storage?
Might be a CPU limitation. Every single one of my Netapps will run out of CPU cycles long before they run out of disk. A clustered pair of the F740's routinely burst over 10000 NFS ops/sec each, but they only need a couple shelves of 18GB drives between them. With an EMC Celerra, you can keep the same pool of drives, but plug in more front-end processors to soak up the NFS load. I believe they support up to 14 of those, sharing the same array of drives.
Hmm. Maybe I'm confused, but isn't RAID 1+0 *less* tolerant of disk failures, not more? Both can tolerate a single disk failure.
RAID 1+0 (striped mirrors) may tolerate multi-drive failures. What I'd like to see is RAID 1+4 (mirrored data and parity drives, preferably with each half of the mirror on a separate FC-AL controller). I don't think any of the NAS vendors offer that yet, although you might be able to do that with Veritas. Anyone at Netapp working on that? :)
RAID 1+0 has essentially twice as many disks, so the chance of a single disk failure is twice as high. Once that happens, the chance of a second disk failure in the same array is about the same was the Netapp. (That's assuming one RAID group).
But since each stripe in a RAID 1+0 is on a mirrored pair of drives, you'd have to lose both drives for the RAID 0 part to fail. With N pairs of drives, the chance is only 1 in 2N-1 for the second failure to hit the other half of a broken mirror. Those are much better odds than the 1 in 1 chance that a double disk failure will cause a RAID 4 volume to die. ;-)
And what's the price difference for a Celerra, and a Symmetrix, with *3 times* as much disk as you need for the Netapp model? Can't you buy 2 Netapps for that price?
Well, thats's the kicker when you buy EMC. Not only do you pay a premium on hardware (yay for "RAID-S"), but the software and support as well. There are certain maintenance operations they will not allow you to perform yourself, and you must call in a field engineer. That goes against my philosophy of systems administration, but it works for some people.
----- Original Message ----- From: "Brian Tao" taob@risc.org To: "Bruce Sterling Woodcock" sirbruce@ix.netcom.com Cc: "Sam Schorr" sschorr@homestead-inc.com; "'Gilles TOURPE'" gilles@bnpcn.com; "toasters@mathworks. com" toasters@mathworks.com Sent: Monday, April 03, 2000 9:43 PM Subject: Re: EMC Celerra vs NetApp Filer
On Mon, 3 Apr 2000, Bruce Sterling Woodcock wrote:
What do you mean by this? Not enough total storage?
Might be a CPU limitation. Every single one of my Netapps willrun out of CPU cycles long before they run out of disk. A clustered pair of the F740's routinely burst over 10000 NFS ops/sec each, but they only need a couple shelves of 18GB drives between them. With an EMC Celerra, you can keep the same pool of drives, but plug in more front-end processors to soak up the NFS load. I believe they support up to 14 of those, sharing the same array of drives.
Strange, but I would think you could still get away with less disk per Netapp and just more Netapps and accomplish the same thing cheaper. But I guess EMC is giving you a good deal.
Hmm. Maybe I'm confused, but isn't RAID 1+0 *less* tolerant of disk failures, not more? Both can tolerate a single disk failure.
RAID 1+0 (striped mirrors) may tolerate multi-drive failures.
I don't see how? When one mirror loses a disk, the whole thing is lost; you switch over to the other mirror, which has n-1 disks as the RAID 4 array. So your initial chance of failure is 2n-1 and then n-1 for a second failure whereas RAID 4 is just n followed by n-1 for the second failure.
What I'd like to see is RAID 1+4 (mirrored data and parity drives, preferably with each half of the mirror on a separate FC-AL controller). I don't think any of the NAS vendors offer that yet, although you might be able to do that with Veritas. Anyone at Netapp working on that? :)
The big thing missing is a RAID 4 hardware controller. All the other vendors are relying on RAID hardware to do high-performance; that's why they don't offer RAID 4, and that's why Netapp will continue to be a better solution. :)
RAID 1+0 has essentially twice as many disks, so the chance of a single disk failure is twice as high. Once that happens, the chance of a second disk failure in the same array is about the same was the Netapp. (That's assuming one RAID group).
But since each stripe in a RAID 1+0 is on a mirrored pair ofdrives, you'd have to lose both drives for the RAID 0 part to fail. With N pairs of drives, the chance is only 1 in 2N-1 for the second failure to hit the other half of a broken mirror.
Oh, are the drives mirror images of each-other too? I didn't realize that. So if Drive 1 in Mirror A fails, breaking the mirror, and then Drive 2 in Mirror B fails, Mirror B will be smart enough to switch over to Drive 2 in Mirror A?
And what's the price difference for a Celerra, and a Symmetrix, with *3 times* as much disk as you need for the Netapp model? Can't you buy 2 Netapps for that price?
Well, thats's the kicker when you buy EMC. Not only do you pay apremium on hardware (yay for "RAID-S"), but the software and support as well. There are certain maintenance operations they will not allow you to perform yourself, and you must call in a field engineer. That goes against my philosophy of systems administration, but it works for some people.
Yes, I know about the support, but even so, is the support WORTH the extra you are paying for the hardware? What exactly is the price difference on what you evaluated?
Bruce
On Mon, 3 Apr 2000, Bruce Sterling Woodcock wrote:
Strange, but I would think you could still get away with less disk per Netapp and just more Netapps and accomplish the same thing cheaper. But I guess EMC is giving you a good deal.
9GB drives are no longer available, 18GB drives will probably be EOL'd this year, and 36GB drives are the current standard. If the performance "sweet spot" is still 14 drives, that's ~400GB per filer. For my particular needs, storage is essentially infinite... NFS ops is what I crave. Give me a quad Alpha filer with two shelves of 15000 RPM 18GB Cheetahs, please. :)
I don't see how? When one mirror loses a disk, the whole thing is lost; you switch over to the other mirror, which has n-1 disks as the RAID 4 array. So your initial chance of failure is 2n-1 and then n-1 for a second failure whereas RAID 4 is just n followed by n-1 for the second failure.
I think you're talking about RAID 0+1 (taking a RAID-0 set and mirroring it on another RAID-0 set, which is just silly). RAID 1+0 does the mirroring first, then the striping/concatenation:
+----+----+ +--------------+ +------------+ | 1A | 1B | | RAID-1 set 1 | | | +----+----+ +--------------+ | | | 2A | 2B | | RAID-1 set 2 | | | +----+----+ +--------------+ | | | 3A | 3B | | RAID-1 set 3 | | | +----+----+ +--------------+ | | | 4A | 4B | --> | RAID-1 set 4 | --> | RAID 0 | +----+----+ +--------------+ |(7 "drives")| | 5A | 5B | | RAID-1 set 5 | | | +----+----+ +--------------+ | | | 6A | 6B | | RAID-1 set 6 | | | +----+----+ +--------------+ | | | 7A | 7B | | RAID-1 set 7 | | | +----+----+ +--------------+ +------------+ [etc...]
The A's and B's are drives of a mirrored pair. You could lose, say, drives 1A, 2A, 4B, 5B and 7A and still have a functional RAID 0, because no single mirror is completely broken. You could lose the entire shelf containing the A drives and not have an outage (something from which today's Netapps cannot automatically recover, clustering or no clustering). Having mirrored pairs in RAID 4 (i.e., having mirrored data and parity drives) and the ability to concurrently rebuild multiple broken mirrors to hot spares would really give Netapp a leg up on other NAS vendors, IMHO.
Oh, are the drives mirror images of each-other too? I didn't realize that. So if Drive 1 in Mirror A fails, breaking the mirror, and then Drive 2 in Mirror B fails, Mirror B will be smart enough to switch over to Drive 2 in Mirror A?
As far as the RAID 0 is concerned, single drive failures within each mirrored pair does not result in that stripe being down, because the remaining half of the mirror is still online. If you lose, say, drives 4A and 4B in a RAID 1+0, then you're toast. In RAID 1+4 (or 1+5), you would still have the parity drive.
Yes, I know about the support, but even so, is the support WORTH the extra you are paying for the hardware? What exactly is the price difference on what you evaluated?
We inherited a Symmetrix 3430 (using it as direct-attach SCSI, not NAS). It was not configured in the most intelligent manner to begin with, and it takes a lot of careful planning and detail work to ensure you didn't build yourself a rickety house of cards. I don't know how much the company paid for the EMC originally, but you also have to factor in the costs for the Veritas VxVM/VxFS licenses to efficiently manage all those disk targets. All this extra complexity not only makes outages more difficult to prevent, but usually means repair times go way up as well, as you try to remember all the intricacies of your whiz-bang RAID configuration. I'd rather just have Netapps humming away in the background, and trust in their magic. ;-)
The A's and B's are drives of a mirrored pair. You could lose,say, drives 1A, 2A, 4B, 5B and 7A and still have a functional RAID 0, because no single mirror is completely broken.
Ahh, okay. So you are using three times as many drives. So your chance of a single disk failure is 3n-1, but the chance of a double disk failure in a single stripe is... well, I'm not gonna sit here and calculate it out. I think in the end you might end up with better data protection (although not as much as one would think since your chance of failure is much higher), but it's a heavy premium. I always thought if any drive in the stripe set was down it would switch completely to the other stripe set, so that's good to know.
[more stuff snipped]
Since we don't know how much all the EMC and related stuff cost, it's hard to accurately compare the two solutions. I bet it is much higher, and you said you would prefer the Netapps, which is good because you were sound failry pro-Celerra earlier, but it turns out you were comparing apples and oranges.
Personally I don't mind who prefers what just so long as all the facts are on the table.
Bruce
"sirbruce" == Bruce Sterling Woodcock sirbruce@ix.netcom.com writes:
sirbruce> Since we don't know how much all the EMC and related stuff sirbruce> cost, it's hard to accurately compare the two solutions. I sirbruce> bet it is much higher, and you said you would prefer the sirbruce> Netapps, which is good because you were sound failry sirbruce> pro-Celerra earlier, but it turns out you were comparing sirbruce> apples and oranges.
It's difficult to actually compare apples to apples when talking about emc and netapp... they both like to tweak things a little differently. Once you beat them about the head and neck enough, they cooperate. :-)
Cost-wise... eh. Going by list prices, EMC is a truckload more expensive. Discounts will vary the difference *GREATLY*. I just can't put enough emphasis on that last bit... At some point, depending on current pricing, EMC does become cheaper than NetApp. Half of our setup is disk intensive and doesn't require that much CPU. The other have is the opposite - we're melting a pair of 740s right now. Going by units of "1 cpu, 1gb cache, 200gb disk", EMC comes in cheaper after a small number of units. For us. YMWV.
sirbruce> Personally I don't mind who prefers what just so long as all sirbruce> the facts are on the table.
It's surprisingly difficult at times to get the facts out of them. We're at the end of 8 months of vendor bashingnegotiations and meetings.
K.
Cost-wise... eh. Going by list prices, EMC is a truckload more expensive. Discounts will vary the difference *GREATLY*. I just can't put enough emphasis on that last bit... At some point, depending on current pricing, EMC does become cheaper than NetApp. Half of our setup is disk intensive and doesn't require that much CPU. The other have is the opposite - we're melting a pair of 740s right now. Going by units of "1 cpu, 1gb cache, 200gb disk", EMC comes in cheaper after a small number of units. For us. YMWV.
But these wouldn't be units that would be equal comparison. What matters is primarily ops/$ and GB/$ for roughly the same levels of data protection. There are other factors (administration, features, etc.) but these are harder to measure objectively. I'm not doubting what you say; I'm just annoyed customers are comparing the two and yet don't appear to have numbers on this. Having more cpus per $ doesn't matter much if they need those extra cpus to handle the load.
Bruce
On Mon, 3 Apr 2000, Bruce Sterling Woodcock wrote:
Ahh, okay. So you are using three times as many drives.
Compared to what? Assuming 10GB usable space per disk, 10 drives in a RAID 0 configuration gives you 100GB. Both RAID 1+0 and 0+1 would need 20 drives to reach 100GB. RAID 4 and 5 would need 11 drives.
So your chance of a single disk failure is 3n-1, but the chance of a double disk failure in a single stripe is... well, I'm not gonna sit here and calculate it out.
In RAID 1+0, the chance that the next disk failure will result in the catastrophic failure of the entire RAID is F/(D-F), where F is the number of already-failed disks, and D is the total number of disks in the RAID. Thus with 14 drives in a RAID 1+0 configuration:
1st failure = 0/(14-0) = 0.0% chance of broken RAID 2nd failure = 1/(14-1) = 7.7% " " " " 3rd failure = 2/(14-2) = 16.6% " " " " 4th failure = 3/(14-3) = 27.3% " " " " 5th failure = 4/(14-4) = 40.0% " " " " 6th failure = 5/(14-5) = 55.6% " " " " 7th failure = 6/(14-6) = 75.0% " " " " 8th failure = 7/(14-7) = 100.0% " " " "
As expected, losing the first drive never results in RAID failure, and if you're really lucky, you can lose up to half the drives (as others have mentioned) and still keep humming along. Unlike RAID 4, having more drives in RAID 1+0 helps you withstand more simultaneous drive failures. There's probably a point where the diminishing MTBF on a large pool of drives catches up to you though, but I don't know when that happens.
I think in the end you might end up with better data protection (although not as much as one would think since your chance of failure is much higher), but it's a heavy premium.
True, these numbers don't translate directly into availability, since you now have almost twice as many drives to worry about, compared to a RAID 4 of similar capacity. Still, this is better than the 100% chance of a RAID 4 dying with 2 or more broken disks.
Since we don't know how much all the EMC and related stuff cost, it's hard to accurately compare the two solutions. I bet it is much higher, and you said you would prefer the Netapps, which is good because you were sound failry pro-Celerra earlier, but it turns out you were comparing apples and oranges.
Naw, I wasn't trying to compare the two, but rather to suggest a reason why someone might have said that an EMC Celerra is "more scalable" than a Netapp. For my needs, I sure as heck wouldn't trade any of my filers in for an EMC.
On Mon, 3 Apr 2000, Bruce Sterling Woodcock wrote:
Ahh, okay. So you are using three times as many drives.
Compared to what?
RAID 4.
Assuming 10GB usable space per disk, 10 drives in a RAID 0 configuration gives you 100GB. Both RAID 1+0 and 0+1 would need 20 drives to reach 100GB. RAID 4 and 5 would need 11 drives.
The diagram I saw was misleading. You're right; you use the space on both "sides" of the RAID 1 stripe so really it is twice as much disk space.
As expected, losing the first drive never results in RAID failure, and if you're really lucky, you can lose up to half the drives (as others have mentioned) and still keep humming along. Unlike RAID 4, having more drives in RAID 1+0 helps you withstand more simultaneous drive failures.
No it doesn't, because your initial chance of the first failure is twice as high. Your figures aren't very helpful; what you need is to use the MTBF to find out what your real chance is for a given number of drives.
There's probably a point where the diminishing MTBF on a large pool of drives catches up to you though, but I don't know when that happens.
Right. That's what I'm trying to determine.
I think in the end you might end up with better data protection (although not as much as one would think since your chance of failure is much higher), but it's a heavy premium.
True, these numbers don't translate directly into availability,since you now have almost twice as many drives to worry about, compared to a RAID 4 of similar capacity. Still, this is better than the 100% chance of a RAID 4 dying with 2 or more broken disks.
I don't think that's necessarily true. But it's possible to create a semi RAID 4+1 configuration by using SnapMirror to regularly mirror one filer onto another. Given what you say, I expect NTAP should offer a RAID 4+1 configuration in the future. It would seem relatively easy to implement on a single filer.
Bruce
True, these numbers don't translate directly into availability,since you now have almost twice as many drives to worry about, compared to a RAID 4 of similar capacity. Still, this is better than the 100% chance of a RAID 4 dying with 2 or more broken disks.
It's a somewhat minor nit, but I feel obliged to point out that while NetApp uses RAID 4, and a RAID 4 set will croak if it suffers a double disk failure, it's not necessarily true that a double disk failure will take out a filer. A sufficiently large volume on a filer will be RAID 4+0 (I think I got the notation right -- it's a concatenation of several RAID 4 group), and you can have multiple volumes. Therefore, you can have multiple disk failures on a filer without losing data so long as there isn't more than one failure in a single RAID 4 group. You could even configure your volumes with a single data disk per RAID group and end up with the same data integrity as RAID 1+0, though we don't optimize for this case so performance may suffer.
Anyone who is interested enough to still be reading this thread might want to read an old tech report of mine, especially section 3:
http://www.netapp.com/tech_library/3027.html
Some of the numbers are dated (they're based on an F630 and our first generation of 9 GB FC-AL disks) and a few pieces of section 3.4 are obsolete, but the basic ideas are still good.
-- Karl Swartz Network Appliance Engineering Work: kls@netapp.com http://www.netapp.com/ Home: kls@chicago.com http://www.chicago.com/~kls/
In message 200004060020.RAA26503@turok.netapp.com, Karl Swartz writes:
It's a somewhat minor nit, but I feel obliged to point out that while NetApp uses RAID 4, and a RAID 4 set will croak if it suffers a double disk failure, it's not necessarily true that a double disk failure will take out a filer.
As someone who has lost 3 disks at the same time (in 3 different raid groups) and not lost a filer, I agree :P
Mark
"Karl" == Karl Swartz kls@netapp.com writes:
>> True, these numbers don't translate directly into availability, >> since you now have almost twice as many drives to worry about, >> compared to a RAID 4 of similar capacity. Still, this is better than >> the 100% chance of a RAID 4 dying with 2 or more broken disks.
Karl> It's a somewhat minor nit, but I feel obliged to point out Karl> that while NetApp uses RAID 4, and a RAID 4 set will croak Karl> if it suffers a double disk failure, it's not necessarily Karl> true that a double disk failure will take out a filer. A Karl> sufficiently large volume on a filer will be RAID 4+0 (I Karl> think I got the notation right -- it's a concatenation of Karl> several RAID 4 group), and you can have multiple volumes.
Actually, concatenation is not defined as one of the raid levels. RAID 0 is striping. So if the NetApp were striping a volume across the RAID 4 groups, then it would be RAID 4+0 (or RAID 0+4). And if you striped multiple volumes across multiple RAID 4 groups, then you'd have something like EMC's RAID-S with Hypervolumes, except using RAID 4 instead of RAID 5. Phew, I believe this thread has now come full-circle and may safely die.
j.
On an EMC if a disk goes out on one side of a mirror (we'll call it side A), side A will be offline (all of the disks in side A, not just the bad drive). Therefore, any new writes will be seen on side B but not on the drives of side A and you can't use side A drives to recover from subsequent side B drive failures. My EMC SE said that only if both sides of the mirror have simultaneous drive failures taking them both down at the same time is there a chance of recovering and that it would take a lot of work. (this assumes that the bad drives weren't mirrors of each other)
Brian Tao wrote:
On Mon, 3 Apr 2000, Bruce Sterling Woodcock wrote:
I don't see how? When one mirror loses a disk, the whole thing is lost; you switch over to the other mirror, which has n-1 disks as the RAID 4 array. So your initial chance of failure is 2n-1 and then n-1 for a second failure whereas RAID 4 is just n followed by n-1 for the second failure
To be somewhat anal about it, the initial chance of failure of mirrored partitions is 2(n-1). :)
But you're right, you have a higher probability of being put at risk using a mirror (just needs 1 disk going bad out of 2n-2 disks) than with RAID4 (1 disk bad out of n disks). Once that 1st disk is lost the probabilities of a second failure are the same for both (n-1).
If EMC had actually implemented their mirroring like Brian Tao mentions below then their mirroring would be much more reliable than RAID4.
I think you're talking about RAID 0+1 (taking a RAID-0 set andmirroring it on another RAID-0 set, which is just silly). RAID 1+0 does the mirroring first, then the striping/concatenation:
+----+----+ +--------------+ +------------+ | 1A | 1B | | RAID-1 set 1 | | | +----+----+ +--------------+ | | | 2A | 2B | | RAID-1 set 2 | | | +----+----+ +--------------+ | | | 3A | 3B | | RAID-1 set 3 | | | +----+----+ +--------------+ | | | 4A | 4B | --> | RAID-1 set 4 | --> | RAID 0 | +----+----+ +--------------+ |(7 "drives")| | 5A | 5B | | RAID-1 set 5 | | | +----+----+ +--------------+ | | | 6A | 6B | | RAID-1 set 6 | | | +----+----+ +--------------+ | | | 7A | 7B | | RAID-1 set 7 | | | +----+----+ +--------------+ +------------+ [etc...]
The A's and B's are drives of a mirrored pair. You could lose,say, drives 1A, 2A, 4B, 5B and 7A and still have a functional RAID 0, because no single mirror is completely broken. You could lose the entire shelf containing the A drives and not have an outage (something from which today's Netapps cannot automatically recover, clustering or no clustering). Having mirrored pairs in RAID 4 (i.e., having mirrored data and parity drives) and the ability to concurrently rebuild multiple broken mirrors to hot spares would really give Netapp a leg up on other NAS vendors, IMHO.
Oh, are the drives mirror images of each-other too? I didn't realize that. So if Drive 1 in Mirror A fails, breaking the mirror, and then Drive 2 in Mirror B fails, Mirror B will be smart enough to switch over to Drive 2 in Mirror A?
As far as the RAID 0 is concerned, single drive failures withineach mirrored pair does not result in that stripe being down, because the remaining half of the mirror is still online. If you lose, say, drives 4A and 4B in a RAID 1+0, then you're toast. In RAID 1+4 (or 1+5), you would still have the parity drive.
-- Brian Tao (BT300, taob@risc.org) "Though this be madness, yet there is method in't"
"Steve" == Steve Gremban gremban@msp.sc.ti.com writes:
Steve> On an EMC if a disk goes out on one side of a mirror (we'll Steve> call it side A), side A will be offline (all of the disks Steve> in side A, not just the bad drive). Therefore, any new Steve> writes will be seen on side B but not on the drives of side Steve> A and you can't use side A drives to recover from Steve> subsequent side B drive failures. My EMC SE said that only Steve> if both sides of the mirror have simultaneous drive Steve> failures taking them both down at the same time is there a Steve> chance of recovering and that it would take a lot of Steve> work. (this assumes that the bad drives weren't mirrors of Steve> each other)
There are two ways of combining RAID 0 and RAID 1. I call these RAID 0+1 and RAID 1+0, though I've seen people describe what I'd call RAID 0+1 and call it RAID 1+0. But technically, they are different things and have to do with the order in which the operations are applied.
If you mirror all your disk pairs first, and then you stripe across the pairs, you have RAID 1+0. In this case, you can lose up to 50% of your disks, as long as you don't lose both members of a pair.
If you split your disks into two sets, stripe each set, then mirror the two stripe sets, you have RAID 0+1. In this case, losing one disk in a stripe set brings that whole set of disks offline. Losing a disk in the opposite stripe set will in this case fail the entire array.
So, RAID 0+1 = mirror -> stripe -> disk RAID 1+0 = stripe -> mirror -> disk
Your odds of a double-disk failure taking out the array on RAID 0+1 go way up as compared to RAID 1+0 for more than 4 disks. The only advantage I know of using 0+1 is that it allows you to combine disks of different sizes into equal size stripe sets which can then be mirrored.
AFAIK, Solaris Disk Suite 4.2 (maybe earlier) provides 0+1 functionality. That is, you can yank half your disks from your array, as long as you don't yank any two disks of the same mirrored pair. This has been demonstrated empirically; I haven't found SDS documentation that actually says SDS supports this feature.
EMC's RAID-S is a specialized RAID-5 implementation that offloads the XOR parity calculations to the drives (among other things, see: http://esdis-it.gsfc.nasa.gov/MSST/conf1996/C4_3Quinn.html).
What does this have to do with Netapp? :)
j.
"Jay" == Jay Soffian jay@loudcloud.com writes:
Jay> AFAIK, Solaris Disk Suite 4.2 (maybe earlier) provides 0+1 ^^^
Typo. SDS provides 1+0, as the rest of the paragraph explains:
Jay> functionality. That is, you can yank half your disks from your array, Jay> as long as you don't yank any two disks of the same mirrored Jay> pair. This has been demonstrated empirically; I haven't found SDS Jay> documentation that actually says SDS supports this feature.
j.
On Tue, 4 Apr 2000, Jay Soffian wrote:
"Jay" == Jay Soffian jay@loudcloud.com writes:
Jay> AFAIK, Solaris Disk Suite 4.2 (maybe earlier) provides 0+1 ^^^Typo. SDS provides 1+0, as the rest of the paragraph explains:
Does it? In both scenarios you can yank half of your drives out as long as you don't yank any two disks of the same mirrored pair. In 1+0 this is achieved by yanking any disk whose mirror has not already been yanked. In 0+1 this is achieved by yanking any disks from one side of the mirror. In both cases you can loose at most 50% of the drives without data loss. As far as I recall ODS does not support 1+0.
Jay> functionality. That is, you can yank half your disks from your array, Jay> as long as you don't yank any two disks of the same mirrored Jay> pair. This has been demonstrated empirically; I haven't found SDS Jay> documentation that actually says SDS supports this feature.j.
"tkaczma" == tkaczma tkaczma@gryf.net writes:
tkaczma> On Tue, 4 Apr 2000, Jay Soffian wrote:
>> "Jay" == Jay Soffian jay@loudcloud.com writes: >> Jay> AFAIK, Solaris Disk Suite 4.2 (maybe earlier) provides 0+1 >> ^^^ >> >> Typo. SDS provides 1+0, as the rest of the paragraph explains:
tkaczma> Does it? In both scenarios you can yank half of your tkaczma> drives out as long as you don't yank any two disks of the tkaczma> same mirrored pair. In 1+0 this is achieved by yanking tkaczma> any disk whose mirror has not already been yanked. In tkaczma> 0+1 this is achieved by yanking any disks from one side tkaczma> of the mirror. In both cases you can loose at most 50% tkaczma> of the drives without data loss. As far as I recall ODS tkaczma> does not support 1+0.
I have not personally confirmed it, but another SA here confirms for me that it support 1+0. In other words, he yanked one drive of each pair, selecting half from one sub-mirror, the other half from the other sub-mirror. That is 1+0. This was with SDS 4.2.
However, since I hate relying on hearsay, I will repeat this experiment on an E450 as soon as I round up an hour to spare.
j.
On Thu, 6 Apr 2000, Jay Soffian wrote:
I have not personally confirmed it, but another SA here confirms for me that it support 1+0. In other words, he yanked one drive of each pair, selecting half from one sub-mirror, the other half from the other sub-mirror. That is 1+0. This was with SDS 4.2.
I doubt it as one needs physical slices to create stripes, but metadevices to create mirrors. Thus ODS's mirroring can be further qualified as RAID 0+1, always. If you find otherwise please let me know how to do it. It'd be great.
Tom
tkaczma@gryf.net wrote:
On Thu, 6 Apr 2000, Jay Soffian wrote:
I have not personally confirmed it, but another SA here confirms for me that it support 1+0. In other words, he yanked one drive of each pair, selecting half from one sub-mirror, the other half from the other sub-mirror. That is 1+0. This was with SDS 4.2.
I doubt it as one needs physical slices to create stripes, but metadevices to create mirrors. Thus ODS's mirroring can be further qualified as RAID 0+1, always. If you find otherwise please let me know how to do it. It'd be great.
You are both right and wrong, SDS recognises 0+1 as a special case and tries to make it a 1+0 if it can. AFAIK ODS was only ever a Solaris 1 product and discontinued years ago, I doubt that it had the 1+0 functionality.
Another possibility for Sun machines is the SRC/P, a rebadged DPT PCI RAID controller which has 1+0 and 5+0 support. These are very fast, unfortunately Sun maintains that there are som quality issues so they are on hold right now.
/Michael
tkaczma@gryf.net wrote:
On Fri, 7 Apr 2000, Michael Salmon wrote:
You are both right and wrong, SDS recognises 0+1 as a special case and tries to make it a 1+0 if it can.
Cool, can you tell me where this behavior is documented?
In the Disksuite 4.2 Answerbook: Reference Guide: Metadevices: Mirrors
It doesn't say 1+0 but it does state that:
DiskSuite cannot guarantee that a mirror will be able to tolerate multiple slice failures and continue operating. However, depending on the mirror's configuration, in many instances DiskSuite can handle a multiple-slice failure scenario. As long as multiple slice failures within a mirror do not contain the same logical blocks, the mirror continues to operate. (The submirrors must also be identically constructed.)
/Michael
On Tue, 4 Apr 2000, Jay Soffian wrote:
So, RAID 0+1 = mirror -> stripe -> disk RAID 1+0 = stripe -> mirror -> disk
Yep, that's how I think of it.
The only advantage I know of using 0+1 is that it allows you to combine disks of different sizes into equal size stripe sets which can then be mirrored.
Ah, I never considered this scenario. You'd have to do some tricky command queue and rate throttling if your stripe sits on two drives of wildly differing performance characteristics!
EMC's RAID-S is a specialized RAID-5 implementation that offloads the XOR parity calculations to the drives (among other things, see: http://esdis-it.gsfc.nasa.gov/MSST/conf1996/C4_3Quinn.html).
Good link.
What does this have to do with Netapp? :)
Who cares? :)
The only advantage I know of using 0+1 is that it allows you to combine disks of different sizes into equal size stripe sets which can then be mirrored.
Ah, I never considered this scenario. You'd have to do sometricky command queue and rate throttling if your stripe sits on two drives of wildly differing performance characteristics!
NTAP supports drives of differing sizes with in the same RAID group in RAID 4, although it has recently been suggested that this could hurt performance more than previously thought. It would be nice if they could release a white paper testing and quantifying this.
Bruce
On Mon, 3 Apr 2000, Bruce Sterling Woodcock wrote:
The big thing missing is a RAID 4 hardware controller. All the other vendors are relying on RAID hardware to do high-performance; that's why they don't offer RAID 4, and that's why Netapp will continue to be a better solution. :)
I thought at least the parity calculations in a Netapp were offloaded to an ASIC, so it doesn't tie up the general-purpose CPU with trivial calculations?
"Brian" == Brian Tao taob@risc.org writes:
Brian> I thought at least the parity calculations in a Netapp were Brian> offloaded to an ASIC, so it doesn't tie up the Brian> general-purpose CPU with trivial calculations?
Rebuilding in raid-4/5 isn't expensive due to the parity calculations. After all, the XOR operations takes few cycles on most CPU's. It the reading of the blocks from each disk and then the resulting write back to the rebuilding disk that are the expensive part of rebuilding under raid 4/5.
j.
Brian,
We have seen the same thing, we are running out of CPU cycles and we continue to add capacity to our LSF pool. I am very familiar with the NetApp product but have no experience with the EMC but we are talking to them in a lot.
My understanding that, yes you could add Celerra processing power to the front end, but are you not still limited by the capacity of the data movers on the back end? One may point several Celerra processors to a file system but there is still only a single data mover to a volume. Please help me if I misunderstand the EMC configuration.
-gdg
Brian Tao wrote:
On Mon, 3 Apr 2000, Bruce Sterling Woodcock wrote:
What do you mean by this? Not enough total storage?
Might be a CPU limitation. Every single one of my Netapps willrun out of CPU cycles long before they run out of disk. A clustered pair of the F740's routinely burst over 10000 NFS ops/sec each, but they only need a couple shelves of 18GB drives between them. With an EMC Celerra, you can keep the same pool of drives, but plug in more front-end processors to soak up the NFS load. I believe they support up to 14 of those, sharing the same array of drives.
-- --------------------------------------------------------------- G D Geen mailto:geen@ti.com Texas Instruments Phone : (972)480.7896 System Administrator FAX : (972)480.7676 --------------------------------------------------------------- Life is what happens while you're busy making other plans. -J. Lennon
Celerra is the name of the NAS product from EMC. It consists of from 1 to 14 data movers which are each kind of like a small filer without DataONTAP and WAFL. The entire collection of data movers is controlled from a "control station" located in the same cabinet as the data movers. Each data mover is connected to the Symmetrix storage subsystem via direct-attached SCSI interfaces and connected to the network via network interfaces.
There is no such thing as Celerra processing power in and of itself. Data movers may be configured within a Celerra for cluster failover, but the failover occurs to a hot standby data mover that is not otherwise being used to serve data. More than one data mover can access the same file system but only one of them has RW acess to that file system. The others would have RO access. In any event each data mover has only one volume (or file system) associated with it. There is no VIF support for NIC failover. There is no fast etherchannel. Gigabit ethernet and ATM are brand new technologies to EMC. Netapp has had both for over two years. There are no Snapshots on a Celerra. For multiprotocol support they use a version of Sambs that runs on top of their UxFS file system.
The EMC concept of scalability is also a bit self serving. The claim that they can scale to 9TB within a box sounds impressive but actually adds up to very little. It's easier to deploy 9 1TB filers than a single 9TB Symmetrix. Also, with netapp you can start at 1 or two and grow to 9 as you need. You don't have to buy this huge hulking giant of a storage subsystem and then grow within it. You can actually grow your storage as you need it. Also, when you hit the 9TB limit on a Sym you then have to buy another (plus another Celerra and more data movers to deploy it with.
I hope I don't sound too negative, but I know both products fairly well and I think filers are a far superior solution. But then again I work for Netapp don't I :-) Celerra usually only appeals to folks who have already invested huge sums of money in EMC Symmetrix boxes and now need a NAS solution that works with them to help justify the purchase of them in the first place. If you don't already have a Symmetrix the solution becomes quite expensive.
From: owner-dl-toasters@netapp.com [mailto:owner-dl-toasters@netapp.com]On Behalf Of G D Geen Sent: Friday, April 07, 2000 12:37 PM To: toasters@mathworks. com Subject: Re: EMC Celerra vs NetApp Filer
Brian,
We have seen the same thing, we are running out of CPU cycles and we continue to add capacity to our LSF pool. I am very familiar with the NetApp product but have no experience with the EMC but we are talking to them in a lot.
My understanding that, yes you could add Celerra processing power to the front end, but are you not still limited by the capacity of the data movers on the back end? One may point several Celerra processors to a file system but there is still only a single data mover to a volume. Please help me if I misunderstand the EMC configuration.
-gdg
Brian Tao wrote:
On Mon, 3 Apr 2000, Bruce Sterling Woodcock wrote:
What do you mean by this? Not enough total storage?
Might be a CPU limitation. Every single one of my Netapps willrun out of CPU cycles long before they run out of disk. A clustered pair of the F740's routinely burst over 10000 NFS ops/sec each, but they only need a couple shelves of 18GB drives between them. With an EMC Celerra, you can keep the same pool of drives, but plug in more front-end processors to soak up the NFS load. I believe they support up to 14 of those, sharing the same array of drives.
-- --------------------------------------------------------------- G D Geen mailto:geen@ti.com Texas Instruments Phone : (972)480.7896 System Administrator FAX : (972)480.7676 --------------------------------------------------------------- Life is what happens while you're busy making other plans. -J. Lennon
-
Brian Tao -
Bruce Sterling Woodcock -
fubar＠mathworks.com -
G D Geen -
Jay Soffian -
kls＠netapp.com -
Michael Salmon -
Paul W. Brosseau -
Sam Schorr -
Steve Gremban -
Stoltz -
tkaczma＠gryf.net