Gentlemen, I need your opinion on a creation I am about to unleash on our unsuspecting production site where I work.
I currently have a 740 filer with 2 shelves and 7 disks.
3 disks are in the first shelf and configured for volume 0 root volume 4 disks are inthe second shelf and configured for volume 1
Now, since I didn't put this box together. I was wondering if it is better to make all of these disks one volume. There are heavy reads and writes to the volume1, but I don't know if having a heavily used volume combined with root functions will kill my performance. Besides this will make a larger volume for more distributed writes (8 total disks versus 4). Also I will be saving 1 disk of data space because I won't need the second parity disk.
Addition information is that they are fibre channel shelves (of course). And that the data is not important (washed away daily).
Assuming that the root volume functions do not critically affect my box (and I don't believe that they do, as most of it is running from memory) I believe my choice is correct.
Just wondering if you guys agree with this hypothesis or not.
Nicholas Pesce Internet Production Services Phone: 847 488-6384 Pager: 888 785 3455 Fax: 847 488 3434
On Mon, 11 Sep 2000, Pesce, Nicholas (FUSA) wrote:
Now, since I didn't put this box together. I was wondering if it is better to make all of these disks one volume. There are heavy reads and writes to the volume1, but I don't know if having a heavily used volume combined with root functions will kill my performance. Besides this will make a larger volume for more distributed writes (8 total disks versus 4). Also I will be saving 1 disk of data space because I won't need the second parity disk.
Search the NOW site for the NFS performance white paper. [0] The systems and numbers are dated (F330 with about 1,000 ops/s!) but Figure 3 "Performance Sensitivity to Number of Disk Drives" is pretty telling. A 14 disk RAID group (12d+1p+1hs) seems the way to go if you are concerned with performance.
[0] I don't have the document number, but the filename is nfsperf.pdf. I keep it handy at all times...
Until next time...
The Mathworks, Inc. 508-647-7000 x7792 3 Apple Hill Drive, Natick, MA 01760-2098 508-647-7001 FAX tmerrill@mathworks.com http://www.mathworks.com ---
Search the NOW site for the NFS performance white paper. [0] The systems and numbers are dated (F330 with about 1,000 ops/s!) but Figure 3 "Performance Sensitivity to Number of Disk Drives" is pretty telling. A 14 disk RAID group (12d+1p+1hs) seems the way to go if you are concerned with performance.
Well, that is somewhat outdated now, given that SCSI has pretty much gone the way of the dodo... which is curious, since Ultra160 (and soon Ultra320) would seem to be very competetive with the current generation of FibreChannel drives...
But to turn this thread on a slight tangent, I was curious about the performance advantages of spreading drives within a RAID group across multiple controllers. (Actually, I should probably just return my local SE's phone call, since I was discussing this with him a few weeks back and he said he had some new info... but heck, in the previous thread, three separate Netapp employees jumped on the response, and maybe y'all are interested too. :-) Given that More Disks Is Bettah, the question becomes whether or not it's worth the trouble (on a filer) to try to optimize the physical placement of those drives.
The last filer I helped manage (Hi, Marion!) is an F630 with six SCSI shelves, attached to three SCSI channels (on two dual-channel controllers, i.e., two PCI slots, obvious to anyone with a 630 :-). We have two volumes, one comprised of two RAID groups, and one comprising a single RAID group. Within each of the three RAID groups I meticulously balanced the assignment of physical drives across the three SCSI channels [1], following the hunch (er, traditional wisdom) that spreading things out like that was A Good Thing.
Now I'm in FibreChannelLand, playing with an F760 that I spec'ed with two FC-AL adapters and six FC-9 shelves, despite the fact that I only ordered 21 disks [2]. So, I figure if I'm configuring for performance it seems I should I split my RAID groups so that the drives are evenly distributed across the two loops. (If I'm configuring for fault tolerance, I'd want to keep each volume's RAID groups solely within one loop, so if that controller blew up or that loop was chopped to bits with an axe or a backhoe, I'd just lose one volume. Yeah, okay, so that's still not real pleasant a prospect, but it's just for argument's sake. :-)
But the word, apparently, is that write performance in ONTAP actually _degrades_ when a RAID group spans controllers, at least in FC-AL land. This is, as I understand it, due to the fact that the current batch of machines has a single PCI bus, and there's more overhead in spreading out I/O to multiple cards than in just blasting a bunch of bits over the bus to one card. I guess that makes some sense, if it's true. But it seems counterintuitive...
What I find surprising about the F760 is that I've been able to saturate the machine at around 35-38MB/sec sustained writes. (Hey! This ties in with the recent "gigabit ethernet performance" thread :-) I did some informal tests with both a single Sun E4500 (Sbus GbE card) and two Sun 420Rs (PCI GbE cards), running multiple "bonnie" or "dd" or "cpio" sessions to the two separate filer volumes. I was guessing that 32MB of NVRAM would be the bottleneck, but the CPU was pegged at 100%...
Still, 35MB/sec sustained writes ain't bad. :-) Our Oracle guy ran some benchmarks and found that the filer came within .1 seconds vs. a locally attached Sun A5200 (albeit a Disksuite volume...) But with a 600Mhz Alpha CPU, multiple gigabit FC-AL loops and a gigabit pipe into the box, all tied together with the much-touted PCI bus [4], I guess I just figured on higher numbers. I'm really curious to see what the next batch of new hardware offers. (And no, I love my toasters, and I'm not *really* complaining; you'll have to pry my 760 from my cold, dead fingers before I'd even THINK of going back to local disks. :-)
Cheers,
-- Chris
[1] As evidenced by the sticker neatly affixed to the front of each canister identifying to which RAID group and volume each drive belongs. If Netapp gives an annual award for "Most Anal-Retentive Sysadmin in a Comedy or Drama", that'd be me.
[2] Room for growth, baby! The idea is that I can buy 21 more drives over time, and plug them in as needed without downtime. If I'd bought three full shelves instead of six half-empty ones, I'd be bouncing the box every time I need to add one, which (I assume) is because these "Fibre Channel" shelves aren't actually using fiber optic cabling. Am I the only one that finds that kinda strange? And what's with these little grounding straps? IANAEE [3], but it seems to me that using "standard" GBIC slots on the shelves would give Netapp and customers the option to choose between FC-over-copper or FC-over-fiber, depending on their particular needs. And with nothin' but fiber between the head and the shelves wouldn't that eliminate the need for those silly grounding straps? Gads, I can't imagine the mess if/when it comes time to cluster this puppy. ("Sorry, I can't upgrade to a cluster, that tangle of cabling offends my aesthetic sensibilities." :-)
[3] I Am Not An Electrical Engineer, but I play one on the org chart. :-) Actually, all this newfangled FC-AL stuff seems pretty weird. The first machine I used with a "Winchester disk" in it used _SASI_ - the old Shugart interface. Ah, 1979, the bad old days...
[4] Imagine my surprise when in testing I found I was pushing nearly 80 MB/sec through my poor old Sbus GbE card, while the PCI cards - in 64-bit, 66Mhz PCI slots - topped out at 52... but that's just out-of-the-box, without tuning. Still, I'm a grumpy old Sun guy, and I'm still not convinced this PCI stuff is all that spiffy. :-P
-- Chris Lamb, Unix Guy MeasureCast, Inc. 503-241-1469 x247 skeezics@measurecast.com
----- Original Message ----- From: "Chris Lamb" skeezics@measurecast.com To: toasters@mathworks.com Sent: Monday, September 11, 2000 9:08 PM Subject: Re: 2 volumes or 1
Search the NOW site for the NFS performance white paper. [0] The systems and numbers are dated (F330 with about 1,000 ops/s!) but Figure 3 "Performance Sensitivity to Number of Disk Drives" is pretty telling. A 14 disk RAID group (12d+1p+1hs) seems the way to go if you are concerned with performance.
Well, that is somewhat outdated now, given that SCSI has pretty much gone the way of the dodo... which is curious, since Ultra160 (and soon Ultra320) would seem to be very competetive with the current generation of FibreChannel drives...
But to turn this thread on a slight tangent, I was curious about the performance advantages of spreading drives within a RAID group across multiple controllers. (Actually, I should probably just return my local SE's phone call, since I was discussing this with him a few weeks back and he said he had some new info... but heck, in the previous thread, three separate Netapp employees jumped on the response, and maybe y'all are interested too. :-) Given that More Disks Is Bettah, the question becomes whether or not it's worth the trouble (on a filer) to try to optimize the physical placement of those drives.
Worth the trouble? No.
However, if you can do it during initial setup, it is Bettah if one RAID group is all on one controller, not spread out like you did it. Which you later found out. (I'm not sure this is true of just FC land; it was probably true with SCSI as well.)
What I find surprising about the F760 is that I've been able to saturate the machine at around 35-38MB/sec sustained writes. (Hey! This ties in with the recent "gigabit ethernet performance" thread :-) I did some informal tests with both a single Sun E4500 (Sbus GbE card) and two Sun 420Rs (PCI GbE cards), running multiple "bonnie" or "dd" or "cpio" sessions to the two separate filer volumes. I was guessing that 32MB of NVRAM would be the bottleneck, but the CPU was pegged at 100%...
Things are highly tuned on Netapp servers so that all resources are used to the fullest extent possible. If you saw that CPU was only 70% with full writes (limited by 32MB NVRAM), you'd want to know why they weren't using the other 30% to improve response times. :)
[4] Imagine my surprise when in testing I found I was pushing nearly 80 MB/sec through my poor old Sbus GbE card, while the PCI cards - in 64-bit, 66Mhz PCI slots - topped out at 52... but that's just out-of-the-box, without tuning. Still, I'm a grumpy old Sun guy, and I'm still not convinced this PCI stuff is all that spiffy. :-P
It's not, but it's not *too* bad, and yet it's a standard everyone can agree on. Rambus ain't that spiffy either, but it looks like it might be in our future (I speak of computing in general, not filers specifically).
Bruce
On Mon, 11 Sep 2000, Bruce Sterling Woodcock wrote:
From: "Chris Lamb" skeezics@measurecast.com
But to turn this thread on a slight tangent, I was curious about the performance advantages of spreading drives within a RAID group across multiple controllers.
[...]
interested too. :-) Given that More Disks Is Bettah, the question becomes whether or not it's worth the trouble (on a filer) to try to optimize the physical placement of those drives.
Worth the trouble? No.
Worth the trouble? Yes IMHO. The NetApp 202 class notes state:
"A RAID group that spans two different controllers shows a 10% performance degradation."
At your next convenient reboot, halt the filer instead, shuffle the disks around, and boot back up if you care about the 10%.
Until next time...
The Mathworks, Inc. 508-647-7000 x7792 3 Apple Hill Drive, Natick, MA 01760-2098 508-647-7001 FAX tmerrill@mathworks.com http://www.mathworks.com ---
----- Original Message ----- From: "Todd C. Merrill" tmerrill@mathworks.com To: "Chris Lamb" skeezics@measurecast.com Cc: toasters@mathworks.com Sent: Tuesday, September 12, 2000 10:09 AM Subject: Re: 2 volumes or 1
On Mon, 11 Sep 2000, Bruce Sterling Woodcock wrote:
From: "Chris Lamb" skeezics@measurecast.com
But to turn this thread on a slight tangent, I was curious about the performance advantages of spreading drives within a RAID group across multiple controllers.
[...]
interested too. :-) Given that More Disks Is Bettah, the question
becomes
whether or not it's worth the trouble (on a filer) to try to optimize
the
physical placement of those drives.
Worth the trouble? No.
Worth the trouble? Yes IMHO. The NetApp 202 class notes state:
"A RAID group that spans two different controllers shows a 10% performance degradation."
We've been over this before. This note, as written, is grossly false. The penalty is only on writes, and only noticeable if you're already saturating your NVRAM such that you are writing all the time. There is no constant 10% performance penalty.
Bruce
-
Bruce Sterling Woodcock -
Chris Lamb -
Pesce, Nicholas (FUSA) -
Todd C. Merrill