Personally, I don't think it's worth it. Odds are if you loose one shelf, you will probably have a broken loop and you'll lose more than one shelf so I don't think you'll get as much redundancy as you think you're getting. Better to dual loop the shelves so you have multiple paths to the shelves. If you really need to protect yourself against losing a shelf, consider syncmirror and double your spindles/shelves. Expensive? Maybe, but that's the way to really get to where you are talking about. It just comes down to whether that level of redundancy is worth the extra cost. Sometimes it is, sometimes it isn't. There's no one right answer there.
Also, spares are global, so you have the extra fun of having the move disks around (or do multiple reconstructs) to get your disks back to the way you want them.
As far as one vs. multiple aggregates. It comes down to a few points:
1) The physical limit of an aggregate size. On most platforms (including yours) it's 16TB raw. On lower memory platforms it can be lower. 2) A need (real or perceived) to have certain data types on physically different spindles. A real need could be usage patterns (heavy workloads could chew up spidle ops). However you can also deal with this using FlexShare. A perceived need could be different groups purchased disks and want them exclusively, or maybe there's a mandate to keep certain types of data totally separate. 3. SyncMirror. SyncMirror'ing in the FlexVol world occurs at the aggragate level. Sometimes you may want some data syncmirrored and some not.
If these don't apply to you, then you will typically get the most flexibility (and usually the best utilization) with one big aggregate. But again, there's not one right answer.
Just one guy's opinion here.
-- Adam Fox adamfox@netapp.com
-----Original Message----- From: Paul Letta [mailto:letta@jlab.org] Sent: Friday, June 15, 2007 4:15 PM To: toasters@mathworks.com Subject: multiple shelf aggregate or single shelf aggregates
We have a new FAS3040 with 3 shelves of 500GB SATA's. It has an onboard controller with 4 ports, and we have 2 expansion controllers with 2 ports each.
I'm trying to think about the aggregate layouts.
I am thinking that since these are SATA drives, I should put all 3 shelves into 1 big aggregate for performance (to get as many spindles as possible) and then just use flex vols.
But then I also am thinking that if I use 3 separate aggregates (1 per shelf), I would have better data protection in case I loose a whole shelf (only loose volumes on that one shelf). Since the raid_size is maxed at 16 for SATA anyways, I am going to end up with a raid group per shelf no matter what (defaults to 14).
I've done some write tests. I had an aggregate made from 1 shelf and another aggregate made up of 2 shelves. I didn't see much difference. I was write testing via NFS mounts. The client was on a GigE, and the filer was on a VIF made up of all 4 GigE ports.
Is the performance gains from having mutli-shelf aggregates worth the loss of the data protection from a whole shelf failure.
As for the FC connections, currently I have all 3 shelves daisy chained with 2 loops, one loop is on the on board and the other loop is on one of the expansion cards.
Suggestions ?
Thanks !
Paul