To answer the raid grouping...Much work has been to automagically assign disks properly.
If you have a brand new systems with lots of disks and create a new large aggregate, you *will* notice
that is is striped every which way it can. In other words, If I have two stacks, each RAID GROUP should
get evenly (or roughly evenly if there are odd numbers of disks) split between the stacks and on each stack,
the disks would get evenly split across the shelves, and at each shelf it will get split per row.
I recently did a new ONTAP8 install and had 4 stacks (two of 600GB SAS and two of 2TB SATA).
What I described happened like it should. If you have a clean system, you should not worry about balancing
as ONTAP will take care of that for you.
If I did a raid group of 14 for instance and I have 2 x 4243 shelves in each stack, I should see something like
stack1, shelf 1 and stack 2 shelf 1 with 4 disks each
stack1, shelf 2 and stack 2 shelf 2 with 3 disks each
On top of all that, ONTAP will talk to half of the disks on each shelf on the A port and the other half on the B port.
Everything is load balanced
--tmac
Tim McCarthy
Principal Consultant
RedHat Certified Engineer
804006984323821 (RHEL4)
805007643429572 (RHEL5)
On Fri, Nov 4, 2011 at 8:26 AM, Petar Forai
<pfo@pseudoterminal.org> wrote:
Hey Guys!
To piggyback on the RG size vs performance discussions...
Can anyone explain why the RG size as such is not _that_ dominant to aggregate (write) performance? My current understanding of WAFL with respect to writes is, that due to the write anywhere mantra it is (most of the time?) doing what on RAID-6 (in SNIA definitions) should be the equivalent of a full stride write.
If this is the case how come there is no requirement for the RG size ( N + P + Q) to be even? I'm sure I'm missing something here.
In the last discussions of RG size I've never seen anyone mention aggr/RG layout over SAS stacks/shelfs which seems a bitt odd to me. From the sequential read POV it should be crucial to balance back end SAS ports but no mentioning of that on any NetApp TR I've come across. There should be some kind of automatic SAS path load balancing going on (on a per disk basis?) automagically but from my current observations ONTAP doesn't care about disk selection/placement when it comes to aggr construction.
NetApp tells you[1] that optimal RG size shouldn't differ by more than +- 1 within an aggregate - does this also apply to large aggregates composed of say:
6 RGs of size 12 plus another 4 RGs each holding 13 disks?
On a somewhat related note, I've not come across any considerations of different SAS HBA capacities - just as an example:
A FAS3240A system that I'm currently looking at, with the IOXM option installed - each head has one dual-port on-board 6Gbps SAS HBA (ports 0a and 0b) and an additional _quad-port_ 6Gbps SAS HBA in the IOXM.
According to the platform documentation[2], in particular the platform block diagram, the IOXM is only 32xPCI-E 1.0 lanes switch connected to the PCI-E 2.0 (32x?) switch (that is connected to the PCI-E root complex) shared by the onboard SAS HBA and the FC HBA. So you have a quad port 6 Gbps HBA hanging on 8x PCI-1.0 in the IOXM which is shared somewhat with the and one dual port 6Gbps 8x PCI-E 2.0 HBA. How do you balance this - if at all :) ?
I'm asking because a client's system currently has 2x DS4324 populated with 48x 1TB drives and another 2x DS4324 with 48x 1TB drives has been purchased and installed so that each 2xDS4324 are on their own stack. The current configuration is one aggregate with RG size of 15 and 3 hot spares. The system is running 8.0.2P3.
According to [3] the maximum aggr size on pre 8.1 60 disks with rg size 17 make for the optimal layout of a single 1T aggr on a FAS3240A. For 8.1 I'd probably go with RG size 18 and 5 of them leaving 2 spares per controller or so.
As 8.1 is basically just around the corner I'd like to have aggr designed with the 8.1 limits in the back of my head.
I would probably go now with a new aggr from the new shelfs with RG size 17 do a aggr copy destroy the old one and add a few disks from the old one until 8.1 is running on the system.
Any opinions?
Thanks,
P
—
[1] TR3838 (NDA) or check [3] about most important content of TR3838
[2] FAS/V3200 Platform Report
[3] http://communities.netapp.com/message/52632#52632
On 03.11.2011, at 18:50, Adam Levin wrote:
>
> On Thu, 3 Nov 2011, Jeff Mohler wrote:
>> And at this level...were in violent agreement. :) :)
>
> Heh, indeed! :)
>
>> Ive never chastised (lightly or otherwise) a customer with RG's that dont
>> balance out perfectly...we discuss it, but if a disk purchase is not in
>> store to solve a perf issue Im consulting over, I dont suggest that capital
>> and effort be taken to resolve it at that time. It's rare, IMHO, to see
>> that overhead manifest itself in the user space. You have to exhaust the
>> CP engine to do it, then even so, you're likely still left with a sizing
>> challenge to resolve. (Legacy HW servicing new user loads, etc)
>
> I think that's a reasonable position to take.
>
>> Later on..a few disk adds and a few injections of reallocate to the patient
>> are in good order.
>
> Yup.
>
> -Adam
>
> _______________________________________________
> Toasters mailing list
> Toasters@teaparty.net
> http://www.teaparty.net/mailman/listinfo/toasters
Petar Forai
PGP-Key-Fingerprint: 4D15 F20B 6BB0 F68D 9580 2828 D17D BB4E 4DFF B82B
_______________________________________________
Toasters mailing list
Toasters@teaparty.net
http://www.teaparty.net/mailman/listinfo/toasters