agreed... for shelf resiliency, SyncMirror to a different stack is a no charge license... double the disk isn't, but for stack and shelf resiliency that might be a better option.

From: tmac tmacmd@gmail.com To: jordan slingerland jordan.slingerland@gmail.com Cc: Toasters toasters@teaparty.net Sent: Tuesday, June 21, 2016 8:26 AM Subject: Re: Raid disk layout - the ability to lose a shelf.

This can be crazy! You will end up with small raid groups eating up parity to sacrifice for what? Sacrificing space for what? Now, given a large environment (like 10 shelves or more)...maybe you can start with this. I did this for a customer once....ONCE We ended up with 16 or 18 disk raidgroups and there was no more than 2 per raidgroup per shelf.We took this one a bit farther too....all even numbered disks (*0,*2,*4,*6,*8) were assigned to node 2. The rest to node 1When a disk fails, assign even to node 2, odd to node 1 This made the aggregates a bit trickier to place, but it happened. Now, when a disk fails, I cannot control where it rebuilds other than a spare. I tried to keep the spares on one shelf thinking in the event of failures, they will likley be different raidgroups. However, one could script some monitoring software to wathc where the spares are and watch for more than 2 disks in a raidgroup showing in the same shelf. Then possibly forcing the running of a "disk copy start" command to nondisruptively move the disk. THIS TAKES LONGER than a reconstruction!!! Why? the process is NICE'd to use limited resources because it it not critical yet.

--tmac Tim McCarthy, Principal Consultant

On Tue, Jun 21, 2016 at 11:11 AM, jordan slingerland jordan.slingerland@gmail.com wrote:

Hello,

I am deploying a new 8040 and it was requested that the aggregates /  raid groups are laid out in such a way that no more than 2 disks in any raid group are within the same shelf.

At first this sounds like it reduces single points of failure and could protect availability from the failure of a full disk shelf.

I argue against this strategy and was wondering if anyone in this list had any feedback.

My thought is that this configuration is marginally increasing availability at the sacrifice of additional risk to data integrity.  With this strategy, each time a disk failed we would endure not only the initial rebuilt to spare, but a second rebuild when a disk replace is executed to put the original shelf/slot/disk back into the the active raid group.

Additional, if a shelf failure were encountered, I question whether it would even be possible to limp along. In an example configuration, we would be down 24 disks, 4 or 5 would rebuild to the remaining spares available.  Those rebuilds along should require significant cpu to occur concurrently and I expect would impact data services significantly.  Additionally, at least 10 other raid groups would be either single or double degraded.  I expect the performance degradation at this point would be so great that the most practical course of action would be to shutdown the system until the failed shelf could be replaced.

Thanks for any input.  I would like to know if anyone has any experience thinking through this type of scenario.  Is considering this configuration interesting or perhaps silly?  Are any best practice recommendations being violated?

Thanks in advance.

--Jordan

_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

For the most part, the shelf is just sheet metal. The controllers, power supplies, and all that are redundant. I'm not in hardware engineering, but I imagine that's the reason there isn't a native ability to isolate data within certain shelves. Catastrophic sheet metal failure is unlikely.

I was a NetApp customer for about 10 years before joining the company and the only shelf "failure" I encountered was a mildly annoying temperature sensor failure. There was only the one sensor, so I had to replace the shelf. It didn't cause downtime or anything.

From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Douglas Siggins Sent: Tuesday, June 21, 2016 5:42 PM To: NGC-tmacmd-gmail.com tmacmd@gmail.com Cc: Toasters toasters@teaparty.net Subject: Re: Raid disk layout - the ability to lose a shelf.

I find the whole exercise a fine idea. However, I've never seen whole shelf failures, usually its one of the modules (thats why there is two). Has anyone ever experienced a whole shelf go offline because of a failure outside of power?

I've only had one of my 4243s with a single slot failure.

On Tue, Jun 21, 2016 at 11:26 AM, tmac <tmacmd@gmail.commailto:tmacmd@gmail.com> wrote: This can be crazy! You will end up with small raid groups eating up parity to sacrifice for what? Sacrificing space for what?

Now, given a large environment (like 10 shelves or more)...maybe you can start with this. I did this for a customer once. ...ONCE

We ended up with 16 or 18 disk raidgroups and there was no more than 2 per raidgroup per shelf. We took this one a bit farther too....all even numbered disks (*0,*2,*4,*6,*8) were assigned to node 2. The rest to node 1 When a disk fails, assign even to node 2, odd to node 1

This made the aggregates a bit trickier to place, but it happened.

Now, when a disk fails, I cannot control where it rebuilds other than a spare. I tried to keep the spares on one shelf thinking in the event of failures, they will likley be different raidgroups.

However, one could script some monitoring software to wathc where the spares are and watch for more than 2 disks in a raidgroup showing in the same shelf. Then possibly forcing the running of a "disk copy start" command to nondisruptively move the disk. THIS TAKES LONGER than a reconstruction!!! Why? the process is NICE'd to use limited resources because it it not critical yet.

--tmac

Tim McCarthy, Principal Consultant

On Tue, Jun 21, 2016 at 11:11 AM, jordan slingerland <jordan.slingerland@gmail.commailto:jordan.slingerland@gmail.com> wrote: Hello, I am deploying a new 8040 and it was requested that the aggregates / raid groups are laid out in such a way that no more than 2 disks in any raid group are within the same shelf. At first this sounds like it reduces single points of failure and could protect availability from the failure of a full disk shelf. I argue against this strategy and was wondering if anyone in this list had any feedback. My thought is that this configuration is marginally increasing availability at the sacrifice of additional risk to data integrity. With this strategy, each time a disk failed we would endure not only the initial rebuilt to spare, but a second rebuild when a disk replace is executed to put the original shelf/slot/disk back into the the active raid group. Additional, if a shelf failure were encountered, I question whether it would even be possible to limp along. In an example configuration, we would be down 24 disks, 4 or 5 would rebuild to the remaining spares available. Those rebuilds along should require significant cpu to occur concurrently and I expect would impact data services significantly. Additionally, at least 10 other raid groups would be either single or double degraded. I expect the performance degradation at this point would be so great that the most practical course of action would be to shutdown the system until the failed shelf could be replaced.

Thanks for any input. I would like to know if anyone has any experience thinking through this type of scenario. Is considering this configuration interesting or perhaps silly? Are any best practice recommendations being violated?

Thanks in advance. --Jordan

_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

I have seen (more than once) NetApp support recommending power cycling shelf. And I did full shelf replacement less than a month ago.

?????????? ? iPhone

21 ???? 2016 ?., ? 18:53, Douglas Siggins <siggins@gmail.commailto:siggins@gmail.com> ???????(?):

I find the whole exercise a fine idea. However, I've never seen whole shelf failures, usually its one of the modules (thats why there is two). Has anyone ever experienced a whole shelf go offline because of a failure outside of power?

I've only had one of my 4243s with a single slot failure.

On Tue, Jun 21, 2016 at 11:26 AM, tmac <tmacmd@gmail.commailto:tmacmd@gmail.com> wrote: This can be crazy! You will end up with small raid groups eating up parity to sacrifice for what? Sacrificing space for what?

Now, given a large environment (like 10 shelves or more)...maybe you can start with this. I did this for a customer once. ...ONCE

We ended up with 16 or 18 disk raidgroups and there was no more than 2 per raidgroup per shelf. We took this one a bit farther too....all even numbered disks (*0,*2,*4,*6,*8) were assigned to node 2. The rest to node 1 When a disk fails, assign even to node 2, odd to node 1

This made the aggregates a bit trickier to place, but it happened.

Now, when a disk fails, I cannot control where it rebuilds other than a spare. I tried to keep the spares on one shelf thinking in the event of failures, they will likley be different raidgroups.

However, one could script some monitoring software to wathc where the spares are and watch for more than 2 disks in a raidgroup showing in the same shelf. Then possibly forcing the running of a "disk copy start" command to nondisruptively move the disk. THIS TAKES LONGER than a reconstruction!!! Why? the process is NICE'd to use limited resources because it it not critical yet.

--tmac

Tim McCarthy, Principal Consultant

On Tue, Jun 21, 2016 at 11:11 AM, jordan slingerland <jordan.slingerland@gmail.commailto:jordan.slingerland@gmail.com> wrote: Hello,

I am deploying a new 8040 and it was requested that the aggregates / raid groups are laid out in such a way that no more than 2 disks in any raid group are within the same shelf.

At first this sounds like it reduces single points of failure and could protect availability from the failure of a full disk shelf.

I argue against this strategy and was wondering if anyone in this list had any feedback.

My thought is that this configuration is marginally increasing availability at the sacrifice of additional risk to data integrity. With this strategy, each time a disk failed we would endure not only the initial rebuilt to spare, but a second rebuild when a disk replace is executed to put the original shelf/slot/disk back into the the active raid group.

Additional, if a shelf failure were encountered, I question whether it would even be possible to limp along. In an example configuration, we would be down 24 disks, 4 or 5 would rebuild to the remaining spares available. Those rebuilds along should require significant cpu to occur concurrently and I expect would impact data services significantly. Additionally, at least 10 other raid groups would be either single or double degraded. I expect the performance degradation at this point would be so great that the most practical course of action would be to shutdown the system until the failed shelf could be replaced.

Thanks for any input. I would like to know if anyone has any experience thinking through this type of scenario. Is considering this configuration interesting or perhaps silly? Are any best practice recommendations being violated?

Thanks in advance.

--Jordan

_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

"A shelf can certainly have problems like any electronic device, I just question whether isolating aggregates to particular shelves solves a real problem."--- That's been my thinking the whole thread.

All great thoughts, but it's really deep in the weeds to waste time over thinking the problem on enterprise HW, using retail JBOD thinking.

We're way down into the .00x afr percentages here even where specific targeted/found bugs actually exist.  Add more decimal points for the much more rare smoke events that can kill a shelf.

ONTAP does a good job at layout given a lot of practical experience/data based on best placement for highest reliability.  _________________________________Jeff MohlerTech Yahoo, Storage Architect, Principal(831)454-6712 YPAC Gold Member Twitter: @PrincipalYahoo CorpIM:  Hipchat & Iris

I experienced a disk shelf failure once, some internal electronics failed, smoke and all that fun ... no data availability.

Another possible option is to enable data mirror across two separate sas domains, it used to need snapmirror_local license on 7Mode OnTAP. On Jun 21, 2016 18:15, "jordan slingerland" jordan.slingerland@gmail.com wrote:

thanks for all the reply so far. That is a valid point but I believe in that situation each raidgroup could be extended by 1 or 2 disks. The initial configuration will be 12 shelves@tmac and 12 disks raid groups. so though the raid groups will end up being smaller than I would typically recommend, not tiny.

On Tue, Jun 21, 2016 at 12:01 PM, Rhorer, Kyle L. (JSC-OD)[THE BOEING COMPANY] kyle.l.rhorer@nasa.gov wrote:

...

Another issue to think about besides resiliency… what happens in this “no more than two RAID group disks per shelf” scheme when they want to add another shelf because they’re running out of capacity?

...

On Jun 21, 2016, at 10:11, jordan slingerland <

jordan.slingerland@gmail.com> wrote:

...

Hello,

I am deploying a new 8040 and it was requested that the aggregates /

raid groups are laid out in such a way that no more than 2 disks in any raid group are within the same shelf.

...

At first this sounds like it reduces single points of failure and could

protect availability from the failure of a full disk shelf.

...

I argue against this strategy and was wondering if anyone in this list

had any feedback.

...

My thought is that this configuration is marginally increasing

availability at the sacrifice of additional risk to data integrity. With this strategy, each time a disk failed we would endure not only the initial rebuilt to spare, but a second rebuild when a disk replace is executed to put the original shelf/slot/disk back into the the active raid group.

...

Additional, if a shelf failure were encountered, I question whether it

would even be possible to limp along. In an example configuration, we would be down 24 disks, 4 or 5 would rebuild to the remaining spares available. Those rebuilds along should require significant cpu to occur concurrently and I expect would impact data services significantly. Additionally, at least 10 other raid groups would be either single or double degraded. I expect the performance degradation at this point would be so great that the most practical course of action would be to shutdown the system until the failed shelf could be replaced.

...

Thanks for any input. I would like to know if anyone has any

experience thinking through this type of scenario. Is considering this configuration interesting or perhaps silly? Are any best practice recommendations being violated?

...

Thanks in advance.

--Jordan _______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

---

Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters