I have seen whole shelf failures due to Bug ID 902420. (Don’t bother trying to look up the bug, it’s one of those wonderful, completely blank ones.)
Doug Clendening
(c) 713-516-4671
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Douglas Siggins Sent: Tuesday, June 21, 2016 10:42 AM To: tmac Cc: Toasters Subject: [**EXTERNAL**] Re: Raid disk layout - the ability to lose a shelf.
I find the whole exercise a fine idea. However, I've never seen whole shelf failures, usually its one of the modules (thats why there is two). Has anyone ever experienced a whole shelf go offline because of a failure outside of power?
I've only had one of my 4243s with a single slot failure.
On Tue, Jun 21, 2016 at 11:26 AM, tmac <tmacmd@gmail.commailto:tmacmd@gmail.com> wrote: This can be crazy! You will end up with small raid groups eating up parity to sacrifice for what? Sacrificing space for what?
Now, given a large environment (like 10 shelves or more)...maybe you can start with this. I did this for a customer once. ...ONCE
We ended up with 16 or 18 disk raidgroups and there was no more than 2 per raidgroup per shelf. We took this one a bit farther too....all even numbered disks (*0,*2,*4,*6,*8) were assigned to node 2. The rest to node 1 When a disk fails, assign even to node 2, odd to node 1
This made the aggregates a bit trickier to place, but it happened.
Now, when a disk fails, I cannot control where it rebuilds other than a spare. I tried to keep the spares on one shelf thinking in the event of failures, they will likley be different raidgroups.
However, one could script some monitoring software to wathc where the spares are and watch for more than 2 disks in a raidgroup showing in the same shelf. Then possibly forcing the running of a "disk copy start" command to nondisruptively move the disk. THIS TAKES LONGER than a reconstruction!!! Why? the process is NICE'd to use limited resources because it it not critical yet.
--tmac
Tim McCarthy, Principal Consultant
On Tue, Jun 21, 2016 at 11:11 AM, jordan slingerland <jordan.slingerland@gmail.commailto:jordan.slingerland@gmail.com> wrote: Hello, I am deploying a new 8040 and it was requested that the aggregates / raid groups are laid out in such a way that no more than 2 disks in any raid group are within the same shelf. At first this sounds like it reduces single points of failure and could protect availability from the failure of a full disk shelf. I argue against this strategy and was wondering if anyone in this list had any feedback. My thought is that this configuration is marginally increasing availability at the sacrifice of additional risk to data integrity. With this strategy, each time a disk failed we would endure not only the initial rebuilt to spare, but a second rebuild when a disk replace is executed to put the original shelf/slot/disk back into the the active raid group. Additional, if a shelf failure were encountered, I question whether it would even be possible to limp along. In an example configuration, we would be down 24 disks, 4 or 5 would rebuild to the remaining spares available. Those rebuilds along should require significant cpu to occur concurrently and I expect would impact data services significantly. Additionally, at least 10 other raid groups would be either single or double degraded. I expect the performance degradation at this point would be so great that the most practical course of action would be to shutdown the system until the failed shelf could be replaced.
Thanks for any input. I would like to know if anyone has any experience thinking through this type of scenario. Is considering this configuration interesting or perhaps silly? Are any best practice recommendations being violated?
Thanks in advance. --Jordan
_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters