We are using the IBM n-series filers with our aggregates using dual-parity. Our filer is a dual-node controller, so each node independently owns different disks from the storage shelves. My question deals with how spare disks are allocated. With how it's currently set up, each node has to reserve two disks that it owns as spares to support dual parity. I don't know that we necessarily need four spares for such a relatively small number of physical disks (21 total). It seems pretty wasteful to have all that space just sitting there, especially when the chances of more than two disks failing before we can get replacements in are pretty slim.
Is there some way to just let the nodes share a couple spares so that either one can grab them if failures occur in one of their owned disks? It would be great if I could free up one or two of those spares for actual use. I know there's a way to force spare disksk into aggregates, but I don't want to do this if that means one node won't have any spares to use.
Thanks, Brian Beaird
I don't believe they can pull from unowned, although that would be nice, but if you only have 21 spindles total, it's just my opinion, but 2 spares per controllers is a bit conservative assuming they are all the same size/speed. You should be able to get away with 1 per controller. Then if one of them needed another spare short term, you could zap it over until you new disks arrived.
-- Adam Fox Systems Engineer adamfox@netapp.com
From: Brian.Beaird@cat.com [mailto:Brian.Beaird@cat.com] Sent: Tuesday, October 28, 2008 4:16 PM To: toasters@mathworks.com Subject: Nodes Sharing Spares
We are using the IBM n-series filers with our aggregates using dual-parity. Our filer is a dual-node controller, so each node independently owns different disks from the storage shelves. My question deals with how spare disks are allocated. With how it's currently set up, each node has to reserve two disks that it owns as spares to support dual parity. I don't know that we necessarily need four spares for such a relatively small number of physical disks (21 total). It seems pretty wasteful to have all that space just sitting there, especially when the chances of more than two disks failing before we can get replacements in are pretty slim.
Is there some way to just let the nodes share a couple spares so that either one can grab them if failures occur in one of their owned disks? It would be great if I could free up one or two of those spares for actual use. I know there's a way to force spare disksk into aggregates, but I don't want to do this if that means one node won't have any spares to use.
Thanks, Brian Beaird
The system will only complain if you have no spares left. Based on the fact that you only have 21 disks (10 on one node, 11 on the other?) one spare per node should be plenty.
While double-parity protects you from double disk failure, it doesn't require you actually have to spares.
From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Brian.Beaird@cat.com Sent: Tuesday, October 28, 2008 3:16 PM To: toasters@mathworks.com Subject: Nodes Sharing Spares
We are using the IBM n-series filers with our aggregates using dual-parity. Our filer is a dual-node controller, so each node independently owns different disks from the storage shelves. My question deals with how spare disks are allocated. With how it's currently set up, each node has to reserve two disks that it owns as spares to support dual parity. I don't know that we necessarily need four spares for such a relatively small number of physical disks (21 total). It seems pretty wasteful to have all that space just sitting there, especially when the chances of more than two disks failing before we can get replacements in are pretty slim.
Is there some way to just let the nodes share a couple spares so that either one can grab them if failures occur in one of their owned disks? It would be great if I could free up one or two of those spares for actual use. I know there's a way to force spare disksk into aggregates, but I don't want to do this if that means one node won't have any spares to use.
Thanks, Brian Beaird
Unfortunately there isn't a way to share spares among the controllers. Each node requires its own set of spare drives for the disks that it owns. You'll want to have at least one spare per drive type, and one of those spares per 2-3 shelves of that drive type depending on who you talk to. For 21 total disks there would be no reason to have 4 spares though.
If they're all the same drive type and split between the two heads then you would be find with one spare of that drive type per controller. If some are SAS or FC and some are ATA then you would want to have the node that owns the SAS/FC drives to have one spare of that type and the head that owns the ATA to have one spare of that type. If, however, you have them split evenly between the nodes then each node would need to have a spare of each type.
Can I ask how you have 21 total physical disks? Is there a half-filled shelf in the system?
From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Brian.Beaird@cat.com Sent: Tuesday, October 28, 2008 3:16 PM To: toasters@mathworks.com Subject: Nodes Sharing Spares
We are using the IBM n-series filers with our aggregates using dual-parity. Our filer is a dual-node controller, so each node independently owns different disks from the storage shelves. My question deals with how spare disks are allocated. With how it's currently set up, each node has to reserve two disks that it owns as spares to support dual parity. I don't know that we necessarily need four spares for such a relatively small number of physical disks (21 total). It seems pretty wasteful to have all that space just sitting there, especially when the chances of more than two disks failing before we can get replacements in are pretty slim.
Is there some way to just let the nodes share a couple spares so that either one can grab them if failures occur in one of their owned disks? It would be great if I could free up one or two of those spares for actual use. I know there's a way to force spare disksk into aggregates, but I don't want to do this if that means one node won't have any spares to use.
Thanks, Brian Beaird
Thanks, everyone. I have manually added a spare on each of the nodes to their aggregates, leaving me with a spare left for each one. Yes, the disks are all the same size.
Also, yes, one of the shelves is not completely full. This has happened with the last few groups of shelves we have gotten, leaving us with an odd number. Not totally sure why it happens, but I'm guessing they just give us however many disks it takes to reach the amount of raw space we ordered. They probably don't care if that makes the shelves even or not.
-----Original Message----- From: Chris Muellner [mailto:chrism@mochadata.com] Sent: Tuesday, October 28, 2008 5:01 PM To: Brian.Beaird@cat.com; toasters@mathworks.com Subject: RE: Nodes Sharing Spares
Unfortunately there isn't a way to share spares among the controllers. Each node requires its own set of spare drives for the disks that it owns. You'll want to have at least one spare per drive type, and one of those spares per 2-3 shelves of that drive type depending on who you talk to. For 21 total disks there would be no reason to have 4 spares though.
If they're all the same drive type and split between the two heads then you would be find with one spare of that drive type per controller. If some are SAS or FC and some are ATA then you would want to have the node that owns the SAS/FC drives to have one spare of that type and the head that owns the ATA to have one spare of that type. If, however, you have them split evenly between the nodes then each node would need to have a spare of each type.
Can I ask how you have 21 total physical disks? Is there a half-filled shelf in the system?
From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Brian.Beaird@cat.com Sent: Tuesday, October 28, 2008 3:16 PM To: toasters@mathworks.com Subject: Nodes Sharing Spares
We are using the IBM n-series filers with our aggregates using dual-parity. Our filer is a dual-node controller, so each node independently owns different disks from the storage shelves. My question deals with how spare disks are allocated. With how it's currently set up, each node has to reserve two disks that it owns as spares to support dual parity. I don't know that we necessarily need four spares for such a relatively small number of physical disks (21 total). It seems pretty wasteful to have all that space just sitting there, especially when the chances of more than two disks failing before we can get replacements in are pretty slim.
Is there some way to just let the nodes share a couple spares so that either one can grab them if failures occur in one of their owned disks? It would be great if I could free up one or two of those spares for actual use. I know there's a way to force spare disksk into aggregates, but I don't want to do this if that means one node won't have any spares to use.
Thanks, Brian Beaird
I'd be tempted to put all 21 drives on one head (with two spares) and just have the other head for failover. Don't know if this is a supported configuration, but it would likely deliver the best performance and redundancy for a 21 disk system. In small systems like this the performance bottleneck is the number of spindles, not the controller. Of course I could be wrong. And it could depend on your use. You might want to check with your ibm/netapp team.
Unfortunately there isn't a way to share spares among the controllers. Each node requires its own set of spare drives for the disks that it owns. You'll want to have at least one spare per drive type, and one of those spares per 2-3 shelves of that drive type depending on who you talk to. For 21 total disks there would be no reason to have 4 spares though.
If they're all the same drive type and split between the two heads then you would be find with one spare of that drive type per controller. If some are SAS or FC and some are ATA then you would want to have the node that owns the SAS/FC drives to have one spare of that type and the head that owns the ATA to have one spare of that type. If, however, you have them split evenly between the nodes then each node would need to have a spare of each type.
Can I ask how you have 21 total physical disks? Is there a half-filled shelf in the system?
From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Brian.Beaird@cat.com Sent: Tuesday, October 28, 2008 3:16 PM To: toasters@mathworks.com Subject: Nodes Sharing Spares
We are using the IBM n-series filers with our aggregates using dual-parity. Our filer is a dual-node controller, so each node independently owns different disks from the storage shelves. My question deals with how spare disks are allocated. With how it's currently set up, each node has to reserve two disks that it owns as spares to support dual parity. I don't know that we necessarily need four spares for such a relatively small number of physical disks (21 total). It seems pretty wasteful to have all that space just sitting there, especially when the chances of more than two disks failing before we can get replacements in are pretty slim.
Is there some way to just let the nodes share a couple spares so that either one can grab them if failures occur in one of their owned disks? It would be great if I could free up one or two of those spares for actual use. I know there's a way to force spare disksk into aggregates, but I don't want to do this if that means one node won't have any spares to use.
Thanks, Brian Beaird
It can't be done. To be able to do takeover the other head must be up and running, so it needs at least root volume which requires at least 2 disks. So minimal split in this configuration is 19/2. We have a number of systems that are running this way; it mostly works, except you get autosupport "spares low" every time head is rebooted.
С уважением / With best regards / Mit freundlichen Grüβen
--- Andrey Borzenkov Senior system engineer -----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of David Knight Sent: Wednesday, October 29, 2008 2:16 AM To: Chris Muellner Cc: Brian.Beaird@cat.com; toasters@mathworks.com Subject: Re: Nodes Sharing Spares
I'd be tempted to put all 21 drives on one head (with two spares) and just have the other head for failover. Don't know if this is a supported configuration, but it would likely deliver the best performance and redundancy for a 21 disk system. In small systems like this the performance bottleneck is the number of spindles, not the controller. Of course I could be wrong. And it could depend on your use. You might want to check with your ibm/netapp team.
Unfortunately there isn't a way to share spares among the controllers. Each node requires its own set of spare drives for the disks that it owns. You'll want to have at least one spare per drive type, and one of those spares per 2-3 shelves of that drive type depending on who you talk to. For 21 total disks there would be no reason to have 4 spares though.
If they're all the same drive type and split between the two heads then you would be find with one spare of that drive type per controller. If some are SAS or FC and some are ATA then you would want to have the node that owns the SAS/FC drives to have one spare of that type and the head that owns the ATA to have one spare of that type. If, however, you have them split evenly between the nodes then each node would need to have a spare of each type.
Can I ask how you have 21 total physical disks? Is there a half-filled shelf in the system?
From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Brian.Beaird@cat.com Sent: Tuesday, October 28, 2008 3:16 PM To: toasters@mathworks.com Subject: Nodes Sharing Spares
We are using the IBM n-series filers with our aggregates using dual-parity. Our filer is a dual-node controller, so each node independently owns different disks from the storage shelves. My question deals with how spare disks are allocated. With how it's currently set up, each node has to reserve two disks that it owns as spares to support dual parity. I don't know that we necessarily need four spares for such a relatively small number of physical disks (21 total). It seems pretty wasteful to have all that space just sitting there, especially when the chances of more than two disks failing before we can get replacements in are pretty slim.
Is there some way to just let the nodes share a couple spares so that either one can grab them if failures occur in one of their owned disks? It would be great if I could free up one or two of those spares for actual use. I know there's a way to force spare disksk into aggregates, but I don't want to do this if that means one node won't have any spares to use.
Thanks, Brian Beaird
Also, in the 19/2 configuration, no core dumps will be available on the head with no spares unless you enable the "take a long time to failover option" - I can get specific option if anyone is interested.
we got bit by this problem with a filer head panic'ing with no spares = no core dumps!
HTH
Jack
Borzenkov, Andrey wrote:
It can't be done. To be able to do takeover the other head must be up and running, so it needs at least root volume which requires at least 2 disks. So minimal split in this configuration is 19/2. We have a number of systems that are running this way; it mostly works, except you get autosupport "spares low" every time head is rebooted.
С уважением / With best regards / Mit freundlichen Grüβen
Andrey Borzenkov Senior system engineer -----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of David Knight Sent: Wednesday, October 29, 2008 2:16 AM To: Chris Muellner Cc: Brian.Beaird@cat.com; toasters@mathworks.com Subject: Re: Nodes Sharing Spares
I'd be tempted to put all 21 drives on one head (with two spares) and just have the other head for failover. Don't know if this is a supported configuration, but it would likely deliver the best performance and redundancy for a 21 disk system. In small systems like this the performance bottleneck is the number of spindles, not the controller. Of course I could be wrong. And it could depend on your use. You might want to check with your ibm/netapp team.
Unfortunately there isn't a way to share spares among the controllers. Each node requires its own set of spare drives for the disks that it owns. You'll want to have at least one spare per drive type, and one of those spares per 2-3 shelves of that drive type depending on who you talk to. For 21 total disks there would be no reason to have 4 spares though.
If they're all the same drive type and split between the two heads then you would be find with one spare of that drive type per controller. If some are SAS or FC and some are ATA then you would want to have the node that owns the SAS/FC drives to have one spare of that type and the head that owns the ATA to have one spare of that type. If, however, you have them split evenly between the nodes then each node would need to have a spare of each type.
Can I ask how you have 21 total physical disks? Is there a half-filled shelf in the system?
From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Brian.Beaird@cat.com Sent: Tuesday, October 28, 2008 3:16 PM To: toasters@mathworks.com Subject: Nodes Sharing Spares
We are using the IBM n-series filers with our aggregates using dual-parity. Our filer is a dual-node controller, so each node independently owns different disks from the storage shelves. My question deals with how spare disks are allocated. With how it's currently set up, each node has to reserve two disks that it owns as spares to support dual parity. I don't know that we necessarily need four spares for such a relatively small number of physical disks (21 total). It seems pretty wasteful to have all that space just sitting there, especially when the chances of more than two disks failing before we can get replacements in are pretty slim.
Is there some way to just let the nodes share a couple spares so that either one can grab them if failures occur in one of their owned disks? It would be great if I could free up one or two of those spares for actual use. I know there's a way to force spare disksk into aggregates, but I don't want to do this if that means one node won't have any spares to use.
Thanks, Brian Beaird
Yes, I'd be interested in this option; I did not know core was written to spare disk to be honest.
С уважением / With best regards / Mit freundlichen Grüβen
--- Andrey Borzenkov Senior system engineer -----Original Message----- From: Jack Lyons [mailto:jack1729@gmail.com] Sent: Tuesday, November 11, 2008 9:57 AM To: Borzenkov, Andrey Cc: David Knight; Chris Muellner; Brian.Beaird@cat.com; toasters@mathworks.com Subject: Re: Nodes Sharing Spares
Also, in the 19/2 configuration, no core dumps will be available on the head with no spares unless you enable the "take a long time to failover option" - I can get specific option if anyone is interested.
we got bit by this problem with a filer head panic'ing with no spares = no core dumps!
HTH
Jack
Borzenkov, Andrey wrote:
It can't be done. To be able to do takeover the other head must be up and running, so it needs at least root volume which requires at least 2 disks. So minimal split in this configuration is 19/2. We have a number of systems that are running this way; it mostly works, except you get autosupport "spares low" every time head is rebooted.
С уважением / With best regards / Mit freundlichen Grüβen
Andrey Borzenkov Senior system engineer -----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of David Knight Sent: Wednesday, October 29, 2008 2:16 AM To: Chris Muellner Cc: Brian.Beaird@cat.com; toasters@mathworks.com Subject: Re: Nodes Sharing Spares
I'd be tempted to put all 21 drives on one head (with two spares) and just have the other head for failover. Don't know if this is a supported configuration, but it would likely deliver the best performance and redundancy for a 21 disk system. In small systems like this the performance bottleneck is the number of spindles, not the controller. Of course I could be wrong. And it could depend on your use. You might want to check with your ibm/netapp team.
Unfortunately there isn't a way to share spares among the controllers. Each node requires its own set of spare drives for the disks that it owns. You'll want to have at least one spare per drive type, and one of those spares per 2-3 shelves of that drive type depending on who you talk to. For 21 total disks there would be no reason to have 4 spares though.
If they're all the same drive type and split between the two heads then you would be find with one spare of that drive type per controller. If some are SAS or FC and some are ATA then you would want to have the node that owns the SAS/FC drives to have one spare of that type and the head that owns the ATA to have one spare of that type. If, however, you have them split evenly between the nodes then each node would need to have a spare of each type.
Can I ask how you have 21 total physical disks? Is there a half-filled shelf in the system?
From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Brian.Beaird@cat.com Sent: Tuesday, October 28, 2008 3:16 PM To: toasters@mathworks.com Subject: Nodes Sharing Spares
We are using the IBM n-series filers with our aggregates using dual-parity. Our filer is a dual-node controller, so each node independently owns different disks from the storage shelves. My question deals with how spare disks are allocated. With how it's currently set up, each node has to reserve two disks that it owns as spares to support dual parity. I don't know that we necessarily need four spares for such a relatively small number of physical disks (21 total). It seems pretty wasteful to have all that space just sitting there, especially when the chances of more than two disks failing before we can get replacements in are pretty slim.
Is there some way to just let the nodes share a couple spares so that either one can grab them if failures occur in one of their owned disks? It would be great if I could free up one or two of those spares for actual use. I know there's a way to force spare disksk into aggregates, but I don't want to do this if that means one node won't have any spares to use.
Thanks, Brian Beaird
-
Borzenkov, Andrey -
Brian.Beaird＠cat.com -
Chris Muellner -
Christopher Mende -
David Knight -
Fox, Adam -
Jack Lyons