Hi, I have read that an initiator should not be zoned to more than 4 target ports. We have a 8.2 cdot cluster with 12 nodes each having two fcp lifs. So there are potentially 12 paths to storage per fabric.
Just curious what strategy has been used to limit the paths? Zoning, port sets, etc.? And any recommendations.
Thank you. Kevin
Hi Kevin First, I think your current config is unsupported, since max FCP nodes in a single cluster is 8 in cDOT 8.2. (Going from memory, but I’m pretty sure that’s the rule.)
Where did you read that zoning guideline? I’d be surprised if that was a NetApp practice. If a switch or server/OS vendor, then it merits investigation. If another storage vendor, then I’d say that’s their problem and you should feel confident in the NetApp practice of a zone for each initiator to target LIFs on each node x fabric.
Another tidbit for you is that in 8.3 we have a feature called Selective LUN Mapping. When new LUN maps are created, by default the only paths visible will be on the node owning the aggr containing the LUN and its HA partner. This will automatically reduce visible paths. Note that the main value of SLM is to reduce path consumption in attached hosts/servers. For example, VMware ESXi allows 256 LUNs and 1024 paths (including non-array devices such as USB, CD, internal disks, etc.). With more than two cDOT nodes, you start to deplete paths faster than LUNs. Having said that, depending on the issue that 4 target practice is meant to solve/prevent, SLM may do the job in another, simpler way.
Share and enjoy!
Peter
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Kevin Richards Sent: Monday, July 13, 2015 1:53 PM To: toasters@teaparty.net Subject: fcp paths in multi-node cluster
Hi, I have read that an initiator should not be zoned to more than 4 target ports. We have a 8.2 cdot cluster with 12 nodes each having two fcp lifs. So there are potentially 12 paths to storage per fabric.
Just curious what strategy has been used to limit the paths? Zoning, port sets, etc.? And any recommendations.
Thank you. Kevin
I’ve actually seen the 4-port recommendation before. I’m working on updating the Best Practice guide for Oracle lately, and I’m currently writing the LIF section, and that’s in there, but it’s a general recommendation. It’s not an ONTAP limitation.
I understand a general desire to have the most broad FC connectivity possible across a cluster, but FC just doesn’t play will with that approach when you scale up. You’ll start bumping into all sorts of limits to the FC protocol in terms of ports and logins and such. If nothing else, hosts don’t generally like more than 8 paths to a LUN. You run into unusual limits related to SAN booting or failover time etc. If you have a full mesh SAN the ports multiply too. Management can get complicated trying to figure out what’s been zoned where, ISL utilization, and so forth.
Leveraging SLM does seem to be the far and away best option. It was essentially done manually before, but SLM makes it easier because it happens automatically. That’s why it’s the default in 8.3. I polled our field experts to try to get the most common practices and everyone yelled “SLM” at me.
Yes, it does mean that you have a LUN appear on 2 nodes only, but in 99% of use cases that should be enough. You can still use all the usual features like vol-move and your SVM can span all the nodes if you like. We’re only talking about using SLM to limit the *initial* number of ports where it’s advertised at the point of creation. You can change that on the fly if the need arises.
As a side note, if you did manage to lose all the ports on both of the nodes in an HA pair, you’ve probably lost the entire pair for some reason, like a power failure. That means the underlying disks are not accessible, so the existence of other ports in the cluster won’t do you any good.
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Learmonth, Peter Sent: Monday, July 13, 2015 11:05 PM To: Kevin Richards; toasters@teaparty.net Subject: RE: fcp paths in multi-node cluster
Hi Kevin First, I think your current config is unsupported, since max FCP nodes in a single cluster is 8 in cDOT 8.2. (Going from memory, but I’m pretty sure that’s the rule.)
Where did you read that zoning guideline? I’d be surprised if that was a NetApp practice. If a switch or server/OS vendor, then it merits investigation. If another storage vendor, then I’d say that’s their problem and you should feel confident in the NetApp practice of a zone for each initiator to target LIFs on each node x fabric.
Another tidbit for you is that in 8.3 we have a feature called Selective LUN Mapping. When new LUN maps are created, by default the only paths visible will be on the node owning the aggr containing the LUN and its HA partner. This will automatically reduce visible paths. Note that the main value of SLM is to reduce path consumption in attached hosts/servers. For example, VMware ESXi allows 256 LUNs and 1024 paths (including non-array devices such as USB, CD, internal disks, etc.). With more than two cDOT nodes, you start to deplete paths faster than LUNs. Having said that, depending on the issue that 4 target practice is meant to solve/prevent, SLM may do the job in another, simpler way.
Share and enjoy!
Peter
From: toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Kevin Richards Sent: Monday, July 13, 2015 1:53 PM To: toasters@teaparty.netmailto:toasters@teaparty.net Subject: fcp paths in multi-node cluster
Hi, I have read that an initiator should not be zoned to more than 4 target ports. We have a 8.2 cdot cluster with 12 nodes each having two fcp lifs. So there are potentially 12 paths to storage per fabric.
Just curious what strategy has been used to limit the paths? Zoning, port sets, etc.? And any recommendations.
Thank you. Kevin
And just in addition to what Peter and Jeffrey said, I'd use portsets to emulate the SLM behaviour, if you're stuck on 8.2. It's easier to modify the portset, when you move a lun, than the zoning IMHO...
Sebastian
On 7/14/2015 8:16 AM, Steiner, Jeffrey wrote:
I’ve actually seen the 4-port recommendation before. I’m working on updating the Best Practice guide for Oracle lately, and I’m currently writing the LIF section, and that’s in there, but it’s a general recommendation. It’s not an ONTAP limitation.
I understand a general desire to have the most broad FC connectivity possible across a cluster, but FC just doesn’t play will with that approach when you scale up. You’ll start bumping into all sorts of limits to the FC protocol in terms of ports and logins and such. If nothing else, hosts don’t generally like more than 8 paths to a LUN. You run into unusual limits related to SAN booting or failover time etc. If you have a full mesh SAN the ports multiply too. Management can get complicated trying to figure out what’s been zoned where, ISL utilization, and so forth.
Leveraging SLM does seem to be the far and away best option. It was essentially done manually before, but SLM makes it easier because it happens automatically. That’s why it’s the default in 8.3. I polled our field experts to try to get the most common practices and everyone yelled “SLM” at me.
Yes, it does mean that you have a LUN appear on 2 nodes only, but in 99% of use cases that should be enough. You can still use all the usual features like vol-move and your SVM can span all the nodes if you like. We’re only talking about using SLM to limit the **initial** number of ports where it’s advertised at the point of creation. You can change that on the fly if the need arises.
As a side note, if you did manage to lose all the ports on both of the nodes in an HA pair, you’ve probably lost the entire pair for some reason, like a power failure. That means the underlying disks are not accessible, so the existence of other ports in the cluster won’t do you any good.
*From:*toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] *On Behalf Of *Learmonth, Peter *Sent:* Monday, July 13, 2015 11:05 PM *To:* Kevin Richards; toasters@teaparty.net *Subject:* RE: fcp paths in multi-node cluster
Hi Kevin
First, I think your current config is unsupported, since max FCP nodes in a single cluster is 8 in cDOT 8.2. (Going from memory, but I’m pretty sure that’s the rule.)
Where did you read that zoning guideline? I’d be surprised if that was a NetApp practice. If a switch or server/OS vendor, then it merits investigation. If another storage vendor, then I’d say that’s their problem and you should feel confident in the NetApp practice of a zone for each initiator to target LIFs on each node x fabric.
Another tidbit for you is that in 8.3 we have a feature called Selective LUN Mapping. When new LUN maps are created, by default the only paths visible will be on the node owning the aggr containing the LUN and its HA partner. This will automatically reduce visible paths. Note that the main value of SLM is to reduce path consumption in attached hosts/servers. For example, VMware ESXi allows 256 LUNs and 1024 paths (including non-array devices such as USB, CD, internal disks, etc.). With more than two cDOT nodes, you start to deplete paths faster than LUNs. Having said that, depending on the issue that 4 target practice is meant to solve/prevent, SLM may do the job in another, simpler way.
Share and enjoy!
Peter
*From:*toasters-bounces@teaparty.net mailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] *On Behalf Of *Kevin Richards *Sent:* Monday, July 13, 2015 1:53 PM *To:* toasters@teaparty.net mailto:toasters@teaparty.net *Subject:* fcp paths in multi-node cluster
Hi,
I have read that an initiator should not be zoned to more than 4 target ports. We have a 8.2 cdot cluster with 12 nodes each having two fcp lifs. So there are potentially 12 paths to storage per fabric.
Just curious what strategy has been used to limit the paths? Zoning, port sets, etc.? And any recommendations.
Thank you.
Kevin
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Thank you all for your input. I appreciate it! I was mistaken, it's a 4-node cluster, but with 4 paths per node. I will look into leveraging port sets and/or moving to 8.3 with SLM.
Kevin
On Tue, Jul 14, 2015 at 7:36 AM, Sebastian Goetze spgoetze@gmail.com wrote:
And just in addition to what Peter and Jeffrey said, I'd use portsets to emulate the SLM behaviour, if you're stuck on 8.2. It's easier to modify the portset, when you move a lun, than the zoning IMHO...
Sebastian
On 7/14/2015 8:16 AM, Steiner, Jeffrey wrote:
I’ve actually seen the 4-port recommendation before. I’m working on updating the Best Practice guide for Oracle lately, and I’m currently writing the LIF section, and that’s in there, but it’s a general recommendation. It’s not an ONTAP limitation.
I understand a general desire to have the most broad FC connectivity possible across a cluster, but FC just doesn’t play will with that approach when you scale up. You’ll start bumping into all sorts of limits to the FC protocol in terms of ports and logins and such. If nothing else, hosts don’t generally like more than 8 paths to a LUN. You run into unusual limits related to SAN booting or failover time etc. If you have a full mesh SAN the ports multiply too. Management can get complicated trying to figure out what’s been zoned where, ISL utilization, and so forth.
Leveraging SLM does seem to be the far and away best option. It was essentially done manually before, but SLM makes it easier because it happens automatically. That’s why it’s the default in 8.3. I polled our field experts to try to get the most common practices and everyone yelled “SLM” at me.
Yes, it does mean that you have a LUN appear on 2 nodes only, but in 99% of use cases that should be enough. You can still use all the usual features like vol-move and your SVM can span all the nodes if you like. We’re only talking about using SLM to limit the **initial** number of ports where it’s advertised at the point of creation. You can change that on the fly if the need arises.
As a side note, if you did manage to lose all the ports on both of the nodes in an HA pair, you’ve probably lost the entire pair for some reason, like a power failure. That means the underlying disks are not accessible, so the existence of other ports in the cluster won’t do you any good.
*From:* toasters-bounces@teaparty.net [ mailto:toasters-bounces@teaparty.net toasters-bounces@teaparty.net] *On Behalf Of *Learmonth, Peter *Sent:* Monday, July 13, 2015 11:05 PM *To:* Kevin Richards; toasters@teaparty.net *Subject:* RE: fcp paths in multi-node cluster
Hi Kevin
First, I think your current config is unsupported, since max FCP nodes in a single cluster is 8 in cDOT 8.2. (Going from memory, but I’m pretty sure that’s the rule.)
Where did you read that zoning guideline? I’d be surprised if that was a NetApp practice. If a switch or server/OS vendor, then it merits investigation. If another storage vendor, then I’d say that’s their problem and you should feel confident in the NetApp practice of a zone for each initiator to target LIFs on each node x fabric.
Another tidbit for you is that in 8.3 we have a feature called Selective LUN Mapping. When new LUN maps are created, by default the only paths visible will be on the node owning the aggr containing the LUN and its HA partner. This will automatically reduce visible paths. Note that the main value of SLM is to reduce path consumption in attached hosts/servers. For example, VMware ESXi allows 256 LUNs and 1024 paths (including non-array devices such as USB, CD, internal disks, etc.). With more than two cDOT nodes, you start to deplete paths faster than LUNs. Having said that, depending on the issue that 4 target practice is meant to solve/prevent, SLM may do the job in another, simpler way.
Share and enjoy!
Peter
*From:* toasters-bounces@teaparty.net [ mailto:toasters-bounces@teaparty.net toasters-bounces@teaparty.net] *On Behalf Of *Kevin Richards *Sent:* Monday, July 13, 2015 1:53 PM *To:* toasters@teaparty.net *Subject:* fcp paths in multi-node cluster
Hi,
I have read that an initiator should not be zoned to more than 4 target ports. We have a 8.2 cdot cluster with 12 nodes each having two fcp lifs. So there are potentially 12 paths to storage per fabric.
Just curious what strategy has been used to limit the paths? Zoning, port sets, etc.? And any recommendations.
Thank you.
Kevin
Toasters mailing listToasters@teaparty.nethttp://www.teaparty.net/mailman/listinfo/toasters
-
Kevin Richards -
Learmonth, Peter -
Sebastian Goetze -
Steiner, Jeffrey