Hi All,
I have two production cDOT clusters out of 8 nodes in two DCs, running OnTAP 8.3.2.
I ended up having a bit of a weird network topology where 6 nodes (3 pairs) are only 1Gb enabled and 2 nodes (1 pair) are 10Gb enabled.
I wonder how can I enforce cluster peering traffic via those 10Gb enabled nodes?
I know that I have to have ICL LIFs on every node in the clusters, and that's fine, I can do that, but would like to set something like a preference for ICL traffic to flow via 10Gb enabled LIFs.
Cheers, Vladimir
Simply choose your 10 gb interface for your ICL. You could also put your ICL on a vlan off that 10gb link.
.
On Jun 27, 2016, at 7:59 AM, Momonth momonth@gmail.com wrote:
Hi All,
I have two production cDOT clusters out of 8 nodes in two DCs, running OnTAP 8.3.2.
I ended up having a bit of a weird network topology where 6 nodes (3 pairs) are only 1Gb enabled and 2 nodes (1 pair) are 10Gb enabled.
I wonder how can I enforce cluster peering traffic via those 10Gb enabled nodes?
I know that I have to have ICL LIFs on every node in the clusters, and that's fine, I can do that, but would like to set something like a preference for ICL traffic to flow via 10Gb enabled LIFs.
Cheers, Vladimir _______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
This is exactly what I did. What this approach doesn't guarantee is to always stick to 10Gb LIFs SM traffic, therefore there is my initial question. On Jun 27, 2016 17:49, "Francis Kim" fkim@berkcom.com wrote:
Simply choose your 10 gb interface for your ICL. You could also put your ICL on a vlan off that 10gb link.
.
On Jun 27, 2016, at 7:59 AM, Momonth momonth@gmail.com wrote:
Hi All,
I have two production cDOT clusters out of 8 nodes in two DCs, running OnTAP 8.3.2.
I ended up having a bit of a weird network topology where 6 nodes (3 pairs) are only 1Gb enabled and 2 nodes (1 pair) are 10Gb enabled.
I wonder how can I enforce cluster peering traffic via those 10Gb
enabled nodes?
I know that I have to have ICL LIFs on every node in the clusters, and that's fine, I can do that, but would like to set something like a preference for ICL traffic to flow via 10Gb enabled LIFs.
Cheers, Vladimir _______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
On the CLI, start by looking at "net int show -role intercluster" on each cluster. Make sure the LIFs for interclusters are on the interfaces you are expecting.
Next, run "cluster peer show -instance" from each cluster. -> look at the local and remote interfaces. These should only be intercluster interfaces (LIFs)
SnapMirror traffic should only flow to/from those interfaces listed.
--tmac
*Tim McCarthy, **Principal Consultant*
On Mon, Jun 27, 2016 at 2:32 PM, Momonth momonth@gmail.com wrote:
This is exactly what I did. What this approach doesn't guarantee is to always stick to 10Gb LIFs SM traffic, therefore there is my initial question. On Jun 27, 2016 17:49, "Francis Kim" fkim@berkcom.com wrote:
Simply choose your 10 gb interface for your ICL. You could also put your ICL on a vlan off that 10gb link.
.
On Jun 27, 2016, at 7:59 AM, Momonth momonth@gmail.com wrote:
Hi All,
I have two production cDOT clusters out of 8 nodes in two DCs, running OnTAP 8.3.2.
I ended up having a bit of a weird network topology where 6 nodes (3 pairs) are only 1Gb enabled and 2 nodes (1 pair) are 10Gb enabled.
I wonder how can I enforce cluster peering traffic via those 10Gb
enabled nodes?
I know that I have to have ICL LIFs on every node in the clusters, and that's fine, I can do that, but would like to set something like a preference for ICL traffic to flow via 10Gb enabled LIFs.
Cheers, Vladimir _______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Sorry, if I was not clear enough in my first message. I meant to say that the 6 nodes on 1Gb network is my infrastructure limitation, ie those node can not be connected to 10Gb network, but only 1Gb. On Jun 27, 2016 20:50, "tmac" tmacmd@gmail.com wrote:
On the CLI, start by looking at "net int show -role intercluster" on each cluster. Make sure the LIFs for interclusters are on the interfaces you are expecting.
Next, run "cluster peer show -instance" from each cluster. -> look at the local and remote interfaces. These should only be intercluster interfaces (LIFs)
SnapMirror traffic should only flow to/from those interfaces listed.
--tmac
*Tim McCarthy, **Principal Consultant*
On Mon, Jun 27, 2016 at 2:32 PM, Momonth momonth@gmail.com wrote:
This is exactly what I did. What this approach doesn't guarantee is to always stick to 10Gb LIFs SM traffic, therefore there is my initial question. On Jun 27, 2016 17:49, "Francis Kim" fkim@berkcom.com wrote:
Simply choose your 10 gb interface for your ICL. You could also put your ICL on a vlan off that 10gb link.
.
On Jun 27, 2016, at 7:59 AM, Momonth momonth@gmail.com wrote:
Hi All,
I have two production cDOT clusters out of 8 nodes in two DCs, running OnTAP 8.3.2.
I ended up having a bit of a weird network topology where 6 nodes (3 pairs) are only 1Gb enabled and 2 nodes (1 pair) are 10Gb enabled.
I wonder how can I enforce cluster peering traffic via those 10Gb
enabled nodes?
I know that I have to have ICL LIFs on every node in the clusters, and that's fine, I can do that, but would like to set something like a preference for ICL traffic to flow via 10Gb enabled LIFs.
Cheers, Vladimir _______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Momonth> Sorry, if I was not clear enough in my first message. I meant Momonth> to say that the 6 nodes on 1Gb network is my infrastructure Momonth> limitation, ie those node can not be connected to 10Gb Momonth> network, but only 1Gb.
It would be nice if you could show the output of 'show int' for one of your 10g nodes and your 1g nodes. But I also find it surprising that you don't have a set of 10g switches for the InterClusterLinks, since that's the standard for Netapp installs of four nodes or more.
Did you just start with two nodes and just keep adding in FAS2250s or something? :-)
John
On Mon, Jun 27, 2016 at 10:44 PM, John Stoffel john@stoffel.org wrote:
It would be nice if you could show the output of 'show int' for one of your 10g nodes and your 1g nodes. But I also find it surprising that you don't have a set of 10g switches for the InterClusterLinks, since that's the standard for Netapp installs of four nodes or more.
Did you just start with two nodes and just keep adding in FAS2250s or something? :-)
We tarted with 8040 x2 nodes, however I've never heard about the 10G infra for intercluster communications ..
Here is my current setup:
na101::> network interface show -role intercluster -fields lif,curr-node,curr-port vserver lif curr-node curr-port ------- ------------------- ------------ --------- na101 na101node-1a-icl-01 na101node-1a e2a <<-- 1Gb from here na101 na101node-1b-icl-01 na101node-1b e2a na101 na101node-2a-icl-01 na101node-2a e2a na101 na101node-2b-icl-01 na101node-2b e2a na101 na101node-3a-icl-01 na101node-3a e2a na101 na101node-3b-icl-01 na101node-3b e0i <<-- 1Gb till here na101 na101node-4a-icl-02 na101node-4a a0a <<-- 10Gb na101 na101node-4b-icl-02 na101node-4b a0a <<-- 10Gb 8 entries were displayed.
na101::> cluster peer show Peer Cluster Name Cluster Serial Number Availability Authentication ------------------------- --------------------- -------------- -------------- na201 XXXXXXXXX Available ok
Like I said it works fine, I just want to make it better =)
You confuse cluster interconnect 10G switches and intercluster peering.
I am actually surprised that such simple question is apparently even not understood. Let's try to ask differently - which local and remote LIFs are selected by Data ONTAP when it connects to another cluster? What are rules?
--- With best regards
Andrei Borzenkov Senior system engineer FTS WEMEAI RUC RU SC TMS FOS
FUJITSU Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation Tel.: +7 495 730 62 20 ( reception) Mob.: +7 916 678 7208 Fax: +7 495 730 62 14 E-mail: Andrei.Borzenkov@ts.fujitsu.com Web: ru.fujitsu.com Company details: ts.fujitsu.com/imprint This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation. Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.
-----Original Message----- From: toasters-bounces@teaparty.net [mailto:toasters- bounces@teaparty.net] On Behalf Of John Stoffel Sent: Monday, June 27, 2016 11:45 PM To: momonth@gmail.com Cc: toasters@teaparty.net Subject: Re: cDOT cluster replication aka peering via specific LIFs
Momonth> Sorry, if I was not clear enough in my first message. I meant Momonth> to say that the 6 nodes on 1Gb network is my infrastructure Momonth> limitation, ie those node can not be connected to 10Gb network, Momonth> but only 1Gb.
It would be nice if you could show the output of 'show int' for one of your 10g nodes and your 1g nodes. But I also find it surprising that you don't have a set of 10g switches for the InterClusterLinks, since that's the standard for Netapp installs of four nodes or more.
Did you just start with two nodes and just keep adding in FAS2250s or something? :-)
John _______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
On Tue, Jun 28, 2016 at 10:57 AM, andrei.borzenkov@ts.fujitsu.com andrei.borzenkov@ts.fujitsu.com wrote:
You confuse cluster interconnect 10G switches and intercluster peering.
"You" here is me (Vladimir) or John?
I am actually surprised that such simple question is apparently even not understood. Let's try to ask differently - which local and remote LIFs are selected by Data ONTAP when it connects to another cluster? What are rules?
According to the documentation nodes of local and remote clusters setup full-mesh connectivity, so LIFs are involved on both sides.
Cheers, Vladimir
Intercluster communications are for communicating between clusters.
The Cluster Network (the private 10G switches) are used for Intracluster activities. i.e. local communications stay on the cluster network.
So, any communications from Cluster na101 to/from Cluster na201 will always go through the InterCluster LIFs.
Therefore, if the communication is from na101:node1 to na201:node1 it will be 1Gig na101:node1 to na201:node2 it will be 1Gig na101:node1 to na201:node3 it will be 1Gig na101:node1 to na201:node4 it will be 1Gig na101:node1 to na201:node5 it will be 1Gig na101:node1 to na201:node6 it will be 1Gig na101:node5 to na201:node6 it will be 10Gig
In other words, if your mirror source/destinations are on nodes 5/6 then you will use 10Gig. If your mirror source/destinations are on anything but those two nodes, you will be using 1Gig.
--tmac
*Tim McCarthy, **Principal Consultant*
On Tue, Jun 28, 2016 at 6:03 AM, Momonth momonth@gmail.com wrote:
On Tue, Jun 28, 2016 at 10:57 AM, andrei.borzenkov@ts.fujitsu.com andrei.borzenkov@ts.fujitsu.com wrote:
You confuse cluster interconnect 10G switches and intercluster peering.
"You" here is me (Vladimir) or John?
I am actually surprised that such simple question is apparently even not
understood. Let's try to ask differently - which local and remote LIFs are selected by Data ONTAP when it connects to another cluster? What are rules?
According to the documentation nodes of local and remote clusters setup full-mesh connectivity, so LIFs are involved on both sides.
Cheers, Vladimir _______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
On Tue, Jun 28, 2016 at 12:32 PM, tmac tmacmd@gmail.com wrote:
Intercluster communications are for communicating between clusters.
The Cluster Network (the private 10G switches) are used for Intracluster activities. i.e. local communications stay on the cluster network.
So, any communications from Cluster na101 to/from Cluster na201 will always go through the InterCluster LIFs.
Therefore, if the communication is from na101:node1 to na201:node1 it will be 1Gig na101:node1 to na201:node2 it will be 1Gig na101:node1 to na201:node3 it will be 1Gig na101:node1 to na201:node4 it will be 1Gig na101:node1 to na201:node5 it will be 1Gig na101:node1 to na201:node6 it will be 1Gig na101:node5 to na201:node6 it will be 10Gig
In other words, if your mirror source/destinations are on nodes 5/6 then you will use 10Gig. If your mirror source/destinations are on anything but those two nodes, you will be using 1Gig.
Awesome, thanks, now the question is:
Is it possible to invoke 10Gig LIFs, available on nodes 5/6, to mirror volumes on nodes 1/2/3/4?
Cheers, Vladimir
To the best of my knowledge: No.
The full mesh is set up so that all nodes can talk to each other on their local LIFs. Your speed will be the slowest link.
refer to my other email (sent privately yesterday). let me know.
--tmac
*Tim McCarthy, **Principal Consultant*
On Tue, Jun 28, 2016 at 6:37 AM, Momonth momonth@gmail.com wrote:
On Tue, Jun 28, 2016 at 12:32 PM, tmac tmacmd@gmail.com wrote:
Intercluster communications are for communicating between clusters.
The Cluster Network (the private 10G switches) are used for Intracluster activities. i.e. local communications stay on the cluster network.
So, any communications from Cluster na101 to/from Cluster na201 will always go through the InterCluster LIFs.
Therefore, if the communication is from na101:node1 to na201:node1 it will be 1Gig na101:node1 to na201:node2 it will be 1Gig na101:node1 to na201:node3 it will be 1Gig na101:node1 to na201:node4 it will be 1Gig na101:node1 to na201:node5 it will be 1Gig na101:node1 to na201:node6 it will be 1Gig na101:node5 to na201:node6 it will be 10Gig
In other words, if your mirror source/destinations are on nodes 5/6 then
you
will use 10Gig. If your mirror source/destinations are on anything but those two nodes,
you
will be using 1Gig.
Awesome, thanks, now the question is:
Is it possible to invoke 10Gig LIFs, available on nodes 5/6, to mirror volumes on nodes 1/2/3/4?
Cheers, Vladimir
On Tue, Jun 28, 2016 at 12:41 PM, tmac tmacmd@gmail.com wrote:
To the best of my knowledge: No.
The full mesh is set up so that all nodes can talk to each other on their local LIFs. Your speed will be the slowest link.
refer to my other email (sent privately yesterday). let me know.
OK, thanks.
Awesome, thanks, now the question is:
Is it possible to invoke 10Gig LIFs, available on nodes 5/6, to mirror volumes on nodes 1/2/3/4?
[Borzenkov, Andrei] As I am on training anyway, I tried what happens if I use only partial intercluster connectivity. The result is
- I can in principle establish peer relationship if not all nodes have intercluster LIFs, but this relationship is shown as "partial" then.
- snapmirror does not work in this case. I.e. if I have node1 with intercluster LIF and node2 with aggregate and volume, attempt to update snapmirror simply times out.
- that happens also if I initially have full mesh and later remove one intercluster LIF from configuration (also modifying cluster peer relationship)
So it looks like connection always happens to node that owns volume and answer to your question is "no". This is with 8.3.2P2.
Thanks Andrei, I appreciate your effort, and I actually concluded the same on 8.2.3.
On Wed, Jul 13, 2016 at 10:12 AM, andrei.borzenkov@ts.fujitsu.com andrei.borzenkov@ts.fujitsu.com wrote:
As I am on training anyway, I tried what happens if I use only partial intercluster connectivity. The result is
I can in principle establish peer relationship if not all nodes have intercluster LIFs, but this relationship is shown as "partial" then.
snapmirror does not work in this case. I.e. if I have node1 with intercluster LIF and node2 with aggregate and volume, attempt to update snapmirror simply times out.
that happens also if I initially have full mesh and later remove one intercluster LIF from configuration (also modifying cluster peer relationship)
So it looks like connection always happens to node that owns volume and answer to your question is "no". This is with 8.3.2P2.
That makes sense. Mirroring is a node to node operation. If it cannot communicate directly between the nodes on the intercluster interfaces, it will fail.
--tmac
*Tim McCarthy, **Principal Consultant*
*Proud Member of the #NetAppATeam https://twitter.com/NetAppATeam*
*I Blog at TMACsRack https://tmacsrack.wordpress.com/*
On Wed, Jul 13, 2016 at 7:18 AM, Momonth momonth@gmail.com wrote:
Thanks Andrei, I appreciate your effort, and I actually concluded the same on 8.2.3.
On Wed, Jul 13, 2016 at 10:12 AM, andrei.borzenkov@ts.fujitsu.com andrei.borzenkov@ts.fujitsu.com wrote:
As I am on training anyway, I tried what happens if I use only partial
intercluster connectivity. The result is
- I can in principle establish peer relationship if not all nodes have
intercluster LIFs, but this relationship is shown as "partial" then.
- snapmirror does not work in this case. I.e. if I have node1 with
intercluster LIF and node2 with aggregate and volume, attempt to update snapmirror simply times out.
- that happens also if I initially have full mesh and later remove one
intercluster LIF from configuration (also modifying cluster peer relationship)
So it looks like connection always happens to node that owns volume and
answer to your question is "no". This is with 8.3.2P2.
"You" here is John.
--- With best regards
Andrei Borzenkov Senior system engineer FTS WEMEAI RUC RU SC TMS FOS
FUJITSU Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation Tel.: +7 495 730 62 20 ( reception) Mob.: +7 916 678 7208 Fax: +7 495 730 62 14 E-mail: Andrei.Borzenkov@ts.fujitsu.com Web: ru.fujitsu.com Company details: ts.fujitsu.com/imprint This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation. Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.
-----Original Message----- From: vladimir.zhigulin@gmail.com [mailto:vladimir.zhigulin@gmail.com] On Behalf Of Momonth Sent: Tuesday, June 28, 2016 1:03 PM To: Borzenkov, Andrei Cc: John Stoffel; toasters@teaparty.net Subject: Re: cDOT cluster replication aka peering via specific LIFs
On Tue, Jun 28, 2016 at 10:57 AM, andrei.borzenkov@ts.fujitsu.com andrei.borzenkov@ts.fujitsu.com wrote:
You confuse cluster interconnect 10G switches and intercluster
peering.
"You" here is me (Vladimir) or John?
I am actually surprised that such simple question is apparently even
not understood. Let's try to ask differently - which local and remote LIFs are selected by Data ONTAP when it connects to another cluster? What are rules?
According to the documentation nodes of local and remote clusters setup full-mesh connectivity, so LIFs are involved on both sides.
Cheers, Vladimir
andrei> You confuse cluster interconnect 10G switches and intercluster peering.
Yup, I did.
andrei> I am actually surprised that such simple question is andrei> apparently even not understood. Let's try to ask differently - andrei> which local and remote LIFs are selected by Data ONTAP when it andrei> connects to another cluster? What are rules?
The original question was a bit lacking on details... and the distinction of IntERcluster and IntRAcluster can be hard when you are -ENOCAFFEINE in the morning.
*grin*
Wonder if your broadcast domains and failover groups should be reviewed.
.
On Jun 27, 2016, at 11:32 AM, Momonth <momonth@gmail.commailto:momonth@gmail.com> wrote:
This is exactly what I did. What this approach doesn't guarantee is to always stick to 10Gb LIFs SM traffic, therefore there is my initial question.
On Jun 27, 2016 17:49, "Francis Kim" <fkim@berkcom.commailto:fkim@berkcom.com> wrote: Simply choose your 10 gb interface for your ICL. You could also put your ICL on a vlan off that 10gb link.
.
On Jun 27, 2016, at 7:59 AM, Momonth <momonth@gmail.commailto:momonth@gmail.com> wrote:
Hi All,
I have two production cDOT clusters out of 8 nodes in two DCs, running OnTAP 8.3.2.
I ended up having a bit of a weird network topology where 6 nodes (3 pairs) are only 1Gb enabled and 2 nodes (1 pair) are 10Gb enabled.
I wonder how can I enforce cluster peering traffic via those 10Gb enabled nodes?
I know that I have to have ICL LIFs on every node in the clusters, and that's fine, I can do that, but would like to set something like a preference for ICL traffic to flow via 10Gb enabled LIFs.
Cheers, Vladimir _______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
That is a possibility
--tmac
*Tim McCarthy, **Principal Consultant*
On Mon, Jun 27, 2016 at 3:06 PM, Francis Kim fkim@berkcom.com wrote:
Wonder if your broadcast domains and failover groups should be reviewed.
.
On Jun 27, 2016, at 11:32 AM, Momonth momonth@gmail.com wrote:
This is exactly what I did. What this approach doesn't guarantee is to always stick to 10Gb LIFs SM traffic, therefore there is my initial question. On Jun 27, 2016 17:49, "Francis Kim" fkim@berkcom.com wrote:
Simply choose your 10 gb interface for your ICL. You could also put your ICL on a vlan off that 10gb link.
.
On Jun 27, 2016, at 7:59 AM, Momonth momonth@gmail.com wrote:
Hi All,
I have two production cDOT clusters out of 8 nodes in two DCs, running OnTAP 8.3.2.
I ended up having a bit of a weird network topology where 6 nodes (3 pairs) are only 1Gb enabled and 2 nodes (1 pair) are 10Gb enabled.
I wonder how can I enforce cluster peering traffic via those 10Gb
enabled nodes?
I know that I have to have ICL LIFs on every node in the clusters, and that's fine, I can do that, but would like to set something like a preference for ICL traffic to flow via 10Gb enabled LIFs.
Cheers, Vladimir _______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
A bit more in depth in response to your question about "preference". WIth ICL you have to create a LIF specifically for IC traffic, as opposed to selecting among existing LIFs for that traffic.
.
On Jun 27, 2016, at 7:59 AM, Momonth momonth@gmail.com wrote:
Hi All,
I have two production cDOT clusters out of 8 nodes in two DCs, running OnTAP 8.3.2.
I ended up having a bit of a weird network topology where 6 nodes (3 pairs) are only 1Gb enabled and 2 nodes (1 pair) are 10Gb enabled.
I wonder how can I enforce cluster peering traffic via those 10Gb enabled nodes?
I know that I have to have ICL LIFs on every node in the clusters, and that's fine, I can do that, but would like to set something like a preference for ICL traffic to flow via 10Gb enabled LIFs.
Cheers, Vladimir _______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
-
andrei.borzenkov＠ts.fujitsu.com -
Francis Kim -
John Stoffel -
Momonth -
tmac