Hi there
We have to migrate our existing AFF8080 which has NFSv3, ISCSI and CIFS data on it to a new location. The plan is to add a temp A300 in the new location, and add it to the AFF8080’s cluster with two cluster switches (they are sub 200M apart) ISCSI and CIFS should be fairly simple to migrate, but NFSv3 is IP-centric, so as we start the migration volume by volume, the data path for some of our vmware host will be extended somewhat… It will go via the AFF8080’s configured IP address over the cluster network to the A300 controller that holds the volume, and back again. Has anyone tried this with success? Of cause another way would be to migrate the VMs from the VMWare side, which I would like to avoid if possible, but I am a bit worried about the added latency over the cluster network…
Should I be worried?
/Heino
Be sure on your distances. I think you are allowed up to 300M on a fiber from filer to switch for 10G. If you are using patch panels, that length drops.
If your NFS is only vmware datastores, just create a new datastore and storage vmotion. Dont overthink it
If your NFS/CIFS is normal user data, Then you should be able to do a "vol move" from the old AFF to the new AFF and then provided networking is the same, you can also modify the home-node/port of the LIF and move it to the A300s with nearly no disruption.
For iSCSI, if you have full multipathing going, and the networking is the same on the old/new aff, then you could (ONE AT A TIME): set adv net int mod -vserv xxx -lif iscsi_lifa -status-admin down net int mod vserv xxx -lif iscsi_lifa -home-node a300-a -home-port new-port net int mod -vserv xxx -lif iscsi_lifa -status-admin up set admin WAIT 8 minutes, let multipathing stabilize. Verify at the hosts all paths up. Do the next one.
--tmac
*Tim McCarthy, **Principal Consultant*
*Proud Member of the #NetAppATeam https://twitter.com/NetAppATeam*
*I Blog at TMACsRack https://tmacsrack.wordpress.com/*
On Tue, Oct 13, 2020 at 7:40 AM Heino Walther hw@beardmann.dk wrote:
Hi there
We have to migrate our existing AFF8080 which has NFSv3, ISCSI and CIFS data on it to a new location.
The plan is to add a temp A300 in the new location, and add it to the AFF8080’s cluster with two cluster switches (they are sub 200M apart)
ISCSI and CIFS should be fairly simple to migrate, but NFSv3 is IP-centric, so as we start the migration volume by volume, the data path for some of our vmware host will be extended somewhat…
It will go via the AFF8080’s configured IP address over the cluster network to the A300 controller that holds the volume, and back again.
Has anyone tried this with success?
Of cause another way would be to migrate the VMs from the VMWare side, which I would like to avoid if possible, but I am a bit worried about the added latency over the cluster network…
Should I be worried?
/Heino
Toasters mailing list Toasters@teaparty.net https://www.teaparty.net/mailman/listinfo/toasters
The length is not the issue.
The issue is that we have 4 large datastores (20TB+) that needs to be migrated… they all have snapmirror relations attached, so I would rather not use vmotion to a new datastore as I will have to reseed all the snapmirrors and because the destination is rotating aggregates we have no cross-volume dedupe benefits. So… vmotion will only be used if everything else doesn’t work 😊
The problem of cause is that we will end up in a situation where data from the NFS datastores will be pulled via one node across the intercluster links to the node that holds the active volume and back again…
I seem to remember that you can do a “vol move” and hold the cut-over process…. I was thinking of doing the initial vol move (basically snapmirror) of the larger volumes, and then do the cut-over of all the volumes and move the LIF as one big “off-hours” process 😊. Just to be sure…
I think I remember issues while running NFS over the cluster interconnect… basically high latency.
CIFS should not be a problem, and also ISCSI can be moved, just by adding LIFS to the target nodes, and MPIO will take care of the rest… (this is mainly single servers with LUNs for SQL Server etc..)
But has anyone tried to run NFS datastores that uses the intercluster link? Hos severe is the latency? Would it make sense to add 4 intercluster links per node rather than two?
/Heino
Fra: tmac tmacmd@gmail.com Dato: tirsdag den 13. oktober 2020 kl. 14.01 Til: Heino Walther hw@beardmann.dk Cc: "toasters@teaparty.net" toasters@teaparty.net Emne: Re: Migration to new cluster nodes...
Be sure on your distances. I think you are allowed up to 300M on a fiber from filer to switch for 10G. If you are using patch panels, that length drops.
If your NFS is only vmware datastores, just create a new datastore and storage vmotion. Dont overthink it
If your NFS/CIFS is normal user data, Then you should be able to do a "vol move" from the old AFF to the new AFF and then provided networking is the same, you can also modify the home-node/port of the LIF and move it to the A300s with nearly no disruption.
For iSCSI, if you have full multipathing going, and the networking is the same on the old/new aff, then you could (ONE AT A TIME): set adv net int mod -vserv xxx -lif iscsi_lifa -status-admin down net int mod vserv xxx -lif iscsi_lifa -home-node a300-a -home-port new-port net int mod -vserv xxx -lif iscsi_lifa -status-admin up set admin WAIT 8 minutes, let multipathing stabilize. Verify at the hosts all paths up. Do the next one.
--tmac
Tim McCarthy, Principal Consultant
Proud Member of the #NetAppATeamhttps://twitter.com/NetAppATeam
I Blog at TMACsRackhttps://tmacsrack.wordpress.com/
On Tue, Oct 13, 2020 at 7:40 AM Heino Walther <hw@beardmann.dkmailto:hw@beardmann.dk> wrote: Hi there
We have to migrate our existing AFF8080 which has NFSv3, ISCSI and CIFS data on it to a new location. The plan is to add a temp A300 in the new location, and add it to the AFF8080’s cluster with two cluster switches (they are sub 200M apart) ISCSI and CIFS should be fairly simple to migrate, but NFSv3 is IP-centric, so as we start the migration volume by volume, the data path for some of our vmware host will be extended somewhat… It will go via the AFF8080’s configured IP address over the cluster network to the A300 controller that holds the volume, and back again. Has anyone tried this with success? Of cause another way would be to migrate the VMs from the VMWare side, which I would like to avoid if possible, but I am a bit worried about the added latency over the cluster network…
Should I be worried?
/Heino
_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net https://www.teaparty.net/mailman/listinfo/toasters
I run over the intercluster quite frequently. But the only time I have ever seen a problem with latency was when the lif was hosted on a FAS and the volume was on an AFF. In those cases I will get double digit latency.
I use NFSv3 for most of my workloads.
The process you describe, using a delayed vol move, will work just fine.
Wayne
From: Toasters toasters-bounces@teaparty.net On Behalf Of Heino Walther Sent: Tuesday, October 13, 2020 8:46 AM To: tmac tmacmd@gmail.com Cc: toasters@teaparty.net Subject: Re: Migration to new cluster nodes...
CAUTION: This email is from an external source. Do not click links or open attachments unless you recognize the sender and know the content is safe. The length is not the issue.
The issue is that we have 4 large datastores (20TB+) that needs to be migrated… they all have snapmirror relations attached, so I would rather not use vmotion to a new datastore as I will have to reseed all the snapmirrors and because the destination is rotating aggregates we have no cross-volume dedupe benefits. So… vmotion will only be used if everything else doesn’t work 😊
The problem of cause is that we will end up in a situation where data from the NFS datastores will be pulled via one node across the intercluster links to the node that holds the active volume and back again…
I seem to remember that you can do a “vol move” and hold the cut-over process…. I was thinking of doing the initial vol move (basically snapmirror) of the larger volumes, and then do the cut-over of all the volumes and move the LIF as one big “off-hours” process 😊. Just to be sure…
I think I remember issues while running NFS over the cluster interconnect… basically high latency.
CIFS should not be a problem, and also ISCSI can be moved, just by adding LIFS to the target nodes, and MPIO will take care of the rest… (this is mainly single servers with LUNs for SQL Server etc..)
But has anyone tried to run NFS datastores that uses the intercluster link? Hos severe is the latency? Would it make sense to add 4 intercluster links per node rather than two?
/Heino
Fra: tmac <tmacmd@gmail.commailto:tmacmd@gmail.com> Dato: tirsdag den 13. oktober 2020 kl. 14.01 Til: Heino Walther <hw@beardmann.dkmailto:hw@beardmann.dk> Cc: "toasters@teaparty.netmailto:toasters@teaparty.net" <toasters@teaparty.netmailto:toasters@teaparty.net> Emne: Re: Migration to new cluster nodes...
Be sure on your distances. I think you are allowed up to 300M on a fiber from filer to switch for 10G. If you are using patch panels, that length drops.
If your NFS is only vmware datastores, just create a new datastore and storage vmotion. Dont overthink it
If your NFS/CIFS is normal user data, Then you should be able to do a "vol move" from the old AFF to the new AFF and then provided networking is the same, you can also modify the home-node/port of the LIF and move it to the A300s with nearly no disruption.
For iSCSI, if you have full multipathing going, and the networking is the same on the old/new aff, then you could (ONE AT A TIME): set adv net int mod -vserv xxx -lif iscsi_lifa -status-admin down net int mod vserv xxx -lif iscsi_lifa -home-node a300-a -home-port new-port net int mod -vserv xxx -lif iscsi_lifa -status-admin up set admin WAIT 8 minutes, let multipathing stabilize. Verify at the hosts all paths up. Do the next one.
--tmac
Tim McCarthy, Principal Consultant
Proud Member of the #NetAppATeamhttps://twitter.com/NetAppATeam
I Blog at TMACsRackhttps://tmacsrack.wordpress.com/
On Tue, Oct 13, 2020 at 7:40 AM Heino Walther <hw@beardmann.dkmailto:hw@beardmann.dk> wrote: Hi there
We have to migrate our existing AFF8080 which has NFSv3, ISCSI and CIFS data on it to a new location. The plan is to add a temp A300 in the new location, and add it to the AFF8080’s cluster with two cluster switches (they are sub 200M apart) ISCSI and CIFS should be fairly simple to migrate, but NFSv3 is IP-centric, so as we start the migration volume by volume, the data path for some of our vmware host will be extended somewhat… It will go via the AFF8080’s configured IP address over the cluster network to the A300 controller that holds the volume, and back again. Has anyone tried this with success? Of cause another way would be to migrate the VMs from the VMWare side, which I would like to avoid if possible, but I am a bit worried about the added latency over the cluster network…
Should I be worried?
/Heino
_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net https://www.teaparty.net/mailman/listinfo/toasters
On 2020-10-13 10:45, Heino Walther wrote:
But has anyone tried to run NFS datastores that uses the intercluster link? Hos severe is the latency? Would it make sense to add 4 intercluster links per node rather than two?
I just did a 8040 -> 8200/A220 migration doing both methods. Some were moved with "vol move" and some were storage vMotion. Total was about 150TB.
We noted no great increased latency on the VMs that were moved, but they were not latency sensitive workloads, and moving from HDD.
-- Jason
Hi Heino,
I have used this migration scenario in setups where there was a distance of a couple of kilometers between two datacenters and did not run into any issues, even though this was an unsupported setup. I was doing vol moves while data serving was also progressively using more and more cluster interconnect bandwidth.
You will still be in a supported setup so I don’t think you will run into a lot of issues. Some things that you can take into consideration to mitigate risks:
- you can limit the number of concurrent vol moves by lowering the vol move governor setting - see the TR mentioned here: https://community.netapp.com/t5/ONTAP-Discussions/How-many-vol-move-operatio... - you can move the most I/O bound datastores first and then once they are cut over move over their data serving LIF(s) to limit traffic on the cluster backend. - you can monitor latency in your VMware environment and abort vol moves if you are running into issues.
Lastly, maybe not all that important but you don’t want to have your cluster switches go down while you are in your stretched 4-node cluster setup. In my unsupported scenario we had 2 switches on each side, 4 in total with ISLs between them. Like I said, this was not a supported setup bit we were confident enough to get the migration done that was without a whole lot of hassle.
Regards, Filip
On Tue, 13 Oct 2020 at 16:48, Heino Walther hw@beardmann.dk wrote:
The length is not the issue.
The issue is that we have 4 large datastores (20TB+) that needs to be migrated… they all have snapmirror relations attached, so I would rather not use vmotion to a new datastore as I will have to reseed all the snapmirrors and because the destination is rotating aggregates we have no cross-volume dedupe benefits. So… vmotion will only be used if everything else doesn’t work 😊
The problem of cause is that we will end up in a situation where data from the NFS datastores will be pulled via one node across the intercluster links to the node that holds the active volume and back again…
I seem to remember that you can do a “vol move” and hold the cut-over process…. I was thinking of doing the initial vol move (basically snapmirror) of the larger volumes, and then do the cut-over of all the volumes and move the LIF as one big “off-hours” process 😊. Just to be sure…
I think I remember issues while running NFS over the cluster interconnect… basically high latency.
CIFS should not be a problem, and also ISCSI can be moved, just by adding LIFS to the target nodes, and MPIO will take care of the rest… (this is mainly single servers with LUNs for SQL Server etc..)
But has anyone tried to run NFS datastores that uses the intercluster link? Hos severe is the latency? Would it make sense to add 4 intercluster links per node rather than two?
/Heino
*Fra: *tmac tmacmd@gmail.com *Dato: *tirsdag den 13. oktober 2020 kl. 14.01 *Til: *Heino Walther hw@beardmann.dk *Cc: *"toasters@teaparty.net" toasters@teaparty.net *Emne: *Re: Migration to new cluster nodes...
Be sure on your distances.
I think you are allowed up to 300M on a fiber from filer to switch for 10G. If you are using patch panels, that length drops.
If your NFS is only vmware datastores, just create a new datastore and storage vmotion. Dont overthink it
If your NFS/CIFS is normal user data, Then you should be able to do a "vol move" from the old AFF to the new AFF and then provided networking is the same, you can also modify the home-node/port of the LIF and move it to the A300s with nearly no disruption.
For iSCSI, if you have full multipathing going, and the networking is the same on the old/new aff, then you could (ONE AT A TIME):
set adv
net int mod -vserv xxx -lif iscsi_lifa -status-admin down
net int mod vserv xxx -lif iscsi_lifa -home-node a300-a -home-port new-port
net int mod -vserv xxx -lif iscsi_lifa -status-admin up
set admin
WAIT 8 minutes, let multipathing stabilize. Verify at the hosts all paths up. Do the next one.
--tmac
*Tim McCarthy, **Principal Consultant*
*Proud Member of the #NetAppATeam https://twitter.com/NetAppATeam*
*I Blog at **TMACsRack https://tmacsrack.wordpress.com/*
On Tue, Oct 13, 2020 at 7:40 AM Heino Walther hw@beardmann.dk wrote:
Hi there
We have to migrate our existing AFF8080 which has NFSv3, ISCSI and CIFS data on it to a new location.
The plan is to add a temp A300 in the new location, and add it to the AFF8080’s cluster with two cluster switches (they are sub 200M apart)
ISCSI and CIFS should be fairly simple to migrate, but NFSv3 is IP-centric, so as we start the migration volume by volume, the data path for some of our vmware host will be extended somewhat…
It will go via the AFF8080’s configured IP address over the cluster network to the A300 controller that holds the volume, and back again.
Has anyone tried this with success?
Of cause another way would be to migrate the VMs from the VMWare side, which I would like to avoid if possible, but I am a bit worried about the added latency over the cluster network…
Should I be worried?
/Heino
Toasters mailing list Toasters@teaparty.net https://www.teaparty.net/mailman/listinfo/toasters
Toasters mailing list Toasters@teaparty.net https://www.teaparty.net/mailman/listinfo/toasters
I got curious here (I'm not very good at all at Cisco related networking stuff), ISL is the Cisco proprietary VLAN protocol (Inter-Switch Link) and its really old. Right? It's deprecated now and not even all Cisco switches supports it.
You set up a 4-switch "ClusterNet" for your temp stuff there, with 2x the redundancy HW-wise, in this unsupported stretch cluster, correct? Why did you chose to use the old ISL?
/M
On 2020-10-13 18:10, Filip Sneppe wrote:
Lastly, maybe not all that important but you don't want to have your cluster switches go down while you are in your stretched 4-node cluster setup. In my unsupported scenario we had 2 switches on each side, 4 in total with ISLs between them. Like I said, this was not a supported setup bit we were confident enough to get the migration done that was without a whole lot of hassle.
I do not think the NetApp Inter Cluster has anything to do with Cisco...
I do use two NetApp branded cisco switches for intercluster connections (cannot remember their model) the two switches are also interconnected together, but basically you just cable at least one link into each of the switches from your nodes, instead of the two small 0.5M interconnect cables that normally interconnects two HA nodes together. (there is a procedure of doing this that netapp describes... (going from switchless to switched))
I only have two switches at hand __. Do I will have to manage.. the distance is under 200M between the two DCs so I think it will work just fine. I might link up four links from each controller as ISL... this is possible in the AFFF8080, cannot remember if we can manage this in the A300 (we also need some data ports)
/Heino
D. 13.10.2020 18.48 skrev "Michael Bergman" michael.bergman@ericsson.com:
I got curious here (I'm not very good at all at Cisco related networking stuff), ISL is the Cisco proprietary VLAN protocol (Inter-Switch Link) and its really old. Right? It's deprecated now and not even all Cisco switches supports it.
You set up a 4-switch "ClusterNet" for your temp stuff there, with 2x the redundancy HW-wise, in this unsupported stretch cluster, correct? Why did you chose to use the old ISL?
/M
On 2020-10-13 18:10, Filip Sneppe wrote: > Lastly, maybe not all that important but you don't want to have your cluster > switches go down while you are in your stretched 4-node cluster setup. In my > unsupported scenario we had 2 switches on each side, 4 in total with ISLs > between them. Like I said, this was not a supported setup bit we were > confident enough to get the migration done that was without a whole lot of > hassle.
-- Michael Bergman Sr Systems Analyst / Storage Architect michael.bergman@ericsson.com BNEW Service Area Networks, Lab Ops Phone +46 10 7152945 Global Network & Tech, Architecture SMS/MMS +46 70 5480835 Ericsson Torshamnsg 36, 16480 Sthlm, Sweden -- This communication is confidential. We only send and receive email on the basis of the terms set out at www.ericsson.com/email_disclaimer
Heino> We have to migrate our existing AFF8080 which has NFSv3, ISCSI Heino> and CIFS data on it to a new location.
How many nodes in your cluster?
Heino> The plan is to add a temp A300 in the new location, and add it Heino> to the AFF8080’s cluster with two cluster switches (they are Heino> sub 200M apart)
That should work, then you just failover LIFs and vol move volumes.
Heino> ISCSI and CIFS should be fairly simple to migrate, but NFSv3 is Heino> IP-centric, so as we start the migration volume by volume, the Heino> data path for some of our vmware host will be extended Heino> somewhat…
CIFS would be the problem child, since moving it from one node to another is a client outage. NFSv3 shouldn't really care.
Heino> It will go via the AFF8080’s configured IP address over the Heino> cluster network to the A300 controller that holds the volume, Heino> and back again.
Until you move the LIF over to the A300...
Heino> Has anyone tried this with success?
Heino> Of cause another way would be to migrate the VMs from the Heino> VMWare side, which I would like to avoid if possible, but I am Heino> a bit worried about the added latency over the cluster network…
I assume your real goal is to move a two node cluster without any downtime, right? If so, adding in the A300 (with enough storage!) to your cluster should do the trick.
Then it will jsut take time to vol move the data from one system to another, and then once the physical move is complete, you move it back again.
I'd probably remove the AFF8080 pair from the cluster before you shut them down and move them. Then you just power them up, re-add to the cluster, and vol move back.
Tedious, but very doable.
Heino> We have to migrate our existing AFF8080 which has NFSv3, ISCSI Heino> and CIFS data on it to a new location.
How many nodes in your cluster?
A two node AFF8080HA and a two node A300 so four nodes. So we will migrate from the AFF8080 to the A300 (which has enough storage attached to cover the AFF8080)
Heino> The plan is to add a temp A300 in the new location, and add it Heino> to the AFF8080’s cluster with two cluster switches (they are Heino> sub 200M apart)
That should work, then you just failover LIFs and vol move volumes.
Yes the indications I have gotten here is that I should not be so afraid about running some data via the inter cluster link... __ And come to think of it, the issues I had in the past may have been with FAS32XX nodes and spinning disks... which may explain quite a bit __ But I will try to move the larger NFS volumes in one go, I think I can have up to 7 or 8 volume moves pending cut-over at a time... so I think this will be that way...
Heino> ISCSI and CIFS should be fairly simple to migrate, but NFSv3 is Heino> IP-centric, so as we start the migration volume by volume, the Heino> data path for some of our vmware host will be extended Heino> somewhat…
CIFS would be the problem child, since moving it from one node to another is a client outage. NFSv3 shouldn't really care.
Yeah, CIFS is a pain, but again we can do cut-overs in a service window, so should not be an issue here.
Heino> It will go via the AFF8080’s configured IP address over the Heino> cluster network to the A300 controller that holds the volume, Heino> and back again.
Until you move the LIF over to the A300...
Heino> Has anyone tried this with success?
Heino> Of cause another way would be to migrate the VMs from the Heino> VMWare side, which I would like to avoid if possible, but I am Heino> a bit worried about the added latency over the cluster network…
I assume your real goal is to move a two node cluster without any downtime, right? If so, adding in the A300 (with enough storage!) to your cluster should do the trick.
We have the storage required on the A300 __
Then it will jsut take time to vol move the data from one system to another, and then once the physical move is complete, you move it back again.
I'd probably remove the AFF8080 pair from the cluster before you shut them down and move them. Then you just power them up, re-add to the cluster, and vol move back.
Tedious, but very doable.
Yes it's Tedious, but I think it's faster than storage vmotion... also if you have ever tried to do vmotion on larger setups, you will quickly find out just how stupid it is... just a CD mounted or whatever can stop the process... __ Also as we have existing snapmirror relations I would prefer volume move to keep the relations intact.
/Heino
-
Filip Sneppe -
Heino Walther -
Jason Gorrie -
John Stoffel -
Michael Bergman -
tmac -
Wayne McCormick