two filers, have a need to copy the data from source:vol:snap1 to dest:vol
what would be the fastest way to do this, mounting and scp is pretty slow for me, is there some sort of mirroring that i can use to accomplish this ?
If you can take the data down for a single pass, then vol copy is pretty easy. If not, and you have a SnapMirror license, you could do your first pass while the data is being accessed, then do an incremental while it's down.
This assumes that you have the same volume type (trad/flex) on each side. If not, you'll have to use a more logical way of moving data like ndmpcopy or Qtree-based SnapMirror.
-- Adam Fox adamfox@netapp.com
________________________________
From: Mailing Lists [mailto:lists@up-south.com] Sent: Tuesday, June 03, 2008 5:41 PM To: toasters@mathworks.com Subject: snapshot copy
two filers, have a need to copy the data from source:vol:snap1 to dest:vol
what would be the fastest way to do this, mounting and scp is pretty slow for me, is there some sort of mirroring that i can use to accomplish this ?
On Tue, Jun 3, 2008 at 3:05 PM, Fox, Adam Adam.Fox@netapp.com wrote:
If you can take the data down for a single pass, then vol copy is pretty easy. If not, and you have a SnapMirror license, you could do your first pass while the data is being accessed, then do an incremental while it's down.
This assumes that you have the same volume type (trad/flex) on each side. If not, you'll have to use a more logical way of moving data like ndmpcopy or Qtree-based SnapMirror.
-- Adam Fox adamfox@netapp.com
snapmirror is not an option,
thanx though,
ndmpcopy might work. It's filer to filer, and works fairly well, unless your problem is regarding a lot of files. It'll work well enough for a few thousand, but getting into several million it gets a bit nasty, but then any file based replication method for millions of files tends to suck.
It doesn't need a license, and is trival to setup and use on the filer console.
-Blake
On Tue, Jun 3, 2008 at 3:14 PM, Mailing Lists lists@up-south.com wrote:
On Tue, Jun 3, 2008 at 3:05 PM, Fox, Adam Adam.Fox@netapp.com wrote:
If you can take the data down for a single pass, then vol copy is pretty easy. If not, and you have a SnapMirror license, you could do your first pass while the data is being accessed, then do an incremental while it's down.
This assumes that you have the same volume type (trad/flex) on each side. If not, you'll have to use a more logical way of moving data like ndmpcopy or Qtree-based SnapMirror.
-- Adam Fox adamfox@netapp.com
snapmirror is not an option,
thanx though,
Ndmcopy is volume level, i need to copy a napshot. The vol is alive.
I mounted the snapshot and am using scp to copy it to dest filer.
Sent via BlackBerry from T-Mobile
-----Original Message----- From: "Blake Golliher" thelastman@gmail.com
Date: Tue, 3 Jun 2008 21:30:39 To:"Mailing Lists" lists@up-south.com Cc:"Fox, Adam" Adam.Fox@netapp.com, toasters@mathworks.com Subject: Re: snapshot copy
ndmpcopy might work. It's filer to filer, and works fairly well, unless your problem is regarding a lot of files. It'll work well enough for a few thousand, but getting into several million it gets a bit nasty, but then any file based replication method for millions of files tends to suck.
It doesn't need a license, and is trival to setup and use on the filer console.
-Blake
On Tue, Jun 3, 2008 at 3:14 PM, Mailing Lists lists@up-south.com wrote:
On Tue, Jun 3, 2008 at 3:05 PM, Fox, Adam Adam.Fox@netapp.com wrote:
If you can take the data down for a single pass, then vol copy is pretty easy. If not, and you have a SnapMirror license, you could do your first pass while the data is being accessed, then do an incremental while it's down.
This assumes that you have the same volume type (trad/flex) on each side. If not, you'll have to use a more logical way of moving data like ndmpcopy or Qtree-based SnapMirror.
-- Adam Fox adamfox@netapp.com
snapmirror is not an option,
thanx though,
You can still ndmpcopy just a directory. Give it a path like /vol/root/etc will just copy the etc directory. Try /vol/root/.snapshot/hourly.0/etc/ that should work.
-Blake
On Tue, Jun 3, 2008 at 9:48 PM, Steve Rieger lists@up-south.com wrote:
Ndmcopy is volume level, i need to copy a napshot. The vol is alive.
I mounted the snapshot and am using scp to copy it to dest filer.
Sent via BlackBerry from T-Mobile
-----Original Message----- From: "Blake Golliher" thelastman@gmail.com
Date: Tue, 3 Jun 2008 21:30:39 To:"Mailing Lists" lists@up-south.com Cc:"Fox, Adam" Adam.Fox@netapp.com, toasters@mathworks.com Subject: Re: snapshot copy
ndmpcopy might work. It's filer to filer, and works fairly well, unless your problem is regarding a lot of files. It'll work well enough for a few thousand, but getting into several million it gets a bit nasty, but then any file based replication method for millions of files tends to suck.
It doesn't need a license, and is trival to setup and use on the filer console.
-Blake
On Tue, Jun 3, 2008 at 3:14 PM, Mailing Lists lists@up-south.com wrote:
On Tue, Jun 3, 2008 at 3:05 PM, Fox, Adam Adam.Fox@netapp.com wrote:
If you can take the data down for a single pass, then vol copy is pretty easy. If not, and you have a SnapMirror license, you could do your first pass while the data is being accessed, then do an incremental while it's down.
This assumes that you have the same volume type (trad/flex) on each side. If not, you'll have to use a more logical way of moving data like ndmpcopy or Qtree-based SnapMirror.
-- Adam Fox adamfox@netapp.com
snapmirror is not an option,
thanx though,
There's always the possibility of getting a demo license of snapmirror.
I know it's not cheap, but this is exactly what it is designed to do and is well worth the $.
Glenn
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Blake Golliher Sent: Wednesday, June 04, 2008 12:52 AM To: lists@up-south.com Cc: Fox, Adam; toasters@mathworks.com Subject: Re: snapshot copy
You can still ndmpcopy just a directory. Give it a path like /vol/root/etc will just copy the etc directory. Try /vol/root/.snapshot/hourly.0/etc/ that should work.
-Blake
On Tue, Jun 3, 2008 at 9:48 PM, Steve Rieger lists@up-south.com wrote:
Ndmcopy is volume level, i need to copy a napshot. The vol is alive.
I mounted the snapshot and am using scp to copy it to dest filer.
Sent via BlackBerry from T-Mobile
-----Original Message----- From: "Blake Golliher" thelastman@gmail.com
Date: Tue, 3 Jun 2008 21:30:39 To:"Mailing Lists" lists@up-south.com Cc:"Fox, Adam" Adam.Fox@netapp.com, toasters@mathworks.com Subject: Re: snapshot copy
ndmpcopy might work. It's filer to filer, and works fairly well, unless your problem is regarding a lot of files. It'll work well enough for a few thousand, but getting into several million it gets a bit nasty, but then any file based replication method for millions of files tends to suck.
It doesn't need a license, and is trival to setup and use on the filer console.
-Blake
On Tue, Jun 3, 2008 at 3:14 PM, Mailing Lists lists@up-south.com
wrote:
On Tue, Jun 3, 2008 at 3:05 PM, Fox, Adam Adam.Fox@netapp.com
wrote:
If you can take the data down for a single pass, then vol copy is
pretty
easy. If not, and you have a SnapMirror license, you could do your first
pass
while the data is being accessed, then do an incremental while it's down.
This assumes that you have the same volume type (trad/flex) on each
side.
If not, you'll have to use a more logical way of moving data like
ndmpcopy
or Qtree-based SnapMirror.
-- Adam Fox adamfox@netapp.com
snapmirror is not an option,
thanx though,
Snapmirror is really not worth if you are just doing a one time data migration. There should be a snapmirror migrate license. But if this guy is trying to setup a persistent mirror, then your right, snapmirror works splendidly for that.
-Blake
On Tue, Jun 3, 2008 at 10:12 PM, Glenn Walker ggwalker@mindspring.com wrote:
There's always the possibility of getting a demo license of snapmirror.
I know it's not cheap, but this is exactly what it is designed to do and is well worth the $.
Glenn
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Blake Golliher Sent: Wednesday, June 04, 2008 12:52 AM To: lists@up-south.com Cc: Fox, Adam; toasters@mathworks.com Subject: Re: snapshot copy
You can still ndmpcopy just a directory. Give it a path like /vol/root/etc will just copy the etc directory. Try /vol/root/.snapshot/hourly.0/etc/ that should work.
-Blake
On Tue, Jun 3, 2008 at 9:48 PM, Steve Rieger lists@up-south.com wrote:
Ndmcopy is volume level, i need to copy a napshot. The vol is alive.
I mounted the snapshot and am using scp to copy it to dest filer.
Sent via BlackBerry from T-Mobile
-----Original Message----- From: "Blake Golliher" thelastman@gmail.com
Date: Tue, 3 Jun 2008 21:30:39 To:"Mailing Lists" lists@up-south.com Cc:"Fox, Adam" Adam.Fox@netapp.com, toasters@mathworks.com Subject: Re: snapshot copy
ndmpcopy might work. It's filer to filer, and works fairly well, unless your problem is regarding a lot of files. It'll work well enough for a few thousand, but getting into several million it gets a bit nasty, but then any file based replication method for millions of files tends to suck.
It doesn't need a license, and is trival to setup and use on the filer console.
-Blake
On Tue, Jun 3, 2008 at 3:14 PM, Mailing Lists lists@up-south.com
wrote:
On Tue, Jun 3, 2008 at 3:05 PM, Fox, Adam Adam.Fox@netapp.com
wrote:
If you can take the data down for a single pass, then vol copy is
pretty
easy. If not, and you have a SnapMirror license, you could do your first
pass
while the data is being accessed, then do an incremental while it's down.
This assumes that you have the same volume type (trad/flex) on each
side.
If not, you'll have to use a more logical way of moving data like
ndmpcopy
or Qtree-based SnapMirror.
-- Adam Fox adamfox@netapp.com
snapmirror is not an option,
thanx though,
Snapmirror a snapshot ? We have snapmirror, snapvault, cifs, nfs, and many more, money is not the issue speed is Sent via BlackBerry from T-Mobile
-----Original Message----- From: "Glenn Walker" ggwalker@mindspring.com
Date: Wed, 4 Jun 2008 01:12:48 To:"Blake Golliher" thelastman@gmail.com,lists@up-south.com Cc:"Fox, Adam" Adam.Fox@netapp.com,toasters@mathworks.com Subject: RE: snapshot copy
There's always the possibility of getting a demo license of snapmirror.
I know it's not cheap, but this is exactly what it is designed to do and is well worth the $.
Glenn
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Blake Golliher Sent: Wednesday, June 04, 2008 12:52 AM To: lists@up-south.com Cc: Fox, Adam; toasters@mathworks.com Subject: Re: snapshot copy
You can still ndmpcopy just a directory. Give it a path like /vol/root/etc will just copy the etc directory. Try /vol/root/.snapshot/hourly.0/etc/ that should work.
-Blake
On Tue, Jun 3, 2008 at 9:48 PM, Steve Rieger lists@up-south.com wrote:
Ndmcopy is volume level, i need to copy a napshot. The vol is alive.
I mounted the snapshot and am using scp to copy it to dest filer.
Sent via BlackBerry from T-Mobile
-----Original Message----- From: "Blake Golliher" thelastman@gmail.com
Date: Tue, 3 Jun 2008 21:30:39 To:"Mailing Lists" lists@up-south.com Cc:"Fox, Adam" Adam.Fox@netapp.com, toasters@mathworks.com Subject: Re: snapshot copy
ndmpcopy might work. It's filer to filer, and works fairly well, unless your problem is regarding a lot of files. It'll work well enough for a few thousand, but getting into several million it gets a bit nasty, but then any file based replication method for millions of files tends to suck.
It doesn't need a license, and is trival to setup and use on the filer console.
-Blake
On Tue, Jun 3, 2008 at 3:14 PM, Mailing Lists lists@up-south.com
wrote:
On Tue, Jun 3, 2008 at 3:05 PM, Fox, Adam Adam.Fox@netapp.com
wrote:
If you can take the data down for a single pass, then vol copy is
pretty
easy. If not, and you have a SnapMirror license, you could do your first
pass
while the data is being accessed, then do an incremental while it's down.
This assumes that you have the same volume type (trad/flex) on each
side.
If not, you'll have to use a more logical way of moving data like
ndmpcopy
or Qtree-based SnapMirror.
-- Adam Fox adamfox@netapp.com
snapmirror is not an option,
thanx though,
is this a trick question? Why do you have a snapmirror license if you weren't going to use it to transfer data from one filer to another? Is this your cryptic way of saying it's to slow and you tried scp instead? There's a story here that I'm not getting. I'd like to see how you do with snapmirror see if that's good enough or not.
-Blake
On Tue, Jun 3, 2008 at 10:54 PM, Steve Rieger lists@up-south.com wrote:
Snapmirror a snapshot ? We have snapmirror, snapvault, cifs, nfs, and many more, money is not the issue speed is Sent via BlackBerry from T-Mobile
-----Original Message----- From: "Glenn Walker" ggwalker@mindspring.com
Date: Wed, 4 Jun 2008 01:12:48 To:"Blake Golliher" thelastman@gmail.com,lists@up-south.com Cc:"Fox, Adam" Adam.Fox@netapp.com,toasters@mathworks.com Subject: RE: snapshot copy
There's always the possibility of getting a demo license of snapmirror.
I know it's not cheap, but this is exactly what it is designed to do and is well worth the $.
Glenn
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Blake Golliher Sent: Wednesday, June 04, 2008 12:52 AM To: lists@up-south.com Cc: Fox, Adam; toasters@mathworks.com Subject: Re: snapshot copy
You can still ndmpcopy just a directory. Give it a path like /vol/root/etc will just copy the etc directory. Try /vol/root/.snapshot/hourly.0/etc/ that should work.
-Blake
On Tue, Jun 3, 2008 at 9:48 PM, Steve Rieger lists@up-south.com wrote:
Ndmcopy is volume level, i need to copy a napshot. The vol is alive.
I mounted the snapshot and am using scp to copy it to dest filer.
Sent via BlackBerry from T-Mobile
-----Original Message----- From: "Blake Golliher" thelastman@gmail.com
Date: Tue, 3 Jun 2008 21:30:39 To:"Mailing Lists" lists@up-south.com Cc:"Fox, Adam" Adam.Fox@netapp.com, toasters@mathworks.com Subject: Re: snapshot copy
ndmpcopy might work. It's filer to filer, and works fairly well, unless your problem is regarding a lot of files. It'll work well enough for a few thousand, but getting into several million it gets a bit nasty, but then any file based replication method for millions of files tends to suck.
It doesn't need a license, and is trival to setup and use on the filer console.
-Blake
On Tue, Jun 3, 2008 at 3:14 PM, Mailing Lists lists@up-south.com
wrote:
On Tue, Jun 3, 2008 at 3:05 PM, Fox, Adam Adam.Fox@netapp.com
wrote:
If you can take the data down for a single pass, then vol copy is
pretty
easy. If not, and you have a SnapMirror license, you could do your first
pass
while the data is being accessed, then do an incremental while it's down.
This assumes that you have the same volume type (trad/flex) on each
side.
If not, you'll have to use a more logical way of moving data like
ndmpcopy
or Qtree-based SnapMirror.
-- Adam Fox adamfox@netapp.com
snapmirror is not an option,
thanx though,
There's quite a bit of details missing here that would help us solve the problem.
* What type of filers are the source and destination * What type of volumes * What kind of data (lots of small files, few large files, etc..) * How busy are the filers * How much data needs to be moved * Is this a one-time or a repeated operation * If repeated, how much is the source changing * What's the physical distance/ping-time between the filers * What kind of bandwidth is supported between the filers * What kind of transfer rate do you need <<--- this is an important one. * What licenses do you have (on each filer) * What have you already tried
-George
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Blake Golliher Sent: Tuesday, June 03, 2008 11:05 PM To: lists@up-south.com Cc: Glenn Walker; Fox, Adam; toasters@mathworks.com Subject: Re: snapshot copy
is this a trick question? Why do you have a snapmirror license if you weren't going to use it to transfer data from one filer to another? Is this your cryptic way of saying it's to slow and you tried scp instead? There's a story here that I'm not getting. I'd like to see how you do with snapmirror see if that's good enough or not.
-Blake
On Tue, Jun 3, 2008 at 10:54 PM, Steve Rieger lists@up-south.com wrote:
Snapmirror a snapshot ? We have snapmirror, snapvault, cifs, nfs, and many more, money is not
the issue speed is
Sent via BlackBerry from T-Mobile
-----Original Message----- From: "Glenn Walker" ggwalker@mindspring.com
Date: Wed, 4 Jun 2008 01:12:48 To:"Blake Golliher" thelastman@gmail.com,lists@up-south.com Cc:"Fox, Adam" Adam.Fox@netapp.com,toasters@mathworks.com Subject: RE: snapshot copy
There's always the possibility of getting a demo license of snapmirror.
I know it's not cheap, but this is exactly what it is designed to do and is well worth the $.
Glenn
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Blake Golliher Sent: Wednesday, June 04, 2008 12:52 AM To: lists@up-south.com Cc: Fox, Adam; toasters@mathworks.com Subject: Re: snapshot copy
You can still ndmpcopy just a directory. Give it a path like /vol/root/etc will just copy the etc directory. Try /vol/root/.snapshot/hourly.0/etc/ that should work.
-Blake
On Tue, Jun 3, 2008 at 9:48 PM, Steve Rieger lists@up-south.com wrote:
Ndmcopy is volume level, i need to copy a napshot. The vol is alive.
I mounted the snapshot and am using scp to copy it to dest filer.
Sent via BlackBerry from T-Mobile
-----Original Message----- From: "Blake Golliher" thelastman@gmail.com
Date: Tue, 3 Jun 2008 21:30:39 To:"Mailing Lists" lists@up-south.com Cc:"Fox, Adam" Adam.Fox@netapp.com, toasters@mathworks.com Subject: Re: snapshot copy
ndmpcopy might work. It's filer to filer, and works fairly well, unless your problem is regarding a lot of files. It'll work well enough for a few thousand, but getting into several million it gets a bit nasty, but then any file based replication method for millions of files tends to suck.
It doesn't need a license, and is trival to setup and use on the filer console.
-Blake
On Tue, Jun 3, 2008 at 3:14 PM, Mailing Lists lists@up-south.com
wrote:
On Tue, Jun 3, 2008 at 3:05 PM, Fox, Adam Adam.Fox@netapp.com
wrote:
If you can take the data down for a single pass, then vol copy is
pretty
easy. If not, and you have a SnapMirror license, you could do your first
pass
while the data is being accessed, then do an incremental while it's down.
This assumes that you have the same volume type (trad/flex) on each
side.
If not, you'll have to use a more logical way of moving data like
ndmpcopy
or Qtree-based SnapMirror.
-- Adam Fox adamfox@netapp.com
snapmirror is not an option,
thanx though,
Steve> Snapmirror a snapshot ?
Well, when you start a snapshot or a snapmirror, it takes snapshot of the vol/qtree to make sure you're sending something consistent.
Steve> We have snapmirror, snapvault, cifs, nfs, and many more, money Steve> is not the issue speed is
Are you moving data over a LAN or WAN connection? I find that snapvault sucks rocks over the WAN (45mb/s, 100ms RTT). It's simpler and much faster to use:
lrep_reader bbcp <file(s)> <destdir> lrep_writer snapvault start -r -S <sfiler>:<srcvol> <dfiler>:<destvol>
It's manual, but it works well. I really recommend BBCP (http://www.slac.stanford.edu/~abh/bbcp/) for moving large files efficiently over Fast Wide WAN links.
John
John Stoffel - Senior Staff Systems Administrator - System LSI Group Toshiba America Electronic Components, Inc. - http://www.toshiba.com/taec john.stoffel@taec.toshiba.com - 508-486-1087
You can always try adjusting the wsize of the snapmirror connection over WAN. Could help...
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of John Stoffel Sent: Wednesday, June 04, 2008 1:39 PM To: lists@up-south.com Cc: Glenn Walker; Blake Golliher; Fox, Adam; toasters@mathworks.com Subject: Re: snapshot copy
Steve> Snapmirror a snapshot ?
Well, when you start a snapshot or a snapmirror, it takes snapshot of the vol/qtree to make sure you're sending something consistent.
Steve> We have snapmirror, snapvault, cifs, nfs, and many more, money Steve> is not the issue speed is
Are you moving data over a LAN or WAN connection? I find that snapvault sucks rocks over the WAN (45mb/s, 100ms RTT). It's simpler and much faster to use:
lrep_reader bbcp <file(s)> <destdir> lrep_writer snapvault start -r -S <sfiler>:<srcvol> <dfiler>:<destvol>
It's manual, but it works well. I really recommend BBCP (http://www.slac.stanford.edu/~abh/bbcp/) for moving large files efficiently over Fast Wide WAN links.
John
John Stoffel - Senior Staff Systems Administrator - System LSI Group Toshiba America Electronic Components, Inc. - http://www.toshiba.com/taec john.stoffel@taec.toshiba.com - 508-486-1087
Glenn> You can always try adjusting the wsize of the snapmirror Glenn> connection over WAN. Could help...
Nope, doesn't help. The default snapmirror tcp_window_size is already 2mb, which *should* be enough for my site, but doesn't do a damm thing to help.
Again, if you have a WAN link with high latency and high bandwidth, snapmirror/snapvault both suck in my experience. Which is why I'm spending my time doing lrep's by hand to rebuild DR snapvaults.
Friggin sucks...
Yes, NetApp's TCP implemention leaves much to be desired over WAN links. We had many long conversations with NetApp engineering, including tracking down a major bug with window scaling.
We found a substatial improvement when switching to ICS (i.e. use multi(src,dest)(src,dest) even with single end points). Apparently when using ICS it uses a different internal network stack/interface which is much better over the WAN. It still wasn't as good as our interim hack of proxying snapmirror connections via local Linux boxes (with their modern TCP stacks), but it ended up being good enough. WAN accelerators helped us greatly too.
--Dave
On Wed, Jun 4, 2008 at 12:58 PM, John Stoffel john.stoffel@taec.toshiba.com wrote:
Glenn> You can always try adjusting the wsize of the snapmirror Glenn> connection over WAN. Could help...
Nope, doesn't help. The default snapmirror tcp_window_size is already 2mb, which *should* be enough for my site, but doesn't do a damm thing to help.
Again, if you have a WAN link with high latency and high bandwidth, snapmirror/snapvault both suck in my experience. Which is why I'm spending my time doing lrep's by hand to rebuild DR snapvaults.
Friggin sucks...
Dave> Yes, NetApp's TCP implemention leaves much to be desired over Dave> WAN links. We had many long conversations with NetApp Dave> engineering, including tracking down a major bug with window Dave> scaling.
Do you know which versions of OnTap this referred to, and did they get any fixes into more recent versions? We're running 7.0.x on our filers, with 7.2.x on our R200s. If I *know* TCP performance will be improved, I'd work to upgrade them all to something newer.
Dave> We found a substatial improvement when switching to ICS Dave> (i.e. use multi(src,dest)(src,dest) even with single end Dave> points). Apparently when using ICS it uses a different internal Dave> network stack/interface which is much better over the WAN.
This is interesting, I've never heard of ICS. Does this work with Snapvault? Maybe I'll need to check the latest OnTap docs to see how to do this. Or could you post an example of how you set it up?
I'd love to try this out and see if it works better.
Dave> It still wasn't as good as our interim hack of proxying Dave> snapmirror connections via local Linux boxes (with their modern Dave> TCP stacks), but it ended up being good enough.
Now this sounds neat too. I should look into this since we have tons of fast opterons at our sites and it wouldn't be hard to setup a proxy for this. Do you have any pointers to how you did this, or what software you used?
Dave> WAN accelerators helped us greatly too.
They help, and we do get compression and speedup, but it's still hard to get the NetApp to actually *fill* the WAN pipe. Really frustrating.
What's worse is that LAN performance is great. Over the WAN I can *see* the sawtooth bandwidth of snapvaults when the TCP window scaling just screws up. Totally frustrating.
Thanks for your comments Dave.
John John Stoffel - Senior Staff Systems Administrator - System LSI Group Toshiba America Electronic Components, Inc. - http://www.toshiba.com/taec john.stoffel@taec.toshiba.com - 508-486-1087
We were having issues with snapmirror performance cross high latency(80ms) WAN, we only able to push replication speed to be 2-3MB/s , our requirement is at least 10MB/s, if anybody was able to do that, please let me know.
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of John Stoffel Sent: Wednesday, June 04, 2008 2:00 PM To: Dave Barr (鬼佬) Cc: John Stoffel; Glenn Walker; lists@up-south.com; Blake Golliher; Fox, Adam; toasters@mathworks.com Subject: Re: snapshot copy
Dave> Yes, NetApp's TCP implemention leaves much to be desired over Dave> WAN links. We had many long conversations with NetApp Dave> engineering, including tracking down a major bug with window Dave> scaling.
Do you know which versions of OnTap this referred to, and did they get any fixes into more recent versions? We're running 7.0.x on our filers, with 7.2.x on our R200s. If I *know* TCP performance will be improved, I'd work to upgrade them all to something newer.
Dave> We found a substatial improvement when switching to ICS Dave> (i.e. use multi(src,dest)(src,dest) even with single end Dave> points). Apparently when using ICS it uses a different internal Dave> network stack/interface which is much better over the WAN.
This is interesting, I've never heard of ICS. Does this work with Snapvault? Maybe I'll need to check the latest OnTap docs to see how to do this. Or could you post an example of how you set it up?
I'd love to try this out and see if it works better.
Dave> It still wasn't as good as our interim hack of proxying Dave> snapmirror connections via local Linux boxes (with their modern Dave> TCP stacks), but it ended up being good enough.
Now this sounds neat too. I should look into this since we have tons of fast opterons at our sites and it wouldn't be hard to setup a proxy for this. Do you have any pointers to how you did this, or what software you used?
Dave> WAN accelerators helped us greatly too.
They help, and we do get compression and speedup, but it's still hard to get the NetApp to actually *fill* the WAN pipe. Really frustrating.
What's worse is that LAN performance is great. Over the WAN I can *see* the sawtooth bandwidth of snapvaults when the TCP window scaling just screws up. Totally frustrating.
Thanks for your comments Dave.
John John Stoffel - Senior Staff Systems Administrator - System LSI Group Toshiba America Electronic Components, Inc. - http://www.toshiba.com/taec john.stoffel@taec.toshiba.com - 508-486-1087
We have WAN links at around 45ms, but have been able to push single replications up to 50MB/s or so (throttled to be good Samaritans on the network).
-----Original Message----- From: Li, Jackie (Yanhui) [mailto:YLi@ea.com] Sent: Wednesday, June 04, 2008 5:08 PM To: John Stoffel; Dave Barr (鬼佬) Cc: Glenn Walker; lists@up-south.com; Blake Golliher; Fox, Adam; toasters@mathworks.com Subject: RE: snapshot copy
We were having issues with snapmirror performance cross high latency(80ms) WAN, we only able to push replication speed to be 2-3MB/s , our requirement is at least 10MB/s, if anybody was able to do that, please let me know.
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of John Stoffel Sent: Wednesday, June 04, 2008 2:00 PM To: Dave Barr (鬼佬) Cc: John Stoffel; Glenn Walker; lists@up-south.com; Blake Golliher; Fox, Adam; toasters@mathworks.com Subject: Re: snapshot copy
Dave> Yes, NetApp's TCP implemention leaves much to be desired over Dave> WAN links. We had many long conversations with NetApp Dave> engineering, including tracking down a major bug with window Dave> scaling.
Do you know which versions of OnTap this referred to, and did they get any fixes into more recent versions? We're running 7.0.x on our filers, with 7.2.x on our R200s. If I *know* TCP performance will be improved, I'd work to upgrade them all to something newer.
Dave> We found a substatial improvement when switching to ICS Dave> (i.e. use multi(src,dest)(src,dest) even with single end Dave> points). Apparently when using ICS it uses a different internal Dave> network stack/interface which is much better over the WAN.
This is interesting, I've never heard of ICS. Does this work with Snapvault? Maybe I'll need to check the latest OnTap docs to see how to do this. Or could you post an example of how you set it up?
I'd love to try this out and see if it works better.
Dave> It still wasn't as good as our interim hack of proxying Dave> snapmirror connections via local Linux boxes (with their modern Dave> TCP stacks), but it ended up being good enough.
Now this sounds neat too. I should look into this since we have tons of fast opterons at our sites and it wouldn't be hard to setup a proxy for this. Do you have any pointers to how you did this, or what software you used?
Dave> WAN accelerators helped us greatly too.
They help, and we do get compression and speedup, but it's still hard to get the NetApp to actually *fill* the WAN pipe. Really frustrating.
What's worse is that LAN performance is great. Over the WAN I can *see* the sawtooth bandwidth of snapvaults when the TCP window scaling just screws up. Totally frustrating.
Thanks for your comments Dave.
John John Stoffel - Senior Staff Systems Administrator - System LSI Group Toshiba America Electronic Components, Inc. - http://www.toshiba.com/taec john.stoffel@taec.toshiba.com - 508-486-1087
Glenn> We have WAN links at around 45ms, but have been able to push Glenn> single replications up to 50MB/s or so (throttled to be good Glenn> Samaritans on the network).
Nice! Wish I could do as well. Even on our shorter (RTT wise) links, we still don't see as good a performance.
Are you using snapmirror or snapvault in this case? Also, I am using R200s and FAS9x0 and FAS270 boxes, none of the newer stuff.
Sounds like I need to review BURT 217596 and see about upgrading OnTap as well.
John
Ahh - we're running nothing older than 7.2.2 over the entire org. The boxes used for the big\fast snapmirrors are 7.2.3 and 7.2.4. Mostly 960s and a 6070 cluster.
Glenn
-----Original Message----- From: John Stoffel [mailto:john.stoffel@taec.toshiba.com] Sent: Thursday, June 05, 2008 9:28 AM To: Glenn Walker Cc: Li, Jackie (Yanhui); John Stoffel; Dave Barr (鬼佬); lists@up-south.com; Blake Golliher; Fox, Adam; toasters@mathworks.com Subject: RE: snapshot copy
Glenn> We have WAN links at around 45ms, but have been able to push Glenn> single replications up to 50MB/s or so (throttled to be good Glenn> Samaritans on the network).
Nice! Wish I could do as well. Even on our shorter (RTT wise) links, we still don't see as good a performance.
Are you using snapmirror or snapvault in this case? Also, I am using R200s and FAS9x0 and FAS270 boxes, none of the newer stuff.
Sounds like I need to review BURT 217596 and see about upgrading OnTap as well.
John
Glenn> Ahh - we're running nothing older than 7.2.2 over the entire Glenn> org. The boxes used for the big\fast snapmirrors are 7.2.3 and Glenn> 7.2.4. Mostly 960s and a 6070 cluster. Glenn
That would possibly explain it then. I'm working now to upgrade to 7.2.5 since it's fixed both the TCP bug we've been talking about, as well as some other bugs that have hit us and caused crashes.
Knock on wood... :]
While you're at it and have the outage window, be sure to get the most current disk and shelf firmware as well. If there was any firmware released after the release of 7.2.5, it will never be packaged in the download, regardless of whan you download it.
Stetson M. Webster Onsite Professional Services Engineer PS - North Amer. - East
NetApp 919.250.0052 Mobile Stetson.Webster@netapp.com www.netapp.com
-----Original Message----- From: John Stoffel [mailto:john.stoffel@taec.toshiba.com] Sent: Friday, June 06, 2008 9:06 AM To: Glenn Walker Cc: John Stoffel; Li, Jackie (Yanhui); Dave Barr (鬼佬); lists@up-south.com; Blake Golliher; Fox, Adam; toasters@mathworks.com Subject: RE: snapshot copy
Glenn> Ahh - we're running nothing older than 7.2.2 over the entire Glenn> org. The boxes used for the big\fast snapmirrors are 7.2.3 and Glenn> 7.2.4. Mostly 960s and a 6070 cluster. Glenn
That would possibly explain it then. I'm working now to upgrade to 7.2.5 since it's fixed both the TCP bug we've been talking about, as well as some other bugs that have hit us and caused crashes.
Knock on wood... :]
On Wed, Jun 4, 2008 at 2:00 PM, John Stoffel john.stoffel@taec.toshiba.com wrote:
Dave> Yes, NetApp's TCP implemention leaves much to be desired over Dave> WAN links. We had many long conversations with NetApp Dave> engineering, including tracking down a major bug with window Dave> scaling.
Do you know which versions of OnTap this referred to, and did they get any fixes into more recent versions? We're running 7.0.x on our filers, with 7.2.x on our R200s. If I *know* TCP performance will be improved, I'd work to upgrade them all to something newer.
Tracking it down I'm pretty sure it's BURT 217596, first fixed in 7.2.1. AFAIK it wasn't backported to 7.0.
Dave> We found a substatial improvement when switching to ICS Dave> (i.e. use multi(src,dest)(src,dest) even with single end Dave> points). Apparently when using ICS it uses a different internal Dave> network stack/interface which is much better over the WAN.
This is interesting, I've never heard of ICS. Does this work with Snapvault? Maybe I'll need to check the latest OnTap docs to see how to do this. Or could you post an example of how you set it up?
Search the snapmirror.conf man page for "multi". Here's an example:
filer1_multi=multi(filer1,filer2)(filer1,filer2) filer1_multi:vol1 filer100:vol1 48 * * *
Dave> It still wasn't as good as our interim hack of proxying Dave> snapmirror connections via local Linux boxes (with their modern Dave> TCP stacks), but it ended up being good enough.
Now this sounds neat too. I should look into this since we have tons of fast opterons at our sites and it wouldn't be hard to setup a proxy for this. Do you have any pointers to how you did this, or what software you used?
Basically it worked by setting up virtual IPs on the proxy box, and with xinetd, setting up netcat (nc) to tunnel the connection to another box. The virtual IPs were there so we could set up a bunch of source/destination pairs, since snapmirror is hardwired to run over a single port. (You can't say to the filer, connect to "proxybox:50023", for example). Each IP on the proxy box was dedicated to serving a proxy through another proxy to a given remote filer. We did modify netcat in a small way, namely to use TCP keepalives. I may be able to send you the patch we used. It's a one liner.
What's worse is that LAN performance is great. Over the WAN I can *see* the sawtooth bandwidth of snapvaults when the TCP window scaling just screws up. Totally frustrating.
Yep that looks totally like what we saw with 217596. I suggest upgrading to 7.2.1+ or request a backport of 217596. If I recall the code change was pretty small.
--Dave
Be aware, there are issues when using ICS with older versions of snapdrive (5.0 works fine, 4.1 experienced issues).
-----Original Message----- From: Dave Barr (鬼佬) [mailto:barr@google.com] Sent: Wednesday, June 04, 2008 5:24 PM To: John Stoffel Cc: Glenn Walker; lists@up-south.com; Blake Golliher; Fox, Adam; toasters@mathworks.com Subject: Re: snapshot copy
On Wed, Jun 4, 2008 at 2:00 PM, John Stoffel john.stoffel@taec.toshiba.com wrote:
Dave> Yes, NetApp's TCP implemention leaves much to be desired over Dave> WAN links. We had many long conversations with NetApp Dave> engineering, including tracking down a major bug with window Dave> scaling.
Do you know which versions of OnTap this referred to, and did they get any fixes into more recent versions? We're running 7.0.x on our filers, with 7.2.x on our R200s. If I *know* TCP performance will be improved, I'd work to upgrade them all to something newer.
Tracking it down I'm pretty sure it's BURT 217596, first fixed in 7.2.1. AFAIK it wasn't backported to 7.0.
Dave> We found a substatial improvement when switching to ICS Dave> (i.e. use multi(src,dest)(src,dest) even with single end Dave> points). Apparently when using ICS it uses a different internal Dave> network stack/interface which is much better over the WAN.
This is interesting, I've never heard of ICS. Does this work with Snapvault? Maybe I'll need to check the latest OnTap docs to see how to do this. Or could you post an example of how you set it up?
Search the snapmirror.conf man page for "multi". Here's an example:
filer1_multi=multi(filer1,filer2)(filer1,filer2) filer1_multi:vol1 filer100:vol1 48 * * *
Dave> It still wasn't as good as our interim hack of proxying Dave> snapmirror connections via local Linux boxes (with their modern Dave> TCP stacks), but it ended up being good enough.
Now this sounds neat too. I should look into this since we have tons of fast opterons at our sites and it wouldn't be hard to setup a proxy for this. Do you have any pointers to how you did this, or what software you used?
Basically it worked by setting up virtual IPs on the proxy box, and with xinetd, setting up netcat (nc) to tunnel the connection to another box. The virtual IPs were there so we could set up a bunch of source/destination pairs, since snapmirror is hardwired to run over a single port. (You can't say to the filer, connect to "proxybox:50023", for example). Each IP on the proxy box was dedicated to serving a proxy through another proxy to a given remote filer. We did modify netcat in a small way, namely to use TCP keepalives. I may be able to send you the patch we used. It's a one liner.
What's worse is that LAN performance is great. Over the WAN I can *see* the sawtooth bandwidth of snapvaults when the TCP window scaling just screws up. Totally frustrating.
Yep that looks totally like what we saw with 217596. I suggest upgrading to 7.2.1+ or request a backport of 217596. If I recall the code change was pretty small.
--Dave
Ndmpcopy is actually directory level and you can give it snapshot name as well. Or use vol copy which is able to copy specific snapshot as well (which depends whether you need to preserve current destination contents or not). Vol copy would be faster.
С уважением / With best regards / Mit freundlichen Grüβen
--- Andrey Borzenkov Senior system engineer
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Steve Rieger Sent: Wednesday, June 04, 2008 8:48 AM To: Blake Golliher Cc: Fox, Adam; toasters@mathworks.com Subject: Re: snapshot copy
Ndmcopy is volume level, i need to copy a napshot. The vol is alive.
I mounted the snapshot and am using scp to copy it to dest filer.
Sent via BlackBerry from T-Mobile
-----Original Message----- From: "Blake Golliher" thelastman@gmail.com
Date: Tue, 3 Jun 2008 21:30:39 To:"Mailing Lists" lists@up-south.com Cc:"Fox, Adam" Adam.Fox@netapp.com, toasters@mathworks.com Subject: Re: snapshot copy
ndmpcopy might work. It's filer to filer, and works fairly well, unless your problem is regarding a lot of files. It'll work well enough for a few thousand, but getting into several million it gets a bit nasty, but then any file based replication method for millions of files tends to suck.
It doesn't need a license, and is trival to setup and use on the filer console.
-Blake
On Tue, Jun 3, 2008 at 3:14 PM, Mailing Lists lists@up-south.com wrote:
On Tue, Jun 3, 2008 at 3:05 PM, Fox, Adam Adam.Fox@netapp.com wrote:
If you can take the data down for a single pass, then vol copy is pretty easy. If not, and you have a SnapMirror license, you could do your first pass while the data is being accessed, then do an incremental while it's down.
This assumes that you have the same volume type (trad/flex) on each side. If not, you'll have to use a more logical way of moving data like ndmpcopy or Qtree-based SnapMirror.
-- Adam Fox adamfox@netapp.com
snapmirror is not an option,
thanx though,
If you have a snapmirror license, you can use it to setup a mirror. If the relationship isn't needed beyond that simply break it. You would use a QTree snapmirror to go directly from snapshot to dest vol OR you could use a volume snapmirror then do a snap restore on the destination.
If you don't have snapmirror you can do ndmpcopy between the filers.
Either option should be significantly faster than scp. ----- Original Message ----- From: Mailing Lists To: toasters@mathworks.com Sent: Tuesday, June 03, 2008 5:41 PM Subject: snapshot copy
two filers, have a need to copy the data from source:vol:snap1 to dest:vol
what would be the fastest way to do this, mounting and scp is pretty slow for me, is there some sort of mirroring that i can use to accomplish this ?
-- eats the blues for breakfast, does unix for rent, plays harp for food, will play the flute for kicks rides for the freedom scrapes for the challenge
I would create a either a vol clone or flexclone depending on the licensing and space available using the snapshot required and then use snapmirror or ndmp copy to transfer the data.
bluezman wrote:
two filers, have a need to copy the data from source:vol:snap1 to dest:vol
what would be the fastest way to do this, mounting and scp is pretty slow for me, is there some sort of mirroring that i can use to accomplish this ?
-- eats the blues for breakfast, does unix for rent, plays harp for food, will play the flute for kicks rides for the freedom scrapes for the challenge
-
Bill Holland -
Blake Golliher -
Borzenkov, Andrey -
Dave Barr (鬼佬) -
Fox, Adam -
George T Chen -
Glenn Walker -
ifstat -
John Stoffel -
Li, Jackie (Yanhui) -
Mailing Lists -
Steve Rieger
-
Webster, Stetson