1.       If this is Netapp to netapp you may try ndmcopy

2.       If you need something more vendor agnostic – you may want to tar the data at the source, copy this as a single file to the remote site and untar it there. You may copy every subdirectory separately in parallel – which will also speed-up things. You may also want to tune TCP window to overcome some of the latency issues – you’d have to use something like rsync over hpn-ssh to leverage it

 

From: Arnold de Leon [mailto:a-toasters@deleons.com]
Sent: Thursday, October 17, 2013 11:37 PM
To: toasters <toasters@teaparty.net>
Subject: Re: Slow copy of a directory full of files via an NFS client across a WAN
 

I just tried NFSv2, similar results.  Same deadly embrace of request/reply which is death with a link with latency.

 

For this current test case this took over 5 minutes to copy 11MB of data in 600 files.

 

For reference

 

NFS (WAN 20ms) to NFS (WAN 20ms): 5 minutes

NFS (WAN 20 ms) to local disk:  50 seconds

Local disk to NFS (WAN 20ms ): 3 minutes 50 seconds

Local disk to local disk: 0.285 seconds

NFS (LAN 1ms) to NFS (LAN 1ms):  5 seconds

 

[I see that Peter also added data.  This is a bit of throwback for me, I've been around long enough to deal with the fun Usenet spool directories and joys of lots of small files]

 

The irony here is all I want is to make a copy of a directory which would be a trivial operation on the filer itself, I just need another reference to the directory.  The filer can make the copy when data is modified.  I'm wondering if I can do this with the OnTap API (I don't know about the APIs yet).  The other possible work around is using a host that is on the same LAN as the filer to do the copies.  I just can't believe that there isn't a simple solution to this.

 

Thanks everyone for the ideas and information.

 

 arnold

 

 

 

 

On Thu, Oct 17, 2013 at 2:28 PM, Mike Horwath <drechsau@gmail.com> wrote:

Do you require NFSv3?

Why not try NFSv2 - that will bypass the multitude of *ATTR calls going on a bit and may be better.

Only testing will tell, of course.

 

---------------------------------------------------------------------
Intel Israel (74) Limited

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

_______________________________________________
Toasters mailing list
Toasters@teaparty.net
http://www.teaparty.net/mailman/listinfo/toasters