toasters November 2013

toasters@lists.teaparty.net

26 participants
11 discussions

RE: Methods to offline transfer a large amount of data between two filers
by Strauss Wolfram 01 Nov '13

01 Nov '13

Hi Mike It doesn't have to be disk based storage but we have to be able to connect to it over Ethernet. So, if there is a way to have a portable tape drive with an Ethernet connection that can be used by smtape, then it would sound like a promising solution. It should be a portable tape drive since we have about ten branch sites and I don't think we are going to install tape drives at every site. Regards, Wolfram Strauss From: Mike Nye [mailto:Mike.Nye@xpanse.com.au] Sent: Freitag, 1. November 2013 11:15 To: Strauss Wolfram Subject: RE: Methods to offline transfer a large amount of data between two filers Hi Wolfram, Does it have to be a RAID box? How about SnapMirror to Tape (smtape) ? Kind regards, Mike Nye [Description: cid:image001.jpg@01C9F9A5.3B7B34A0] [Description: cid:image002.jpg@01C9F9A5.3B7B34A0] Mike Nye Team Lead Systems Engineer mike.nye(a)xpanse.com.au<mailto:mike.nye@xpanse.com.au> www.xpanse.com.au<http://www.xpanse.com.au/> Mob: Office: Fax: +61 407 772 465 +61 8 9322 6767 +61 8 9322 6077 18 Emerald Tce West Perth WA 6005 [Description: cid:image003.jpg@01C9F9A5.3B7B34A0] From: toasters-bounces(a)teaparty.net<mailto:toasters-bounces@teaparty.net> [mailto:toasters-bounces@teaparty.net] On Behalf Of Strauss Wolfram Sent: Friday, 1 November 2013 6:07 PM To: toasters(a)teaparty.net<mailto:toasters@teaparty.net> Subject: Methods to offline transfer a large amount of data between two filers The scenario is as follows: Main site with a filer used as a Nearstore System, in particular as a Snapvault destination. A branch site with a smaller filer backing up its data every night with Snapvault to the main site. Both sites are connected over a rather low bandwidth WAN connection. Restoring single files from the main site's Nearstore System to the branch site's filer is no big deal. But what if we have to restore a complete volume in the Terabyte size? Over the WAN connection this would take days to complete. Here comes in the offline transfer scenario like so 1. Make a dump of the volume from the Nearstore System to a portable raid box 2. Hop in the car with the raid box and drive to the branch site (about 100 kilometers away) 3. Restore the volume from the raid box to the branch site filer The crux is how to dump a volume to the portable raid box fast enough. I haven't found a way to directly connect external storage to a filer so we definitely need to use a computer to connect the raid box to and then dump the data from the filer over the network onto this computer. We expect to more or less saturate a 1 Gbits/s link. This would allow us to carry out a restore of a 1 Terabyte volume within one working day. So far we have tried - Copy the files directly over NFS / CIFS -> too slow as expected, especially if there are many files and what if the volume contains LUNs and/or qtrees? - Using NetApp's LREP tool (offline SnapVault) -> astonishingly too slow, we only achieved about 200 Mbits/s although neither the filer nor the laptop used in the scenario hit any resource limit - NetApp's dump command to dump the volume into a file on a second volume and then transfer the dump file over NFS onto the laptop and the raid box. On the branch site copy the dump file over NFS to a second volume on the filer and then restore the main volume from the dump file. -> Especially the creation of the dump file was quite slow and the method doubles space requirements on both filers Out of curiosity, since all the above methods were too slow, I tried the following. I connected a LUN over iSCSI to the laptop used in the scenario and copied the files from the LUN onto the raid box. I easily managed to saturate the 1 Gbits/s link even with many small files residing in the LUN. So basically, the filer, the laptop and the raid box are capable of saturating 1 Gbits/s. What I'm missing is a method to dump a hole volume, including all its sub structures like qtrees and LUNs, in a decent way out of a filer. Does anybody have tried something similar and found a performing solution or did anybody succeed with one of the three above mentions methods and if so, how exactly did your scenario look like? Regards, Wolfram Strauss

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters November 2013