---------------------- Forwarded by Thomas L. White/CHASE on 04/20/2001 11:31 AM ---------------------------
Thomas L. White 04/20/2001 11:32 AM
To: jkrueger@qualcomm.com cc: Subject: Re: ndmpcopy patch -- "level i"
my curiosity here is whether I should use the -i or 1-9 what would you suggest. And what do you know about the method used for level i or 1-9 ? to wit: will the ndmpcopy -i serve up every file and have the destination filer decidde if it needs to update or not ??? in other words will the incremental take 25 minutes or 5 hours ??
tw ---------------------- Forwarded by Thomas L. White/CHASE on 04/20/2001 11:23 AM ---------------------------
Thomas L. White 04/20/2001 09:54 AM
To: Jeffrey Krueger jkrueger@qualcomm.com cc: ptran@broadcom.com Subject: Re: ndmpcopy patch -- "level i" (Document link: Thomas L. White)
Jeff: We have a situation here that requires me to rebuild a (400G on 21X36G drive) volume onto 72G drives so - last night I ran ndmpcopy without any arguments. It seems to be running fine - 310G has been written so-far and I plan to let it run to completion this a.m. My problem is my boss - he has a 0 confidence level re: the incremental step of ndmpcopy and wants the next "update" to occur by way of robocopy and rdist. Since we plan to physically swap the drives saturday - I would guess this approach would not be finished by then. so - are you aware of any tool that can compare "source volume" to "destination volume" and give him the assurance that this pass and the update pass tonight completed successfully ???
thanks for your indulgence.
tw
Jeffrey Krueger jkrueger@qualcomm.com@mathworks.com on 04/19/2001 09:58:38 PM
Sent by: owner-toasters@mathworks.com
To: Peter Tran ptran@broadcom.com cc: Jeffrey Krueger jkrueger@qualcomm.com, toasters@mathworks.com Subject: Re: ndmpcopy patch -- "level i"
Peter:
Yes, we did have a problem with a level 1 copy based on a level 0 copy of data with ndmpcopy last year. Afterwards, we could not reproduce the problem and at some point another toaster user said they had the same problem and it was tracked down to a bug in a previous version of ONTAP. We had been wary of ndmpcopy for level 1-9 copies, but still used it with caution and had no difficulties.
The "level i" feature of ndmpcopy is even more attractive and has worked very nicely for us in the past few months. Our confidence level with ndmpcopy is very high these days.
As for if it can "be the trusted tool for safely and effectively migrating/moving data between filers", that's a question that only NetApp support can answer! =)
-- Jeff
On Thu, Apr 19, 2001 at 05:21:01PM -0700, Peter Tran wrote:
Jeff,
I recall seeing an email from you stating that you had problem with incremental NDMPcopy the last time you used it. Specifically many of the files in the destination directory were corrupted or mangled. You had to resort to rsync to fix this problem. Is this still the case with incremental NDMPcopy. What's your confidence level with it now? Can incremental NDMPcopy be the "trusted" tool for safely and effectively migrating/moving data between the filers? Thanks.
Peter
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com]On Behalf Of Jeffrey Krueger Sent: Thursday, April 19, 2001 3:39 PM To: toasters@mathworks.com Cc: ndmp-users@ndmp.org Subject: ndmpcopy patch -- "level i"
Thanks to Bruce for posting his patch to NDMPcopy! Here's the one that we've been using and all the information we've collected about it.
I've uploaded the following patch for ndmpcopy to ftp.ndmp.org in
addition
to attaching it to this e-mail. Please read the entire mail before attempting to apply the patch or use the functionality it provides.
WHAT DOES IT DO?
This is a very small, straight-forward patch that will enable ndmpcopy to make an indefinite number of incremental copies of data. The ability to
do
this was not included in ndmpcopy, but was a part of jndmpcopy - the Java version of ndmpcopy. The feature is used by passing the command-line argument "-level i" to ndmpcopy once the patch is compiled in.
The incremental feature is very useful for migrating a directory from one filer/volume to another. It allows you to make an initial copy and then perform periodical incremental updates of the changes until a cut-over period. During the cut-over period, you can make the source read-only or unaccessible and perform a final incremental copy while updating clients
to
point at the new location.
Without "level i", you can only do 9 incremental copies (level 1-9).
With
this enhancement, you can do as many incremental copies as you need.
HOW DOES IT WORK?
Our analysis of jndmpcopy showed that this feature was driven by simply replacing the NDMP environment variable "LEVEL" which is normally set to
a
number 0-9, with the NDMP environment variable "REPLICATE" and a value of "y". Again, this action is triggered in the patch by specifying "-level
i"
on the command line.
Normally, ndmpcopy gives the following output near the beginning of a
copy:
disposal: LOG: DUMP: disposal: LOG: Date of this level 0 dump: Tue Apr
17
14:22:43 2001. disposal: LOG: DUMP: disposal: LOG: Date of last level 0 dump: the
epoch.
When using the "-level i" option, that output will look like this:
disposal: LOG: DUMP: disposal: LOG: Date of this replication dump: Tue Apr 17 14:24:45 2001. disposal: LOG: DUMP: disposal: LOG: Date of this last update : the
epoch.
It adds a line to /etc/dumpdates for the directory you copied which contains "i" for the level. Presumably this shouldn't be messed with as
it
keeps track of what the next level "i" for that directory needs to copy.
I
also assume that this means you can't do a level "i" of /qtree from
filerA
to filerB, then do another one from filerA to filerC, then back to filerB again.
This works like a dandy for us running Data ONTAP 5.3.4R3P2 and 5.3.7R3.
CAVEATS
o USE THIS PATCH AT YOUR OWN RISK! The patch isn't supported by myself,
my
company, or NetApp. Before using this feature, you should have a fair understanding of C programming and a good understanding of how NetApp filers work, especially with NDMP.
o I'm unsure what versions of ONTAP support the "REPLICATE" NDMP environment variable. So far I can't find any documentation of this, although jndmpcopy eludes to it working with ONTAP 5.3 and higher. Maybe someone from NetApp can chime in here?
o Level i copies will *not* work with level 0-9 copies of the same source and destination. That is, you cannot do a "-level 0" ndmpcopy to move data initially, then use "-level i" to make updates. For that matter, you can't use any other method to do the initial copy such as "vol
copy",
dump/restore, rsync, etc. If you want to use "-level i" at all, you
must
use it to do the initial copy and all updates.
o Don't modify the destination until you're done updating it. Once the destination is modified, another "level i" will likely *not* be able to update that destination.
o Be aware of "level i" entries in /etc/dumpdates! Don't delete a "level
i"
entry for a source if you still want to be able to do further updates
to
the same destination. Conversely, if you're done using this method to move a directory, and want to use it again on the same source (or a source which eventually has the same name) to a different (new, blank) destination, you'll have to remove the "level i" entry out of /etc/dumpdates for the source directory. Also, don't try to perform "level i" copies of the same source to multiple destinations.
HOW DO I INSTALL IT?
There are two separate patches attached, one for version 1.1 and one for 1.2 of ndmpcopy (they vary only slightly because of the usage statement). If you are new to patching the following instructions may help you.
- Copy the patch into the base directory of ndmpcopy.
ex. % cp ndmpcopy-level_i.patch ~/ndmpcopy-1.1
- Enter the base directory of ndmpcopy.
ex. % cd ~/ndmpcopy-1.1
- Patch the source (example uses GNU patch)
ex. % patch -p 1 -b -z .orig -i ndmpcopy_1.1-level_i.patch
- Build the package
ex. % make clean; make
ANYTHING ELSE, JEFF-THE-WINDBAG?
The contribution of this patch to the NetApp/NDMP community is intended
to
help folks out with administering their filers. This feature is pretty handy for migrating data between filers and/or volumes. Although I'm not offering any support of this patch, I am interested in any feedback regarding it so that it can be improved for everyone's benefit. In particular, I'm looking for:
o NetApp/NDMP official information about the REPLICATE variable -
versions
of ONTAP it can be used with, confirmation of the identified caveats, etc.
o User's experiences with the feature - does it work, does it fail, under what circumstances, etc.
o Programming tips - suggestions for improving the patch.
Also, I will be posting this along with this information in this e-mail
and
any feedback I get to my personal website. The URL will be posted in a follow-up mail in a few weeks.
Good luck!
-- Jeff
--
----------------------------------------------------------------------------
Jeff Krueger, NetApp CA E-Mail: jeff@qualcomm.com Senior Engineer Phone: 858-651-6709 NetApp Filers / UNIX Infrastructure Fax: 858-651-6627 QUALCOMM, Inc. IT Engineering Web: www.qualcomm.com