Jeff,
I recall seeing an email from you stating that you had problem with incremental NDMPcopy the last time you used it. Specifically many of the files in the destination directory were corrupted or mangled. You had to resort to rsync to fix this problem. Is this still the case with incremental NDMPcopy. What's your confidence level with it now? Can incremental NDMPcopy be the "trusted" tool for safely and effectively migrating/moving data between the filers? Thanks.
Peter
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com]On Behalf Of Jeffrey Krueger Sent: Thursday, April 19, 2001 3:39 PM To: toasters@mathworks.com Cc: ndmp-users@ndmp.org Subject: ndmpcopy patch -- "level i"
Thanks to Bruce for posting his patch to NDMPcopy! Here's the one that we've been using and all the information we've collected about it.
I've uploaded the following patch for ndmpcopy to ftp.ndmp.org in addition to attaching it to this e-mail. Please read the entire mail before attempting to apply the patch or use the functionality it provides.
---------------- WHAT DOES IT DO? ---------------- This is a very small, straight-forward patch that will enable ndmpcopy to make an indefinite number of incremental copies of data. The ability to do this was not included in ndmpcopy, but was a part of jndmpcopy - the Java version of ndmpcopy. The feature is used by passing the command-line argument "-level i" to ndmpcopy once the patch is compiled in.
The incremental feature is very useful for migrating a directory from one filer/volume to another. It allows you to make an initial copy and then perform periodical incremental updates of the changes until a cut-over period. During the cut-over period, you can make the source read-only or unaccessible and perform a final incremental copy while updating clients to point at the new location.
Without "level i", you can only do 9 incremental copies (level 1-9). With this enhancement, you can do as many incremental copies as you need.
----------------- HOW DOES IT WORK? ----------------- Our analysis of jndmpcopy showed that this feature was driven by simply replacing the NDMP environment variable "LEVEL" which is normally set to a number 0-9, with the NDMP environment variable "REPLICATE" and a value of "y". Again, this action is triggered in the patch by specifying "-level i" on the command line.
Normally, ndmpcopy gives the following output near the beginning of a copy:
disposal: LOG: DUMP: disposal: LOG: Date of this level 0 dump: Tue Apr 17 14:22:43 2001. disposal: LOG: DUMP: disposal: LOG: Date of last level 0 dump: the epoch.
When using the "-level i" option, that output will look like this:
disposal: LOG: DUMP: disposal: LOG: Date of this replication dump: Tue Apr 17 14:24:45 2001. disposal: LOG: DUMP: disposal: LOG: Date of this last update : the epoch.
It adds a line to /etc/dumpdates for the directory you copied which contains "i" for the level. Presumably this shouldn't be messed with as it keeps track of what the next level "i" for that directory needs to copy. I also assume that this means you can't do a level "i" of /qtree from filerA to filerB, then do another one from filerA to filerC, then back to filerB again.
This works like a dandy for us running Data ONTAP 5.3.4R3P2 and 5.3.7R3.
------- CAVEATS ------- o USE THIS PATCH AT YOUR OWN RISK! The patch isn't supported by myself, my company, or NetApp. Before using this feature, you should have a fair understanding of C programming and a good understanding of how NetApp filers work, especially with NDMP.
o I'm unsure what versions of ONTAP support the "REPLICATE" NDMP environment variable. So far I can't find any documentation of this, although jndmpcopy eludes to it working with ONTAP 5.3 and higher. Maybe someone from NetApp can chime in here?
o Level i copies will *not* work with level 0-9 copies of the same source and destination. That is, you cannot do a "-level 0" ndmpcopy to move data initially, then use "-level i" to make updates. For that matter, you can't use any other method to do the initial copy such as "vol copy", dump/restore, rsync, etc. If you want to use "-level i" at all, you must use it to do the initial copy and all updates.
o Don't modify the destination until you're done updating it. Once the destination is modified, another "level i" will likely *not* be able to update that destination.
o Be aware of "level i" entries in /etc/dumpdates! Don't delete a "level i" entry for a source if you still want to be able to do further updates to the same destination. Conversely, if you're done using this method to move a directory, and want to use it again on the same source (or a source which eventually has the same name) to a different (new, blank) destination, you'll have to remove the "level i" entry out of /etc/dumpdates for the source directory. Also, don't try to perform "level i" copies of the same source to multiple destinations.
-------------------- HOW DO I INSTALL IT? -------------------- There are two separate patches attached, one for version 1.1 and one for 1.2 of ndmpcopy (they vary only slightly because of the usage statement). If you are new to patching the following instructions may help you.
1) Copy the patch into the base directory of ndmpcopy. ex. % cp ndmpcopy-level_i.patch ~/ndmpcopy-1.1
2) Enter the base directory of ndmpcopy. ex. % cd ~/ndmpcopy-1.1
3) Patch the source (example uses GNU patch) ex. % patch -p 1 -b -z .orig -i ndmpcopy_1.1-level_i.patch
4) Build the package ex. % make clean; make
-------------------------------- ANYTHING ELSE, JEFF-THE-WINDBAG? -------------------------------- The contribution of this patch to the NetApp/NDMP community is intended to help folks out with administering their filers. This feature is pretty handy for migrating data between filers and/or volumes. Although I'm not offering any support of this patch, I am interested in any feedback regarding it so that it can be improved for everyone's benefit. In particular, I'm looking for:
o NetApp/NDMP official information about the REPLICATE variable - versions of ONTAP it can be used with, confirmation of the identified caveats, etc.
o User's experiences with the feature - does it work, does it fail, under what circumstances, etc.
o Programming tips - suggestions for improving the patch.
Also, I will be posting this along with this information in this e-mail and any feedback I get to my personal website. The URL will be posted in a follow-up mail in a few weeks.
Good luck!
-- Jeff
-- ---------------------------------------------------------------------------- Jeff Krueger, NetApp CA E-Mail: jeff@qualcomm.com Senior Engineer Phone: 858-651-6709 NetApp Filers / UNIX Infrastructure Fax: 858-651-6627 QUALCOMM, Inc. IT Engineering Web: www.qualcomm.com