I am getting ready to do a big data migration. I am planning on doing an initial level 0, and then multiple level 1's leading up to the big day.

I know the that native ndmpcopy in OnTap 7.1 can only do one level 1, and one level 2 for each level 0.

But I have read that if you get ndmpcopy or jnmdpcopy from ndmp.org that it supports multiple level 1 dumps based on the same level 0.

But I am getting the same errors I got with the native ndmpcopy when I use ndmpcopy and jndmpcopy from a unix host. I just grabbed the latest copies for ndmp.org yesterday.

It doesn't make sense to me why you would do multiple level 1 copies because a level 1 does not copy files that have changed since the previous level 1. To do this you must use a level 2 copy.

A level 1 copies files that have changed since the previous level 0. So if you do multiple level 1 copies, then you end up recopying a lot of files.

And as you have discovered, ndmpcopy doesn't like it when you try to do a level 1 copy on top of another one. You must go to level 2.

Here is what I have done:

Run a level 0 copy while the filer is still available to the users. For large copies, this may take many hours, maybe even more than a day.

Run a level 1 copy soon after the level 0 finishes. This will be much quicker, but may still take a few hours because it copies all files that changed since the START of the level 0 copy. The filer is still available to the users.

Start your downtime. Disable NFS, CIFS, etc on the filer. You may need to shut down or idle clients beforehand.

During the downtime run a level 2 copy that copies files that changed since the START of the level 1 copy. This should take considerably less time than the level 1 copy, minimizing your downtime.

Since most of us must schedule downtime in advance, the trick is to estimate how long the level 0 and level 1 dumps will run and start them so that the level 1 copy finishes as close as possible to the start of the downtime. If the level 0 copy finishes earlier than expected, then hold off starting the level 1 copy.

Since you are copying one source volume to multiple destination volumes, all on the same filer, I recommend running your level 0 copies sequentially, followed by your level 1 copies, and then the level 2s.

Steve Losen scl@virginia.edu phone: 434-924-0640

University of Virginia ITC Unix Support

jndmpcopy has a -level i option that allows you to do unlimited incrementals, not just 1,2,etc. Maybe that is what you are looking for?

Moshe

Paul Letta letta@jlab.org Sent by: owner-toasters@mathworks.com 10/11/2006 15:21 Please respond to letta@jlab.org

To toasters@mathworks.com cc

Subject ndmpcopy, jndmpcopy and multiple level 1 transfers

I am getting ready to do a big data migration. I am planning on doing an initial level 0, and then multiple level 1's leading up to the big day.

I know the that native ndmpcopy in OnTap 7.1 can only do one level 1, and one level 2 for each level 0.

But I have read that if you get ndmpcopy or jnmdpcopy from ndmp.org that it supports multiple level 1 dumps based on the same level 0.

But I am getting the same errors I got with the native ndmpcopy when I use ndmpcopy and jndmpcopy from a unix host. I just grabbed the latest copies for ndmp.org yesterday.

java jndmpcopy.JNdmpcopy -version

java version "1.4.2" gcj (GCC) 3.4.6 20060404 (Red Hat 3.4.6-3)

So I did a level 0, and then one level 1 that both worked just fine. Then I re-ran the level 1 with the same command, and get the errors below. This is the same error I get if I try this with the native ndmpcopy command on the filer itself.

I am moving about 1TB of data (about 16M files) off of 1 traditional volume onto multiple new flex vols on the same filer. If what I'm doing is flat out wrong, what would others suggest ? tar wouldn't give me the incrementals and rsync would take a long time with this number of files.

Suggestions ?

Thanks, Paul

java jndmpcopy.JNdmpcopy filer:/vol/vol0/source

filer:/vol/destination -sa xxxxxxx -da xxxxxxx -level 1 Nov 09 03:53:25 [ndmpd:0]: Now waiting for source and destination to finish. SrcFiler: Nov 09 03:53:25 [ndmpd:0]: DUMP: creating "/vol/vol0/../snapshot_for_backup.153" snapshot. SrcFiler: Nov 09 03:53:26 [ndmpd:0]: DUMP: Using subtree dump SrcFiler: Nov 09 03:53:42 [ndmpd:0]: DUMP: Date of this level 1 dump: Thu Nov 9 15:53:25 2006. SrcFiler: Nov 09 03:53:42 [ndmpd:0]: DUMP: Date of last level 0 dump: Thu Nov 9 15:48:10 2006. SrcFiler: Nov 09 03:53:42 [ndmpd:0]: DUMP: Dumping /vol/vol0/source to NDMP connection SrcFiler: Nov 09 03:53:42 [ndmpd:0]: DUMP: mapping (Pass I)[regular files] SrcFiler: Nov 09 03:53:43 [ndmpd:0]: DUMP: mapping (Pass II)[directories] SrcFiler: Nov 09 03:54:09 [ndmpd:0]: DUMP: estimated 18 KB. SrcFiler: Nov 09 03:54:09 [ndmpd:0]: DUMP: dumping (Pass III) [directories] DstFiler: Nov 09 03:54:13 [ndmpd:0]: RESTORE: Thu Nov 9 15:54:13 2006: Begin incremental restore DstFiler: Nov 09 03:54:13 [ndmpd:0]: RESTORE: This incremental dump does not follow the previous dump that you have restored. DstFiler: Nov 09 03:54:13 [ndmpd:0]: RESTORE: This dump is based on Thu Nov 9 15:48:10 2006. DstFiler: Nov 09 03:54:13 [ndmpd:0]: RESTORE: It should be based on Thu Nov 9 15:50:28 2006. DstFiler: Nov 09 03:54:13 [ndmpd:0]: RESTORE: Could not use tape. DstFiler: Nov 09 03:54:13 [ndmpd:0]: RESTORE: RESTORE IS ABORTED Nov 09 03:54:13 [ndmpd:0]: DstFiler just halted. Nov 09 03:54:13 [ndmpd:0]: Reason: ndmp_data_halted_internal_error : Dump aborted. SrcFiler: Nov 09 03:54:13 [ndmpd:0]: DUMP: dumping (Pass IV) [regular files] SrcFiler: Nov 09 03:54:17 [ndmpd:0]: DUMP: Network communication error SrcFiler: Nov 09 03:54:17 [ndmpd:0]: DUMP: DUMP IS ABORTED SrcFiler: Nov 09 03:54:18 [ndmpd:0]: DUMP: Deleting "/vol/vol0/../snapshot_for_backup.153" snapshot. Nov 09 03:54:19 [ndmpd:0]: SrcFiler just halted. Nov 09 03:54:19 [ndmpd:0]: Reason: ndmp_data_halted_internal_error : Dump aborted. Nov 09 03:54:19 [ndmpd:0]: jndmpcopy finished