I had a need to transfer transfer a large amount of data between two filers, one running OnTAP 5.3.4 and one running OnTAP 5.3.4R2. I ran a level 0 ndmpcopy the day before the cutover, and then a level 1 the night of the cutover, in order to minimize downtime. However, after doing a compare of file sizes after the transfer, I found that some files which had been modified since the level 0 did not have the proper size on the destination.
Here is an example. Toaster1 is the source, and toaster2 is the destination. I have added some comments in UPPER CASE.
-------- # cd /toaster1/test # ls -l total 80 -rw-r--r-- 1 root other 37817 Jan 31 15:54 testfile SIZE OF ORIGINAL FILE IS 37817 BYTES.
# rsh toaster1 snap create vol0 testndmp creating snapshot....................................
# ndmpcopy toaster1:/vol/vol0/.snapshot/testndmp/test toaster2:/vol/vol0/test -level 0 Password: Connecting to toaster1. Connecting to toaster2. toaster1: CONNECT: Connection established. toaster2: CONNECT: Connection established. toaster1: LOG: DUMP: toaster1: LOG: Using subtree dump toaster1: LOG: DUMP: toaster1: LOG: Date of this level 0 dump: Mon Jan 31 15:55:38 2000. toaster1: LOG: DUMP: toaster1: LOG: Date of last level 0 dump: the epoch. [NORMAL LOOKING DUMP/RESTORE OUTPUT] toaster1: LOG: DUMP: toaster1: LOG: DUMP IS DONE toaster1: HALT: The operation was successful! Waiting for toaster2 to halt too. toaster2: HALT: The operation was successful! The transfer is complete. Elapsed time: 0 hours, 0 minutes, 37 seconds.
# rsh toaster1 snap delete testndmp deleting snapshot....................................
# cd /toaster2/test # ls -l total 68928 -rw-r--r-- 1 root other 37817 Jan 31 15:54 testfile -rw-rw-rw- 1 root root 35213312 Jan 31 15:57 restore_symboltable SIZE OF COPY IS 37817 BYTES, AS EXPECTED.
# cd /toaster1/test # vi testfile # ls -l total 80 -rw-r--r-- 1 root other 37639 Jan 31 16:02 testfile I HAVE MODIFIED THE FILE, SO IT IS NOW SHORTER (37639 BYTES).
# rsh -n toaster1 snap create vol0 testndmp creating snapshot....................................
# ndmpcopy toaster1:/vol/vol0/.snapshot/testndmp/test toaster2:/vol/vol0/test -level 1 Password: Connecting to toaster1. Connecting to toaster2. toaster1: CONNECT: Connection established. toaster2: CONNECT: Connection established. toaster1: LOG: DUMP: toaster1: LOG: Using subtree dump toaster1: LOG: DUMP: toaster1: LOG: Date of this level 1 dump: Mon Jan 31 16:03:47 2000. toaster1: LOG: DUMP: toaster1: LOG: Date of last level 0 dump: Mon Jan 31 15:55:38 2000. [NORMAL LOOKING DUMP/RESTORE OUTPUT] toaster1: LOG: DUMP: toaster1: LOG: DUMP IS DONE toaster1: HALT: The operation was successful! Waiting for toaster2 to halt too. toaster2: HALT: The operation was successful! The transfer is complete. Elapsed time: 0 hours, 0 minutes, 38 seconds.
# cd /toaster2/test # ls -l total 68928 -rw-r--r-- 1 root other 37817 Jan 31 16:02 testfile -rw-rw-rw- 1 root root 35213312 Jan 31 16:05 restore_symboltable THE TIMESTAMP ON THE COPY HAS CHANGED, BUT IT IS STILL 37817 BYTES!
# cd / # umount /toaster2 # mount toaster2:/vol/vol0 /toaster2 # cd /toaster2/test # ls -l total 68928 -rw-r--r-- 1 root other 37817 Jan 31 16:02 testfile -rw-rw-rw- 1 root root 35213312 Jan 31 16:05 restore_symboltable UNMOUNTED AND REMOUNTED TO MAKE SURE IT WASN'T A CACHING ISSUE. --------
Can anyone tell me what I might have done wrong?
-- Deron Johnson djohnson@amgen.com
Hi.
This is burt 20564, which we are going to bring online at NOW as soon as possible.
The behavior you see is a bug. On incrementals, the restore does not truncate the files that have been truncated on the source.
Let's say you had edited testfile (originally 37817 bytes) and totally changed the first 20000 bytes, and deleted the remaining 17817 bytes.
The incremental NDMPcopy would have written the new 20000 bytes, but would NOT delete the final 17817. So you'd have a mishmash of two versions of the file.
If you had made the file longer, everything would have worked. If the file had stayed the same length, everything would have worked. If the file had been deleted, everything would have worked. If the file had been renamed or moved, everything would have worked.
Unfortunately, you hit the bug - 20564.
It has been fixed in 5.3.5, so you should contact NetApp support and find out when it is available.
Sorry about the inconvenience. Next time just make your files longer. We want you to use more storage, not less. ;)
Seriously, though, I am sorry that this affected you, and feel free to contact me with more questions about it.
Stephen Manley Data Availability and Management Leadoff Hitter
I had a need to transfer transfer a large amount of data between two filers, one running OnTAP 5.3.4 and one running OnTAP 5.3.4R2. I ran a level 0 ndmpcopy the day before the cutover, and then a level 1 the night of the cutover, in order to minimize downtime. However, after doing a compare of file sizes after the transfer, I found that some files which had been modified since the level 0 did not have the proper size on the destination.
Here is an example. Toaster1 is the source, and toaster2 is the destination. I have added some comments in UPPER CASE.
# cd /toaster1/test # ls -l total 80 -rw-r--r-- 1 root other 37817 Jan 31 15:54 testfile SIZE OF ORIGINAL FILE IS 37817 BYTES.
# rsh toaster1 snap create vol0 testndmp creating snapshot....................................
# ndmpcopy toaster1:/vol/vol0/.snapshot/testndmp/test toaster2:/vol/vol0/test -level 0 Password: Connecting to toaster1. Connecting to toaster2. toaster1: CONNECT: Connection established. toaster2: CONNECT: Connection established. toaster1: LOG: DUMP: toaster1: LOG: Using subtree dump toaster1: LOG: DUMP: toaster1: LOG: Date of this level 0 dump: Mon Jan 31 15:55:38 2000. toaster1: LOG: DUMP: toaster1: LOG: Date of last level 0 dump: the epoch. [NORMAL LOOKING DUMP/RESTORE OUTPUT] toaster1: LOG: DUMP: toaster1: LOG: DUMP IS DONE toaster1: HALT: The operation was successful! Waiting for toaster2 to halt too. toaster2: HALT: The operation was successful! The transfer is complete. Elapsed time: 0 hours, 0 minutes, 37 seconds.
# rsh toaster1 snap delete testndmp deleting snapshot....................................
# cd /toaster2/test # ls -l total 68928 -rw-r--r-- 1 root other 37817 Jan 31 15:54 testfile -rw-rw-rw- 1 root root 35213312 Jan 31 15:57 restore_symboltable SIZE OF COPY IS 37817 BYTES, AS EXPECTED.
# cd /toaster1/test # vi testfile # ls -l total 80 -rw-r--r-- 1 root other 37639 Jan 31 16:02 testfile I HAVE MODIFIED THE FILE, SO IT IS NOW SHORTER (37639 BYTES).
# rsh -n toaster1 snap create vol0 testndmp creating snapshot....................................
# ndmpcopy toaster1:/vol/vol0/.snapshot/testndmp/test toaster2:/vol/vol0/test -level 1 Password: Connecting to toaster1. Connecting to toaster2. toaster1: CONNECT: Connection established. toaster2: CONNECT: Connection established. toaster1: LOG: DUMP: toaster1: LOG: Using subtree dump toaster1: LOG: DUMP: toaster1: LOG: Date of this level 1 dump: Mon Jan 31 16:03:47 2000. toaster1: LOG: DUMP: toaster1: LOG: Date of last level 0 dump: Mon Jan 31 15:55:38 2000. [NORMAL LOOKING DUMP/RESTORE OUTPUT] toaster1: LOG: DUMP: toaster1: LOG: DUMP IS DONE toaster1: HALT: The operation was successful! Waiting for toaster2 to halt too. toaster2: HALT: The operation was successful! The transfer is complete. Elapsed time: 0 hours, 0 minutes, 38 seconds.
# cd /toaster2/test # ls -l total 68928 -rw-r--r-- 1 root other 37817 Jan 31 16:02 testfile -rw-rw-rw- 1 root root 35213312 Jan 31 16:05 restore_symboltable THE TIMESTAMP ON THE COPY HAS CHANGED, BUT IT IS STILL 37817 BYTES!
# cd / # umount /toaster2 # mount toaster2:/vol/vol0 /toaster2 # cd /toaster2/test # ls -l total 68928 -rw-r--r-- 1 root other 37817 Jan 31 16:02 testfile -rw-rw-rw- 1 root root 35213312 Jan 31 16:05 restore_symboltable UNMOUNTED AND REMOUNTED TO MAKE SURE IT WASN'T A CACHING ISSUE.
Can anyone tell me what I might have done wrong?
-- Deron Johnson djohnson@amgen.com
YOW!! This looks like a pretty severe problem! Does this affect ALL incremental restores using NDMP on DOT < 5.3.5???? Or is it just incrementals using NDMPcopy? This would be a rather large bug to have if if it affected more than just NDMPcopy but why would one have to upgrade DOT if it was a bug in NDMPcopy - so can you provide more info? What OS versions are affected, under what conditions, is it any NDMP incremental operations, (ie tape restore as well as NDMPcopy?) etc?
Justin Acklin Philips Semiconductor STB
Stephen Manley wrote:
Hi.
This is burt 20564, which we are going to bring online at NOW as soon as possible.
The behavior you see is a bug. On incrementals, the restore does not truncate the files that have been truncated on the source.
Let's say you had edited testfile (originally 37817 bytes) and totally changed the first 20000 bytes, and deleted the remaining 17817 bytes.
The incremental NDMPcopy would have written the new 20000 bytes, but would NOT delete the final 17817. So you'd have a mishmash of two versions of the file.
If you had made the file longer, everything would have worked. If the file had stayed the same length, everything would have worked. If the file had been deleted, everything would have worked. If the file had been renamed or moved, everything would have worked.
Unfortunately, you hit the bug - 20564.
It has been fixed in 5.3.5, so you should contact NetApp support and find out when it is available.
Sorry about the inconvenience. Next time just make your files longer. We want you to use more storage, not less. ;)
Seriously, though, I am sorry that this affected you, and feel free to contact me with more questions about it.
Stephen Manley Data Availability and Management Leadoff Hitter
I had a need to transfer transfer a large amount of data between two filers, one running OnTAP 5.3.4 and one running OnTAP 5.3.4R2. I ran a level 0 ndmpcopy the day before the cutover, and then a level 1 the night of the cutover, in order to minimize downtime. However, after doing a compare of file sizes after the transfer, I found that some files which had been modified since the level 0 did not have the proper size on the destination.
Here is an example. Toaster1 is the source, and toaster2 is the destination. I have added some comments in UPPER CASE.
# cd /toaster1/test # ls -l total 80 -rw-r--r-- 1 root other 37817 Jan 31 15:54 testfile SIZE OF ORIGINAL FILE IS 37817 BYTES.
# rsh toaster1 snap create vol0 testndmp creating snapshot....................................
# ndmpcopy toaster1:/vol/vol0/.snapshot/testndmp/test toaster2:/vol/vol0/test -level 0 Password: Connecting to toaster1. Connecting to toaster2. toaster1: CONNECT: Connection established. toaster2: CONNECT: Connection established. toaster1: LOG: DUMP: toaster1: LOG: Using subtree dump toaster1: LOG: DUMP: toaster1: LOG: Date of this level 0 dump: Mon Jan 31 15:55:38 2000. toaster1: LOG: DUMP: toaster1: LOG: Date of last level 0 dump: the epoch. [NORMAL LOOKING DUMP/RESTORE OUTPUT] toaster1: LOG: DUMP: toaster1: LOG: DUMP IS DONE toaster1: HALT: The operation was successful! Waiting for toaster2 to halt too. toaster2: HALT: The operation was successful! The transfer is complete. Elapsed time: 0 hours, 0 minutes, 37 seconds.
# rsh toaster1 snap delete testndmp deleting snapshot....................................
# cd /toaster2/test # ls -l total 68928 -rw-r--r-- 1 root other 37817 Jan 31 15:54 testfile -rw-rw-rw- 1 root root 35213312 Jan 31 15:57 restore_symboltable SIZE OF COPY IS 37817 BYTES, AS EXPECTED.
# cd /toaster1/test # vi testfile # ls -l total 80 -rw-r--r-- 1 root other 37639 Jan 31 16:02 testfile I HAVE MODIFIED THE FILE, SO IT IS NOW SHORTER (37639 BYTES).
# rsh -n toaster1 snap create vol0 testndmp creating snapshot....................................
# ndmpcopy toaster1:/vol/vol0/.snapshot/testndmp/test toaster2:/vol/vol0/test -level 1 Password: Connecting to toaster1. Connecting to toaster2. toaster1: CONNECT: Connection established. toaster2: CONNECT: Connection established. toaster1: LOG: DUMP: toaster1: LOG: Using subtree dump toaster1: LOG: DUMP: toaster1: LOG: Date of this level 1 dump: Mon Jan 31 16:03:47 2000. toaster1: LOG: DUMP: toaster1: LOG: Date of last level 0 dump: Mon Jan 31 15:55:38 2000. [NORMAL LOOKING DUMP/RESTORE OUTPUT] toaster1: LOG: DUMP: toaster1: LOG: DUMP IS DONE toaster1: HALT: The operation was successful! Waiting for toaster2 to halt too. toaster2: HALT: The operation was successful! The transfer is complete. Elapsed time: 0 hours, 0 minutes, 38 seconds.
# cd /toaster2/test # ls -l total 68928 -rw-r--r-- 1 root other 37817 Jan 31 16:02 testfile -rw-rw-rw- 1 root root 35213312 Jan 31 16:05 restore_symboltable THE TIMESTAMP ON THE COPY HAS CHANGED, BUT IT IS STILL 37817 BYTES!
# cd / # umount /toaster2 # mount toaster2:/vol/vol0 /toaster2 # cd /toaster2/test # ls -l total 68928 -rw-r--r-- 1 root other 37817 Jan 31 16:02 testfile -rw-rw-rw- 1 root root 35213312 Jan 31 16:05 restore_symboltable UNMOUNTED AND REMOUNTED TO MAKE SURE IT WASN'T A CACHING ISSUE.
Can anyone tell me what I might have done wrong?
-- Deron Johnson djohnson@amgen.com
3
YOW!! This looks like a pretty severe problem! Does this affect ALL incremental restores using NDMP on DOT < 5.3.5???? Or is it just incrementals using NDMPcopy? This would be a rather large bug to have if if it affected more than just NDMPcopy but why would one have to upgrade DOT if it was a bug in NDMPcopy - so can you provide more info? What OS versions are affected, under what conditions, is it any NDMP incremental operations, (ie tape restore as well as NDMPcopy?) etc?
Hopefully this will help.
burt20564 is the bug that incremental NDMPcopies do not truncate files. burt20564 was introduced in 5.3 and fixed in 5.3.5
NDMP operations NOT AFFECTED by burt20564 ----------------------------------------- 1. NDMP level 0 restores _from tape_ 2. NDMP incremental restores _from tape_ 3. NDMPcopy level 0's . 4. Ontap releases before 5.3 5. Ontap release including and after 5.3.5
NDMP operations that ARE AFFECTED by burt20564 ---------------------------------------------- 1. incremental NDMPcopies using Ontap 5.3 to 5.3.4*
As a workaround, you can either just use level 0 NDMPcopies or a dump piped to restore from a Unix client.
For example, unix% rsh filer dump 1uf - /vol/vol0/dir | rsh filer2 restore xfD - /vol/vol1/rstdir The performance will not be as good as NDMPcopy, though.
Non-NDMP dump/restore operations NOT AFFECTED by burt20564 ----------------------------------------------------------- 1. restore 'r' or 'R' level 0 restores 2. restore 'x' level 0 restores 3. restore 'x' incremental restores
Non-NDMP dump/restore operations that ARE AFFECTED by burt20564 --------------------------------------------------------------- 1. restore 'r' or 'R' incremental restores
Thus, as a workaround, use the 'x' option for incremental restores.
As you might be able to guess, incremental NDMPcopies use the underlying restore 'r' option, which is why they are both broken. NDMP clients (e.g. BudTool, Legato, Veritas, WorkStation Solutions) use the 'x' option, so they don't encounter the problem.
Of course, when 5.3.5 is available, I encourage upgrades for those who want to use incremental NDMPcopies. Once again, I apologize for the inconvenience (and the panic I caused Justin).
If you have any more concerns, give me a holler.
Stephen Manley Data Availability and Management Spokesmodel
Stephen Manley wrote:
Hi.
This is burt 20564, which we are going to bring online at NOW as soon as possible.
The behavior you see is a bug. On incrementals, the restore does not truncate the files that have been truncated on the source.
Let's say you had edited testfile (originally 37817 bytes) and totally changed the first 20000 bytes, and deleted the remaining 17817 bytes.
The incremental NDMPcopy would have written the new 20000 bytes, but would NOT delete the final 17817. So you'd have a mishmash of two versions of the file.
If you had made the file longer, everything would have worked. If the file had stayed the same length, everything would have worked. If the file had been deleted, everything would have worked. If the file had been renamed or moved, everything would have worked.
Unfortunately, you hit the bug - 20564.
It has been fixed in 5.3.5, so you should contact NetApp support and find out when it is available.
Sorry about the inconvenience. Next time just make your files longer. We want you to use more storage, not less. ;)
Seriously, though, I am sorry that this affected you, and feel free to contact me with more questions about it.
Stephen Manley Data Availability and Management Leadoff Hitter
I had a need to transfer transfer a large amount of data between two filers, one running OnTAP 5.3.4 and one running OnTAP 5.3.4R2. I ran a level 0 ndmpcopy the day before the cutover, and then a level 1 the night of the cutover, in order to minimize downtime. However, after doing a compare of file sizes after the transfer, I found that some files which had been modified since the level 0 did not have the proper size on the destination.
Here is an example. Toaster1 is the source, and toaster2 is the destination. I have added some comments in UPPER CASE.
# cd /toaster1/test # ls -l total 80 -rw-r--r-- 1 root other 37817 Jan 31 15:54 testfile SIZE OF ORIGINAL FILE IS 37817 BYTES.
# rsh toaster1 snap create vol0 testndmp creating snapshot....................................
# ndmpcopy toaster1:/vol/vol0/.snapshot/testndmp/test toaster2:/vol/vol0/test -level 0 Password: Connecting to toaster1. Connecting to toaster2. toaster1: CONNECT: Connection established. toaster2: CONNECT: Connection established. toaster1: LOG: DUMP: toaster1: LOG: Using subtree dump toaster1: LOG: DUMP: toaster1: LOG: Date of this level 0 dump: Mon Jan 31 15:55:38 2000. toaster1: LOG: DUMP: toaster1: LOG: Date of last level 0 dump: the epoch. [NORMAL LOOKING DUMP/RESTORE OUTPUT] toaster1: LOG: DUMP: toaster1: LOG: DUMP IS DONE toaster1: HALT: The operation was successful! Waiting for toaster2 to halt too. toaster2: HALT: The operation was successful! The transfer is complete. Elapsed time: 0 hours, 0 minutes, 37 seconds.
# rsh toaster1 snap delete testndmp deleting snapshot....................................
# cd /toaster2/test # ls -l total 68928 -rw-r--r-- 1 root other 37817 Jan 31 15:54 testfile -rw-rw-rw- 1 root root 35213312 Jan 31 15:57 restore_symboltable SIZE OF COPY IS 37817 BYTES, AS EXPECTED.
# cd /toaster1/test # vi testfile # ls -l total 80 -rw-r--r-- 1 root other 37639 Jan 31 16:02 testfile I HAVE MODIFIED THE FILE, SO IT IS NOW SHORTER (37639 BYTES).
# rsh -n toaster1 snap create vol0 testndmp creating snapshot....................................
# ndmpcopy toaster1:/vol/vol0/.snapshot/testndmp/test toaster2:/vol/vol0/test -level 1 Password: Connecting to toaster1. Connecting to toaster2. toaster1: CONNECT: Connection established. toaster2: CONNECT: Connection established. toaster1: LOG: DUMP: toaster1: LOG: Using subtree dump toaster1: LOG: DUMP: toaster1: LOG: Date of this level 1 dump: Mon Jan 31 16:03:47 2000. toaster1: LOG: DUMP: toaster1: LOG: Date of last level 0 dump: Mon Jan 31 15:55:38 2000. [NORMAL LOOKING DUMP/RESTORE OUTPUT] toaster1: LOG: DUMP: toaster1: LOG: DUMP IS DONE toaster1: HALT: The operation was successful! Waiting for toaster2 to halt too. toaster2: HALT: The operation was successful! The transfer is complete. Elapsed time: 0 hours, 0 minutes, 38 seconds.
# cd /toaster2/test # ls -l total 68928 -rw-r--r-- 1 root other 37817 Jan 31 16:02 testfile -rw-rw-rw- 1 root root 35213312 Jan 31 16:05 restore_symboltable THE TIMESTAMP ON THE COPY HAS CHANGED, BUT IT IS STILL 37817 BYTES!
# cd / # umount /toaster2 # mount toaster2:/vol/vol0 /toaster2 # cd /toaster2/test # ls -l total 68928 -rw-r--r-- 1 root other 37817 Jan 31 16:02 testfile -rw-rw-rw- 1 root root 35213312 Jan 31 16:05 restore_symboltable UNMOUNTED AND REMOUNTED TO MAKE SURE IT WASN'T A CACHING ISSUE.
Can anyone tell me what I might have done wrong?
-- Deron Johnson djohnson@amgen.com
3
-
Deron Johnson -
Justin -
stephen＠netapp.com