I'm running into a recurring issue with backing up my NearStore and a long F760 filer.
In the procoess of backing some large filesystems (qtrees), it appears that NDMP is basically timing out and fails. I'm running Veritas NetBackup 6.0 (finally upgrading from 4.5 recently). I think the issue has to do with NDMP starting the dump commands on the filer, which then does a three-way backup to another filer with local tape drives.
The debug from ndmp logs shows for a manual full backup of a filer:
Sep 05 21:05:57 EDT [ndmpd:58]: Log message: DUMP: creating "/vol/vol0/../snapshot_for_backup.370" snapshot. Sep 05 21:06:05 EDT [ndmpd:58]: Log message: DUMP: Using Partial Volume Dump with Exclude Lists Sep 05 21:07:31 EDT [ndmpd:58]: Log message: DUMP: Date of this level 1 dump: Tue Sep 5 21:05:57 2006. Sep 05 21:07:31 EDT [ndmpd:58]: Log message: DUMP: Date of last level 0 dump: Fri Aug 18 19:20:57 2006. Sep 05 21:07:31 EDT [ndmpd:58]: Log message: DUMP: Dumping /vol/vol0/files3.rt to NDMP connection Sep 05 21:07:31 EDT [ndmpd:58]: Log message: DUMP: mapping (Pass I)[regular files]
It never completes the mapping of the files in Pass I. It just sits there. The filesystems I'm backing up are between 250-650GB with a LOT of small files (millions) and many subdirs. Tons of mail and html files.
My guess is that the amount of files and dirs are getting large that NDMP can't map them fully and is timing out after 8 hours. It shouldn't take that long to map files for the size of data it's doing; I've seem mapping times be much less for larger sets of data.
I know that the filesystem topology can cause NDMP backups to be slow, depending on file sizes, data layout, filer load, network etc, but this is happening on moderately busy filers or bone idle NearStores.
Anyone run across lengthy DUMP times?
Chewing through NOW and Veritas support site hasn't turned up anything obvious.
Just curious if others have run into NDMP/dump issues like this.
-Scott
Suggestions: 1. Try to dump to null as a test. 2. What is your ONTAP release? I believe there have been improvements in dump/restore performance for such environments in more recent releases.
Eyal.
On 9/6/06, Scott T. Mikusko guerilla@sw.xo.com wrote:
I'm running into a recurring issue with backing up my NearStore and a long F760 filer.
In the procoess of backing some large filesystems (qtrees), it appears that NDMP is basically timing out and fails. I'm running Veritas NetBackup 6.0 (finally upgrading from 4.5 recently). I think the issue has to do with NDMP starting the dump commands on the filer, which then does a three-way backup to another filer with local tape drives.
The debug from ndmp logs shows for a manual full backup of a filer:
Sep 05 21:05:57 EDT [ndmpd:58]: Log message: DUMP: creating "/vol/vol0/../snapshot_for_backup.370" snapshot. Sep 05 21:06:05 EDT [ndmpd:58]: Log message: DUMP: Using Partial Volume Dump with Exclude Lists Sep 05 21:07:31 EDT [ndmpd:58]: Log message: DUMP: Date of this level 1 dump: Tue Sep 5 21:05:57 2006. Sep 05 21:07:31 EDT [ndmpd:58]: Log message: DUMP: Date of last level 0 dump: Fri Aug 18 19:20:57 2006. Sep 05 21:07:31 EDT [ndmpd:58]: Log message: DUMP: Dumping /vol/vol0/files3.rt to NDMP connection Sep 05 21:07:31 EDT [ndmpd:58]: Log message: DUMP: mapping (Pass I)[regular files]
It never completes the mapping of the files in Pass I. It just sits there. The filesystems I'm backing up are between 250-650GB with a LOT of small files (millions) and many subdirs. Tons of mail and html files.
My guess is that the amount of files and dirs are getting large that NDMP can't map them fully and is timing out after 8 hours. It shouldn't take that long to map files for the size of data it's doing; I've seem mapping times be much less for larger sets of data.
I know that the filesystem topology can cause NDMP backups to be slow, depending on file sizes, data layout, filer load, network etc, but this is happening on moderately busy filers or bone idle NearStores.
Anyone run across lengthy DUMP times?
Chewing through NOW and Veritas support site hasn't turned up anything obvious.
Just curious if others have run into NDMP/dump issues like this.
-Scott
I think phase 1 is where it does a catalogue of all the files in a file system to be backed up. If you have millions of files this could take a long long time. So snapmirror to tape might be an option, thought I'm not sure if your backup software supports it. Also, you could try backing up smaller chunks a time to keep that first phase from killing you.
You could also look into extending the time out function of your ndmp software. That might be an option as well.
-Blake
On 9/6/06, Eyal Traitel etraitel@gmail.com wrote:
Suggestions:
- Try to dump to null as a test.
- What is your ONTAP release? I believe there have been improvements in
dump/restore performance for such environments in more recent releases.
Eyal.
On 9/6/06, Scott T. Mikusko guerilla@sw.xo.com wrote:
I'm running into a recurring issue with backing up my NearStore and a long F760 filer.
In the procoess of backing some large filesystems (qtrees), it appears that NDMP is basically timing out and fails. I'm running Veritas NetBackup 6.0 (finally upgrading from 4.5 recently). I think the issue has to do with NDMP starting the dump commands on the filer, which then does a three-way backup to another filer with local tape drives.
The debug from ndmp logs shows for a manual full backup of a filer:
Sep 05 21:05:57 EDT [ndmpd:58]: Log message: DUMP: creating "/vol/vol0/../snapshot_for_backup.370" snapshot. Sep 05 21:06:05 EDT [ndmpd:58]: Log message: DUMP: Using Partial Volume Dump with Exclude Lists Sep 05 21:07:31 EDT [ndmpd:58]: Log message: DUMP: Date of this level 1 dump: Tue Sep 5 21:05:57 2006. Sep 05 21:07:31 EDT [ndmpd:58]: Log message: DUMP: Date of last level 0 dump: Fri Aug 18 19:20:57 2006. Sep 05 21:07:31 EDT [ndmpd:58]: Log message: DUMP: Dumping /vol/vol0/files3.rt to NDMP connection Sep 05 21:07:31 EDT [ndmpd:58]: Log message: DUMP: mapping (Pass I)[regular files]
It never completes the mapping of the files in Pass I. It just sits there. The filesystems I'm backing up are between 250-650GB with a LOT of small files (millions) and many subdirs. Tons of mail and html files.
My guess is that the amount of files and dirs are getting large that NDMP can't map them fully and is timing out after 8 hours. It shouldn't take that long to map files for the size of data it's doing; I've seem mapping times be much less for larger sets of data.
I know that the filesystem topology can cause NDMP backups to be slow, depending on file sizes, data layout, filer load, network etc, but this is happening on moderately busy filers or bone idle NearStores.
Anyone run across lengthy DUMP times?
Chewing through NOW and Veritas support site hasn't turned up anything obvious.
Just curious if others have run into NDMP/dump issues like this.
-Scott
-- Yours, Eyal.
This is described on now.netapp.com as Bug 96894 (Phase I of NDMP based backups is slow) and fixed in Data ONTAP 6.4.2P4, ONTAP 6.4.5 (GA), 6.5.3P4 (GA), 6.5.4 (FCS), 7.0.0.1 (GA)
Description: For NDMP backups with File History enabled, Phase I is slow. The extent of slowness may depend on fragmentation of the inode file and the file system. With Data Ontap 6.4 onwards, NDMP backups with File History add some processing in Phase I. This processing is required for DAR Enhancements (A restore time feature), which are a part of Data Ontap 6.4 or beyond. The slowness is
attributable to performance bottlenecks in this additional processing.
Workaround: Backups without file history continue to perform as expected. For backups where performance of Phase I is bothersome, file history can be turned off temporarily until a fix is available. For a workaround to turn off the additional processing during Phase I (Offset map generation), without disabling the file history, please track bug 102211.
regards
alfred
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Blake Golliher Sent: Thursday, September 07, 2006 3:17 AM To: Eyal Traitel Cc: Scott T. Mikusko; toasters@mathworks.com Subject: Re: NDMP backups, timeouts
I think phase 1 is where it does a catalogue of all the files in a file system to be backed up. If you have millions of files this could take a long long time. So snapmirror to tape might be an option, thought I'm not sure if your backup software supports it. Also, you could try backing up smaller chunks a time to keep that first phase from killing you.
You could also look into extending the time out function of your ndmp software. That might be an option as well.
-Blake
On 9/6/06, Eyal Traitel etraitel@gmail.com wrote:
Suggestions:
- Try to dump to null as a test.
- What is your ONTAP release? I believe there have been improvements
in dump/restore performance for such environments in more recent
releases.
Eyal.
On 9/6/06, Scott T. Mikusko guerilla@sw.xo.com wrote:
I'm running into a recurring issue with backing up my NearStore and a long F760 filer.
In the procoess of backing some large filesystems (qtrees), it appears that NDMP is basically timing out and fails. I'm running Veritas NetBackup 6.0 (finally upgrading from 4.5 recently). I think the issue has to do with NDMP starting the dump commands on the filer, which then does a three-way backup to another filer with
local tape drives.
The debug from ndmp logs shows for a manual full backup of a filer:
Sep 05 21:05:57 EDT [ndmpd:58]: Log message: DUMP: creating "/vol/vol0/../snapshot_for_backup.370" snapshot. Sep 05 21:06:05 EDT [ndmpd:58]: Log message: DUMP: Using Partial Volume Dump with Exclude Lists Sep 05 21:07:31 EDT [ndmpd:58]: Log message: DUMP: Date of this level 1 dump: Tue Sep 5 21:05:57 2006. Sep 05 21:07:31 EDT [ndmpd:58]: Log message: DUMP: Date of last level 0 dump: Fri Aug 18 19:20:57 2006. Sep 05 21:07:31 EDT [ndmpd:58]: Log message: DUMP: Dumping /vol/vol0/files3.rt to NDMP connection Sep 05 21:07:31 EDT [ndmpd:58]: Log message: DUMP: mapping (Pass I)[regular files]
It never completes the mapping of the files in Pass I. It just sits
there.
The filesystems I'm backing up are between 250-650GB with a LOT of small files (millions) and many subdirs. Tons of mail and html
files.
My guess is that the amount of files and dirs are getting large that
NDMP can't map them fully and is timing out after 8 hours. It shouldn't take that long to map files for the size of data it's doing; I've seem mapping times be much less for larger sets of data.
I know that the filesystem topology can cause NDMP backups to be slow, depending on file sizes, data layout, filer load, network etc,
but this is happening on moderately busy filers or bone idle
NearStores.
Anyone run across lengthy DUMP times?
Chewing through NOW and Veritas support site hasn't turned up anything obvious.
Just curious if others have run into NDMP/dump issues like this.
-Scott
-- Yours, Eyal.
-
Blake Golliher -
Eyal Traitel -
Lerch, Alfred -
Scott T. Mikusko