I wanted to relay some the information that I received regarding my problem
with slow file system dumps from my toaster to my Solaris 8 remote tape.
As well as thank those persons who responded.
I received three suggestions.
The first was to try using a different blocking factor, I had tried using
different blocking factors but not larger as in this suggestion. I am
evaluating a blocking factor of 128 which has doubled my throughput. Not
what it should be but still a 100% improvement! As has been noted you have
to keep this in mind on a restore as well. (rsh $FILER dump 0ubf 128
$SUNBOX:/dev/rmt/1 /vol/vol2).
The second suggestion that I received indicated a similar issue with a very
wide but not necessarily deep directory structure where NDMP dumps went from
50+GB/hour to around 8kbps. Apparently "NetApp have indicated it is an
NDMP/dump issue and there is no fix". However this site upgraded to 6.4.x
and saw much greater throughput, (no data). I have not yet decided to
upgrade Data ONTAP.
The third suggestion that I received noted that "Lots of small files,
especially lots of small files in a single
directory is a dump performance killer. Basically you spend a lot of time
doing the meta data stuff (path, permissions, etc) and little time to get
the tape streaming to dump the data. I'd be willing to bet the same amount
of data but in a few large files would back up significantly quicker...at
least that's been my experience." This is indeed true. He had two
suggestions. The first was to try using the native hidden command "dd" on
the filer which is really less friendly than the dump command but might be
worth a try. "DD" has its draw backs and limitations and it seems like it
might be an administrative nightmare. The second suggestion was to use
snapmirror, which is unfortunately a bit cost prohibitive.
Thanks for all your help. I'll let you know if an OS upgrade helps if I try
it. Other suggestions are always welcomed.
-Jamie
-----Original Message-----
From: Rogers, Jamie [mailto:jrogers@aware.com]
Sent: Tuesday, November 11, 2003 4:10 PM
To: 'toasters(a)mathworks.com'
Subject: slow dumps
Toasters,
I have an F760 running data ontap 6.3.1 with 4 FC-9 (7x32) shelves attached.
I use Ultra 10 with a HP SureStore718 LVD autoloader for remote backups. I
use rsh (rsh filer dump 0uf sunbox:/dev/rmt/1 /vol/vol1) via a cron job to
run the dump. The Ultra10 and the filer are connected on a private network
via 100TX crossover cable. Local file system backups (from the Ultra 10) to
this tape drive go at about 4+ Meg/sec, great. The root volume of my filer
and /vol3 dump at 4+ Megs/sec, this is great. However 2 volumes, vol1 and
vol2, dump at 571Kb/sec. This performance is independent of the interface
used, I get the same results using my 100TX or my gigabit. Likewise I get
the same results when the dump is run from the filer, that is eliminating
rsh. Sysstat looks normal during both dumps.
The only difference in the volumes is the number of files, volumes 1 and 2
each have about 2 million files where as volume 3 has 1.2 million files and
the root is about 10,000 files with out quotas. The data on volumes 1 and 2
consists of lots of small files and changes frequently whereas volumes 0 and
3 are pretty static. I have already cut the volumes up into 4 qtrees of 30G
each. A quota report does not look out of the ordinary and no single quota
tree has more than 950,000 files, see quota report below.
What could be causing the dumps to progress so slowly? Any suggestions to
speed up my backups?
K-Bytes Files
Type ID Volume Tree Used Limit Used Limit Quota
Specifier
----- -------- -------- -------- -------- -------- ------- -------
---------------
tree 1 vol4 zeus21 0 31457280 3 -
/vol/vol4/zeus21
tree 2 vol4 zeus22 0 31457280 3 -
/vol/vol4/zeus22
tree 3 vol4 zeus23 0 31457280 3 -
/vol/vol4/zeus23
tree 4 vol4 zeus24 3969544 31457280 22085 -
/vol/vol4/zeus24
tree 1 vol3 zeus17 3076 31457280 5 -
/vol/vol3/zeus17
tree 2 vol3 zeus18 0 31457280 3 -
/vol/vol3/zeus18
tree 3 vol3 tools_19 21525624 31457280 343289 -
/vol/vol3/tools_19
tree 4 vol3 tools_20 22765236 31457280 701033 -
/vol/vol3/tools_20
tree 1 vol2 zeus13 21631732 31457280 439528 -
/vol/vol2/zeus13
tree 2 vol2 zeus14 30732636 31457280 716547 -
/vol/vol2/zeus14
tree 3 vol2 zeus15 21069420 31457280 721908 -
/vol/vol2/zeus15
tree 4 vol2 zeus16 0 31457280 3 -
/vol/vol2/zeus16
tree 2 vol1 zeus11 26029280 31457280 933922 -
/vol/vol1/zeus11
tree 3 vol1 zeus12 29793368 31457280 788538 -
/vol/vol1/zeus12
Thanks,
Jamie
-Jamie.