I wanted to relay some the information that I received regarding my problem with slow file system dumps from my toaster to my Solaris 8 remote tape. As well as thank those persons who responded.
I received three suggestions. The first was to try using a different blocking factor, I had tried using different blocking factors but not larger as in this suggestion. I am evaluating a blocking factor of 128 which has doubled my throughput. Not what it should be but still a 100% improvement! As has been noted you have to keep this in mind on a restore as well. (rsh $FILER dump 0ubf 128 $SUNBOX:/dev/rmt/1 /vol/vol2). The second suggestion that I received indicated a similar issue with a very wide but not necessarily deep directory structure where NDMP dumps went from 50+GB/hour to around 8kbps. Apparently "NetApp have indicated it is an NDMP/dump issue and there is no fix". However this site upgraded to 6.4.x and saw much greater throughput, (no data). I have not yet decided to upgrade Data ONTAP. The third suggestion that I received noted that "Lots of small files, especially lots of small files in a single directory is a dump performance killer. Basically you spend a lot of time doing the meta data stuff (path, permissions, etc) and little time to get the tape streaming to dump the data. I'd be willing to bet the same amount of data but in a few large files would back up significantly quicker...at least that's been my experience." This is indeed true. He had two suggestions. The first was to try using the native hidden command "dd" on the filer which is really less friendly than the dump command but might be worth a try. "DD" has its draw backs and limitations and it seems like it might be an administrative nightmare. The second suggestion was to use snapmirror, which is unfortunately a bit cost prohibitive.
Thanks for all your help. I'll let you know if an OS upgrade helps if I try it. Other suggestions are always welcomed.
-Jamie
-----Original Message----- From: Rogers, Jamie [mailto:jrogers@aware.com] Sent: Tuesday, November 11, 2003 4:10 PM To: 'toasters@mathworks.com' Subject: slow dumps
Toasters, I have an F760 running data ontap 6.3.1 with 4 FC-9 (7x32) shelves attached. I use Ultra 10 with a HP SureStore718 LVD autoloader for remote backups. I use rsh (rsh filer dump 0uf sunbox:/dev/rmt/1 /vol/vol1) via a cron job to run the dump. The Ultra10 and the filer are connected on a private network via 100TX crossover cable. Local file system backups (from the Ultra 10) to this tape drive go at about 4+ Meg/sec, great. The root volume of my filer and /vol3 dump at 4+ Megs/sec, this is great. However 2 volumes, vol1 and vol2, dump at 571Kb/sec. This performance is independent of the interface used, I get the same results using my 100TX or my gigabit. Likewise I get the same results when the dump is run from the filer, that is eliminating rsh. Sysstat looks normal during both dumps.
The only difference in the volumes is the number of files, volumes 1 and 2 each have about 2 million files where as volume 3 has 1.2 million files and the root is about 10,000 files with out quotas. The data on volumes 1 and 2 consists of lots of small files and changes frequently whereas volumes 0 and 3 are pretty static. I have already cut the volumes up into 4 qtrees of 30G each. A quota report does not look out of the ordinary and no single quota tree has more than 950,000 files, see quota report below.
What could be causing the dumps to progress so slowly? Any suggestions to speed up my backups?
K-Bytes Files Type ID Volume Tree Used Limit Used Limit Quota Specifier ----- -------- -------- -------- -------- -------- ------- ------- --------------- tree 1 vol4 zeus21 0 31457280 3 - /vol/vol4/zeus21 tree 2 vol4 zeus22 0 31457280 3 - /vol/vol4/zeus22 tree 3 vol4 zeus23 0 31457280 3 - /vol/vol4/zeus23 tree 4 vol4 zeus24 3969544 31457280 22085 - /vol/vol4/zeus24 tree 1 vol3 zeus17 3076 31457280 5 - /vol/vol3/zeus17 tree 2 vol3 zeus18 0 31457280 3 - /vol/vol3/zeus18 tree 3 vol3 tools_19 21525624 31457280 343289 - /vol/vol3/tools_19 tree 4 vol3 tools_20 22765236 31457280 701033 - /vol/vol3/tools_20 tree 1 vol2 zeus13 21631732 31457280 439528 - /vol/vol2/zeus13 tree 2 vol2 zeus14 30732636 31457280 716547 - /vol/vol2/zeus14 tree 3 vol2 zeus15 21069420 31457280 721908 - /vol/vol2/zeus15 tree 4 vol2 zeus16 0 31457280 3 - /vol/vol2/zeus16 tree 2 vol1 zeus11 26029280 31457280 933922 - /vol/vol1/zeus11 tree 3 vol1 zeus12 29793368 31457280 788538 - /vol/vol1/zeus12
Thanks, Jamie
-Jamie.