Donald,
You describe this as a specific problem - nevertheless, are we sure this is not just working as designed? All these FlexVols are located in the same aggregate - so parallel dumps from the same aggregate going slower makes sense to me. dump is pretty efficient once it reaches the later phases - I would suggest you to debug this in a different method - use the PerfStat tool to capture the statit and sysstat outputs, and review whether the disks are highly utilized during a single dump. If this is the case already, then there's nothing wrong - you're already maxing out your shelves I/O by this dump. Unless the FAS960 platform is maxed by itself (which I guestimate to NOT be the case), adding shelves would actually help you achieve faster dumps, not slower.
My side non-proven suggestion - stage your NDMP backups so that the first phases will not all occur in parallel. This may help improve your total backup time.
Anyhow, I'd love to see numbers when I hear a problem such as yours - what's your ONTAP? how many MB/sec are you seeing in sysstat etc...
Eyal.
http://filers.blogspot.com
http://stupidstorage.blogspot.com
On 10/10/06,
Glascock, Donald <glascock.donald@mayo.edu> wrote:
Hi, Toaster folks --
We're a small research group on our third Filer, which is
a single FAS960 with five shelves on two cross-connected 2-Gb
loops. We trigger flexVolume-level dumps of our one aggregate
directly to 2-Gb-fibre-connected LTO-3 drives.
When we dump two volumes simultaneously to two tape drives,
the time taken to dump each volume increases by a third to a
half again, compared to dumping each volume independently.
For example, a stand-alone dump may take two hours, but if
another dump of another volume to another drive is run
simultaneously, then that two-hour dump becomes a three-hour
dump. Adding a third simultaneous dump to a third drive causes
an even greater increase in dump times.
For diagnostic purposes, I've launched a conventional dump
of one volume triggered via NDMP our backup software, waited
for the dump to reach Phase-IV (the writing-data phase), and
launched from the FAS960's console a dump of another volume
to the FAS960's null device. Soon after this second dump
starts, the write speed on the tape drive associated with
the first dump falls off by quite a bit. This write speed
picks up instantly when I kill the second dump. Since the
second dump was only in Phase-I (inspecting files, but not
writing them), and since this second dump was to the local
null device anyway, I'm led to think that the performance
issues lie somewhere within the Filer's head and Disk I/O
subsystem.
With one-at-a-time backups of our volumes, our full backup
window is about eight hours (for about two terabytes). We
were planning to buy two more shelves in early 2007.
Should I expect a FAS960 to handle two or three simultaneous
full dumps without a significant loss in dump speed? Are you
getting nearly-linear performance out of your simultaneous dumps?
Thanks for your time & have a great day!
Don Glascock
--
Donald S. Glascock
Mailstop RO-SN-2-SPPDG
Special Purpose Processor Development Group
Mayo Foundation
4001 41st Street NW
Rochester, MN 55901
Glascock.Donald@Mayo.EDU +1.507.538.5467 +1.507.284.9171 (fax)
http://www.mayo.edu/sppdg/
"No matter where you go, there you are."
--
Yours,
Eyal.