Hi, Toaster folks --
We're a small research group on our third Filer, which is a single FAS960 with five shelves on two cross-connected 2-Gb loops. We trigger flexVolume-level dumps of our one aggregate directly to 2-Gb-fibre-connected LTO-3 drives.
When we dump two volumes simultaneously to two tape drives, the time taken to dump each volume increases by a third to a half again, compared to dumping each volume independently. For example, a stand-alone dump may take two hours, but if another dump of another volume to another drive is run simultaneously, then that two-hour dump becomes a three-hour dump. Adding a third simultaneous dump to a third drive causes an even greater increase in dump times.
For diagnostic purposes, I've launched a conventional dump of one volume triggered via NDMP our backup software, waited for the dump to reach Phase-IV (the writing-data phase), and launched from the FAS960's console a dump of another volume to the FAS960's null device. Soon after this second dump starts, the write speed on the tape drive associated with the first dump falls off by quite a bit. This write speed picks up instantly when I kill the second dump. Since the second dump was only in Phase-I (inspecting files, but not writing them), and since this second dump was to the local null device anyway, I'm led to think that the performance issues lie somewhere within the Filer's head and Disk I/O subsystem.
With one-at-a-time backups of our volumes, our full backup window is about eight hours (for about two terabytes). We were planning to buy two more shelves in early 2007.
Should I expect a FAS960 to handle two or three simultaneous full dumps without a significant loss in dump speed? Are you getting nearly-linear performance out of your simultaneous dumps?
Thanks for your time & have a great day!
Don Glascock
Donald,
You describe this as a specific problem - nevertheless, are we sure this is not just working as designed? All these FlexVols are located in the same aggregate - so parallel dumps from the same aggregate going slower makes sense to me. dump is pretty efficient once it reaches the later phases - I would suggest you to debug this in a different method - use the PerfStat tool to capture the statit and sysstat outputs, and review whether the disks are highly utilized during a single dump. If this is the case already, then there's nothing wrong - you're already maxing out your shelves I/O by this dump. Unless the FAS960 platform is maxed by itself (which I guestimate to NOT be the case), adding shelves would actually help you achieve faster dumps, not slower.
My side non-proven suggestion - stage your NDMP backups so that the first phases will not all occur in parallel. This may help improve your total backup time.
Anyhow, I'd love to see numbers when I hear a problem such as yours - what's your ONTAP? how many MB/sec are you seeing in sysstat etc...
Eyal. http://filers.blogspot.com http://stupidstorage.blogspot.com
On 10/10/06, Glascock, Donald glascock.donald@mayo.edu wrote:
Hi, Toaster folks --
We're a small research group on our third Filer, which is a single FAS960 with five shelves on two cross-connected 2-Gb loops. We trigger flexVolume-level dumps of our one aggregate directly to 2-Gb-fibre-connected LTO-3 drives.
When we dump two volumes simultaneously to two tape drives, the time taken to dump each volume increases by a third to a half again, compared to dumping each volume independently. For example, a stand-alone dump may take two hours, but if another dump of another volume to another drive is run simultaneously, then that two-hour dump becomes a three-hour dump. Adding a third simultaneous dump to a third drive causes an even greater increase in dump times.
For diagnostic purposes, I've launched a conventional dump of one volume triggered via NDMP our backup software, waited for the dump to reach Phase-IV (the writing-data phase), and launched from the FAS960's console a dump of another volume to the FAS960's null device. Soon after this second dump starts, the write speed on the tape drive associated with the first dump falls off by quite a bit. This write speed picks up instantly when I kill the second dump. Since the second dump was only in Phase-I (inspecting files, but not writing them), and since this second dump was to the local null device anyway, I'm led to think that the performance issues lie somewhere within the Filer's head and Disk I/O subsystem.
With one-at-a-time backups of our volumes, our full backup window is about eight hours (for about two terabytes). We were planning to buy two more shelves in early 2007.
Should I expect a FAS960 to handle two or three simultaneous full dumps without a significant loss in dump speed? Are you getting nearly-linear performance out of your simultaneous dumps?
Thanks for your time & have a great day!
Don Glascock
-- Donald S. Glascock Mailstop RO-SN-2-SPPDG Special Purpose Processor Development Group Mayo Foundation 4001 41st Street NW Rochester, MN 55901 Glascock.Donald@Mayo.EDU +1.507.538.5467 +1.507.284.9171 (fax) http://www.mayo.edu/sppdg/ "No matter where you go, there you are."
Don,
How are your disks laid out is all the data in one large aggregate?
What type of data do you have on your system? Lots of small files, fewer larger files? Very compressible files?
The Phase 1 scanning of files has a significant impact on filer CPU whilst it is performing this. If you have a lot of small files then this process can take along time to complete. If Phase 1 is taking a significant amount of time to perform then running 2 together will extend the backups.
When you run one backup check the output of a sysstat to see what rate the data is being written to tape at.
LTO 3 runs at 80MB/sec native and upto 160MB/sec with compressible data. A 2GB FC runs at 200MB/sec, 2 tape drives on one FC loop will potentially start to slow each other down with compressible data.
If you backup window is too small then you have to start looking at ways to decrease the amount of data you are backing up each night. Do you have to do full backups of every volume every night? Why not rotate the volumes you are doing full backups of around the week and then just do incrementals of the other volumes every night. Or perform full backups at a weekend and then incrementals every other night.
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Glascock, Donald Sent: 10 October 2006 21:24 To: toasters@mathworks.com Subject: FAS960: backup speed vs simultaneous dumps...?
Hi, Toaster folks --
We're a small research group on our third Filer, which is a single FAS960 with five shelves on two cross-connected 2-Gb loops. We trigger flexVolume-level dumps of our one aggregate directly to 2-Gb-fibre-connected LTO-3 drives.
When we dump two volumes simultaneously to two tape drives, the time taken to dump each volume increases by a third to a half again, compared to dumping each volume independently. For example, a stand-alone dump may take two hours, but if another dump of another volume to another drive is run simultaneously, then that two-hour dump becomes a three-hour dump. Adding a third simultaneous dump to a third drive causes an even greater increase in dump times.
For diagnostic purposes, I've launched a conventional dump of one volume triggered via NDMP our backup software, waited for the dump to reach Phase-IV (the writing-data phase), and launched from the FAS960's console a dump of another volume to the FAS960's null device. Soon after this second dump starts, the write speed on the tape drive associated with the first dump falls off by quite a bit. This write speed picks up instantly when I kill the second dump. Since the second dump was only in Phase-I (inspecting files, but not writing them), and since this second dump was to the local null device anyway, I'm led to think that the performance issues lie somewhere within the Filer's head and Disk I/O subsystem.
With one-at-a-time backups of our volumes, our full backup window is about eight hours (for about two terabytes). We were planning to buy two more shelves in early 2007.
Should I expect a FAS960 to handle two or three simultaneous full dumps without a significant loss in dump speed? Are you getting nearly-linear performance out of your simultaneous dumps?
Thanks for your time & have a great day!
Don Glascock
Hi, again, Toaster folks --
Thanks, Darren and Eyal, for taking the time to reply.
From your observations, it sounds like our choice for "ultimate flexibility" with regard to volume sizing by creating only one aggregate on our FAS960 was also a choice away from fast backup speed for simultaneous dumps. Is that a fair assessment?
Is anyone successfully running full simultaneous dumps to separate tape drives (utilizing separate FC paths from the filer to the fabric, and from the fabric to the drives) from two or more aggregates on one filer (whatever model you may have) at rated speed?
Eyal writes:
[...] adding shelves would actually help you achieve faster dumps, not slower.
Would the shelves be best added to our existing two loops, or would we do better to buy yet another HBA for them? Intuitively, a busier loop doesn't sound like an optimal configuration; on the other hand, we've been told that our FAS960's existing five HBAs have pretty much maxed "all of the busses" in our FAS960.
While I'm trying to keep this discussion mostly about the system architecture and what we should be able to expect for backup performance, I can tell you that we're running DOT 7.0.4 while waiting for a few more folks to run 7.2 for a while, and we are getting ~80 MB/sec when dumping volumes individually. We dump fulls once per month, and we can split up the monthly dump process across a few nights if we absolutely have to. When we dump two volumes simultaneously, the write speeds of the drives hover around 40-50 MB/sec, and when we write three, they hover around 30-40 MB/sec.
Thanks again for your time & have a great day!
Don Glascock
Don,
Are you getting more MB/sec when dumping to null, or when reading from CIFS/NFS? To answer your first assesment - it´s a theoretical question, I would assume that it´s not necessarily correct. Let´s say you would have configured 2 separate volumes instead of one large aggregate - each one of them would have less disks so you would not necessarily gain faster performance....
For your loops question - look at your 80MB/sec result - you are far away from loops being your bottleneck. I have seen many filers in the last few years and cannot remember cases where loops were maxing out.
Eyal. http://filers.blogspot.com http://stupidstorage.blogspot.com
On 10/11/06, Glascock, Donald glascock.donald@mayo.edu wrote:
Hi, again, Toaster folks --
Thanks, Darren and Eyal, for taking the time to reply.
From your observations, it sounds like our choice for "ultimate flexibility" with regard to volume sizing by creating only one aggregate on our FAS960 was also a choice away from fast backup speed for simultaneous dumps. Is that a fair assessment?
Is anyone successfully running full simultaneous dumps to separate tape drives (utilizing separate FC paths from the filer to the fabric, and from the fabric to the drives) from two or more aggregates on one filer (whatever model you may have) at rated speed?
Eyal writes:
[...] adding shelves would actually help you achieve faster dumps, not slower.
Would the shelves be best added to our existing two loops, or would we do better to buy yet another HBA for them? Intuitively, a busier loop doesn't sound like an optimal configuration; on the other hand, we've been told that our FAS960's existing five HBAs have pretty much maxed "all of the busses" in our FAS960.
While I'm trying to keep this discussion mostly about the system architecture and what we should be able to expect for backup performance, I can tell you that we're running DOT 7.0.4 while waiting for a few more folks to run 7.2 for a while, and we are getting ~80 MB/sec when dumping volumes individually. We dump fulls once per month, and we can split up the monthly dump process across a few nights if we absolutely have to. When we dump two volumes simultaneously, the write speeds of the drives hover around 40-50 MB/sec, and when we write three, they hover around 30-40 MB/sec.
Thanks again for your time & have a great day!
Don Glascock
-- Donald S. Glascock Mailstop RO-SN-2-SPPDG Special Purpose Processor Development Group Mayo Foundation 4001 41st Street NW Rochester, MN 55901 Glascock.Donald@Mayo.EDU +1.507.538.5467 +1.507.284.9171 (fax) http://www.mayo.edu/sppdg/ "No matter where you go, there you are."
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Glascock, Donald Sent: Tuesday, October 10, 2006 3:24 PM To: toasters@mathworks.com Subject: FAS960: backup speed vs simultaneous dumps...?
Hi, Toaster folks --
We're a small research group on our third Filer, which is a single FAS960 with five shelves on two cross-connected 2-Gb loops. We trigger flexVolume-level dumps of our one aggregate directly to 2-Gb-fibre-connected LTO-3 drives.
When we dump two volumes simultaneously to two tape drives, the time taken to dump each volume increases by a third to a half again, compared to dumping each volume independently. For example, a stand-alone dump may take two hours, but if another dump of another volume to another drive is run simultaneously, then that two-hour dump becomes a three-hour dump. Adding a third simultaneous dump to a third drive causes an even greater increase in dump times.
For diagnostic purposes, I've launched a conventional dump of one volume triggered via NDMP our backup software, waited for the dump to reach Phase-IV (the writing-data phase), and launched from the FAS960's console a dump of another volume to the FAS960's null device. Soon after this second dump starts, the write speed on the tape drive associated with the first dump falls off by quite a bit. This write speed picks up instantly when I kill the second dump. Since the second dump was only in Phase-I (inspecting files, but not writing them), and since this second dump was to the local null device anyway, I'm led to think that the performance issues lie somewhere within the Filer's head and Disk I/O subsystem.
With one-at-a-time backups of our volumes, our full backup window is about eight hours (for about two terabytes). We were planning to buy two more shelves in early 2007.
Should I expect a FAS960 to handle two or three simultaneous full dumps without a significant loss in dump speed? Are you getting nearly-linear performance out of your simultaneous dumps?
Thanks for your time & have a great day!
Don Glascock
-- Donald S. Glascock Mailstop RO-SN-2-SPPDG Special Purpose Processor Development Group Mayo Foundation 4001 41st Street NW Rochester, MN 55901 Glascock.Donald@Mayo.EDU +1.507.538.5467 +1.507.284.9171 (fax) http://www.mayo.edu/sppdg/ "No matter where you go, there you are."
-
Darren Soothill -
Eyal Traitel -
Glascock, Donald