Hi, We have been facing some explosive growth during the past 6 months and have grown past our current backup solution. We are currently backing up via a linux box that has our 740 nfs mounted. Backups are taking sometimes 24-30 hours for incrementals and all weekend for fulls.
What I am looking for is info on how others are backing up 200G volumes on 740's or 760's. I just purchased a pair of 760's and am going to need a nice stable reliable backup system for them.
Ideas?
--Brian
Go to NDMP solution. We use Veritas, but no point in speaking about it too much, since too much have been sent on the list on these issues....
You can learn on ndmp in http://www.ndmp.org and from NetApp's site...
Eyal.
Brian Hostetter wrote:
Hi, We have been facing some explosive growth during the past 6 months and have grown past our current backup solution. We are currently backing up via a linux box that has our 740 nfs mounted. Backups are taking sometimes 24-30 hours for incrementals and all weekend for fulls.
What I am looking for is info on how others are backing up 200G volumes on 740's or 760's. I just purchased a pair of 760's and am going to need a nice stable reliable backup system for them.
Ideas?
--Brian
On Wed, 12 Apr 2000, Brian Hostetter wrote:
What I am looking for is info on how others are backing up 200G volumes on 740's or 760's. I just purchased a pair of 760's and am going to need a nice stable reliable backup system for them.
My recommendation is for locally attached tape libraries on the NetApp boxes. I currently have two F-740's with AIT-2 library units on each. One of them has over 200 GB of active space and it takes about 5 or 6 hours to do a full backup.
Brian> We have been facing some explosive growth during the past 6 Brian> months and have grown past our current backup solution. We are Brian> currently backing up via a linux box that has our 740 nfs Brian> mounted. Backups are taking sometimes 24-30 hours for Brian> incrementals and all weekend for fulls.
Ouch, not fun. But can you give us a bit more detail on how your network and backup server are setup? Then we can help with a range of suggestions, ranging from NFS tweaks to full blown replacement of hardware.
Brian> What I am looking for is info on how others are backing up 200G Brian> volumes on 740's or 760's. I just purchased a pair of 760's Brian> and am going to need a nice stable reliable backup system for Brian> them.
We're currently using Legato Networker (Solstice Backup actually) 5.1 to backup NetApps that have been NFS mounted to the backup servers, which have ATL jukeboxes with dual DLT7000 in them. It works well on the server that's an Ultra 2 with dual 300Mhz CPUs. The other one is slow because it's just a Sparc 20 with Dual CPUs. Haven't had the chance to replace it yet.
Anyway, there could be lots of problems with your backups speeds. Some things to look at and possibly try out:
1. Change your NFS mount options
2. Build a seperate backup network between the backup server and NetApp(s). This is a good idea in general.
3. Check the load on your server when doing backups. - How busy is the CPU? - Is it swapping heavily? - Are there other programs running at the same time? - What's the size of memory on the system? - What Linux version are you running? - Do you have the latest NFS patches applied? - What tape drive(s) are you using? - Which tape jukebox are you using?
4. What backup software are you using and does it have any performance indicators?
5. Are you willing to spend the money on NDMP capable software and a DLT7000 jukebox attached directory to the filer(s)? That would get the network out of the equation almost completely.
Lots and lots of things to look at here.
John John Stoffel - Senior Unix Systems Administrator - Lucent Technologies stoffel@lucent.com - http://www.lucent.com - 978-952-7548 john.stoffel@ascend.com - http://www.ascend.com
On Wed, 12 Apr 2000, John Stoffel wrote:
- Are you willing to spend the money on NDMP capable software and a DLT7000 jukebox attached directory to the filer(s)? That would get the network out of the equation almost completely.
I've seen several comments like this in the past and I don't understand them. Why do you feel it is neccessary to spend money on NDMP capable software to be able to use a directly attached tape drive? It probably took me a couple of hours to put togethor a shell script which will back up the filer using the locally attached tape drive. The script runs on the admin host and uses rsh commands to connect to the filer and run the dump program. Two hours of my time is a lot less expensive than any of the commercial backup solutions, especially since it would have taken me almost 2 hours to fill out the paperwork for the purchase. Plus, it was an interesting problem to solve so I enjoyed doing it.
I think the reason you see many comments like this is because many folks are realizing that backup strategies are becoming more and more important as data spaces grow.
In realization of this, many IT managers are trying to get away from homegrown solutions because they sometimes are not documented well nor maintained properly. In addition, if the person that develops the shell script leaves the company, they take the legacy knowledge with them and inevitably others may not be able to support the system in place.
Commercial products deliver a support network, documentation, and accountability for the IT manager that outlives his tenure at the company. Many feel (including myself) that this is a more responsible strategy being in businesses where data integrity is crucial to the success of the company.
Thanks, Derek Kelly www.genomecorp.com
On 4/13/00 at 10:44 AM, dhb@ssd.ray.com (David H. Brierley) wrote:
On Wed, 12 Apr 2000, John Stoffel wrote:
- Are you willing to spend the money on NDMP capable software and a DLT7000 jukebox attached directory to the filer(s)? That would get the network out of the equation almost completely.
I've seen several comments like this in the past and I don't
understand
them. Why do you feel it is neccessary to spend money on NDMP capable software to be able to use a directly attached tape drive? It
probably
took me a couple of hours to put togethor a shell script which will back up the filer using the locally attached tape drive. The script runs on the admin host and uses rsh commands to connect to the filer and run the dump program. Two hours of my time is a lot less
expensive
than any of the commercial backup solutions, especially since it would have taken me almost 2 hours to fill out the paperwork for the purchase. Plus, it was an interesting problem to solve so I enjoyed doing it.
-- David H. Brierley Raytheon Electronic Systems, Naval & Maritime Integrated Systems Engineering Technology, Operating Systems Support Group
In addition, the time that you want to save is not in the backups, it's in the recovery portion. Commercial backup solutions (at least the good ones) allow you to track what tape a file is on and what date it was backed up on, so you can restore to point-in-time eaily.
With most homegrown solutions, you either have to restore it all, look through a paper list of tapes and dates, or just keep on trying until you find the right tape.
On Thu, 13 Apr 2000, Derek Kelly wrote:
I think the reason you see many comments like this is because many folks are realizing that backup strategies are becoming more and more important as data spaces grow.
In realization of this, many IT managers are trying to get away from homegrown solutions because they sometimes are not documented well nor maintained properly. In addition, if the person that develops the shell script leaves the company, they take the legacy knowledge with them and inevitably others may not be able to support the system in place.
Commercial products deliver a support network, documentation, and accountability for the IT manager that outlives his tenure at the company. Many feel (including myself) that this is a more responsible strategy being in businesses where data integrity is crucial to the success of the company.
Thanks, Derek Kelly www.genomecorp.com
On 4/13/00 at 10:44 AM, dhb@ssd.ray.com (David H. Brierley) wrote:
On Wed, 12 Apr 2000, John Stoffel wrote:
- Are you willing to spend the money on NDMP capable software and a DLT7000 jukebox attached directory to the filer(s)? That would get the network out of the equation almost completely.
I've seen several comments like this in the past and I don't
understand
them. Why do you feel it is neccessary to spend money on NDMP capable software to be able to use a directly attached tape drive? It
probably
took me a couple of hours to put togethor a shell script which will back up the filer using the locally attached tape drive. The script runs on the admin host and uses rsh commands to connect to the filer and run the dump program. Two hours of my time is a lot less
expensive
than any of the commercial backup solutions, especially since it would have taken me almost 2 hours to fill out the paperwork for the purchase. Plus, it was an interesting problem to solve so I enjoyed doing it.
-- David H. Brierley Raytheon Electronic Systems, Naval & Maritime Integrated Systems Engineering Technology, Operating Systems Support Group
-- -------------------------------------------------------------------------- Cliff Nadler Collective Technologies, Inc. (a Pencom Company) cnadler@colltech.com (617)-443-1144 -------------------------------------------------------------------------- To win, you must treat a pressure situation as an opportunity to succeed, not an opportunity to fail Gardner Dickinson
"David" == David H Brierley dhb@ssd.ray.com writes:
David> On Wed, 12 Apr 2000, John Stoffel wrote:
- Are you willing to spend the money on NDMP capable software and a
DLT7000 jukebox attached directory to the filer(s)? That would get the network out of the equation almost completely.
David> I've seen several comments like this in the past and I don't David> understand them. Why do you feel it is neccessary to spend David> money on NDMP capable software to be able to use a directly David> attached tape drive?
Because I want to be able to do all my backups to one jukebox if at all possible. This keeps down costs, keeps down maintenance issues in terms of where you have to go to swap tapes, etc. In our situation, I have filers on four seperate server closets (they're not rooms, dammit! :-) so I don't want to have to walk around the building to swap tapes.
Also, I can see that I wasn't quite clear in my previous statement, since it will get the NetApp's backup traffic off the network, but if you use the NetApp as the NDMP target for other hosts, their traffic will be going over the network.
And of course there's the issue of indexes, since it does no good to backup your files if you can't restore them in a timely manner, and having to dig through tapes to find a file is slow. That's why all the major backup software vendors have online file indexes of files backed up.
David> It probably took me a couple of hours to put togethor a shell David> script which will back up the filer using the locally attached David> tape drive. The script runs on the admin host and uses rsh David> commands to connect to the filer and run the dump program. Two David> hours of my time is a lot less expensive than any of the David> commercial backup solutions, especially since it would have David> taken me almost 2 hours to fill out the paperwork for the David> purchase. Plus, it was an interesting problem to solve so I David> enjoyed doing it.
It's a tradeoff. And everyone is in a different boat. I've been in both the Educational environment where we didn't have the money for backup software, so I wrote the scripts to manage backups and tapes and all. It's not trivial to do, especially since the core of it's really simple. It's the error checking and making the script resistent to errors (both user and system) that's hard. I've heard it said that 90% of all code is bounds and error checking and I can believe it. It's a hard problem to solve well.
Some of the tradeoffs include:
- how many systems you need to backup. It goes from 1 to 100 to N. The more you have to backup, the better a central tape library looks.
- Is your time cheaper than software costs? Usually this is true in the non-profit space. For those cases, I also look at the Amanda backup package.
- How much data do you have to backup? And how long can you take to back it up? Time is money in the business world.
- How fast do you need to restore a single file from backups. This is a big hidden gotcha that you need to think about. And sometimes you can't justify it until after you've been bitten.
And I'm sure people can chime in with more.
For us, the software costs alot less than my time, since I'd have to baby-sit homegrown scripts all the time to get them working, and no homegrown script offers the speed of a Legato or Veritas or other costly backup solution, they just don't do the data interleaving. Yet.
John John Stoffel - Senior Unix Systems Administrator - Lucent Technologies stoffel@lucent.com - http://www.lucent.com - 978-952-7548 john.stoffel@ascend.com - http://www.ascend.com
On Wed, 12 Apr 2000, John Stoffel wrote:
- Are you willing to spend the money on NDMP capable software and a DLT7000 jukebox attached directory to the filer(s)? That would get the network out of the equation almost completely.
I've seen several comments like this in the past and I don't understand them. Why do you feel it is neccessary to spend money on NDMP capable software to be able to use a directly attached tape drive? It probably
Yes you can use the native dump/restore on the filer and a shell script to manage it. (Which we may end of having to do, depending on the 'choice' of NDMP software solutions.) But you can do much more by using a commercial package.
1) You get to use NDMP and avoide the insecurities associated with rsh. 2) The ease of scheduling/rescheduling of backups with a commercial package. 3) Ease of selection of files to restore. 4) File history (snapshots are great, but when users don't tell you they need a file until the snapshot copy has rolled over, they don't help you too much). 5) You get to beat someone else up when you're having problems with your backups. (okay relying on someone else can be a hindrance too.)
Those are the ones that pop immediately into mind.
purchase. Plus, it was an interesting problem to solve so I enjoyed doing it.
Agreed, I enjoyed putting together the script to do our Linux cluster.
On Wed, 12 Apr 2000, John Stoffel wrote:
- Are you willing to spend the money on NDMP capable software and a DLT7000 jukebox attached directory to the filer(s)? That would get the network out of the equation almost completely.
I've seen several comments like this in the past and I don't understand them. Why do you feel it is neccessary to spend money on NDMP capable software to be able to use a directly attached tape drive? It probably
Yes you can use the native dump/restore on the filer and a shell script to manage it. (Which we may end of having to do, depending on the 'choice' of NDMP software solutions.) But you can do much more by using a commercial package.
- You get to use NDMP and avoide the insecurities associated with rsh.
- The ease of scheduling/rescheduling of backups with a commercial
package. 3) Ease of selection of files to restore. 4) File history (snapshots are great, but when users don't tell you they need a file until the snapshot copy has rolled over, they don't help you too much). 5) You get to beat someone else up when you're having problems with your backups. (okay relying on someone else can be a hindrance too.)
Those are the ones that pop immediately into mind.
One more.
Your filer cannot automatically swap tapes in a library, just a stacker. To get full advantage of libraries, you need an NDMP solution. As data grows, you may want a big library.
Stephen Manley Data Availability and Management Lifeguard
We have been facing some explosive growth during the past 6 months and have grown past our current backup solution. We are currently backing up via a linux box that has our 740 nfs mounted. Backups are taking sometimes 24-30 hours for incrementals and all weekend for fulls.
What I am looking for is info on how others are backing up 200G volumes on 740's or 760's. I just purchased a pair of 760's and am going to need a nice stable reliable backup system for them.
Direct attach the tape drive (library) to your filer (via SCSI or fibre). That eliminates the bandwith problem. Then you need NDMP backup software to do the backup. We use BudTool but it's going to EOL soon, so we're in the process of evaluating an alternative (Veritas NetBackup vs Legato Networker vs Workstation Solution QuickRestore).
-
Brian Hostetter -
Cliff Nadler -
David H. Brierley -
Derek Kelly -
Eyal Traitel -
Jeff Bryer -
John Stoffel -
Stephen Manley