On Mon 14 Feb, 2000, Lisa Paton lcp@head-cfa.harvard.edu wrote:
Thanks for the review Lisa, word from teh front-line is far more valuable than marketing bumf.
<snip>
supported. I liked the fact that both QR and Veritas used an ISO standard tar format for the backup. This ruled Legato out early, since they use
Hmmm, I suspect from what I've learned from Veritas and others, that you don't see tar format on the tapes used for the basic backups. The reasoning being that the backup software usually interleaves several streams of data from clients to ensure streaming onto tapes (to extend tape lifetimes as well as reduce the overall backup window). Cloning the tapes in some manner allows the backup software to reduce the chance of data loss AND also to rationalise the interleaving so you do get tape files with coherent tar format inside.. but some folks don't clone tapes, so they may well never get tar-readable tapes.
I'd be happy to hear otherwise, but I wouldn't place too much value in that ISO standard tar formatting. I have to say that when I first heard, from Veritas Reps trying to sell their product to me, that the guys at CDC had nobbled gnu tar into an enterprise-class backup product (GPL notwithstanding) I just chuckled. Valid approach, but when you're backing up 100's of GB's or more, and spending through the nose for an enterprise-class backup product that manages every detail for you, the value of personally being able to grub through a tape using tar begins to pale a bit.
a proprietary format. I was impressed that Jim Ward (President of Workstation Solutions) came out to meet with me personally, AND could answer technical questions that the Veritas SE couldn't answer. WS impressed me as a small, technically savvy company that was very responsive to me during the evaluation period. I had a couple small glitches that were
Now that sounds like NetApp - who still send out plenipotentiaries to find out what their customers are really doing, and really need.
Lisa Paton -- End of excerpt from Lisa Paton
On Mon, 14 Feb 2000, mark wrote:
supported. I liked the fact that both QR and Veritas used an ISO standard tar format for the backup. This ruled Legato out early, since they use
Hmmm, I suspect from what I've learned from Veritas and others, that you don't see tar format on the tapes used for the basic backups. The reasoning being that the backup software usually interleaves several streams of data from clients to ensure streaming onto tapes (to extend tape lifetimes as well as reduce the overall backup window). Cloning the tapes in some manner allows the backup software to reduce the chance of data loss AND also to rationalize the interleaving so you do get tape files with coherent tar format inside.. but some folks don't clone tapes, so they may well never get tar-readable tapes.
My understanding of Budtool and Quick Restore was that you could pull tar files off of the tape directly. Generally, the desire for this ability is:
1) if the machine with the backup software goes down, you can still do recoveries, and 2) you can restore to differing systems.
I will grant that the Veritas solution of interleaving streams does sound well thought out and probably more efficient, but from an admin's perspective, you have to figure some day the worst will happen.
I'd be happy to hear otherwise, but I wouldn't place too much value in that ISO standard tar formatting. I have to say that when I first heard, from Veritas Reps trying to sell their product to me, that the guys at CDC had nobbled gnu tar into an enterprise-class backup product (GPL notwithstanding) I just chuckled.
Reminds me of Microsoft's argument against solaris : "Sun is trying to resurrect the 1960s-era mainframe paradigm and return computing to old-style timesharing." (www.dot-truth.com) Amazing how marketing appeals to feelings, not empirical data.
an enterprise-class backup product that manages every detail for you, the value of personally being able to grub through a tape using tar begins to pale a bit.
Until the system with the backup-product fails....
But that all said, your comments are appreciated and gives me food for thought. I guess it weighs in as one of the older Unix paradoxes - ease vs. utility (can't think of a better way to phrase that). Sure, who wouldn't want the easiest to use backup software? But from an admin perspective, we have to be able to pick up the pieces if it all explodes and put it back together ASAP.
----------- Jay Orr Systems Administrator Fujitsu Nexion Inc. St. Louis, MO
On Tue, 15 Feb 2000, Jay Orr wrote:
I will grant that the Veritas solution of interleaving streams does sound well thought out and probably more efficient, but from an admin's perspective, you have to figure some day the worst will happen.
Multiplexing data streams to tape always felt like a bad idea to me, although I agree that it is probably the only feasible way to keep a tape drive streaming if you have slow data sources. It's just one more thing that could go wrong when you most need your data. I think NetBackup has a cloning feature that "de-multiplexes" the streams back into contiguous chunks though.
Just to beat a dead horse here, I'd like to make some more comments on the multiplexing issue. I'm in favor, you can ingnore the rest of this if you don't care about this issue and are tired of it.
John
Brian> Multiplexing data streams to tape always felt like a bad idea Brian> to me, although I agree that it is probably the only feasible Brian> way to keep a tape drive streaming if you have slow data Brian> sources. It's just one more thing that could go wrong when you Brian> most need your data. I think NetBackup has a cloning feature Brian> that "de-multiplexes" the streams back into contiguous chunks Brian> though. -- Brian Tao (BT300, taob@risc.org) "Though this be Brian> madness, yet there is method in't"
The reason people use multiplexing is simple. Slow data steam from client(s), tape drive that requires a certain speed of data flow to be efficient.
The options, with no commentary, are:
1. Don't stream. To hell with the drive, to hell with fast backups, just put it all on one chunk on the drive so I can get it back by myself if need be if the data format is something standard.
2. Stage the backups to a local disk first, since they don't care about data flow performance. Then spool from local disk to local tape and get faster speeds. Requires alot of spare local disk. Can get data off tape if the format is something standard
3. Backup multiple streams of data at once to a single tape drive, interleaving data as you go. Now you can't get your data back without the multiplexing software.
Veritas uses both options 1 and 3. If you want fast backups, you have to multiplex (option 3), otherwise through put drops through the floor because you're using option 1. You have to pin the sales-rep down a bit to get them to admit that options 1 & 3 do not go together in NetBackup.
Amanda uses option 2, which helps, but means you need to spend money on disk you can't use for anything else, and which sits around idle alot of the time.
Legato uses option 3, which has worked for me for quite a few years now. There can be problems, and you can get screwed, but a bad tape will make a hash of any of these three options.
Now the big question is, how many people have had to actually grovel through the raw backup tapes created by the software above (or other software) to do a restore? Please don't include anything over three years old in this, since the explosion of data storage over the past three years has been phenominal, and grovelling through a 1gb tape is nothing when compared to searching through a 50Gb DLT 7000 backup.
My feeling is that it's just not done anymore. Mostly because if you depend on having your data around that badly, you will use other methods to insure it's safe keeping, such as:
- some form of RAID to protect against lost of hardware - some form of snapshot backups to protect against accidental deletion. In this case, speed of restore is crucial for one or a few files, generally not a full filesystem. - cloning and moving offsite of backups.
Another consideration is the backup window. If you only have 8 hours to do a backup, and you can't meet that deadline without using multiplexing, what are you going to do?
1. buy more tape drives and/or libraries. Lots of bucks. 2. Interleave the data and let the software worry about pulling what you need off tape later.
Generally, option 2 here is cheaper and works quite well.
Yet another consideration is how do you backup large databases efficiently? With regular filesystems which hold user directories, mail spools, etc, it doesn't matter as much if some of the data is backed up at 1am, and some at 2am. Even if they are making changes. The interelation of the files among themselves isn't that tight.
But with databases the equation changes, you need to lock down the entire database from change to make sure you get a consistent, clean backup. And since you want to minimize this window, multiplexing can be one way to maximize throughput from the system to tape. Or more tape drives, etc.
Online Indexing of saved files:
The next area of backup options is indexing. This is good if you want to browse your savesets and only restore the files you need, instead of the full saveset. Both Legato and NetBackup offer this, Amanda doesn't (well... it might be working now, but I haven't checked recently). The big advantage of indexing is that you can find what you want quickly and only restore what you need. No more need for a big disk to restore the full saveset to, them pull out what you need, then delete the excess.
Legato got a bad rep in their version 4.x software because thier indexing scheme was less than robust. I never had a problem with getting data off good tapes, even if I had to scan the entire tape to rebuild the online indexes. And of course the *size* of these indexes is a pain as well. I wish they had just taken some known good backend DB and used that instead. But with Legato 5.x, it's gotten alot more stable.
Disaster recovery:
If you have a disaster and need to restore alot of data, you have to think of how bad a disaster it is.
- the entire site went up in flames. Unless you have the hardware and procedures in place, it's not going to make much difference whether you need to rebuild the backup server first to restore the data, or if you have to wander through the tapes by hand looking for the data you need. In either case, it's not going to be a simple thing.
- If a major server goes down, but the backup server is fine, what's the problem? You're going to have to restore alot of data and that takes time. Multiplexing might slow down the restore if it's spread across the tape in lots of chunks, but I don't think that single savesets in one section of the tape will make that big a difference.
Again, if it's that important that you stay up, why aren't you using RAID in the first place?
- The legato disaster recover procedure works just fine. You get 15 day enabler codes right out of the box, so you don't have to worry about licenses in the short term. This gives you the time to get back up before having to deal with the backup system licenseing.
Conclusion:
1. I've written way too much here. :] 2. I like multiplexing. 3. Next dead horse beater, step right up!
<snip>
- Backup multiple streams of data at once to a single tape drive, interleaving data as you go. Now you can't get your data back without the multiplexing software.
<snip>
Legato uses option 3, which has worked for me for quite a few years now. There can be problems, and you can get screwed, but a bad tape will make a hash of any of these three options.
<snip>
Legato got a bad rep in their version 4.x software because thier indexing scheme was less than robust. I never had a problem with getting data off good tapes, even if I had to scan the entire tape to rebuild the online indexes. And of course the *size* of these indexes is a pain as well. I wish they had just taken some known good backend DB and used that instead. But with Legato 5.x, it's gotten alot more stable.
<snip>
Disaster recovery:
If you have a disaster and need to restore alot of data, you have to think of how bad a disaster it is.
If a major server goes down, but the backup server is fine, what's the problem? You're going to have to restore alot of data and that takes time. Multiplexing might slow down the restore if it's spread across the tape in lots of chunks, but I don't think that single savesets in one section of the tape will make that big a difference.
The legato disaster recover procedure works just fine. You get 15 day enabler codes right out of the box, so you don't have to worry about licenses in the short term. This gives you the time to get back up before having to deal with the backup system licenseing.
Ok, if I can resurrect this subject again, I have some more questions . Looking at the cost of replacing budtool is mucho $$$$, so we are thinking about considering the legato migration.
1) could you elaborate on "There can be problems, and you can get screwed"? I presume you mean more then having a bad tape....
2) "...gotten a lot more stable" What problems still exist with Legato 5?
3) You say with interleaving restore takes longer - by what magnitude? Restores are pretty long now with budtool tar's.
4) What is "legato disaster recover procedure" ?
----------- Jay Orr Systems Administrator Fujitsu Nexion Inc. St. Louis, MO
-
Brian Tao -
Jay Orr -
John Stoffel -
mds＠gbnet.net