toasters

toasters@lists.teaparty.net

13531 discussions

Re: Data OnTap 5.3.5P2 and NDMP Backups
by Moshe Linzer 04 May '00

04 May '00

Pablo, I am seeing the same problem with Budtool 4.6.1a (and previously with 4.6). Budtool support is claiming that it is a hardware issue. They insisted I install OnTap 5.3.5, which is certified with 4.6.1a, but this did not solved the problem. After discussion with NetApp, and reading some of the recent posts here, I am going to 5.3.5R2P2, which is supposed to fix various NDMP problems. We are using a Breece Hill Q2.15 with 2 DLT7000 drives. The error I receive in the budtool log is: 04/27/00 23:38:39> Unload volume : B0113 04/27/00 23:40:02> ------------------------------------------------------------------ 04/27/00 23:40:02> ------ W A R N I N G ------- 04/27/00 23:40:02> Close Volume B0113 failed: Error 0 04/27/00 23:40:02> This failure, by itself, does not mean that the volume 04/27/00 23:40:02> or any of the data already on the volume is unusable. 04/27/00 23:40:02> ------------------------------------------------------------------ Then I have to manually eject the disk from the drive. I will write back if the new OnTap version helps. Moshe >From owner-toasters(a)mathworks.com Thu May 4 03:42:40 2000 >Date: Wed, 3 May 2000 17:17:05 -0500 (CDT) >From: Paul Mikulencak <paul.mikulencak(a)amd.com> >To: "Linn, Greg" <Greg.Linn(a)netapp.com> >Cc: "'toasters(a)mathworks.com'" <toasters(a)mathworks.com> >Subject: Re: Data OnTap 5.3.5P2 and NDMP Backups >Mime-Version: 1.0 >Precedence: bulk > > >On Wed, 3 May 2000, Linn, Greg wrote: > >> A NDMP-based backup problem has been identified in our Data OnTap 5.3.5P2 patch >> release. >> >> During development of Data OnTap 5.3.6, a deadlock condition was discovered >> which impacted the NDMP Java implementation. This condition typically manifests >> itself as an NDMP connection failure during resource depletion conditions. We >> believe it was first introduced 5.3.5P2. The problem has been resolved in >> 5.3.5R2P2 and in the forthcoming 5.3.6 release. >> >> If you are running 5.3.5P2, and are experiencing NDMP backup problems similar to >> the connection failure described above, we recommend that you upgrade to >> 5.3.5R2P2. > > >greg (and all), > > >i'd appreciate a pointer to a description of the "problem" in detail. > >we're wrestling with something here that i've been assuming was a hardware >problem on our storagetek 9740 tape changer, but which hasn't responded >to anything i or the storagetek field engineer have tried. > >our backups (budtool, 4.6.1, with the jumbo patch, dlt7000 drives scsi >attached to each F760 filer) have been failing because fully written >tapes are incompletely ejected from their drives. > >nothing in the messages file on either the budtool host or the netapps in >question. tape drives have been replaced (for one filer, three times), >and we even replaced the robotic hand. manual loading and unloading of >tapes via the storagetek touch panel work without failure. sysconfig -t >always shows the drives it should, and budtool always sees them. > >i have a hard time laying this at the feet of the particular revision of >data ontap we're running (currently 5.3.5R2P1, formerly 5.3.4R2), but >most of the hardware angles have been explored. i've been reading >messages on the toasters list that indicates that a bunch of folks are >having problems with their backups, but nothing quite like what we're >experiencing. > >anyone have any feedback? > > >pablo > > >email paul.mikulencak(a)amd.com >ext. 602-2448 "the word pessimism bothers me >pager 624-0929 because it is often used instead >cube 3.277.042 of the word lucidity." > > robert bresson > > ----------------------------------------------------------------------------- Moshe Linzer | Mastery of Unix offers real freedom. Unix Systems Manager | The price of freedom is always dear, but National Semiconductor, Israel | I'd rather pay for my freedom than live in Phone: 972-9-970-2247 | a bitmapped, pop-up-happy dungeon like NT. Fax: 972-9-970-2001 | Email: moshel(a)nsc.com | - Thomas Scoville, Unix Review -----------------------------------------------------------------------------

2 1

Veritas
by Shaw, Kevin 04 May '00

04 May '00

I'm looking at dropping Legato and purchasing Veritas for my backups. What are the pros and cons of Veritas? How is Veritas service and support? Is there a better enterprise backup solution? Thanks in advance.

5 4

RE: Network Appliance Upgrade to ONTAP 5.3.4R3P2 and Backups
by Moshe Linzer 04 May '00

04 May '00

Budtool has certified Budtool 4.6.1a in combination with OnTap 5.3.5. Unfortunately, NetApp says that 5.3.5 is buggy and recommends a patched release, which is not certified by Budtool. 5.3.5P2 did not work for me, because of the NDMP problems referred to in many emails (Budtool could not connect to NDMP on filer). 5.3.5R2P2 is SUPPOSED to fix those problems, and I will be testing that release today. Moshe >From owner-toasters(a)mathworks.com Tue May 2 21:28:08 2000 >Date: Tue, 2 May 2000 11:21:02 -0500 (CDT) >From: Jay Orr <orrjl(a)stl.nexen.com> >To: toasters(a)mathworks.com >Subject: RE: Network Appliance Upgrade to ONTAP 5.3.4R3P2 and Backups >MIME-Version: 1.0 >Precedence: bulk > > >Anyone know if budtool is keeping up? I've been pondering about upgrading >our F330's to 5.3.5r2 ..... > >On Tue, 2 May 2000, Kelly Wyatt wrote: > >> Keep in mind that VERITAS currently has no plans to qualify any 5.3.5 release. If you are concerned about keeping NetBackup and ONTAP in a supported mode you need to be at 5.3.4 (qualified). (They are currently qualifying 5.3.4R2 per email I received from VERITAS support. >> >> Kelly >> >> >> -- >> >> Kelly Wyatt, Kelly.Wyatt(a)SAS.com >> Systems Programmer >> Integrated Solutions Consulting >> SAS Institute Inc. / SAS Campus Drive / Cary, NC 27513 >> http://www.sas.com >> The power to know TM >> >> >> >> >> -----Original Message----- >> From: Micheler Klaus (MDCA Villach) [mailto:Klaus.Micheler@infineon.com] >> Sent: Tuesday, May 02, 2000 2:10 AM >> To: 'toasters(a)mathworks.com'; 'Kelly.Wyatt(a)sas.com'; 'armijo(a)cs.unm.edu' >> Subject: RE: Network Appliance Upgrade to ONTAP 5.3.4R3P2 and Backups >> >> >> >> On Thu, Apr 20, 2000 at 11:37:35AM -0400, Kelly Wyatt wrote: >> >> We upgraded out filers to NetApp Release 5.3.4R3P2 this weekend. Since >> then, NBU 3.2 keeps downing the tape drives on >> several, but not all of the filers (3 of 9). This seems to happen before, >> during and after backups. >> >> >> >> Has anyone else seen this? Thanks!! >> >> >we saw this on previous versions of NBU, 3.1.*, the work around was really >> >> >lame. cron executed a script which did something like: vmoprcmd -up # for >> >each drive. >> >> Hi there! >> >> We also have the same problem with this release of Ontap and Netbackup v3.2. >> According to Netapp the cause of this problem is a faulty Java Class >> (Garbage collection) which NDMP uses. >> This should also be the reason for the problems with Ontap 5.3.5xxx, where >> the problem got critical and backup stopped working. >> >> Netapp recommended us to switch to Ontap 5.3.5R2P1, where the NDMP-BUG is >> (hopefully) fixed. >> >> regards, Klaus Micheler >> Infineon Technologies DC >> > >----------- >Jay Orr >Systems Administrator >Fujitsu Nexion Inc. >St. Louis, MO > ----------------------------------------------------------------------------- Moshe Linzer | Mastery of Unix offers real freedom. Unix Systems Manager | The price of freedom is always dear, but National Semiconductor, Israel | I'd rather pay for my freedom than live in Phone: 972-9-970-2247 | a bitmapped, pop-up-happy dungeon like NT. Fax: 972-9-970-2001 | Email: moshel(a)nsc.com | - Thomas Scoville, Unix Review -----------------------------------------------------------------------------

1 0

RE: Data OnTap 5.3.5P2 and NDMP Backups
by Linn, Greg 04 May '00

04 May '00

Paul, I doubt that the 5.3.5P2 NDMP problem could account for the incomplete tape eject condition you described. Symptoms of the 5.3.5P2 problem are that NDMP on the filer deadlocks and stops responding to network requests from the backup application. This condition can be verified by attempting to telnet to the NDMP port (10K) on the filer. -Greg -----Original Message----- From: Paul Mikulencak [mailto:paul.mikulencak@amd.com] Sent: Wednesday, May 03, 2000 3:17 PM To: Linn, Greg Cc: 'toasters(a)mathworks.com' Subject: Re: Data OnTap 5.3.5P2 and NDMP Backups On Wed, 3 May 2000, Linn, Greg wrote: > A NDMP-based backup problem has been identified in our Data OnTap 5.3.5P2 patch > release. > > During development of Data OnTap 5.3.6, a deadlock condition was discovered > which impacted the NDMP Java implementation. This condition typically manifests > itself as an NDMP connection failure during resource depletion conditions. We > believe it was first introduced 5.3.5P2. The problem has been resolved in > 5.3.5R2P2 and in the forthcoming 5.3.6 release. > > If you are running 5.3.5P2, and are experiencing NDMP backup problems similar to > the connection failure described above, we recommend that you upgrade to > 5.3.5R2P2. greg (and all), i'd appreciate a pointer to a description of the "problem" in detail. we're wrestling with something here that i've been assuming was a hardware problem on our storagetek 9740 tape changer, but which hasn't responded to anything i or the storagetek field engineer have tried. our backups (budtool, 4.6.1, with the jumbo patch, dlt7000 drives scsi attached to each F760 filer) have been failing because fully written tapes are incompletely ejected from their drives. nothing in the messages file on either the budtool host or the netapps in question. tape drives have been replaced (for one filer, three times), and we even replaced the robotic hand. manual loading and unloading of tapes via the storagetek touch panel work without failure. sysconfig -t always shows the drives it should, and budtool always sees them. i have a hard time laying this at the feet of the particular revision of data ontap we're running (currently 5.3.5R2P1, formerly 5.3.4R2), but most of the hardware angles have been explored. i've been reading messages on the toasters list that indicates that a bunch of folks are having problems with their backups, but nothing quite like what we're experiencing. anyone have any feedback? pablo email paul.mikulencak(a)amd.com ext. 602-2448 "the word pessimism bothers me pager 624-0929 because it is often used instead cube 3.277.042 of the word lucidity." robert bresson

1 0

Veritas Netbackup woes?
by jmiddlebrooks＠datalink.com 04 May '00

04 May '00

Anybody seen Netbackup Status codes 99 when using Veritas Netbackup NDMP to back up your filer? I am consistantly getting Status code 99. I have a HP master server and I am running Ontap 5.3.4r2 on my F760 which has four drives attached to it, two off the onboard SCSI port and two off a second SCSI card in the Filer. What path do you use within Netbackup and are there any special attributes I should use? Anything special to put in the bp.conf? What is the best version of Ontap to run that has no Netbackup bugs? I have installed and used NDMP with a SUN master server several times and not experienced any problems. This is my only experience with a HP master and I am having no luck getting backups to be successfull all the time. I know these sound like Veritas questions but I was just curious as to what actual users are doing with success. Thanks in advance Jason Middlebrooks Systems Engineer Datalink Corp. 888-933-9327 x2970

1 0

Re: raid failure
by Robert L. Millner 03 May '00

03 May '00

"AT" == Aiello, Tony <Tony.Aiello(a)netapp.com>: "GDG" == G D Geen <geen(a)msp.sc.ti.com>: AT> As of the 5.3.2 release we added functionality to reassign blocks as they AT> occur. Prior to that we enabled automatic reassignment features on the disk AT> drives to do this. We found that the disks did not handle the reassignment AT> in all cases we'd like so we took control of that function. That would be The fact that whatever version this customer was running was not able to handle that common type of disk failure scares the hell out of me for what 5.3.5R2P1 will not be able to handle and how it will bite me when something breaks. Why didn't this make it in to ONTAP before something as useless as the Java GUI (ok, thats a personal opinion; but try and convince me that its worth more than the ability to appropriately handle disk failures). Netapp had years of experience from other vendors to draw from when it was formed. Part of that experience should have been comprehensive knowledge of the kinds of failures people see in the field. GDG> First and foremost, I take complete responsibility for my filers. I did so GDG> in my message to my management which was forwarded up through VP level. Of course, and there's always more that can be done at everyone's end. There's a long list of things I'd like to both do and see Netapp do. Important problems that vendors missed in their own tests make me really worry for that vendor's Q/A and what a product will cost me later when something pathological happens (after all, that's part of the cost of running the box). Nobody gets everything right immediately and that includes Netapp. Nobody's server is perfect for my environment. Everything is a balance of costs and some of those costs are how well it works on its own. What they miss has alot to do with what I consider my mistake to have been. Was my mistake not buying an EMC or a Sun in the first place or was my mistake not noticing the autosupport message (for example)? GDG> Having said that, I do agree with you. This disk should have failed by the GDG> filer two weeks prior. We at TI are now pushing NetApp to be proactive in We have been discussing certain things we'd like to see in their product as well (in case any one is curious I can elaborate more here). GDG> I still prefer to look at the autosupport messages. There is so much that I GDG> glean from these. The information is not just what disks to watch out for Everything in the autosupport messages can be gleaned from commands you run on the filer and checking the logs. I'd be really interested in hearing what kinds of tools you are working on if you don't mind sharing that info. GDG> I cannot disclose what fifteen hours of down time cost Texas Instruments, GDG> Inc. as that is proprietary information though we were fortunate that this GDG> was over a holiday weekend. *shiver* Rob

2 1

Data OnTap 5.3.5P2 and NDMP Backups
by Linn, Greg 03 May '00

03 May '00

A NDMP-based backup problem has been identified in our Data OnTap 5.3.5P2 patch release. During development of Data OnTap 5.3.6, a deadlock condition was discovered which impacted the NDMP Java implementation. This condition typically manifests itself as an NDMP connection failure during resource depletion conditions. We believe it was first introduced 5.3.5P2. The problem has been resolved in 5.3.5R2P2 and in the forthcoming 5.3.6 release. If you are running 5.3.5P2, and are experiencing NDMP backup problems similar to the connection failure described above, we recommend that you upgrade to 5.3.5R2P2. Greg Linn Manager, NDMP Development linn(a)netapp.com 408.822.3752 telephone 408.822.4457 fax

2 1

Re: raid failure
by Robert L. Millner 03 May '00

03 May '00

SL> filter out those hourly status messages. There's no way I can wade SL> through those weekly emails because /etc/messages is usually about SL> 5000 lines long. I'd probably forget to check the logs by hand, but Swatch is another really good tool for doing this. It can be used to compact a large number of entries into a summary and weed out useless information. Rob

1 0

RE: raid failure
by Aiello, Tony 03 May '00

03 May '00

Hello, I don't see the reference to the version of OnTap used but perhaps I can relate some information. As of the 5.3.2 release we added functionality to reassign blocks as they occur. Prior to that we enabled automatic reassignment features on the disk drives to do this. We found that the disks did not handle the reassignment in all cases we'd like so we took control of that function. That would be why you could see multiple reports of bad blocks showing up in subsequent scrubs. The disk did not do the reassignment and so this bad spot was left on the media. As of 5.3.2 messages would appear to the effect of: Sun Apr 30 04:38:50 MDT [isp2100_main]: Disk 5.14: sector 33601609 will be reassigned Reassignment means the device uses a different piece of media to store information for some block address. Not all errors returned from a disk can be handled by a block reassignment - really only those that come back as unrecoverable media errors can you repair by performing a block level reassignment. Should the reassigment fail for some reason then the disk is failed as sector-wise errors can lead to large reliability issues. Tony -------------------------- Tony Aiello, Mgr. Storage Software mailto:taiello@netapp.com Ph:(408)822-6515 > -----Original Message----- > From: Robert L. Millner [mailto:rmillner@transmeta.com] > Sent: Wednesday, May 03, 2000 10:08 AM > To: toasters > Subject: Re: raid failure > > > Hey, > > GDG> autosupport messages too. As I went back through the autosupport > GDG> logs that are e-mailed to me each week, I found that the problem > GDG> began approximately two weeks earlier. Every time a the > disk tried > GDG> to read a particular sector of the disk, an error messages would > GDG> appear the messages log indicating such an event had > occurred. Had > GDG> I not been busily working other issues, to the detriment of my > GDG> filers, I would have failed this disk at lease a week prior. > > > My immediate question to Netapp in this case would be why was the > periodic disk scrubbing not sufficient to cause the failed > sectors to be > replaced (this was going on for two weeks)? Why upon detection of the > block failure (after all, if a log message is generated, then > the filer > knows it happened) was the data not immediately reconstructed > elsewhere > and the disk blocks marked as unusable? A block sized RAID > reconstruction and re-write should be a trivial problem for > the filer to > solve. This is the kind of detail I'd expect a storage > vendor to place > a much higher priority on than having a java GUI. This is a > well known > way that disks fail; not some mysterious voodoo issue. I worry about > what other well known failure modes were left out till a later release > of ONTAP. > > > GDG> occurred. Had I not been busily working other issues, to the > GDG> detriment of my filers, I would have failed this disk at lease > GDG> a week prior. > > Had Netapp not been busily working other issues, to the > detriment of you > and your user's time and data this disk would have failed > itself or the > filer would have taken some other corrective action on its own. You > should have your own automated methods for looking for > problems (like a > script which analyses the logs and reports problems back to > you). Don't > be afraid to turn into a nasty bastard in a situation like this. None > of my users would hesitate for a moment and that may be your last > recourse to making sure that people understand the priority of certain > kinds of issues. > > > I realize that I am being brutal to Netapp here but that kind > of failure > would cost us more than twice what we have invested in our > entire Netapp > infrastructure in time to rebuild the data. It gives me that cold, > prickly, paranoid feeling about all the data we have on our filers. I > also realize that there are other potential problems that would have > caused a dual disk failure in one raid group. This specific problem > should have been dealt with more gracefully by the filer on > its own. If > it didn't, then your case alone should have been enough to > put it on the > 'Must Fix This Immediately!' list. > > > > Rob > > "You're just the little bundle of negative reinforcement I've > been looking for." -Mr. Gone >

2 1

Re: raid failure
by Jay Orr 03 May '00

03 May '00

On 3 May 2000, Mark D Fowle wrote: > I have heard a few horror stories lately about netapps and multi-disk raid > failures. Has anyone out there experienced this > and what did you do for recovery ? Where there any warnings? I have not had > this happen and would like to do as much > as possible to prevent it. I'll say this much for these Hardy Beasts - we had our A/C die on us overnight once, and I came in to find our filer dead. We had to swap out ALL the parts on the filer to bring it back up (it's a F330 we've had a few years and the room was 90+ degrees). However, didn't loose a drive! Knock on wood, we've never had a two-disk failure. To me, this illustrates that the chances of a double drive failure are quite low. Also, I'm always look at the drive lights as I walk by to make sure I didn't miss a log message about a drive failure. my $0.02... ----------- Jay Orr Systems Administrator Fujitsu Nexion Inc. St. Louis, MO

2 1

Re: raid failure
by Robert L. Millner 03 May '00

03 May '00

Hey, GDG> autosupport messages too. As I went back through the autosupport GDG> logs that are e-mailed to me each week, I found that the problem GDG> began approximately two weeks earlier. Every time a the disk tried GDG> to read a particular sector of the disk, an error messages would GDG> appear the messages log indicating such an event had occurred. Had GDG> I not been busily working other issues, to the detriment of my GDG> filers, I would have failed this disk at lease a week prior. My immediate question to Netapp in this case would be why was the periodic disk scrubbing not sufficient to cause the failed sectors to be replaced (this was going on for two weeks)? Why upon detection of the block failure (after all, if a log message is generated, then the filer knows it happened) was the data not immediately reconstructed elsewhere and the disk blocks marked as unusable? A block sized RAID reconstruction and re-write should be a trivial problem for the filer to solve. This is the kind of detail I'd expect a storage vendor to place a much higher priority on than having a java GUI. This is a well known way that disks fail; not some mysterious voodoo issue. I worry about what other well known failure modes were left out till a later release of ONTAP. GDG> occurred. Had I not been busily working other issues, to the GDG> detriment of my filers, I would have failed this disk at lease GDG> a week prior. Had Netapp not been busily working other issues, to the detriment of you and your user's time and data this disk would have failed itself or the filer would have taken some other corrective action on its own. You should have your own automated methods for looking for problems (like a script which analyses the logs and reports problems back to you). Don't be afraid to turn into a nasty bastard in a situation like this. None of my users would hesitate for a moment and that may be your last recourse to making sure that people understand the priority of certain kinds of issues. I realize that I am being brutal to Netapp here but that kind of failure would cost us more than twice what we have invested in our entire Netapp infrastructure in time to rebuild the data. It gives me that cold, prickly, paranoid feeling about all the data we have on our filers. I also realize that there are other potential problems that would have caused a dual disk failure in one raid group. This specific problem should have been dealt with more gracefully by the filer on its own. If it didn't, then your case alone should have been enough to put it on the 'Must Fix This Immediately!' list. Rob "You're just the little bundle of negative reinforcement I've been looking for." -Mr. Gone

2 1

RE: raid failure
by Walters, Mike 03 May '00

03 May '00

Just another tool for your armoury against the unthinkable: SnapMirror. This gives you the ability to keep an asynchronous copy of your volumes on another filer with little overhead. I won't bore you with detail (which you may already know), but you might want to have a scan down http://www.netapp.com/tech_library/3066.html for data protection strategies. Cheers Mike > -----Original Message----- > From: owner-dl-toasters(a)netapp.com > [mailto:owner-dl-toasters@netapp.com]On Behalf Of Mark D Fowle > Sent: 03 May 2000 12:00 > To: toasters > Subject: raid failure > > > I have heard a few horror stories lately about netapps and > multi-disk raid > failures. Has anyone out there experienced this > and what did you do for recovery ? Where there any > warnings? I have not had > this happen and would like to do as much > as possible to prevent it. > > Thanks, > ============================================================ > =================== > ======= > Mark Fowle > Caterpillar/BCP > Cary North Carolina > ============================================================ > =================== > ======= >

1 0

RE: raid failure
by Mohler, Jeff 03 May '00

03 May '00

Absolutely, ONTAP will just stop in its tracks, and will be just fine when you resolve the underlying hardware problem. -----Original Message----- From: Mike Mueller [mailto:Mike.Mueller@jpl.nasa.gov] Sent: Wednesday, May 03, 2000 9:38 AM To: toasters(a)mathworks.com Subject: Re: raid failure What about the case when a whole shelf goes away at once (power is lost to the shelf for instance)? This seems more likely than a multiple disk failure. Is this recoverable? -- Mike

1 0

RE: raid failure
by Mohler, Jeff 03 May '00

03 May '00

I have. If the disk failure is simply a bad block on another drive, Netapp can sometimes product a patch that will "skip" that block and allow you to continue to use the filesystem and then after it's done rebuilding from the first failed disk, you can fail and rebuild the second failed disk. But this can be time consuming since you'll have to recheck your filesystem for possible corruption, and even after you fix that you have some data files somewhere that are corrupted that you don't know about. --- There are actual 'secret ninja' commands & tricks to do this with the OS a customer is currently running, that can be used to get around a severely abused drive or two in the case of an emergency rebuild. I have had to use this with customers who more often than not..have just relocated thier toaster from one rack/building to another, which says a LOT about proper drive treatment..Ive seen people really get slam-happy on replacing drives in the shelves as well..seeding thier own future drive failure issues. Bruce describes the situation rather nicely. BUT..those secret commands and options..are things you will get from Tech Support should you have to need them (truly a rare experience for TS to have to use them) and you wont get em from me. *grin* PS: Renew your support, is your data worth it? http://now.netapp.com

1 0

Re: raid failure
by Bruce Sterling Woodcock 03 May '00

03 May '00

----- Original Message ----- From: "Mark D Fowle" <Fowle_Mark_D(a)CAT.com> To: "toasters" <toasters(a)mathworks.com> Sent: Wednesday, May 03, 2000 4:00 AM Subject: raid failure > I have heard a few horror stories lately about netapps and > multi-disk raid failures. I have heard a lot more horror stories regarding single-disk failures in non-raid situations. That's the whole point of getting raid... to move that pain from the chance of a single disk failure to a double disk failure, thus reducing it substantially. > Has anyone out there experienced this > and what did you do for recovery ? I have. If the disk failure is simply a bad block on another drive, Netapp can sometimes product a patch that will "skip" that block and allow you to continue to use the filesystem and then after it's done rebuilding from the first failed disk, you can fail and rebuild the second failed disk. But this can be time consuming since you'll have to recheck your filesystem for possible corruption, and even after you fix that you have some data files somewhere that are corrupted that you don't know about. Generally, in the double-disk failure case, like every other RAID vendor (unless you're running like a +1 configuration), you lose whatever is in that raid group/filesystem and you have to restore it from a mirrored copy or tape backup. > Where there any warnings? Back before RAID scrubbing, sometimes there were no warnings, because when 1 drive failed and the system went through every block on the other disks rebuilding the filesystem, it would hit a block that had gone bad some time before and yet had not been accessed and boom, you had a double disk failure. Now that you have RAID scrubbing, all those blocks are checked every week or so, and you will get any indications of a possibly bad drive. Also the filer does provide warnings in advance if it's having trouble talking to a particular drive and you can fail it proactively and replace it without waiting for it to fail (and increasing the risk that another drive will fail during that time). > I have not had this happen and would like to do as much > as possible to prevent it. There's not much to be done to prevent it... disks have a MTBF and the chance of two failures in the same time period are non- zero. The best thing to do is follow Netapp's directions regarding the operating environment and rack-mounting of your drives (so you don't have excessive vibration or temperatures or whatnot), watch your logs and fail drives if they appear to be going bad and replace them ASAP. Set your raid reconstruct speed high or disable access to that filesystem during reconstruction to make sure the reconstruction happens as quickly as possible to shorten the time window for a second disk failure. Make sure your RAID scrubs run periodically and check their results. Use smaller RAID group sizes and have frequent backups so that when a failure does occur it is as limited and painless as possible. And only use disk drives from Network Appliance. Bruce

1 0

RE: Network Appliance Upgrade to ONTAP 5.3.4R3P2 and Backups
by Steve Kappel 03 May '00

03 May '00

Kelly Wyatt wrote: >Keep in mind that VERITAS currently has no plans to qualify any 5.3.5 release. >If you are concerned about keeping NetBackup and ONTAP in a supported mode you >need to be at 5.3.4 (qualified). (They are currently qualifying 5.3.4R2 per >email I received from VERITAS support. I don't know where this information is coming from but nothing beyond 5.3.4 is being qualified by VERITAS at this time. There are no plans to qualify any 5.3.4x or 5.3.5 release. Obviously we will be qualifying some future release at some point. Klaus Micheler wrote: >I hope Netapp and Veritas will improve their quality with better >certifications in the near future >and tell us facts, not feelings. Please remember that the absence of qualification of a particular release is not the same as not supporting it. When VERITAS identifies an OnTap release as qualified, it means that we have done a level of testing within VERITAS to suggest to us that the release meets the minimum standards for use with NetBackup for NDMP. It does not mean that there are no issues. Releases that we have not qualified have not been tested by VERITAS. The lack of qualification does not automatically mean we don't support that release. It means that NetApp should be your first contact if you run into a problem that is new with that release. The simple truth is that it would be impossible for VERITAS to test every OnTap release. At some point we must assume that NetApp maintains full upward compatibility with subsequent releases and that reasonable regression tests have been performed. I would point out that the problems that have been seen have been identified by NetApp as an OnTap bug. This problem affects more than one backup application. The fix does not include a new version of NetBackup for NDMP or a NetBackup for NDMP patch. This is strictly an OnTap issue. If you have futher questions about this, please contact me directly. I'm the project lead and developer for NetBackup NDMP. __________________________________________________________________________ Steve Kappel steve.kappel(a)veritas.com VERITAS Software steve.kappel(a)iname.com (Personal)

2 1

raid failure
by Mark D Fowle 03 May '00

03 May '00

I have heard a few horror stories lately about netapps and multi-disk raid failures. Has anyone out there experienced this and what did you do for recovery ? Where there any warnings? I have not had this happen and would like to do as much as possible to prevent it. Thanks, =============================================================================== ======= Mark Fowle Caterpillar/BCP Cary North Carolina =============================================================================== =======

1 0

Article on RAIT backup strategies
by Derek Kelly 02 May '00

02 May '00

Here's a link to an interesting article on RAIT (Redundant Array of Independent Tapes) strategies... http://www2.itworld.com/cma/ett_article_frame/0,2848,1_495,00.html ...Well, at least *I* thought it was interesting. =) Regards, Derek Kelly GTC

1 0

RE: Network Appliance Upgrade to ONTAP 5.3.4R3P2 and Backups
by Kelly Wyatt 02 May '00

02 May '00

I understand *now* that this has been acknowledged as an ONTAP bug. That was not the case (or VERITAS support did not know that) when I opened my case two weeks ago. I have email from my case owner that 5.3.4R2 is currently being tested. That is where that came from. My point in going to an unqualified release as a concern is that I have been stuck in the middle in the past. If I proceed to a version of ONTAP and backups break, then downgrading to fix backups is not necessarily a pretty thing to do. I guess my biggest concern is... if backups can break, so can restores... I have been backing up filers with NetBackup and NDMP for almost two years. I have seen a lot of the good, the bad and the ugly :-). I tend to be cautious as a result. Perhaps I should not have said "in a supported mode" but rather, in a "known working configuration". In general, backups are still working. We run a script every three hours that ups the drives, and any backups suspended by a downed drive pick up and proceed. And I agree... testing every version of ONTAP would be impossible. Kelly -- Kelly Wyatt, Kelly.Wyatt(a)SAS.com Systems Programmer Integrated Solutions Consulting SAS Institute Inc. / SAS Campus Drive / Cary, NC 27513 http://www.sas.com The power to know TM -----Original Message----- From: Steve Kappel [mailto:steve.kappel@raistlin.min.ov.com] Sent: Tuesday, May 02, 2000 2:46 PM To: Kelly.Wyatt(a)sas.com; Klaus.Micheler(a)infineon.com Cc: toasters(a)mathworks.com Subject: RE: Network Appliance Upgrade to ONTAP 5.3.4R3P2 and Backups Kelly Wyatt wrote: >Keep in mind that VERITAS currently has no plans to qualify any 5.3.5 release. >If you are concerned about keeping NetBackup and ONTAP in a supported mode you >need to be at 5.3.4 (qualified). (They are currently qualifying 5.3.4R2 per >email I received from VERITAS support. I don't know where this information is coming from but nothing beyond 5.3.4 is being qualified by VERITAS at this time. There are no plans to qualify any 5.3.4x or 5.3.5 release. Obviously we will be qualifying some future release at some point. Klaus Micheler wrote: >I hope Netapp and Veritas will improve their quality with better >certifications in the near future >and tell us facts, not feelings. Please remember that the absence of qualification of a particular release is not the same as not supporting it. When VERITAS identifies an OnTap release as qualified, it means that we have done a level of testing within VERITAS to suggest to us that the release meets the minimum standards for use with NetBackup for NDMP. It does not mean that there are no issues. Releases that we have not qualified have not been tested by VERITAS. The lack of qualification does not automatically mean we don't support that release. It means that NetApp should be your first contact if you run into a problem that is new with that release. The simple truth is that it would be impossible for VERITAS to test every OnTap release. At some point we must assume that NetApp maintains full upward compatibility with subsequent releases and that reasonable regression tests have been performed. I would point out that the problems that have been seen have been identified by NetApp as an OnTap bug. This problem affects more than one backup application. The fix does not include a new version of NetBackup for NDMP or a NetBackup for NDMP patch. This is strictly an OnTap issue. If you have futher questions about this, please contact me directly. I'm the project lead and developer for NetBackup NDMP. __________________________________________________________________________ Steve Kappel steve.kappel(a)veritas.com VERITAS Software steve.kappel(a)iname.com (Personal)

1 0

RE: Network Appliance Upgrade to ONTAP 5.3.4R3P2 and Backups
by Kelly Wyatt 02 May '00

02 May '00

We have drives down several times a day. My updrives script runs every three hours. Starting last night, here is what the script did: 6PM Up-ping drive 0,1,8,9 9PM Up-ping drive 0,1,8,9 12AM Up-ping drive 0,1,2,6,9 3AM Up-ping drive 0,1,6,9 6AM Up-ping drive 0,1,6,8,9 All we have to do is up the drive & backups proceed properly. It seems like the downing of the drive is triggered by and end of job or need for a second tape. I also get the (ndmp_connect_open_and_auth), ndmp_result = -5, DOWN'ing it error in the messages file. -- Kelly Wyatt, Kelly.Wyatt(a)SAS.com Systems Programmer Integrated Solutions Consulting SAS Institute Inc. / SAS Campus Drive / Cary, NC 27513 http://www.sas.com The power to know TM

1 0

RE: Network Appliance Upgrade to ONTAP 5.3.4R3P2 and Backups
by Kelly Wyatt 02 May '00

02 May '00

Keep in mind that VERITAS currently has no plans to qualify any 5.3.5 release. If you are concerned about keeping NetBackup and ONTAP in a supported mode you need to be at 5.3.4 (qualified). (They are currently qualifying 5.3.4R2 per email I received from VERITAS support. Kelly -- Kelly Wyatt, Kelly.Wyatt(a)SAS.com Systems Programmer Integrated Solutions Consulting SAS Institute Inc. / SAS Campus Drive / Cary, NC 27513 http://www.sas.com The power to know TM -----Original Message----- From: Micheler Klaus (MDCA Villach) [mailto:Klaus.Micheler@infineon.com] Sent: Tuesday, May 02, 2000 2:10 AM To: 'toasters(a)mathworks.com'; 'Kelly.Wyatt(a)sas.com'; 'armijo(a)cs.unm.edu' Subject: RE: Network Appliance Upgrade to ONTAP 5.3.4R3P2 and Backups On Thu, Apr 20, 2000 at 11:37:35AM -0400, Kelly Wyatt wrote: >> We upgraded out filers to NetApp Release 5.3.4R3P2 this weekend. Since then, NBU 3.2 keeps downing the tape drives on several, but not all of the filers (3 of 9). This seems to happen before, during and after backups. >> >> Has anyone else seen this? Thanks!! >we saw this on previous versions of NBU, 3.1.*, the work around was really >lame. cron executed a script which did something like: vmoprcmd -up # for >each drive. Hi there! We also have the same problem with this release of Ontap and Netbackup v3.2. According to Netapp the cause of this problem is a faulty Java Class (Garbage collection) which NDMP uses. This should also be the reason for the problems with Ontap 5.3.5xxx, where the problem got critical and backup stopped working. Netapp recommended us to switch to Ontap 5.3.5R2P1, where the NDMP-BUG is (hopefully) fixed. regards, Klaus Micheler Infineon Technologies DC

2 1

AW: Network Appliance Upgrade to ONTAP 5.3.4R3P2 and Backups
by Micheler Klaus (MDCA Villach) 02 May '00

02 May '00

At the moment we run 5.3.4R3P2 and have only the problem with with DOWNING Drives (irregular, about 1-2 times a week). It looks like netapp has problems with Ontap Releases gretaer than 5.3.4R2. The last 3 months we used Ontap 5.3.4R2 without any problems. The switch to Ontap 5.3.5P2 (note that netapp and veritas promised us it will work) was a disaster! After switching to the new version everything looked fine. But after about 5 hours the drives went down and setting them up was impossible. Error Message: Apr 29 00:49:17 viws111 avrd[17817]: fatal error on device 3 (ndmp_connect_open_and_auth), ndmp_result = -5, DOWN'ing it. (messages on Netbackup Master) After that and some calls by veritas and netapp we downgraded to Ontap 5.3.4R3P2 and only(!) have the downing drives problem. Netapp then recommended us to switch to Ontap 5.3.5R2P1 and promised us to have that NDMP Bug fixed. By our next shutdown we would have switched to that version of Ontap. But now it looks like we will stay on 5.3.4x so long as we can afford it. I hope Netapp and Veritas will improve their quality with better certifications in the near future and tell us facts, not feelings. regards, Klaus Micheler -----Ursprüngliche Nachricht----- Von: Derek Kelly [mailto:derek.kelly@genomecorp.com] Gesendet am: Dienstag, 02. Mai 2000 16:35 An: Micheler Klaus (MDCA Villach) Cc: 'toasters(a)mathworks.com'; 'Kelly.Wyatt(a)sas.com'; 'armijo(a)cs.unm.edu' Betreff: RE: Network Appliance Upgrade to ONTAP 5.3.4R3P2 and Backups Yeah- we too are having all kinds of backup related problems with the one filer we have upgraded to 5.3.5R2P1. In our case, we can't back up the machine at all. Aaarrgh, this one really hurts. Not to mention the fact that NetApp had us upgrade to this version partially because of ECC errors we were getting. This is a brandy new F760 to boot. I'm not looking forward to begging for the downtime needed to fix this one. It'll be like... Me: "Uh, hi, um- we need some dowwntime one of the Filers." Them: "Why?" Me: "Ah- because we can't back it up suddenly." Them: "What!?- How long has THIS been going on?" Me: "Since we upgraded the OS last Friday." Them: "You mean that we could lose data from as far back as Friday?" Me: "Unfortunately. Yes." ... Well you all get the picture. I guess I'm just venting because not being able to backup a filer is a PRETTY BIG BUG! Derek Kelly On 5/2/00 at 8:10 AM, Klaus.Micheler(a)infineon.com (Micheler Klaus (MDCA Villach)) wrote: > > On Thu, Apr 20, 2000 at 11:37:35AM -0400, Kelly Wyatt wrote: > >> We upgraded out filers to NetApp Release 5.3.4R3P2 this weekend. > Since > then, NBU 3.2 keeps downing the tape drives on > several, but not all of the filers (3 of 9). This seems to happen > before, > during and after backups. > >> > >> Has anyone else seen this? Thanks!! > > >we saw this on previous versions of NBU, 3.1.*, the work around was > really > > >lame. cron executed a script which did something like: vmoprcmd -up # > for > >each drive. > > Hi there! > > We also have the same problem with this release of Ontap and Netbackup > v3.2. > According to Netapp the cause of this problem is a faulty Java Class > (Garbage collection) which NDMP uses. > This should also be the reason for the problems with Ontap 5.3.5xxx, > where > the problem got critical and backup stopped working. > > Netapp recommended us to switch to Ontap 5.3.5R2P1, where the NDMP-BUG > is > (hopefully) fixed. > > regards, Klaus Micheler > Infineon Technologies DC

1 0

FYI: TrueSAN Vs NetApp
by Philip Thomas 01 May '00

01 May '00

I got a chuckle out of their last "comparison" of TrueSAN...Vs F7x0 http://www.truesan.com/analysis.html#netapp ---philip thomas

2 1

What does this message mean
by Timothy Demarest 28 Apr '00

28 Apr '00

I keep seeing these messages on the Filer console: Server busy. Waiting 10 seconds then retrying Server busy. Waiting 10 seconds then retrying What do they mean? Tim

1 0

upgrade to 5.3.5R2
by Corris Randall 28 Apr '00

28 Apr '00

I use my Filer for mostly NFS access, and some CIFS access. People are starting to buy Windows 2000, which won't work in my setting because we don't have an NT domain (everything lives in a workgroup). Has anyone had any bad experiences with 5.3.5R2 ? -corris

1 0

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters