toasters

toasters@lists.teaparty.net

13515 discussions

Re: Apology and Request (was Re: raid failure)
by dnb＠ccs.neu.edu 05 May '00

05 May '00

Cool, that's great to hear. Thanks for the response. Respectfully, David N. Blank-Edelman Director of Technology College of Computer Science Northeastern University John Denholm <johnd(a)ThePLAnet.net> writes: > Yeah, that's better now. We used to get that a lot - part of the > problem can be the interaction of Cisco switches and Netapps - the > Cisco can take a short while to figure out that the link is back up > to a Netapp and the mail falls on the floor while waiting for that. > > Some time ago, presumably at the insistence of ourselves and others, > Netapp inserted a 30 second wait and retry loop into bootup, and now > I think it gives autosupport 4 or 5 tries over a couple of minutes. > We always get autosupport off them now. Not that, I admit, we've > had a crash message in a while, but we never used to get them on > reboots either :< > > As for dealing with network or DNS or whatever not coming back, one thing you > can do is specify your mailhost wholely by IP - while fractionally more work to > maintain if the boxes on your network change frequently, it requires a little > less to go right. > > If you are worried about missing a crash, use snmp monitoring and grab > system.sysUptime.0, which is the uptime in hundredths of seconds. I run a > large number of netapps of different breeds, and I just run a perl script which > grabs the uptime off every one every 10 minutes. Any crashes are immediately > apparent. It could also write to file and compare current uptime to previous > uptime - if uptime drops, it can sms you, mail you, sound sires, flash lights, > whatever :> > > > Now I just wish they'd get it right with the caches. They still only try > mailing autosupport once :p > > J > > --------------------------------------------------------------------------- > # John Denholm johnd(a)theplanet.net # > # Webcache & Filer Administrator, Planet Online +44 113 207 6357 # > --------------------------------------------------------------------------- > > Error 404: There is no spoon

1 0

RE: Raid groups
by Mattel, Martin 05 May '00

05 May '00

Just want to add one word, as more spindles you have, as faster you can write. But if you have a graph for spindles / speed, you get a non-linear line. The delta is decreasing but is always positive. The delta is "feelable" for data-disk # up to 4-5, over that it is measurable but not significant. If you have 35 drives, I suggest: 1 HS 2 Parity 32 Data Drives That means, your raidsize will be 17 [2x (16+1)] Take care, that if you add another shelf, you need at least 2 drives in there (for a singe system) to build another raidgroup to your volume when you "vol add ´volume´ 1", because it is adding 2 (1 data, 1 parity). MM > -----Original Message----- > From: Mohler, Jeff > Sent: Donnerstag, 04. Mai 2000 23:55 > To: 'Sateesh Mucharla'; toasters(a)mathworks.com > Subject: RE: Raid groups > > > Question: > > Why would you have one large raid group? > > Your mathematical chances of losing the entire volume from a > 2-drive failure are > astonomically higher than if you use smaller RG's, it will > also take ages longer > to rebuild a failed drive in that large of a RG instead of a > small(er) one. > > Also consider that the small raid group will be a > 'bottleneck' or sorts. > > Its my understanding that every CP (consistency point) that > is written, > round-robins thru all available Raid Groups (RGs) in a > volume. As I see it, > that helps balance the data for when you want to read it back > from disk helping > to minimize head-seek times. X amount of data will take less > time and disk > revolutions to write out on the large RG, but -may- take multiple disk > revolutions to complete on the small RG..and extend to the > length of time it > would take to read the data as well. All things being equal, > equal sized RGs > would offer a more consistent performer. > > A balanced set of RGs will serve you better in the long run. > > -----Original Message----- > From: Sateesh Mucharla [mailto:sateesh@ampere.nsc.com] > Sent: Thursday, May 04, 2000 12:21 PM > To: toasters(a)mathworks.com > Subject: Raid groups > > > I am buying a F760 with 35 drives (all 36GB). > > As there is limitation of 28 drives per raid group ( i read > it NetApp Sysadmin > manual), i am thinking of having two raid groups one with 26 > and the other with > 7 drives. Rest two will be hot spares. But i am open for > better configuration. > > I would like to know whether i can configure all 33 drives > into one raid group - > > Is that possible? If so what are the disadvantages. (With 33 > drives in one raid > group, i loose only one drive as a parity drive - this is an > advantage). > > Can somebody suggest a better configuration ? > > Thanks > > Sateesh Mucharla > National Semiconductors >

1 0

Re: Apology and Request (was Re: raid failure)
by dnb＠ccs.neu.edu 05 May '00

05 May '00

Dave Hitz <hitz(a)netapp.com> writes: > I also have a request. Please turn on autosupport. That's > extremely important to us for two reasons: Quick question: once upon a time, machine-gone-kablooey autosupport mail was only sent when a filer booted. It was the Yoda scenario ("there is no retry, only do!"). If something went awry in that sending, the message was just dropped on the floor, no queuing, no retrying. We had a situation where our filer bounced twice, but on the first return to service it didn't have good net (or NIS, or name resolution, can't recall which). This meant that the first "here's the problem" mail never got to the mothership, but the "just booted, everything is peachy" mail did. Does the crash autosupport facility do anything more sophisticated these days? (he asks, because his filer hasn't crashed in many moons...). Thanks. Respectfully, David N. Blank-Edelman Director of Technology College of Computer Science Northeastern University

2 1

RE: 18GB drive performance versus 36BG
by Mohler, Jeff 05 May '00

05 May '00

What does your dataset look like? Homedirs, software development, chip development/simulation? And how many users..etc. There is also a new high-performance (spindle rate/seek rate) 9G drive now as well for those special environments. -----Original Message----- From: Mike(a)broaddaylight.com [mailto:Mike@broaddaylight.com] Sent: Thursday, May 04, 2000 4:36 PM To: toasters(a)mathworks.com Subject: 18GB drive performance versus 36BG I am considereing ordering a second F740 with 36BG drives. Big drives are nice, but there must be some tradeoffs: performance, restoration? If there is a significant performance difference I may go with the 18GB drives Michael Hopkins mike(a)BroadDaylight.com <mailto:mike@BroadDaylight.com> Systems Administrator phone 408.450.7450 Broad Daylight, Inc. fax 408.327.0177

2 1

18GB drive performance versus 36BG
by Mike＠broaddaylight.com 04 May '00

04 May '00

I am considereing ordering a second F740 with 36BG drives. Big drives are nice, but there must be some tradeoffs: performance, restoration? If there is a significant performance difference I may go with the 18GB drives Michael Hopkins mike(a)BroadDaylight.com <mailto:mike@BroadDaylight.com> Systems Administrator phone 408.450.7450 Broad Daylight, Inc. fax 408.327.0177

1 0

Seeking opinions on dump UI change
by Stephen Manley 04 May '00

04 May '00

Hey everybody. In the wake of Dave's email about UI changes, and the feelings of people on this list, I thought I'd pose a question to y'all. Currently, if you are running a dump on your console, and you hit end of tape or end of media, dump will pop up a message that says: toaster1> dump 0ufX rst0a *.o,*core* /vol/vol0/home DUMP: Dumping tape file 1 on rst0a ... DUMP: Change tape file on toaster1. DUMP: Mount tape file #2 DUMP: Is the new tape file mounted and ready to go?: ("yes" or "no") [If you respond "yes", it will write data to the tape device rst0a.] [If you respond "no", it will say:] DUMP: Do you want to abort?: ("yes" or "no") -> A problem that some people have raised is that there is no way to -> try to dump to rst1a or rst2a at this point They'd like to be able to -> write to a different device, but the question/answer framework will -> not currently allow you to do that. On the other hand, restore says: toaster2> restore rfD rst0a /vol/vol1/home_new ... RESTORE: Mount tape file 2 on toaster2 RESTORE: Enter ``none'' if there are no more tapes RESTORE: otherwise enter tape device name (default: rst0a) The benefit, of course, is that you can specify a different device if you desire. So here's the question - if we unified the interface for dump/restore on this point, would the added flexibility outweigh the pain of changing scripts? (This would come out in the next major release). So it would look like: toaster1> dump 0ufX rst0a *.o,*core* /vol/vol0/home DUMP: Dumping tape file 1 on rst0a ... DUMP: Mount tape file #2 on toaster1 DUMP: Enter ``none'' if there are no more tapes DUMP: otherwise enter tape device name (default: rst0a) (Note: it changed the "Mount tape file" message as well as the Q and A). Or, as an alternative, would people prefer something like: DUMP: Change tape file on toaster1. DUMP: Mount tape file #2 DUMP: Is the new tape file mounted and ready to go?: ("yes" or "no") no DUMP: Enter ``none'' if there are no more tapes DUMP: otherwise enter tape device name (default: rst0a) so that you could continue to respond "yes" and see no behavior change? But you would still get the advantage of being able to specify a new drive. The negative, of course, is that dump and restore ask the questions differently... So three questions: 1) Would you like to see: A) Unify dump and restore interface for asking questions B) The alternative I gave C) Something else entirely 2) If we did change this in the next major release, where would you like to see the "HEY! We changed this!" note? 3) Other suggestions for dump/restore UI changes you'd like? Thanks for reading this and thanks for your feedback, Stephen Manley Data Availability and Management Pollster

2 1

RE: Raid groups
by Mohler, Jeff 04 May '00

04 May '00

Question: Why would you have one large raid group? Your mathematical chances of losing the entire volume from a 2-drive failure are astonomically higher than if you use smaller RG's, it will also take ages longer to rebuild a failed drive in that large of a RG instead of a small(er) one. Also consider that the small raid group will be a 'bottleneck' or sorts. Its my understanding that every CP (consistency point) that is written, round-robins thru all available Raid Groups (RGs) in a volume. As I see it, that helps balance the data for when you want to read it back from disk helping to minimize head-seek times. X amount of data will take less time and disk revolutions to write out on the large RG, but -may- take multiple disk revolutions to complete on the small RG..and extend to the length of time it would take to read the data as well. All things being equal, equal sized RGs would offer a more consistent performer. A balanced set of RGs will serve you better in the long run. -----Original Message----- From: Sateesh Mucharla [mailto:sateesh@ampere.nsc.com] Sent: Thursday, May 04, 2000 12:21 PM To: toasters(a)mathworks.com Subject: Raid groups I am buying a F760 with 35 drives (all 36GB). As there is limitation of 28 drives per raid group ( i read it NetApp Sysadmin manual), i am thinking of having two raid groups one with 26 and the other with 7 drives. Rest two will be hot spares. But i am open for better configuration. I would like to know whether i can configure all 33 drives into one raid group - Is that possible? If so what are the disadvantages. (With 33 drives in one raid group, i loose only one drive as a parity drive - this is an advantage). Can somebody suggest a better configuration ? Thanks Sateesh Mucharla National Semiconductors

1 0

RE: raid failure
by Mohler, Jeff 04 May '00

04 May '00

I'd prefer either to have the tools myself, or only hear back if something was amiss. The danger with getting regular "everything is ok" type emails is that you could fail to notice an email informing you of a potential problem. --- Someone correct me if Im wrong.. But if you are properly registered on NOW with all of your filer serial numbers, you WILL be notified if something is amiss that requires NetApp action. When the case is opened against ASUP Email, you get an Email about a new case thats opened with you as the owner of the hardware in question.

1 0

Re: raid failure
by Richard L. Rhodes 04 May '00

04 May '00

On 3 May 2000, at 21:21, Robert L. Millner wrote: > mistake to have been. Was my mistake not buying an EMC or a Sun in the > first place or was my mistake not noticing the autosupport message (for > example)? We have some big EMC systems - they aren't perfect either! I've had: - databases crash due to scsi bus errors that emc couldn't explain (it was an emc problem, the same exact scsi error occured on several systems at the exact same time) - 30 min outages for microcode updates that turned into 4 hours - lost a 4 port scsi board in the emc box taking systems down Our emc sales rep insists that they've never lost data. Maybe . . . but I don't see where it's any less vulnerable to double- disk failures than any other raid system. Rick

2 1

Re: Data OnTap 5.3.5P2 and NDMP Backups
by Moshe Linzer 04 May '00

04 May '00

Pablo, I am seeing the same problem with Budtool 4.6.1a (and previously with 4.6). Budtool support is claiming that it is a hardware issue. They insisted I install OnTap 5.3.5, which is certified with 4.6.1a, but this did not solved the problem. After discussion with NetApp, and reading some of the recent posts here, I am going to 5.3.5R2P2, which is supposed to fix various NDMP problems. We are using a Breece Hill Q2.15 with 2 DLT7000 drives. The error I receive in the budtool log is: 04/27/00 23:38:39> Unload volume : B0113 04/27/00 23:40:02> ------------------------------------------------------------------ 04/27/00 23:40:02> ------ W A R N I N G ------- 04/27/00 23:40:02> Close Volume B0113 failed: Error 0 04/27/00 23:40:02> This failure, by itself, does not mean that the volume 04/27/00 23:40:02> or any of the data already on the volume is unusable. 04/27/00 23:40:02> ------------------------------------------------------------------ Then I have to manually eject the disk from the drive. I will write back if the new OnTap version helps. Moshe >From owner-toasters(a)mathworks.com Thu May 4 03:42:40 2000 >Date: Wed, 3 May 2000 17:17:05 -0500 (CDT) >From: Paul Mikulencak <paul.mikulencak(a)amd.com> >To: "Linn, Greg" <Greg.Linn(a)netapp.com> >Cc: "'toasters(a)mathworks.com'" <toasters(a)mathworks.com> >Subject: Re: Data OnTap 5.3.5P2 and NDMP Backups >Mime-Version: 1.0 >Precedence: bulk > > >On Wed, 3 May 2000, Linn, Greg wrote: > >> A NDMP-based backup problem has been identified in our Data OnTap 5.3.5P2 patch >> release. >> >> During development of Data OnTap 5.3.6, a deadlock condition was discovered >> which impacted the NDMP Java implementation. This condition typically manifests >> itself as an NDMP connection failure during resource depletion conditions. We >> believe it was first introduced 5.3.5P2. The problem has been resolved in >> 5.3.5R2P2 and in the forthcoming 5.3.6 release. >> >> If you are running 5.3.5P2, and are experiencing NDMP backup problems similar to >> the connection failure described above, we recommend that you upgrade to >> 5.3.5R2P2. > > >greg (and all), > > >i'd appreciate a pointer to a description of the "problem" in detail. > >we're wrestling with something here that i've been assuming was a hardware >problem on our storagetek 9740 tape changer, but which hasn't responded >to anything i or the storagetek field engineer have tried. > >our backups (budtool, 4.6.1, with the jumbo patch, dlt7000 drives scsi >attached to each F760 filer) have been failing because fully written >tapes are incompletely ejected from their drives. > >nothing in the messages file on either the budtool host or the netapps in >question. tape drives have been replaced (for one filer, three times), >and we even replaced the robotic hand. manual loading and unloading of >tapes via the storagetek touch panel work without failure. sysconfig -t >always shows the drives it should, and budtool always sees them. > >i have a hard time laying this at the feet of the particular revision of >data ontap we're running (currently 5.3.5R2P1, formerly 5.3.4R2), but >most of the hardware angles have been explored. i've been reading >messages on the toasters list that indicates that a bunch of folks are >having problems with their backups, but nothing quite like what we're >experiencing. > >anyone have any feedback? > > >pablo > > >email paul.mikulencak(a)amd.com >ext. 602-2448 "the word pessimism bothers me >pager 624-0929 because it is often used instead >cube 3.277.042 of the word lucidity." > > robert bresson > > ----------------------------------------------------------------------------- Moshe Linzer | Mastery of Unix offers real freedom. Unix Systems Manager | The price of freedom is always dear, but National Semiconductor, Israel | I'd rather pay for my freedom than live in Phone: 972-9-970-2247 | a bitmapped, pop-up-happy dungeon like NT. Fax: 972-9-970-2001 | Email: moshel(a)nsc.com | - Thomas Scoville, Unix Review -----------------------------------------------------------------------------

2 1

Veritas
by Shaw, Kevin 04 May '00

04 May '00

I'm looking at dropping Legato and purchasing Veritas for my backups. What are the pros and cons of Veritas? How is Veritas service and support? Is there a better enterprise backup solution? Thanks in advance.

5 4

RE: Network Appliance Upgrade to ONTAP 5.3.4R3P2 and Backups
by Moshe Linzer 04 May '00

04 May '00

Budtool has certified Budtool 4.6.1a in combination with OnTap 5.3.5. Unfortunately, NetApp says that 5.3.5 is buggy and recommends a patched release, which is not certified by Budtool. 5.3.5P2 did not work for me, because of the NDMP problems referred to in many emails (Budtool could not connect to NDMP on filer). 5.3.5R2P2 is SUPPOSED to fix those problems, and I will be testing that release today. Moshe >From owner-toasters(a)mathworks.com Tue May 2 21:28:08 2000 >Date: Tue, 2 May 2000 11:21:02 -0500 (CDT) >From: Jay Orr <orrjl(a)stl.nexen.com> >To: toasters(a)mathworks.com >Subject: RE: Network Appliance Upgrade to ONTAP 5.3.4R3P2 and Backups >MIME-Version: 1.0 >Precedence: bulk > > >Anyone know if budtool is keeping up? I've been pondering about upgrading >our F330's to 5.3.5r2 ..... > >On Tue, 2 May 2000, Kelly Wyatt wrote: > >> Keep in mind that VERITAS currently has no plans to qualify any 5.3.5 release. If you are concerned about keeping NetBackup and ONTAP in a supported mode you need to be at 5.3.4 (qualified). (They are currently qualifying 5.3.4R2 per email I received from VERITAS support. >> >> Kelly >> >> >> -- >> >> Kelly Wyatt, Kelly.Wyatt(a)SAS.com >> Systems Programmer >> Integrated Solutions Consulting >> SAS Institute Inc. / SAS Campus Drive / Cary, NC 27513 >> http://www.sas.com >> The power to know TM >> >> >> >> >> -----Original Message----- >> From: Micheler Klaus (MDCA Villach) [mailto:Klaus.Micheler@infineon.com] >> Sent: Tuesday, May 02, 2000 2:10 AM >> To: 'toasters(a)mathworks.com'; 'Kelly.Wyatt(a)sas.com'; 'armijo(a)cs.unm.edu' >> Subject: RE: Network Appliance Upgrade to ONTAP 5.3.4R3P2 and Backups >> >> >> >> On Thu, Apr 20, 2000 at 11:37:35AM -0400, Kelly Wyatt wrote: >> >> We upgraded out filers to NetApp Release 5.3.4R3P2 this weekend. Since >> then, NBU 3.2 keeps downing the tape drives on >> several, but not all of the filers (3 of 9). This seems to happen before, >> during and after backups. >> >> >> >> Has anyone else seen this? Thanks!! >> >> >we saw this on previous versions of NBU, 3.1.*, the work around was really >> >> >lame. cron executed a script which did something like: vmoprcmd -up # for >> >each drive. >> >> Hi there! >> >> We also have the same problem with this release of Ontap and Netbackup v3.2. >> According to Netapp the cause of this problem is a faulty Java Class >> (Garbage collection) which NDMP uses. >> This should also be the reason for the problems with Ontap 5.3.5xxx, where >> the problem got critical and backup stopped working. >> >> Netapp recommended us to switch to Ontap 5.3.5R2P1, where the NDMP-BUG is >> (hopefully) fixed. >> >> regards, Klaus Micheler >> Infineon Technologies DC >> > >----------- >Jay Orr >Systems Administrator >Fujitsu Nexion Inc. >St. Louis, MO > ----------------------------------------------------------------------------- Moshe Linzer | Mastery of Unix offers real freedom. Unix Systems Manager | The price of freedom is always dear, but National Semiconductor, Israel | I'd rather pay for my freedom than live in Phone: 972-9-970-2247 | a bitmapped, pop-up-happy dungeon like NT. Fax: 972-9-970-2001 | Email: moshel(a)nsc.com | - Thomas Scoville, Unix Review -----------------------------------------------------------------------------

1 0

RE: Data OnTap 5.3.5P2 and NDMP Backups
by Linn, Greg 04 May '00

04 May '00

Paul, I doubt that the 5.3.5P2 NDMP problem could account for the incomplete tape eject condition you described. Symptoms of the 5.3.5P2 problem are that NDMP on the filer deadlocks and stops responding to network requests from the backup application. This condition can be verified by attempting to telnet to the NDMP port (10K) on the filer. -Greg -----Original Message----- From: Paul Mikulencak [mailto:paul.mikulencak@amd.com] Sent: Wednesday, May 03, 2000 3:17 PM To: Linn, Greg Cc: 'toasters(a)mathworks.com' Subject: Re: Data OnTap 5.3.5P2 and NDMP Backups On Wed, 3 May 2000, Linn, Greg wrote: > A NDMP-based backup problem has been identified in our Data OnTap 5.3.5P2 patch > release. > > During development of Data OnTap 5.3.6, a deadlock condition was discovered > which impacted the NDMP Java implementation. This condition typically manifests > itself as an NDMP connection failure during resource depletion conditions. We > believe it was first introduced 5.3.5P2. The problem has been resolved in > 5.3.5R2P2 and in the forthcoming 5.3.6 release. > > If you are running 5.3.5P2, and are experiencing NDMP backup problems similar to > the connection failure described above, we recommend that you upgrade to > 5.3.5R2P2. greg (and all), i'd appreciate a pointer to a description of the "problem" in detail. we're wrestling with something here that i've been assuming was a hardware problem on our storagetek 9740 tape changer, but which hasn't responded to anything i or the storagetek field engineer have tried. our backups (budtool, 4.6.1, with the jumbo patch, dlt7000 drives scsi attached to each F760 filer) have been failing because fully written tapes are incompletely ejected from their drives. nothing in the messages file on either the budtool host or the netapps in question. tape drives have been replaced (for one filer, three times), and we even replaced the robotic hand. manual loading and unloading of tapes via the storagetek touch panel work without failure. sysconfig -t always shows the drives it should, and budtool always sees them. i have a hard time laying this at the feet of the particular revision of data ontap we're running (currently 5.3.5R2P1, formerly 5.3.4R2), but most of the hardware angles have been explored. i've been reading messages on the toasters list that indicates that a bunch of folks are having problems with their backups, but nothing quite like what we're experiencing. anyone have any feedback? pablo email paul.mikulencak(a)amd.com ext. 602-2448 "the word pessimism bothers me pager 624-0929 because it is often used instead cube 3.277.042 of the word lucidity." robert bresson

1 0

Veritas Netbackup woes?
by jmiddlebrooks＠datalink.com 04 May '00

04 May '00

Anybody seen Netbackup Status codes 99 when using Veritas Netbackup NDMP to back up your filer? I am consistantly getting Status code 99. I have a HP master server and I am running Ontap 5.3.4r2 on my F760 which has four drives attached to it, two off the onboard SCSI port and two off a second SCSI card in the Filer. What path do you use within Netbackup and are there any special attributes I should use? Anything special to put in the bp.conf? What is the best version of Ontap to run that has no Netbackup bugs? I have installed and used NDMP with a SUN master server several times and not experienced any problems. This is my only experience with a HP master and I am having no luck getting backups to be successfull all the time. I know these sound like Veritas questions but I was just curious as to what actual users are doing with success. Thanks in advance Jason Middlebrooks Systems Engineer Datalink Corp. 888-933-9327 x2970

1 0

Re: raid failure
by Robert L. Millner 03 May '00

03 May '00

"AT" == Aiello, Tony <Tony.Aiello(a)netapp.com>: "GDG" == G D Geen <geen(a)msp.sc.ti.com>: AT> As of the 5.3.2 release we added functionality to reassign blocks as they AT> occur. Prior to that we enabled automatic reassignment features on the disk AT> drives to do this. We found that the disks did not handle the reassignment AT> in all cases we'd like so we took control of that function. That would be The fact that whatever version this customer was running was not able to handle that common type of disk failure scares the hell out of me for what 5.3.5R2P1 will not be able to handle and how it will bite me when something breaks. Why didn't this make it in to ONTAP before something as useless as the Java GUI (ok, thats a personal opinion; but try and convince me that its worth more than the ability to appropriately handle disk failures). Netapp had years of experience from other vendors to draw from when it was formed. Part of that experience should have been comprehensive knowledge of the kinds of failures people see in the field. GDG> First and foremost, I take complete responsibility for my filers. I did so GDG> in my message to my management which was forwarded up through VP level. Of course, and there's always more that can be done at everyone's end. There's a long list of things I'd like to both do and see Netapp do. Important problems that vendors missed in their own tests make me really worry for that vendor's Q/A and what a product will cost me later when something pathological happens (after all, that's part of the cost of running the box). Nobody gets everything right immediately and that includes Netapp. Nobody's server is perfect for my environment. Everything is a balance of costs and some of those costs are how well it works on its own. What they miss has alot to do with what I consider my mistake to have been. Was my mistake not buying an EMC or a Sun in the first place or was my mistake not noticing the autosupport message (for example)? GDG> Having said that, I do agree with you. This disk should have failed by the GDG> filer two weeks prior. We at TI are now pushing NetApp to be proactive in We have been discussing certain things we'd like to see in their product as well (in case any one is curious I can elaborate more here). GDG> I still prefer to look at the autosupport messages. There is so much that I GDG> glean from these. The information is not just what disks to watch out for Everything in the autosupport messages can be gleaned from commands you run on the filer and checking the logs. I'd be really interested in hearing what kinds of tools you are working on if you don't mind sharing that info. GDG> I cannot disclose what fifteen hours of down time cost Texas Instruments, GDG> Inc. as that is proprietary information though we were fortunate that this GDG> was over a holiday weekend. *shiver* Rob

2 1

Data OnTap 5.3.5P2 and NDMP Backups
by Linn, Greg 03 May '00

03 May '00

A NDMP-based backup problem has been identified in our Data OnTap 5.3.5P2 patch release. During development of Data OnTap 5.3.6, a deadlock condition was discovered which impacted the NDMP Java implementation. This condition typically manifests itself as an NDMP connection failure during resource depletion conditions. We believe it was first introduced 5.3.5P2. The problem has been resolved in 5.3.5R2P2 and in the forthcoming 5.3.6 release. If you are running 5.3.5P2, and are experiencing NDMP backup problems similar to the connection failure described above, we recommend that you upgrade to 5.3.5R2P2. Greg Linn Manager, NDMP Development linn(a)netapp.com 408.822.3752 telephone 408.822.4457 fax

2 1

Re: raid failure
by Robert L. Millner 03 May '00

03 May '00

SL> filter out those hourly status messages. There's no way I can wade SL> through those weekly emails because /etc/messages is usually about SL> 5000 lines long. I'd probably forget to check the logs by hand, but Swatch is another really good tool for doing this. It can be used to compact a large number of entries into a summary and weed out useless information. Rob

1 0

RE: raid failure
by Aiello, Tony 03 May '00

03 May '00

Hello, I don't see the reference to the version of OnTap used but perhaps I can relate some information. As of the 5.3.2 release we added functionality to reassign blocks as they occur. Prior to that we enabled automatic reassignment features on the disk drives to do this. We found that the disks did not handle the reassignment in all cases we'd like so we took control of that function. That would be why you could see multiple reports of bad blocks showing up in subsequent scrubs. The disk did not do the reassignment and so this bad spot was left on the media. As of 5.3.2 messages would appear to the effect of: Sun Apr 30 04:38:50 MDT [isp2100_main]: Disk 5.14: sector 33601609 will be reassigned Reassignment means the device uses a different piece of media to store information for some block address. Not all errors returned from a disk can be handled by a block reassignment - really only those that come back as unrecoverable media errors can you repair by performing a block level reassignment. Should the reassigment fail for some reason then the disk is failed as sector-wise errors can lead to large reliability issues. Tony -------------------------- Tony Aiello, Mgr. Storage Software mailto:taiello@netapp.com Ph:(408)822-6515 > -----Original Message----- > From: Robert L. Millner [mailto:rmillner@transmeta.com] > Sent: Wednesday, May 03, 2000 10:08 AM > To: toasters > Subject: Re: raid failure > > > Hey, > > GDG> autosupport messages too. As I went back through the autosupport > GDG> logs that are e-mailed to me each week, I found that the problem > GDG> began approximately two weeks earlier. Every time a the > disk tried > GDG> to read a particular sector of the disk, an error messages would > GDG> appear the messages log indicating such an event had > occurred. Had > GDG> I not been busily working other issues, to the detriment of my > GDG> filers, I would have failed this disk at lease a week prior. > > > My immediate question to Netapp in this case would be why was the > periodic disk scrubbing not sufficient to cause the failed > sectors to be > replaced (this was going on for two weeks)? Why upon detection of the > block failure (after all, if a log message is generated, then > the filer > knows it happened) was the data not immediately reconstructed > elsewhere > and the disk blocks marked as unusable? A block sized RAID > reconstruction and re-write should be a trivial problem for > the filer to > solve. This is the kind of detail I'd expect a storage > vendor to place > a much higher priority on than having a java GUI. This is a > well known > way that disks fail; not some mysterious voodoo issue. I worry about > what other well known failure modes were left out till a later release > of ONTAP. > > > GDG> occurred. Had I not been busily working other issues, to the > GDG> detriment of my filers, I would have failed this disk at lease > GDG> a week prior. > > Had Netapp not been busily working other issues, to the > detriment of you > and your user's time and data this disk would have failed > itself or the > filer would have taken some other corrective action on its own. You > should have your own automated methods for looking for > problems (like a > script which analyses the logs and reports problems back to > you). Don't > be afraid to turn into a nasty bastard in a situation like this. None > of my users would hesitate for a moment and that may be your last > recourse to making sure that people understand the priority of certain > kinds of issues. > > > I realize that I am being brutal to Netapp here but that kind > of failure > would cost us more than twice what we have invested in our > entire Netapp > infrastructure in time to rebuild the data. It gives me that cold, > prickly, paranoid feeling about all the data we have on our filers. I > also realize that there are other potential problems that would have > caused a dual disk failure in one raid group. This specific problem > should have been dealt with more gracefully by the filer on > its own. If > it didn't, then your case alone should have been enough to > put it on the > 'Must Fix This Immediately!' list. > > > > Rob > > "You're just the little bundle of negative reinforcement I've > been looking for." -Mr. Gone >

2 1

Re: raid failure
by Jay Orr 03 May '00

03 May '00

On 3 May 2000, Mark D Fowle wrote: > I have heard a few horror stories lately about netapps and multi-disk raid > failures. Has anyone out there experienced this > and what did you do for recovery ? Where there any warnings? I have not had > this happen and would like to do as much > as possible to prevent it. I'll say this much for these Hardy Beasts - we had our A/C die on us overnight once, and I came in to find our filer dead. We had to swap out ALL the parts on the filer to bring it back up (it's a F330 we've had a few years and the room was 90+ degrees). However, didn't loose a drive! Knock on wood, we've never had a two-disk failure. To me, this illustrates that the chances of a double drive failure are quite low. Also, I'm always look at the drive lights as I walk by to make sure I didn't miss a log message about a drive failure. my $0.02... ----------- Jay Orr Systems Administrator Fujitsu Nexion Inc. St. Louis, MO

2 1

Re: raid failure
by Robert L. Millner 03 May '00

03 May '00

Hey, GDG> autosupport messages too. As I went back through the autosupport GDG> logs that are e-mailed to me each week, I found that the problem GDG> began approximately two weeks earlier. Every time a the disk tried GDG> to read a particular sector of the disk, an error messages would GDG> appear the messages log indicating such an event had occurred. Had GDG> I not been busily working other issues, to the detriment of my GDG> filers, I would have failed this disk at lease a week prior. My immediate question to Netapp in this case would be why was the periodic disk scrubbing not sufficient to cause the failed sectors to be replaced (this was going on for two weeks)? Why upon detection of the block failure (after all, if a log message is generated, then the filer knows it happened) was the data not immediately reconstructed elsewhere and the disk blocks marked as unusable? A block sized RAID reconstruction and re-write should be a trivial problem for the filer to solve. This is the kind of detail I'd expect a storage vendor to place a much higher priority on than having a java GUI. This is a well known way that disks fail; not some mysterious voodoo issue. I worry about what other well known failure modes were left out till a later release of ONTAP. GDG> occurred. Had I not been busily working other issues, to the GDG> detriment of my filers, I would have failed this disk at lease GDG> a week prior. Had Netapp not been busily working other issues, to the detriment of you and your user's time and data this disk would have failed itself or the filer would have taken some other corrective action on its own. You should have your own automated methods for looking for problems (like a script which analyses the logs and reports problems back to you). Don't be afraid to turn into a nasty bastard in a situation like this. None of my users would hesitate for a moment and that may be your last recourse to making sure that people understand the priority of certain kinds of issues. I realize that I am being brutal to Netapp here but that kind of failure would cost us more than twice what we have invested in our entire Netapp infrastructure in time to rebuild the data. It gives me that cold, prickly, paranoid feeling about all the data we have on our filers. I also realize that there are other potential problems that would have caused a dual disk failure in one raid group. This specific problem should have been dealt with more gracefully by the filer on its own. If it didn't, then your case alone should have been enough to put it on the 'Must Fix This Immediately!' list. Rob "You're just the little bundle of negative reinforcement I've been looking for." -Mr. Gone

2 1

RE: raid failure
by Walters, Mike 03 May '00

03 May '00

Just another tool for your armoury against the unthinkable: SnapMirror. This gives you the ability to keep an asynchronous copy of your volumes on another filer with little overhead. I won't bore you with detail (which you may already know), but you might want to have a scan down http://www.netapp.com/tech_library/3066.html for data protection strategies. Cheers Mike > -----Original Message----- > From: owner-dl-toasters(a)netapp.com > [mailto:owner-dl-toasters@netapp.com]On Behalf Of Mark D Fowle > Sent: 03 May 2000 12:00 > To: toasters > Subject: raid failure > > > I have heard a few horror stories lately about netapps and > multi-disk raid > failures. Has anyone out there experienced this > and what did you do for recovery ? Where there any > warnings? I have not had > this happen and would like to do as much > as possible to prevent it. > > Thanks, > ============================================================ > =================== > ======= > Mark Fowle > Caterpillar/BCP > Cary North Carolina > ============================================================ > =================== > ======= >

1 0

RE: raid failure
by Mohler, Jeff 03 May '00

03 May '00

Absolutely, ONTAP will just stop in its tracks, and will be just fine when you resolve the underlying hardware problem. -----Original Message----- From: Mike Mueller [mailto:Mike.Mueller@jpl.nasa.gov] Sent: Wednesday, May 03, 2000 9:38 AM To: toasters(a)mathworks.com Subject: Re: raid failure What about the case when a whole shelf goes away at once (power is lost to the shelf for instance)? This seems more likely than a multiple disk failure. Is this recoverable? -- Mike

1 0

RE: raid failure
by Mohler, Jeff 03 May '00

03 May '00

I have. If the disk failure is simply a bad block on another drive, Netapp can sometimes product a patch that will "skip" that block and allow you to continue to use the filesystem and then after it's done rebuilding from the first failed disk, you can fail and rebuild the second failed disk. But this can be time consuming since you'll have to recheck your filesystem for possible corruption, and even after you fix that you have some data files somewhere that are corrupted that you don't know about. --- There are actual 'secret ninja' commands & tricks to do this with the OS a customer is currently running, that can be used to get around a severely abused drive or two in the case of an emergency rebuild. I have had to use this with customers who more often than not..have just relocated thier toaster from one rack/building to another, which says a LOT about proper drive treatment..Ive seen people really get slam-happy on replacing drives in the shelves as well..seeding thier own future drive failure issues. Bruce describes the situation rather nicely. BUT..those secret commands and options..are things you will get from Tech Support should you have to need them (truly a rare experience for TS to have to use them) and you wont get em from me. *grin* PS: Renew your support, is your data worth it? http://now.netapp.com

1 0

Re: raid failure
by Bruce Sterling Woodcock 03 May '00

03 May '00

----- Original Message ----- From: "Mark D Fowle" <Fowle_Mark_D(a)CAT.com> To: "toasters" <toasters(a)mathworks.com> Sent: Wednesday, May 03, 2000 4:00 AM Subject: raid failure > I have heard a few horror stories lately about netapps and > multi-disk raid failures. I have heard a lot more horror stories regarding single-disk failures in non-raid situations. That's the whole point of getting raid... to move that pain from the chance of a single disk failure to a double disk failure, thus reducing it substantially. > Has anyone out there experienced this > and what did you do for recovery ? I have. If the disk failure is simply a bad block on another drive, Netapp can sometimes product a patch that will "skip" that block and allow you to continue to use the filesystem and then after it's done rebuilding from the first failed disk, you can fail and rebuild the second failed disk. But this can be time consuming since you'll have to recheck your filesystem for possible corruption, and even after you fix that you have some data files somewhere that are corrupted that you don't know about. Generally, in the double-disk failure case, like every other RAID vendor (unless you're running like a +1 configuration), you lose whatever is in that raid group/filesystem and you have to restore it from a mirrored copy or tape backup. > Where there any warnings? Back before RAID scrubbing, sometimes there were no warnings, because when 1 drive failed and the system went through every block on the other disks rebuilding the filesystem, it would hit a block that had gone bad some time before and yet had not been accessed and boom, you had a double disk failure. Now that you have RAID scrubbing, all those blocks are checked every week or so, and you will get any indications of a possibly bad drive. Also the filer does provide warnings in advance if it's having trouble talking to a particular drive and you can fail it proactively and replace it without waiting for it to fail (and increasing the risk that another drive will fail during that time). > I have not had this happen and would like to do as much > as possible to prevent it. There's not much to be done to prevent it... disks have a MTBF and the chance of two failures in the same time period are non- zero. The best thing to do is follow Netapp's directions regarding the operating environment and rack-mounting of your drives (so you don't have excessive vibration or temperatures or whatnot), watch your logs and fail drives if they appear to be going bad and replace them ASAP. Set your raid reconstruct speed high or disable access to that filesystem during reconstruction to make sure the reconstruction happens as quickly as possible to shorten the time window for a second disk failure. Make sure your RAID scrubs run periodically and check their results. Use smaller RAID group sizes and have frequent backups so that when a failure does occur it is as limited and painless as possible. And only use disk drives from Network Appliance. Bruce

1 0

RE: Network Appliance Upgrade to ONTAP 5.3.4R3P2 and Backups
by Steve Kappel 03 May '00

03 May '00

Kelly Wyatt wrote: >Keep in mind that VERITAS currently has no plans to qualify any 5.3.5 release. >If you are concerned about keeping NetBackup and ONTAP in a supported mode you >need to be at 5.3.4 (qualified). (They are currently qualifying 5.3.4R2 per >email I received from VERITAS support. I don't know where this information is coming from but nothing beyond 5.3.4 is being qualified by VERITAS at this time. There are no plans to qualify any 5.3.4x or 5.3.5 release. Obviously we will be qualifying some future release at some point. Klaus Micheler wrote: >I hope Netapp and Veritas will improve their quality with better >certifications in the near future >and tell us facts, not feelings. Please remember that the absence of qualification of a particular release is not the same as not supporting it. When VERITAS identifies an OnTap release as qualified, it means that we have done a level of testing within VERITAS to suggest to us that the release meets the minimum standards for use with NetBackup for NDMP. It does not mean that there are no issues. Releases that we have not qualified have not been tested by VERITAS. The lack of qualification does not automatically mean we don't support that release. It means that NetApp should be your first contact if you run into a problem that is new with that release. The simple truth is that it would be impossible for VERITAS to test every OnTap release. At some point we must assume that NetApp maintains full upward compatibility with subsequent releases and that reasonable regression tests have been performed. I would point out that the problems that have been seen have been identified by NetApp as an OnTap bug. This problem affects more than one backup application. The fix does not include a new version of NetBackup for NDMP or a NetBackup for NDMP patch. This is strictly an OnTap issue. If you have futher questions about this, please contact me directly. I'm the project lead and developer for NetBackup NDMP. __________________________________________________________________________ Steve Kappel steve.kappel(a)veritas.com VERITAS Software steve.kappel(a)iname.com (Personal)

2 1

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters