toasters

toasters@lists.teaparty.net

3 participants
13514 discussions

Re: raid failure
by G D Geen 06 May '00

06 May '00

I have lived through this harrowing experience. Yes there were warning signs but as I was busy performing other tasks I did not see them. The moral of my story is to read your weekly message logs and have good back ups. I also file every weekly autosupport message. I have message logs from everyone of my filers, we currently have twenty-one, dating back to 1996 when I started this position. It began with a phone call in the middle of the night. On the other end was a co-worker confused as to why the filer was no longer serving data. She had made the determination that there was a failed disk, but little did she know what the next fifteen hours would bring. The reported disk failure was being reconstructed but would never complete. That is when I got the phone call. By this time it was just after 10:00 PM and I rushed into work. I immediately got the run down on the problem she was seeing and what actions were taken. From that point I made a few stabs at getting the RAID to reconstruct. My attempts also failed, now we had to call NetApp support. I made the priority one call and was soon contacted by a NetApp engineer. After a couple of hours poking around on the filer, we were able to determine the sequence of events leading up to the nights exercises in futility. Approximately two weeks prior, the filer had reported in the messages logs that a disk in RAID group zero on volume zero, the root volume, had an unrecoverable read error but did not fail the disk. This night, now closer to morning, the system failed a different disk in RAID group zero on volume zero. See the connection? The net result was the filer could not reconstruct the data because of the unrecoverable read error on the earlier disk. The filer could not make any sense of the data coming off the corrupted sector of the disk. By this time our local support people were awaken by the central support team and had joined us. We were starting to draw a crowd. It was closing in on 2:00 AM when we tried to reconstruct the disk with the bad read sector manually but after a couple of failed attempts even the NetApp engineer was beginning to sound depressed. Then came the question. The one that all data managers hate to hear. How good are your backups? To this question I replied, "we are about to find out." Since our root volume failed, the first step was to make a new root volume. I made /vol/vol1 our new root volume. Next we destroyed /vol/vol0 and removed all the troubled disk drives. Since all of these disks are now spares, we may create a new RAID group and a new /vol/vol0. Here it is important to note that there must be a /etc on the volume *before* Veritas can begin its data restore. Once the volume was recreated, we began our restore of approximately 150 Gigabytes of data which took us well into the morning. The NetApp engineer in California and I had a long conversation over the phone while the data restores were beginning. He was certain that there should have been some sort of sign that this was about to happen. I assured him that I would look and would forward my results to our local support people, who are copied on the autosupport messages too. As I went back through the autosupport logs that are e-mailed to me each week, I found that the problem began approximately two weeks earlier. Every time a the disk tried to read a particular sector of the disk, an error messages would appear the messages log indicating such an event had occurred. Had I not been busily working other issues, to the detriment of my filers, I would have failed this disk at lease a week prior. -gdg Mark D Fowle wrote: > I have heard a few horror stories lately about netapps and multi-disk raid > failures. Has anyone out there experienced this > and what did you do for recovery ? Where there any warnings? I have not had > this happen and would like to do as much > as possible to prevent it. > > Thanks, > =============================================================================== > ======= > Mark Fowle > Caterpillar/BCP > Cary North Carolina > =============================================================================== > ======= -- --------------------------------------------------------------- G D Geen mailto:geen@ti.com Texas Instruments Phone : (214)480.7896 System Administrator FAX : (214)480.7676 --------------------------------------------------------------- Life is what happens while you're busy making other plans. -J. Lennon

7 7

Re: raid failure
by Richard L. Rhodes 06 May '00

06 May '00

On 3 May 2000, at 18:29, owner-toasters(a)mathworks.com wrote: > Unfortunately, this only works if we get the autosupport in the first > place. Apparently there are a number of customers who think we don't > want to be bothered. We do! If your filer or NetCache doesn't list > 'autosupport(a)netapp.com' in the 'autosupport.to' option, please add > it if your policies allow this. > I'm a new netapp customer (less than a year). I've found autosupport a little frustrating. For myself, the problem is the lack of feedback. I set up autosupport to send messages to several internal addresses and NetApp. I get the email just fine, but I have no way of knowing if NetApp got the email. I'd use the "options autosupport.doit just testing" command to send a test. I'd get the email, but I had no way to know if NetApp got the email. I almost called support to ask if anyone could tell me whether they got the emails! I'd like to see several things: 1) an auto reply message sent back from NetApp acknowledging that they got the autosupport email. 2) If you run it through some tool for analysis, I'd like to have the output emailed back to me. Here's another example. Since upgrading DataOntap, the autosupport email says that my gigabit ethernet cards are in the wrong slots. These cards are in the slots that the factory put them in. NetApp support has said nothing and the filers are working fine - so I've done nothing. Am I wrong? Rick

4 3

Raid groups
by Sateesh Mucharla 06 May '00

06 May '00

I am buying a F760 with 35 drives (all 36GB). As there is limitation of 28 drives per raid group ( i read it NetApp Sysadmin manual), i am thinking of having two raid groups one with 26 and the other with 7 drives. Rest two will be hot spares. But i am open for better configuration. I would like to know whether i can configure all 33 drives into one raid group - Is that possible? If so what are the disadvantages. (With 33 drives in one raid group, i loose only one drive as a parity drive - this is an advantage). Can somebody suggest a better configuration ? Thanks Sateesh Mucharla National Semiconductors

3 2

autosupport RE: Apology and Request (was Re: raid failure)
by Klinkner, Steve 05 May '00

05 May '00

> -----Original Message----- > From: dnb(a)ccs.neu.edu [mailto:dnb@ccs.neu.edu] > Dave Hitz <hitz(a)netapp.com> writes: > > > I also have a request. Please turn on autosupport. That's > > extremely important to us for two reasons: > > Quick question: once upon a time, machine-gone-kablooey autosupport > mail was only sent when a filer booted. It was the Yoda scenario > ("there is no retry, only do!"). If something went awry in that > sending, the message was just dropped on the floor, no queuing, no > retrying. > > We had a situation where our filer bounced twice, but on the first > return to service it didn't have good net (or NIS, or name resolution, > can't recall which). This meant that the first "here's the problem" > mail never got to the mothership, but the "just booted, everything is > peachy" mail did. Does the crash autosupport facility do anything more > sophisticated these days? (he asks, because his filer hasn't > crashed in > many moons...). ONTAP 5.3.4 got a change which has autosupport retry delivery for about 15 minutes. This should help with transient failures such as the mailhost rebooting, temporary loss of DNS, et cetera. In the "next major release" the retry interval has been boosted to an hour. In addition, autosupport will wait until boot is fully complete before trying to do *anything*. This will prevent autosupport from issuing silly syslog messages saying it can't deliver mail because the network isn't up (prior to 5.3.4, that behavior could cause loss of a message, in 5.3.4 it is annoying/silly, and in the "next major release" it will go away completely). ---------------------------------------------------------------------- steve klinkner klink(a)netapp.com

1 0

Re: Apology and Request (was Re: raid failure)
by dnb＠ccs.neu.edu 05 May '00

05 May '00

Cool, that's great to hear. Thanks for the response. Respectfully, David N. Blank-Edelman Director of Technology College of Computer Science Northeastern University John Denholm <johnd(a)ThePLAnet.net> writes: > Yeah, that's better now. We used to get that a lot - part of the > problem can be the interaction of Cisco switches and Netapps - the > Cisco can take a short while to figure out that the link is back up > to a Netapp and the mail falls on the floor while waiting for that. > > Some time ago, presumably at the insistence of ourselves and others, > Netapp inserted a 30 second wait and retry loop into bootup, and now > I think it gives autosupport 4 or 5 tries over a couple of minutes. > We always get autosupport off them now. Not that, I admit, we've > had a crash message in a while, but we never used to get them on > reboots either :< > > As for dealing with network or DNS or whatever not coming back, one thing you > can do is specify your mailhost wholely by IP - while fractionally more work to > maintain if the boxes on your network change frequently, it requires a little > less to go right. > > If you are worried about missing a crash, use snmp monitoring and grab > system.sysUptime.0, which is the uptime in hundredths of seconds. I run a > large number of netapps of different breeds, and I just run a perl script which > grabs the uptime off every one every 10 minutes. Any crashes are immediately > apparent. It could also write to file and compare current uptime to previous > uptime - if uptime drops, it can sms you, mail you, sound sires, flash lights, > whatever :> > > > Now I just wish they'd get it right with the caches. They still only try > mailing autosupport once :p > > J > > --------------------------------------------------------------------------- > # John Denholm johnd(a)theplanet.net # > # Webcache & Filer Administrator, Planet Online +44 113 207 6357 # > --------------------------------------------------------------------------- > > Error 404: There is no spoon

1 0

RE: Raid groups
by Mattel, Martin 05 May '00

05 May '00

Just want to add one word, as more spindles you have, as faster you can write. But if you have a graph for spindles / speed, you get a non-linear line. The delta is decreasing but is always positive. The delta is "feelable" for data-disk # up to 4-5, over that it is measurable but not significant. If you have 35 drives, I suggest: 1 HS 2 Parity 32 Data Drives That means, your raidsize will be 17 [2x (16+1)] Take care, that if you add another shelf, you need at least 2 drives in there (for a singe system) to build another raidgroup to your volume when you "vol add ´volume´ 1", because it is adding 2 (1 data, 1 parity). MM > -----Original Message----- > From: Mohler, Jeff > Sent: Donnerstag, 04. Mai 2000 23:55 > To: 'Sateesh Mucharla'; toasters(a)mathworks.com > Subject: RE: Raid groups > > > Question: > > Why would you have one large raid group? > > Your mathematical chances of losing the entire volume from a > 2-drive failure are > astonomically higher than if you use smaller RG's, it will > also take ages longer > to rebuild a failed drive in that large of a RG instead of a > small(er) one. > > Also consider that the small raid group will be a > 'bottleneck' or sorts. > > Its my understanding that every CP (consistency point) that > is written, > round-robins thru all available Raid Groups (RGs) in a > volume. As I see it, > that helps balance the data for when you want to read it back > from disk helping > to minimize head-seek times. X amount of data will take less > time and disk > revolutions to write out on the large RG, but -may- take multiple disk > revolutions to complete on the small RG..and extend to the > length of time it > would take to read the data as well. All things being equal, > equal sized RGs > would offer a more consistent performer. > > A balanced set of RGs will serve you better in the long run. > > -----Original Message----- > From: Sateesh Mucharla [mailto:sateesh@ampere.nsc.com] > Sent: Thursday, May 04, 2000 12:21 PM > To: toasters(a)mathworks.com > Subject: Raid groups > > > I am buying a F760 with 35 drives (all 36GB). > > As there is limitation of 28 drives per raid group ( i read > it NetApp Sysadmin > manual), i am thinking of having two raid groups one with 26 > and the other with > 7 drives. Rest two will be hot spares. But i am open for > better configuration. > > I would like to know whether i can configure all 33 drives > into one raid group - > > Is that possible? If so what are the disadvantages. (With 33 > drives in one raid > group, i loose only one drive as a parity drive - this is an > advantage). > > Can somebody suggest a better configuration ? > > Thanks > > Sateesh Mucharla > National Semiconductors >

1 0

Re: Apology and Request (was Re: raid failure)
by dnb＠ccs.neu.edu 05 May '00

05 May '00

Dave Hitz <hitz(a)netapp.com> writes: > I also have a request. Please turn on autosupport. That's > extremely important to us for two reasons: Quick question: once upon a time, machine-gone-kablooey autosupport mail was only sent when a filer booted. It was the Yoda scenario ("there is no retry, only do!"). If something went awry in that sending, the message was just dropped on the floor, no queuing, no retrying. We had a situation where our filer bounced twice, but on the first return to service it didn't have good net (or NIS, or name resolution, can't recall which). This meant that the first "here's the problem" mail never got to the mothership, but the "just booted, everything is peachy" mail did. Does the crash autosupport facility do anything more sophisticated these days? (he asks, because his filer hasn't crashed in many moons...). Thanks. Respectfully, David N. Blank-Edelman Director of Technology College of Computer Science Northeastern University

2 1

RE: 18GB drive performance versus 36BG
by Mohler, Jeff 05 May '00

05 May '00

What does your dataset look like? Homedirs, software development, chip development/simulation? And how many users..etc. There is also a new high-performance (spindle rate/seek rate) 9G drive now as well for those special environments. -----Original Message----- From: Mike(a)broaddaylight.com [mailto:Mike@broaddaylight.com] Sent: Thursday, May 04, 2000 4:36 PM To: toasters(a)mathworks.com Subject: 18GB drive performance versus 36BG I am considereing ordering a second F740 with 36BG drives. Big drives are nice, but there must be some tradeoffs: performance, restoration? If there is a significant performance difference I may go with the 18GB drives Michael Hopkins mike(a)BroadDaylight.com <mailto:mike@BroadDaylight.com> Systems Administrator phone 408.450.7450 Broad Daylight, Inc. fax 408.327.0177

2 1

18GB drive performance versus 36BG
by Mike＠broaddaylight.com 04 May '00

04 May '00

I am considereing ordering a second F740 with 36BG drives. Big drives are nice, but there must be some tradeoffs: performance, restoration? If there is a significant performance difference I may go with the 18GB drives Michael Hopkins mike(a)BroadDaylight.com <mailto:mike@BroadDaylight.com> Systems Administrator phone 408.450.7450 Broad Daylight, Inc. fax 408.327.0177

1 0

Seeking opinions on dump UI change
by Stephen Manley 04 May '00

04 May '00

Hey everybody. In the wake of Dave's email about UI changes, and the feelings of people on this list, I thought I'd pose a question to y'all. Currently, if you are running a dump on your console, and you hit end of tape or end of media, dump will pop up a message that says: toaster1> dump 0ufX rst0a *.o,*core* /vol/vol0/home DUMP: Dumping tape file 1 on rst0a ... DUMP: Change tape file on toaster1. DUMP: Mount tape file #2 DUMP: Is the new tape file mounted and ready to go?: ("yes" or "no") [If you respond "yes", it will write data to the tape device rst0a.] [If you respond "no", it will say:] DUMP: Do you want to abort?: ("yes" or "no") -> A problem that some people have raised is that there is no way to -> try to dump to rst1a or rst2a at this point They'd like to be able to -> write to a different device, but the question/answer framework will -> not currently allow you to do that. On the other hand, restore says: toaster2> restore rfD rst0a /vol/vol1/home_new ... RESTORE: Mount tape file 2 on toaster2 RESTORE: Enter ``none'' if there are no more tapes RESTORE: otherwise enter tape device name (default: rst0a) The benefit, of course, is that you can specify a different device if you desire. So here's the question - if we unified the interface for dump/restore on this point, would the added flexibility outweigh the pain of changing scripts? (This would come out in the next major release). So it would look like: toaster1> dump 0ufX rst0a *.o,*core* /vol/vol0/home DUMP: Dumping tape file 1 on rst0a ... DUMP: Mount tape file #2 on toaster1 DUMP: Enter ``none'' if there are no more tapes DUMP: otherwise enter tape device name (default: rst0a) (Note: it changed the "Mount tape file" message as well as the Q and A). Or, as an alternative, would people prefer something like: DUMP: Change tape file on toaster1. DUMP: Mount tape file #2 DUMP: Is the new tape file mounted and ready to go?: ("yes" or "no") no DUMP: Enter ``none'' if there are no more tapes DUMP: otherwise enter tape device name (default: rst0a) so that you could continue to respond "yes" and see no behavior change? But you would still get the advantage of being able to specify a new drive. The negative, of course, is that dump and restore ask the questions differently... So three questions: 1) Would you like to see: A) Unify dump and restore interface for asking questions B) The alternative I gave C) Something else entirely 2) If we did change this in the next major release, where would you like to see the "HEY! We changed this!" note? 3) Other suggestions for dump/restore UI changes you'd like? Thanks for reading this and thanks for your feedback, Stephen Manley Data Availability and Management Pollster

2 1

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters