toasters

toasters@lists.teaparty.net

13520 discussions

RE: raid failure
by Walters, Mike 03 May '00

03 May '00

Just another tool for your armoury against the unthinkable: SnapMirror. This gives you the ability to keep an asynchronous copy of your volumes on another filer with little overhead. I won't bore you with detail (which you may already know), but you might want to have a scan down http://www.netapp.com/tech_library/3066.html for data protection strategies. Cheers Mike > -----Original Message----- > From: owner-dl-toasters(a)netapp.com > [mailto:owner-dl-toasters@netapp.com]On Behalf Of Mark D Fowle > Sent: 03 May 2000 12:00 > To: toasters > Subject: raid failure > > > I have heard a few horror stories lately about netapps and > multi-disk raid > failures. Has anyone out there experienced this > and what did you do for recovery ? Where there any > warnings? I have not had > this happen and would like to do as much > as possible to prevent it. > > Thanks, > ============================================================ > =================== > ======= > Mark Fowle > Caterpillar/BCP > Cary North Carolina > ============================================================ > =================== > ======= >

1 0

RE: raid failure
by Mohler, Jeff 03 May '00

03 May '00

Absolutely, ONTAP will just stop in its tracks, and will be just fine when you resolve the underlying hardware problem. -----Original Message----- From: Mike Mueller [mailto:Mike.Mueller@jpl.nasa.gov] Sent: Wednesday, May 03, 2000 9:38 AM To: toasters(a)mathworks.com Subject: Re: raid failure What about the case when a whole shelf goes away at once (power is lost to the shelf for instance)? This seems more likely than a multiple disk failure. Is this recoverable? -- Mike

1 0

RE: raid failure
by Mohler, Jeff 03 May '00

03 May '00

I have. If the disk failure is simply a bad block on another drive, Netapp can sometimes product a patch that will "skip" that block and allow you to continue to use the filesystem and then after it's done rebuilding from the first failed disk, you can fail and rebuild the second failed disk. But this can be time consuming since you'll have to recheck your filesystem for possible corruption, and even after you fix that you have some data files somewhere that are corrupted that you don't know about. --- There are actual 'secret ninja' commands & tricks to do this with the OS a customer is currently running, that can be used to get around a severely abused drive or two in the case of an emergency rebuild. I have had to use this with customers who more often than not..have just relocated thier toaster from one rack/building to another, which says a LOT about proper drive treatment..Ive seen people really get slam-happy on replacing drives in the shelves as well..seeding thier own future drive failure issues. Bruce describes the situation rather nicely. BUT..those secret commands and options..are things you will get from Tech Support should you have to need them (truly a rare experience for TS to have to use them) and you wont get em from me. *grin* PS: Renew your support, is your data worth it? http://now.netapp.com

1 0

Re: raid failure
by Bruce Sterling Woodcock 03 May '00

03 May '00

----- Original Message ----- From: "Mark D Fowle" <Fowle_Mark_D(a)CAT.com> To: "toasters" <toasters(a)mathworks.com> Sent: Wednesday, May 03, 2000 4:00 AM Subject: raid failure > I have heard a few horror stories lately about netapps and > multi-disk raid failures. I have heard a lot more horror stories regarding single-disk failures in non-raid situations. That's the whole point of getting raid... to move that pain from the chance of a single disk failure to a double disk failure, thus reducing it substantially. > Has anyone out there experienced this > and what did you do for recovery ? I have. If the disk failure is simply a bad block on another drive, Netapp can sometimes product a patch that will "skip" that block and allow you to continue to use the filesystem and then after it's done rebuilding from the first failed disk, you can fail and rebuild the second failed disk. But this can be time consuming since you'll have to recheck your filesystem for possible corruption, and even after you fix that you have some data files somewhere that are corrupted that you don't know about. Generally, in the double-disk failure case, like every other RAID vendor (unless you're running like a +1 configuration), you lose whatever is in that raid group/filesystem and you have to restore it from a mirrored copy or tape backup. > Where there any warnings? Back before RAID scrubbing, sometimes there were no warnings, because when 1 drive failed and the system went through every block on the other disks rebuilding the filesystem, it would hit a block that had gone bad some time before and yet had not been accessed and boom, you had a double disk failure. Now that you have RAID scrubbing, all those blocks are checked every week or so, and you will get any indications of a possibly bad drive. Also the filer does provide warnings in advance if it's having trouble talking to a particular drive and you can fail it proactively and replace it without waiting for it to fail (and increasing the risk that another drive will fail during that time). > I have not had this happen and would like to do as much > as possible to prevent it. There's not much to be done to prevent it... disks have a MTBF and the chance of two failures in the same time period are non- zero. The best thing to do is follow Netapp's directions regarding the operating environment and rack-mounting of your drives (so you don't have excessive vibration or temperatures or whatnot), watch your logs and fail drives if they appear to be going bad and replace them ASAP. Set your raid reconstruct speed high or disable access to that filesystem during reconstruction to make sure the reconstruction happens as quickly as possible to shorten the time window for a second disk failure. Make sure your RAID scrubs run periodically and check their results. Use smaller RAID group sizes and have frequent backups so that when a failure does occur it is as limited and painless as possible. And only use disk drives from Network Appliance. Bruce

1 0

RE: Network Appliance Upgrade to ONTAP 5.3.4R3P2 and Backups
by Steve Kappel 03 May '00

03 May '00

Kelly Wyatt wrote: >Keep in mind that VERITAS currently has no plans to qualify any 5.3.5 release. >If you are concerned about keeping NetBackup and ONTAP in a supported mode you >need to be at 5.3.4 (qualified). (They are currently qualifying 5.3.4R2 per >email I received from VERITAS support. I don't know where this information is coming from but nothing beyond 5.3.4 is being qualified by VERITAS at this time. There are no plans to qualify any 5.3.4x or 5.3.5 release. Obviously we will be qualifying some future release at some point. Klaus Micheler wrote: >I hope Netapp and Veritas will improve their quality with better >certifications in the near future >and tell us facts, not feelings. Please remember that the absence of qualification of a particular release is not the same as not supporting it. When VERITAS identifies an OnTap release as qualified, it means that we have done a level of testing within VERITAS to suggest to us that the release meets the minimum standards for use with NetBackup for NDMP. It does not mean that there are no issues. Releases that we have not qualified have not been tested by VERITAS. The lack of qualification does not automatically mean we don't support that release. It means that NetApp should be your first contact if you run into a problem that is new with that release. The simple truth is that it would be impossible for VERITAS to test every OnTap release. At some point we must assume that NetApp maintains full upward compatibility with subsequent releases and that reasonable regression tests have been performed. I would point out that the problems that have been seen have been identified by NetApp as an OnTap bug. This problem affects more than one backup application. The fix does not include a new version of NetBackup for NDMP or a NetBackup for NDMP patch. This is strictly an OnTap issue. If you have futher questions about this, please contact me directly. I'm the project lead and developer for NetBackup NDMP. __________________________________________________________________________ Steve Kappel steve.kappel(a)veritas.com VERITAS Software steve.kappel(a)iname.com (Personal)

2 1

raid failure
by Mark D Fowle 03 May '00

03 May '00

I have heard a few horror stories lately about netapps and multi-disk raid failures. Has anyone out there experienced this and what did you do for recovery ? Where there any warnings? I have not had this happen and would like to do as much as possible to prevent it. Thanks, =============================================================================== ======= Mark Fowle Caterpillar/BCP Cary North Carolina =============================================================================== =======

1 0

Article on RAIT backup strategies
by Derek Kelly 02 May '00

02 May '00

Here's a link to an interesting article on RAIT (Redundant Array of Independent Tapes) strategies... http://www2.itworld.com/cma/ett_article_frame/0,2848,1_495,00.html ...Well, at least *I* thought it was interesting. =) Regards, Derek Kelly GTC

1 0

RE: Network Appliance Upgrade to ONTAP 5.3.4R3P2 and Backups
by Kelly Wyatt 02 May '00

02 May '00

I understand *now* that this has been acknowledged as an ONTAP bug. That was not the case (or VERITAS support did not know that) when I opened my case two weeks ago. I have email from my case owner that 5.3.4R2 is currently being tested. That is where that came from. My point in going to an unqualified release as a concern is that I have been stuck in the middle in the past. If I proceed to a version of ONTAP and backups break, then downgrading to fix backups is not necessarily a pretty thing to do. I guess my biggest concern is... if backups can break, so can restores... I have been backing up filers with NetBackup and NDMP for almost two years. I have seen a lot of the good, the bad and the ugly :-). I tend to be cautious as a result. Perhaps I should not have said "in a supported mode" but rather, in a "known working configuration". In general, backups are still working. We run a script every three hours that ups the drives, and any backups suspended by a downed drive pick up and proceed. And I agree... testing every version of ONTAP would be impossible. Kelly -- Kelly Wyatt, Kelly.Wyatt(a)SAS.com Systems Programmer Integrated Solutions Consulting SAS Institute Inc. / SAS Campus Drive / Cary, NC 27513 http://www.sas.com The power to know TM -----Original Message----- From: Steve Kappel [mailto:steve.kappel@raistlin.min.ov.com] Sent: Tuesday, May 02, 2000 2:46 PM To: Kelly.Wyatt(a)sas.com; Klaus.Micheler(a)infineon.com Cc: toasters(a)mathworks.com Subject: RE: Network Appliance Upgrade to ONTAP 5.3.4R3P2 and Backups Kelly Wyatt wrote: >Keep in mind that VERITAS currently has no plans to qualify any 5.3.5 release. >If you are concerned about keeping NetBackup and ONTAP in a supported mode you >need to be at 5.3.4 (qualified). (They are currently qualifying 5.3.4R2 per >email I received from VERITAS support. I don't know where this information is coming from but nothing beyond 5.3.4 is being qualified by VERITAS at this time. There are no plans to qualify any 5.3.4x or 5.3.5 release. Obviously we will be qualifying some future release at some point. Klaus Micheler wrote: >I hope Netapp and Veritas will improve their quality with better >certifications in the near future >and tell us facts, not feelings. Please remember that the absence of qualification of a particular release is not the same as not supporting it. When VERITAS identifies an OnTap release as qualified, it means that we have done a level of testing within VERITAS to suggest to us that the release meets the minimum standards for use with NetBackup for NDMP. It does not mean that there are no issues. Releases that we have not qualified have not been tested by VERITAS. The lack of qualification does not automatically mean we don't support that release. It means that NetApp should be your first contact if you run into a problem that is new with that release. The simple truth is that it would be impossible for VERITAS to test every OnTap release. At some point we must assume that NetApp maintains full upward compatibility with subsequent releases and that reasonable regression tests have been performed. I would point out that the problems that have been seen have been identified by NetApp as an OnTap bug. This problem affects more than one backup application. The fix does not include a new version of NetBackup for NDMP or a NetBackup for NDMP patch. This is strictly an OnTap issue. If you have futher questions about this, please contact me directly. I'm the project lead and developer for NetBackup NDMP. __________________________________________________________________________ Steve Kappel steve.kappel(a)veritas.com VERITAS Software steve.kappel(a)iname.com (Personal)

1 0

RE: Network Appliance Upgrade to ONTAP 5.3.4R3P2 and Backups
by Kelly Wyatt 02 May '00

02 May '00

We have drives down several times a day. My updrives script runs every three hours. Starting last night, here is what the script did: 6PM Up-ping drive 0,1,8,9 9PM Up-ping drive 0,1,8,9 12AM Up-ping drive 0,1,2,6,9 3AM Up-ping drive 0,1,6,9 6AM Up-ping drive 0,1,6,8,9 All we have to do is up the drive & backups proceed properly. It seems like the downing of the drive is triggered by and end of job or need for a second tape. I also get the (ndmp_connect_open_and_auth), ndmp_result = -5, DOWN'ing it error in the messages file. -- Kelly Wyatt, Kelly.Wyatt(a)SAS.com Systems Programmer Integrated Solutions Consulting SAS Institute Inc. / SAS Campus Drive / Cary, NC 27513 http://www.sas.com The power to know TM

1 0

RE: Network Appliance Upgrade to ONTAP 5.3.4R3P2 and Backups
by Kelly Wyatt 02 May '00

02 May '00

Keep in mind that VERITAS currently has no plans to qualify any 5.3.5 release. If you are concerned about keeping NetBackup and ONTAP in a supported mode you need to be at 5.3.4 (qualified). (They are currently qualifying 5.3.4R2 per email I received from VERITAS support. Kelly -- Kelly Wyatt, Kelly.Wyatt(a)SAS.com Systems Programmer Integrated Solutions Consulting SAS Institute Inc. / SAS Campus Drive / Cary, NC 27513 http://www.sas.com The power to know TM -----Original Message----- From: Micheler Klaus (MDCA Villach) [mailto:Klaus.Micheler@infineon.com] Sent: Tuesday, May 02, 2000 2:10 AM To: 'toasters(a)mathworks.com'; 'Kelly.Wyatt(a)sas.com'; 'armijo(a)cs.unm.edu' Subject: RE: Network Appliance Upgrade to ONTAP 5.3.4R3P2 and Backups On Thu, Apr 20, 2000 at 11:37:35AM -0400, Kelly Wyatt wrote: >> We upgraded out filers to NetApp Release 5.3.4R3P2 this weekend. Since then, NBU 3.2 keeps downing the tape drives on several, but not all of the filers (3 of 9). This seems to happen before, during and after backups. >> >> Has anyone else seen this? Thanks!! >we saw this on previous versions of NBU, 3.1.*, the work around was really >lame. cron executed a script which did something like: vmoprcmd -up # for >each drive. Hi there! We also have the same problem with this release of Ontap and Netbackup v3.2. According to Netapp the cause of this problem is a faulty Java Class (Garbage collection) which NDMP uses. This should also be the reason for the problems with Ontap 5.3.5xxx, where the problem got critical and backup stopped working. Netapp recommended us to switch to Ontap 5.3.5R2P1, where the NDMP-BUG is (hopefully) fixed. regards, Klaus Micheler Infineon Technologies DC

2 1

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters