I have lived through this harrowing experience. Yes there were warning signs but as I was busy performing other tasks I did not see them. The moral of my story is to read your weekly message logs and have good back ups. I also file every weekly autosupport message. I have message logs from everyone of my filers, we currently have twenty-one, dating back to 1996 when I started this position.

It began with a phone call in the middle of the night. On the other end was a co-worker confused as to why the filer was no longer serving data. She had made the determination that there was a failed disk, but little did she know what the next fifteen hours would bring. The reported disk failure was being reconstructed but would never complete. That is when I got the phone call. By this time it was just after 10:00 PM and I rushed into work. I immediately got the run down on the problem she was seeing and what actions were taken. From that point I made a few stabs at getting the RAID to reconstruct. My attempts also failed, now we had to call NetApp support. I made the priority one call and was soon contacted by a NetApp engineer. After a couple of hours poking around on the filer, we were able to determine the sequence of events leading up to the nights exercises in futility. Approximately two weeks prior, the filer had reported in the messages logs that a disk in RAID group zero on volume zero, the root volume, had an unrecoverable read error but did not fail the disk. This night, now closer to morning, the system failed a different disk in RAID group zero on volume zero. See the connection? The net result was the filer could not reconstruct the data because of the unrecoverable read error on the earlier disk. The filer could not make any sense of the data coming off the corrupted sector of the disk. By this time our local support people were awaken by the central support team and had joined us. We were starting to draw a crowd. It was closing in on 2:00 AM when we tried to reconstruct the disk with the bad read sector manually but after a couple of failed attempts even the NetApp engineer was beginning to sound depressed. Then came the question. The one that all data managers hate to hear. How good are your backups? To this question I replied, "we are about to find out."

Since our root volume failed, the first step was to make a new root volume. I made /vol/vol1 our new root volume. Next we destroyed /vol/vol0 and removed all the troubled disk drives. Since all of these disks are now spares, we may create a new RAID group and a new /vol/vol0. Here it is important to note that there must be a /etc on the volume *before* Veritas can begin its data restore. Once the volume was recreated, we began our restore of approximately 150 Gigabytes of data which took us well into the morning.

The NetApp engineer in California and I had a long conversation over the phone while the data restores were beginning. He was certain that there should have been some sort of sign that this was about to happen. I assured him that I would look and would forward my results to our local support people, who are copied on the autosupport messages too. As I went back through the autosupport logs that are e-mailed to me each week, I found that the problem began approximately two weeks earlier. Every time a the disk tried to read a particular sector of the disk, an error messages would appear the messages log indicating such an event had occurred. Had I not been busily working other issues, to the detriment of my filers, I would have failed this disk at lease a week prior.

-gdg

Mark D Fowle wrote:

I have heard a few horror stories lately about netapps and multi-disk raid failures. Has anyone out there experienced this and what did you do for recovery ? Where there any warnings? I have not had this happen and would like to do as much as possible to prevent it.

Thanks,

======= Mark Fowle Caterpillar/BCP Cary North Carolina =============================================================================== =======

-- --------------------------------------------------------------- G D Geen mailto:geen@ti.com Texas Instruments Phone : (214)480.7896 System Administrator FAX : (214)480.7676 --------------------------------------------------------------- Life is what happens while you're busy making other plans. -J. Lennon