Simon,
I had a shelf hang on the R100 and R150 this week resulting in about 10 failed drives, so the reconstruction is taking forever - hence my interest in keeping an eye on the reconstruction progress. It seems like the systems can only do two reconstructs at a time, so I've had raid groups waiting for a drive reconstruction for about 24 hours so far... If you know of a way to increase the number of parrallel reconstructions I'd be interested to hear it.
1. You are correct... they only reconstruct 2 at a time. 2. You need to make sure that you have increased your timeout for reconstruct from 24 hours to something much longer in this case. (options raid.timeout XX) 3. We had the same thing happen on one of our R150s and were seeing potential for it on our other R150. The errors that were indicative of impending doom were of this nature...
messages.5:Wed Jul 28 19:39:05 CDT [adpch_timeoutd:warning]: adpch_timeoutd: 5a.0(0x0631a900, 0x2a:0696b11b:0090): command timeout, quiescing bus to allow outstanding I/O to complete. messages.5:Wed Jul 28 19:39:24 CDT [adpch_asyncd:warning]: adpch asyncd: Resetting SCSI 5a target 0 to clear outstanding I/O. messages.5:Wed Jul 28 19:39:28 CDT [adpch_intrd:warning]: adpch_timeoutd: 5a: unquiescing bus messages.5:Wed Jul 28 19:39:30 CDT [adpch_asyncd:warning]: adpch asyncd: Resetting SCSI 5a target 2 to clear outstanding I/O. messages.5:Wed Jul 28 19:39:35 CDT [adpch_asyncd:warning]: adpch asyncd: Resetting SCSI 5a target 3 to clear outstanding I/O. messages.5:Wed Jul 28 19:39:41 CDT [adpch_asyncd:warning]: adpch asyncd: Resetting SCSI 5a target 4 to clear outstanding I/O. messages.5:Wed Jul 28 19:39:46 CDT [adpch_asyncd:warning]: adpch asyncd: Resetting SCSI 5a target 8 to clear outstanding I/O. messages.5:Wed Jul 28 19:39:52 CDT [adpch_asyncd:warning]: adpch asyncd: Resetting SCSI 5a target 9 to clear outstanding I/O. messages.5:Wed Jul 28 19:39:57 CDT [adpch_asyncd:warning]: adpch asyncd: Resetting SCSI 5a target 10 to clear outstanding I/O. messages.5:Wed Jul 28 19:40:03 CDT [adpch_asyncd:warning]: adpch asyncd: Resetting SCSI 5a target 11 to clear outstanding I/O. messages.5:Wed Jul 28 19:40:08 CDT [adpch_asyncd:warning]: adpch asyncd: Resetting SCSI 5a target 12 to clear outstanding I/O. messages.5:Wed Jul 28 19:40:14 CDT [adpch_asyncd:warning]: adpch asyncd: Resetting SCSI 5a target 14 to clear outstanding I/O. messages.5:Wed Jul 28 19:40:56 CDT [adpch_timeoutd:warning]: adpch_timeoutd: 5a.0(0x06279600, 0x2a:0696d524:0090): command timeout, quiescing bus to allow outstanding I/O to complete. messages.5:Wed Jul 28 19:41:09 CDT [adpch_asyncd:warning]: adpch asyncd: Resetting SCSI 5a target 0 to clear outstanding I/O. messages.5:Wed Jul 28 19:41:14 CDT [adpch_asyncd:warning]: adpch asyncd: Resetting SCSI bus 5a to clear outstanding I/O. messages.5:Wed Jul 28 19:41:16 CDT [adpch_intrd:warning]: adpch_timeoutd: 5a: unquiescing bus messages.5:Wed Jul 28 19:41:16 CDT [adpch_intrd:warning]: adpch_osmevent[5a]: internal adapter failure detected, resetting adapter messages.5:Wed Jul 28 19:41:21 CDT [adpch_intrd:warning]: adpch_osmevent[5a]: internal adapter failure detected, resetting adapter messages.5:Wed Jul 28 19:41:46 CDT [adpch_timeoutd:warning]: adpch_timeoutd: 5a.0(0x06314300, 0x2a:0696f4c8:0090): command timeout, low outstanding I/O to complete. messages.5:Wed Jul 28 19:42:03 CDT [adpch_asyncd:warning]: adpch asyncd: Resetting SCSI 5a target 0 to clear outstanding I/O. messages.5:Wed Jul 28 19:42:08 CDT [adpch_asyncd:warning]: adpch asyncd: Resetting SCSI bus 5a to clear outstanding I/O. messages.5:Wed Jul 28 19:42:10 CDT [adpch_intrd:warning]: adpch_timeoutd: 5a: unquiescing bus messages.5:Wed Jul 28 19:42:10 CDT [adpch_intrd:warning]: adpch_osmevent[5a]: internal adapter failure detected, resetting adapter messages.5:Wed Jul 28 19:42:15 CDT [adpch_intrd:warning]: adpch_osmevent[5a]: internal adapter failure detected, resetting adapter
4. NA instructed us to upgrade from 6.4.4 to 6.4.5P2. We haven't seen any more adpch errors since. We also haven't lost a complete shelf since, THANK GOODNESS!
C-
Cheers, and it's good to have finally found a group of other netapp users.
Simon.
-----Original Message----- From: McCarthy, Tim [mailto:timothy.mccarthy@netapp.com] Subject: RE: New Simplified Monitoring Tool
Hey, you have a failed disk. Better call NGS ;)
-----Original Message----- From: Ben Rockwood [mailto:BRockwood@homestead-inc.com] Subject: New Simplified Monitoring Tool
I'd see if anyone else was interested in it or could propose suggestions for it.
This email message and any attachments are for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient or his/her representative, please contact the sender by reply email and destroy all copies of the original message.