######## scsi.cmd.checkCondition:error on .87,89
Fri Feb 4 11:29:54 GMT [scsi.cmd.checkCondition:error]: Device 8a.87: -- quiescing drive to allow outstanding I/O to complete.
######## FIRST MENTION OF FAILED DISK .87 ( DISK NOW LIVES IN SPARES POOL )
Fri Feb 4 11:30:20 GMT [raid.config.filesystem.disk.failed:error]: Disk -- [3HYY4K82000073408FS8]
####### scsi error continue on disk .89 Fri Feb 4 11:30:35 GMT [scsi.cmd.checkCondition:error]: Device 8a.89: Check Condition: CDB 0x2a:09cbc800:0080: Sense Data SCSI:aborted command - Fibre Channel frame CRC error (0xb - 0x47 0x0 0x3)(40387).
####### SUCCESSFUL RETRY ON DISK .87 Fri Feb 4 11:30:35 GMT [scsi.cmd.retrySuccess:info]: Device 8a.87: request successful after retry #3: cdb 0x2a:09cbc600:0080.
##### FIRST MENTION OF FC CRC FAME ERROR Fri Feb 4 11:30:36 GMT [ispfc_main:error]: Disk 8a.89: op 0x2a:09cbc800:0080 sector 164349952 SCSI:aborted command - Fibre Channel -- - Fibre Channel frame CRC error (0xb - 0x47 0x0 0x3)(41776)
###### FILER GIVES UP ON DISK .89. Fri Feb 4 11:30:36 GMT [scsi.cmd.noMorePaths:error]: Device 8a.89: No more paths to device: cdb 0x2a:09cbc800:0080. All retries have failed.
###### FIRST MENTION OF DISK 82 Fri Feb 4 11:30:55 GMT [scsi.cmd.checkCondition:error]: Device 8a.82: Check Condition: CDB 0x2a:09ac4b80:0080: Sense Data SCSI:aborted command -- - Fibre Channel frame CRC error (0xb - 0x47 0x0 0x3)(18075).
###### FIRST MENTION OF DISK 84 Fri Feb 4 11:30:58 GMT [scsi.cmd.checkCondition:error]: Device 8a.84: Check Condition: CDB 0x2a:09ac4c80:0080: Sense Data SCSI:aborted command -- - Fibre Channel frame CRC error (0xb - 0x47 0x0 0x3)(22992).
####### FIRST MENTION OF DISK 91 AND 90 (FC TIMEOUT) Fri Feb 4 11:31:00 GMT [ispfc_timeout_0:warning]: 8a.91 (0x0000005b) (0x034f4e10,0x2a:09cbcb80:0080,0/0,474/0/0,5192/0): command timeout, -- quiescing drive to allow outstanding I/O to complete.
######## GOING DOWN THE LIST NOW WITH 88 85 84 82 Fri Feb 4 11:31:00 GMT [ispfc_timeout_0:warning]: 8a.88 (0x00000058) (0x034f6f00,0x2a:09cbcb80:0080,0/0,387/0/0,5192/0): command timeout, -- request successful after retry #3: cdb 0x2a:09ac4c80:0080.
###### FIRST MENTION OF PASTTIMETOLIVE:ERROR Fri Feb 4 11:32:36 GMT [scsi.cmd.pastTimeToLive:error]: Device 8a.80: request failed after try #1: cdb 0x1c. -- request failed after try #1: cdb 0x2a:09cbcc00:0080.
###### FIRST MENTION OF WRITEERROR, DISKS 91 90 89 88 82 ( WILL STOP USING THEM!!??) Fri Feb 4 11:32:45 GMT [raid.label.io.writeError:error]: Label write on 8a.91 (S/N 3HYY4H2800007341NB79) failed with storage error 21. --
######### THE DREADED MULTI-DISK ERROR PANIC: raid volfsm: vol boot: fatal multi-disk error. in process config_thread on release NetApp Release 6.4.5 on Fri Feb 4 11:32:46 GMT -- DUMPCORE: START
#### GLOBAL TIMEOUT ( DISKS 92 86 85 84 83 81 80 ) .......................Fri Feb 4 11:32:46 GMT [config_thread:warning]: 8a.93 (0x0000005d) (0x03468f44,0x2a:109d43e8:0420,0/0,426/0/0,6197/0): --
###############3 ##REBOOT #1 (SWAPPED MULTI-MODE CABLE BETWEEN LRC AND FC CARD) ###############3
............................................................................................................................................ -- instance on primary side
#### DISK 87, PREVIOUSLY FAILED, IS NOW ORPHANED
Fri Feb 4 11:48:55 GMT [raid.assim.cls.notInCls:error]: Orphaning disk -- Loading volume boot
####### WAFL REPLAY FAILES AND PANICS THE FILER Replaying WAFL log wafl_verify_nvlog: replay failed twice in a row in log 0 request 8543 -- ............................................................................................................................................
###### REBOOT - SWITCHED CABLE TO PORT B ON FC CARD
ispfc: Fibre Channel adapter 8a appears to be unattached/disconnected. -- ore failure:
## MENTIONS DISK 80 ON A CHANNEL, MUST HAVE BEEN SOMETHING FROM LAST BOOT Error writing buffer to block 34873601 on disk 8a.80. Fri Feb 4 11:54:19 GMT [mgr.stack.saved:notice]: Reboot with saved panic information in log file
#### MENTION 2 OF MULT-DISK ERROR ( PROBABLY FROM LAST BOOT ) Fri Feb 4 11:54:19 GMT [mgr.stack.string:notice]: Panic string: raid volfsm: vol boot: fatal multi-disk error. in process config_thread on -- config_thread
##### GOING TO REBUILD ON DISK 80 Fri Feb 4 11:54:20 GMT [raid.rg.recons.info:notice]: One disk is missing from RAID group /boot/plex0/rg0 -- prototype is 8b.93
###### WEIRD STACK FRAME CRAP MENTIONS MULTIDISK ERROR Fri Feb 4 11:54:20 GMT [mgr.stack.framename:notice]: Stack frame 0: vol_multidisk_error(0xa2247c) + 0x5d -- is performed.
##### PASTTIMETOLIVE ERRORS RETURN ON DISK 80, 82,84, Fri Feb 4 12:13:59 GMT [scsi.cmd.pastTimeToLive:error]: Device 8b.80: request failed after try #1: cdb 0x1c. -- Check Condition: CDB 0x2a:09b24e98:0080: Sense Data SCSI:aborted command
##### MORE FC FRAME CRC ERRORS ( DISK 82 ) - Fibre Channel frame CRC error (0xb - 0x47 0x0 0x3)(57649). Fri Feb 4 12:14:23 GMT [scsi.cmd.checkCondition:error]: Device 8b.82: -- 10.3.38.28 was slow, op was v3 read, 63 > 60 (in seconds)
####### SCSI ERROR DISK 84 Fri Feb 4 12:14:48 GMT [scsi.cmd.checkCondition:error]: Device 8b.84: Check Condition: CDB 0x2a:09b25418:0080: Sense Data SCSI:aborted command -- - Fibre Channel frame CRC error (0xb - 0x47 0x0 0x3)(20148).
####### SCSI ERROR DISK 81 Fri Feb 4 12:14:53 GMT [ispfc_timeout_1:warning]: 8b.81 (0x01000051) (0x034f47b0,0x2a:09b25298:0080,0/0,20150/0/0,8028/0): command timeout, quiescing drive to allow outstanding I/O to complete.
###### SCSI DISK 80 Fri Feb 4 12:15:33 GMT [scsi.cmd.pastTimeToLive:error]: Device 8b.80: request failed after try #1: cdb 0x1c. -- Volume State Status Options
#### PAST TOME TO LIVE ON DISKS 80 84 82 Fri Feb 4 12:16:19 GMT [scsi.cmd.pastTimeToLive:error]: Device 8b.80: request failed after try #1: cdb 0x1c. -- request failed after try #1: cdb 0x2a:09b25298:0080.
#### FC TIMOUT ON DISKS 92 85 84 83 82 81 Fri Feb 4 12:16:49 GMT [ispfc_timeout_1:warning]: 8b.92 (0x0100005c) (0x034f4d00,0x2a:09b25698:0080,0/0,19912/0/0,8729/0): command timeout, --
#### FIRST MENTION OF SCSI ABORTED BY HOST DISK 92 Command aborted by host adapter: HA status 0x4: cdb 0x2a:09b25698:0080. Fri Feb 4 12:16:52 GMT [scsi.cmd.abortedByHost:error]: Device 8b.92: -- request failed after try #1: cdb 0x2a:09b25518:0080.
##### DEVICE RECOVERY DISK 84 Fri Feb 4 12:16:52 GMT [ispfc_timeout_1:error]: 8b.84 (0x01000054): global device timer timeout, initiating device recovery.
##### SCSI ABBORTED BY HOST DISK 85 Fri Feb 4 12:16:52 GMT [scsi.cmd.abortedByHost:error]: Device 8b.85: Command aborted by host adapter: HA status 0x4: cdb 0x2f:0025a800:0400. -- F
#### MULTI-DISK ERROR #3 PANIC: raid volfsm: vol boot: fatal multi-disk error. in process config_thread on release NetApp Release 6.4.5 on Fri Feb 4 12:17:25 GMT -- 2032 to 2048MB
##### BOOT IN READONLY
Intel Open Firmware by FirmWorks -- EMS NVRAM cache
### DISK FAILED ERROR DISK 89 (SECOND FAILED DISK) Fri Feb 4 12:17:25 GMT [raid.config.filesystem.disk.failed:error]: Disk /boot/plex0/rg0/8b.89 Shelf 5 Bay 9 [NETAPP X236_SCHT6146F10 NA05] S/N -- Autosupport, please see: http://now.netapp.com/autosupport/Fri
#### DISK RECONSTRUCTION FAILED (DUH, READONLY FILESYSTEM) Feb 4 12:37:23 GMT [asup.throttle.msgHistoryErr:error]: Problems tracking message for throttling: DISK RECONSTRUCTION FAILED