New subject: Fibre Channel tape Connections

13 Aug 2007


      Hello,
   I can't stress the below bulletin enough.  Our site experienced a 12+ 
hour outage on a FAS3040's first day of production due to this bug.
On Sunday, I did a migration from 2 FAS940's and 1 F880 to a new FAS3040 
with 3 SATA shelves.  The first day of production on Monday was great, 
it took the load with ease and response was great.  Level 0 backups were 
due.  So at about 5pm on Monday, I started level 0's of the volumes on 
the new 3040.  At 7pm, it panic'ed and halted.  It said it lost 1 disk 
in the root aggr and was rebuilding, during which time it said it lost 2 
more.
I wasn't on the phone with L3 tech support more than 5 minutes before 
they said I was hit with this firmware bug.  The shelves came with V33, 
and I hadn't put them in production before the May 31st email alert 
about the FW upgrade to V34 came out.
So by around 11pm Monday we started the WAFL_check on the root aggr. 
And it finished about 5am.  But there were a lot of lost files and 
support did not want us to accept the changes by this WAFL_check and do 
it against a snapshot from before the crash.  We elected to accept the 
lost files and then search the snapshot and copy the lost files back to 
the live file system ourselves. But the file system was offline until 
about 11am, 3 hours into a work day for 2200 users -- not good.  And 
then the 3040 was so slow during that day doing a double reconstruction 
that many services were not responsive at all, our email spool being the 
most visible.
So,
If you have shelves with this firmware -- update it as soon as possible !
The firmware upgrade was the first thing I did the night of the crash, 
and it only took about 15 minutes to do 3 shelves with 2 modules each.
The good news is, we got a 4th shelf that was came later than the rest, 
and it came with V34 firmware.
Paul
9 August 2007 	Customer Support Bulletin 	CSB-0708-03
Critical Upgrade – AT-FCX FW to v34
*Summary**:*
This Customer Support Bulletin is meant to reinforce and amplify the
need to expeditiously upgrade to the AT-FCX v34 Firmware Release that is
described in CSB-0705-03 . Customers are required to schedule a 
maintenance window and upgrade to AT-FCX FW v34 as soon as possible.
*What is the potential service impact **if AT-FCX firmware is not
upgraded to v34**:*
Systems with AT-FCX I/O modules (X/SP-5612A-R5) not upgraded to firmware
v34 can experience the following problems (see link above for more details):
* WAFL® file system inconsistency
     * Write-data loss
     * Extended system outages
     * The need to run WAFL recovery utilities to restore data integrity
     * Unexpected reservation conflicts that in turn can cause additional
       service outages
*What are the failure symptoms if AT-FCX firmware is not upgraded to v34?*
Systems with AT-FCX I/O modules (X/SP-5612A-R5) not upgraded to firmware
v34 would see the following failure symptoms related to pre-v34 firmware:**
o Multiple RAID lost-write or checksum errors are seen on
             multiple drives attached to an AT-FCX shelf running shelf
             I/O module firmware version 33 or lower. These errors are
             caused by misdirected writes and/or reads to a disk. In some
             cases, not only is the location wrong, but the wrong disk is
             involved.
           o The AT-FCX I/O module can occasionally panic during shelf
             firmware downloading.  This will result in loss of disk
             connectivity and storage multi-disk panic.
*Additional Benefits*
In addition to protection against the above-mentioned problems, the
following fixes are included in firmware v34:
* Corrective action that recognizes the failure of the first I/O
       AFTER a cache flush has failed. This I/O is returned with a
       senseKey/asc/ascq of 9/88/01 error that causes Data ONTAP® to take
       the drive out of service. The data on the drive will then be
       reconstructed normally without data loss.
     * Corrects an issue in which the reservation database is not updated
       properly, resulting in unexpected reservation conflicts that in
       turn can cause service outages.
     * Avoids a known problem in which a controller failover can occur
       while downloading shelf firmware.
*Support*
Please contact NetApp Global Services and/or Professional Services
     for questions or assistance concerning the firmware upgrade of
     AT-FCX (X/SP-5612A-R5) to v34.**
*NOW Self-Service Options*
* To register online for NOW access, go to
       http://now.netapp.com/newuser/
* To unsubscribe from the Customer Product Announcement distribution
       list, deselect the Field Alerts option in the Subscriptions
https://now.netapp.com/eservice/personal/loadSubscription.do?moduleName=MYPROFILE
       area on NOW.
https://now.netapp.com/eservice/personal/loadSubscription.do?moduleName=MYPROFILE
* To review, order, and renew software subscriptions, click Manage
       Service Contracts on the Service and Support
       http://now.netapp.com/Self-Service/Forms/SupportHome.asp home
       page on NOW.
The purpose of this communication is for NetApp Global Services to
notify its installed base end users about urgent and important product
information that may affect product performance or reliability. The
information contained herein and the distribution list are Network
Appliance™ confidential materials that are subject to restrictions on
redistribution and that cannot be shared outside of this email
distribution list.
© 2007 Network Appliance, Inc. All rights reserved. Specifications
subject to change without notice. NetApp, the Network Appliance logo,
Data ONTAP, and WAFL are registered trademarks and Network Appliance and
NOW are trademarks of Network Appliance, Inc. in the U.S. and other
countries. All other brands or products are trademarks or registered
trademarks of their respective holders and should be treated as such.

[Fwd: Network Appliance Customer Support Bulletin: Critical Upgrade - AT-FCX FW to v34 (CSB-0708-03)]