Does anyone else find the wording of syslog messages confusing? I almost had a small heart attack when I saw this after failing out a marginal drive, thinking that two drives had just died. "Disk 8" and "Data disk 7" are the same thing, and I wish the filer would use one or the either.
Mon Aug 4 15:45:27 EDT [raid_disk_admin]: Unload of disk 8 has completed successfully. Mon Aug 4 15:45:27 EDT [raid_stripe_owner]: Read on data disk 7 failed, reverting to degraded mode.
Brian Tao wrote:
Does anyone else find the wording of syslog messages confusing? Ialmost had a small heart attack when I saw this after failing out a marginal drive, thinking that two drives had just died. "Disk 8" and "Data disk 7" are the same thing, and I wish the filer would use one or the either.
Mon Aug 4 15:45:27 EDT [raid_disk_admin]: Unload of disk 8 has completed successfully. Mon Aug 4 15:45:27 EDT [raid_stripe_owner]: Read on data disk 7 failed, reverting to degraded mode.
Admittedly, this is confusing. I hope your heart is feeling better now.
Just one clarification. The netapp will not "fail" two drives. If it has a "read failure" on a drive while it is in degraded mode, and all retrys fail, it will reboot to try and "jar" the offending SCSI device back to life, but will not actually "fail" it in the RAID sense.
In a worst case scenario, i.e., you have a double disk failure, a system in degraded mode cannot complete reconstruction because of all the noise being made by the second "failing" drive causing reboots before reconstruction is complete.
This scenario is actually useful for rescuing data because of the statelessness of NFS. Data can be accessed from clients with the occasional NFS Server not responding due to the reboots.
That is basically what a "double disk failure" is. It does not mean that suddenly you cannot access any of your data. You can choose to use "rdist" or some other file copy over NFS to copy the data, or decide that your backups are reliable, and restore on a new filesystem. It might even be able to wait until the weekend with reboots every hour or so.
So basically, give your heart a rest, and don't panic if a message appears. This is a very very *VERY* rare occurence.
Ken.
In a worst case scenario, i.e., you have a double disk failure, a system in degraded mode cannot complete reconstruction because of all the noise being made by the second "failing" drive causing reboots before reconstruction is complete.
It cannot complete reconstruction because, in an N-disk system, if one disk fails, you need the information on the remaining N-1 disks to reconstruct what was on the now-dead disk - and if all N-1 of them aren't accessible, you lose.
That is basically what a "double disk failure" is. It does not mean that suddenly you cannot access any of your data.
Err, umm, if more than one of the disks in your RAID array have "failed" to the extent that more than one of them is inaccessible to the filer, you cannot, in fact, access any of your data any more, except for copies on your backup tapes. Maybe the reboot will make enough of the disks accessible that the problem is transient, but that's not guaranteed.
(E.g., to take an extreme example, ripping two drives out of your filer and pounding them to scrap metal with a sledgehammer will certainly render data on them inaccessible. Hard read or write errors on two drives will have the same effect, if a reboot doesn't make the errors go away.)
Guy Harris wrote:
Err, umm, if more than one of the disks in your RAID array have "failed" to the extent that more than one of them is inaccessible to the filer, you cannot, in fact, access any of your data any more, except for copies on your backup tapes. Maybe the reboot will make enough of the disks accessible that the problem is transient, but that's not guaranteed.
(E.g., to take an extreme example, ripping two drives out of your filer and pounding them to scrap metal with a sledgehammer will certainly render data on them inaccessible. Hard read or write errors on two drives will have the same effect, if a reboot doesn't make the errors go away.)
Okay, Guy is correct as always. However, I have been in Escalations for a year now, and I have seen the nastiest situations during that time, and I have not yet seen one like what Guy describes above. I guess most people don't keep a sledgehammer in the office :)
Incidentally, CEO's are almost as bad as a sledgehammer. I had a call where a proud CEO walked into the "sysadmin territory" of his company and tried to show his business buddies how he could rip drives our of his nifty little filer and it would stay up. It was hilarious.
At 01:19 PM 8/5/97 -0700, Kenneth Whittaker wrote:
Incidentally, CEO's are almost as bad as a sledgehammer. I had a call where a proud CEO walked into the "sysadmin territory" of his company and tried to show his business buddies how he could rip drives our of his nifty little filer and it would stay up. It was hilarious.
Heh. I had something like that, only it wasn't even a CEO. 'twas a new sysadmin. I had been telling him about this really neat concept of RAID and HOT SPARES (he came from an academic environment, where saving money was more important than redundancy, so he'd never gotten to play with raid before).
He was very impressed, and when we hired another junior sysadmin, he started to tell the junior guy all about RAID and HOT SPARES. He'd forgotten (or missed) the step where you have to tell a software raid system when you're gonna hot swap something. Oops.
Amy
The wording does seem a bit confusing, and more so that there are 3 ways to refer to a disk, by RAID number, by DISK ID number and by SCSI number. The RAID disk is useful to determine if its a parity disk, a data disk, or a spare, but otherwise doesn't seem to tell a user much. The DISK ID is used for the raid command, but you still need to know the scsi address to actually remove/replace a disk. It seems that it could be reduced to just referring to/listing the SCSI addresses and whether its a data/parity/spare disk, then raid commands would be like 'raid fail 8a.7', and the messages would be 'unload of disk 8a.6 completed' and 'read on data disk 8a.6 failed'. Is there an obvious reason this isn't done?
RAID Disk DISK_ID# HA.SCSI# --------- -------- -------- parity 1 8a.1 data 1 2 8a.2
Brian Tao wrote:
Does anyone else find the wording of syslog messages confusing? Ialmost had a small heart attack when I saw this after failing out a marginal drive, thinking that two drives had just died. "Disk 8" and "Data disk 7" are the same thing, and I wish the filer would use one or the either.
Mon Aug 4 15:45:27 EDT [raid_disk_admin]: Unload of disk 8 has completed successfully. Mon Aug 4 15:45:27 EDT [raid_stripe_owner]: Read on data disk 7 failed, reverting to degraded mode.
-- Brian Tao (BT300, taob@netcom.ca) "Though this be madness, yet there is method in't"
Is there an obvious reason this isn't done?
Yeah - the filer's data structures don't keep information around in such a fashion as to allow us to refer to a disk by any name other than disk ID number by the time it's been unloaded, but disk ID numbers are kind of a gross way of referring to disks (they have no correlation with anything other than the order in which the filer saw the disks, and can change over a reboot).
We plan to fix this in a future release. The plan is to use the HA.SCSI# ("HA" means "host adapter", "8a" being the name of the first SCSI adapter in card slot 8) names universally.
-
Amy Chused -
Brian Tao -
Dave Heiland -
guy＠netapp.com -
Kenneth Whittaker