Hey everybody,
I just had the pleasure to try to replace a faulty disk in a filer which is out-of-service with a "new" one from after market. Of course, this didn't work out as easy as it should, so maybe some could help me get this disk going.
Here are the facts: FAS2040 NetApp Release 8.1.4P8 7-Mode
The disk at 0d.01.13 was broken, so I pulled it a few days ago - now that the replacment disk has arrived, I tried to plug it in and this is what happens:
Thu Oct 22 14:30:10 CEST [riddler:scsi.cmd.notReadyCondition:notice]: Disk device 0d.01.13: Device returns not yet ready: CDB 0x1b: Sense Data SCSI:not ready - Drive spinning up (0x2 - 0x4 0x1 0x28)(2).
The interesting thing, I can see the disk by using `storage show disk`:
riddler> storage show disk DISK SHELF BAY SERIAL VENDOR MODEL REV --------------------- --------- ---------------- -------- ---------- ---- 0d.01.13 1 13 J838WSRL NETAPP X302_HJUPI NA04
But I cannot see it in disk show.
riddler> disk show 0d.01.13 DISK OWNER POOL SERIAL NUMBER HOME ------------ ------------- ----- ------------- -------------
Sysconfig -v tells me, that there is a newer firmware on the "new" disk:
01.13: NETAPP X302_HJUPI01TSSM NA04 847.5GB (1953525168 512B/sect) 01.14: NETAPP X302_WVULC01TSSA NA03 847.5GB (1953525168 512B/sect)
So I first tried to downgrade firmware using:
disk_fw_update -R 0d.01.13
but that did not really work out, no error message, but just no change on the firmware of the disk - also nothing in the logs. So I read somewhere that this might be due to a bad/incompatible label on the disk and found out that the only way to get rid of the label is to boot into maintenance mode (thus having a downtime of parts of the HA-pair :-/) and recreate the label there.
So I booted into maintenance mode and have run the following command:
*> label makespare 0d.01.13 ********************************************************* Label editing is highly dangerous and should be attempted ONLY by NetApp Customer Support Engineers. Misuse can cause ALL data to be permanently lost. ********************************************************* 'Label save': Error while trying to save Label 1 on disk 0d.03.22: error 15. Label save: 1 of 35 disks could not save Label 1. 'Label save': Error while trying to save Label 2 on disk 0d.03.22: error 15. Label save: 1 of 35 disks could not save Label 2. The current set of labels on the disks have been saved. You can use the 'label restore' command to revert to these labels, the 'label keep' command to erase these labels, or 'label save' to preserve the current set of labels. ********************************************************* *** Run WAFL recovery tools (wafliron or wafliron_oc) after this. ***
Disk 0d.01.13 does not exist.
The last line suggests, that the disk still does not get recognized properly - and additionally, it (for whatever reason) claims to not be able to write the label 1 on disk 0d.03.22, although this is there and online:
riddler> disk show 0d.03.22 DISK OWNER POOL SERIAL NUMBER HOME ------------ ------------- ----- ------------- ------------- 0d.03.22 riddler (142248200) Pool0 N002XRML riddler (142248200)
I've stopped my expiriments at this point and am very thankful for fresh ideas :)
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
This is not due to a bad label. You would have seen a message when taking ownership of the disk about a bad label if that were the case. The disk would also show up in the failed disk pool (aggr status -f) with a bad or invalid label tag on it (I can't remember exactly what it says). The bad label only comes in to play if a disk was in a higher version of Data ONTAP, and was not marked as a spare, zeroed, and then ownership removed prior to inserting it in your system. On your version of ONTAP the label version is 11. So it would have had to have been in a system running 8.2 or higher.
Regarding 0d.03.22, is that disk a spare? What does your "sysconfig -r" output look like?
On Thu, Oct 22, 2015 at 10:06 AM, Alexander Griesser < AGriesser@anexia-it.com> wrote:
Hey everybody,
I just had the pleasure to try to replace a faulty disk in a filer which is out-of-service with a „new“ one from after market.
Of course, this didn’t work out as easy as it should, so maybe some could help me get this disk going.
Here are the facts:
FAS2040
NetApp Release 8.1.4P8 7-Mode
The disk at 0d.01.13 was broken, so I pulled it a few days ago – now that the replacment disk has arrived, I tried to plug it in and this is what happens:
Thu Oct 22 14:30:10 CEST [riddler:scsi.cmd.notReadyCondition:notice]: Disk device 0d.01.13: Device returns not yet ready: CDB 0x1b: Sense Data SCSI:not ready - Drive spinning up (0x2 - 0x4 0x1 0x28)(2).
The interesting thing, I can see the disk by using `storage show disk`:
riddler> storage show disk
DISK SHELF BAY SERIAL VENDOR MODEL REV
0d.01.13 1 13 J838WSRL NETAPP X302_HJUPI NA04
But I cannot see it in disk show.
riddler> disk show 0d.01.13
DISK OWNER POOL SERIAL NUMBER HOME
Sysconfig –v tells me, that there is a newer firmware on the „new“ disk:
01.13: NETAPP X302_HJUPI01TSSM NA04 847.5GB (1953525168512B/sect)
01.14: NETAPP X302_WVULC01TSSA NA03 847.5GB (1953525168512B/sect)
So I first tried to downgrade firmware using:
disk_fw_update –R 0d.01.13
but that did not really work out, no error message, but just no change on the firmware of the disk – also nothing in the logs.
So I read somewhere that this might be due to a bad/incompatible label on the disk and found out that the only way to get rid of the label is to boot into maintenance mode (thus having a downtime of parts of the HA-pair :-/) and recreate the label there.
So I booted into maintenance mode and have run the following command:
*> label makespare 0d.01.13
Label editing is highly dangerous and should be attempted
ONLY by NetApp Customer Support Engineers.
Misuse can cause ALL data to be permanently lost.
'Label save': Error while trying to save Label 1 on disk 0d.03.22: error 15.
Label save: 1 of 35 disks could not save Label 1.
'Label save': Error while trying to save Label 2 on disk 0d.03.22: error 15.
Label save: 1 of 35 disks could not save Label 2.
The current set of labels on the disks have been saved. You can
use the 'label restore' command to revert to these labels, the
'label keep' command to erase these labels, or 'label save' to
preserve the current set of labels.
*** Run WAFL recovery tools (wafliron or wafliron_oc) after this.
Disk 0d.01.13 does not exist.
The last line suggests, that the disk still does not get recognized properly – and additionally, it (for whatever reason) claims to not be able to write the label 1 on disk 0d.03.22, although this is there and online:
riddler> disk show 0d.03.22
DISK OWNER POOL SERIAL NUMBER HOME
0d.03.22 riddler (142248200) Pool0 N002XRML riddler (142248200)
I’ve stopped my expiriments at this point and am very thankful for fresh ideas J
Best,
*Alexander Griesser*
Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Oh, yes – you’re right.
0d.03.22 is a spare disk – so now the label message makes sense! Sysconfig –r does not show the disk 0d.01.13 either and aggr status –f is also clean:
riddler> aggr status -f
Broken disks (empty)
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Von: Brad Thompson [mailto:brad.thompson877@gmail.com] Gesendet: Donnerstag, 22. Oktober 2015 16:36 An: Alexander Griesser AGriesser@anexia-it.com Cc: toasters@teaparty.net Betreff: Re: Used Replacement disk not recognized
This is not due to a bad label. You would have seen a message when taking ownership of the disk about a bad label if that were the case. The disk would also show up in the failed disk pool (aggr status -f) with a bad or invalid label tag on it (I can't remember exactly what it says). The bad label only comes in to play if a disk was in a higher version of Data ONTAP, and was not marked as a spare, zeroed, and then ownership removed prior to inserting it in your system. On your version of ONTAP the label version is 11. So it would have had to have been in a system running 8.2 or higher. Regarding 0d.03.22, is that disk a spare? What does your "sysconfig -r" output look like?
On Thu, Oct 22, 2015 at 10:06 AM, Alexander Griesser <AGriesser@anexia-it.commailto:AGriesser@anexia-it.com> wrote: Hey everybody,
I just had the pleasure to try to replace a faulty disk in a filer which is out-of-service with a „new“ one from after market. Of course, this didn’t work out as easy as it should, so maybe some could help me get this disk going.
Here are the facts: FAS2040 NetApp Release 8.1.4P8 7-Mode
The disk at 0d.01.13 was broken, so I pulled it a few days ago – now that the replacment disk has arrived, I tried to plug it in and this is what happens:
Thu Oct 22 14:30:10 CEST [riddler:scsi.cmd.notReadyCondition:notice]: Disk device 0d.01.13: Device returns not yet ready: CDB 0x1b: Sense Data SCSI:not ready - Drive spinning up (0x2 - 0x4 0x1 0x28)(2).
The interesting thing, I can see the disk by using `storage show disk`:
riddler> storage show disk DISK SHELF BAY SERIAL VENDOR MODEL REV --------------------- --------- ---------------- -------- ---------- ---- 0d.01.13 1 13 J838WSRL NETAPP X302_HJUPI NA04
But I cannot see it in disk show.
riddler> disk show 0d.01.13 DISK OWNER POOL SERIAL NUMBER HOME ------------ ------------- ----- ------------- -------------
Sysconfig –v tells me, that there is a newer firmware on the „new“ disk:
01.13: NETAPP X302_HJUPI01TSSM NA04 847.5GB (1953525168 512B/sect) 01.14: NETAPP X302_WVULC01TSSA NA03 847.5GB (1953525168 512B/sect)
So I first tried to downgrade firmware using:
disk_fw_update –R 0d.01.13
but that did not really work out, no error message, but just no change on the firmware of the disk – also nothing in the logs. So I read somewhere that this might be due to a bad/incompatible label on the disk and found out that the only way to get rid of the label is to boot into maintenance mode (thus having a downtime of parts of the HA-pair :-/) and recreate the label there.
So I booted into maintenance mode and have run the following command:
*> label makespare 0d.01.13 ********************************************************* Label editing is highly dangerous and should be attempted ONLY by NetApp Customer Support Engineers. Misuse can cause ALL data to be permanently lost. ********************************************************* 'Label save': Error while trying to save Label 1 on disk 0d.03.22: error 15. Label save: 1 of 35 disks could not save Label 1. 'Label save': Error while trying to save Label 2 on disk 0d.03.22: error 15. Label save: 1 of 35 disks could not save Label 2. The current set of labels on the disks have been saved. You can use the 'label restore' command to revert to these labels, the 'label keep' command to erase these labels, or 'label save' to preserve the current set of labels. ********************************************************* *** Run WAFL recovery tools (wafliron or wafliron_oc) after this. ***
Disk 0d.01.13 does not exist.
The last line suggests, that the disk still does not get recognized properly – and additionally, it (for whatever reason) claims to not be able to write the label 1 on disk 0d.03.22, although this is there and online:
riddler> disk show 0d.03.22 DISK OWNER POOL SERIAL NUMBER HOME ------------ ------------- ----- ------------- ------------- 0d.03.22 riddler (142248200) Pool0 N002XRML riddler (142248200)
I’ve stopped my expiriments at this point and am very thankful for fresh ideas ☺
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
One thing to look at... Do you have the latest disk qualification package? Doubt that's it though. It looks like the disk is not properly spinning up... You replaced a bad drive with a not so good drive 😢 --tmac
Sent from Mobile Outlook
On Thu, Oct 22, 2015 at 7:10 AM -0700, "Alexander Griesser" AGriesser@anexia-it.com wrote:
Hey everybody,
I just had the pleasure to try to replace a faulty disk in a filer which is out-of-service with a „new“ one from after market.
Of course, this didn’t work out as easy as it should, so maybe some could help me get this disk going.
Here are the facts:
FAS2040
NetApp Release 8.1.4P8 7-Mode
The disk at 0d.01.13 was broken, so I pulled it a few days ago – now that the replacment disk has arrived, I tried to plug it in and this is what happens:
Thu Oct 22 14:30:10 CEST [riddler:scsi.cmd.notReadyCondition:notice]: Disk device 0d.01.13: Device returns not yet ready: CDB 0x1b: Sense Data SCSI:not ready - Drive spinning up (0x2 - 0x4 0x1 0x28)(2).
The interesting thing, I can see the disk by using `storage show disk`:
riddler> storage show disk
DISK SHELF BAY SERIAL VENDOR MODEL REV
--------------------- --------- ---------------- -------- ---------- ----
0d.01.13 1 13 J838WSRL NETAPP X302_HJUPI NA04
But I cannot see it in disk show.
riddler> disk show 0d.01.13
DISK OWNER POOL SERIAL NUMBER HOME
------------ ------------- ----- ------------- -------------
Sysconfig –v tells me, that there is a newer firmware on the „new“ disk:
01.13: NETAPP X302_HJUPI01TSSM NA04 847.5GB (1953525168 512B/sect)
01.14: NETAPP X302_WVULC01TSSA NA03 847.5GB (1953525168 512B/sect)
So I first tried to downgrade firmware using:
disk_fw_update –R 0d.01.13
but that did not really work out, no error message, but just no change on the firmware of the disk – also nothing in the logs.
So I read somewhere that this might be due to a bad/incompatible label on the disk and found out that the only way to get rid of the label is to boot into maintenance mode (thus having a downtime of parts of the HA-pair :-/) and recreate the label there.
So I booted into maintenance mode and have run the following command:
*> label makespare 0d.01.13
*********************************************************
Label editing is highly dangerous and should be attempted
ONLY by NetApp Customer Support Engineers.
Misuse can cause ALL data to be permanently lost.
*********************************************************
'Label save': Error while trying to save Label 1 on disk 0d.03.22: error 15.
Label save: 1 of 35 disks could not save Label 1.
'Label save': Error while trying to save Label 2 on disk 0d.03.22: error 15.
Label save: 1 of 35 disks could not save Label 2.
The current set of labels on the disks have been saved. You can
use the 'label restore' command to revert to these labels, the
'label keep' command to erase these labels, or 'label save' to
preserve the current set of labels.
*********************************************************
*** Run WAFL recovery tools (wafliron or wafliron_oc) after this. ***
Disk 0d.01.13 does not exist.
The last line suggests, that the disk still does not get recognized properly – and additionally, it (for whatever reason) claims to not be able to write the label 1 on disk 0d.03.22, although this is there and online:
riddler> disk show 0d.03.22
DISK OWNER POOL SERIAL NUMBER HOME
------------ ------------- ----- ------------- -------------
0d.03.22 riddler (142248200) Pool0 N002XRML riddler (142248200)
I’ve stopped my expiriments at this point and am very thankful for fresh ideas J
Best,
Alexander Griesser
Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Well, in fact, I ordered two replacement drives and both show the exact same symptom… So I’m either the unluckiest guy on the world, or this is unrelated to the actual problem ☺
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Von: Tim McCarthy [mailto:tmacmd@gmail.com] Gesendet: Donnerstag, 22. Oktober 2015 16:38 An: toasters@teaparty.net; Alexander Griesser AGriesser@anexia-it.com Betreff: Re: Used Replacement disk not recognized
One thing to look at...
Do you have the latest disk qualification package?
Doubt that's it though. It looks like the disk is not properly spinning up... You replaced a bad drive with a not so good drive 😢
--tmac Sent from Mobile Outlookhttp://aka.ms/Ox5hz3
On Thu, Oct 22, 2015 at 7:10 AM -0700, "Alexander Griesser" <AGriesser@anexia-it.commailto:AGriesser@anexia-it.com> wrote: Hey everybody,
I just had the pleasure to try to replace a faulty disk in a filer which is out-of-service with a „new“ one from after market. Of course, this didn’t work out as easy as it should, so maybe some could help me get this disk going.
Here are the facts: FAS2040 NetApp Release 8.1.4P8 7-Mode
The disk at 0d.01.13 was broken, so I pulled it a few days ago – now that the replacment disk has arrived, I tried to plug it in and this is what happens:
Thu Oct 22 14:30:10 CEST [riddler:scsi.cmd.notReadyCondition:notice]: Disk device 0d.01.13: Device returns not yet ready: CDB 0x1b: Sense Data SCSI:not ready - Drive spinning up (0x2 - 0x4 0x1 0x28)(2).
The interesting thing, I can see the disk by using `storage show disk`:
riddler> storage show disk DISK SHELF BAY SERIAL VENDOR MODEL REV --------------------- --------- ---------------- -------- ---------- ---- 0d.01.13 1 13 J838WSRL NETAPP X302_HJUPI NA04
But I cannot see it in disk show.
riddler> disk show 0d.01.13 DISK OWNER POOL SERIAL NUMBER HOME ------------ ------------- ----- ------------- -------------
Sysconfig –v tells me, that there is a newer firmware on the „new“ disk:
01.13: NETAPP X302_HJUPI01TSSM NA04 847.5GB (1953525168 512B/sect) 01.14: NETAPP X302_WVULC01TSSA NA03 847.5GB (1953525168 512B/sect)
So I first tried to downgrade firmware using:
disk_fw_update –R 0d.01.13
but that did not really work out, no error message, but just no change on the firmware of the disk – also nothing in the logs. So I read somewhere that this might be due to a bad/incompatible label on the disk and found out that the only way to get rid of the label is to boot into maintenance mode (thus having a downtime of parts of the HA-pair :-/) and recreate the label there.
So I booted into maintenance mode and have run the following command:
*> label makespare 0d.01.13 ********************************************************* Label editing is highly dangerous and should be attempted ONLY by NetApp Customer Support Engineers. Misuse can cause ALL data to be permanently lost. ********************************************************* 'Label save': Error while trying to save Label 1 on disk 0d.03.22: error 15. Label save: 1 of 35 disks could not save Label 1. 'Label save': Error while trying to save Label 2 on disk 0d.03.22: error 15. Label save: 1 of 35 disks could not save Label 2. The current set of labels on the disks have been saved. You can use the 'label restore' command to revert to these labels, the 'label keep' command to erase these labels, or 'label save' to preserve the current set of labels. ********************************************************* *** Run WAFL recovery tools (wafliron or wafliron_oc) after this. ***
Disk 0d.01.13 does not exist.
The last line suggests, that the disk still does not get recognized properly – and additionally, it (for whatever reason) claims to not be able to write the label 1 on disk 0d.03.22, although this is there and online:
riddler> disk show 0d.03.22 DISK OWNER POOL SERIAL NUMBER HOME ------------ ------------- ----- ------------- ------------- 0d.03.22 riddler (142248200) Pool0 N002XRML riddler (142248200)
I’ve stopped my expiriments at this point and am very thankful for fresh ideas ☺
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Uh oh. That *could* be the indicator of a bad slot If you have them.... Issue the command to remove a working spare Insert one of your replacement disks into that slot and put the spare you removed into the suspect slot If you see the same problems on the same slot, you likely have a bad slot on the shelf Occasionally, this could be caused by a disk error in an adjacent slot... Although I have not seen that since the DS14 shelves. Worth a look. There are commands to query the disk statistics ( sas or fc counters) Do not have them handy. Maybe someone else can advise there. Sent from Mobile Outlook
_____________________________ From: Alexander Griesser agriesser@anexia-it.com Sent: Thursday, October 22, 2015 10:40 AM Subject: AW: Used Replacement disk not recognized To: Tim McCarthy tmacmd@gmail.com, toasters@teaparty.net
Well, in fact, I ordered two replacement drives and both show the exact same symptom…
So I’m either the unluckiest guy on the world, or this is unrelated to the actual problem J
Best,
Alexander Griesser
Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Von: Tim McCarthy [mailto:tmacmd@gmail.com] Gesendet: Donnerstag, 22. Oktober 2015 16:38 An: toasters@teaparty.net; Alexander Griesser AGriesser@anexia-it.com Betreff: Re: Used Replacement disk not recognized
One thing to look at...
Do you have the latest disk qualification package?
Doubt that's it though. It looks like the disk is not properly spinning up... You replaced a bad drive with a not so good drive 😢
--tmac
Sent from Mobile Outlook
On Thu, Oct 22, 2015 at 7:10 AM -0700, "Alexander Griesser" AGriesser@anexia-it.com wrote:
Hey everybody,
I just had the pleasure to try to replace a faulty disk in a filer which is out-of-service with a „new“ one from after market.
Of course, this didn’t work out as easy as it should, so maybe some could help me get this disk going.
Here are the facts:
FAS2040
NetApp Release 8.1.4P8 7-Mode
The disk at 0d.01.13 was broken, so I pulled it a few days ago – now that the replacment disk has arrived, I tried to plug it in and this is what happens:
Thu Oct 22 14:30:10 CEST [riddler:scsi.cmd.notReadyCondition:notice]: Disk device 0d.01.13: Device returns not yet ready: CDB 0x1b: Sense Data SCSI:not ready - Drive spinning up (0x2 - 0x4 0x1 0x28)(2).
The interesting thing, I can see the disk by using `storage show disk`:
riddler> storage show disk
DISK SHELF BAY SERIAL VENDOR MODEL REV
--------------------- --------- ---------------- -------- ---------- ----
0d.01.13 1 13 J838WSRL NETAPP X302_HJUPI NA04
But I cannot see it in disk show.
riddler> disk show 0d.01.13
DISK OWNER POOL SERIAL NUMBER HOME
------------ ------------- ----- ------------- -------------
Sysconfig –v tells me, that there is a newer firmware on the „new“ disk:
01.13: NETAPP X302_HJUPI01TSSM NA04 847.5GB (1953525168 512B/sect)
01.14: NETAPP X302_WVULC01TSSA NA03 847.5GB (1953525168 512B/sect)
So I first tried to downgrade firmware using:
disk_fw_update –R 0d.01.13
but that did not really work out, no error message, but just no change on the firmware of the disk – also nothing in the logs.
So I read somewhere that this might be due to a bad/incompatible label on the disk and found out that the only way to get rid of the label is to boot into maintenance mode (thus having a downtime of parts of the HA-pair :-/) and recreate the label there.
So I booted into maintenance mode and have run the following command:
*> label makespare 0d.01.13
*********************************************************
Label editing is highly dangerous and should be attempted
ONLY by NetApp Customer Support Engineers.
Misuse can cause ALL data to be permanently lost.
*********************************************************
'Label save': Error while trying to save Label 1 on disk 0d.03.22: error 15.
Label save: 1 of 35 disks could not save Label 1.
'Label save': Error while trying to save Label 2 on disk 0d.03.22: error 15.
Label save: 1 of 35 disks could not save Label 2.
The current set of labels on the disks have been saved. You can
use the 'label restore' command to revert to these labels, the
'label keep' command to erase these labels, or 'label save' to
preserve the current set of labels.
*********************************************************
*** Run WAFL recovery tools (wafliron or wafliron_oc) after this. ***
Disk 0d.01.13 does not exist.
The last line suggests, that the disk still does not get recognized properly – and additionally, it (for whatever reason) claims to not be able to write the label 1 on disk 0d.03.22, although this is there and online:
riddler> disk show 0d.03.22
DISK OWNER POOL SERIAL NUMBER HOME
------------ ------------- ----- ------------- -------------
0d.03.22 riddler (142248200) Pool0 N002XRML riddler (142248200)
I’ve stopped my expiriments at this point and am very thankful for fresh ideas J
Best,
Alexander Griesser
Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Firmware is not older or newer – those two disks are from different manufacturers so they have different firmware; both are using the latest versions publically available.
KB https://kb.netapp.com/support/index?page=content&id=2010827&locale=e... implies that Drive spinning up may be seen with X302 and is otherwise harmless, although there are so many different variants of X302 it is hard to guess whether it applies in your case. I would suggest open the case.
Last command you pasted suggests that everything is OK now; or what is remaining problem?
--- With best regards
Andrei Borzenkov Senior system engineer FTS WEMEAI RUC RU SC TMS FOS [cid:image001.gif@01CBF835.B3FEDA90] FUJITSU Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation Tel.: +7 495 730 62 20 ( reception) Mob.: +7 916 678 7208 Fax: +7 495 730 62 14 E-mail: Andrei.Borzenkov@ts.fujitsu.commailto:Andrei.Borzenkov@ts.fujitsu.com Web: ru.fujitsu.comhttp://ts.fujitsu.com/ Company details: ts.fujitsu.com/imprinthttp://ts.fujitsu.com/imprint.html This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation. Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Alexander Griesser Sent: Thursday, October 22, 2015 5:06 PM To: toasters@teaparty.net Subject: Used Replacement disk not recognized
Hey everybody,
I just had the pleasure to try to replace a faulty disk in a filer which is out-of-service with a „new“ one from after market. Of course, this didn’t work out as easy as it should, so maybe some could help me get this disk going.
Here are the facts: FAS2040 NetApp Release 8.1.4P8 7-Mode
The disk at 0d.01.13 was broken, so I pulled it a few days ago – now that the replacment disk has arrived, I tried to plug it in and this is what happens:
Thu Oct 22 14:30:10 CEST [riddler:scsi.cmd.notReadyCondition:notice]: Disk device 0d.01.13: Device returns not yet ready: CDB 0x1b: Sense Data SCSI:not ready - Drive spinning up (0x2 - 0x4 0x1 0x28)(2).
The interesting thing, I can see the disk by using `storage show disk`:
riddler> storage show disk DISK SHELF BAY SERIAL VENDOR MODEL REV --------------------- --------- ---------------- -------- ---------- ---- 0d.01.13 1 13 J838WSRL NETAPP X302_HJUPI NA04
But I cannot see it in disk show.
riddler> disk show 0d.01.13 DISK OWNER POOL SERIAL NUMBER HOME ------------ ------------- ----- ------------- -------------
Sysconfig –v tells me, that there is a newer firmware on the „new“ disk:
01.13: NETAPP X302_HJUPI01TSSM NA04 847.5GB (1953525168 512B/sect) 01.14: NETAPP X302_WVULC01TSSA NA03 847.5GB (1953525168 512B/sect)
So I first tried to downgrade firmware using:
disk_fw_update –R 0d.01.13
but that did not really work out, no error message, but just no change on the firmware of the disk – also nothing in the logs. So I read somewhere that this might be due to a bad/incompatible label on the disk and found out that the only way to get rid of the label is to boot into maintenance mode (thus having a downtime of parts of the HA-pair :-/) and recreate the label there.
So I booted into maintenance mode and have run the following command:
*> label makespare 0d.01.13 ********************************************************* Label editing is highly dangerous and should be attempted ONLY by NetApp Customer Support Engineers. Misuse can cause ALL data to be permanently lost. ********************************************************* 'Label save': Error while trying to save Label 1 on disk 0d.03.22: error 15. Label save: 1 of 35 disks could not save Label 1. 'Label save': Error while trying to save Label 2 on disk 0d.03.22: error 15. Label save: 1 of 35 disks could not save Label 2. The current set of labels on the disks have been saved. You can use the 'label restore' command to revert to these labels, the 'label keep' command to erase these labels, or 'label save' to preserve the current set of labels. ********************************************************* *** Run WAFL recovery tools (wafliron or wafliron_oc) after this. ***
Disk 0d.01.13 does not exist.
The last line suggests, that the disk still does not get recognized properly – and additionally, it (for whatever reason) claims to not be able to write the label 1 on disk 0d.03.22, although this is there and online:
riddler> disk show 0d.03.22 DISK OWNER POOL SERIAL NUMBER HOME ------------ ------------- ----- ------------- ------------- 0d.03.22 riddler (142248200) Pool0 N002XRML riddler (142248200)
I’ve stopped my expiriments at this point and am very thankful for fresh ideas :)
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Hi Andrei,
the last command states "Disk 0d.01.13 does not exist", I don't think that this is cool... The problem is that even after the reboot, the disk does not show up anywhere except for sysconfig or storage show disk.
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Von: andrei.borzenkov@ts.fujitsu.com [mailto:andrei.borzenkov@ts.fujitsu.com] Gesendet: Donnerstag, 22. Oktober 2015 16:41 An: Alexander Griesser AGriesser@anexia-it.com; toasters@teaparty.net Betreff: RE: Used Replacement disk not recognized
Firmware is not older or newer - those two disks are from different manufacturers so they have different firmware; both are using the latest versions publically available.
KB https://kb.netapp.com/support/index?page=content&id=2010827&locale=e... implies that Drive spinning up may be seen with X302 and is otherwise harmless, although there are so many different variants of X302 it is hard to guess whether it applies in your case. I would suggest open the case.
Last command you pasted suggests that everything is OK now; or what is remaining problem?
--- With best regards
Andrei Borzenkov Senior system engineer FTS WEMEAI RUC RU SC TMS FOS [cid:image001.gif@01CBF835.B3FEDA90] FUJITSU Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation Tel.: +7 495 730 62 20 ( reception) Mob.: +7 916 678 7208 Fax: +7 495 730 62 14 E-mail: Andrei.Borzenkov@ts.fujitsu.commailto:Andrei.Borzenkov@ts.fujitsu.com Web: ru.fujitsu.comhttp://ts.fujitsu.com/ Company details: ts.fujitsu.com/imprinthttp://ts.fujitsu.com/imprint.html This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation. Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.
From: toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Alexander Griesser Sent: Thursday, October 22, 2015 5:06 PM To: toasters@teaparty.netmailto:toasters@teaparty.net Subject: Used Replacement disk not recognized
Hey everybody,
I just had the pleasure to try to replace a faulty disk in a filer which is out-of-service with a "new" one from after market. Of course, this didn't work out as easy as it should, so maybe some could help me get this disk going.
Here are the facts: FAS2040 NetApp Release 8.1.4P8 7-Mode
The disk at 0d.01.13 was broken, so I pulled it a few days ago - now that the replacment disk has arrived, I tried to plug it in and this is what happens:
Thu Oct 22 14:30:10 CEST [riddler:scsi.cmd.notReadyCondition:notice]: Disk device 0d.01.13: Device returns not yet ready: CDB 0x1b: Sense Data SCSI:not ready - Drive spinning up (0x2 - 0x4 0x1 0x28)(2).
The interesting thing, I can see the disk by using `storage show disk`:
riddler> storage show disk DISK SHELF BAY SERIAL VENDOR MODEL REV --------------------- --------- ---------------- -------- ---------- ---- 0d.01.13 1 13 J838WSRL NETAPP X302_HJUPI NA04
But I cannot see it in disk show.
riddler> disk show 0d.01.13 DISK OWNER POOL SERIAL NUMBER HOME ------------ ------------- ----- ------------- -------------
Sysconfig -v tells me, that there is a newer firmware on the "new" disk:
01.13: NETAPP X302_HJUPI01TSSM NA04 847.5GB (1953525168 512B/sect) 01.14: NETAPP X302_WVULC01TSSA NA03 847.5GB (1953525168 512B/sect)
So I first tried to downgrade firmware using:
disk_fw_update -R 0d.01.13
but that did not really work out, no error message, but just no change on the firmware of the disk - also nothing in the logs. So I read somewhere that this might be due to a bad/incompatible label on the disk and found out that the only way to get rid of the label is to boot into maintenance mode (thus having a downtime of parts of the HA-pair :-/) and recreate the label there.
So I booted into maintenance mode and have run the following command:
*> label makespare 0d.01.13 ********************************************************* Label editing is highly dangerous and should be attempted ONLY by NetApp Customer Support Engineers. Misuse can cause ALL data to be permanently lost. ********************************************************* 'Label save': Error while trying to save Label 1 on disk 0d.03.22: error 15. Label save: 1 of 35 disks could not save Label 1. 'Label save': Error while trying to save Label 2 on disk 0d.03.22: error 15. Label save: 1 of 35 disks could not save Label 2. The current set of labels on the disks have been saved. You can use the 'label restore' command to revert to these labels, the 'label keep' command to erase these labels, or 'label save' to preserve the current set of labels. ********************************************************* *** Run WAFL recovery tools (wafliron or wafliron_oc) after this. ***
Disk 0d.01.13 does not exist.
The last line suggests, that the disk still does not get recognized properly - and additionally, it (for whatever reason) claims to not be able to write the label 1 on disk 0d.03.22, although this is there and online:
riddler> disk show 0d.03.22 DISK OWNER POOL SERIAL NUMBER HOME ------------ ------------- ----- ------------- ------------- 0d.03.22 riddler (142248200) Pool0 N002XRML riddler (142248200)
I've stopped my expiriments at this point and am very thankful for fresh ideas :)
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Does it appear in “disk show -v”?
--- With best regards
Andrei Borzenkov Senior system engineer FTS WEMEAI RUC RU SC TMS FOS [cid:image001.gif@01CBF835.B3FEDA90] FUJITSU Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation Tel.: +7 495 730 62 20 ( reception) Mob.: +7 916 678 7208 Fax: +7 495 730 62 14 E-mail: Andrei.Borzenkov@ts.fujitsu.commailto:Andrei.Borzenkov@ts.fujitsu.com Web: ru.fujitsu.comhttp://ts.fujitsu.com/ Company details: ts.fujitsu.com/imprinthttp://ts.fujitsu.com/imprint.html This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation. Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.
From: Alexander Griesser [mailto:AGriesser@anexia-it.com] Sent: Thursday, October 22, 2015 5:44 PM To: Borzenkov, Andrei; toasters@teaparty.net Subject: AW: Used Replacement disk not recognized
Hi Andrei,
the last command states „Disk 0d.01.13 does not exist“, I don’t think that this is cool… The problem is that even after the reboot, the disk does not show up anywhere except for sysconfig or storage show disk.
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Von: andrei.borzenkov@ts.fujitsu.commailto:andrei.borzenkov@ts.fujitsu.com [mailto:andrei.borzenkov@ts.fujitsu.com] Gesendet: Donnerstag, 22. Oktober 2015 16:41 An: Alexander Griesser <AGriesser@anexia-it.commailto:AGriesser@anexia-it.com>; toasters@teaparty.netmailto:toasters@teaparty.net Betreff: RE: Used Replacement disk not recognized
Firmware is not older or newer – those two disks are from different manufacturers so they have different firmware; both are using the latest versions publically available.
KB https://kb.netapp.com/support/index?page=content&id=2010827&locale=e... implies that Drive spinning up may be seen with X302 and is otherwise harmless, although there are so many different variants of X302 it is hard to guess whether it applies in your case. I would suggest open the case.
Last command you pasted suggests that everything is OK now; or what is remaining problem?
--- With best regards
Andrei Borzenkov Senior system engineer FTS WEMEAI RUC RU SC TMS FOS [cid:image001.gif@01CBF835.B3FEDA90] FUJITSU Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation Tel.: +7 495 730 62 20 ( reception) Mob.: +7 916 678 7208 Fax: +7 495 730 62 14 E-mail: Andrei.Borzenkov@ts.fujitsu.commailto:Andrei.Borzenkov@ts.fujitsu.com Web: ru.fujitsu.comhttp://ts.fujitsu.com/ Company details: ts.fujitsu.com/imprinthttp://ts.fujitsu.com/imprint.html This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation. Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.
From: toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Alexander Griesser Sent: Thursday, October 22, 2015 5:06 PM To: toasters@teaparty.netmailto:toasters@teaparty.net Subject: Used Replacement disk not recognized
Hey everybody,
I just had the pleasure to try to replace a faulty disk in a filer which is out-of-service with a „new“ one from after market. Of course, this didn’t work out as easy as it should, so maybe some could help me get this disk going.
Here are the facts: FAS2040 NetApp Release 8.1.4P8 7-Mode
The disk at 0d.01.13 was broken, so I pulled it a few days ago – now that the replacment disk has arrived, I tried to plug it in and this is what happens:
Thu Oct 22 14:30:10 CEST [riddler:scsi.cmd.notReadyCondition:notice]: Disk device 0d.01.13: Device returns not yet ready: CDB 0x1b: Sense Data SCSI:not ready - Drive spinning up (0x2 - 0x4 0x1 0x28)(2).
The interesting thing, I can see the disk by using `storage show disk`:
riddler> storage show disk DISK SHELF BAY SERIAL VENDOR MODEL REV --------------------- --------- ---------------- -------- ---------- ---- 0d.01.13 1 13 J838WSRL NETAPP X302_HJUPI NA04
But I cannot see it in disk show.
riddler> disk show 0d.01.13 DISK OWNER POOL SERIAL NUMBER HOME ------------ ------------- ----- ------------- -------------
Sysconfig –v tells me, that there is a newer firmware on the „new“ disk:
01.13: NETAPP X302_HJUPI01TSSM NA04 847.5GB (1953525168 512B/sect) 01.14: NETAPP X302_WVULC01TSSA NA03 847.5GB (1953525168 512B/sect)
So I first tried to downgrade firmware using:
disk_fw_update –R 0d.01.13
but that did not really work out, no error message, but just no change on the firmware of the disk – also nothing in the logs. So I read somewhere that this might be due to a bad/incompatible label on the disk and found out that the only way to get rid of the label is to boot into maintenance mode (thus having a downtime of parts of the HA-pair :-/) and recreate the label there.
So I booted into maintenance mode and have run the following command:
*> label makespare 0d.01.13 ********************************************************* Label editing is highly dangerous and should be attempted ONLY by NetApp Customer Support Engineers. Misuse can cause ALL data to be permanently lost. ********************************************************* 'Label save': Error while trying to save Label 1 on disk 0d.03.22: error 15. Label save: 1 of 35 disks could not save Label 1. 'Label save': Error while trying to save Label 2 on disk 0d.03.22: error 15. Label save: 1 of 35 disks could not save Label 2. The current set of labels on the disks have been saved. You can use the 'label restore' command to revert to these labels, the 'label keep' command to erase these labels, or 'label save' to preserve the current set of labels. ********************************************************* *** Run WAFL recovery tools (wafliron or wafliron_oc) after this. ***
Disk 0d.01.13 does not exist.
The last line suggests, that the disk still does not get recognized properly – and additionally, it (for whatever reason) claims to not be able to write the label 1 on disk 0d.03.22, although this is there and online:
riddler> disk show 0d.03.22 DISK OWNER POOL SERIAL NUMBER HOME ------------ ------------- ----- ------------- ------------- 0d.03.22 riddler (142248200) Pool0 N002XRML riddler (142248200)
I’ve stopped my expiriments at this point and am very thankful for fresh ideas :)
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Bingo:
riddler> disk show -v DISK OWNER POOL SERIAL NUMBER HOME CHKSUM ------------ ------------- ----- ------------- ------------- ------ 0d.01.13 w-nas-04 (142255463) Pool0 J838WSRL w-nas-04 (142255463) Block
"w-nas-04" is none of my filers - so I need to remove the ownership here now, I guess.
riddler*> disk remove_ownership 0d.01.13 -f disk remove_ownership: Disk 0d.01.13 is not owned by this node. riddler*> disk reassign -o w-nas-04 -n riddler disk reassign: Can only be run in maintenance mode or during takeover in advanced mode riddler*>
So, I guess I have to - again - boot into maintenance mode or is there a trick which would allow me to remove the ownership without having to shut everything down again?
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Von: andrei.borzenkov@ts.fujitsu.com [mailto:andrei.borzenkov@ts.fujitsu.com] Gesendet: Donnerstag, 22. Oktober 2015 16:47 An: Alexander Griesser AGriesser@anexia-it.com; toasters@teaparty.net Betreff: RE: Used Replacement disk not recognized
Does it appear in "disk show -v"?
--- With best regards
Andrei Borzenkov Senior system engineer FTS WEMEAI RUC RU SC TMS FOS [cid:image001.gif@01CBF835.B3FEDA90] FUJITSU Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation Tel.: +7 495 730 62 20 ( reception) Mob.: +7 916 678 7208 Fax: +7 495 730 62 14 E-mail: Andrei.Borzenkov@ts.fujitsu.commailto:Andrei.Borzenkov@ts.fujitsu.com Web: ru.fujitsu.comhttp://ts.fujitsu.com/ Company details: ts.fujitsu.com/imprinthttp://ts.fujitsu.com/imprint.html This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation. Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.
From: Alexander Griesser [mailto:AGriesser@anexia-it.com] Sent: Thursday, October 22, 2015 5:44 PM To: Borzenkov, Andrei; toasters@teaparty.netmailto:toasters@teaparty.net Subject: AW: Used Replacement disk not recognized
Hi Andrei,
the last command states "Disk 0d.01.13 does not exist", I don't think that this is cool... The problem is that even after the reboot, the disk does not show up anywhere except for sysconfig or storage show disk.
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Von: andrei.borzenkov@ts.fujitsu.commailto:andrei.borzenkov@ts.fujitsu.com [mailto:andrei.borzenkov@ts.fujitsu.com] Gesendet: Donnerstag, 22. Oktober 2015 16:41 An: Alexander Griesser <AGriesser@anexia-it.commailto:AGriesser@anexia-it.com>; toasters@teaparty.netmailto:toasters@teaparty.net Betreff: RE: Used Replacement disk not recognized
Firmware is not older or newer - those two disks are from different manufacturers so they have different firmware; both are using the latest versions publically available.
KB https://kb.netapp.com/support/index?page=content&id=2010827&locale=e... implies that Drive spinning up may be seen with X302 and is otherwise harmless, although there are so many different variants of X302 it is hard to guess whether it applies in your case. I would suggest open the case.
Last command you pasted suggests that everything is OK now; or what is remaining problem?
--- With best regards
Andrei Borzenkov Senior system engineer FTS WEMEAI RUC RU SC TMS FOS [cid:image001.gif@01CBF835.B3FEDA90] FUJITSU Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation Tel.: +7 495 730 62 20 ( reception) Mob.: +7 916 678 7208 Fax: +7 495 730 62 14 E-mail: Andrei.Borzenkov@ts.fujitsu.commailto:Andrei.Borzenkov@ts.fujitsu.com Web: ru.fujitsu.comhttp://ts.fujitsu.com/ Company details: ts.fujitsu.com/imprinthttp://ts.fujitsu.com/imprint.html This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation. Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.
From: toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Alexander Griesser Sent: Thursday, October 22, 2015 5:06 PM To: toasters@teaparty.netmailto:toasters@teaparty.net Subject: Used Replacement disk not recognized
Hey everybody,
I just had the pleasure to try to replace a faulty disk in a filer which is out-of-service with a "new" one from after market. Of course, this didn't work out as easy as it should, so maybe some could help me get this disk going.
Here are the facts: FAS2040 NetApp Release 8.1.4P8 7-Mode
The disk at 0d.01.13 was broken, so I pulled it a few days ago - now that the replacment disk has arrived, I tried to plug it in and this is what happens:
Thu Oct 22 14:30:10 CEST [riddler:scsi.cmd.notReadyCondition:notice]: Disk device 0d.01.13: Device returns not yet ready: CDB 0x1b: Sense Data SCSI:not ready - Drive spinning up (0x2 - 0x4 0x1 0x28)(2).
The interesting thing, I can see the disk by using `storage show disk`:
riddler> storage show disk DISK SHELF BAY SERIAL VENDOR MODEL REV --------------------- --------- ---------------- -------- ---------- ---- 0d.01.13 1 13 J838WSRL NETAPP X302_HJUPI NA04
But I cannot see it in disk show.
riddler> disk show 0d.01.13 DISK OWNER POOL SERIAL NUMBER HOME ------------ ------------- ----- ------------- -------------
Sysconfig -v tells me, that there is a newer firmware on the "new" disk:
01.13: NETAPP X302_HJUPI01TSSM NA04 847.5GB (1953525168 512B/sect) 01.14: NETAPP X302_WVULC01TSSA NA03 847.5GB (1953525168 512B/sect)
So I first tried to downgrade firmware using:
disk_fw_update -R 0d.01.13
but that did not really work out, no error message, but just no change on the firmware of the disk - also nothing in the logs. So I read somewhere that this might be due to a bad/incompatible label on the disk and found out that the only way to get rid of the label is to boot into maintenance mode (thus having a downtime of parts of the HA-pair :-/) and recreate the label there.
So I booted into maintenance mode and have run the following command:
*> label makespare 0d.01.13 ********************************************************* Label editing is highly dangerous and should be attempted ONLY by NetApp Customer Support Engineers. Misuse can cause ALL data to be permanently lost. ********************************************************* 'Label save': Error while trying to save Label 1 on disk 0d.03.22: error 15. Label save: 1 of 35 disks could not save Label 1. 'Label save': Error while trying to save Label 2 on disk 0d.03.22: error 15. Label save: 1 of 35 disks could not save Label 2. The current set of labels on the disks have been saved. You can use the 'label restore' command to revert to these labels, the 'label keep' command to erase these labels, or 'label save' to preserve the current set of labels. ********************************************************* *** Run WAFL recovery tools (wafliron or wafliron_oc) after this. ***
Disk 0d.01.13 does not exist.
The last line suggests, that the disk still does not get recognized properly - and additionally, it (for whatever reason) claims to not be able to write the label 1 on disk 0d.03.22, although this is there and online:
riddler> disk show 0d.03.22 DISK OWNER POOL SERIAL NUMBER HOME ------------ ------------- ----- ------------- ------------- 0d.03.22 riddler (142248200) Pool0 N002XRML riddler (142248200)
I've stopped my expiriments at this point and am very thankful for fresh ideas :)
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
disk assign 0d.01.13 -f
--- With best regards
Andrei Borzenkov Senior system engineer FTS WEMEAI RUC RU SC TMS FOS [cid:image001.gif@01CBF835.B3FEDA90] FUJITSU Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation Tel.: +7 495 730 62 20 ( reception) Mob.: +7 916 678 7208 Fax: +7 495 730 62 14 E-mail: Andrei.Borzenkov@ts.fujitsu.commailto:Andrei.Borzenkov@ts.fujitsu.com Web: ru.fujitsu.comhttp://ts.fujitsu.com/ Company details: ts.fujitsu.com/imprinthttp://ts.fujitsu.com/imprint.html This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation. Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.
From: Alexander Griesser [mailto:AGriesser@anexia-it.com] Sent: Thursday, October 22, 2015 5:50 PM To: Borzenkov, Andrei; toasters@teaparty.net Subject: AW: Used Replacement disk not recognized
Bingo:
riddler> disk show -v DISK OWNER POOL SERIAL NUMBER HOME CHKSUM ------------ ------------- ----- ------------- ------------- ------ 0d.01.13 w-nas-04 (142255463) Pool0 J838WSRL w-nas-04 (142255463) Block
„w-nas-04“ is none of my filers – so I need to remove the ownership here now, I guess.
riddler*> disk remove_ownership 0d.01.13 -f disk remove_ownership: Disk 0d.01.13 is not owned by this node. riddler*> disk reassign -o w-nas-04 -n riddler disk reassign: Can only be run in maintenance mode or during takeover in advanced mode riddler*>
So, I guess I have to – again – boot into maintenance mode or is there a trick which would allow me to remove the ownership without having to shut everything down again?
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Von: andrei.borzenkov@ts.fujitsu.commailto:andrei.borzenkov@ts.fujitsu.com [mailto:andrei.borzenkov@ts.fujitsu.com] Gesendet: Donnerstag, 22. Oktober 2015 16:47 An: Alexander Griesser <AGriesser@anexia-it.commailto:AGriesser@anexia-it.com>; toasters@teaparty.netmailto:toasters@teaparty.net Betreff: RE: Used Replacement disk not recognized
Does it appear in “disk show -v”?
--- With best regards
Andrei Borzenkov Senior system engineer FTS WEMEAI RUC RU SC TMS FOS [cid:image001.gif@01CBF835.B3FEDA90] FUJITSU Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation Tel.: +7 495 730 62 20 ( reception) Mob.: +7 916 678 7208 Fax: +7 495 730 62 14 E-mail: Andrei.Borzenkov@ts.fujitsu.commailto:Andrei.Borzenkov@ts.fujitsu.com Web: ru.fujitsu.comhttp://ts.fujitsu.com/ Company details: ts.fujitsu.com/imprinthttp://ts.fujitsu.com/imprint.html This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation. Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.
From: Alexander Griesser [mailto:AGriesser@anexia-it.com] Sent: Thursday, October 22, 2015 5:44 PM To: Borzenkov, Andrei; toasters@teaparty.netmailto:toasters@teaparty.net Subject: AW: Used Replacement disk not recognized
Hi Andrei,
the last command states „Disk 0d.01.13 does not exist“, I don’t think that this is cool… The problem is that even after the reboot, the disk does not show up anywhere except for sysconfig or storage show disk.
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Von: andrei.borzenkov@ts.fujitsu.commailto:andrei.borzenkov@ts.fujitsu.com [mailto:andrei.borzenkov@ts.fujitsu.com] Gesendet: Donnerstag, 22. Oktober 2015 16:41 An: Alexander Griesser <AGriesser@anexia-it.commailto:AGriesser@anexia-it.com>; toasters@teaparty.netmailto:toasters@teaparty.net Betreff: RE: Used Replacement disk not recognized
Firmware is not older or newer – those two disks are from different manufacturers so they have different firmware; both are using the latest versions publically available.
KB https://kb.netapp.com/support/index?page=content&id=2010827&locale=e... implies that Drive spinning up may be seen with X302 and is otherwise harmless, although there are so many different variants of X302 it is hard to guess whether it applies in your case. I would suggest open the case.
Last command you pasted suggests that everything is OK now; or what is remaining problem?
--- With best regards
Andrei Borzenkov Senior system engineer FTS WEMEAI RUC RU SC TMS FOS [cid:image001.gif@01CBF835.B3FEDA90] FUJITSU Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation Tel.: +7 495 730 62 20 ( reception) Mob.: +7 916 678 7208 Fax: +7 495 730 62 14 E-mail: Andrei.Borzenkov@ts.fujitsu.commailto:Andrei.Borzenkov@ts.fujitsu.com Web: ru.fujitsu.comhttp://ts.fujitsu.com/ Company details: ts.fujitsu.com/imprinthttp://ts.fujitsu.com/imprint.html This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation. Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.
From: toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Alexander Griesser Sent: Thursday, October 22, 2015 5:06 PM To: toasters@teaparty.netmailto:toasters@teaparty.net Subject: Used Replacement disk not recognized
Hey everybody,
I just had the pleasure to try to replace a faulty disk in a filer which is out-of-service with a „new“ one from after market. Of course, this didn’t work out as easy as it should, so maybe some could help me get this disk going.
Here are the facts: FAS2040 NetApp Release 8.1.4P8 7-Mode
The disk at 0d.01.13 was broken, so I pulled it a few days ago – now that the replacment disk has arrived, I tried to plug it in and this is what happens:
Thu Oct 22 14:30:10 CEST [riddler:scsi.cmd.notReadyCondition:notice]: Disk device 0d.01.13: Device returns not yet ready: CDB 0x1b: Sense Data SCSI:not ready - Drive spinning up (0x2 - 0x4 0x1 0x28)(2).
The interesting thing, I can see the disk by using `storage show disk`:
riddler> storage show disk DISK SHELF BAY SERIAL VENDOR MODEL REV --------------------- --------- ---------------- -------- ---------- ---- 0d.01.13 1 13 J838WSRL NETAPP X302_HJUPI NA04
But I cannot see it in disk show.
riddler> disk show 0d.01.13 DISK OWNER POOL SERIAL NUMBER HOME ------------ ------------- ----- ------------- -------------
Sysconfig –v tells me, that there is a newer firmware on the „new“ disk:
01.13: NETAPP X302_HJUPI01TSSM NA04 847.5GB (1953525168 512B/sect) 01.14: NETAPP X302_WVULC01TSSA NA03 847.5GB (1953525168 512B/sect)
So I first tried to downgrade firmware using:
disk_fw_update –R 0d.01.13
but that did not really work out, no error message, but just no change on the firmware of the disk – also nothing in the logs. So I read somewhere that this might be due to a bad/incompatible label on the disk and found out that the only way to get rid of the label is to boot into maintenance mode (thus having a downtime of parts of the HA-pair :-/) and recreate the label there.
So I booted into maintenance mode and have run the following command:
*> label makespare 0d.01.13 ********************************************************* Label editing is highly dangerous and should be attempted ONLY by NetApp Customer Support Engineers. Misuse can cause ALL data to be permanently lost. ********************************************************* 'Label save': Error while trying to save Label 1 on disk 0d.03.22: error 15. Label save: 1 of 35 disks could not save Label 1. 'Label save': Error while trying to save Label 2 on disk 0d.03.22: error 15. Label save: 1 of 35 disks could not save Label 2. The current set of labels on the disks have been saved. You can use the 'label restore' command to revert to these labels, the 'label keep' command to erase these labels, or 'label save' to preserve the current set of labels. ********************************************************* *** Run WAFL recovery tools (wafliron or wafliron_oc) after this. ***
Disk 0d.01.13 does not exist.
The last line suggests, that the disk still does not get recognized properly – and additionally, it (for whatever reason) claims to not be able to write the label 1 on disk 0d.03.22, although this is there and online:
riddler> disk show 0d.03.22 DISK OWNER POOL SERIAL NUMBER HOME ------------ ------------- ----- ------------- ------------- 0d.03.22 riddler (142248200) Pool0 N002XRML riddler (142248200)
I’ve stopped my expiriments at this point and am very thankful for fresh ideas :)
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Many thanks!
riddler> vol status -s
Pool1 spare disks (empty)
Pool0 spare disks
RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks) --------- ------ ------------- ---- ---- ---- ----- -------------- -------------- Spare disks for block checksum spare 0d.01.13 0d 1 13 SA:B 0 BSAS 7200 847555/1735794176 847884/1736466816 spare 0d.03.22 0d 3 22 SA:B 0 BSAS 7200 847555/1735794176 847884/1736466816
I'm up and running :)
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Von: andrei.borzenkov@ts.fujitsu.com [mailto:andrei.borzenkov@ts.fujitsu.com] Gesendet: Donnerstag, 22. Oktober 2015 16:52 An: Alexander Griesser AGriesser@anexia-it.com; toasters@teaparty.net Betreff: RE: Used Replacement disk not recognized
disk assign 0d.01.13 -f
--- With best regards
Andrei Borzenkov Senior system engineer FTS WEMEAI RUC RU SC TMS FOS [cid:image001.gif@01CBF835.B3FEDA90] FUJITSU Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation Tel.: +7 495 730 62 20 ( reception) Mob.: +7 916 678 7208 Fax: +7 495 730 62 14 E-mail: Andrei.Borzenkov@ts.fujitsu.commailto:Andrei.Borzenkov@ts.fujitsu.com Web: ru.fujitsu.comhttp://ts.fujitsu.com/ Company details: ts.fujitsu.com/imprinthttp://ts.fujitsu.com/imprint.html This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation. Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.
From: Alexander Griesser [mailto:AGriesser@anexia-it.com] Sent: Thursday, October 22, 2015 5:50 PM To: Borzenkov, Andrei; toasters@teaparty.netmailto:toasters@teaparty.net Subject: AW: Used Replacement disk not recognized
Bingo:
riddler> disk show -v DISK OWNER POOL SERIAL NUMBER HOME CHKSUM ------------ ------------- ----- ------------- ------------- ------ 0d.01.13 w-nas-04 (142255463) Pool0 J838WSRL w-nas-04 (142255463) Block
"w-nas-04" is none of my filers - so I need to remove the ownership here now, I guess.
riddler*> disk remove_ownership 0d.01.13 -f disk remove_ownership: Disk 0d.01.13 is not owned by this node. riddler*> disk reassign -o w-nas-04 -n riddler disk reassign: Can only be run in maintenance mode or during takeover in advanced mode riddler*>
So, I guess I have to - again - boot into maintenance mode or is there a trick which would allow me to remove the ownership without having to shut everything down again?
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Von: andrei.borzenkov@ts.fujitsu.commailto:andrei.borzenkov@ts.fujitsu.com [mailto:andrei.borzenkov@ts.fujitsu.com] Gesendet: Donnerstag, 22. Oktober 2015 16:47 An: Alexander Griesser <AGriesser@anexia-it.commailto:AGriesser@anexia-it.com>; toasters@teaparty.netmailto:toasters@teaparty.net Betreff: RE: Used Replacement disk not recognized
Does it appear in "disk show -v"?
--- With best regards
Andrei Borzenkov Senior system engineer FTS WEMEAI RUC RU SC TMS FOS [cid:image001.gif@01CBF835.B3FEDA90] FUJITSU Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation Tel.: +7 495 730 62 20 ( reception) Mob.: +7 916 678 7208 Fax: +7 495 730 62 14 E-mail: Andrei.Borzenkov@ts.fujitsu.commailto:Andrei.Borzenkov@ts.fujitsu.com Web: ru.fujitsu.comhttp://ts.fujitsu.com/ Company details: ts.fujitsu.com/imprinthttp://ts.fujitsu.com/imprint.html This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation. Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.
From: Alexander Griesser [mailto:AGriesser@anexia-it.com] Sent: Thursday, October 22, 2015 5:44 PM To: Borzenkov, Andrei; toasters@teaparty.netmailto:toasters@teaparty.net Subject: AW: Used Replacement disk not recognized
Hi Andrei,
the last command states "Disk 0d.01.13 does not exist", I don't think that this is cool... The problem is that even after the reboot, the disk does not show up anywhere except for sysconfig or storage show disk.
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Von: andrei.borzenkov@ts.fujitsu.commailto:andrei.borzenkov@ts.fujitsu.com [mailto:andrei.borzenkov@ts.fujitsu.com] Gesendet: Donnerstag, 22. Oktober 2015 16:41 An: Alexander Griesser <AGriesser@anexia-it.commailto:AGriesser@anexia-it.com>; toasters@teaparty.netmailto:toasters@teaparty.net Betreff: RE: Used Replacement disk not recognized
Firmware is not older or newer - those two disks are from different manufacturers so they have different firmware; both are using the latest versions publically available.
KB https://kb.netapp.com/support/index?page=content&id=2010827&locale=e... implies that Drive spinning up may be seen with X302 and is otherwise harmless, although there are so many different variants of X302 it is hard to guess whether it applies in your case. I would suggest open the case.
Last command you pasted suggests that everything is OK now; or what is remaining problem?
--- With best regards
Andrei Borzenkov Senior system engineer FTS WEMEAI RUC RU SC TMS FOS [cid:image001.gif@01CBF835.B3FEDA90] FUJITSU Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation Tel.: +7 495 730 62 20 ( reception) Mob.: +7 916 678 7208 Fax: +7 495 730 62 14 E-mail: Andrei.Borzenkov@ts.fujitsu.commailto:Andrei.Borzenkov@ts.fujitsu.com Web: ru.fujitsu.comhttp://ts.fujitsu.com/ Company details: ts.fujitsu.com/imprinthttp://ts.fujitsu.com/imprint.html This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation. Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.
From: toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Alexander Griesser Sent: Thursday, October 22, 2015 5:06 PM To: toasters@teaparty.netmailto:toasters@teaparty.net Subject: Used Replacement disk not recognized
Hey everybody,
I just had the pleasure to try to replace a faulty disk in a filer which is out-of-service with a "new" one from after market. Of course, this didn't work out as easy as it should, so maybe some could help me get this disk going.
Here are the facts: FAS2040 NetApp Release 8.1.4P8 7-Mode
The disk at 0d.01.13 was broken, so I pulled it a few days ago - now that the replacment disk has arrived, I tried to plug it in and this is what happens:
Thu Oct 22 14:30:10 CEST [riddler:scsi.cmd.notReadyCondition:notice]: Disk device 0d.01.13: Device returns not yet ready: CDB 0x1b: Sense Data SCSI:not ready - Drive spinning up (0x2 - 0x4 0x1 0x28)(2).
The interesting thing, I can see the disk by using `storage show disk`:
riddler> storage show disk DISK SHELF BAY SERIAL VENDOR MODEL REV --------------------- --------- ---------------- -------- ---------- ---- 0d.01.13 1 13 J838WSRL NETAPP X302_HJUPI NA04
But I cannot see it in disk show.
riddler> disk show 0d.01.13 DISK OWNER POOL SERIAL NUMBER HOME ------------ ------------- ----- ------------- -------------
Sysconfig -v tells me, that there is a newer firmware on the "new" disk:
01.13: NETAPP X302_HJUPI01TSSM NA04 847.5GB (1953525168 512B/sect) 01.14: NETAPP X302_WVULC01TSSA NA03 847.5GB (1953525168 512B/sect)
So I first tried to downgrade firmware using:
disk_fw_update -R 0d.01.13
but that did not really work out, no error message, but just no change on the firmware of the disk - also nothing in the logs. So I read somewhere that this might be due to a bad/incompatible label on the disk and found out that the only way to get rid of the label is to boot into maintenance mode (thus having a downtime of parts of the HA-pair :-/) and recreate the label there.
So I booted into maintenance mode and have run the following command:
*> label makespare 0d.01.13 ********************************************************* Label editing is highly dangerous and should be attempted ONLY by NetApp Customer Support Engineers. Misuse can cause ALL data to be permanently lost. ********************************************************* 'Label save': Error while trying to save Label 1 on disk 0d.03.22: error 15. Label save: 1 of 35 disks could not save Label 1. 'Label save': Error while trying to save Label 2 on disk 0d.03.22: error 15. Label save: 1 of 35 disks could not save Label 2. The current set of labels on the disks have been saved. You can use the 'label restore' command to revert to these labels, the 'label keep' command to erase these labels, or 'label save' to preserve the current set of labels. ********************************************************* *** Run WAFL recovery tools (wafliron or wafliron_oc) after this. ***
Disk 0d.01.13 does not exist.
The last line suggests, that the disk still does not get recognized properly - and additionally, it (for whatever reason) claims to not be able to write the label 1 on disk 0d.03.22, although this is there and online:
riddler> disk show 0d.03.22 DISK OWNER POOL SERIAL NUMBER HOME ------------ ------------- ----- ------------- ------------- 0d.03.22 riddler (142248200) Pool0 N002XRML riddler (142248200)
I've stopped my expiriments at this point and am very thankful for fresh ideas :)
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Alright, but now I do have a label issue I guess :)
An even older filer (FAS2040, OnTap 8.0.1P2 7-Mode) is writing this to the log when I insert one of the refurbished disks:
Fri Oct 23 20:24:52 CEST [megatron: raid.disk.missing:info]: Disk 0d.01.5 Shelf 1 Bay 5 [NETAPP X410_HVIPC288A15 NA00] S/N [JXV0TL2J] is missing from the system Fri Oct 23 20:26:52 CEST [megatron: diskown.changingOwner:info]: changing ownership for disk 0d.01.5 (S/N JXXDSRMJ) from unowned (ID 4294967295) to megatron (ID 142227074) Fri Oct 23 20:26:52 CEST [megatron: raid.assim.disk.nolabels:error]: Disk 0d.01.5 Shelf 1 Bay 5 [NETAPP X410_HVIPC288A15 NA02] S/N [JXXDSRMJ] has no valid labels. It will be taken out of service to prevent possible data loss. Fri Oct 23 20:26:52 CEST [megatron: raid.config.disk.bad.label:error]: Disk 0d.01.5 Shelf 1 Bay 5 [NETAPP X410_HVIPC288A15 NA02] S/N [JXXDSRMJ] has bad label. Fri Oct 23 20:26:52 CEST [megatron: callhome.dsk.label:CRITICAL]: Call home for DISK BAD LABEL Fri Oct 23 20:27:00 CEST [megatron: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0d, shelf 1, bay 5, failed.
I guess this means that I will have to erase the label here, correct?
aggr status -r Broken disks
RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks) --------- ------ ------------- ---- ---- ---- ----- -------------- -------------- bad label 0d.01.5 0d 1 5 SA:B 0 SAS 15000 272000/557056000 274845/562884296
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Von: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] Im Auftrag von Alexander Griesser Gesendet: Donnerstag, 22. Oktober 2015 16:55 An: andrei.borzenkov@ts.fujitsu.com; toasters@teaparty.net Betreff: AW: Used Replacement disk not recognized
Many thanks!
riddler> vol status -s
Pool1 spare disks (empty)
Pool0 spare disks
RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks) --------- ------ ------------- ---- ---- ---- ----- -------------- -------------- Spare disks for block checksum spare 0d.01.13 0d 1 13 SA:B 0 BSAS 7200 847555/1735794176 847884/1736466816 spare 0d.03.22 0d 3 22 SA:B 0 BSAS 7200 847555/1735794176 847884/1736466816
I'm up and running :)
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Von: andrei.borzenkov@ts.fujitsu.commailto:andrei.borzenkov@ts.fujitsu.com [mailto:andrei.borzenkov@ts.fujitsu.com] Gesendet: Donnerstag, 22. Oktober 2015 16:52 An: Alexander Griesser <AGriesser@anexia-it.commailto:AGriesser@anexia-it.com>; toasters@teaparty.netmailto:toasters@teaparty.net Betreff: RE: Used Replacement disk not recognized
disk assign 0d.01.13 -f
--- With best regards
Andrei Borzenkov Senior system engineer FTS WEMEAI RUC RU SC TMS FOS [cid:image001.gif@01CBF835.B3FEDA90] FUJITSU Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation Tel.: +7 495 730 62 20 ( reception) Mob.: +7 916 678 7208 Fax: +7 495 730 62 14 E-mail: Andrei.Borzenkov@ts.fujitsu.commailto:Andrei.Borzenkov@ts.fujitsu.com Web: ru.fujitsu.comhttp://ts.fujitsu.com/ Company details: ts.fujitsu.com/imprinthttp://ts.fujitsu.com/imprint.html This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation. Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.
From: Alexander Griesser [mailto:AGriesser@anexia-it.com] Sent: Thursday, October 22, 2015 5:50 PM To: Borzenkov, Andrei; toasters@teaparty.netmailto:toasters@teaparty.net Subject: AW: Used Replacement disk not recognized
Bingo:
riddler> disk show -v DISK OWNER POOL SERIAL NUMBER HOME CHKSUM ------------ ------------- ----- ------------- ------------- ------ 0d.01.13 w-nas-04 (142255463) Pool0 J838WSRL w-nas-04 (142255463) Block
"w-nas-04" is none of my filers - so I need to remove the ownership here now, I guess.
riddler*> disk remove_ownership 0d.01.13 -f disk remove_ownership: Disk 0d.01.13 is not owned by this node. riddler*> disk reassign -o w-nas-04 -n riddler disk reassign: Can only be run in maintenance mode or during takeover in advanced mode riddler*>
So, I guess I have to - again - boot into maintenance mode or is there a trick which would allow me to remove the ownership without having to shut everything down again?
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Von: andrei.borzenkov@ts.fujitsu.commailto:andrei.borzenkov@ts.fujitsu.com [mailto:andrei.borzenkov@ts.fujitsu.com] Gesendet: Donnerstag, 22. Oktober 2015 16:47 An: Alexander Griesser <AGriesser@anexia-it.commailto:AGriesser@anexia-it.com>; toasters@teaparty.netmailto:toasters@teaparty.net Betreff: RE: Used Replacement disk not recognized
Does it appear in "disk show -v"?
--- With best regards
Andrei Borzenkov Senior system engineer FTS WEMEAI RUC RU SC TMS FOS [cid:image001.gif@01CBF835.B3FEDA90] FUJITSU Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation Tel.: +7 495 730 62 20 ( reception) Mob.: +7 916 678 7208 Fax: +7 495 730 62 14 E-mail: Andrei.Borzenkov@ts.fujitsu.commailto:Andrei.Borzenkov@ts.fujitsu.com Web: ru.fujitsu.comhttp://ts.fujitsu.com/ Company details: ts.fujitsu.com/imprinthttp://ts.fujitsu.com/imprint.html This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation. Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.
From: Alexander Griesser [mailto:AGriesser@anexia-it.com] Sent: Thursday, October 22, 2015 5:44 PM To: Borzenkov, Andrei; toasters@teaparty.netmailto:toasters@teaparty.net Subject: AW: Used Replacement disk not recognized
Hi Andrei,
the last command states "Disk 0d.01.13 does not exist", I don't think that this is cool... The problem is that even after the reboot, the disk does not show up anywhere except for sysconfig or storage show disk.
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Von: andrei.borzenkov@ts.fujitsu.commailto:andrei.borzenkov@ts.fujitsu.com [mailto:andrei.borzenkov@ts.fujitsu.com] Gesendet: Donnerstag, 22. Oktober 2015 16:41 An: Alexander Griesser <AGriesser@anexia-it.commailto:AGriesser@anexia-it.com>; toasters@teaparty.netmailto:toasters@teaparty.net Betreff: RE: Used Replacement disk not recognized
Firmware is not older or newer - those two disks are from different manufacturers so they have different firmware; both are using the latest versions publically available.
KB https://kb.netapp.com/support/index?page=content&id=2010827&locale=e... implies that Drive spinning up may be seen with X302 and is otherwise harmless, although there are so many different variants of X302 it is hard to guess whether it applies in your case. I would suggest open the case.
Last command you pasted suggests that everything is OK now; or what is remaining problem?
--- With best regards
Andrei Borzenkov Senior system engineer FTS WEMEAI RUC RU SC TMS FOS [cid:image001.gif@01CBF835.B3FEDA90] FUJITSU Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation Tel.: +7 495 730 62 20 ( reception) Mob.: +7 916 678 7208 Fax: +7 495 730 62 14 E-mail: Andrei.Borzenkov@ts.fujitsu.commailto:Andrei.Borzenkov@ts.fujitsu.com Web: ru.fujitsu.comhttp://ts.fujitsu.com/ Company details: ts.fujitsu.com/imprinthttp://ts.fujitsu.com/imprint.html This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation. Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.
From: toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Alexander Griesser Sent: Thursday, October 22, 2015 5:06 PM To: toasters@teaparty.netmailto:toasters@teaparty.net Subject: Used Replacement disk not recognized
Hey everybody,
I just had the pleasure to try to replace a faulty disk in a filer which is out-of-service with a "new" one from after market. Of course, this didn't work out as easy as it should, so maybe some could help me get this disk going.
Here are the facts: FAS2040 NetApp Release 8.1.4P8 7-Mode
The disk at 0d.01.13 was broken, so I pulled it a few days ago - now that the replacment disk has arrived, I tried to plug it in and this is what happens:
Thu Oct 22 14:30:10 CEST [riddler:scsi.cmd.notReadyCondition:notice]: Disk device 0d.01.13: Device returns not yet ready: CDB 0x1b: Sense Data SCSI:not ready - Drive spinning up (0x2 - 0x4 0x1 0x28)(2).
The interesting thing, I can see the disk by using `storage show disk`:
riddler> storage show disk DISK SHELF BAY SERIAL VENDOR MODEL REV --------------------- --------- ---------------- -------- ---------- ---- 0d.01.13 1 13 J838WSRL NETAPP X302_HJUPI NA04
But I cannot see it in disk show.
riddler> disk show 0d.01.13 DISK OWNER POOL SERIAL NUMBER HOME ------------ ------------- ----- ------------- -------------
Sysconfig -v tells me, that there is a newer firmware on the "new" disk:
01.13: NETAPP X302_HJUPI01TSSM NA04 847.5GB (1953525168 512B/sect) 01.14: NETAPP X302_WVULC01TSSA NA03 847.5GB (1953525168 512B/sect)
So I first tried to downgrade firmware using:
disk_fw_update -R 0d.01.13
but that did not really work out, no error message, but just no change on the firmware of the disk - also nothing in the logs. So I read somewhere that this might be due to a bad/incompatible label on the disk and found out that the only way to get rid of the label is to boot into maintenance mode (thus having a downtime of parts of the HA-pair :-/) and recreate the label there.
So I booted into maintenance mode and have run the following command:
*> label makespare 0d.01.13 ********************************************************* Label editing is highly dangerous and should be attempted ONLY by NetApp Customer Support Engineers. Misuse can cause ALL data to be permanently lost. ********************************************************* 'Label save': Error while trying to save Label 1 on disk 0d.03.22: error 15. Label save: 1 of 35 disks could not save Label 1. 'Label save': Error while trying to save Label 2 on disk 0d.03.22: error 15. Label save: 1 of 35 disks could not save Label 2. The current set of labels on the disks have been saved. You can use the 'label restore' command to revert to these labels, the 'label keep' command to erase these labels, or 'label save' to preserve the current set of labels. ********************************************************* *** Run WAFL recovery tools (wafliron or wafliron_oc) after this. ***
Disk 0d.01.13 does not exist.
The last line suggests, that the disk still does not get recognized properly - and additionally, it (for whatever reason) claims to not be able to write the label 1 on disk 0d.03.22, although this is there and online:
riddler> disk show 0d.03.22 DISK OWNER POOL SERIAL NUMBER HOME ------------ ------------- ----- ------------- ------------- 0d.03.22 riddler (142248200) Pool0 N002XRML riddler (142248200)
I've stopped my expiriments at this point and am very thankful for fresh ideas :)
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Alex,
If you were running 8.02 there is a nice fix. ☹
8.01 you are out of luck. https://kb.netapp.com/support/index?page=content&id=2011366&locale=e...
Have you got a newer system you can plug it into to clear it?
Regards
Duncan Cummings NetApp Specialist Interactive Pty Ltd Telephone +61 7 3323 0800 Facsimile +61 7 3323 0899 Mobile +61 403 383 050 www.interactive.com.auhttp://www.interactive.com.au
-------Confidentiality & Legal Privilege------------- "This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately. Please delete this message from your computer. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you." From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Alexander Griesser Sent: Saturday, 24 October 2015 4:31 AM To: toasters@teaparty.net Subject: AW: Used Replacement disk not recognized
Alright, but now I do have a label issue I guess ☺
An even older filer (FAS2040, OnTap 8.0.1P2 7-Mode) is writing this to the log when I insert one of the refurbished disks:
Fri Oct 23 20:24:52 CEST [megatron: raid.disk.missing:info]: Disk 0d.01.5 Shelf 1 Bay 5 [NETAPP X410_HVIPC288A15 NA00] S/N [JXV0TL2J] is missing from the system Fri Oct 23 20:26:52 CEST [megatron: diskown.changingOwner:info]: changing ownership for disk 0d.01.5 (S/N JXXDSRMJ) from unowned (ID 4294967295) to megatron (ID 142227074) Fri Oct 23 20:26:52 CEST [megatron: raid.assim.disk.nolabels:error]: Disk 0d.01.5 Shelf 1 Bay 5 [NETAPP X410_HVIPC288A15 NA02] S/N [JXXDSRMJ] has no valid labels. It will be taken out of service to prevent possible data loss. Fri Oct 23 20:26:52 CEST [megatron: raid.config.disk.bad.label:error]: Disk 0d.01.5 Shelf 1 Bay 5 [NETAPP X410_HVIPC288A15 NA02] S/N [JXXDSRMJ] has bad label. Fri Oct 23 20:26:52 CEST [megatron: callhome.dsk.label:CRITICAL]: Call home for DISK BAD LABEL Fri Oct 23 20:27:00 CEST [megatron: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0d, shelf 1, bay 5, failed.
I guess this means that I will have to erase the label here, correct?
aggr status -r Broken disks
RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks) --------- ------ ------------- ---- ---- ---- ----- -------------- -------------- bad label 0d.01.5 0d 1 5 SA:B 0 SAS 15000 272000/557056000 274845/562884296
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Von: toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] Im Auftrag von Alexander Griesser Gesendet: Donnerstag, 22. Oktober 2015 16:55 An: andrei.borzenkov@ts.fujitsu.commailto:andrei.borzenkov@ts.fujitsu.com; toasters@teaparty.netmailto:toasters@teaparty.net Betreff: AW: Used Replacement disk not recognized
Many thanks!
riddler> vol status -s
Pool1 spare disks (empty)
Pool0 spare disks
RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks) --------- ------ ------------- ---- ---- ---- ----- -------------- -------------- Spare disks for block checksum spare 0d.01.13 0d 1 13 SA:B 0 BSAS 7200 847555/1735794176 847884/1736466816 spare 0d.03.22 0d 3 22 SA:B 0 BSAS 7200 847555/1735794176 847884/1736466816
I’m up and running ☺
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Von: andrei.borzenkov@ts.fujitsu.commailto:andrei.borzenkov@ts.fujitsu.com [mailto:andrei.borzenkov@ts.fujitsu.com] Gesendet: Donnerstag, 22. Oktober 2015 16:52 An: Alexander Griesser <AGriesser@anexia-it.commailto:AGriesser@anexia-it.com>; toasters@teaparty.netmailto:toasters@teaparty.net Betreff: RE: Used Replacement disk not recognized
disk assign 0d.01.13 -f
--- With best regards
Andrei Borzenkov Senior system engineer FTS WEMEAI RUC RU SC TMS FOS [cid:image001.gif@01CBF835.B3FEDA90] FUJITSU Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation Tel.: +7 495 730 62 20 ( reception) Mob.: +7 916 678 7208 Fax: +7 495 730 62 14 E-mail: Andrei.Borzenkov@ts.fujitsu.commailto:Andrei.Borzenkov@ts.fujitsu.com Web: ru.fujitsu.comhttp://ts.fujitsu.com/ Company details: ts.fujitsu.com/imprinthttp://ts.fujitsu.com/imprint.html This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation. Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.
From: Alexander Griesser [mailto:AGriesser@anexia-it.com] Sent: Thursday, October 22, 2015 5:50 PM To: Borzenkov, Andrei; toasters@teaparty.netmailto:toasters@teaparty.net Subject: AW: Used Replacement disk not recognized
Bingo:
riddler> disk show -v DISK OWNER POOL SERIAL NUMBER HOME CHKSUM ------------ ------------- ----- ------------- ------------- ------ 0d.01.13 w-nas-04 (142255463) Pool0 J838WSRL w-nas-04 (142255463) Block
„w-nas-04“ is none of my filers – so I need to remove the ownership here now, I guess.
riddler*> disk remove_ownership 0d.01.13 -f disk remove_ownership: Disk 0d.01.13 is not owned by this node. riddler*> disk reassign -o w-nas-04 -n riddler disk reassign: Can only be run in maintenance mode or during takeover in advanced mode riddler*>
So, I guess I have to – again – boot into maintenance mode or is there a trick which would allow me to remove the ownership without having to shut everything down again?
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Von: andrei.borzenkov@ts.fujitsu.commailto:andrei.borzenkov@ts.fujitsu.com [mailto:andrei.borzenkov@ts.fujitsu.com] Gesendet: Donnerstag, 22. Oktober 2015 16:47 An: Alexander Griesser <AGriesser@anexia-it.commailto:AGriesser@anexia-it.com>; toasters@teaparty.netmailto:toasters@teaparty.net Betreff: RE: Used Replacement disk not recognized
Does it appear in “disk show -v”?
--- With best regards
Andrei Borzenkov Senior system engineer FTS WEMEAI RUC RU SC TMS FOS [cid:image001.gif@01CBF835.B3FEDA90] FUJITSU Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation Tel.: +7 495 730 62 20 ( reception) Mob.: +7 916 678 7208 Fax: +7 495 730 62 14 E-mail: Andrei.Borzenkov@ts.fujitsu.commailto:Andrei.Borzenkov@ts.fujitsu.com Web: ru.fujitsu.comhttp://ts.fujitsu.com/ Company details: ts.fujitsu.com/imprinthttp://ts.fujitsu.com/imprint.html This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation. Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.
From: Alexander Griesser [mailto:AGriesser@anexia-it.com] Sent: Thursday, October 22, 2015 5:44 PM To: Borzenkov, Andrei; toasters@teaparty.netmailto:toasters@teaparty.net Subject: AW: Used Replacement disk not recognized
Hi Andrei,
the last command states „Disk 0d.01.13 does not exist“, I don’t think that this is cool… The problem is that even after the reboot, the disk does not show up anywhere except for sysconfig or storage show disk.
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Von: andrei.borzenkov@ts.fujitsu.commailto:andrei.borzenkov@ts.fujitsu.com [mailto:andrei.borzenkov@ts.fujitsu.com] Gesendet: Donnerstag, 22. Oktober 2015 16:41 An: Alexander Griesser <AGriesser@anexia-it.commailto:AGriesser@anexia-it.com>; toasters@teaparty.netmailto:toasters@teaparty.net Betreff: RE: Used Replacement disk not recognized
Firmware is not older or newer – those two disks are from different manufacturers so they have different firmware; both are using the latest versions publically available.
KB https://kb.netapp.com/support/index?page=content&id=2010827&locale=e... implies that Drive spinning up may be seen with X302 and is otherwise harmless, although there are so many different variants of X302 it is hard to guess whether it applies in your case. I would suggest open the case.
Last command you pasted suggests that everything is OK now; or what is remaining problem?
--- With best regards
Andrei Borzenkov Senior system engineer FTS WEMEAI RUC RU SC TMS FOS [cid:image001.gif@01CBF835.B3FEDA90] FUJITSU Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation Tel.: +7 495 730 62 20 ( reception) Mob.: +7 916 678 7208 Fax: +7 495 730 62 14 E-mail: Andrei.Borzenkov@ts.fujitsu.commailto:Andrei.Borzenkov@ts.fujitsu.com Web: ru.fujitsu.comhttp://ts.fujitsu.com/ Company details: ts.fujitsu.com/imprinthttp://ts.fujitsu.com/imprint.html This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation. Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.
From: toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Alexander Griesser Sent: Thursday, October 22, 2015 5:06 PM To: toasters@teaparty.netmailto:toasters@teaparty.net Subject: Used Replacement disk not recognized
Hey everybody,
I just had the pleasure to try to replace a faulty disk in a filer which is out-of-service with a „new“ one from after market. Of course, this didn’t work out as easy as it should, so maybe some could help me get this disk going.
Here are the facts: FAS2040 NetApp Release 8.1.4P8 7-Mode
The disk at 0d.01.13 was broken, so I pulled it a few days ago – now that the replacment disk has arrived, I tried to plug it in and this is what happens:
Thu Oct 22 14:30:10 CEST [riddler:scsi.cmd.notReadyCondition:notice]: Disk device 0d.01.13: Device returns not yet ready: CDB 0x1b: Sense Data SCSI:not ready - Drive spinning up (0x2 - 0x4 0x1 0x28)(2).
The interesting thing, I can see the disk by using `storage show disk`:
riddler> storage show disk DISK SHELF BAY SERIAL VENDOR MODEL REV --------------------- --------- ---------------- -------- ---------- ---- 0d.01.13 1 13 J838WSRL NETAPP X302_HJUPI NA04
But I cannot see it in disk show.
riddler> disk show 0d.01.13 DISK OWNER POOL SERIAL NUMBER HOME ------------ ------------- ----- ------------- -------------
Sysconfig –v tells me, that there is a newer firmware on the „new“ disk:
01.13: NETAPP X302_HJUPI01TSSM NA04 847.5GB (1953525168 512B/sect) 01.14: NETAPP X302_WVULC01TSSA NA03 847.5GB (1953525168 512B/sect)
So I first tried to downgrade firmware using:
disk_fw_update –R 0d.01.13
but that did not really work out, no error message, but just no change on the firmware of the disk – also nothing in the logs. So I read somewhere that this might be due to a bad/incompatible label on the disk and found out that the only way to get rid of the label is to boot into maintenance mode (thus having a downtime of parts of the HA-pair :-/) and recreate the label there.
So I booted into maintenance mode and have run the following command:
*> label makespare 0d.01.13 ********************************************************* Label editing is highly dangerous and should be attempted ONLY by NetApp Customer Support Engineers. Misuse can cause ALL data to be permanently lost. ********************************************************* 'Label save': Error while trying to save Label 1 on disk 0d.03.22: error 15. Label save: 1 of 35 disks could not save Label 1. 'Label save': Error while trying to save Label 2 on disk 0d.03.22: error 15. Label save: 1 of 35 disks could not save Label 2. The current set of labels on the disks have been saved. You can use the 'label restore' command to revert to these labels, the 'label keep' command to erase these labels, or 'label save' to preserve the current set of labels. ********************************************************* *** Run WAFL recovery tools (wafliron or wafliron_oc) after this. ***
Disk 0d.01.13 does not exist.
The last line suggests, that the disk still does not get recognized properly – and additionally, it (for whatever reason) claims to not be able to write the label 1 on disk 0d.03.22, although this is there and online:
riddler> disk show 0d.03.22 DISK OWNER POOL SERIAL NUMBER HOME ------------ ------------- ----- ------------- ------------- 0d.03.22 riddler (142248200) Pool0 N002XRML riddler (142248200)
I’ve stopped my expiriments at this point and am very thankful for fresh ideas ☺
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Hi Duncan,
unfortunately, I do not have a newer system with a free slot to plug the disk in there, but I’ll try to find one before I start the upgrade to 8.0.2 ☺
Thanks, will keep you posted!
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Von: Duncan Cummings [mailto:dcummings@interactive.com.au] Gesendet: Sonntag, 25. Oktober 2015 02:45 An: Alexander Griesser AGriesser@anexia-it.com; toasters@teaparty.net Betreff: RE: Used Replacement disk not recognized
Alex,
If you were running 8.02 there is a nice fix. ☹
8.01 you are out of luck. https://kb.netapp.com/support/index?page=content&id=2011366&locale=e...
Have you got a newer system you can plug it into to clear it?
Regards
Duncan Cummings NetApp Specialist Interactive Pty Ltd Telephone +61 7 3323 0800 Facsimile +61 7 3323 0899 Mobile +61 403 383 050 www.interactive.com.auhttp://www.interactive.com.au
-------Confidentiality & Legal Privilege------------- "This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately. Please delete this message from your computer. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you." From: toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Alexander Griesser Sent: Saturday, 24 October 2015 4:31 AM To: toasters@teaparty.netmailto:toasters@teaparty.net Subject: AW: Used Replacement disk not recognized
Alright, but now I do have a label issue I guess ☺
An even older filer (FAS2040, OnTap 8.0.1P2 7-Mode) is writing this to the log when I insert one of the refurbished disks:
Fri Oct 23 20:24:52 CEST [megatron: raid.disk.missing:info]: Disk 0d.01.5 Shelf 1 Bay 5 [NETAPP X410_HVIPC288A15 NA00] S/N [JXV0TL2J] is missing from the system Fri Oct 23 20:26:52 CEST [megatron: diskown.changingOwner:info]: changing ownership for disk 0d.01.5 (S/N JXXDSRMJ) from unowned (ID 4294967295) to megatron (ID 142227074) Fri Oct 23 20:26:52 CEST [megatron: raid.assim.disk.nolabels:error]: Disk 0d.01.5 Shelf 1 Bay 5 [NETAPP X410_HVIPC288A15 NA02] S/N [JXXDSRMJ] has no valid labels. It will be taken out of service to prevent possible data loss. Fri Oct 23 20:26:52 CEST [megatron: raid.config.disk.bad.label:error]: Disk 0d.01.5 Shelf 1 Bay 5 [NETAPP X410_HVIPC288A15 NA02] S/N [JXXDSRMJ] has bad label. Fri Oct 23 20:26:52 CEST [megatron: callhome.dsk.label:CRITICAL]: Call home for DISK BAD LABEL Fri Oct 23 20:27:00 CEST [megatron: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0d, shelf 1, bay 5, failed.
I guess this means that I will have to erase the label here, correct?
aggr status -r Broken disks
RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks) --------- ------ ------------- ---- ---- ---- ----- -------------- -------------- bad label 0d.01.5 0d 1 5 SA:B 0 SAS 15000 272000/557056000 274845/562884296
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Von: toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] Im Auftrag von Alexander Griesser Gesendet: Donnerstag, 22. Oktober 2015 16:55 An: andrei.borzenkov@ts.fujitsu.commailto:andrei.borzenkov@ts.fujitsu.com; toasters@teaparty.netmailto:toasters@teaparty.net Betreff: AW: Used Replacement disk not recognized
Many thanks!
riddler> vol status -s
Pool1 spare disks (empty)
Pool0 spare disks
RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks) --------- ------ ------------- ---- ---- ---- ----- -------------- -------------- Spare disks for block checksum spare 0d.01.13 0d 1 13 SA:B 0 BSAS 7200 847555/1735794176 847884/1736466816 spare 0d.03.22 0d 3 22 SA:B 0 BSAS 7200 847555/1735794176 847884/1736466816
I’m up and running ☺
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Von: andrei.borzenkov@ts.fujitsu.commailto:andrei.borzenkov@ts.fujitsu.com [mailto:andrei.borzenkov@ts.fujitsu.com] Gesendet: Donnerstag, 22. Oktober 2015 16:52 An: Alexander Griesser <AGriesser@anexia-it.commailto:AGriesser@anexia-it.com>; toasters@teaparty.netmailto:toasters@teaparty.net Betreff: RE: Used Replacement disk not recognized
disk assign 0d.01.13 -f
--- With best regards
Andrei Borzenkov Senior system engineer FTS WEMEAI RUC RU SC TMS FOS [cid:image001.gif@01CBF835.B3FEDA90] FUJITSU Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation Tel.: +7 495 730 62 20 ( reception) Mob.: +7 916 678 7208 Fax: +7 495 730 62 14 E-mail: Andrei.Borzenkov@ts.fujitsu.commailto:Andrei.Borzenkov@ts.fujitsu.com Web: ru.fujitsu.comhttp://ts.fujitsu.com/ Company details: ts.fujitsu.com/imprinthttp://ts.fujitsu.com/imprint.html This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation. Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.
From: Alexander Griesser [mailto:AGriesser@anexia-it.com] Sent: Thursday, October 22, 2015 5:50 PM To: Borzenkov, Andrei; toasters@teaparty.netmailto:toasters@teaparty.net Subject: AW: Used Replacement disk not recognized
Bingo:
riddler> disk show -v DISK OWNER POOL SERIAL NUMBER HOME CHKSUM ------------ ------------- ----- ------------- ------------- ------ 0d.01.13 w-nas-04 (142255463) Pool0 J838WSRL w-nas-04 (142255463) Block
„w-nas-04“ is none of my filers – so I need to remove the ownership here now, I guess.
riddler*> disk remove_ownership 0d.01.13 -f disk remove_ownership: Disk 0d.01.13 is not owned by this node. riddler*> disk reassign -o w-nas-04 -n riddler disk reassign: Can only be run in maintenance mode or during takeover in advanced mode riddler*>
So, I guess I have to – again – boot into maintenance mode or is there a trick which would allow me to remove the ownership without having to shut everything down again?
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Von: andrei.borzenkov@ts.fujitsu.commailto:andrei.borzenkov@ts.fujitsu.com [mailto:andrei.borzenkov@ts.fujitsu.com] Gesendet: Donnerstag, 22. Oktober 2015 16:47 An: Alexander Griesser <AGriesser@anexia-it.commailto:AGriesser@anexia-it.com>; toasters@teaparty.netmailto:toasters@teaparty.net Betreff: RE: Used Replacement disk not recognized
Does it appear in “disk show -v”?
--- With best regards
Andrei Borzenkov Senior system engineer FTS WEMEAI RUC RU SC TMS FOS [cid:image001.gif@01CBF835.B3FEDA90] FUJITSU Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation Tel.: +7 495 730 62 20 ( reception) Mob.: +7 916 678 7208 Fax: +7 495 730 62 14 E-mail: Andrei.Borzenkov@ts.fujitsu.commailto:Andrei.Borzenkov@ts.fujitsu.com Web: ru.fujitsu.comhttp://ts.fujitsu.com/ Company details: ts.fujitsu.com/imprinthttp://ts.fujitsu.com/imprint.html This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation. Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.
From: Alexander Griesser [mailto:AGriesser@anexia-it.com] Sent: Thursday, October 22, 2015 5:44 PM To: Borzenkov, Andrei; toasters@teaparty.netmailto:toasters@teaparty.net Subject: AW: Used Replacement disk not recognized
Hi Andrei,
the last command states „Disk 0d.01.13 does not exist“, I don’t think that this is cool… The problem is that even after the reboot, the disk does not show up anywhere except for sysconfig or storage show disk.
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Von: andrei.borzenkov@ts.fujitsu.commailto:andrei.borzenkov@ts.fujitsu.com [mailto:andrei.borzenkov@ts.fujitsu.com] Gesendet: Donnerstag, 22. Oktober 2015 16:41 An: Alexander Griesser <AGriesser@anexia-it.commailto:AGriesser@anexia-it.com>; toasters@teaparty.netmailto:toasters@teaparty.net Betreff: RE: Used Replacement disk not recognized
Firmware is not older or newer – those two disks are from different manufacturers so they have different firmware; both are using the latest versions publically available.
KB https://kb.netapp.com/support/index?page=content&id=2010827&locale=e... implies that Drive spinning up may be seen with X302 and is otherwise harmless, although there are so many different variants of X302 it is hard to guess whether it applies in your case. I would suggest open the case.
Last command you pasted suggests that everything is OK now; or what is remaining problem?
--- With best regards
Andrei Borzenkov Senior system engineer FTS WEMEAI RUC RU SC TMS FOS [cid:image001.gif@01CBF835.B3FEDA90] FUJITSU Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation Tel.: +7 495 730 62 20 ( reception) Mob.: +7 916 678 7208 Fax: +7 495 730 62 14 E-mail: Andrei.Borzenkov@ts.fujitsu.commailto:Andrei.Borzenkov@ts.fujitsu.com Web: ru.fujitsu.comhttp://ts.fujitsu.com/ Company details: ts.fujitsu.com/imprinthttp://ts.fujitsu.com/imprint.html This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation. Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.
From: toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Alexander Griesser Sent: Thursday, October 22, 2015 5:06 PM To: toasters@teaparty.netmailto:toasters@teaparty.net Subject: Used Replacement disk not recognized
Hey everybody,
I just had the pleasure to try to replace a faulty disk in a filer which is out-of-service with a „new“ one from after market. Of course, this didn’t work out as easy as it should, so maybe some could help me get this disk going.
Here are the facts: FAS2040 NetApp Release 8.1.4P8 7-Mode
The disk at 0d.01.13 was broken, so I pulled it a few days ago – now that the replacment disk has arrived, I tried to plug it in and this is what happens:
Thu Oct 22 14:30:10 CEST [riddler:scsi.cmd.notReadyCondition:notice]: Disk device 0d.01.13: Device returns not yet ready: CDB 0x1b: Sense Data SCSI:not ready - Drive spinning up (0x2 - 0x4 0x1 0x28)(2).
The interesting thing, I can see the disk by using `storage show disk`:
riddler> storage show disk DISK SHELF BAY SERIAL VENDOR MODEL REV --------------------- --------- ---------------- -------- ---------- ---- 0d.01.13 1 13 J838WSRL NETAPP X302_HJUPI NA04
But I cannot see it in disk show.
riddler> disk show 0d.01.13 DISK OWNER POOL SERIAL NUMBER HOME ------------ ------------- ----- ------------- -------------
Sysconfig –v tells me, that there is a newer firmware on the „new“ disk:
01.13: NETAPP X302_HJUPI01TSSM NA04 847.5GB (1953525168 512B/sect) 01.14: NETAPP X302_WVULC01TSSA NA03 847.5GB (1953525168 512B/sect)
So I first tried to downgrade firmware using:
disk_fw_update –R 0d.01.13
but that did not really work out, no error message, but just no change on the firmware of the disk – also nothing in the logs. So I read somewhere that this might be due to a bad/incompatible label on the disk and found out that the only way to get rid of the label is to boot into maintenance mode (thus having a downtime of parts of the HA-pair :-/) and recreate the label there.
So I booted into maintenance mode and have run the following command:
*> label makespare 0d.01.13 ********************************************************* Label editing is highly dangerous and should be attempted ONLY by NetApp Customer Support Engineers. Misuse can cause ALL data to be permanently lost. ********************************************************* 'Label save': Error while trying to save Label 1 on disk 0d.03.22: error 15. Label save: 1 of 35 disks could not save Label 1. 'Label save': Error while trying to save Label 2 on disk 0d.03.22: error 15. Label save: 1 of 35 disks could not save Label 2. The current set of labels on the disks have been saved. You can use the 'label restore' command to revert to these labels, the 'label keep' command to erase these labels, or 'label save' to preserve the current set of labels. ********************************************************* *** Run WAFL recovery tools (wafliron or wafliron_oc) after this. ***
Disk 0d.01.13 does not exist.
The last line suggests, that the disk still does not get recognized properly – and additionally, it (for whatever reason) claims to not be able to write the label 1 on disk 0d.03.22, although this is there and online:
riddler> disk show 0d.03.22 DISK OWNER POOL SERIAL NUMBER HOME ------------ ------------- ----- ------------- ------------- 0d.03.22 riddler (142248200) Pool0 N002XRML riddler (142248200)
I’ve stopped my expiriments at this point and am very thankful for fresh ideas ☺
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
make sure to add the -s unowned, because the disk is owned, but not by the local node.
On Thu, Oct 22, 2015 at 10:51 AM, andrei.borzenkov@ts.fujitsu.com < andrei.borzenkov@ts.fujitsu.com> wrote:
disk assign 0d.01.13 -f
With best regards
*Andre**i** Borzenkov*
Senior system engineer
FTS WEMEAI RUC RU SC TMS FOS
[image: cid:image001.gif@01CBF835.B3FEDA90]
*FUJITSU*
Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation
Tel.: +7 495 730 62 20 ( reception)
Mob.: +7 916 678 7208
Fax: +7 495 730 62 14
E-mail: Andrei.Borzenkov@ts.fujitsu.com
Web: ru.fujitsu.com http://ts.fujitsu.com/
Company details: ts.fujitsu.com/imprint http://ts.fujitsu.com/imprint.html
This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation.
Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.
*From:* Alexander Griesser [mailto:AGriesser@anexia-it.com] *Sent:* Thursday, October 22, 2015 5:50 PM *To:* Borzenkov, Andrei; toasters@teaparty.net *Subject:* AW: Used Replacement disk not recognized
Bingo:
riddler> disk show -v
DISK OWNER POOL SERIAL NUMBER HOME CHKSUM
0d.01.13 w-nas-04 (142255463) Pool0 J838WSRL w-nas-04 (142255463) Block
„w-nas-04“ is none of my filers – so I need to remove the ownership here now, I guess.
riddler*> disk remove_ownership 0d.01.13 -f
disk remove_ownership: Disk 0d.01.13 is not owned by this node.
riddler*> disk reassign -o w-nas-04 -n riddler
disk reassign: Can only be run in maintenance mode or during takeover in advanced mode
riddler*>
So, I guess I have to – again – boot into maintenance mode or is there a trick which would allow me to remove the ownership without having to shut everything down again?
Best,
*Alexander Griesser*
Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
*Von:* andrei.borzenkov@ts.fujitsu.com [ mailto:andrei.borzenkov@ts.fujitsu.com andrei.borzenkov@ts.fujitsu.com] *Gesendet:* Donnerstag, 22. Oktober 2015 16:47 *An:* Alexander Griesser AGriesser@anexia-it.com; toasters@teaparty.net *Betreff:* RE: Used Replacement disk not recognized
Does it appear in “disk show -v”?
With best regards
*Andre**i** Borzenkov*
Senior system engineer
FTS WEMEAI RUC RU SC TMS FOS
[image: cid:image001.gif@01CBF835.B3FEDA90]
*FUJITSU*
Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation
Tel.: +7 495 730 62 20 ( reception)
Mob.: +7 916 678 7208
Fax: +7 495 730 62 14
E-mail: Andrei.Borzenkov@ts.fujitsu.com
Web: ru.fujitsu.com http://ts.fujitsu.com/
Company details: ts.fujitsu.com/imprint http://ts.fujitsu.com/imprint.html
This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation.
Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.
*From:* Alexander Griesser [mailto:AGriesser@anexia-it.com AGriesser@anexia-it.com] *Sent:* Thursday, October 22, 2015 5:44 PM *To:* Borzenkov, Andrei; toasters@teaparty.net *Subject:* AW: Used Replacement disk not recognized
Hi Andrei,
the last command states „Disk 0d.01.13 does not exist“, I don’t think that this is cool…
The problem is that even after the reboot, the disk does not show up anywhere except for sysconfig or storage show disk.
Best,
*Alexander Griesser*
Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
*Von:* andrei.borzenkov@ts.fujitsu.com [ mailto:andrei.borzenkov@ts.fujitsu.com andrei.borzenkov@ts.fujitsu.com] *Gesendet:* Donnerstag, 22. Oktober 2015 16:41 *An:* Alexander Griesser AGriesser@anexia-it.com; toasters@teaparty.net *Betreff:* RE: Used Replacement disk not recognized
Firmware is not older or newer – those two disks are from different manufacturers so they have different firmware; both are using the latest versions publically available.
KB https://kb.netapp.com/support/index?page=content&id=2010827&locale=e... implies that Drive spinning up may be seen with X302 and is otherwise harmless, although there are so many different variants of X302 it is hard to guess whether it applies in your case. I would suggest open the case.
Last command you pasted suggests that everything is OK now; or what is remaining problem?
With best regards
*Andre**i** Borzenkov*
Senior system engineer
FTS WEMEAI RUC RU SC TMS FOS
[image: cid:image001.gif@01CBF835.B3FEDA90]
*FUJITSU*
Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation
Tel.: +7 495 730 62 20 ( reception)
Mob.: +7 916 678 7208
Fax: +7 495 730 62 14
E-mail: Andrei.Borzenkov@ts.fujitsu.com
Web: ru.fujitsu.com http://ts.fujitsu.com/
Company details: ts.fujitsu.com/imprint http://ts.fujitsu.com/imprint.html
This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation.
Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.
*From:* toasters-bounces@teaparty.net [ mailto:toasters-bounces@teaparty.net toasters-bounces@teaparty.net] *On Behalf Of *Alexander Griesser *Sent:* Thursday, October 22, 2015 5:06 PM *To:* toasters@teaparty.net *Subject:* Used Replacement disk not recognized
Hey everybody,
I just had the pleasure to try to replace a faulty disk in a filer which is out-of-service with a „new“ one from after market.
Of course, this didn’t work out as easy as it should, so maybe some could help me get this disk going.
Here are the facts:
FAS2040
NetApp Release 8.1.4P8 7-Mode
The disk at 0d.01.13 was broken, so I pulled it a few days ago – now that the replacment disk has arrived, I tried to plug it in and this is what happens:
Thu Oct 22 14:30:10 CEST [riddler:scsi.cmd.notReadyCondition:notice]: Disk device 0d.01.13: Device returns not yet ready: CDB 0x1b: Sense Data SCSI:not ready - Drive spinning up (0x2 - 0x4 0x1 0x28)(2).
The interesting thing, I can see the disk by using `storage show disk`:
riddler> storage show disk
DISK SHELF BAY SERIAL VENDOR MODEL REV
0d.01.13 1 13 J838WSRL NETAPP X302_HJUPI NA04
But I cannot see it in disk show.
riddler> disk show 0d.01.13
DISK OWNER POOL SERIAL NUMBER HOME
Sysconfig –v tells me, that there is a newer firmware on the „new“ disk:
01.13: NETAPP X302_HJUPI01TSSM NA04 847.5GB (1953525168512B/sect)
01.14: NETAPP X302_WVULC01TSSA NA03 847.5GB (1953525168512B/sect)
So I first tried to downgrade firmware using:
disk_fw_update –R 0d.01.13
but that did not really work out, no error message, but just no change on the firmware of the disk – also nothing in the logs.
So I read somewhere that this might be due to a bad/incompatible label on the disk and found out that the only way to get rid of the label is to boot into maintenance mode (thus having a downtime of parts of the HA-pair :-/) and recreate the label there.
So I booted into maintenance mode and have run the following command:
*> label makespare 0d.01.13
Label editing is highly dangerous and should be attempted
ONLY by NetApp Customer Support Engineers.
Misuse can cause ALL data to be permanently lost.
'Label save': Error while trying to save Label 1 on disk 0d.03.22: error 15.
Label save: 1 of 35 disks could not save Label 1.
'Label save': Error while trying to save Label 2 on disk 0d.03.22: error 15.
Label save: 1 of 35 disks could not save Label 2.
The current set of labels on the disks have been saved. You can
use the 'label restore' command to revert to these labels, the
'label keep' command to erase these labels, or 'label save' to
preserve the current set of labels.
*** Run WAFL recovery tools (wafliron or wafliron_oc) after this.
Disk 0d.01.13 does not exist.
The last line suggests, that the disk still does not get recognized properly – and additionally, it (for whatever reason) claims to not be able to write the label 1 on disk 0d.03.22, although this is there and online:
riddler> disk show 0d.03.22
DISK OWNER POOL SERIAL NUMBER HOME
0d.03.22 riddler (142248200) Pool0 N002XRML riddler (142248200)
I’ve stopped my expiriments at this point and am very thankful for fresh ideas J
Best,
*Alexander Griesser*
Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
This command will fix you without having to take the system down.
disk assign 0d.01.13 -s unowned -f
Then, assign the disk.
On Thu, Oct 22, 2015 at 10:49 AM, Alexander Griesser < AGriesser@anexia-it.com> wrote:
Bingo:
riddler> disk show -v
DISK OWNER POOL SERIAL NUMBER HOME CHKSUM
0d.01.13 w-nas-04 (142255463) Pool0 J838WSRL w-nas-04 (142255463) Block
„w-nas-04“ is none of my filers – so I need to remove the ownership here now, I guess.
riddler*> disk remove_ownership 0d.01.13 -f
disk remove_ownership: Disk 0d.01.13 is not owned by this node.
riddler*> disk reassign -o w-nas-04 -n riddler
disk reassign: Can only be run in maintenance mode or during takeover in advanced mode
riddler*>
So, I guess I have to – again – boot into maintenance mode or is there a trick which would allow me to remove the ownership without having to shut everything down again?
Best,
*Alexander Griesser*
Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
*Von:* andrei.borzenkov@ts.fujitsu.com [mailto: andrei.borzenkov@ts.fujitsu.com] *Gesendet:* Donnerstag, 22. Oktober 2015 16:47 *An:* Alexander Griesser AGriesser@anexia-it.com; toasters@teaparty.net *Betreff:* RE: Used Replacement disk not recognized
Does it appear in “disk show -v”?
With best regards
*Andre**i** Borzenkov*
Senior system engineer
FTS WEMEAI RUC RU SC TMS FOS
[image: cid:image001.gif@01CBF835.B3FEDA90]
*FUJITSU*
Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation
Tel.: +7 495 730 62 20 ( reception)
Mob.: +7 916 678 7208
Fax: +7 495 730 62 14
E-mail: Andrei.Borzenkov@ts.fujitsu.com
Web: ru.fujitsu.com http://ts.fujitsu.com/
Company details: ts.fujitsu.com/imprint http://ts.fujitsu.com/imprint.html
This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation.
Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.
*From:* Alexander Griesser [mailto:AGriesser@anexia-it.com AGriesser@anexia-it.com] *Sent:* Thursday, October 22, 2015 5:44 PM *To:* Borzenkov, Andrei; toasters@teaparty.net *Subject:* AW: Used Replacement disk not recognized
Hi Andrei,
the last command states „Disk 0d.01.13 does not exist“, I don’t think that this is cool…
The problem is that even after the reboot, the disk does not show up anywhere except for sysconfig or storage show disk.
Best,
*Alexander Griesser*
Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
*Von:* andrei.borzenkov@ts.fujitsu.com [ mailto:andrei.borzenkov@ts.fujitsu.com andrei.borzenkov@ts.fujitsu.com] *Gesendet:* Donnerstag, 22. Oktober 2015 16:41 *An:* Alexander Griesser AGriesser@anexia-it.com; toasters@teaparty.net *Betreff:* RE: Used Replacement disk not recognized
Firmware is not older or newer – those two disks are from different manufacturers so they have different firmware; both are using the latest versions publically available.
KB https://kb.netapp.com/support/index?page=content&id=2010827&locale=e... implies that Drive spinning up may be seen with X302 and is otherwise harmless, although there are so many different variants of X302 it is hard to guess whether it applies in your case. I would suggest open the case.
Last command you pasted suggests that everything is OK now; or what is remaining problem?
With best regards
*Andre**i** Borzenkov*
Senior system engineer
FTS WEMEAI RUC RU SC TMS FOS
[image: cid:image001.gif@01CBF835.B3FEDA90]
*FUJITSU*
Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation
Tel.: +7 495 730 62 20 ( reception)
Mob.: +7 916 678 7208
Fax: +7 495 730 62 14
E-mail: Andrei.Borzenkov@ts.fujitsu.com
Web: ru.fujitsu.com http://ts.fujitsu.com/
Company details: ts.fujitsu.com/imprint http://ts.fujitsu.com/imprint.html
This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation.
Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.
*From:* toasters-bounces@teaparty.net [ mailto:toasters-bounces@teaparty.net toasters-bounces@teaparty.net] *On Behalf Of *Alexander Griesser *Sent:* Thursday, October 22, 2015 5:06 PM *To:* toasters@teaparty.net *Subject:* Used Replacement disk not recognized
Hey everybody,
I just had the pleasure to try to replace a faulty disk in a filer which is out-of-service with a „new“ one from after market.
Of course, this didn’t work out as easy as it should, so maybe some could help me get this disk going.
Here are the facts:
FAS2040
NetApp Release 8.1.4P8 7-Mode
The disk at 0d.01.13 was broken, so I pulled it a few days ago – now that the replacment disk has arrived, I tried to plug it in and this is what happens:
Thu Oct 22 14:30:10 CEST [riddler:scsi.cmd.notReadyCondition:notice]: Disk device 0d.01.13: Device returns not yet ready: CDB 0x1b: Sense Data SCSI:not ready - Drive spinning up (0x2 - 0x4 0x1 0x28)(2).
The interesting thing, I can see the disk by using `storage show disk`:
riddler> storage show disk
DISK SHELF BAY SERIAL VENDOR MODEL REV
0d.01.13 1 13 J838WSRL NETAPP X302_HJUPI NA04
But I cannot see it in disk show.
riddler> disk show 0d.01.13
DISK OWNER POOL SERIAL NUMBER HOME
Sysconfig –v tells me, that there is a newer firmware on the „new“ disk:
01.13: NETAPP X302_HJUPI01TSSM NA04 847.5GB (1953525168512B/sect)
01.14: NETAPP X302_WVULC01TSSA NA03 847.5GB (1953525168512B/sect)
So I first tried to downgrade firmware using:
disk_fw_update –R 0d.01.13
but that did not really work out, no error message, but just no change on the firmware of the disk – also nothing in the logs.
So I read somewhere that this might be due to a bad/incompatible label on the disk and found out that the only way to get rid of the label is to boot into maintenance mode (thus having a downtime of parts of the HA-pair :-/) and recreate the label there.
So I booted into maintenance mode and have run the following command:
*> label makespare 0d.01.13
Label editing is highly dangerous and should be attempted
ONLY by NetApp Customer Support Engineers.
Misuse can cause ALL data to be permanently lost.
'Label save': Error while trying to save Label 1 on disk 0d.03.22: error 15.
Label save: 1 of 35 disks could not save Label 1.
'Label save': Error while trying to save Label 2 on disk 0d.03.22: error 15.
Label save: 1 of 35 disks could not save Label 2.
The current set of labels on the disks have been saved. You can
use the 'label restore' command to revert to these labels, the
'label keep' command to erase these labels, or 'label save' to
preserve the current set of labels.
*** Run WAFL recovery tools (wafliron or wafliron_oc) after this.
Disk 0d.01.13 does not exist.
The last line suggests, that the disk still does not get recognized properly – and additionally, it (for whatever reason) claims to not be able to write the label 1 on disk 0d.03.22, although this is there and online:
riddler> disk show 0d.03.22
DISK OWNER POOL SERIAL NUMBER HOME
0d.03.22 riddler (142248200) Pool0 N002XRML riddler (142248200)
I’ve stopped my expiriments at this point and am very thankful for fresh ideas J
Best,
*Alexander Griesser*
Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.com
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
-
Alexander Griesser -
andrei.borzenkov＠ts.fujitsu.com -
Brad Thompson -
Duncan Cummings -
Tim McCarthy