Hey there,
I was adding two shelves to one of my FAS8020s (cDOT, if that matters) recently and one IOM on each of the newly added shelves is missing in `storage show acp` or `acpadmin list_all` directly on the nodeshell.
smokescreen> acpadmin list_all IP MAC Reset Last Contact Protocol Assigner Shelf Current Inband IOM Address Address Cnt (seconds ago) Version ACPA ID S/N State ID Type ---------------------------------------------------------------------------------------------------------------------- 192.168.0.1 00:a0:98:71:a0:00 000 163 1.2.2.8 536878789 SHFMS1414001364 0x5 0a.14.B IOM6 192.168.0.73 00:a0:98:72:d0:48 000 309 1.2.2.8 536878789 SHJMS1416001181 0x5 2a.21.B IOM6 192.168.0.101 00:a0:98:72:d0:64 000 373 1.2.2.8 536878789 SHJMS1416001174 0x5 2a.22.B IOM6 192.168.0.105 00:a0:98:72:d0:68 000 407 1.2.2.8 536878620 SHJMS1416001174 0x5 0b.22.A IOM6 192.168.0.191 00:a0:98:71:a0:be 000 154 1.2.2.8 536878789 SHFMS1414001364 0x5 2b.14.A IOM6 192.168.0.225 00:a0:98:83:14:e1 000 252 1.2.2.8 536878789 SHJHU1515000294 0x4 ------- IOM6 192.168.1.29 00:a0:98:72:d1:1c 000 335 1.2.2.8 536878789 SHJMS1416001181 0x5 0b.21.A IOM6 192.168.2.103 00:a0:98:83:12:67 000 251 1.2.2.8 536878789 SHJHU1515000294 0x5 0b.23.A IOM6 192.168.2.173 00:a0:98:72:ba:ac 000 392 1.2.2.8 536878789 SHFMS1417000556 0x5 2b.11.A IOM6 192.168.3.7 00:a0:98:72:cb:06 000 337 1.2.2.8 536878620 SHFMS1417000567 0x5 0a.12.B IOM6 192.168.3.13 00:a0:98:81:87:0c 000 384 1.2.2.8 536878789 SHFHU1513000260 0x4 ------- IOM6 192.168.3.19 00:a0:98:81:87:12 000 285 1.2.2.8 536878620 SHFHU1513000260 0x5 2b.15.A IOM6 192.168.3.21 00:a0:98:72:cb:14 000 407 1.2.2.8 536878789 SHFMS1417000567 0x5 2b.12.A IOM6 192.168.3.25 00:a0:98:72:2f:18 000 305 1.2.2.8 536878789 SHFMS1416000221 0x5 2b.13.A IOM6 192.168.3.95 00:a0:98:72:cb:5e 000 471 1.2.2.8 536878620 SHFMS1417000556 0x5 0a.11.B IOM6 192.168.3.131 00:a0:98:72:2f:82 000 386 1.2.2.8 536878789 SHFMS1416000221 0x5 0a.13.B IOM6
smokescreen> storage show acp
Alternate Control Path: Enabled Ethernet Interface: e0P ACP Status: Active ACP IP Address: 192.168.2.215 ACP Subnet: 192.168.0.0 ACP Netmask: 255.255.252.0 ACP Connectivity Status: Additional Connectivity ACP Partner Connectivity Status: Additional Connectivity
Shelf Module Reset Cnt IP Address FW Version Module Type Status ----------------- ------------ --------------- ------------ ------------ ------- 0b.23.A 000 192.168.2.103 02.08 IOM6 active 0b.21.A 000 192.168.1.29 02.08 IOM6 active 0b.22.A 000 192.168.0.105 02.08 IOM6 active 2b.15.A 000 192.168.3.19 02.08 IOM6 active 2b.12.A 000 192.168.3.21 02.08 IOM6 active 2b.14.A 000 192.168.0.191 02.08 IOM6 active 2b.11.A 000 192.168.2.173 02.08 IOM6 active 2b.13.A 000 192.168.3.25 02.08 IOM6 active 2a.21.B 000 192.168.0.73 02.08 IOM6 active 2a.22.B 000 192.168.0.101 02.08 IOM6 active 0a.14.B 000 192.168.0.1 02.08 IOM6 active 0a.13.B 000 192.168.3.131 02.08 IOM6 active 0a.12.B 000 192.168.3.7 02.08 IOM6 active 0a.11.B 000 192.168.3.95 02.08 IOM6 active NA 000 192.168.0.225 02.08 IOM6 inactive (no in-band connectivity) NA 000 192.168.3.13 02.08 IOM6 inactive (no in-band connectivity)
I've done some research and found that usually a TO/GB helps in this situation, but I'm - of course - trying to avoid that. So, any ideas how I can try to make those pesky IOMs respond on the ACP? I tried to disable and re-enable ACP, but that didn't help (maybe I did it wrong, have never fully disabled ACP on a cDOT system), so any pointers are welcome.
Also, config advisor suggests to install missing shelf firmware, so I was wondering if it is safe to install the shelf firmware while an IOM is missing on the ACP?
Thanks,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: ag@anexia.atmailto:ag@anexia.at Web: http://www.anexia.athttp://www.anexia.at/
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Alexander -
Good morning, it looks like you may need to reseat/reset the IOMs associated with these ACP ports in order to get them recognized. Hopefully your storage is all configured for MP-HA so you can do this.
What you will want to do is physically pull out the IOM, wait 4 minutes and then reseat the module. It's important that you give it the full 4 minutes or the reset will not work. Provided everything is properly cabled for MP-HA there will be no outage incurred, so you will want to verify that ahead of time.
This issue has been popping up on toasters with some frequency later so you may want to visit some of the archived discussions for more details.
Thanks and good luck!
Anthony Bar tbar@berkcom.commailto:tbar@berkcom.com 650.207.5368tel:650.207.5368
On May 8, 2015, at 12:01 PM, Alexander Griesser <ag@anexia.atmailto:ag@anexia.at> wrote:
Hey there,
I was adding two shelves to one of my FAS8020s (cDOT, if that matters) recently and one IOM on each of the newly added shelves is missing in `storage show acp` or `acpadmin list_all` directly on the nodeshell.
smokescreen> acpadmin list_all IP MAC Reset Last Contact Protocol Assigner Shelf Current Inband IOM Address Address Cnt (seconds ago) Version ACPA ID S/N State ID Type ---------------------------------------------------------------------------------------------------------------------- 192.168.0.1 00:a0:98:71:a0:00 000 163 1.2.2.8 536878789 SHFMS1414001364 0x5 0a.14.B IOM6 192.168.0.73 00:a0:98:72:d0:48 000 309 1.2.2.8 536878789 SHJMS1416001181 0x5 2a.21.B IOM6 192.168.0.101 00:a0:98:72:d0:64 000 373 1.2.2.8 536878789 SHJMS1416001174 0x5 2a.22.B IOM6 192.168.0.105 00:a0:98:72:d0:68 000 407 1.2.2.8 536878620 SHJMS1416001174 0x5 0b.22.A IOM6 192.168.0.191 00:a0:98:71:a0:be 000 154 1.2.2.8 536878789 SHFMS1414001364 0x5 2b.14.A IOM6 192.168.0.225 00:a0:98:83:14:e1 000 252 1.2.2.8 536878789 SHJHU1515000294 0x4 ------- IOM6 192.168.1.29 00:a0:98:72:d1:1c 000 335 1.2.2.8 536878789 SHJMS1416001181 0x5 0b.21.A IOM6 192.168.2.103 00:a0:98:83:12:67 000 251 1.2.2.8 536878789 SHJHU1515000294 0x5 0b.23.A IOM6 192.168.2.173 00:a0:98:72:ba:ac 000 392 1.2.2.8 536878789 SHFMS1417000556 0x5 2b.11.A IOM6 192.168.3.7 00:a0:98:72:cb:06 000 337 1.2.2.8 536878620 SHFMS1417000567 0x5 0a.12.B IOM6 192.168.3.13 00:a0:98:81:87:0c 000 384 1.2.2.8 536878789 SHFHU1513000260 0x4 ------- IOM6 192.168.3.19 00:a0:98:81:87:12 000 285 1.2.2.8 536878620 SHFHU1513000260 0x5 2b.15.A IOM6 192.168.3.21 00:a0:98:72:cb:14 000 407 1.2.2.8 536878789 SHFMS1417000567 0x5 2b.12.A IOM6 192.168.3.25 00:a0:98:72:2f:18 000 305 1.2.2.8 536878789 SHFMS1416000221 0x5 2b.13.A IOM6 192.168.3.95 00:a0:98:72:cb:5e 000 471 1.2.2.8 536878620 SHFMS1417000556 0x5 0a.11.B IOM6 192.168.3.131 00:a0:98:72:2f:82 000 386 1.2.2.8 536878789 SHFMS1416000221 0x5 0a.13.B IOM6
smokescreen> storage show acp
Alternate Control Path: Enabled Ethernet Interface: e0P ACP Status: Active ACP IP Address: 192.168.2.215 ACP Subnet: 192.168.0.0 ACP Netmask: 255.255.252.0 ACP Connectivity Status: Additional Connectivity ACP Partner Connectivity Status: Additional Connectivity
Shelf Module Reset Cnt IP Address FW Version Module Type Status ----------------- ------------ --------------- ------------ ------------ ------- 0b.23.A 000 192.168.2.103 02.08 IOM6 active 0b.21.A 000 192.168.1.29 02.08 IOM6 active 0b.22.A 000 192.168.0.105 02.08 IOM6 active 2b.15.A 000 192.168.3.19 02.08 IOM6 active 2b.12.A 000 192.168.3.21 02.08 IOM6 active 2b.14.A 000 192.168.0.191 02.08 IOM6 active 2b.11.A 000 192.168.2.173 02.08 IOM6 active 2b.13.A 000 192.168.3.25 02.08 IOM6 active 2a.21.B 000 192.168.0.73 02.08 IOM6 active 2a.22.B 000 192.168.0.101 02.08 IOM6 active 0a.14.B 000 192.168.0.1 02.08 IOM6 active 0a.13.B 000 192.168.3.131 02.08 IOM6 active 0a.12.B 000 192.168.3.7 02.08 IOM6 active 0a.11.B 000 192.168.3.95 02.08 IOM6 active NA 000 192.168.0.225 02.08 IOM6 inactive (no in-band connectivity) NA 000 192.168.3.13 02.08 IOM6 inactive (no in-band connectivity)
I’ve done some research and found that usually a TO/GB helps in this situation, but I’m – of course – trying to avoid that. So, any ideas how I can try to make those pesky IOMs respond on the ACP? I tried to disable and re-enable ACP, but that didn’t help (maybe I did it wrong, have never fully disabled ACP on a cDOT system), so any pointers are welcome.
Also, config advisor suggests to install missing shelf firmware, so I was wondering if it is safe to install the shelf firmware while an IOM is missing on the ACP?
Thanks,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: ag@anexia.atmailto:ag@anexia.at Web: http://www.anexia.athttp://www.anexia.at/
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
I’ve recently run into the same situation.
The obvious safest route to take here is to open up a support case with NetApp.
However, if you can’t open a case for whatever reason, the following has worked for me in my particular case. YMMV.
I first made sure the controllers were showing MultiPath-HA.
After trying the same thing you have, plus pulling and replacing the SAS cables and ACP cables, I had to resort to physically pulling the “missing” IOM, waiting a bit—others strongly suggest waiting at least 4 minutes, though I didn’t wait more than maybe 10 seconds—and reseating the IOM before ACP recognized the IOM module.
Francis Kim | Engineer 510-644-1599 x334 | fkim@berkcom.commailto:fkim@berkcom.com
BerkCom | www.berkcom.comhttp://www.berkcom.com/ NetApp | Cisco | Supermicro | Brocade | VMware
On May 8, 2015, at 12:02 PM, Alexander Griesser <ag@anexia.atmailto:ag@anexia.at> wrote:
Hey there,
I was adding two shelves to one of my FAS8020s (cDOT, if that matters) recently and one IOM on each of the newly added shelves is missing in `storage show acp` or `acpadmin list_all` directly on the nodeshell.
smokescreen> acpadmin list_all IP MAC Reset Last Contact Protocol Assigner Shelf Current Inband IOM Address Address Cnt (seconds ago) Version ACPA ID S/N State ID Type ---------------------------------------------------------------------------------------------------------------------- 192.168.0.1 00:a0:98:71:a0:00 000 163 1.2.2.8 536878789 SHFMS1414001364 0x5 0a.14.B IOM6 192.168.0.73 00:a0:98:72:d0:48 000 309 1.2.2.8 536878789 SHJMS1416001181 0x5 2a.21.B IOM6 192.168.0.101 00:a0:98:72:d0:64 000 373 1.2.2.8 536878789 SHJMS1416001174 0x5 2a.22.B IOM6 192.168.0.105 00:a0:98:72:d0:68 000 407 1.2.2.8 536878620 SHJMS1416001174 0x5 0b.22.A IOM6 192.168.0.191 00:a0:98:71:a0:be 000 154 1.2.2.8 536878789 SHFMS1414001364 0x5 2b.14.A IOM6 192.168.0.225 00:a0:98:83:14:e1 000 252 1.2.2.8 536878789 SHJHU1515000294 0x4 ------- IOM6 192.168.1.29 00:a0:98:72:d1:1c 000 335 1.2.2.8 536878789 SHJMS1416001181 0x5 0b.21.A IOM6 192.168.2.103 00:a0:98:83:12:67 000 251 1.2.2.8 536878789 SHJHU1515000294 0x5 0b.23.A IOM6 192.168.2.173 00:a0:98:72:ba:ac 000 392 1.2.2.8 536878789 SHFMS1417000556 0x5 2b.11.A IOM6 192.168.3.7 00:a0:98:72:cb:06 000 337 1.2.2.8 536878620 SHFMS1417000567 0x5 0a.12.B IOM6 192.168.3.13 00:a0:98:81:87:0c 000 384 1.2.2.8 536878789 SHFHU1513000260 0x4 ------- IOM6 192.168.3.19 00:a0:98:81:87:12 000 285 1.2.2.8 536878620 SHFHU1513000260 0x5 2b.15.A IOM6 192.168.3.21 00:a0:98:72:cb:14 000 407 1.2.2.8 536878789 SHFMS1417000567 0x5 2b.12.A IOM6 192.168.3.25 00:a0:98:72:2f:18 000 305 1.2.2.8 536878789 SHFMS1416000221 0x5 2b.13.A IOM6 192.168.3.95 00:a0:98:72:cb:5e 000 471 1.2.2.8 536878620 SHFMS1417000556 0x5 0a.11.B IOM6 192.168.3.131 00:a0:98:72:2f:82 000 386 1.2.2.8 536878789 SHFMS1416000221 0x5 0a.13.B IOM6
smokescreen> storage show acp
Alternate Control Path: Enabled Ethernet Interface: e0P ACP Status: Active ACP IP Address: 192.168.2.215 ACP Subnet: 192.168.0.0 ACP Netmask: 255.255.252.0 ACP Connectivity Status: Additional Connectivity ACP Partner Connectivity Status: Additional Connectivity
Shelf Module Reset Cnt IP Address FW Version Module Type Status ----------------- ------------ --------------- ------------ ------------ ------- 0b.23.A 000 192.168.2.103 02.08 IOM6 active 0b.21.A 000 192.168.1.29 02.08 IOM6 active 0b.22.A 000 192.168.0.105 02.08 IOM6 active 2b.15.A 000 192.168.3.19 02.08 IOM6 active 2b.12.A 000 192.168.3.21 02.08 IOM6 active 2b.14.A 000 192.168.0.191 02.08 IOM6 active 2b.11.A 000 192.168.2.173 02.08 IOM6 active 2b.13.A 000 192.168.3.25 02.08 IOM6 active 2a.21.B 000 192.168.0.73 02.08 IOM6 active 2a.22.B 000 192.168.0.101 02.08 IOM6 active 0a.14.B 000 192.168.0.1 02.08 IOM6 active 0a.13.B 000 192.168.3.131 02.08 IOM6 active 0a.12.B 000 192.168.3.7 02.08 IOM6 active 0a.11.B 000 192.168.3.95 02.08 IOM6 active NA 000 192.168.0.225 02.08 IOM6 inactive (no in-band connectivity) NA 000 192.168.3.13 02.08 IOM6 inactive (no in-band connectivity)
I’ve done some research and found that usually a TO/GB helps in this situation, but I’m – of course – trying to avoid that. So, any ideas how I can try to make those pesky IOMs respond on the ACP? I tried to disable and re-enable ACP, but that didn’t help (maybe I did it wrong, have never fully disabled ACP on a cDOT system), so any pointers are welcome.
Also, config advisor suggests to install missing shelf firmware, so I was wondering if it is safe to install the shelf firmware while an IOM is missing on the ACP?
Thanks,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: ag@anexia.atmailto:ag@anexia.at Web: http://www.anexia.athttp://www.anexia.at/
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Make sure your Multipath HA is working.
go to shelf 23 and disconnect the B IOM from the backplane (no need to fully remove, just electrical disconnect) go to shelf 15 and disconnect the B IOM
WAIT AT LEAST 4 MINUTES!!!
Insert them back in to the shelves.
Re-check after 15-20 minutes. That should do it. If there are still some missing, pull and wait at least 4 minutes. Make sure to only do one side (a or b) at a time!
--tmac
*Tim McCarthy* *Principal Consultant*
On Fri, May 8, 2015 at 3:02 PM, Alexander Griesser ag@anexia.at wrote:
Hey there,
I was adding two shelves to one of my FAS8020s (cDOT, if that matters) recently and one IOM on each of the newly added shelves is missing in `storage show acp` or `acpadmin list_all` directly on the nodeshell.
smokescreen> acpadmin list_all
IP MAC Reset Last Contact Protocol Assigner Shelf Current Inband IOM
Address Address Cnt (seconds ago) Version ACPA ID S/N State ID Type
192.168.0.1 00:a0:98:71:a0:00 000 163 1.2.2.8 536878789 SHFMS1414001364 0x5 0a.14.B IOM6
192.168.0.73 00:a0:98:72:d0:48 000 309 1.2.2.8 536878789 SHJMS1416001181 0x5 2a.21.B IOM6
192.168.0.101 00:a0:98:72:d0:64 000 373 1.2.2.8 536878789 SHJMS1416001174 0x5 2a.22.B IOM6
192.168.0.105 00:a0:98:72:d0:68 000 407 1.2.2.8 536878620 SHJMS1416001174 0x5 0b.22.A IOM6
192.168.0.191 00:a0:98:71:a0:be 000 154 1.2.2.8 536878789 SHFMS1414001364 0x5 2b.14.A IOM6
192.168.0.225 00:a0:98:83:14:e1 000 252 1.2.2.8 536878789 SHJHU1515000294 0x4 ------- IOM6
192.168.1.29 00:a0:98:72:d1:1c 000 335 1.2.2.8 536878789 SHJMS1416001181 0x5 0b.21.A IOM6
192.168.2.103 00:a0:98:83:12:67 000 251 1.2.2.8 536878789 SHJHU1515000294 0x5 0b.23.A IOM6
192.168.2.173 00:a0:98:72:ba:ac 000 392 1.2.2.8 536878789 SHFMS1417000556 0x5 2b.11.A IOM6
192.168.3.7 00:a0:98:72:cb:06 000 337 1.2.2.8 536878620 SHFMS1417000567 0x5 0a.12.B IOM6
192.168.3.13 00:a0:98:81:87:0c 000 384 1.2.2.8 536878789 SHFHU1513000260 0x4 ------- IOM6
192.168.3.19 00:a0:98:81:87:12 000 285 1.2.2.8 536878620 SHFHU1513000260 0x5 2b.15.A IOM6
192.168.3.21 00:a0:98:72:cb:14 000 407 1.2.2.8 536878789 SHFMS1417000567 0x5 2b.12.A IOM6
192.168.3.25 00:a0:98:72:2f:18 000 305 1.2.2.8 536878789 SHFMS1416000221 0x5 2b.13.A IOM6
192.168.3.95 00:a0:98:72:cb:5e 000 471 1.2.2.8 536878620 SHFMS1417000556 0x5 0a.11.B IOM6
192.168.3.131 00:a0:98:72:2f:82 000 386 1.2.2.8 536878789 SHFMS1416000221 0x5 0a.13.B IOM6
smokescreen> storage show acp
Alternate Control Path: Enabled
Ethernet Interface: e0P
ACP Status: Active
ACP IP Address: 192.168.2.215
ACP Subnet: 192.168.0.0
ACP Netmask: 255.255.252.0
ACP Connectivity Status: Additional Connectivity
ACP Partner Connectivity Status: Additional Connectivity
Shelf Module Reset Cnt IP Address FW Version Module Type Status
0b.23.A 000 192.168.2.103 02.08 IOM6 active
0b.21.A 000 192.168.1.29 02.08 IOM6 active
0b.22.A 000 192.168.0.105 02.08 IOM6 active
2b.15.A 000 192.168.3.19 02.08 IOM6 active
2b.12.A 000 192.168.3.21 02.08 IOM6 active
2b.14.A 000 192.168.0.191 02.08 IOM6 active
2b.11.A 000 192.168.2.173 02.08 IOM6 active
2b.13.A 000 192.168.3.25 02.08 IOM6 active
2a.21.B 000 192.168.0.73 02.08 IOM6 active
2a.22.B 000 192.168.0.101 02.08 IOM6 active
0a.14.B 000 192.168.0.1 02.08 IOM6 active
0a.13.B 000 192.168.3.131 02.08 IOM6 active
0a.12.B 000 192.168.3.7 02.08 IOM6 active
0a.11.B 000 192.168.3.95 02.08 IOM6 active
NA 000 192.168.0.225 02.08 IOM6 inactive (no in-band connectivity)
NA 000 192.168.3.13 02.08 IOM6 inactive (no in-band connectivity)
I’ve done some research and found that usually a TO/GB helps in this situation, but I’m – of course – trying to avoid that.
So, any ideas how I can try to make those pesky IOMs respond on the ACP? I tried to disable and re-enable ACP, but that didn’t help (maybe I did it wrong, have never fully disabled ACP on a cDOT system), so any pointers are welcome.
Also, config advisor suggests to install missing shelf firmware, so I was wondering if it is safe to install the shelf firmware while an IOM is missing on the ACP?
Thanks,
*Alexander Griesser*
Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: ag@anexia.at
Web: http://www.anexia.at
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
-
Alexander Griesser -
Francis Kim -
tmac -
Tony Bar