Make sure your Multipath HA is working.

go to shelf 23 and disconnect the B IOM from the backplane (no need to fully remove, just electrical disconnect)
go to shelf 15 and disconnect the B IOM

WAIT AT LEAST 4 MINUTES!!!

Insert them back in to the shelves.

Re-check after 15-20 minutes. 
That should do it. If there are still some missing, pull and wait at least 4 minutes. Make sure to only do one side (a or b) at a time!


--tmac

Tim McCarthy
Principal Consultant



On Fri, May 8, 2015 at 3:02 PM, Alexander Griesser <ag@anexia.at> wrote:

Hey there,

 

I was adding two shelves to one of my FAS8020s (cDOT, if that matters) recently and one IOM on each of the newly added shelves is missing in `storage show acp` or `acpadmin list_all` directly on the nodeshell.

 

smokescreen> acpadmin list_all

IP              MAC                Reset  Last Contact  Protocol   Assigner    Shelf             Current Inband   IOM

Address         Address            Cnt    (seconds ago) Version    ACPA ID     S/N               State   ID       Type

----------------------------------------------------------------------------------------------------------------------

192.168.0.1     00:a0:98:71:a0:00  000      163         1.2.2.8    536878789   SHFMS1414001364   0x5    0a.14.B  IOM6

192.168.0.73    00:a0:98:72:d0:48  000      309         1.2.2.8    536878789   SHJMS1416001181   0x5    2a.21.B  IOM6

192.168.0.101   00:a0:98:72:d0:64  000      373         1.2.2.8    536878789   SHJMS1416001174   0x5    2a.22.B  IOM6

192.168.0.105   00:a0:98:72:d0:68  000      407         1.2.2.8    536878620   SHJMS1416001174   0x5    0b.22.A  IOM6

192.168.0.191   00:a0:98:71:a0:be  000      154         1.2.2.8    536878789   SHFMS1414001364   0x5    2b.14.A  IOM6

192.168.0.225   00:a0:98:83:14:e1  000      252         1.2.2.8    536878789   SHJHU1515000294   0x4    -------  IOM6

192.168.1.29    00:a0:98:72:d1:1c  000      335         1.2.2.8    536878789   SHJMS1416001181   0x5    0b.21.A  IOM6

192.168.2.103   00:a0:98:83:12:67  000      251         1.2.2.8    536878789   SHJHU1515000294   0x5    0b.23.A  IOM6

192.168.2.173   00:a0:98:72:ba:ac  000      392         1.2.2.8    536878789   SHFMS1417000556   0x5    2b.11.A  IOM6

192.168.3.7     00:a0:98:72:cb:06  000      337         1.2.2.8    536878620   SHFMS1417000567   0x5    0a.12.B  IOM6

192.168.3.13    00:a0:98:81:87:0c  000      384         1.2.2.8    536878789   SHFHU1513000260   0x4    -------  IOM6

192.168.3.19    00:a0:98:81:87:12  000      285         1.2.2.8    536878620   SHFHU1513000260   0x5    2b.15.A  IOM6

192.168.3.21    00:a0:98:72:cb:14  000      407         1.2.2.8    536878789   SHFMS1417000567   0x5    2b.12.A  IOM6

192.168.3.25    00:a0:98:72:2f:18  000      305         1.2.2.8    536878789   SHFMS1416000221   0x5    2b.13.A  IOM6

192.168.3.95    00:a0:98:72:cb:5e  000      471         1.2.2.8    536878620   SHFMS1417000556   0x5    0a.11.B  IOM6

192.168.3.131   00:a0:98:72:2f:82  000      386         1.2.2.8    536878789   SHFMS1416000221   0x5    0a.13.B  IOM6

 

smokescreen> storage show acp

 

Alternate Control Path:          Enabled

Ethernet Interface:              e0P

ACP Status:                      Active

ACP IP Address:                  192.168.2.215

ACP Subnet:                      192.168.0.0

ACP Netmask:                     255.255.252.0

ACP Connectivity Status:         Additional Connectivity

ACP Partner Connectivity Status: Additional Connectivity

 

Shelf Module      Reset Cnt    IP Address      FW Version   Module Type  Status

----------------- ------------ --------------- ------------ ------------ -------

0b.23.A           000          192.168.2.103   02.08        IOM6         active

0b.21.A           000          192.168.1.29    02.08        IOM6         active

0b.22.A           000          192.168.0.105   02.08        IOM6         active

2b.15.A           000          192.168.3.19    02.08        IOM6         active

2b.12.A           000          192.168.3.21    02.08        IOM6         active

2b.14.A           000          192.168.0.191   02.08        IOM6         active

2b.11.A           000          192.168.2.173   02.08        IOM6         active

2b.13.A           000          192.168.3.25    02.08        IOM6         active

2a.21.B           000          192.168.0.73    02.08        IOM6         active

2a.22.B           000          192.168.0.101   02.08        IOM6         active

0a.14.B           000          192.168.0.1     02.08        IOM6         active

0a.13.B           000          192.168.3.131   02.08        IOM6         active

0a.12.B           000          192.168.3.7     02.08        IOM6         active

0a.11.B           000          192.168.3.95    02.08        IOM6         active

NA               000          192.168.0.225   02.08        IOM6         inactive (no in-band connectivity)

NA               000          192.168.3.13    02.08        IOM6         inactive (no in-band connectivity)

 

I’ve done some research and found that usually a TO/GB helps in this situation, but I’m – of course – trying to avoid that.

So, any ideas how I can try to make those pesky IOMs respond on the ACP? I tried to disable and re-enable ACP, but that didn’t help (maybe I did it wrong, have never fully disabled ACP on a cDOT system), so any pointers are welcome.

 

Also, config advisor suggests to install missing shelf firmware, so I was wondering if it is safe to install the shelf firmware while an IOM is missing on the ACP?

 

Thanks,

 

Alexander Griesser

Head of Systems Operations

 

ANEXIA Internetdienstleistungs GmbH

 

E-Mail: ag@anexia.at

Web: http://www.anexia.at

 

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt

Geschäftsführer: Alexander Windbichler

Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

 


_______________________________________________
Toasters mailing list
Toasters@teaparty.net
http://www.teaparty.net/mailman/listinfo/toasters