Alright, thanks – I’ll go for the reseating then since this seems to be the most reliable solution which also shouldn’t have any other impact to the system.

Best,

Alexander Griesser

Head of Systems Operations

ANEXIA Internetdienstleistungs GmbH

E-Mail: ag@anexia.at

Web: http://www.anexia.at

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt

Geschäftsführer: Alexander Windbichler

Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

Von: tmac [mailto:tmacmd@gmail.com]
Gesendet: Samstag, 09. Mai 2015 14:15
An: Borzenkov, Andrei
Cc: Alexander Griesser; Clark, André M.; toasters@teaparty.net
Betreff: Re: AW: ACP - missing IOMs

The other method that seems to work every time that will work for you is a cluster takeover and giveback.

I doubt rebooting the IOM will fix it.

The two methods I have used that seem to work every time:

1. Pull the IOM for 4 minutes and re-insert (more details earlier in thread)

2. Takeover/Giveback.

The short pull of the IOM was unreliable. Sometimes it worked, sometimes it did not and if it did not, it would usually cause other IOMS to misbehave for some reason.

--tmac

Tim McCarthy

Principal Consultant

On Sat, May 9, 2015 at 5:59 AM, Borzenkov, Andrei <andrei.borzenkov@ts.fujitsu.com> wrote:

ACP is not necessary for FW update. There is also separate ACP firmware - here I am not sure whether it is update over SAS or Ethernet.

There is also possibility to reboot IOM remotely, but you need to ask support how to do it.

Отправлено с iPhone

9 мая 2015 г., в 11:25, Alexander Griesser <ag@anexia.at> написал(а):

Thanks everyone,

since the filer is a few hundred kilometers away ATM I was hoping for a remote fix J System is MPHA of course and I verified that with Config Advisor as well as on the CLI, so that shouldn’t be a problem.

Does it make sense to open a Case to make them aware of the fact? Could that also be a firmware issue on the IOMs or ACP that might be fixed by applying a new firmware? I’m not sure how firmware updates are applied to the IOMs – is that happening through the SAS cables or is ACP necessary for that? My idea was to start a firmware update on the shelves to see if that fixes the issue, since a firmware update also reboots the IOMs one at a time – but if ACP connection is necessary for the FW update to complete I’m out of luck here and will have to dispatch remote hands.

Best,

Alexander Griesser

Head of Systems Operations

ANEXIA Internetdienstleistungs GmbH

E-Mail: ag@anexia.at

Web: http://www.anexia.at

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt

Geschäftsführer: Alexander Windbichler

Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

Von: Clark, André M. [mailto:Andre.Clark@redeight.com]
Gesendet: Freitag, 08. Mai 2015 23:18
An: tmac; Alexander Griesser
Cc: toasters@teaparty.net
Betreff: RE: ACP - missing IOMs

I’ve seen this issues on numerous installs now, including brand new systems. This is something that we should not have to be doing all the time. There has to be something else going on here.

Regards,

André M. Clark | Sr. Solutions Architect | 917.388.8236
Tell me I will forget... Show me I may remember... Involve me I WILL UNDERSTAND!!!
Start by doing what's necessary, then what's possible, and suddenly you are doing the impossible!!!

Red8, An Insight Investments Company

www.redeight.com

From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of tmac
Sent: Friday, 8 May, 2015 15:12
To: Alexander Griesser
Cc: toasters@teaparty.net
Subject: Re: ACP - missing IOMs

Make sure your Multipath HA is working.

go to shelf 23 and disconnect the B IOM from the backplane (no need to fully remove, just electrical disconnect)

go to shelf 15 and disconnect the B IOM

WAIT AT LEAST 4 MINUTES!!!

Insert them back in to the shelves.

Re-check after 15-20 minutes.

That should do it. If there are still some missing, pull and wait at least 4 minutes. Make sure to only do one side (a or b) at a time!

--tmac

Tim McCarthy

Principal Consultant

On Fri, May 8, 2015 at 3:02 PM, Alexander Griesser <ag@anexia.at> wrote:

Hey there,

I was adding two shelves to one of my FAS8020s (cDOT, if that matters) recently and one IOM on each of the newly added shelves is missing in `storage show acp` or `acpadmin list_all` directly on the nodeshell.

smokescreen> acpadmin list_all

IP              MAC                Reset Last Contact Protocol   Assigner    Shelf             Current Inband   IOM

Address         Address            Cnt    (seconds ago) Version    ACPA ID    S/N               State   ID       Type

----------------------------------------------------------------------------------------------------------------------

192.168.0.1     00:a0:98:71:a0:00 000      163         1.2.2.8    536878789   SHFMS1414001364 0x5    0a.14.B IOM6

192.168.0.73    00:a0:98:72:d0:48 000      309         1.2.2.8    536878789   SHJMS1416001181   0x5    2a.21.B IOM6

192.168.0.101   00:a0:98:72:d0:64 000      373         1.2.2.8    536878789   SHJMS1416001174   0x5    2a.22.B IOM6

192.168.0.105   00:a0:98:72:d0:68 000      407         1.2.2.8    536878620   SHJMS1416001174   0x5    0b.22.A IOM6

192.168.0.191   00:a0:98:71:a0:be 000      154         1.2.2.8    536878789   SHFMS1414001364   0x5    2b.14.A IOM6

192.168.0.225   00:a0:98:83:14:e1 000      252         1.2.2.8    536878789   SHJHU1515000294   0x4    ------- IOM6

192.168.1.29    00:a0:98:72:d1:1c 000      335         1.2.2.8    536878789   SHJMS1416001181   0x5    0b.21.A IOM6

192.168.2.103   00:a0:98:83:12:67 000      251         1.2.2.8    536878789   SHJHU1515000294   0x5    0b.23.A IOM6

192.168.2.173   00:a0:98:72:ba:ac 000      392         1.2.2.8    536878789   SHFMS1417000556   0x5    2b.11.A IOM6

192.168.3.7     00:a0:98:72:cb:06 000      337         1.2.2.8    536878620   SHFMS1417000567   0x5    0a.12.B IOM6

192.168.3.13    00:a0:98:81:87:0c 000      384         1.2.2.8    536878789   SHFHU1513000260   0x4    ------- IOM6

192.168.3.19    00:a0:98:81:87:12 000      285         1.2.2.8    536878620 SHFHU1513000260   0x5    2b.15.A IOM6

192.168.3.21    00:a0:98:72:cb:14 000      407         1.2.2.8    536878789   SHFMS1417000567   0x5    2b.12.A IOM6

192.168.3.25    00:a0:98:72:2f:18 000      305         1.2.2.8    536878789   SHFMS1416000221   0x5    2b.13.A IOM6

192.168.3.95    00:a0:98:72:cb:5e 000      471         1.2.2.8    536878620   SHFMS1417000556   0x5    0a.11.B IOM6

192.168.3.131   00:a0:98:72:2f:82 000      386         1.2.2.8    536878789   SHFMS1416000221   0x5    0a.13.B IOM6

smokescreen> storage show acp

Alternate Control Path:          Enabled

Ethernet Interface:              e0P

ACP Status:                      Active

ACP IP Address:                  192.168.2.215

ACP Subnet:                      192.168.0.0

ACP Netmask:                     255.255.252.0

ACP Connectivity Status:         Additional Connectivity

ACP Partner Connectivity Status: Additional Connectivity

Shelf Module      Reset Cnt    IP Address      FW Version   Module Type Status

----------------- ------------ --------------- ------------ ------------ -------

0b.23.A           000          192.168.2.103   02.08        IOM6         active

0b.21.A           000          192.168.1.29    02.08        IOM6         active

0b.22.A           000          192.168.0.105   02.08        IOM6         active

2b.15.A           000          192.168.3.19    02.08        IOM6         active

2b.12.A           000          192.168.3.21    02.08        IOM6         active

2b.14.A           000          192.168.0.191   02.08        IOM6         active

2b.11.A           000          192.168.2.173   02.08        IOM6         active

2b.13.A           000          192.168.3.25    02.08        IOM6         active

2a.21.B           000          192.168.0.73    02.08        IOM6         active

2a.22.B           000          192.168.0.101   02.08        IOM6         active

0a.14.B           000          192.168.0.1     02.08        IOM6         active

0a.13.B           000          192.168.3.131   02.08        IOM6         active

0a.12.B           000          192.168.3.7     02.08        IOM6         active

0a.11.B           000          192.168.3.95    02.08        IOM6         active

NA               000          192.168.0.225   02.08        IOM6         inactive (no in-band connectivity)

NA               000          192.168.3.13    02.08        IOM6         inactive (no in-band connectivity)

I’ve done some research and found that usually a TO/GB helps in this situation, but I’m – of course – trying to avoid that.

So, any ideas how I can try to make those pesky IOMs respond on the ACP? I tried to disable and re-enable ACP, but that didn’t help (maybe I did it wrong, have never fully disabled ACP on a cDOT system), so any pointers are welcome.

Also, config advisor suggests to install missing shelf firmware, so I was wondering if it is safe to install the shelf firmware while an IOM is missing on the ACP?

Thanks,

Alexander Griesser

Head of Systems Operations

ANEXIA Internetdienstleistungs GmbH

E-Mail: ag@anexia.at

Web: http://www.anexia.at

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt

Geschäftsführer: Alexander Windbichler

Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

_______________________________________________
Toasters mailing list
Toasters@teaparty.net
http://www.teaparty.net/mailman/listinfo/toasters

_______________________________________________
Toasters mailing list
Toasters@teaparty.net
http://www.teaparty.net/mailman/listinfo/toasters