Alright, thanks – I’ll go for the reseating then since this seems to be the most reliable solution which also shouldn’t have any other impact to the system.

Best,

Alexander Griesser Head of Systems Operations

ANEXIA Internetdienstleistungs GmbH

E-Mail: ag@anexia.atmailto:ag@anexia.at Web: http://www.anexia.athttp://www.anexia.at/

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

Von: tmac [mailto:tmacmd@gmail.com] Gesendet: Samstag, 09. Mai 2015 14:15 An: Borzenkov, Andrei Cc: Alexander Griesser; Clark, André M.; toasters@teaparty.net Betreff: Re: AW: ACP - missing IOMs

The other method that seems to work every time that will work for you is a cluster takeover and giveback.

I doubt rebooting the IOM will fix it.

The two methods I have used that seem to work every time:

1. Pull the IOM for 4 minutes and re-insert (more details earlier in thread) 2. Takeover/Giveback.

The short pull of the IOM was unreliable. Sometimes it worked, sometimes it did not and if it did not, it would usually cause other IOMS to misbehave for some reason.

--tmac

Tim McCarthy Principal Consultant

On Sat, May 9, 2015 at 5:59 AM, Borzenkov, Andrei <andrei.borzenkov@ts.fujitsu.commailto:andrei.borzenkov@ts.fujitsu.com> wrote: ACP is not necessary for FW update. There is also separate ACP firmware - here I am not sure whether it is update over SAS or Ethernet.

There is also possibility to reboot IOM remotely, but you need to ask support how to do it.

Отправлено с iPhone

9 мая 2015 г., в 11:25, Alexander Griesser <ag@anexia.atmailto:ag@anexia.at> написал(а): Thanks everyone,

since the filer is a few hundred kilometers away ATM I was hoping for a remote fix ☺ System is MPHA of course and I verified that with Config Advisor as well as on the CLI, so that shouldn’t be a problem.

Does it make sense to open a Case to make them aware of the fact? Could that also be a firmware issue on the IOMs or ACP that might be fixed by applying a new firmware? I’m not sure how firmware updates are applied to the IOMs – is that happening through the SAS cables or is ACP necessary for that? My idea was to start a firmware update on the shelves to see if that fixes the issue, since a firmware update also reboots the IOMs one at a time – but if ACP connection is necessary for the FW update to complete I’m out of luck here and will have to dispatch remote hands.

Best,

Alexander Griesser Head of Systems Operations

ANEXIA Internetdienstleistungs GmbH

E-Mail: ag@anexia.atmailto:ag@anexia.at Web: http://www.anexia.athttp://www.anexia.at/

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

Von: Clark, André M. [mailto:Andre.Clark@redeight.com] Gesendet: Freitag, 08. Mai 2015 23:18 An: tmac; Alexander Griesser Cc: toasters@teaparty.netmailto:toasters@teaparty.net Betreff: RE: ACP - missing IOMs

I’ve seen this issues on numerous installs now, including brand new systems. This is something that we should not have to be doing all the time. There has to be something else going on here.

Regards,

André M. Clarkmailto:Andre.Clark@redeight.com | Sr. Solutions Architect | 917.388.8236tel:917.388.8236 Tell me I will forget... Show me I may remember... Involve me I WILL UNDERSTAND!!! Start by doing what's necessary, then what's possible, and suddenly you are doing the impossible!!! Red8, An Insight Investments Company www.redeight.comhttp://www.redeight.com/

From: toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of tmac Sent: Friday, 8 May, 2015 15:12 To: Alexander Griesser Cc: toasters@teaparty.netmailto:toasters@teaparty.net Subject: Re: ACP - missing IOMs

Make sure your Multipath HA is working.

go to shelf 23 and disconnect the B IOM from the backplane (no need to fully remove, just electrical disconnect) go to shelf 15 and disconnect the B IOM

WAIT AT LEAST 4 MINUTES!!!

Insert them back in to the shelves.

Re-check after 15-20 minutes. That should do it. If there are still some missing, pull and wait at least 4 minutes. Make sure to only do one side (a or b) at a time!

--tmac

Tim McCarthy Principal Consultant

On Fri, May 8, 2015 at 3:02 PM, Alexander Griesser <ag@anexia.atmailto:ag@anexia.at> wrote: Hey there,

I was adding two shelves to one of my FAS8020s (cDOT, if that matters) recently and one IOM on each of the newly added shelves is missing in `storage show acp` or `acpadmin list_all` directly on the nodeshell.

smokescreen> acpadmin list_all IP MAC Reset Last Contact Protocol Assigner Shelf Current Inband IOM Address Address Cnt (seconds ago) Version ACPA ID S/N State ID Type ---------------------------------------------------------------------------------------------------------------------- 192.168.0.1 00:a0:98:71:a0:00 000 163 1.2.2.8 536878789 SHFMS1414001364 0x5 0a.14.B IOM6 192.168.0.73 00:a0:98:72:d0:48 000 309 1.2.2.8 536878789 SHJMS1416001181 0x5 2a.21.B IOM6 192.168.0.101 00:a0:98:72:d0:64 000 373 1.2.2.8 536878789 SHJMS1416001174 0x5 2a.22.B IOM6 192.168.0.105 00:a0:98:72:d0:68 000 407 1.2.2.8 536878620 SHJMS1416001174 0x5 0b.22.A IOM6 192.168.0.191 00:a0:98:71:a0:be 000 154 1.2.2.8 536878789 SHFMS1414001364 0x5 2b.14.A IOM6 192.168.0.225 00:a0:98:83:14:e1 000 252 1.2.2.8 536878789 SHJHU1515000294 0x4 ------- IOM6 192.168.1.29 00:a0:98:72:d1:1c 000 335 1.2.2.8 536878789 SHJMS1416001181 0x5 0b.21.A IOM6 192.168.2.103 00:a0:98:83:12:67 000 251 1.2.2.8 536878789 SHJHU1515000294 0x5 0b.23.A IOM6 192.168.2.173 00:a0:98:72:ba:ac 000 392 1.2.2.8 536878789 SHFMS1417000556 0x5 2b.11.A IOM6 192.168.3.7 00:a0:98:72:cb:06 000 337 1.2.2.8 536878620 SHFMS1417000567 0x5 0a.12.B IOM6 192.168.3.13 00:a0:98:81:87:0c 000 384 1.2.2.8 536878789 SHFHU1513000260 0x4 ------- IOM6 192.168.3.19 00:a0:98:81:87:12 000 285 1.2.2.8 536878620 SHFHU1513000260 0x5 2b.15.A IOM6 192.168.3.21 00:a0:98:72:cb:14 000 407 1.2.2.8 536878789 SHFMS1417000567 0x5 2b.12.A IOM6 192.168.3.25 00:a0:98:72:2f:18 000 305 1.2.2.8 536878789 SHFMS1416000221 0x5 2b.13.A IOM6 192.168.3.95 00:a0:98:72:cb:5e 000 471 1.2.2.8 536878620 SHFMS1417000556 0x5 0a.11.B IOM6 192.168.3.131 00:a0:98:72:2f:82 000 386 1.2.2.8 536878789 SHFMS1416000221 0x5 0a.13.B IOM6

smokescreen> storage show acp

Alternate Control Path: Enabled Ethernet Interface: e0P ACP Status: Active ACP IP Address: 192.168.2.215 ACP Subnet: 192.168.0.0 ACP Netmask: 255.255.252.0 ACP Connectivity Status: Additional Connectivity ACP Partner Connectivity Status: Additional Connectivity

Shelf Module Reset Cnt IP Address FW Version Module Type Status ----------------- ------------ --------------- ------------ ------------ ------- 0b.23.A 000 192.168.2.103 02.08 IOM6 active 0b.21.A 000 192.168.1.29 02.08 IOM6 active 0b.22.A 000 192.168.0.105 02.08 IOM6 active 2b.15.A 000 192.168.3.19 02.08 IOM6 active 2b.12.A 000 192.168.3.21 02.08 IOM6 active 2b.14.A 000 192.168.0.191 02.08 IOM6 active 2b.11.A 000 192.168.2.173 02.08 IOM6 active 2b.13.A 000 192.168.3.25 02.08 IOM6 active 2a.21.B 000 192.168.0.73 02.08 IOM6 active 2a.22.B 000 192.168.0.101 02.08 IOM6 active 0a.14.B 000 192.168.0.1 02.08 IOM6 active 0a.13.B 000 192.168.3.131 02.08 IOM6 active 0a.12.B 000 192.168.3.7 02.08 IOM6 active 0a.11.B 000 192.168.3.95 02.08 IOM6 active NA 000 192.168.0.225 02.08 IOM6 inactive (no in-band connectivity) NA 000 192.168.3.13 02.08 IOM6 inactive (no in-band connectivity)

I’ve done some research and found that usually a TO/GB helps in this situation, but I’m – of course – trying to avoid that. So, any ideas how I can try to make those pesky IOMs respond on the ACP? I tried to disable and re-enable ACP, but that didn’t help (maybe I did it wrong, have never fully disabled ACP on a cDOT system), so any pointers are welcome.

Also, config advisor suggests to install missing shelf firmware, so I was wondering if it is safe to install the shelf firmware while an IOM is missing on the ACP?

Thanks,

Alexander Griesser Head of Systems Operations

ANEXIA Internetdienstleistungs GmbH

E-Mail: ag@anexia.atmailto:ag@anexia.at Web: http://www.anexia.athttp://www.anexia.at/

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

You are welcome!

--tmac

*Tim McCarthy* *Principal Consultant*

On Wed, May 13, 2015 at 10:22 AM, Momonth momonth@gmail.com wrote:

Hi,

On Sat, May 9, 2015 at 2:15 PM, tmac tmacmd@gmail.com wrote:

...

The two methods I have used that seem to work every time:

1. Pull the IOM for 4 minutes and re-insert (more details earlier in

thread)

That works like a charm, just fixed a bunch of DS4246's like that. Thanks!

...

2. Takeover/Giveback.

The short pull of the IOM was unreliable. Sometimes it worked, sometimes

it

...

did not and if it did not, it would usually cause other IOMS to misbehave for some reason.

Never tried. Sorry

Sent from Mobile Outlook

On Wed, May 13, 2015 at 7:58 AM -0700, "Brandon Kitchen" bkitchen@datalink.com wrote:

Ever had to do this with the controller modules themselves in a 2554?

Have a 2554 where one controller doesn't show the other's IOM6E via ACP.

-BK

On May 13, 2015, at 10:23 AM, tmac tmacmd@gmail.com wrote:

You are welcome!

--tmac

Tim McCarthy Principal Consultant

On Wed, May 13, 2015 at 10:22 AM, Momonth momonth@gmail.com wrote:

Hi,

On Sat, May 9, 2015 at 2:15 PM, tmac tmacmd@gmail.com wrote:

The two methods I have used that seem to work every time:

1. Pull the IOM for 4 minutes and re-insert (more details earlier in thread)

That works like a charm, just fixed a bunch of DS4246's like that. Thanks!

2. Takeover/Giveback.

The short pull of the IOM was unreliable. Sometimes it worked, sometimes it

did not and if it did not, it would usually cause other IOMS to misbehave

for some reason.

_______________________________________________

Toasters mailing list

Toasters@teaparty.net

http://www.teaparty.net/mailman/listinfo/toasters

I can confirm that this same process works with the IOM6E in the 2554 controller as well.

Did a takeover, pulled the down controller module out for 4 minutes, and ACP was happy when it came back up.

-BK

On May 14, 2015, at 1:50 AM, Momonth momonth@gmail.com wrote:

No, never worked with those, sorry.

On Wed, May 13, 2015 at 4:58 PM, Brandon Kitchen bkitchen@datalink.com wrote:

...

Ever had to do this with the controller modules themselves in a 2554?

Have a 2554 where one controller doesn't show the other's IOM6E via ACP.

-BK

Tried that first. Didn't work in my case.

It's possible I may not have waited long enough though.

On May 14, 2015, at 5:12 PM, Tim McCarthy <tmacmd@gmail.commailto:tmacmd@gmail.com> wrote:

If you do a takeover wait a few minutes and give back there should be no need to pull the iom modules. The proper reset happens at that time

Sent from Mobile Outlookhttp://taps.io/outlookmobile

On Thu, May 14, 2015 at 1:59 PM -0700, "Brandon Kitchen" <bkitchen@datalink.commailto:bkitchen@datalink.com> wrote:

I can confirm that this same process works with the IOM6E in the 2554 controller as well.

Did a takeover, pulled the down controller module out for 4 minutes, and ACP was happy when it came back up.

-BK

On May 14, 2015, at 1:50 AM, Momonth wrote:

No, never worked with those, sorry.

On Wed, May 13, 2015 at 4:58 PM, Brandon Kitchen wrote:

...

Ever had to do this with the controller modules themselves in a 2554?

Have a 2554 where one controller doesn't show the other's IOM6E via ACP.

-BK