We have 2 3270 filers, each using a LACP portchannel over 2 1G interfaces to a CISCO 6500.
When we enable LACP debugging, we get this.
Apr 1 14:08:04 kudan.its.uow.edu.au netapp: [ID 702911 daemon.info] (e4b) Mux state change: (MUX_COLLECT_DISTRIBUTE) to (MUX_COLLECTING) Apr 1 14:08:04 kudan.its.uow.edu.au netapp: [ID 702911 daemon.info] (e4b) Mux state change: (MUX_COLLECTING) to (MUX_ATTACHED) Apr 1 14:08:04 kudan.its.uow.edu.au netapp: [ID 702911 daemon.info] (e4b) Mux state change: (MUX_ATTACHED) to (MUX_DETACHED) Apr 1 14:08:04 kudan.its.uow.edu.au netapp: [ID 702911 daemon.info] (e4b) Mux state change: (MUX_DETACHED) to (MUX_WAITING) Apr 1 14:09:04 kudan.its.uow.edu.au netapp: [ID 702911 daemon.info] (e4b) Mux state change: (MUX_WAITING) to (MUX_ATTACHED) Apr 1 14:09:04 kudan.its.uow.edu.au netapp: [ID 702911 daemon.info] (e4b) Mux state change: (MUX_ATTACHED) to (MUX_COLLECTING) Apr 1 14:09:04 kudan.its.uow.edu.au netapp: [ID 702911 daemon.info] (e4b) Mux state change: (MUX_COLLECTING) to (MUX_COLLECT_DISTRIBUTE) Apr 1 14:16:25 kudan.its.uow.edu.au netapp: [ID 702911 daemon.info] (e4a) Mux state change: (MUX_COLLECT_DISTRIBUTE) to (MUX_COLLECTING) Apr 1 14:16:25 kudan.its.uow.edu.au netapp: [ID 702911 daemon.info] (e4a) Mux state change: (MUX_COLLECTING) to (MUX_ATTACHED) Apr 1 14:16:25 kudan.its.uow.edu.au netapp: [ID 702911 daemon.info] (e4a) Mux state change: (MUX_ATTACHED) to (MUX_DETACHED) Apr 1 14:16:25 kudan.its.uow.edu.au netapp: [ID 702911 daemon.info] (e4a) Mux state change: (MUX_DETACHED) to (MUX_WAITING) Apr 1 14:16:25 kudan.its.uow.edu.au netapp: [ID 702911 daemon.info] (e4a) Mux state change: (MUX_WAITING) to (MUX_ATTACHED) Apr 1 14:16:25 kudan.its.uow.edu.au netapp: [ID 702911 daemon.info] (e4a) Mux state change: (MUX_ATTACHED) to (MUX_COLLECTING) Apr 1 14:16:25 kudan.its.uow.edu.au netapp: [ID 702911 daemon.info] (e4a) Mux state change: (MUX_COLLECTING) to (MUX_COLLECT_DISTRIBUTE) Apr 1 14:18:26 kudan.its.uow.edu.au netapp: [ID 702911 daemon.info] (e4a) Mux state change: (MUX_COLLECT_DISTRIBUTE) to (MUX_COLLECTING) Apr 1 14:18:26 kudan.its.uow.edu.au netapp: [ID 702911 daemon.info] (e4a) Mux state change: (MUX_COLLECTING) to (MUX_ATTACHED) Apr 1 14:18:26 kudan.its.uow.edu.au netapp: [ID 702911 daemon.info] (e4a) Mux state change: (MUX_ATTACHED) to (MUX_DETACHED) Apr 1 14:18:26 kudan.its.uow.edu.au netapp: [ID 702911 daemon.info] (e4a) Mux state change: (MUX_DETACHED) to (MUX_WAITING) Apr 1 14:18:26 kudan.its.uow.edu.au netapp: [ID 702911 daemon.info] (e4a) Mux state change: (MUX_WAITING) to (MUX_ATTACHED) Apr 1 14:18:26 kudan.its.uow.edu.au netapp: [ID 702911 daemon.info] (e4a) Mux state change: (MUX_ATTACHED) to (MUX_COLLECTING) Apr 1 14:18:26 kudan.its.uow.edu.au netapp: [ID 702911 daemon.info] (e4a) Mux state change: (MUX_COLLECTING) to (MUX_COLLECT_DISTRIBUTE) Apr 1 14:08:04 kudan.its.uow.edu.au netapp: [ID 702911 daemon.info] (e4b) Mux state change: (MUX_COLLECT_DISTRIBUTE) to (MUX_COLLECTING) Apr 1 14:08:04 kudan.its.uow.edu.au netapp: [ID 702911 daemon.info] (e4b) Mux state change: (MUX_COLLECTING) to (MUX_ATTACHED) Apr 1 14:08:04 kudan.its.uow.edu.au netapp: [ID 702911 daemon.info] (e4b) Mux state change: (MUX_ATTACHED) to (MUX_DETACHED)
The settings on the switch and the netapp seem to agree. If our understanding of this is correct, there is a LACP packet every 30 seconds (active setting). If 3 are missed in a row, the netapp downs that leg of the portchannel. Later, a packet arrives and the netapp ups the leg of the portchannel.
Does anybody with a deep understanding of this know why we get these messages and why it appears the netapp is uping and downing the interface and if this is serious.
Any help appreciated.
Regards, pdg
Peter Gray Ph (direct): +61 2 4221 3770 Information Technology Services Ph (switch): +61 2 4221 3555 University of Wollongong Fax: +61 2 4229 1958 Wollongong NSW 2522 Email: pdg@uow.edu.au Australia URL: http://pdg.uow.edu.au
On 2014-4-1 5:24 , Peter D. Gray wrote:
We have 2 3270 filers, each using a LACP portchannel over 2 1G interfaces to a CISCO 6500.
When we enable LACP debugging, we get this.
Apr 1 14:08:04 kudan.its.uow.edu.au netapp: [ID 702911 daemon.info] (e4b) Mux state change: (MUX_COLLECT_DISTRIBUTE) to (MUX_COLLECTING)
[instability].
It's probably interesting to known which OS version (cisco AND ontap). Are you also seeing this in syslog on the cisco? Anything interesting in /etc/log/lacp_log on the filer?
We have a pair of 3250's connected with 2*1G LACP to a Cisco 6500, and we're also seeing LACP instability. Ontap 8.2P3, Cisco IOS 12.2.
I did a simple "ping" measurement, and there is occasional packet loss, sometimes up to 15% (measured over 2 minute intervals, standard 1-ping-per-second), which is quite substantial. It seems like the link flaps only occur under heavy load (where "heavy load" in our case is more than a few hundred mbit).
As far as we can tell, this started the moment we upgraded Ontap from 8.1.x to 8.2 (at least, the Cisco started complaining at that exact moment).
There's a case open with netapp for this problem.... for a while now.