Phil –

This is a very high number of pause frames in my opinion, per this KB -- https://kb.netapp.com/support/index?id=3013252&page=content&locale=en_US – each pause frame can add up to 3.355ms of latency which can definitely cause problems with iSCSI or FCoE traffic.

Is there a specific reason you need flow control enabled? It’s not only not best practice for NetApp, but generally not best practice for any block storage which uses Ethernet as a transmission media. Unless there’s some specific reason you’re enabling it, I would follow the recommendations and turn it off. Remember, you want to keep latency under 5ms for block storage – especially for database applications.

Good luck to you!

Anthony Bar | Director of Engineering

650.207.5368 | tbar@berkcom.com

Berkeley Communications | www.berkcom.com

From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Philbert Rupkins
Sent: Friday, April 17, 2015 10:23 AM
To: toasters@teaparty.net
Subject: Excessive Flow Control Frames?

Toasters,

We have ethernet flow control set to full on our NetApp's 10g interfaces. The FEX ports to which our 10g interfaces are connected also have flow control TX/RX enabled. I realize enabling flow control is against best practices.

I am trying to determine if we are seeing an excessive number of flow control frames. We received over 12,000 PAUSE frames from the FEX within a 10 minute period which sounds excessive to me. Can anybody confirm that 12,000 is an excessive number of PAUSE frames that would likely result in connectivity issues?

Here are statistics for one of our 10g interfaces during a recent 10 minute interval. Notice that the NetApp is receiving PAUSE frames (RECEIVE-Xoff) from the FEX and never transmitting (TRANSMIT-Xoff).

RECEIVE

Jabber: 0 | Bus overruns: 0 | Xon: 12412

Xoff: 12412 | Jumbo: 0

TRANSMIT

Queue overflows: 0 | No buffers: 0 | Xon: 0

Xoff: 0 | Jumbo: 0 | TSO non-TCP drop: 0

What I see is the FEX telling the NetApp to slow down. I just dont know if the FEX is slowing down the NetApp enough that it would cause problems.

Have a great day,
Phil