Constantly receiving the below alert by email (for both e0e & e0f even though they are in an ifgrp). Can't find anything about it on the KB and would be interested to know how to increase the threshold and thoughts on doing so?
---
Severity: LOG_ERR
Message: netif.rateLimitThreshold: High rate limit on network interface e0e for broadcast protocol ARP being detected: 5001 pkts/sec.
Description: This message occurs when the protocol rate threshold is reached on a network interface.
Action: Fix the faulty network configuration or incorrect setup that enables a sudden spike in broadcast packets to bring down the node. If you still want a high ARP rate threshold, use the "bootarg.arp.ratelimit.threshold" boot argument to set that threshold.
Source: NwkThd_04
Index: 3128725
[Note: This email message is sent using a deprecated event routing mechanism.
For information, search the knowledgebase of the NetApp support web site for "convert existing event configurations in Data ONTAP 9.0."]
I happened to have a window open to our bug tracker, and there's nothing mentioned about a known bug.
It does seem like this should be a warning rather than an error. A lot of ARP traffic was once connected with some DDOS attacks, but I don't think that's really applicable any more. Anything vulnerable to such a thing should have been patched ages go.
What else is on that subnet? Do you have a lot of other hosts which would be communicating with one another?
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Chris Hague Sent: Monday, September 25, 2017 10:54 AM To: Toasters@teaparty.net Toasters@teaparty.net Subject: Event Threshold
Constantly receiving the below alert by email (for both e0e & e0f even though they are in an ifgrp). Can't find anything about it on the KB and would be interested to know how to increase the threshold and thoughts on doing so?
---
Severity: LOG_ERR
Message: netif.rateLimitThreshold: High rate limit on network interface e0e for broadcast protocol ARP being detected: 5001 pkts/sec.
Description: This message occurs when the protocol rate threshold is reached on a network interface.
Action: Fix the faulty network configuration or incorrect setup that enables a sudden spike in broadcast packets to bring down the node. If you still want a high ARP rate threshold, use the "bootarg.arp.ratelimit.threshold" boot argument to set that threshold.
Source: NwkThd_04
Index: 3128725
[Note: This email message is sent using a deprecated event routing mechanism.
For information, search the knowledgebase of the NetApp support web site for "convert existing event configurations in Data ONTAP 9.0."]
That's the thing Jeffrey, it only mentions e0e & e0f, but those are bonded in an ifgrp which then has multiple VLAN's running over them.
From: Steiner, Jeffrey [mailto:Jeffrey.Steiner@netapp.com] Sent: 25 September 2017 10:01 To: Chris Hague Chris_Hague@ajg.com; Toasters@teaparty.net Toasters@teaparty.net Subject: RE: Event Threshold
I happened to have a window open to our bug tracker, and there's nothing mentioned about a known bug.
It does seem like this should be a warning rather than an error. A lot of ARP traffic was once connected with some DDOS attacks, but I don't think that's really applicable any more. Anything vulnerable to such a thing should have been patched ages go.
What else is on that subnet? Do you have a lot of other hosts which would be communicating with one another?
From: toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Chris Hague Sent: Monday, September 25, 2017 10:54 AM To: <Toasters@teaparty.netmailto:Toasters@teaparty.net> <Toasters@teaparty.netmailto:Toasters@teaparty.net> Subject: Event Threshold
Constantly receiving the below alert by email (for both e0e & e0f even though they are in an ifgrp). Can't find anything about it on the KB and would be interested to know how to increase the threshold and thoughts on doing so?
---
Severity: LOG_ERR
Message: netif.rateLimitThreshold: High rate limit on network interface e0e for broadcast protocol ARP being detected: 5001 pkts/sec.
Description: This message occurs when the protocol rate threshold is reached on a network interface.
Action: Fix the faulty network configuration or incorrect setup that enables a sudden spike in broadcast packets to bring down the node. If you still want a high ARP rate threshold, use the "bootarg.arp.ratelimit.threshold" boot argument to set that threshold.
Source: NwkThd_04
Index: 3128725
[Note: This email message is sent using a deprecated event routing mechanism.
For information, search the knowledgebase of the NetApp support web site for "convert existing event configurations in Data ONTAP 9.0."]
You could open a case and see if there are debug settings that will let you see who's doing the ARP activity, but if you have access to regular linux hosts you could sample the subnet for ARP activity with "tcpdump -I [interface] -v arp" and see if something looks really out of the ordinary. It's a little more awkward to do it directly on ONTAP. I think you'd have to just run tcpdump and then get the trace files off the system, load them into Wireshark or similar, and filter for ARP.
You wouldn't normally have an ONTAP system deployed on the same subnet as, for example, the a lot of end users with their laptops and desktops and such, so a lot of ARPing would be out of the ordinary. There's nothing wrong with it, it just would be unusual. Normally you have to route to get between ONTAP and the end users. If you don't see anything strange in the tcpdump output, the next step would probably be to open a case and see if there's a tunable for this "error" message.
From: Chris Hague [mailto:Chris_Hague@ajg.com] Sent: Monday, September 25, 2017 11:41 AM To: Steiner, Jeffrey Jeffrey.Steiner@netapp.com; Toasters@teaparty.net Toasters@teaparty.net Subject: RE: Event Threshold
That's the thing Jeffrey, it only mentions e0e & e0f, but those are bonded in an ifgrp which then has multiple VLAN's running over them.
From: Steiner, Jeffrey [mailto:Jeffrey.Steiner@netapp.com] Sent: 25 September 2017 10:01 To: Chris Hague <Chris_Hague@ajg.commailto:Chris_Hague@ajg.com>; <Toasters@teaparty.netmailto:Toasters@teaparty.net> <Toasters@teaparty.netmailto:Toasters@teaparty.net> Subject: RE: Event Threshold
I happened to have a window open to our bug tracker, and there's nothing mentioned about a known bug.
It does seem like this should be a warning rather than an error. A lot of ARP traffic was once connected with some DDOS attacks, but I don't think that's really applicable any more. Anything vulnerable to such a thing should have been patched ages go.
What else is on that subnet? Do you have a lot of other hosts which would be communicating with one another?
From: toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Chris Hague Sent: Monday, September 25, 2017 10:54 AM To: <Toasters@teaparty.netmailto:Toasters@teaparty.net> <Toasters@teaparty.netmailto:Toasters@teaparty.net> Subject: Event Threshold
Constantly receiving the below alert by email (for both e0e & e0f even though they are in an ifgrp). Can't find anything about it on the KB and would be interested to know how to increase the threshold and thoughts on doing so?
---
Severity: LOG_ERR
Message: netif.rateLimitThreshold: High rate limit on network interface e0e for broadcast protocol ARP being detected: 5001 pkts/sec.
Description: This message occurs when the protocol rate threshold is reached on a network interface.
Action: Fix the faulty network configuration or incorrect setup that enables a sudden spike in broadcast packets to bring down the node. If you still want a high ARP rate threshold, use the "bootarg.arp.ratelimit.threshold" boot argument to set that threshold.
Source: NwkThd_04
Index: 3128725
[Note: This email message is sent using a deprecated event routing mechanism.
For information, search the knowledgebase of the NetApp support web site for "convert existing event configurations in Data ONTAP 9.0."]
Thanks again.
There is a CIFS vfiler on this system running on a VLAN on those ports. This has an IP address which is not on the same subnet as the users but is routeable.
From: Steiner, Jeffrey [mailto:Jeffrey.Steiner@netapp.com] Sent: 25 September 2017 10:57 To: Chris Hague Chris_Hague@ajg.com; Toasters@teaparty.net Toasters@teaparty.net Subject: RE: Event Threshold
You could open a case and see if there are debug settings that will let you see who's doing the ARP activity, but if you have access to regular linux hosts you could sample the subnet for ARP activity with "tcpdump -I [interface] -v arp" and see if something looks really out of the ordinary. It's a little more awkward to do it directly on ONTAP. I think you'd have to just run tcpdump and then get the trace files off the system, load them into Wireshark or similar, and filter for ARP.
You wouldn't normally have an ONTAP system deployed on the same subnet as, for example, the a lot of end users with their laptops and desktops and such, so a lot of ARPing would be out of the ordinary. There's nothing wrong with it, it just would be unusual. Normally you have to route to get between ONTAP and the end users. If you don't see anything strange in the tcpdump output, the next step would probably be to open a case and see if there's a tunable for this "error" message.
From: Chris Hague [mailto:Chris_Hague@ajg.com] Sent: Monday, September 25, 2017 11:41 AM To: Steiner, Jeffrey <Jeffrey.Steiner@netapp.commailto:Jeffrey.Steiner@netapp.com>; <Toasters@teaparty.netmailto:Toasters@teaparty.net> <Toasters@teaparty.netmailto:Toasters@teaparty.net> Subject: RE: Event Threshold
That's the thing Jeffrey, it only mentions e0e & e0f, but those are bonded in an ifgrp which then has multiple VLAN's running over them.
From: Steiner, Jeffrey [mailto:Jeffrey.Steiner@netapp.com] Sent: 25 September 2017 10:01 To: Chris Hague <Chris_Hague@ajg.commailto:Chris_Hague@ajg.com>; <Toasters@teaparty.netmailto:Toasters@teaparty.net> <Toasters@teaparty.netmailto:Toasters@teaparty.net> Subject: RE: Event Threshold
I happened to have a window open to our bug tracker, and there's nothing mentioned about a known bug.
It does seem like this should be a warning rather than an error. A lot of ARP traffic was once connected with some DDOS attacks, but I don't think that's really applicable any more. Anything vulnerable to such a thing should have been patched ages go.
What else is on that subnet? Do you have a lot of other hosts which would be communicating with one another?
From: toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Chris Hague Sent: Monday, September 25, 2017 10:54 AM To: <Toasters@teaparty.netmailto:Toasters@teaparty.net> <Toasters@teaparty.netmailto:Toasters@teaparty.net> Subject: Event Threshold
Constantly receiving the below alert by email (for both e0e & e0f even though they are in an ifgrp). Can't find anything about it on the KB and would be interested to know how to increase the threshold and thoughts on doing so?
---
Severity: LOG_ERR
Message: netif.rateLimitThreshold: High rate limit on network interface e0e for broadcast protocol ARP being detected: 5001 pkts/sec.
Description: This message occurs when the protocol rate threshold is reached on a network interface.
Action: Fix the faulty network configuration or incorrect setup that enables a sudden spike in broadcast packets to bring down the node. If you still want a high ARP rate threshold, use the "bootarg.arp.ratelimit.threshold" boot argument to set that threshold.
Source: NwkThd_04
Index: 3128725
[Note: This email message is sent using a deprecated event routing mechanism.
For information, search the knowledgebase of the NetApp support web site for "convert existing event configurations in Data ONTAP 9.0."]
That wouldn't explain the ARPing. That should only propagate within the subnet.
I just thought of a way the ARPing could be caused by a misconfiguration. I'm not sure if ONTAP filters this, but let's say someone accidentally added a subnet to one of the ports used by ONTAP. You can have leakage of ARP's from a foreign subnet to the ONTAP NICs. It might just drop an ARP that isn't on the same subnet.
I used to have a bunch of Nortel routing switches that would insist on putting the 'default' subnet on every newly configured interface whether you wanted it to be there or not. If we forgot to remove the default subnet, weird things would happen as hosts saw activity from subnets that shouldn't have been visible on those ports.
From: Chris Hague [mailto:Chris_Hague@ajg.com] Sent: Monday, September 25, 2017 12:04 PM To: Steiner, Jeffrey Jeffrey.Steiner@netapp.com; Toasters@teaparty.net Toasters@teaparty.net Subject: RE: Event Threshold
Thanks again.
There is a CIFS vfiler on this system running on a VLAN on those ports. This has an IP address which is not on the same subnet as the users but is routeable.
From: Steiner, Jeffrey [mailto:Jeffrey.Steiner@netapp.com] Sent: 25 September 2017 10:57 To: Chris Hague <Chris_Hague@ajg.commailto:Chris_Hague@ajg.com>; <Toasters@teaparty.netmailto:Toasters@teaparty.net> <Toasters@teaparty.netmailto:Toasters@teaparty.net> Subject: RE: Event Threshold
You could open a case and see if there are debug settings that will let you see who's doing the ARP activity, but if you have access to regular linux hosts you could sample the subnet for ARP activity with "tcpdump -I [interface] -v arp" and see if something looks really out of the ordinary. It's a little more awkward to do it directly on ONTAP. I think you'd have to just run tcpdump and then get the trace files off the system, load them into Wireshark or similar, and filter for ARP.
You wouldn't normally have an ONTAP system deployed on the same subnet as, for example, the a lot of end users with their laptops and desktops and such, so a lot of ARPing would be out of the ordinary. There's nothing wrong with it, it just would be unusual. Normally you have to route to get between ONTAP and the end users. If you don't see anything strange in the tcpdump output, the next step would probably be to open a case and see if there's a tunable for this "error" message.
From: Chris Hague [mailto:Chris_Hague@ajg.com] Sent: Monday, September 25, 2017 11:41 AM To: Steiner, Jeffrey <Jeffrey.Steiner@netapp.commailto:Jeffrey.Steiner@netapp.com>; <Toasters@teaparty.netmailto:Toasters@teaparty.net> <Toasters@teaparty.netmailto:Toasters@teaparty.net> Subject: RE: Event Threshold
That's the thing Jeffrey, it only mentions e0e & e0f, but those are bonded in an ifgrp which then has multiple VLAN's running over them.
From: Steiner, Jeffrey [mailto:Jeffrey.Steiner@netapp.com] Sent: 25 September 2017 10:01 To: Chris Hague <Chris_Hague@ajg.commailto:Chris_Hague@ajg.com>; <Toasters@teaparty.netmailto:Toasters@teaparty.net> <Toasters@teaparty.netmailto:Toasters@teaparty.net> Subject: RE: Event Threshold
I happened to have a window open to our bug tracker, and there's nothing mentioned about a known bug.
It does seem like this should be a warning rather than an error. A lot of ARP traffic was once connected with some DDOS attacks, but I don't think that's really applicable any more. Anything vulnerable to such a thing should have been patched ages go.
What else is on that subnet? Do you have a lot of other hosts which would be communicating with one another?
From: toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Chris Hague Sent: Monday, September 25, 2017 10:54 AM To: <Toasters@teaparty.netmailto:Toasters@teaparty.net> <Toasters@teaparty.netmailto:Toasters@teaparty.net> Subject: Event Threshold
Constantly receiving the below alert by email (for both e0e & e0f even though they are in an ifgrp). Can't find anything about it on the KB and would be interested to know how to increase the threshold and thoughts on doing so?
---
Severity: LOG_ERR
Message: netif.rateLimitThreshold: High rate limit on network interface e0e for broadcast protocol ARP being detected: 5001 pkts/sec.
Description: This message occurs when the protocol rate threshold is reached on a network interface.
Action: Fix the faulty network configuration or incorrect setup that enables a sudden spike in broadcast packets to bring down the node. If you still want a high ARP rate threshold, use the "bootarg.arp.ratelimit.threshold" boot argument to set that threshold.
Source: NwkThd_04
Index: 3128725
[Note: This email message is sent using a deprecated event routing mechanism.
For information, search the knowledgebase of the NetApp support web site for "convert existing event configurations in Data ONTAP 9.0."]
-
Chris Hague -
Steiner, Jeffrey