Hello,
Anybody have experience setting Flow Control to none on an array (controllers) that are already being used in production? We have a number of filers that were deployed with Flow Control set to Full on the 10GE interfaces. NetApp best practices recommend setting Flow Control to none.
Is it preferable to:
a) Update the RC file with the Flow Control settings and do a takeover/giveback to apply the new flow control settings?
b) Modify the interface Flow Control settings directly with the system up and running? Then update the RC file so those settings will persist across future takeover/givebacks? I know the interface would 'reset' and I'm not sure how long of an outage we can expect.
c) Something else?
Our busiest ethernet based clients are VMWare ESXi hosts with pretty busy NFS datastores.
Thank you all in advance! Phil
Either method will work, but personally I would do a takeover/giveback during a maintenance windows, just to verify that the rc file changes are correct. Nothing worse than a typo causing a time-bomb.
I seem to recall somewhere that changing flow control settings may cause a momentary network outage, but I don’t have a reference for that. I’m sure the impact would be less than the pause you get during a takeover/giveback though.
HTH, Jeremy
On 8 Nov 2014, at 8:20 am, Philbert Rupkins <philbertrupkins@gmail.commailto:philbertrupkins@gmail.com> wrote:
Hello,
Anybody have experience setting Flow Control to none on an array (controllers) that are already being used in production? We have a number of filers that were deployed with Flow Control set to Full on the 10GE interfaces. NetApp best practices recommend setting Flow Control to none.
Is it preferable to:
a) Update the RC file with the Flow Control settings and do a takeover/giveback to apply the new flow control settings?
b) Modify the interface Flow Control settings directly with the system up and running? Then update the RC file so those settings will persist across future takeover/givebacks? I know the interface would 'reset' and I'm not sure how long of an outage we can expect.
c) Something else?
Our busiest ethernet based clients are VMWare ESXi hosts with pretty busy NFS datastores.
Thank you all in advance! Phil _______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
-- Jeremy Webber Senior Systems Engineer Animal Logic Pty Ltd T: +61 2 8310 3577 <— please note changed number M: +61 2 9383 4800 F: +61 2 9383 4801
Setting flow control will create a blip, as the port gets "rebooted" during the process. Depending on load, generally less than a minute.
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Jeremy Webber Sent: Sunday, November 09, 2014 6:29 PM To: Philbert Rupkins Cc: toasters@teaparty.net Subject: Re: Setting Flow Control to None on Production Filers
Either method will work, but personally I would do a takeover/giveback during a maintenance windows, just to verify that the rc file changes are correct. Nothing worse than a typo causing a time-bomb.
I seem to recall somewhere that changing flow control settings may cause a momentary network outage, but I don't have a reference for that. I'm sure the impact would be less than the pause you get during a takeover/giveback though.
HTH, Jeremy
On 8 Nov 2014, at 8:20 am, Philbert Rupkins <philbertrupkins@gmail.commailto:philbertrupkins@gmail.com> wrote:
Hello,
Anybody have experience setting Flow Control to none on an array (controllers) that are already being used in production? We have a number of filers that were deployed with Flow Control set to Full on the 10GE interfaces. NetApp best practices recommend setting Flow Control to none.
Is it preferable to:
a) Update the RC file with the Flow Control settings and do a takeover/giveback to apply the new flow control settings?
b) Modify the interface Flow Control settings directly with the system up and running? Then update the RC file so those settings will persist across future takeover/givebacks? I know the interface would 'reset' and I'm not sure how long of an outage we can expect.
c) Something else?
Our busiest ethernet based clients are VMWare ESXi hosts with pretty busy NFS datastores.
Thank you all in advance! Phil _______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
-- Jeremy Webber Senior Systems Engineer Animal Logic Pty Ltd T: +61 2 8310 3577 <- please note changed number M: +61 2 9383 4800 F: +61 2 9383 4801
I changed the MTU for an iSCSI interface…. freaked me out, but was about 15 to 20 seconds, and was not under load.
On Nov 9, 2014, at 5:35 PM, Parisi, Justin Justin.Parisi@netapp.com wrote:
Setting flow control will create a blip, as the port gets “rebooted” during the process. Depending on load, generally less than a minute.
From: toasters-bounces@teaparty.net mailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net mailto:toasters-bounces@teaparty.net] On Behalf Of Jeremy Webber Sent: Sunday, November 09, 2014 6:29 PM To: Philbert Rupkins Cc: toasters@teaparty.net mailto:toasters@teaparty.net Subject: Re: Setting Flow Control to None on Production Filers
Either method will work, but personally I would do a takeover/giveback during a maintenance windows, just to verify that the rc file changes are correct. Nothing worse than a typo causing a time-bomb.
I seem to recall somewhere that changing flow control settings may cause a momentary network outage, but I don’t have a reference for that. I’m sure the impact would be less than the pause you get during a takeover/giveback though.
HTH, Jeremy
On 8 Nov 2014, at 8:20 am, Philbert Rupkins <philbertrupkins@gmail.com mailto:philbertrupkins@gmail.com> wrote:
Hello,
Anybody have experience setting Flow Control to none on an array (controllers) that are already being used in production? We have a number of filers that were deployed with Flow Control set to Full on the 10GE interfaces. NetApp best practices recommend setting Flow Control to none.
Is it preferable to:
a) Update the RC file with the Flow Control settings and do a takeover/giveback to apply the new flow control settings?
b) Modify the interface Flow Control settings directly with the system up and running? Then update the RC file so those settings will persist across future takeover/givebacks? I know the interface would 'reset' and I'm not sure how long of an outage we can expect.
c) Something else?
Our busiest ethernet based clients are VMWare ESXi hosts with pretty busy NFS datastores.
Thank you all in advance! Phil _______________________________________________ Toasters mailing list Toasters@teaparty.net mailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters http://www.teaparty.net/mailman/listinfo/toasters
-- Jeremy Webber Senior Systems Engineer Animal Logic Pty Ltd T: +61 2 8310 3577 <— please note changed number M: +61 2 9383 4800 F: +61 2 9383 4801
Toasters mailing list Toasters@teaparty.net mailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters http://www.teaparty.net/mailman/listinfo/toasters
Just in case, I would suggest preparing the commands to bring down your interface and destroy any vlan interfaces and ifgroups in the appropriate order. That way if things go south you can act quickly and tear down the interface stack and source your /etc/rc
Hopefully that is not necessary but it cannot hurt to be prepared.
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of steve klise Sent: Sunday, November 09, 2014 8:44 PM To: Parisi, Justin Cc: toasters@teaparty.net Subject: Re: Setting Flow Control to None on Production Filers
I changed the MTU for an iSCSI interface…. freaked me out, but was about 15 to 20 seconds, and was not under load.
On Nov 9, 2014, at 5:35 PM, Parisi, Justin <Justin.Parisi@netapp.commailto:Justin.Parisi@netapp.com> wrote:
Setting flow control will create a blip, as the port gets “rebooted” during the process. Depending on load, generally less than a minute.
From: toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Jeremy Webber Sent: Sunday, November 09, 2014 6:29 PM To: Philbert Rupkins Cc: toasters@teaparty.netmailto:toasters@teaparty.net Subject: Re: Setting Flow Control to None on Production Filers
Either method will work, but personally I would do a takeover/giveback during a maintenance windows, just to verify that the rc file changes are correct. Nothing worse than a typo causing a time-bomb.
I seem to recall somewhere that changing flow control settings may cause a momentary network outage, but I don’t have a reference for that. I’m sure the impact would be less than the pause you get during a takeover/giveback though.
HTH, Jeremy
On 8 Nov 2014, at 8:20 am, Philbert Rupkins <philbertrupkins@gmail.commailto:philbertrupkins@gmail.com> wrote:
Hello,
Anybody have experience setting Flow Control to none on an array (controllers) that are already being used in production? We have a number of filers that were deployed with Flow Control set to Full on the 10GE interfaces. NetApp best practices recommend setting Flow Control to none.
Is it preferable to:
a) Update the RC file with the Flow Control settings and do a takeover/giveback to apply the new flow control settings?
b) Modify the interface Flow Control settings directly with the system up and running? Then update the RC file so those settings will persist across future takeover/givebacks? I know the interface would 'reset' and I'm not sure how long of an outage we can expect.
c) Something else?
Our busiest ethernet based clients are VMWare ESXi hosts with pretty busy NFS datastores.
Thank you all in advance! Phil _______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
-- Jeremy Webber Senior Systems Engineer Animal Logic Pty Ltd T: +61 2 8310 3577 <— please note changed number M: +61 2 9383 4800 F: +61 2 9383 4801
_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Thanks everybody. This helps.
We'll probably just update the RC file (setting flow control to none for our 10g nics) and do a takeover/giveback. Sounds like directly modifying flow control, mtu and other physical interface settings on the fly cause an interface reset and tangible outage anyway. Also, not quite sure what would happen to our interface groups if we didnt do a takeover/giveback.
We plan to set flow control to none on our 2240 nics (via the RC file during giveback) AND have our network guys configure the switch ports to flow control inbound=none; flow control outbound=none. This is in line with the various NetApp best practices documents I've read.
Note: the switch ports are currently configured for flow control inbound=none;flow control outbound=on. as stated above, well be changing the switch port configs to none/none.
We can't modify the flow control settings on the 10g nics (CNA's) on our 3240's. The plan with the 3240's is perform the takeover/giveback but the only change will be to the switch ports. My understanding is disabling flow control at the switch ports will effectively disable it for the storage controller as well. We are not an FCoE shop to priority flow control (pfc) is no concern.
I really wish somebody had read the various best practices documents before setting these arrays up. I repeatedly see the recommendation to set flow control to send=none, receive=none for 10g infrastructure. Always a pain to reconfigure such a low level setting on an array already running production workloads. We see a lot of pause frames received by the controllers and think this may have something to do with very random/infrequent NFS datastore disconnects/reconnect activity. The NFS datastore disconnect/reconnect is very brief, and nobody notices it, but we dont want to see this snowball and flow control (pause frames) is currently raising a red flag. end rant.
I'll update this thread once we've made the change to let everybody know how it went. -Phil
On Mon, Nov 10, 2014 at 7:40 AM, Jordan Slingerland < Jordan.Slingerland@independenthealth.com> wrote:
Just in case, I would suggest preparing the commands to bring down your interface and destroy any vlan interfaces and ifgroups in the appropriate order. That way if things go south you can act quickly and tear down the interface stack and source your /etc/rc
Hopefully that is not necessary but it cannot hurt to be prepared.
*From:* toasters-bounces@teaparty.net [mailto: toasters-bounces@teaparty.net] *On Behalf Of *steve klise *Sent:* Sunday, November 09, 2014 8:44 PM *To:* Parisi, Justin
*Cc:* toasters@teaparty.net *Subject:* Re: Setting Flow Control to None on Production Filers
I changed the MTU for an iSCSI interface…. freaked me out, but was about 15 to 20 seconds, and was not under load.
On Nov 9, 2014, at 5:35 PM, Parisi, Justin Justin.Parisi@netapp.com wrote:
Setting flow control will create a blip, as the port gets “rebooted” during the process. Depending on load, generally less than a minute.
*From:* toasters-bounces@teaparty.net [ mailto:toasters-bounces@teaparty.net toasters-bounces@teaparty.net] *On Behalf Of *Jeremy Webber *Sent:* Sunday, November 09, 2014 6:29 PM *To:* Philbert Rupkins *Cc:* toasters@teaparty.net *Subject:* Re: Setting Flow Control to None on Production Filers
Either method will work, but personally I would do a takeover/giveback during a maintenance windows, just to verify that the rc file changes are correct. Nothing worse than a typo causing a time-bomb.
I seem to recall somewhere that changing flow control settings may cause a momentary network outage, but I don’t have a reference for that. I’m sure the impact would be less than the pause you get during a takeover/giveback though.
HTH,
Jeremy
On 8 Nov 2014, at 8:20 am, Philbert Rupkins philbertrupkins@gmail.com wrote:
Hello,
Anybody have experience setting Flow Control to none on an array (controllers) that are already being used in production? We have a number of filers that were deployed with Flow Control set to Full on the 10GE interfaces. NetApp best practices recommend setting Flow Control to none.
Is it preferable to:
a) Update the RC file with the Flow Control settings and do a takeover/giveback to apply the new flow control settings?
b) Modify the interface Flow Control settings directly with the system up and running? Then update the RC file so those settings will persist across future takeover/givebacks? I know the interface would 'reset' and I'm not sure how long of an outage we can expect.
c) Something else?
Our busiest ethernet based clients are VMWare ESXi hosts with pretty busy NFS datastores.
Thank you all in advance!
Phil
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
-- Jeremy Webber Senior Systems Engineer Animal Logic Pty Ltd T: +61 2 8310 3577 <— please note changed number M: +61 2 9383 4800 F: +61 2 9383 4801
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
-
Jeremy Webber -
Jordan Slingerland -
Parisi, Justin -
Philbert Rupkins -
steve klise