On Fri, Aug 10, 2012 at 05:40:04PM +0000, Jayanathan, David wrote:

How are your routing groups setup, from the "network routing-groups show" command, and what metric is assigned to each? Do you have your vserver configured to allow protocol services to use Data LIFs? You can verify by running a "vserver show -vserver <vservername>".

Thanks, David

Hi David;

These commands don't appear to be valid on our version of Data ONTAP.

Here is the top half of our routing tables (the routable stuff):

Internet: Destination Gateway Flags Refs Use Interface default 10.44.15.254 UGS 18 12334464694 e0M 10.44.12/22 link#7 UC 0 0 e0M 10.44.12.6 0:a0:98:1a:bd:c6 UHL 0 0 e0M 10.44.12.7 0:a0:98:1a:bc:c4 UHL 0 12 lo 10.44.15.254 0:0:c:7:ac:64 UHL 1 0 e0M 10.44.15.255 ff:ff:ff:ff:ff:ff UHL 0 32258 e0M 10.49/16 link#14 UC 0 0 ifgrp0-49 10.49.2.69 0:50:56:88:0:32 UHL 0 6 ifgrp0-49 10.49.4.7 0:1c:23:e2:40:c7 UHL 0 6 ifgrp0-49 10.49.4.43 14:fe:b5:c9:39:8e UHL 0 0 ifgrp0-49 10.49.4.68 d0:67:e5:f8:86:68 UHL 0 3 ifgrp0-49 10.49.4.76 0:15:17:45:b5:41 UHL 0 9 ifgrp0-49 10.49.5.106 0:15:17:60:e4:c UHL 0 3 ifgrp0-49 10.49.6.49 0:25:90:3d:47:6e UHL 0 28662 ifgrp0-49 10.49.6.156 0:25:90:63:88:24 UHL 0 7 ifgrp0-49 10.49.6.161 90:e2:ba:13:3:68 UHL 0 12718 ifgrp0-49 10.49.6.162 90:e2:ba:12:fb:3c UHL 0 8764 ifgrp0-49 10.49.6.167 0:25:90:b:15:da UHL 0 4 ifgrp0-49 10.49.11.72 0:15:c5:e1:41:58 UHL 0 18 ifgrp0-49 10.49.13.10 0:21:9b:9d:6:6f UHL 0 18 ifgrp0-49 10.49.13.53 7a:2b:cb:16:7f:a0 UHL 0 12 ifgrp0-49 10.49.51.84 0:50:56:ab:47:9a UHL 1 1742 ifgrp0-49 10.49.51.95 0:50:56:ab:48:a6 UHL 0 1907277490 ifgrp0-49 10.49.51.249 0:15:17:45:66:ec UHL 0 21 ifgrp0-49 10.49.53.11 0:21:9b:ca:96:5 UHL 0 18 ifgrp0-49 10.49.104.131 84:2b:2b:17:12:c5 UHL 0 91806 ifgrp0-49 10.49.104.132 0:26:b9:28:b1:9c UHL 0 0 ifgrp0-49 10.49.104.207 0:50:56:88:5:97 UHL 0 2 ifgrp0-49 10.49.106.139 0:26:b9:28:b1:a9 UHL 0 161 ifgrp0-49 10.49.106.145 0:50:56:88:1:53 UHL 0 37 ifgrp0-49 10.49.107.101 0:21:9b:8f:b5:38 UHL 0 3 ifgrp0-49 10.49.110.207 0:18:8b:3f:38:d3 UHL 0 0 ifgrp0-49 10.49.110.239 0:50:56:ab:47:96 UHL 0 0 ifgrp0-49 10.49.254.252 0:c:31:46:4c:a UHL 0 0 ifgrp0-49 10.49.254.253 0:b:bf:c9:e4:8a UHL 0 0 ifgrp0-49 10.49.255.255 ff:ff:ff:ff:ff:ff UHL 0 32258 ifgrp0-49

We just need to move the default gateway to one of the ifgrp interfaces isntead of e0M is my impression.

Ray

-----Original Message----- From: Ray Van Dolson [mailto:rvandolson@esri.com] Sent: Thursday, August 09, 2012 4:40 PM To: Jayanathan, David Cc: toasters@teaparty.net Subject: Re: e0M port as default gateway?

On Thu, Aug 09, 2012 at 10:54:57PM +0000, Jayanathan, David wrote:

...

Hi Ray,

We allow all management activity to go through the same interfaces that data goes through. Only on our Cluster Mode implementations do we use e0M for management ports, otherwise we only plug in e0M in order to have the RLM/SP connections on the network.

Hope this helps!

-David

Thanks, David.

We are running cluster mode on these systems (two controllers).

Key thing we're seeing is that NFS/CIFS file serving activity grinds to a halt when we lose connectivity to AD/NIS servers (the communication for which currently goes through e0M due to default gateway setup).

I presume this is expected behavior, but am a bit surprised that NFS file serving would hang for activity not involving an NIS UID. Presumably any NFS requests gets mapped to a UID and would require an NIS lookup?

Anyone run into this before?

Ray

...

-----Original Message----- From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Ray Van Dolson Sent: Thursday, August 09, 2012 2:26 PM To: toasters@teaparty.net Subject: e0M port as default gateway?

Hello all;

We have some N6240's (rebadged FAS3xxx series) on which the default gateway is set to an IP that routes out the e0M interface.

As such, if the e0M port (wrench port) goes offline, essentially functionality on that controller ceases as communication with AD, NIS, etc. stops working.

We have 10GbE ports on these devices -- we could easily configure the default gateways to use one of those instead, but I am unclear as to what best practice is here. We access management functionality through the IP bound on the e0M port now.

How do the rest of you set things up?

Ray

Don't forget to update your /etc/rc file as well. That's a VERY common oversight seen in Support. It sounds like this:

Hello NetApp Support, Our cluster failed over (or the head rebooted after an upgrade, etc) and now it won't come back up. It's not accessible, only via console or RLM. The system looks fine, but we can't access it.

This can be several months (or years) later and it's not always obvious what the problem is. So just beware ;-)

Stetson Webster Lead Storage Architect – NetApp IT NCIE-SAN, NCIE-BR, SCSN-E, VCP

NetApp stetson@netapp.commailto:name@netapp.com www.netapp.com/us/http://www.netapp.com/us/ [Businesses built on NetApp go further, faster.]http://www.netapp.com/us/?REF_SOURCE=emmbonrunnersignaturehp-us Follow us: [Facebook] http://www.facebook.com/NetApp?REF_SOURCE=ems-facebook [Twitter] http://twitter.com/#netapp?REF_SOURCE=ems-twitter [Linked In] http://www.linkedin.com/groups/NetApp-111681/about?REF_SOURCE=ems-linkedin [YouTube] http://www.youtube.com/user/NetAppTV?REF_SOURCE=ems-youtube [Slideshare] http://www.slideshare.net/NetApp?REF_SOURCE=ems-slideshare [Community] https://communities.netapp.com/welcome?REF_SOURCE=ems-cty Tweet us: #netapp

From: Mark Flint <mf1@sanger.ac.ukmailto:mf1@sanger.ac.uk> Date: Tuesday, September 25, 2012 6:34 AM To: Ray Van Dolson <rvandolson@esri.commailto:rvandolson@esri.com> Cc: "toasters@teaparty.netmailto:toasters@teaparty.net" <toasters@teaparty.netmailto:toasters@teaparty.net> Subject: Re: e0M port as default gateway?

You'll need a 'route delete default', a 'route -f' to flush the table, then a 'route add default 10.44.X.X Z (the Z represents the metric, either 0 or 1, depending on hop count, I think in most cases it's 0, but someone more knowledgable will be along shortly to correct me :) )

`Mark

Mark Flint Principle Systems Administrator Storage & Backups Team System Support Group Wellcome Trust Sanger Institute Wellcome Trust Genome Campus Hinxton Cambridgeshire UK

email: mark.flint@sanger.ac.ukmailto:mark.flint@sanger.ac.uk Tel: 07990535591

On 11 Aug 2012, at 00:17, Ray Van Dolson wrote:

On Fri, Aug 10, 2012 at 05:40:04PM +0000, Jayanathan, David wrote: How are your routing groups setup, from the "network routing-groups show" command, and what metric is assigned to each? Do you have your vserver configured to allow protocol services to use Data LIFs? You can verify by running a "vserver show -vserver <vservername>".

Thanks, David

Hi David;

These commands don't appear to be valid on our version of Data ONTAP.

Here is the top half of our routing tables (the routable stuff):

Internet: Destination Gateway Flags Refs Use Interface default 10.44.15.254 UGS 18 12334464694 e0M 10.44.12/22 link#7 UC 0 0 e0M 10.44.12.6 0:a0:98:1a:bd:c6 UHL 0 0 e0M 10.44.12.7 0:a0:98:1a:bc:c4 UHL 0 12 lo 10.44.15.254 0:0:c:7:ac:64 UHL 1 0 e0M 10.44.15.255 ff:ff:ff:ff:ff:ff UHL 0 32258 e0M 10.49/16 link#14 UC 0 0 ifgrp0-49 10.49.2.69 0:50:56:88:0:32 UHL 0 6 ifgrp0-49 10.49.4.7 0:1c:23:e2:40:c7 UHL 0 6 ifgrp0-49 10.49.4.43 14:fe:b5:c9:39:8e UHL 0 0 ifgrp0-49 10.49.4.68 d0:67:e5:f8:86:68 UHL 0 3 ifgrp0-49 10.49.4.76 0:15:17:45:b5:41 UHL 0 9 ifgrp0-49 10.49.5.106 0:15:17:60:e4:c UHL 0 3 ifgrp0-49 10.49.6.49 0:25:90:3d:47:6e UHL 0 28662 ifgrp0-49 10.49.6.156 0:25:90:63:88:24 UHL 0 7 ifgrp0-49 10.49.6.161 90:e2:ba:13:3:68 UHL 0 12718 ifgrp0-49 10.49.6.162 90:e2:ba:12:fb:3c UHL 0 8764 ifgrp0-49 10.49.6.167 0:25:90:b:15:da UHL 0 4 ifgrp0-49 10.49.11.72 0:15:c5:e1:41:58 UHL 0 18 ifgrp0-49 10.49.13.10 0:21:9b:9d:6:6f UHL 0 18 ifgrp0-49 10.49.13.53 7a:2b:cb:16:7f:a0 UHL 0 12 ifgrp0-49 10.49.51.84 0:50:56:ab:47:9a UHL 1 1742 ifgrp0-49 10.49.51.95 0:50:56:ab:48:a6 UHL 0 1907277490 ifgrp0-49 10.49.51.249 0:15:17:45:66:ec UHL 0 21 ifgrp0-49 10.49.53.11 0:21:9b:ca:96:5 UHL 0 18 ifgrp0-49 10.49.104.131 84:2b:2b:17:12:c5 UHL 0 91806 ifgrp0-49 10.49.104.132 0:26:b9:28:b1:9c UHL 0 0 ifgrp0-49 10.49.104.207 0:50:56:88:5:97 UHL 0 2 ifgrp0-49 10.49.106.139 0:26:b9:28:b1:a9 UHL 0 161 ifgrp0-49 10.49.106.145 0:50:56:88:1:53 UHL 0 37 ifgrp0-49 10.49.107.101 0:21:9b:8f:b5:38 UHL 0 3 ifgrp0-49 10.49.110.207 0:18:8b:3f:38:d3 UHL 0 0 ifgrp0-49 10.49.110.239 0:50:56:ab:47:96 UHL 0 0 ifgrp0-49 10.49.254.252 0:c:31:46:4c:a UHL 0 0 ifgrp0-49 10.49.254.253 0:b:bf:c9:e4:8a UHL 0 0 ifgrp0-49 10.49.255.255 ff:ff:ff:ff:ff:ff UHL 0 32258 ifgrp0-49

We just need to move the default gateway to one of the ifgrp interfaces isntead of e0M is my impression.

Ray

-----Original Message----- From: Ray Van Dolson [mailto:rvandolson@esri.com] Sent: Thursday, August 09, 2012 4:40 PM To: Jayanathan, David Cc: toasters@teaparty.netmailto:toasters@teaparty.net Subject: Re: e0M port as default gateway?

On Thu, Aug 09, 2012 at 10:54:57PM +0000, Jayanathan, David wrote: Hi Ray,

We allow all management activity to go through the same interfaces that data goes through. Only on our Cluster Mode implementations do we use e0M for management ports, otherwise we only plug in e0M in order to have the RLM/SP connections on the network.

Hope this helps!

-David

Thanks, David.

We are running cluster mode on these systems (two controllers).

Key thing we're seeing is that NFS/CIFS file serving activity grinds to a halt when we lose connectivity to AD/NIS servers (the communication for which currently goes through e0M due to default gateway setup).

I presume this is expected behavior, but am a bit surprised that NFS file serving would hang for activity not involving an NIS UID. Presumably any NFS requests gets mapped to a UID and would require an NIS lookup?

Anyone run into this before?

Ray

-----Original Message----- From: toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Ray Van Dolson Sent: Thursday, August 09, 2012 2:26 PM To: toasters@teaparty.netmailto:toasters@teaparty.net Subject: e0M port as default gateway?

Hello all;

We have some N6240's (rebadged FAS3xxx series) on which the default gateway is set to an IP that routes out the e0M interface.

As such, if the e0M port (wrench port) goes offline, essentially functionality on that controller ceases as communication with AD, NIS, etc. stops working.

We have 10GbE ports on these devices -- we could easily configure the default gateways to use one of those instead, but I am unclear as to what best practice is here. We access management functionality through the IP bound on the e0M port now.

How do the rest of you set things up?

Ray _______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

-- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.