Hi Matt
I'm still trying to fully understand the new functionailty of the ESX3.5 SW iSCSI stack, but it seems they've made some improvements in MCS (Multi Connection Sessions) or some other technology that allows the iSCSI SW stack to be aware of and fail over between paths.
 
I'm 90% sure of this so far, but ...
 
I would leave the paths as they are.  You can use tpgroups (documented in the man pages and Block Access Guide) on the filer to restrict iSCSI traffic to certain ports for certain igroups, but I think you're better off leaving the other path as a failover.  In ESX, set the preferred path of the LUN(s) to go through the most direct network or the back-end / storage network if that's what you have.
 
esxcfg-mpath -l on the CLI should show you similar info to what you see in the VI Client.  vmkiscsi-ls (one word / no space) will list all the target portals that iSCSI knows about.  They should look like this:  (I've snipped some other junk)
[root@esx1 mcs_test]# esxcfg-mpath -l
Disk vmhba32:2:105 /dev/sdg (10240MB) has 2 paths and policy of Fixed
 iScsi sw iqn.1998-01.com.vmware:esx1-2e0f4af9<->iqn.1992-08.com.netapp:sn.118042901 vmhba32:2:105 On active preferred
 iScsi sw iqn.1998-01.com.vmware:esx1-2e0f4af9<->iqn.1992-08.com.netapp:sn.118042901 vmhba32:4:105 On
 
[root@esx1 mcs_test]# vmkiscsi-ls
*******************************************************************************
        SFNet iSCSI Driver Version ... 3.6.3 (27-Jun-2005 )
*******************************************************************************
TARGET NAME             : iqn.1992-08.com.netapp:sn.118042901
TARGET ALIAS            :
HOST NO                 : 3
BUS NO                  : 0
TARGET ID               : 2
TARGET ADDRESS          : 192.168.42.61:3260
SESSION STATUS          : ESTABLISHED AT Thu Dec 20 19:43:21 2007
NO. OF PORTALS          : 1
PORTAL ADDRESS 1        : 192.168.42.61:3260,1001
SESSION ID              : ISID 00023d000001 TSIH 0e
*******************************************************************************
TARGET NAME             : iqn.1992-08.com.netapp:sn.118042901
TARGET ALIAS            :
HOST NO                 : 3
BUS NO                  : 0
TARGET ID               : 3
TARGET ADDRESS          : 10.41.87.61:3260
SESSION STATUS          : NOT ESTABLISHED
NO. OF PORTALS          : 1
PORTAL ADDRESS 1        : 10.41.87.61:3260,1000
SESSION ID              : ISID 00023d000002 TSIH 00
*******************************************************************************
TARGET NAME             : iqn.1992-08.com.netapp:sn.118042901
TARGET ALIAS            :
HOST NO                 : 3
BUS NO                  : 0
TARGET ID               : 4
TARGET ADDRESS          : 192.168.42.62:3260
SESSION STATUS          : ESTABLISHED AT Thu Dec 20 19:47:54 2007
NO. OF PORTALS          : 1
PORTAL ADDRESS 1        : 192.168.42.62:3260,1003
SESSION ID              : ISID 00023d000003 TSIH 11
*******************************************************************************
 
Note the TARGET ID for the portals listed as ESTABLISHED will match the target ID in the vmhba LUN name, in my case, 2 and 4.  So from the TARGET ID, you can get the corresponding IP address, and that gives you the preferred target.  You use esxcfg-mpath to set the preferred path:
 
[root@esx1 mcs_test]# esxcfg-mpath -L vmhba32:2:105 -P vmhba32:4:105 -f -p fixed
Setting vmhba32:2:105 -- vmhba32:4:105 as preferred path
[root@esx1 mcs_test]# esxcfg-mpath -q -L vmhba32:2:105
Disk vmhba32:2:105 /dev/sdg (10240MB) has 2 paths and policy of Fixed
 iScsi sw iqn.1998-01.com.vmware:esx1-2e0f4af9<->iqn.1992-08.com.netapp:sn.118042901 vmhba32:2:105 On
 iScsi sw iqn.1998-01.com.vmware:esx1-2e0f4af9<->iqn.1992-08.com.netapp:sn.118042901 vmhba32:4:105 On active preferred
 
Your iSCSI traffic will use the storage network, until it breaks, at which time I/O (and possibly more) will hang for ~30 seconds while ESX figures out that the path failed and fails over to the other target on the other network.  I tested that by downing the preferred interface, and the service console hung for a few seconds, then picked back up, and esxcfg-mapth showed the pref path dead, and active on the other path.  vmkiscsi-ls showed the session status as "Dropped" with a timestamp.
 
I have not yet played with the new path policies, so there may be a better option than preferred and fixed.  I'm also wondering if this might lead to load balancing in iSCSI...
 
Please let us know how this goes for you!
 
Thanks
 
Peter


From: Davies,Matt [mailto:MDAVIES@generalatlantic.com]
Sent: Thursday, December 20, 2007 2:07 AM
To: toasters@mathworks.com
Subject: Netapp and VMware ESX 3.5

Hello all,

 

Having decided to upgrade our VMware ESX servers to 3.5 from 3.0.2, we have come across a strange problem.

 

On each ESX server we have 3 VSwitches, 2 of them each contain a service console port and VMkernel port. One of Vswitches we have been using for Iscsi in ESX 3.0.2 without any problems. (The first VMkernel port is enabled for Vmotion and is routable)

 

Once we upgraded to 3.5 we have noticed that the filer now shows 2 sessions from the esx host, one from each of the VMkernel ports, however one of the paths is routed and one of the paths is switched. On the ESX host we see each LUN twice, with different paths but with the same canonical path.

 

Does anyone know how to restrict what targets are being sent by the filer based on the interface ? We cannot disable Iscsi on any of the interfaces on the filer.

 

Or does anyone have any other bright ideas ?

 

 

Thanks

 

Matt

 

 

____________________________

Matt Davies

Director of International IT Operations

General Atlantic

83 Pall Mall

London

SW1Y 5ES

 

Tel: +44 207 484 3203

Fax: +44 207 484 2803

Mobile: +44 777 559 4265

____________________________

 

 



____________________________________________________________

This e-mail (including all attachments) is confidential and may be privileged. 
It is for the exclusive use of the addressee only.  If you are not the addressee,
you are hereby notified that any dissemination of this communication is strictly
prohibited. If you have received this communication in error, please erase all
copies of the message and its attachments and notify us immediately at
help@generalatlantic.com.  Thank You.