Hi Matt I'm still trying to fully understand the new functionailty of the ESX3.5 SW iSCSI stack, but it seems they've made some improvements in MCS (Multi Connection Sessions) or some other technology that allows the iSCSI SW stack to be aware of and fail over between paths.
I'm 90% sure of this so far, but ...
I would leave the paths as they are. You can use tpgroups (documented in the man pages and Block Access Guide) on the filer to restrict iSCSI traffic to certain ports for certain igroups, but I think you're better off leaving the other path as a failover. In ESX, set the preferred path of the LUN(s) to go through the most direct network or the back-end / storage network if that's what you have.
esxcfg-mpath -l on the CLI should show you similar info to what you see in the VI Client. vmkiscsi-ls (one word / no space) will list all the target portals that iSCSI knows about. They should look like this: (I've snipped some other junk) [root@esx1 mcs_test]# esxcfg-mpath -l Disk vmhba32:2:105 /dev/sdg (10240MB) has 2 paths and policy of Fixed iScsi sw iqn.1998-01.com.vmware:esx1-2e0f4af9<->iqn.1992-08.com.netapp:sn.1180429 01 vmhba32:2:105 On active preferred iScsi sw iqn.1998-01.com.vmware:esx1-2e0f4af9<->iqn.1992-08.com.netapp:sn.1180429 01 vmhba32:4:105 On
[root@esx1 mcs_test]# vmkiscsi-ls ************************************************************************ ******* SFNet iSCSI Driver Version ... 3.6.3 (27-Jun-2005 ) ************************************************************************ ******* TARGET NAME : iqn.1992-08.com.netapp:sn.118042901 TARGET ALIAS : HOST NO : 3 BUS NO : 0 TARGET ID : 2 TARGET ADDRESS : 192.168.42.61:3260 SESSION STATUS : ESTABLISHED AT Thu Dec 20 19:43:21 2007 NO. OF PORTALS : 1 PORTAL ADDRESS 1 : 192.168.42.61:3260,1001 SESSION ID : ISID 00023d000001 TSIH 0e ************************************************************************ ******* TARGET NAME : iqn.1992-08.com.netapp:sn.118042901 TARGET ALIAS : HOST NO : 3 BUS NO : 0 TARGET ID : 3 TARGET ADDRESS : 10.41.87.61:3260 SESSION STATUS : NOT ESTABLISHED NO. OF PORTALS : 1 PORTAL ADDRESS 1 : 10.41.87.61:3260,1000 SESSION ID : ISID 00023d000002 TSIH 00 ************************************************************************ ******* TARGET NAME : iqn.1992-08.com.netapp:sn.118042901 TARGET ALIAS : HOST NO : 3 BUS NO : 0 TARGET ID : 4 TARGET ADDRESS : 192.168.42.62:3260 SESSION STATUS : ESTABLISHED AT Thu Dec 20 19:47:54 2007 NO. OF PORTALS : 1 PORTAL ADDRESS 1 : 192.168.42.62:3260,1003 SESSION ID : ISID 00023d000003 TSIH 11 ************************************************************************ *******
Note the TARGET ID for the portals listed as ESTABLISHED will match the target ID in the vmhba LUN name, in my case, 2 and 4. So from the TARGET ID, you can get the corresponding IP address, and that gives you the preferred target. You use esxcfg-mpath to set the preferred path:
[root@esx1 mcs_test]# esxcfg-mpath -L vmhba32:2:105 -P vmhba32:4:105 -f -p fixed Setting vmhba32:2:105 -- vmhba32:4:105 as preferred path
[root@esx1 mcs_test]# esxcfg-mpath -q -L vmhba32:2:105 Disk vmhba32:2:105 /dev/sdg (10240MB) has 2 paths and policy of Fixed iScsi sw iqn.1998-01.com.vmware:esx1-2e0f4af9<->iqn.1992-08.com.netapp:sn.1180429 01 vmhba32:2:105 On iScsi sw iqn.1998-01.com.vmware:esx1-2e0f4af9<->iqn.1992-08.com.netapp:sn.1180429 01 vmhba32:4:105 On active preferred
Your iSCSI traffic will use the storage network, until it breaks, at which time I/O (and possibly more) will hang for ~30 seconds while ESX figures out that the path failed and fails over to the other target on the other network. I tested that by downing the preferred interface, and the service console hung for a few seconds, then picked back up, and esxcfg-mapth showed the pref path dead, and active on the other path. vmkiscsi-ls showed the session status as "Dropped" with a timestamp.
I have not yet played with the new path policies, so there may be a better option than preferred and fixed. I'm also wondering if this might lead to load balancing in iSCSI...
Please let us know how this goes for you!
Thanks
Peter
________________________________
From: Davies,Matt [mailto:MDAVIES@generalatlantic.com] Sent: Thursday, December 20, 2007 2:07 AM To: toasters@mathworks.com Subject: Netapp and VMware ESX 3.5
Hello all,
Having decided to upgrade our VMware ESX servers to 3.5 from 3.0.2, we have come across a strange problem.
On each ESX server we have 3 VSwitches, 2 of them each contain a service console port and VMkernel port. One of Vswitches we have been using for Iscsi in ESX 3.0.2 without any problems. (The first VMkernel port is enabled for Vmotion and is routable)
Once we upgraded to 3.5 we have noticed that the filer now shows 2 sessions from the esx host, one from each of the VMkernel ports, however one of the paths is routed and one of the paths is switched. On the ESX host we see each LUN twice, with different paths but with the same canonical path.
Does anyone know how to restrict what targets are being sent by the filer based on the interface ? We cannot disable Iscsi on any of the interfaces on the filer.
Or does anyone have any other bright ideas ?
Thanks
Matt
____________________________
Matt Davies
Director of International IT Operations
General Atlantic
83 Pall Mall
London
SW1Y 5ES
Tel: +44 207 484 3203
Fax: +44 207 484 2803
Mobile: +44 777 559 4265
____________________________
____________________________________________________________
This e-mail (including all attachments) is confidential and may be privileged. It is for the exclusive use of the addressee only. If you are not the addressee, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please erase all copies of the message and its attachments and notify us immediately at help@generalatlantic.com. Thank You.