Peter,
Thanks for the info, it seems
that Vmware have started using a different Iscsi initiator, on ESX 3.5
(in 3.0.2 it was a Cisco Driver, and in 3.5 is the SFnet driver)
It seems to have picked the preferred
path via the directly connected network, rather than the routed network. The
main cause of concern at the moment is that it appears to be establishing a
ISCSI session from the VMkernel port associated with Vmotion.
We have restricted the traffic
on the router, to stop this from occurring that the moment.
If you come up with any more
information on what has improved or not improved with Iscsi in ESX 3.5 when
working with Netapp, then please let us know, as it appears even VMware don’t
know.
Thanks
Matt
From: Learmonth, Peter
[mailto:Peter.Learmonth@netapp.com]
Sent: 21 December 2007 04:23
To: Davies,Matt; toasters@mathworks.com
Subject: RE: Netapp and VMware ESX 3.5
Hi Matt
I'm still trying to fully understand the new functionailty of the
ESX3.5 SW iSCSI stack, but it seems they've made some improvements in MCS
(Multi Connection Sessions) or some other technology that allows the iSCSI SW
stack to be aware of and fail over between paths.
I'm 90% sure of this so far, but ...
I would leave the paths as they are. You can
use tpgroups (documented in the man pages and Block Access Guide) on
the filer to restrict iSCSI traffic to certain ports for certain igroups, but I
think you're better off leaving the other path as a failover. In ESX, set
the preferred path of the LUN(s) to go through the most direct network or the
back-end / storage network if that's what you have.
esxcfg-mpath -l on the CLI should show you similar info to what you
see in the VI Client. vmkiscsi-ls (one word / no space) will list all the
target portals that iSCSI knows about. They should look like this:
(I've snipped some other junk)
[root@esx1 mcs_test]# esxcfg-mpath -l
Disk vmhba32:2:105 /dev/sdg (10240MB) has 2 paths and policy of Fixed
iScsi sw
iqn.1998-01.com.vmware:esx1-2e0f4af9<->iqn.1992-08.com.netapp:sn.118042901
vmhba32:2:105 On active preferred
iScsi sw
iqn.1998-01.com.vmware:esx1-2e0f4af9<->iqn.1992-08.com.netapp:sn.118042901
vmhba32:4:105 On
[root@esx1 mcs_test]# vmkiscsi-ls
*******************************************************************************
SFNet iSCSI Driver Version .. 3.6.3
(27-Jun-2005 )
*******************************************************************************
TARGET
NAME :
iqn.1992-08.com.netapp:sn.118042901
TARGET ALIAS
:
HOST
NO
: 3
BUS
NO
: 0
TARGET
ID
: 2
TARGET ADDRESS :
192.168.42.61:3260
SESSION STATUS :
ESTABLISHED AT Thu Dec 20 19:43:21 2007
NO. OF PORTALS : 1
PORTAL ADDRESS 1 :
192.168.42.61:3260,1001
SESSION
ID
: ISID 00023d000001 TSIH 0e
*******************************************************************************
TARGET
NAME :
iqn.1992-08.com.netapp:sn.118042901
TARGET ALIAS
:
HOST
NO
: 3
BUS
NO
: 0
TARGET
ID
: 3
TARGET ADDRESS :
10.41.87.61:3260
SESSION STATUS : NOT
ESTABLISHED
NO. OF PORTALS : 1
PORTAL ADDRESS 1 :
10.41.87.61:3260,1000
SESSION
ID
: ISID 00023d000002 TSIH 00
*******************************************************************************
TARGET
NAME :
iqn.1992-08.com.netapp:sn.118042901
TARGET ALIAS
:
HOST NO
: 3
BUS
NO
: 0
TARGET
ID
: 4
TARGET ADDRESS :
192.168.42.62:3260
SESSION STATUS :
ESTABLISHED AT Thu Dec 20 19:47:54 2007
NO. OF PORTALS : 1
PORTAL ADDRESS 1 : 192.168.42.62:3260,1003
SESSION
ID
: ISID 00023d000003 TSIH 11
*******************************************************************************
Note the TARGET ID for the portals listed as ESTABLISHED will match
the target ID in the vmhba LUN name, in my case, 2 and 4. So from the
TARGET ID, you can get the corresponding IP address, and that gives you the
preferred target. You use esxcfg-mpath to set the preferred path:
[root@esx1 mcs_test]# esxcfg-mpath -L vmhba32:2:105 -P
vmhba32:4:105 -f -p fixed
Setting vmhba32:2:105 -- vmhba32:4:105 as preferred path
[root@esx1 mcs_test]# esxcfg-mpath -q -L vmhba32:2:105
Disk vmhba32:2:105 /dev/sdg (10240MB) has 2 paths and policy of Fixed
iScsi sw
iqn.1998-01.com.vmware:esx1-2e0f4af9<->iqn.1992-08.com.netapp:sn.118042901
vmhba32:2:105 On
iScsi sw
iqn.1998-01.com.vmware:esx1-2e0f4af9<->iqn.1992-08.com.netapp:sn.118042901
vmhba32:4:105 On active preferred
Your iSCSI traffic will use the storage network, until it breaks,
at which time I/O (and possibly more) will hang for ~30 seconds while ESX
figures out that the path failed and fails over to the other target on the
other network. I tested that by downing the preferred interface, and the
service console hung for a few seconds, then picked back up, and esxcfg-mapth showed
the pref path dead, and active on the other path. vmkiscsi-ls showed the
session status as "Dropped" with a timestamp.
I have not yet played with the new path policies, so there may be a
better option than preferred and fixed. I'm also wondering if this might
lead to load balancing in iSCSI...
Please let us know how this goes for you!
Thanks
Peter
From:
Davies,Matt [mailto:MDAVIES@generalatlantic.com]
Sent: Thursday, December 20, 2007 2:07 AM
To: toasters@mathworks.com
Subject: Netapp and VMware ESX 3.5
Hello all,
Having decided to upgrade our
VMware ESX servers to 3.5 from 3.0.2, we have come across a strange problem.
On each ESX server we have 3
VSwitches, 2 of them each contain a service console port and VMkernel port. One
of Vswitches we have been using for Iscsi in ESX 3.0.2 without any problems.
(The first VMkernel port is enabled for Vmotion and is routable)
Once we upgraded to 3.5 we have
noticed that the filer now shows 2 sessions from the esx host, one from each of
the VMkernel ports, however one of the paths is routed and one of the paths is
switched. On the ESX host we see each LUN twice, with different paths but with
the same canonical path.
Does anyone know how to restrict
what targets are being sent by the filer based on the interface ? We cannot
disable Iscsi on any of the interfaces on the filer.
Or does anyone have any other
bright ideas ?
Thanks
Matt
____________________________
Matt Davies
Director of
International IT Operations
General Atlantic
83 Pall Mall
London
SW1Y 5ES
Tel: +44 207 484
3203
Fax: +44 207 484
2803
Mobile: +44 777
559 4265
____________________________
____________________________________________________________
This e-mail (including all attachments) is
confidential and may be privileged.
It is for the exclusive use of the addressee only. If you are not the
addressee,
you are hereby notified that any dissemination of this communication is
strictly
prohibited. If you have received this communication in error, please erase all
copies of the message and its attachments and notify us immediately at
help@generalatlantic.com.
Thank You.