We're switching the software iSCSI and don't have the issue. As far as we can tell, it was specific to the QLogic HBAs, but only occurred on a few servers.

________________________________

From: Hadrian Baron [mailto:Hadrian.Baron@vegas.com] Sent: Thursday, January 24, 2008 5:49 PM To: Glenn Walker; toasters@mathworks.com Subject: RE: Cleaning up Messy SW iSCSI connections

It's software based iSCSI, so the normal gigE NIC becomes the HBA. It sounds like a similar issue though.

- Hadrian

From: Glenn Walker [mailto:ggwalker@mindspring.com] Sent: Thursday, January 24, 2008 2:46 PM To: Hadrian Baron; toasters@mathworks.com Subject: RE: Cleaning up Messy SW iSCSI connections

Not using an HBA are you? We had similar problems with the QLogic 4050C HBAs. The TCP session would drop on the HBA and a reset would be required to get it going again.

________________________________

From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Hadrian Baron Sent: Thursday, January 24, 2008 1:28 PM To: toasters@mathworks.com Subject: Cleaning up Messy SW iSCSI connections

Hello Toasters,

We have Snapdrive + MS iSCSI on a few boxes. I've noticed that once in awhile the iSCSI connection will become unavailable. If you browse the iSCSI front-end in Snapdrive, it will show unavailable where it would show the IP addresses for target & portal.

If you browse the iSCSI control panel, it shows a target that is "reconnecting", but you cannot disconnect it to properly re-establish the connection.

Does anyone have a way to properly clean up these connections? Typically disabling iSCSI / Snapdrive services, bouncing the box, and turning them up will let me re-establish, but this is painful.

If anyone is wondering, it's Snapdrive 4.2.1 + MS iSCSI 2.05.

Thanks all,

- Hadrian

We're not using MPIO.

We have made the advanced tweaks to the driver (jumbo, auto duplex, enable flow control) and tcpip configuration (disable netbios over tcp, no register of dns, no wins). Usually we don't adjust the priority of the iSCSI network interface unless it's a cluster, but I suppose that is a good idea. Our iSCSI network isn't routed so it'd be pointless for the box to send traffic that way anyways.

Drivers are up to date. I know iSCSI 2.06 came out lately, and Netapp's policy is to use the latest MS install. If the network (or filer even) were to have a blip, iSCSI usually is resilient enough to re-establish the connection automatically. It's the cases that it doesn't re-establish gracefully that we would like to handle without a reboot.

- Hadrian

From: Klise, Steve [mailto:klises@caminomedical.org] Sent: Thursday, January 24, 2008 3:26 PM To: Hadrian Baron; Glenn Walker; toasters@mathworks.com Subject: RE: Cleaning up Messy SW iSCSI connections

Are you using MPIO? Also, do you have the advanced options setup properly (ISCSI is not set high in the priority list, and the networking services are not enabled). I would make sure the drivers are up to date, and the network is not having any issues..

From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Hadrian Baron Sent: Thursday, January 24, 2008 2:49 PM To: 'Glenn Walker'; toasters@mathworks.com Subject: RE: Cleaning up Messy SW iSCSI connections

It's software based iSCSI, so the normal gigE NIC becomes the HBA. It sounds like a similar issue though.

- Hadrian

From: Glenn Walker [mailto:ggwalker@mindspring.com] Sent: Thursday, January 24, 2008 2:46 PM To: Hadrian Baron; toasters@mathworks.com Subject: RE: Cleaning up Messy SW iSCSI connections

Not using an HBA are you? We had similar problems with the QLogic 4050C HBAs. The TCP session would drop on the HBA and a reset would be required to get it going again.

________________________________ From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Hadrian Baron Sent: Thursday, January 24, 2008 1:28 PM To: toasters@mathworks.com Subject: Cleaning up Messy SW iSCSI connections

Hello Toasters,

We have Snapdrive + MS iSCSI on a few boxes. I've noticed that once in awhile the iSCSI connection will become unavailable. If you browse the iSCSI front-end in Snapdrive, it will show unavailable where it would show the IP addresses for target & portal.

If you browse the iSCSI control panel, it shows a target that is "reconnecting", but you cannot disconnect it to properly re-establish the connection.

Does anyone have a way to properly clean up these connections? Typically disabling iSCSI / Snapdrive services, bouncing the box, and turning them up will let me re-establish, but this is painful.

If anyone is wondering, it's Snapdrive 4.2.1 + MS iSCSI 2.05.

Thanks all,

- Hadrian

Kim,

Thanks for the in-depth answer. We've already made the iscsi disable interface tweak on our filers to ensure sysadmins don't inadvertently establish iSCSI sessions over the LAN links.

Going into the iSCSI control panel and trying to log-off the reconnecting target does nothing. It was the first thing I tried J

- Hadrian

From: Weller, Kim [mailto:Kim.Weller@netapp.com] Sent: Thursday, January 24, 2008 4:41 PM To: Klise, Steve; Hadrian Baron; Glenn Walker; toasters@mathworks.com Subject: RE: Cleaning up Messy SW iSCSI connections

Hi Hadrian,

I've encountered this issue before during testing of cable pulls...This would occur if you had multiple physical interfaces, some on your storage network, some on your public network, etc., and the server couldn't actually get to the storage device on one or more of them. Issue results because of how MS iSCSI Initiator handles reconnecting sessions after a service or network outage. Basically, it cycles through all the available IPs returned from a send targets command to the storage (which you can see in a packet trace if you go to the trouble). It hangs in a state of reconnecting if it gets to an interface that exists on the storage to which there is no path from the server. One way to prevent this behavior is to disable ISCSI on the filer interfaces that you don't actually use for ISCSI traffic.

(Usage: iscsi interface disable { -a | <interface> ... }

There is a way to get it to disconnect: Go into the 'details' of the 'reconnecting' target, checking the box next to the active session (which will probably not be reflected as active on the filer with iscsi session show), and click 'Log Off.' At that point you should be able to refresh the list of targets and log on as normal.

Hope this helps.

Kim Weller

Solution Architect, SAN and Virtualization

Office: 713.968.3621

Mobile: 713.818.4433

________________________________ From: Klise, Steve [mailto:klises@caminomedical.org] Sent: Thursday, January 24, 2008 5:26 PM To: Hadrian Baron; Glenn Walker; toasters@mathworks.com Subject: RE: Cleaning up Messy SW iSCSI connections Are you using MPIO? Also, do you have the advanced options setup properly (ISCSI is not set high in the priority list, and the networking services are not enabled). I would make sure the drivers are up to date, and the network is not having any issues..

From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Hadrian Baron Sent: Thursday, January 24, 2008 2:49 PM To: 'Glenn Walker'; toasters@mathworks.com Subject: RE: Cleaning up Messy SW iSCSI connections

It's software based iSCSI, so the normal gigE NIC becomes the HBA. It sounds like a similar issue though.

- Hadrian

From: Glenn Walker [mailto:ggwalker@mindspring.com] Sent: Thursday, January 24, 2008 2:46 PM To: Hadrian Baron; toasters@mathworks.com Subject: RE: Cleaning up Messy SW iSCSI connections

Not using an HBA are you? We had similar problems with the QLogic 4050C HBAs. The TCP session would drop on the HBA and a reset would be required to get it going again.

________________________________ From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Hadrian Baron Sent: Thursday, January 24, 2008 1:28 PM To: toasters@mathworks.com Subject: Cleaning up Messy SW iSCSI connections

Hello Toasters,

We have Snapdrive + MS iSCSI on a few boxes. I've noticed that once in awhile the iSCSI connection will become unavailable. If you browse the iSCSI front-end in Snapdrive, it will show unavailable where it would show the IP addresses for target & portal.

If you browse the iSCSI control panel, it shows a target that is "reconnecting", but you cannot disconnect it to properly re-establish the connection.

Does anyone have a way to properly clean up these connections? Typically disabling iSCSI / Snapdrive services, bouncing the box, and turning them up will let me re-establish, but this is painful.

If anyone is wondering, it's Snapdrive 4.2.1 + MS iSCSI 2.05.

Thanks all,

- Hadrian