Don't worry about it, I will try on Monday with new servers that have both broadcom and intel NICs in them and are what through out I get, are if that has any effect.
Are the NICs in the servers you are testing with, inbuilt, PCI or PCI express ? And what manufacturer ?
Cheers
Matt
-----Original Message----- From: Langdon, Laughlin T. (Lock) Langdon.Lock@mayo.edu To: Davies,Matt; slinkywizard@integra.net slinkywizard@integra.net CC: jack1729@gmail.com jack1729@gmail.com; chetan.ganjihal@netapp.com chetan.ganjihal@netapp.com; sgaroutte@gmail.com sgaroutte@gmail.com; ggwalker@mindspring.com ggwalker@mindspring.com; toasters@mathworks.com toasters@mathworks.com Sent: Fri Mar 30 22:19:26 2007 Subject: RE: CIFS overhead with Netapp Filers
srry.. zipped that part of your post :)
The weird thing is that you are getting the same speed (low end) for server to server that I get to my filer 250Mbps.. but I get upwards of 500-600Mbps from server to server..
________________________________
From: Davies,Matt [mailto:MDAVIES@generalatlantic.com] Sent: Fri 3/30/2007 1:57 PM To: Langdon, Laughlin T. (Lock); slinkywizard@integra.net Cc: jack1729@gmail.com; chetan.ganjihal@netapp.com; sgaroutte@gmail.com; ggwalker@mindspring.com; toasters@mathworks.com Subject: Re: CIFS overhead with Netapp Filers
Did that, same speeds, sorry......
Matt
-----Original Message----- From: Langdon, Laughlin T. (Lock) Langdon.Lock@mayo.edu To: Davies,Matt; slinkywizard@integra.net slinkywizard@integra.net CC: jack1729@gmail.com jack1729@gmail.com; chetan.ganjihal@netapp.com chetan.ganjihal@netapp.com; sgaroutte@gmail.com sgaroutte@gmail.com; ggwalker@mindspring.com ggwalker@mindspring.com; toasters@mathworks.com toasters@mathworks.com Sent: Fri Mar 30 21:50:27 2007 Subject: RE: CIFS overhead with Netapp Filers
can you take that same file and copy it from a server to a server on the same network etc?
________________________________
From: Davies,Matt [mailto:MDAVIES@generalatlantic.com] Sent: Fri 3/30/2007 1:16 PM To: Langdon, Laughlin T. (Lock); slinkywizard@integra.net Cc: jack1729@gmail.com; chetan.ganjihal@netapp.com; sgaroutte@gmail.com; ggwalker@mindspring.com; toasters@mathworks.com Subject: RE: CIFS overhead with Netapp Filers
In the interim, I decided to unbox and startup our new FAS270 and quickly set it up. With just a single nic
I have taken a 500MB zip file, and copied from server A to the filer and from server A to another server. Both servers are running single Nics
Peak rate was around 250Mb/s, average rate was around 130Mb/s to both the filer and the server.
The filer is running 7.0.5
I have made no changes to TCP window size on either the servers or the filers, the servers are Win2k3 SP1.
I also did the same test on our production FAS250 and got the same result, which is running 7.1
Sorry I haven’t been much help
Cheers
Matt
From: Langdon, Laughlin T. (Lock) [mailto:Langdon.Lock@mayo.edu] Sent: 30 March 2007 20:40 To: Davies,Matt; slinkywizard@integra.net Cc: jack1729@gmail.com; chetan.ganjihal@netapp.com; sgaroutte@gmail.com; ggwalker@mindspring.com; toasters@mathworks.com Subject: RE: CIFS overhead with Netapp Filers
:) Already been down that path.. Heck I've gone single port, with a crossover cable.. same reuslts.. That's why I'm here.. to see if we're the only ones experiencing this, or if it's just what NetaApp does (I hope not ;)
________________________________
From: Davies,Matt [mailto:MDAVIES@generalatlantic.com] Sent: Fri 3/30/2007 12:38 PM To: Langdon, Laughlin T. (Lock); slinkywizard@integra.net Cc: jack1729@gmail.com; chetan.ganjihal@netapp.com; sgaroutte@gmail.com; ggwalker@mindspring.com; toasters@mathworks.com Subject: RE: CIFS overhead with Netapp Filers
Shame, that might have answered the problem.
Have you tried just using 1 ethernet port on the filer ? Have you tried using ports on the same switch, I have seen some problems with cross stack etherchannels, usually if the stack isn’t connected correctly.
The following command will give you the details.
UKswitchstack#sh switch stack-ring speed
Stack Ring Speed : 32G
Stack Ring Configuration: Full
Stack Ring Protocol : StackWise
UKswitchstack#
My new filer arrived today, but will try and reproduce on Monday
But just so you know:
After enabling QOS on cat3750, certain application (mostly bursty and TCP based) may experience significant performance degradation. This issue is resolved starting Cisco IOS 12.2(25)SEE1.
With the changes in the new release, the following global config. commands are still required :
------------------------------------------------------------------------------
mls qos queue-set output 1 threshold 2 3200 3200 100 3200
mls qos queue-set output 1 threshold 3 3200 3200 100 3200
------------------------------------------------------------------------------
Cheers
Matt
From: Langdon, Laughlin T. (Lock) [mailto:Langdon.Lock@mayo.edu] Sent: 30 March 2007 20:17 To: Davies,Matt; slinkywizard@integra.net Cc: jack1729@gmail.com; chetan.ganjihal@netapp.com; sgaroutte@gmail.com; ggwalker@mindspring.com; toasters@mathworks.com Subject: RE: CIFS overhead with Netapp Filers
sorry, Yes I meant 3750. No QOS.
________________________________
From: Davies,Matt [mailto:MDAVIES@generalatlantic.com] Sent: Fri 3/30/2007 11:47 AM To: Langdon, Laughlin T. (Lock); slinkywizard@integra.net Cc: jack1729@gmail.com; chetan.ganjihal@netapp.com; sgaroutte@gmail.com; ggwalker@mindspring.com; toasters@mathworks.com Subject: Re: CIFS overhead with Netapp Filers
I presume you mean the 3750 ?
Is QoS enabled on the switch ?
-----Original Message----- From: Langdon, Laughlin T. (Lock) Langdon.Lock@mayo.edu To: Davies,Matt; slinkywizard@integra.net slinkywizard@integra.net CC: Jack Lyons jack1729@gmail.com; Ganjihal, Chetan chetan.ganjihal@netapp.com; Shane Garoutte sgaroutte@gmail.com; Glenn Walker ggwalker@mindspring.com; toasters@mathworks.com toasters@mathworks.com Sent: Fri Mar 30 19:37:47 2007 Subject: RE: CIFS overhead with Netapp Filers
2 Cisco 3950s cross connected in the back. Should be configed for LACP, 3 ports on one switch 3 ports on the other switch.
________________________________
From: Davies,Matt [mailto:MDAVIES@generalatlantic.com] Sent: Thu 3/29/2007 1:34 PM To: Langdon, Laughlin T. (Lock); slinkywizard@integra.net Cc: Jack Lyons; Ganjihal, Chetan; Shane Garoutte; Glenn Walker; toasters@mathworks.com Subject: RE: CIFS overhead with Netapp Filers
May be a stupid question, but what network switch are you using ?
Cheers
Matt
________________________________
From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Langdon, Laughlin T. (Lock) Sent: 29 March 2007 21:00 To: slinkywizard@integra.net Cc: Jack Lyons; Ganjihal, Chetan; Shane Garoutte; Glenn Walker; toasters@mathworks.com Subject: RE: CIFS overhead with Netapp Filers
yes, that all looks true
Aggregate is maxed out size of 39 disks Raid DP. I’ve also tried 10k FC drives, and 7.5k SATA drives. Same performance.
no jumbo frames. No cifs tuning.
can anyone else try this? take a large file and copy it from a windows server to a windows server. then copy the sam file from the same server to a netapp filer. you can use the nul trigger to really isolate network usage (doesn;t actually write the file) What Mbps do you get on each xfer?
I get approx 500 mbps to a windows server, and 250 mbps to the filer. same file, same tcp window size, same server.. only delta is once to server once to filer.
If you want to you can also do a read from the filer and the server to get the read performance. ________________________________
From: slinkymax0r [mailto:slinkywizard@integra.net] Sent: Thu 3/29/2007 10:48 AM To: Langdon, Laughlin T. (Lock) Cc: Jack Lyons; Ganjihal, Chetan; Shane Garoutte; Glenn Walker; toasters@mathworks.com Subject: RE: CIFS overhead with Netapp Filers
my previous posts mention that I'm running 7.2.1.. now I've updated to 7.2.1.1
Snapshot of thie issue:
User is running windows 2003. Throughput ramps up with multiple copies, just doesn't seem to "fill the pipe" with a single file copy operation.
No network errors, TCP and lower level tuning on the stack hasn't improved the issue.* (Any Netdiag output to share?)
No idea if CIFs buffer tuning has been done on the filer, per NetApp CIFS troubleshooting instructions.
No info on Aggregate Layout or volume options (Might be helpful).
No idea if jumbo frames are part of the mix.
Accurate so far?
Regards, Max
____________________________________________________________
This e-mail (including all attachments) is confidential and may be privileged. It is for the exclusive use of the addressee only. If you are not the addressee, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please erase all copies of the message and its attachments and notify us immediately at help@generalatlantic.com. Thank You.
built-in broadcom TOE disabled Dell 2950
________________________________
From: Davies,Matt [mailto:MDAVIES@generalatlantic.com] Sent: Fri 3/30/2007 2:28 PM To: Langdon, Laughlin T. (Lock); slinkywizard@integra.net Cc: jack1729@gmail.com; chetan.ganjihal@netapp.com; sgaroutte@gmail.com; ggwalker@mindspring.com; toasters@mathworks.com Subject: Re: CIFS overhead with Netapp Filers
Don't worry about it, I will try on Monday with new servers that have both broadcom and intel NICs in them and are what through out I get, are if that has any effect.
Are the NICs in the servers you are testing with, inbuilt, PCI or PCI express ? And what manufacturer ?
Cheers
Matt
-----Original Message----- From: Langdon, Laughlin T. (Lock) Langdon.Lock@mayo.edu To: Davies,Matt; slinkywizard@integra.net slinkywizard@integra.net CC: jack1729@gmail.com jack1729@gmail.com; chetan.ganjihal@netapp.com chetan.ganjihal@netapp.com; sgaroutte@gmail.com sgaroutte@gmail.com; ggwalker@mindspring.com ggwalker@mindspring.com; toasters@mathworks.com toasters@mathworks.com Sent: Fri Mar 30 22:19:26 2007 Subject: RE: CIFS overhead with Netapp Filers
srry.. zipped that part of your post :)
The weird thing is that you are getting the same speed (low end) for server to server that I get to my filer 250Mbps.. but I get upwards of 500-600Mbps from server to server..
________________________________
From: Davies,Matt [mailto:MDAVIES@generalatlantic.com] Sent: Fri 3/30/2007 1:57 PM To: Langdon, Laughlin T. (Lock); slinkywizard@integra.net Cc: jack1729@gmail.com; chetan.ganjihal@netapp.com; sgaroutte@gmail.com; ggwalker@mindspring.com; toasters@mathworks.com Subject: Re: CIFS overhead with Netapp Filers
Did that, same speeds, sorry......
Matt
-----Original Message----- From: Langdon, Laughlin T. (Lock) Langdon.Lock@mayo.edu To: Davies,Matt; slinkywizard@integra.net slinkywizard@integra.net CC: jack1729@gmail.com jack1729@gmail.com; chetan.ganjihal@netapp.com chetan.ganjihal@netapp.com; sgaroutte@gmail.com sgaroutte@gmail.com; ggwalker@mindspring.com ggwalker@mindspring.com; toasters@mathworks.com toasters@mathworks.com Sent: Fri Mar 30 21:50:27 2007 Subject: RE: CIFS overhead with Netapp Filers
can you take that same file and copy it from a server to a server on the same network etc?
________________________________
From: Davies,Matt [mailto:MDAVIES@generalatlantic.com] Sent: Fri 3/30/2007 1:16 PM To: Langdon, Laughlin T. (Lock); slinkywizard@integra.net Cc: jack1729@gmail.com; chetan.ganjihal@netapp.com; sgaroutte@gmail.com; ggwalker@mindspring.com; toasters@mathworks.com Subject: RE: CIFS overhead with Netapp Filers
In the interim, I decided to unbox and startup our new FAS270 and quickly set it up. With just a single nic
I have taken a 500MB zip file, and copied from server A to the filer and from server A to another server. Both servers are running single Nics
Peak rate was around 250Mb/s, average rate was around 130Mb/s to both the filer and the server.
The filer is running 7.0.5
I have made no changes to TCP window size on either the servers or the filers, the servers are Win2k3 SP1.
I also did the same test on our production FAS250 and got the same result, which is running 7.1
Sorry I haven't been much help
Cheers
Matt
From: Langdon, Laughlin T. (Lock) [mailto:Langdon.Lock@mayo.edu] Sent: 30 March 2007 20:40 To: Davies,Matt; slinkywizard@integra.net Cc: jack1729@gmail.com; chetan.ganjihal@netapp.com; sgaroutte@gmail.com; ggwalker@mindspring.com; toasters@mathworks.com Subject: RE: CIFS overhead with Netapp Filers
:) Already been down that path.. Heck I've gone single port, with a crossover cable.. same reuslts.. That's why I'm here.. to see if we're the only ones experiencing this, or if it's just what NetaApp does (I hope not ;)
________________________________
From: Davies,Matt [mailto:MDAVIES@generalatlantic.com] Sent: Fri 3/30/2007 12:38 PM To: Langdon, Laughlin T. (Lock); slinkywizard@integra.net Cc: jack1729@gmail.com; chetan.ganjihal@netapp.com; sgaroutte@gmail.com; ggwalker@mindspring.com; toasters@mathworks.com Subject: RE: CIFS overhead with Netapp Filers
Shame, that might have answered the problem.
Have you tried just using 1 ethernet port on the filer ? Have you tried using ports on the same switch, I have seen some problems with cross stack etherchannels, usually if the stack isn't connected correctly.
The following command will give you the details.
UKswitchstack#sh switch stack-ring speed
Stack Ring Speed : 32G
Stack Ring Configuration: Full
Stack Ring Protocol : StackWise
UKswitchstack#
My new filer arrived today, but will try and reproduce on Monday
But just so you know:
After enabling QOS on cat3750, certain application (mostly bursty and TCP based) may experience significant performance degradation. This issue is resolved starting Cisco IOS 12.2(25)SEE1.
With the changes in the new release, the following global config. commands are still required :
------------------------------------------------------------------------------
mls qos queue-set output 1 threshold 2 3200 3200 100 3200
mls qos queue-set output 1 threshold 3 3200 3200 100 3200
------------------------------------------------------------------------------
Cheers
Matt
From: Langdon, Laughlin T. (Lock) [mailto:Langdon.Lock@mayo.edu] Sent: 30 March 2007 20:17 To: Davies,Matt; slinkywizard@integra.net Cc: jack1729@gmail.com; chetan.ganjihal@netapp.com; sgaroutte@gmail.com; ggwalker@mindspring.com; toasters@mathworks.com Subject: RE: CIFS overhead with Netapp Filers
sorry, Yes I meant 3750. No QOS.
________________________________
From: Davies,Matt [mailto:MDAVIES@generalatlantic.com] Sent: Fri 3/30/2007 11:47 AM To: Langdon, Laughlin T. (Lock); slinkywizard@integra.net Cc: jack1729@gmail.com; chetan.ganjihal@netapp.com; sgaroutte@gmail.com; ggwalker@mindspring.com; toasters@mathworks.com Subject: Re: CIFS overhead with Netapp Filers
I presume you mean the 3750 ?
Is QoS enabled on the switch ?
-----Original Message----- From: Langdon, Laughlin T. (Lock) Langdon.Lock@mayo.edu To: Davies,Matt; slinkywizard@integra.net slinkywizard@integra.net CC: Jack Lyons jack1729@gmail.com; Ganjihal, Chetan chetan.ganjihal@netapp.com; Shane Garoutte sgaroutte@gmail.com; Glenn Walker ggwalker@mindspring.com; toasters@mathworks.com toasters@mathworks.com Sent: Fri Mar 30 19:37:47 2007 Subject: RE: CIFS overhead with Netapp Filers
2 Cisco 3950s cross connected in the back. Should be configed for LACP, 3 ports on one switch 3 ports on the other switch.
________________________________
From: Davies,Matt [mailto:MDAVIES@generalatlantic.com] Sent: Thu 3/29/2007 1:34 PM To: Langdon, Laughlin T. (Lock); slinkywizard@integra.net Cc: Jack Lyons; Ganjihal, Chetan; Shane Garoutte; Glenn Walker; toasters@mathworks.com Subject: RE: CIFS overhead with Netapp Filers
May be a stupid question, but what network switch are you using ?
Cheers
Matt
________________________________
From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Langdon, Laughlin T. (Lock) Sent: 29 March 2007 21:00 To: slinkywizard@integra.net Cc: Jack Lyons; Ganjihal, Chetan; Shane Garoutte; Glenn Walker; toasters@mathworks.com Subject: RE: CIFS overhead with Netapp Filers
yes, that all looks true
Aggregate is maxed out size of 39 disks Raid DP. I've also tried 10k FC drives, and 7.5k SATA drives. Same performance.
no jumbo frames. No cifs tuning.
can anyone else try this? take a large file and copy it from a windows server to a windows server. then copy the sam file from the same server to a netapp filer. you can use the nul trigger to really isolate network usage (doesn;t actually write the file) What Mbps do you get on each xfer?
I get approx 500 mbps to a windows server, and 250 mbps to the filer. same file, same tcp window size, same server.. only delta is once to server once to filer.
If you want to you can also do a read from the filer and the server to get the read performance. ________________________________
From: slinkymax0r [mailto:slinkywizard@integra.net] Sent: Thu 3/29/2007 10:48 AM To: Langdon, Laughlin T. (Lock) Cc: Jack Lyons; Ganjihal, Chetan; Shane Garoutte; Glenn Walker; toasters@mathworks.com Subject: RE: CIFS overhead with Netapp Filers
my previous posts mention that I'm running 7.2.1.. now I've updated to 7.2.1.1
Snapshot of thie issue:
User is running windows 2003. Throughput ramps up with multiple copies, just doesn't seem to "fill the pipe" with a single file copy operation.
No network errors, TCP and lower level tuning on the stack hasn't improved the issue.* (Any Netdiag output to share?)
No idea if CIFs buffer tuning has been done on the filer, per NetApp CIFS troubleshooting instructions.
No info on Aggregate Layout or volume options (Might be helpful).
No idea if jumbo frames are part of the mix.
Accurate so far?
Regards, Max
____________________________________________________________
This e-mail (including all attachments) is confidential and may be privileged. It is for the exclusive use of the addressee only. If you are not the addressee, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please erase all copies of the message and its attachments and notify us immediately at help@generalatlantic.com. Thank You.