Hi. Wow you're down the street. I have about 250 VMs on 5 servers (IBM x3850m2 with 4 quadcore processors and 64 gig of RAM. We are using Etherchannel to bind two gigE connections to a pair of stacked Cisco 3750 G switches. Then the switches are connected via a 10gig Etherchannel connection to each filer head. So about 50 VMs per server or 25 per gig port.
Here's the results from esxtop's network information. It's got
8:26:08am up 97 days 19:20, 297 worlds; CPU load average: 0.29, 0.31, 0.37
PORT ID UPLINK USED BY DTYP DNAME PKTTX/s MbTX/s PKTRX/s MbRX/s %DRPTX %DRPRX 83886083 Y vmnic6 H vSwitch1 156.40 4.55 0.57 0.00 0.00 0.00 83886085 N 0:vmk-tcpip-172.1.2. H vSwitch1 156.40 4.55 117.49 0.40 0.00 0.00 50331651 Y vmnic5 H vSwitch3 819.40 0.78 854.49 1.45 0.00 0.00 50331746 N 1438:gsovmfmspoll05 H vSwitch3 596.24 0.58 606.16 1.14 0.00 0.00 50331649 Y vmnic1 H vSwitch3 302.31 0.37 388.53 0.29 0.00 0.00 50331690 N 1271:GSOVMFMSPOLL02 H vSwitch3 179.10 0.15 196.46 0.16 0.00 0.00 50331679 N 1249:GSOVMFMSPOLL01 H vSwitch3 129.51 0.14 166.70 0.11 0.00 0.00 50332486 N 1385:gsovmwebsense01 H vSwitch3 120.35 0.13 115.59 0.12 0.00 0.00 50331691 N 1276:GSOVMFMSPOLL03 H vSwitch3 44.06 0.04 57.03 0.05 0.00 0.00 50332424 N 1573:pcresource H vSwitch3 8.77 0.04 11.83 0.01 0.00 0.00 50331747 N 1446:GSOVMFMSPOLL04 H vSwitch3 21.55 0.03 25.56 0.04 0.00 0.00 50332308 N 1470:gsovmdproc1 H vSwitch3 21.17 0.02 24.22 0.09 0.00 0.00 16777217 Y vmnic0 H vSwitch0 10.68 0.02 59.32 0.05 0.00 0.00 16777220 N 0:vswif0 H vSwitch0 10.68 0.02 8.77 0.03 0.00 0.00 50332389 N 1295:gsovmgpprod01 H vSwitch3 3.62 0.01 6.48 0.03 0.00 0.00 33554433 Y vmnic2 H vSwitch2 7.25 0.00 13.73 0.01 0.00 0.00 50331738 N 1414:gsovmdhrdc01 H vSwitch3 2.29 0.00 6.10 0.03 0.00 0.00 33554487 N 1216:gsovmdns01 H vSwitch2 2.48 0.00 8.58 0.00 0.00 0.00 50332382 N 1687:gsovmsoftpdc H vSwitch3 1.53 0.00 2.29 0.00 0.00 0.00 50332455 N 1702:gsovmsmcctst H vSwitch3 2.67 0.00 6.29 0.02 0.00 0.00 50332301 N 1390:gsovmhyp01 H vSwitch3 0.95 0.00 4.20 0.02 0.00 0.00 33554489 N 1290:GSOVM-IK-DMZXP H vSwitch2 0.95 0.00 7.25 0.00 0.00 0.00 33554485 N 1694:gsovmwwwstg H vSwitch2 0.76 0.00 7.06 0.00 0.00 0.00
-----Original Message----- From: clane@co.durham.nc.us [mailto:clane@co.durham.nc.us] Sent: Wednesday, August 27, 2008 3:30 PM To: Page, Jeremy; toasters@mathworks.com Subject: RE: issues with 10G Ethernet.
What sort of ratio are you getting with VM on NFS on those 10gbE cards? (i.e. vm sessions per port and/or esx hosts per port)
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Page, Jeremy Sent: Wednesday, August 27, 2008 9:26 AM To: toasters@mathworks.com Subject: RE: issues with 10G Ethernet.
We are doing the same for our NFS mounted VMs (a couple of hundred) with no real issues. It would be nice of the TOE functionality worked with VIFs though. 7.2.4 here. Are the bugs significant enough to warrant an upgrade if we've been stable for months?
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Glenn Walker Sent: Wednesday, August 27, 2008 8:48 AM To: James Beal; toasters@mathworks.com Subject: RE: issues with 10G Ethernet.
We're using the dual-port 10GbE TOE cards with no issues - that said, TOE is not enabled because we're using VIF (Filer turns TOE off with VIF - not desired by us, but no choice right now as we cannot have SPOF).
Perhaps disabling TOE would get you more reliability?
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of James Beal Sent: Tuesday, August 26, 2008 8:02 AM To: toasters@mathworks.com Subject: issues with 10G Ethernet.
I have a call open with netapp about this however I thought I would ask the list if anyone else has had similar issues.
We have a cluster which is connected to our core via TOE-10G Ethernet (T320E-XFP). The problem we see is that the sometimes the netapp appears stop receiving any packets. I haven't done a packet trace while the problem is in effect as I have been a bit of a Muppet when I have noticed the problem as I had forgotten the rlm user name but I have made notes so next time it happens I will do a trace..
This has happened on both units in our cluster. We have replaced one of the 10gig cards and that has made no difference.
Our networking people show no errors on the line cards. Power cycling the units brings the network card back.
We have turned toe off on the cards and that appears not to make any difference to the reliability of the system.
This has so far occurred 3 times which is really very poor.
sysconfig -v 4 slot 4: Dual TOE-10G Ethernet Controller (T320E-XFP) Device Type: CT-31-1 Version Number: T3-SRAM1.1.0-BR1040-20-C0-FW4.6.0-DR03 Serial Number: PT3807035 e4a MAC Address: 00:07:43:05:13:ac (auto-10g_sr-fd-up) e4b MAC Address: 00:07:43:05:13:ad (auto-10g_sr-fd-cfg_down)
ifconfig e4a e4a: flags=948043<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500 inet 172.17.66.26 netmask 0xffffff00 broadcast 172.17.66.255 partner inet 172.17.66.27 (not in use) ether 00:07:43:05:13:ac (auto-10g_sr-fd-up) flowcontrol full
This message (including any attachments) contains confidential and/or proprietary information intended only for the addressee. Any unauthorized disclosure, copying, distribution or reliance on the contents of this information is strictly prohibited and may constitute a violation of law. If you are not the intended recipient, please notify the sender immediately by responding to this e-mail, and delete the message from your system. If you have any questions about this e-mail please notify the sender immediately.
This message (including any attachments) contains confidential and/or proprietary information intended only for the addressee. Any unauthorized disclosure, copying, distribution or reliance on the contents of this information is strictly prohibited and may constitute a violation of law. If you are not the intended recipient, please notify the sender immediately by responding to this e-mail, and delete the message from your system. If you have any questions about this e-mail please notify the sender immediately.