My boss bought two Cisco 3650 Gig switches of which he will expect me to interconnect our Filer ( FAS 940 ), 3 SunFire 880's, 3 Sun E450's and 6 Gateway 975 Twin Xeon SuSe boxes to. The 880's and the SuSe boxes will all run Oracle over the switch back to separate Volumes on the Filer. The filer's Gig card is a "X1025C-D" Gbit, NIC Fiber card. I've asked that we purchase another Gbit card on the Filer so that I can do a failover VIF. I'm concerned about utilization on the Gbit interface on the Filer.
Anybody have a similar environment and been there already? I'd like to trade notes if so.
Thanks
-Rob
I'm moving to a similar environment here, 3 Oracle Servers and 2 Filers. From what I see, you've got 12 clients hitting 1 Filer over 1 GBit link. I've noticed that Oracle likes to have lots of fast access to it's datafiles. In this case, a full table scan from one of your Oracle servers will probably slow down the rest of the Oracle servers (even if you've got different volumes, you're still traveling over a 1 GBit link.) This means that each client will average 85Mbps. I don't know how much I/O you'll need to support, but I'd recommend that you use a mutli-mode vif, which will load balance the links to the Filer. It sounds like everyone will be on the same network, so you can use the IP based routing, with will tend to balance everyone over different interfaces. It will still give you failover, but it'll obviously be degraded. If you've got the $'s, I'd put in 4 GigE interfaces, each on a different slot (to prevent overloading the PCI slot bandwidth), and set up a multi-vif, which I believe goes up to 4 interfaces. I can't find it in the docs for some reason right now.
For the Suns, I'd recommend using dual GigE cards, and using IP Multi-pathing. It's simply failover, but it's better than nothing. If you've bought it, Sun Trunking is nice also. I've never had cause to set anything up on Linux, although I'd love to hear what others have done.
For the switches. Make sure that you name the ports, and use MRTG or something similar to monitor the switch ports. It can give you valuable insight to which Oracle servers are bandwidth hogs.
If anyone else has anything else to add, particularly on the subjects on switch configuration, I'd love to hear it.
Jason
On Apr 16, 2004, at 3:37 PM, Robert Borowicz wrote:
oss bought two Cisco 3650 Gig switches of which he will expect me to interconnect our Filer ( FAS 940 ), 3 SunFire 880's, 3 Sun E450's and 6 Gateway 975 Twin Xeon SuSe boxes to. The 880's and the SuSe boxes will all run Oracle over the switch back to separate Volumes on the Filer. The filer's Gig card is a "X1025C-D" Gbit, NIC Fiber card. I've asked that we purchase another Gbit card on the Filer so that I can do a failover VIF. I'm concerned about utilization on the Gbit interface on the Filer. Anybody have a similar environment and been there already? I'd like to trade notes if so. Thanks -Rob
here a page of the vif on linux : bonding easy to set and greatly tunable (as always) http://linux-ip.net/html/ether-bonding.html
Jason Frank wrote:
I'm moving to a similar environment here, 3 Oracle Servers and 2 Filers. From what I see, you've got 12 clients hitting 1 Filer over 1 GBit link. I've noticed that Oracle likes to have lots of fast access to it's datafiles. In this case, a full table scan from one of your Oracle servers will probably slow down the rest of the Oracle servers (even if you've got different volumes, you're still traveling over a 1 GBit link.) This means that each client will average 85Mbps. I don't know how much I/O you'll need to support, but I'd recommend that you use a mutli-mode vif, which will load balance the links to the Filer. It sounds like everyone will be on the same network, so you can use the IP based routing, with will tend to balance everyone over different interfaces. It will still give you failover, but it'll obviously be degraded. If you've got the $'s, I'd put in 4 GigE interfaces, each on a different slot (to prevent overloading the PCI slot bandwidth), and set up a multi-vif, which I believe goes up to 4 interfaces. I can't find it in the docs for some reason right now.
For the Suns, I'd recommend using dual GigE cards, and using IP Multi-pathing. It's simply failover, but it's better than nothing. If you've bought it, Sun Trunking is nice also. I've never had cause to set anything up on Linux, although I'd love to hear what others have done.
For the switches. Make sure that you name the ports, and use MRTG or something similar to monitor the switch ports. It can give you valuable insight to which Oracle servers are bandwidth hogs.
If anyone else has anything else to add, particularly on the subjects on switch configuration, I'd love to hear it.
Jason
On Apr 16, 2004, at 3:37 PM, Robert Borowicz wrote:
oss bought two Cisco 3650 Gig switches of which he will expect me to interconnect our Filer ( FAS 940 ), 3 SunFire 880's, 3 Sun E450's and 6 Gateway 975 Twin Xeon SuSe boxes to. The 880's and the SuSe boxes will all run Oracle over the switch back to separate Volumes on the Filer. The filer's Gig card is a "X1025C-D" Gbit, NIC Fiber card. I've asked that we purchase another Gbit card on the Filer so that I can do a failover VIF. I'm concerned about utilization on the Gbit interface on the Filer.
Anybody have a similar environment and been there already? I'd like to trade notes if so.
Thanks
-Rob
-
Jason Frank -
Robert Borowicz -
Stephane Bentebba