toasters January 2000

toasters@lists.teaparty.net

120 participants
129 discussions

Re: Sun Client Maximum Throughputs (again)
by mds＠gbnet.net 06 Jan '00

06 Jan '00

On Wed 7 Jan, 1930, Darrell Fuhriman <darrell(a)grumblesmurf.net> wrote: > Brian Tao <taob(a)risc.org> writes: > > > Do you find that with 2 x 400-MHz CPU's that you have enough > > cycles to stream 600 Mbps off the gigabit ethernet and sustain high > > throughput to your tape drives? > > Yep, plenty. :) Of course, in our testing we've found that Sun > does much better at this than other platforms, say DEC, which > just crapped itself entirely. :) <nosey> Heh. Have you been extra careful balancing your sbus/pci cards? What tape units do you use? What host adaptors too? Did you try IBM's, SGI's or HP's, out of sheer curiosity..? </nosey> > Darrell >-- End of excerpt from Darrell Fuhriman -- -Mark ... an Englishman in London ...

1 0

Re: Sun Client Maximum Throughputs (again)
by Darrell Fuhriman 06 Jan '00

06 Jan '00

mds(a)gbnet.net (mark) writes: > These two do what? > > ndd -set /dev/ge adv_pauseTX 0 > > ndd -set /dev/ge adv_pauseRX 0 Disable Link Level Flow Control. According to the docs, you shouldn't need to muck with them (TX defaults to 0, RX to 1) unless you're silly and aren't using your Gb card in the 66Mhz slot. I think the concept is of LLFC is silly when you're using only TCP like we are, hence the forcibly disabling. > Much bigger buffers than I got for 100Mb/s networks, makes sense. Have you > found these value by experiment or otherwise? > > ndd -set /dev/tcp tcp_recv_hiwat 262144 > > ndd -set /dev/tcp tcp_xmit_hiwat 262144 Honest truth is, I can't remember. I think it was a combination of chats with sun, and our own experimenting. > How come this is set so low? > > ndd -set /dev/tcp tcp_xmit_lowat 2048 That's actually the default on 2.6, maybe it's a hold over from earlier OS revs? > How do each of these three things affect things? > > ndd -set /dev/tcp tcp_wscale_always 1 This enables the wscale option. ;) Essentially it allows the window size to exceed the original TCP default of 64k. I suspect it also puts a minimum size of 64k on the window size as a side effect. > > ndd -set /dev/tcp tcp_tstamp_always 1 This places a (sender determined) timestamp in each packet, which is simply returned to it by the receiver in the ACK. The result of this is that you get more accurate RTT calculations. This means fewer unnecessary retransmits, and better congestion control in fat pipes. > > ndd -set /dev/tcp tcp_rexmit_interval_max 60000 I'm not sure why this is set to what it is (it's 1/4 of the default), we put it in at Sun's recommendation. Docs say: If you are running your Solaris 2.6 system with a large TCP window and you experience slowdowns during high network traffic, you should decrease the TCP default maximum retransmission timeout interval variable Which doesn't help much. :) Darrell

1 0

Re: Sun Client Maximum Throughputs (again)
by Darrell Fuhriman 06 Jan '00

06 Jan '00

Brian Tao <taob(a)risc.org> writes: > Do you find that with 2 x 400-MHz CPU's that you have enough > cycles to stream 600 Mbps off the gigabit ethernet and sustain high > throughput to your tape drives? Yep, plenty. :) Of course, in our testing we've found that Sun does much better at this than other platforms, say DEC, which just crapped itself entirely. :) Darrell

1 0

Can a C720 take a 4GB controller and shelf?
by Jaye Mathisen 06 Jan '00

06 Jan '00

I'm eyeballing swapping the FCAL drives on my new 720 with the 4GB shelves on my 630's and 540's, to get the higher cap drives on the 630. I don't think I can put FCAl on the 540, so it's a moot point. Just thinking about it. Have a spare 720, so I can always try it I guess.

1 0

Re: Sun Client Maximum Throughputs (again)
by Darrell Fuhriman 06 Jan '00

06 Jan '00

mds(a)gbnet.net (mark) writes: > What I'm thinking of is that AFAICR tcp_xmit_hiwat and tcp_recv_hiwat > are set to 8192 (bytes) and the equivalent *lowat's to 2048. This is We're using this on our 2x400 MHz E450's and we see ~60MB/s using switched gigabit. ndd -set /dev/ge adv_1000fdx_cap 1 ndd -set /dev/ge adv_1000hdx_cap 0 ndd -set /dev/ge adv_1000autoneg_cap 0 ndd -set /dev/ge adv_pauseTX 0 ndd -set /dev/ge adv_pauseRX 0 ndd -set /dev/tcp tcp_recv_hiwat 262144 ndd -set /dev/tcp tcp_xmit_hiwat 262144 ndd -set /dev/tcp tcp_xmit_lowat 2048 ndd -set /dev/tcp tcp_wscale_always 1 ndd -set /dev/tcp tcp_tstamp_always 1 ndd -set /dev/tcp tcp_rexmit_interval_max 60000 Note these are dedicated backup servers, so the very high tcp_X_hiwait may not be appropriate if you're not doing lots of bulk data transfer. The tcp_tstamp_always is a big win. Darrell

3 2

Re: Sun Client Maximum Throughputs (again)
by mds＠gbnet.net 06 Jan '00

06 Jan '00

On Wed 7 Jan, 1930, Darrell Fuhriman <darrell(a)grumblesmurf.net> wrote: > mds(a)gbnet.net (mark) writes: > > > What I'm thinking of is that AFAICR tcp_xmit_hiwat and tcp_recv_hiwat > > are set to 8192 (bytes) and the equivalent *lowat's to 2048. This is > > We're using this on our 2x400 MHz E450's and we see ~60MB/s using > switched gigabit. That's pretty respectable! Could you go through how some of these settings help - I'm not 100% clear on some of these meself: These three pin the duplicity of the line, fair enough. > ndd -set /dev/ge adv_1000fdx_cap 1 > ndd -set /dev/ge adv_1000hdx_cap 0 > ndd -set /dev/ge adv_1000autoneg_cap 0 These two do what? > ndd -set /dev/ge adv_pauseTX 0 > ndd -set /dev/ge adv_pauseRX 0 Much bigger buffers than I got for 100Mb/s networks, makes sense. Have you found these value by experiment or otherwise? > ndd -set /dev/tcp tcp_recv_hiwat 262144 > ndd -set /dev/tcp tcp_xmit_hiwat 262144 How come this is set so low? > ndd -set /dev/tcp tcp_xmit_lowat 2048 How do each of these three things affect things? > ndd -set /dev/tcp tcp_wscale_always 1 > ndd -set /dev/tcp tcp_tstamp_always 1 > ndd -set /dev/tcp tcp_rexmit_interval_max 60000 > > Note these are dedicated backup servers, so the very high > tcp_X_hiwait may not be appropriate if you're not doing lots of > bulk data transfer. > > The tcp_tstamp_always is a big win. Tantalising - I'd like to know more. > Darrell > >-- End of excerpt from Darrell Fuhriman -- -Mark ... an Englishman in London ...

1 0

Netapp w. multiple NIC in WINS ?
by Johan Gillquist 05 Jan '00

05 Jan '00

Hi, I have installed a quad ethernet NIC i my 720 and I wonder how it will present it self in the NT Domain. Does any one know how to be sure trafic goes to the correct interface ? How does WINS react on a NT/Netapp host with multiple NIC's ? My NT servers are on the same subnet as the "e1a" interface and NFS clients on the "e0". Regards /Johan

2 1

RE: Sun Client Maximum Throughputs (again)
by Fox, Adam 05 Jan '00

05 Jan '00

You've probably already checked this, but back from my Tech Support days, we always told folks to manually set speed and duplex on both sides. Speed is less of an issue because if the speed isn't matched, you'd be passing far less good data than what you are seeing, but duplex mis-matches can certainly cause very bad performance over 100Mb/s interfaces. The other big thing is retransmissions of 32K UDP requests. However, since these are cross-over cables I hope this isn't the issue. I have seen this on very busy switched networks that 32K UDP can really get slow due to nasty retransmission times. Hope this helps. -- Adam Fox > I didn't see any satisfactory answers to this the last time > around, but I'm doing a bit of benchmarking for a killer tape backup > server (streaming ~60MB/sec to tape). I have an older Sun E450 with 2 > x 250-MHz CPU's and a Quad Fast Ethernet NIC in one of the PCI slots. > > qfe0 is directly attached to an idle F740, and likewise for qfe1 > to another idle F740. Please tell me I should be able to see more > than an aggregate 10 MB/s doing dumps over rsh with that setup? I'm > shuffling the dump streams off to /dev/null so local disk speed is not > an issue. On another system, a 2 x 300-MHz E450 with two single-port > NIC's, I'm able to pull about 15 MB/s total. Does the Sun QFE just > suck, and I should stick to individual single-port NIC's? > > I want to lead up to a 4 x 450-MHz E420R with Gigabit Ethernet, > and hope I can achieve 50 MB/s or more from eight F740's. Has anyone > tried this configuration? > -- > Brian Tao (BT300, taob(a)risc.org) > "Though this be madness, yet there is method in't" >

1 0

Re: Sun Client Maximum Throughputs (again)
by mds＠gbnet.net 05 Jan '00

05 Jan '00

On Wed 7 Jan, 1930, Brian Tao <taob(a)risc.org> wrote: > On Tue, 29 Jun 1999, Brian Pawlowski wrote: > > > > I thought I could use a 2 x 400MHz Ultra Enterprise E250 > > with a new Sun GbE card in it as a killer client, and > > I find that with V3 32KB UDP packets, the sucker > > rolls over at > > > > ~30 MB/s > > > > with 2 CPUs pinned at 99% in system time. > > > > This *sucks*. > > I didn't see any satisfactory answers to this the last time Ditto. <snip> > NIC's, I'm able to pull about 15 MB/s total. Does the Sun QFE just > suck, and I should stick to individual single-port NIC's? Have you tweaked up the high water and low water marks for transmission and reception, out of interest? What I'm thinking of is that AFAICR tcp_xmit_hiwat and tcp_recv_hiwat are set to 8192 (bytes) and the equivalent *lowat's to 2048. This is predicated on 10Mb/s ethernet networks. Not 100Mb/s contentionless networks. This is true in Sol2.5.1, but I've not checked in other Solarises.. They can be checked with: ndd /dev/tcp tcp_xmit_hiwat ndd /dev/tcp tcp_recv_hiwat ndd /dev/tcp tcp_xmit_lowat They can be set higher with the machine up using: ndd -set /dev/tcp tcp_xmit_hiwat 32768 ndd -set /dev/tcp tcp_recv_hiwat 32768 ndd -set /dev/tcp tcp_xmit_lowat 24576 or with appropriate lines in /etc/system. UDP can also be checked and set the same way: ndd /dev/udp udp_xmit_hiwat ndd /dev/udp udp_recv_hiwat ndd /dev/udp udp_xmit_lowat ndd -set /dev/udp udp_xmit_hiwat 32768 ndd -set /dev/udp udp_recv_hiwat 32768 ndd -set /dev/udp udp_xmit_lowat 24576 I'm wondering if people seeking higher NFS performance might want to investigate these settings? (I'd be interested in seeing what people have their nfs:nfsv3_nra set to too 8) All this is straight from the Sun FDDI 5.0 manual, but we're talking 100Mb/s either way, and I've used these settings before. I'm not in a position to test whether they'll have any benefit - or whether Solarises >2.5.1 choose better defaults to start with. Perhaps you can check and set me straight either way? I'm guessing but I think there might be a little mileage in setting them higher still for Gb/s networks. But the faster CPU's and better NIC smarts will offset some buffering requirements in your proposed system. At least they *should*. 8) The FDDI manual goes on to talk about setting the socket options..but that's a whole other kettle of fish. Using netstat -k <interface> might yield interesting numbers, in particular norcvbuf and noxmtbuf which are the buffer allocation failure counts. In Sol2.6 and on, according to Cockfroft at least. If the above defaults are in place make a note of the interrupts and any info from vmstat 5, mpstat 5 and netstat -k that looks interesting as the settings stand doing a dump, *then* change them and rerun the test. I've got a bit of an axe to grind about the hme and qfe interfaces on Sun's.. and this would be v. interesting to me. > -- > Brian Tao (BT300, taob(a)risc.org) > "Though this be madness, yet there is method in't" > >-- End of excerpt from Brian Tao -- -Mark ... an Englishman in London ...

1 0

Sun Client Maximum Throughputs
by beepy＠netapp.com 05 Jan '00

05 Jan '00

Do any of our customers out there have some measurements of Sun client throughput? I thought I could use a 2 x 400MHz Ultra Enterprise E250 with a new Sun GbE card in it as a killer client, and I find that with V3 32KB UDP packets, the sucker rolls over at ~30 MB/s with 2 CPUs pinned at 99% in system time. This *sucks*. In an experimental filer setup reading from memory (all cached data) the F760 CPU is at 20%. I am totally client bound. Has anyone sized Sun clients out there? Are the PCI clients dogs? Is the new GbE card from Sun a loser? Should I go grab a new Alteon card for the Sun? What is your killer high performance client? It seems a function of the Sun E250 - 100BaseT seems to scale only to 30 MB/s before running out of CPU... Broken-hearted in Amsterdam, beepy

9 13

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters January 2000