RE: NFS Performance

Can't see what kind of NIC you are working with (100Mbps or Gb Ethernet), but maybe interesting suggestion for you: Check client NIC settings to Switch port settings and from switch port to Filer NIC settings. Ie, obviously Auto-negotiation and Flow-control settings are not always as they should be, had this problem with Cisco switch. After changing switch port settings (negotiation fixed instead of auto and Flow control set as Tx=full rx=full) performance increased to acceptable levels. Hope this helps

Ben

-----Oorspronkelijk bericht----- Van: owner-toasters@mathworks.com

[mailto:owner-toasters@mathworks.com]

Namens John Stoffel Verzonden: dinsdag 13 september 2005 18:05 Aan: Ben Rockwood CC: list toasters Onderwerp: Re: NFS Performance

Ben> I've had several filers in production for years but in all that Ben> time never used NFS, just CIFS. I'm now starting to use NFS for Ben> some migrations but I'm really baffled by the poor performance Ben> I'm getting. Part of the problem is that I'm not sure what kind Ben> of performance people usually get.

Ben> When I started pulling data off my 940's I was only getting Ben> around 4-5MB/s. I'm grabbing data by way of GNU Tar on Solaris.

I just did some tests on my F940 here, talking to a Solaris 8 box over Gigabit. I had a directory called 'foo' which is 73M in size, with 2200+ directories and 2300+ files. None of the files was that big I think.

Here's my runs with tar and time:

# time tar cf - foo > /dev/null
0.32u 1.65s 0:03.09 63.7%

Pretty fast, 3seconds implies 25mb/sec throughput. But wait... gnutar can optimize /dev/null writes and not do the work...

# time tar cf - foo > /tmp/foo.tar
0.32u 3.27s 0:12.25 29.3%

Ok, now it took 12secs, which works out to 6+mb/sec. Not great

hmmm...

# time tar cf - foo > /tmp/foo.tar
0.24u 2.42s 0:03.85 69.0%
# time tar cf - foo > /tmp/foo.tar
0.31u 2.11s 0:03.18 76.1%
# time tar cf - foo > /tmp/foo.tar
0.26u 2.09s 0:03.27 71.8%
# time tar cf - foo > /tmp/foo1.tar
0.30u 2.19s 0:03.08 80.8%

Ah, but now look what happens. It's back to 3seconds, or 25mb/sec through put over the wire. It looks like the NetApp cache has kicked in here. Which is good, since it shows that the network isn't the bottleneck here.

So I repeated my tests with another directory, bar, with 500mb total size, 790 directories, 1000 files. So they're obviously bigger files in general.

# time tar cf - bar > /tmp/foo.tar
0.64u 17.61s 0:31.51 57.9%
# time tar cf - bar > /tmp/foo.tar
0.62u 14.24s 0:16.01 92.8%
# time tar cf - bar > /tmp/foo.tar
0.60u 14.80s 0:17.29 89.0%

This time I was seeing around 30mb/sec for the cached versions, and around 15mb/sec for the non-cached initial load. Not too bad.

So for your setup, I suspect that your network is either heavily loaded, or that you have a bad network cable, or that you have a switch in the middle which is swamped or just isn't upto snuff.

I'm also running 7.0.1 on the NetApp. The volume which held the data was mounted via TCP onto an E450 with 4gb of RAM and 4x450mhz CPUs, which are certainly not the fastest.

Ben> The data is web content, so its a mix of images (500K or so), Ben> HTML (1K), and Thumbnails (<5K). No matter what I seem to do I Ben> can't boost performance above that 5MB/s mark. I've tried NFSv2 Ben> and NFSv3, TCP and UDP, 8K and 32K, Gigabit clients and FastEther Ben> clients, Solaris9/UltraSPARC and Solaris10/AMD64, so on and so Ben> forth, with no effect.

This all points me to suspecting that your network hardware or cabling has problems. What does 'netstat -ni' say on both ends? And can you setup a direct connection between the filer and a client to see if that can be made to run better? Get the rest of the network out of the loop.

Good luck, John John Stoffel - Senior Staff Systems Administrator - System LSI

Group

Toshiba America Electronic Components, Inc. - http://www.toshiba.com/taec john.stoffel@taec.toshiba.com - 508-486-1087