-----Original Message----- As far as I'm concerned, this is an urban legend :-) I use UDP v3 on Solaris with NetApp, and I've always had a good performance, no worse than Linux or HPUX. When you say you experienced it yourself, what exactly did you see? Was it reproducible with iozone? What were the numbers? -----Original Message-----
In our environment, YMMV, Solaris (100mb) to NetApp (100mb) works fine over UDP. Solaris (100mb) to NetApp (gigabit) was horribly slow for certain operations. Normal file reads were fine, but a compare (`cmp`) of two files from the same filer were unbelievably slow. A cmp that would take 15 seconds to the 100mb interface of the NetApp would take an hour or more to the gigabit interface. To get Sun to actually admit it was their problem, we did a Solaris (100mb) to Solaris (gigabit) test over UDP, and it took 5minutes or so. Not as bad as to NetApp, but bad enough for them to file a bug, which Sun Engineering ignored since there is a work-around (use TCP or the 100mb interface of the filer). This was with Solaris 8. Solaris 7 did not have this problem in our environment.
Since nearly all our clients are 100mb, and only our QA group was running into this, we set their automount entries to point to the 100mb interface.
John
Not urban legend for us. We've experienced this just last week.
We have a new F810 with a dedicated GigE connection to an E880 (Sol8), intending to use the F810 as our mail spool device.
NFS mounting across UDP we can write to the NetApp at 60MB/s. Going from the NetApp to the SUN it goes at 25-30MB/s once or twice, and then slows right down to 1-2MB/s.
Mounting NFS across TCP, we see 50MB/s to the NetApp, and 25MB/s going back from the NetApp to the SUN.
With our production F740 mounting NFS/UDP across 100mb to our SUNs (Sol 2.6 and 8) we've had no performance problems.
-----Original Message----- As far as I'm concerned, this is an urban legend :-) I use UDP v3 on Solaris with NetApp, and I've always had a good performance, no worse than Linux or HPUX. When you say you experienced it yourself, what exactly did you see? Was it reproducible with iozone? What were the numbers? -----Original Message-----
In our environment, YMMV, Solaris (100mb) to NetApp (100mb) works fine over UDP. Solaris (100mb) to NetApp (gigabit) was horribly slow for certain operations. Normal file reads were fine, but a compare (`cmp`) of two files from the same filer were unbelievably slow. A cmp that would take 15 seconds to the 100mb interface of the NetApp would take an hour or more to the gigabit interface. To get Sun to actually admit it was their problem, we did a Solaris (100mb) to Solaris (gigabit) test over UDP, and it took 5minutes or so. Not as bad as to NetApp, but bad enough for them to file a bug, which Sun Engineering ignored since there is a work-around (use TCP or the 100mb interface of the filer). This was with Solaris 8. Solaris 7 did not have this problem in our environment.
Since nearly all our clients are 100mb, and only our QA group was running into this, we set their automount entries to point to the 100mb interface.
John
On Tue, Aug 20, 2002 at 01:56:08PM -0700, Jeff Bryer wrote:
Not urban legend for us. We've experienced this just last week.
We have a new F810 with a dedicated GigE connection to an E880 (Sol8), intending to use the F810 as our mail spool device.
NFS mounting across UDP we can write to the NetApp at 60MB/s. Going from the NetApp to the SUN it goes at 25-30MB/s once or twice, and then slows right down to 1-2MB/s.
Mounting NFS across TCP, we see 50MB/s to the NetApp, and 25MB/s going back from the NetApp to the SUN.
With our production F740 mounting NFS/UDP across 100mb to our SUNs (Sol 2.6 and 8) we've had no performance problems.
After reading through responses, I get a feel that
1) the alleged Solaris bug only appears with Gb connection involved ( it wasn't clear in the original messages ). I only have 100Mb Suns with 1 exception.
2) the problem only occurs with NFS reads - NFS writes are fine ( only Jeff Bryer said that explicitely )
There're some things I still don't understand. Jeff says that his F810 is connected directly with a Sun - no switch in between I presume. If it's the case, then it pretty much eliminates network switches as a source of problem.
I have a V880 server with a Gb interface, and it talks to a NetApp's Gb interface through Extreme Network's Summit48 switch. Preliminary testing didn't indicate any NFS read problem, and I'm still deciding on the course of further testing.
I also found Sun bug #4434256, which talks about slow reads over UDP.
I'll quote it here:
\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ When Solaris8 read a remote file using NFSv3/UDP, Solaris8 sometimes wait 0.8 sec before sending the READ request to the server.
Following is a network configuration of customer's site.
FirstEther Gigabit E250 ------------- SSR2000 ------------ NetApp (Sol 8)
2332 0.00005 NetApp -> Sol8 UDP IP fragment ID=17850 Offset=31080 MF=1 2333 0.00006 NetApp -> Sol8 UDP IP fragment ID=17850 Offset=32560 MF=0 2334 0.84602 Sol8 -> NetApp NFS C READ3 FH=5AA1 at 7503872 for 32768 ~~~~~~~ 2335 1.73994 Sol8 -> NetApp NFS C READ3 FH=5AA1 at 7503872 for 32768 (retransmit) 2336 0.00051 NetApp -> Sol8 UDP IP fragment ID=18106 Offset=0 MF=1 2337 0.00010 NetApp -> Sol8 UDP IP fragment ID=18106 Offset=1480 MF=1
Since there is a speed difference between client and server, the switch has caused buffer overflow. So It caused retransmission of READ packet from Solaris 8. In order to lessen transmission of read request Packet, the user has set nfs3_nra parameter to 1.
However,It is not necessary to wait to send a FIRST read request packet even in such a environment. \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ Can any confirm this? The workaround of course is to use NFS over TCP.
Igor
On Tue, 20 Aug 2002, Igor Schein wrote:
On Tue, Aug 20, 2002 at 01:56:08PM -0700, Jeff Bryer wrote:
Not urban legend for us. We've experienced this just last week.
/.../
After reading through responses, I get a feel that
- the alleged Solaris bug only appears with Gb connection involved (
it wasn't clear in the original messages ). I only have 100Mb Suns with 1 exception.
/.../
Hmmm. I've had a "unique" situation, then. The symptoms that I first reported (a year ago? longer?) were peculiar only to Oracle's I/O to the filer, _not_ an issue with NFS per se.
I noticed that under extreme stress, UDP performance on Solaris 7 was topping out around 40,000 packets/sec - a limit in the network stack or in the switch, or in the Gigabit driver? We've switched to TCP and seen some improvement, but the underlying cause is still a mystery. Since our performance is acceptable and we don't have the time/equipment to really do a thorough set of tests to get to the bottom of it, we've decided to ignore it. :-)
We're (still) running 5.3.7R3 on an F760 with six FC9's, two volumes (8x 36GB each) for Oracle on two FC-AL loops, Gigabit-II card; Cisco 3508G switch; Sun E4500, Sbus Gigabit 2.0, Solaris 7 HW11/99+latest patches, driver updates, etc. Using "bonnie" or "cpio" or various other tools to test plain old NFS performance, the E4500 can easily saturate the single PCI bus in the F760, getting write speeds of 35-37MB/sec, read speeds sometimes into the 50MB's range. Our 420R's with PCI Gigabit cards can top it out too. We've never had a problem with typical NFS loads.
Oracle, on the other hand, seems to throttle itself - we typically see a single Oracle (8.1.7.3) I/O slave reading or writing 3-5MB/sec (UDP) or now about 7-8MB/sec (TCP). We've concluded that this is NOT a problem with the filer, or the network - with multiple Oracle I/O procs running on multiple CPUs and doing parititioned table scans, for instance, Oracle *can* saturate the filer. But for small non-partitioned apps where you get no parallelism, Oracle - or its interaction with Solaris - introduces a bottleneck that nobody has been able to identify...
Anyway, we'll hopefully be jumping up to Solaris 8 (or 9) and Oracle 9i and ONTAP 6.x, at which point we might be able to take the time to do some tests and see if newer versions of all the various software bits will make things go faster...
As to the original poster's question, though, our experience with Oracle on the filer has been overwhelmingly positive. I have no hesitation recommending it - subject, of course, to the caveat that you should test your particular application to determine if it's a good fit or not.
-- Chris
-- Chris Lamb, Unix Guy MeasureCast, Inc. 503-241-1469 x247 skeezics@measurecast.com
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Igor Schein Sent: Tuesday, August 20, 2002 6:24 PM To: Jeff Bryer Cc: toasters@mathworks.com Subject: Re: Oracle using NetApp
After reading through responses, I get a feel that
1) the alleged Solaris bug only appears with Gb connection involved ( it wasn't clear in the original messages ). I only have 100Mb Suns with 1 exception.
---------------------
Not in our case. We experienced problems with NFS over UDP on Sun Ultra 1, Ultra 2 and Ultra 10 boxes but not on Linux boxes, all of which are connected to an Extreme Black Diamond over 100Mbps links. We had the problem with both a 100Mbps connection to the filer and a 1Gbps connection.
On Tue, Aug 20, 2002 at 09:24:24PM -0400, Igor Schein wrote:
I have a V880 server with a Gb interface, and it talks to a NetApp's Gb interface through Extreme Network's Summit48 switch. Preliminary testing didn't indicate any NFS read problem, and I'm still deciding on the course of further testing.
I ran NFS read tests with the forementioned V880 server as NFS client. There was no significant difference between NFS TCP and NFS UDP. The average read speed was 32MBs, which is weak, but I think this can be configured by tweaking some parameters on both Solaris and NetApp side. Can anyone recommend any specific settings that can improve NFS read?
Thanks
Igor
On Wed, Aug 21, 2002 at 02:10:12PM -0400, Igor Schein wrote:
I ran NFS read tests with the forementioned V880 server as NFS client. There was no significant difference between NFS TCP and NFS UDP. The average read speed was 32MBs, which is weak, but I think this can be configured by tweaking some parameters on both Solaris and NetApp side. Can anyone recommend any specific settings that can improve NFS read?
NetApp has documented the parameters used in one of their benchmarks at http://www.netapp.com/tech_library/3145.html#5.
In particular, we have found that nfs:nfs3_max_threads and nfs:nfs3_nra can make quite a difference.
-- Deron Johnson djohnson@amgen.com
-
Chris Lamb -
Deron Johnson -
Igor Schein -
Jeff Bryer -
John Clear -
Michael Fahy