Hello,
Thought I'd try out this list in the hope of sharing some info with others as opposed to bugging NetApp support only for my own sake ;) (but I'll CC them at least)
I'm setting up an F540 as a newsspool NFS server. That is, I will store Usenet news articles on it. A handful of machines will mount the filesystem read-only and these machines will accept user newsreader connections. Then I will have one single machine that mount the filesystem read/write - it will be the machine that stores incoming articles and removes old ones from the filer.
Now, for news I've got the advice to use NFS v2 because clients using v3 issue the readdir+ operation unique to v3 and this operation does some extra stuff (as opposed to v2's readdir) that most applications take advantage of but news software (INN nnrpd) has no use of the extra data returned by readdir+ and this makes it a performance loss to use v3 in this particular case.
But... I've read that NFS writes are much faster in NFS v3 than v2. This, of course, leads me to my question (ta-daa) :
Actually, I'll make it two questions. If the anser to the first one is no, the second one is not so interesting anymore ;)
1. Are write operations slower on a NetApp server if you use NFS v2 compared to when you're using NFS v3? (and if so, about how much does it differ?)
2. (if the answer to the above question was 'yes') Could I leave the option "nfs.v3.enable" ON, on the NetApp and mount the filesystem from the reader clients using e.g "mount -o vers=2,proto=udp" (can't remember the exact syntax) while at the same time, from the client that is supposed to be doing a lot of *writing* to the NetApp, mount using version 3 to speed up the writes? This machine won't be running nnrpd so it won't be doing any readdir+ in vain.
Regards,
/Ragnar, Algonet
Now, for news I've got the advice to use NFS v2 because clients using v3 issue the readdir+ operation unique to v3 and this operation does some extra stuff (as opposed to v2's readdir) that most applications take advantage of but news software (INN nnrpd) has no use of the extra data returned by readdir+ and this makes it a performance loss to use v3 in this particular case.
Right. READDIR+ effectively does a stat() on each file that it returns, and that data is not used by the netnews software.
But... I've read that NFS writes are much faster in NFS v3 than v2. This, of course, leads me to my question (ta-daa) :
That's news to me, unless you're referring to the ability ot NFSv3 to use larger block sizes. For netnews, that's not likely to be of much benefit since most articles are very small.
- Are write operations slower on a NetApp server if you use NFS v2 compared to when you're using NFS v3? (and if so, about how much does it differ?)
Except for any block size effects, the performance should be nearly identical.
- (if the answer to the above question was 'yes') Could I leave the option "nfs.v3.enable" ON, on the NetApp and mount the filesystem from the reader clients using e.g "mount -o vers=2,proto=udp" (can't remember the exact syntax) while at the same time, from the client that is supposed to be doing a lot of *writing* to the NetApp, mount using version 3 to speed up the writes? This machine won't be running nnrpd so it won't be doing any readdir+ in vain.
Unfortunately, it's the "feeder" machine (the one which is writing to the filer) that gets hit the hardest by using NFSv3. One operation that is hit particularly hard is renumbering the active file, though there are others that behave similarly.
But... I've read that NFS writes are much faster in NFS v3 than v2. This, of course, leads me to my question (ta-daa) :
That's news to me, unless you're referring to the ability ot NFSv3 to use larger block sizes.
He could also be referring to asynchronous writes. We *currently* don't do asynchronous writes; if a client asks us to do an async write, we do it synchronously - i.e., we log it to NVRAM - and return an indication in the reply that it was done synchronously. (We may do them asynchronously at some point.)
On Thu, 13 Mar 1997, Karl Swartz wrote:
But... I've read that NFS writes are much faster in NFS v3 than v2. This, of course, leads me to my question (ta-daa) :
That's news to me, unless you're referring to the ability ot NFSv3 to use larger block sizes. For netnews, that's not likely to be of much benefit since most articles are very small.
Maybe I should add that I read this in a book about performance tuning for Solaris systems ;) But it did look as if this write performance improvement was connected to the version of the NFS protocol. Quote from 'Configuration and capacity planning for Solaris servers' by Brian Wong :
In response to a variety of technical requests from customers, the NFS protocol was revised in 1993, with the resulting protocol being designated Version 3. The new protocol is similar to its predecessor, but it makes improvements in several key areas. In particular, the new protcol implements the following changes:
- It permits write operations to be performed much more quickly through the use of a two-phase commit protocol, while still maintaining the server's stateless view of the protocol.
[ some more features cut out ]
Naturally, both the client and server systems must implement these features; since many hosts do not, some of the new features are optional and are negotiated between client and server when a file system is mounted by the client"
Is this not applicable to NetApp servers?
Unfortunately, it's the "feeder" machine (the one which is writing to the filer) that gets hit the hardest by using NFSv3. One operation that is hit particularly hard is renumbering the active file, though there are others that behave similarly.
But INND doesn't scan directories very much does it? I can imagine that the renumber operation would suffer but it doesn't do renumber all that frequently. Isn't it nnrpd that accounts for most of the readdir+ performance waste when using NFS v3?
Regards,
/Ragnar, Algonet
In response to a variety of technical requests from customers,
Uh-huh. :-)
the NFS protocol was revised in 1993, with the resulting protocol being designated Version 3. The new protocol is similar to its predecessor, but it makes improvements in several key areas. In particular, the new protcol implements the following changes:
- It permits write operations to be performed much more quickly through the use of a two-phase commit protocol, while still maintaining the server's stateless view of the protocol.
[ some more features cut out ]
Naturally, both the client and server systems must implement these features; since many hosts do not, some of the new features are optional and are negotiated between client and server when a file system is mounted by the client"
Is this not applicable to NetApp servers?
Well...
The async write of V3 had the most impact on non-HW accelerated NFS servers. The NetApp box uses NVRAM to support a logging scheme that make writes on NFS V2 and V3 faster.
Also, the NVRAM accelerates other modifying operations (create and remove, etc.) that are not addressed in V3.
beepy
But... I've read that NFS writes are much faster in NFS v3 than v2. This, of course, leads me to my question (ta-daa) :
That's news to me, unless you're referring to the ability ot NFSv3 to use larger block sizes. For netnews, that's not likely to be of much benefit since most articles are very small.
...
Quote from 'Configuration and capacity planning for Solaris servers' by Brian Wong :
...
In particular, the new protcol implements the following changes:
- It permits write operations to be performed much more quickly through the use of a two-phase commit protocol, while still maintaining the server's stateless view of the protocol.
Asynchronous writes, as Guy guessed. He and Beepy noted that NetApp does not currently implement async writes (we accept them and note in the response that they were done synchronously) and that, with our NVRAM, they wouldn't provide a substantial benefit as they can with general purpose servers having no special hardware support.
Beyond that, "NFS Version 3 Design and Implementation" (from Summer '94 Usenix) notes that async writes "are most effective for large files." Netnews mostly generates lots of little files, so even if async writes had much value on a filer, they wouldn't have much effect for netnews.
Is this not applicable to NetApp servers?
Bottom line: no.
Unfortunately, it's the "feeder" machine (the one which is writing to the filer) that gets hit the hardest by using NFSv3. One operation that is hit particularly hard is renumbering the active file, though there are others that behave similarly.
But INND doesn't scan directories very much does it?
No, it doesn't, but other things that run on the same machine do.
I can imagine that the renumber operation would suffer but it doesn't do renumber all that frequently.
In the case of renumbering the active file, we saw one site drop from 12 hours to a few tens of minutes. Still not too horrible if you only run it once per night, except that it chews up a *lot* of resources on the filer and can have a severe impact on useful work.
Isn't it nnrpd that accounts for most of the readdir+ performance waste when using NFS v3?
*Any* operation than triggers READDIR+ may severely impact performance. It only happens from nnrpd when you change or list a group other than the one you've most recently looked at (or it's been a while). Since that's not something users do that frequently, "most" of the waste depends on how many users you have. Active file renumbering hammers at it continuously until it's done, so unless you have a pretty large number of users, I doubt nnrpd would account for "most" of the hit.
-- Karl
On Fri, 14 Mar 1997, Karl Swartz wrote:
Unfortunately, it's the "feeder" machine (the one which is writing to the filer) that gets hit the hardest by using NFSv3. One operation that is hit particularly hard is renumbering the active file, though there are others that behave similarly.
But INND doesn't scan directories very much does it?
No, it doesn't, but other things that run on the same machine do.
Hmm, but I still haven't a clue about *what* programs generate the readdir+ requests? If not nnrpd and not innd, then what's left? On a newsfeed machine I run only those programs plus innfeed or possibly innxmit and I doubt that those scan directories a lot (or do they?).
depends on how many users you have. Active file renumbering hammers at it continuously until it's done, so unless you have a pretty large number of users, I doubt nnrpd would account for "most" of the hit.
Hmm.... would it be possible to use two mount points for the same filesystem where one is mounted using NFS v2 and one using NFS v3 and then let ones own software choose whether to do an operation using v2 or v3 depending on what path it tries to open?
Maybe this doesn't matter much if there's no performance advantage in general to using v3 instead of v2 of course... Then there's no reason for me to bother about v3 at all as I'm only going to use the NetApp as a newsspool host and nothing else.
Regards,
/Ragnar, Algonet
Unfortunately, it's the "feeder" machine (the one which is writing to the filer) that gets hit the hardest by using NFSv3. One operation that is hit particularly hard is renumbering the active file, though there are others that behave similarly.
But INND doesn't scan directories very much does it?
No, it doesn't, but other things that run on the same machine do.
Hmm, but I still haven't a clue about *what* programs generate the readdir+ requests? If not nnrpd and not innd, then what's left? On a newsfeed machine I run only those programs plus innfeed or possibly innxmit and I doubt that those scan directories a lot (or do they?).
Most people run expire and fastrm on their newsfeed machines. If you choose not to, we'll be happy to sell you more disk! :-) There's also expireover, and whatever other grot gets run from news.daily.
I was mistaken about the active file renumbering, though, which is part of innd and not a separate executable.
depends on how many users you have. Active file renumbering hammers at it continuously until it's done, so unless you have a pretty large number of users, I doubt nnrpd would account for "most" of the hit.
Hmm.... would it be possible to use two mount points for the same filesystem where one is mounted using NFS v2 and one using NFS v3 and then let ones own software choose whether to do an operation using v2 or v3 depending on what path it tries to open?
With active file renumbering being part of innd, no, not without some hacking on the code. Even for the separate executables, keeping the two straight might prove difficult. Why, though? For news, at least, NFS v3 has little, if any, benefit, and can badly hurt.
-- Karl
On Mon, 17 Mar 1997, Karl Swartz wrote:
Most people run expire and fastrm on their newsfeed machines. If you choose not to, we'll be happy to sell you more disk! :-) There's also expireover, and whatever other grot gets run from news.daily.
I bet you would. Alas, we're not Dejanews ;)
Hmm.... would it be possible to use two mount points for the same filesystem where one is mounted using NFS v2 and one using NFS v3 and then let ones own software choose whether to do an operation using v2 or v3 depending on what path it tries to open?
With active file renumbering being part of innd, no, not without some hacking on the code. Even for the separate executables, keeping the two straight might prove difficult. Why, though? For news, at least, NFS v3 has little, if any, benefit, and can badly hurt.
Yeah, the presumption was that writes were quicker in v3 but as they aren't, there is no need to use v3 at all. I'll just give it a miss I think!
Thanks for straightening things out.
Regards,
/Ragnar, Algonet
-
beepy＠netapp.com -
guy＠netapp.com -
kls＠netapp.com -
Ragnar Lonn