I don't have a really short list, but I can share a medium sized paper that I wrote back when NFS v3 was first coming out.

Hmm -- I guess the paper did have a short list in it:

o 64-bit file sizes o performance improvements o large block transfers (greater than 8 KB) o safe asynchronous writes o improved attribute returns o READDIRPLUS

There's other stuff too, but this is (apparently) what I found interesting at the time.

Dave

---------------------------------------------------------------------- The Evolution of NFS

by

Dave Hitz hitz@netapp.com

Pre-Introduction ----------------

I originally wrote this in mid-1995, so many of the "future changes" that it predicts, such as wide-spread acceptance of NFS version 3 and NFS-over-TCP have already come to pass. With those minor exceptions, I don't think that time has changed it's accuracy much. -- Dave Hitz, November 1996

Introduction ------------

NFS version 3 is arriving almost exactly ten years after Sun originally introduced NFS. This leaves some people wondering: What took so long? Will it be another 10 years before NFS gets another fresh coat of paint?

In part, these questions reflect a conflation between NFS the protocol and NFS the implementation. While the NFS protocol itself remained unchanged until NFS version 3, NFS implementations have changed substantially in the past 10 years, and they will continue to change in the future even without another protocol revision.

---------------------------------------------------------------------- o NFS changes since 1985 o close-to-open file consistency o automounter o performance improvements o Non-volatile RAM to improve write performance o dynamic retry time adjustment o improved retry cache heuristics o client side disk caching

o NFS changes to come soon (probably in 1995) o NFS version 3 o 64-bit file sizes o performance improvements o large block transfers (greater than 8 KB) o safe asynchronous writes o improved attribute returns o READDIRPLUS

o NFS changes to come later (1996 and beyond) o NFS over TCP. o NFS with Kerberos Authentication o NFS version 4 ---------------------------------------------------------------------- Figure 1: The Evolution of NFS. ----------------------------------------------------------------------

The rest of this paper will describe in detail the changes to NFS that are summarized in figure 1. To many it is a surprise that a protocol can change so much without a protocol revision. The NFS specification defines the exact format of NFS packets transmitted over the network, but it leaves great flexibility in the hardware and software that actually send the packets. In addition, a protocol can have some flexibility designed in from the start. For instance, NFS implementations have traditionally used UDP for remote procedure calls (RPC), but the RPC specification allows either UDP or TCP. Finally, services such as the automounter can be added to improve NFS without any change at all to the protocol or its implementation.

NFS changes since 1985 ----------------------

The changes since 1985, when Sun first released NFS, have been made either at the implementation level, or by adding related features that have improved NFS without changing the protocol itself.

Close-to-open file consistency

In the very early days of NFS, updates made to a file on one NFS client might not show up on another NFS client for many seconds. At first this wasn't a problem because users rarely used files from two NFS clients at once. This inconsistency became unacceptable as window systems became popular, making it easy for a single to use more than one computer at a time. One might edit a source file on one NFS client, for instance, but compile it on another.

Modern NFS implementations make accessing a file from multiple NFS clients safe by supporting "close-to-open" consistency. This means that if you write and then close a file on one client, and then open and read that same file on another client, the data on the second client is guaranteed to be up-to-date.

This is implemented in the NFS client by writing all modified file data to the server in the close(2) system call, and by checking with the NFS server to make sure that any locally cached data is up-to-date in the open(2) system call.

Open-to-close consistency is a perfect example of how an implementation change can dramatically improve a protocol without a formal revision.

Automounter

The automounter was added in order to allow system administrators to create a global network name space for their organization.

The initial scheme of requiring all NFS mountpoints to be added to all clients' /etc/fstab files was cumbersome, but with NIS and the automounter, it is possible to manage a corporate wide name space centrally.

The automounter is an example of how a new feature can improve NFS without requiring any change to the protocol itself, or to its implementation.

Performance Improvements

There have been too many performance enhancements to NFS since it was introduced in 1985 to discuss in detail, but a few are especially interesting:

o NV-RAM to improve write performance o dynamic retry time adjustment. o improved retry cache heuristics o client side disk caching

The biggest performance problem with the early NFS implementations was the requirement that the server write data to disk before responding to client requests. This has been largely solved, at least in high performance servers, with the use of non-volatile RAM. By storing the data from NFS operations in NV-RAM, servers are free to respond to client requests without waiting for disk.

Dynamic retry allows clients to adjust their NFS retry values over time based on the performance they see from the server. If a client notices that a server is slow, it increases the retry timeout value to avoid useless retries. If a client notices that a server is fast, it reduces the retry timeout value, so that retries occur faster when a packet is lost.

Chet Juszczak described a set of improved retry cache heuristics in a USENIX paper a few years ago, and they have now been incorporated into most NFS server implementations. The basic idea is that in some cases the server can tell that a client's retry request is probably bogus, and suppress a retransmission of the reply. This reduces network congestion under heavy load.

The most interesting performance improvement is probably the "CacheFS" feature in SunOS 2.4 which allows NFS clients to cache NFS files on disk, instead of in memory. Many people have the misconception that client caching on disk is an AFS feature that NFS cannot support, but in fact, there is nothing in the protocol itself that indicates where clients can cache data. Whether to store cached data in memory or on disk is an implementation detail that has nothing to do with the format of bits that are sent over the network.

NFS version 3 -------------

Despite all the changes that have occurred without a protocol revision, there are some changes that do require the protocol itself to be modified. The driving force behind NFS version 3 was the desire to handle 64 bit file sizes. This has become important as chips like the MIPS, Alpha, and SPARC have started to support 64 bit integers.

Since the initial protocol specification defined file sizes as being 32 bits long, supporting 64 bit files required the NFS protocol revision to be updated.

Protocol revisions are rare, so it isn't sensible to make just one change. As a result, NFS version 3 includes several other changes along with the large file size support. The most interesting are a collection of performance improvements described below.

Large Block Transfers (greater than 8 KB)

NFS version 2 restricts read and write operations to 8 KB. In NFS version 3, the client and server can negotiate any size they like for reads and writes. Typical sizes will probably be on the order of 32 or 64 KB.

Allow the client and server to negotiate the optimal transfer size provides flexibility that will allow NFS version 3 implementations to evolve in the future, if necessary, in case new networking technology makes even larger block sizes desirable.

Safe Asynchronous Writes

This feature allows the server to reply to writes immediately, instead of waiting for the data to be put safely on disk or in NV-RAM. A new operation, called COMMIT, lets clients check with the server at some point after the WRITE operation, to verify that the server actually has written the data. The client is required to keep its own copy of the written data until the COMMIT succeeds, and if the COMMIT fails, the client is required to resend its copy of the written data.

For systems without NV-RAM, this feature improves write performance for large files. On servers that do use NV-RAM, it can reduces the CPU used to copy data into NV-RAM, thus increasing the total capacity of the server.

NV-RAM will continue to be critical to fast NFS service, because the safe asynchronous write feature doesn't help small file writes very much, and it doesn't help operations such as CREATE, REMOVE, and RENAME at all.

Improved Attribute Returns

In NFS version 2, some operations return less information than they should. For instance, the SYMLINK operation creates a new link, but it does not return the file handle or attributes of the link. As a result, an NFS client must send a LOOKUP operation immediately after the SYMLINK.

In NFS version 3, operations return additional information as appropriate, thus reducing the total number of operations that need to be sent.

The READDIRPLUS Operation

In NFS version 2, the READDIR operations returns the names of the files in a directory, but not the attributes. So to handle a command like "ls -l", the READDIR must be followed by a LOOKUP operation for each file in the directory. An "ls -l" on a directory with 100 entries would require 101 NFS operations.

NFS version 3 supports a READDIRPLUS operation that returns both directory names and file attributes. As a result, "ls -l" could be handled with just one READDIRPLUS operation. This is especially useful in speeding up recursive tree walking commands like "find" and "ls -R".

NFS Changes to Come Later (1996 and Beyond) -------------------------------------------

Even after NFS version 3 becomes available, there will be additional changes that do not require protocol changes.

NFS over TCP.

For local area networks, UDP may be more efficient than TCP, because TCP tends to batch small requests rather than sending them out immediately.

On the other hand, TCP can improve performance for wide area networks where lost packets are more likely and the simple retry timeout mechanism of RPC over UDP tends not to work well.

Running NFS over TCP instead of UDP does not require a protocol revision because the remote procedure call (RPC) layer, which NFS uses, is already defined to work on either UDP or TCP. Most current implementations of NFS use UDP, but supporting TCP as well is simply an implementation issue. Some NFS vendors already support RPC over TCP, and many others will add TCP support in up-coming releases.

NFS with Kerberos Authentication

As more people start to use NFS over wide area networks, and especially as TCP improves the performance of NFS over WANs, security becomes more of an issue.

Although some people have experimented with Kerberos in NFS, it is not yet widely available. It seems likely that Sun will move to productize this more formally, and that other vendors will follow suit.

Again, this does not require an NFS protocol modification because the RPC protocol which NFS uses is flexible to support multiple authentication mechanisms. A few vendors are starting to experiment with Kerberos authentication, and it should become more common as NFS over TCP allows NFS to be more widely used over insecure wide-area networks such as the internet.

Conclusion ----------

I believe that most long-lived, wide-spread standards survived largely because they could adapt to changing requirements over time. FORTRAN, for instance, has remained an important programming language for decades because it has repeatedly incorporated the most successful features of newer languages. The C language has done likewise in its evolution to C++.

Similarly, the research community provides a continuing source of ideas for the evolution of NFS with projects like the Andrew File System (AFS), Sprite, Spritely NFS, NQ-NFS, and others. AFS is a good example, because many features that appeared first in AFS have now migrated to NFS, including close-to-open file consistency, network wide name spaces, large block transfers, and client side disk caching.

As a result of its history of adopting new ideas, I expect NFS to remain strong for many years to come. A scientific programmer once said: "I don't know what programming language I'll be using in the year 2001, but I'm sure that its name will be FORTRAN." A similar claim might be made of NFS.