Why not a box with a special OS (ala WAFL) that runs on what servers recognize as a fibre hub, intercepts the SCSI calls from all attached servers, and does all the nifty stuff needed to make the SAN(?) look like locally attached storage? Advantage? Separate file I/O from TCP session traffic but still allow file sharing in some form or another.
-----Original Message----- From: Dave Hitz [mailto:hitz@netapp.com] Sent: Saturday, March 27, 1999 1:41 PM To: willh@infi.net Cc: toasters@mathworks.com; barsellt@infi.net Subject: Re: NAC SAN
Tom's statement below (to a [NetApp] SAN when we build one) brings up a familiar argument between myself and my boss. My boss contends that NetApp is NOT capable of migrating to a "real SAN."
I've been delinquent in my duty of writing a NAS and SAN paper. I'm about half way done. Whem I'm done, I'll post it here, probably in early form, so that you guys can critique it for me.
Very briefly, I believe that NAS and SAN are different but related technologies, and that the definitions are as follows:
NAS (Network Attached Storage) Storage accessed over TCP/IP, using industry standard file sharing protocols like NFS, HTTP and Windows Networking.
SAN (Storage Area Network) Storage accessed over a Fibre Channel switching fabric, using encapsulated SCSI.
I also think that both NAS and SAN are useful, and that -- in the long run -- both will be critical components in any large data infrastructure. I think there's a strong analogy to the TCP/IP networking world. Routers and switches are very similar in some ways, yet subtly different in others. In the mid-eighties, arguments raged about which was "better". (Back then switches were called bridges.) Today we understand that any large network infrastructure needs to include both.
Some people would quibble with my definitions. Some would argue that if you run NFS over TCP/IP over fibre channel, then that is a SAN. I figure that we didn't invent new names for NFS over 100bT, FDDI, ATM, or Gigabit -- what's so special about fibre channel? Others argue that a SAN is any network dedicated entirely to storage traffic. Does this mean a SAN turns to a NAS if you fire up a telnet on it? A year ago these definitions were less settled, but today I think these are the definitions that are emerging.
With these definitions, the key difference is between file system protocols and raw disk protocols. With NAS, the filesystem runs on the storage system itself. (For NetApp, that's WAFL.) With SAN, the filesystem runs on the hosts attached to the storage system.
People familiar with EMC will recognize this model. With an EMC Symetrix, you get a big box of RAID protected disks, and you can attach multiple hosts to that one box. Thus, I view SAN as "open EMC". On the one hand, SAN could be viewed as a threat to EMC. But on the other hand, EMC has a solid history of products -- software and hardware -- based on this storage model, which means that they ought to have a real head-start on SAN. I think the SAN vendors have a great opportunity if they really can create "open EMC". After all, EMC is doing $4 Billion a year in business with this model!
In any case, I think NAS has some serious advantages over SAN. One big issue with running the file system on the separate hosts is that it makes true data sharing very difficult. The on-disk byte formats are very different between UNIX and NT, and even between different flavors of UNIX. Given the differences between UNIT and NT filesystem semantics, it's no surprise that the on-disk format is different. With NAS, the filesystem is by the data, so it can convert the on-disk format into an industry standard protocol appropriate to the host that wants the data: NFS for UNIX, NT Shares for NT, and HTTP for web browsers.
It is technically possible to build a "global file system" that has a common on-disk format for multiple operating systems, thus allowing NAS-like sharing for SAN. However, I'm skeptical of that model. First of all, there are no such products today. And even when they become available, sharing won't be based on industry standards. A system will only be able to participate in sharing if it runs a special SAN file system from a particular vendor. My believe is that the lesson of 20 years of networking is that we build heterogeneous environments by creating open standards and allowing everyone to support them -- not by saying that anyone who wants to participate in sharing must buy code from one particular company.
Having said all this, though, NetApp does have plans to leverage SAN technology. We have announced an OEM agreement with Brocade. Here are the benefits we expect to achieve:
- Attach to lots of disks.
A fibre channel loop is limited to 127 drives, but with a 16 port fibre channel switch, the limit goes to almost 2000 drives. With additional switches, you can support an arbitrary number of drives.
- Share tape drives between multiple servers.
I believe that backup applications are the killer application for SAN. When people buy expensive tape jukeboxes, they want to be able to attach them to multiple servers. With today's SCSI tapes that's a painful job. But with fibre channel and fibre channel switches, it's easy to connect a single tape to any number of hosts.
- Disaster Tolerant Solutions
Fibre channel switches can do WAN tunneling to provide connections to remote disks without any changes to the server code. This makes it much easier to build remote mirrors for data replication and disaster protection.
In conclusion, I believe that NAS and SAN are both important technologies, and the any large data infrastructure in the future will include both. I believe that NAS focuses on issues associated with users, applications, and data sharing. I believe that SAN focuses on issues associated with disk and tape devices, connection to large numbers of them, and moving data back and forth between them. In the long run, any large environment will have both sets of issues, and will need both types of solutions. In the short run, NAS makes good sense as a starting point, because it is here today and the standards it relies on have been solid for over a decade.
I'll go ahead an post the whole paper when I finish, but I still thought it might be useful to share my thoughts in this brief, half-baked form.
Dave
Why not a box with a special OS (ala WAFL) that runs on what servers recognize as a fibre hub, intercepts the SCSI calls from all attached servers, and does all the nifty stuff needed to make the SAN(?) look like locally attached storage? Advantage? Separate file I/O from TCP session traffic but still allow file sharing in some form or another.
SCSI is a raw disk block protocol. SCSI requests are very simple. They say things like "read block twelve-million and two" from disk, or "write block so-and-so to disk." There is no way in SCSI to express file level ideas like "create a file named foo" or "read the first block in a file name bar".
If you make the data look like locally attached storage, then all the different hosts will access it using their own local filesystems, so the different hosts will each use a different format, and the data on disk cannot be shared.
If you want the data to be shared, then you have to find some industry standard filesystem sharing techniques that everyone has implemented. Today the only such standards are the NAS standards (like NFS, HTTP, Windows Networking).
There are a variety of theoretically possible solutions that we could develop, possibly involving new file sharing protocols that could run over SCSI, or "global file systems" that everyone agrees to run on their system. But there are lots of tricky issues to resolve, and the standards work has barely begun, so even in the best possible case, it'd be many years before we could get a true heterogeneous data sharing solution.
Dave
On Mon, 29 Mar 1999, Dave Hitz wrote:
If you make the data look like locally attached storage, then all the different hosts will access it using their own local filesystems, so the different hosts will each use a different format, and the data on disk cannot be shared.
Yes, but if you made the "disk" REALLY intelligent it would be able to emulate each of the boxes' filesystem. This requires a lot of knowledge by the disk about who accesses it and in which way.
If you want the data to be shared, then you have to find some industry standard filesystem sharing techniques that everyone has implemented.
This is much better, easier, and cheaper than making the disk VERY smart.
Today the only such standards are the NAS standards (like NFS, HTTP, Windows Networking).
Well, there are FTP, AFS, and some others. As you see they really aren't that standard, afterall there are at least three of them.
There are a variety of theoretically possible solutions that we could develop, possibly involving new file sharing protocols that could run over SCSI, or "global file systems" that everyone agrees to run on their system. But there are lots of tricky issues to resolve, and the standards work has barely begun, so even in the best possible case, it'd be many years before we could get a true heterogeneous data sharing solution.
I think it will be years before Microsoft agrees with the others. I think that UN*X people will come to agreement much earlier than that.
Tom
On Mon, 29 Mar 1999, Sam Schorr wrote:
Why not a box with a special OS (ala WAFL) that runs on what servers recognize as a fibre hub, intercepts the SCSI calls from all attached servers, and does all the nifty stuff needed to make the SAN(?) look like locally attached storage? Advantage? Separate file I/O from TCP session traffic but still allow file sharing in some form or another.
I thought about this, but if you really want to take it to the extreme, then the switch has to know exactly how it is used. Imagine any "erver" box that uses mirroring or striping. Unless you know exactly how data is placed you won't be able to recognize how the filesystem looks. Then there is caching. Usually systems don't worry about cache coherency on locally attached storage. The assume (fairly correctly too) that they are the only ones using the storage.
Tom
-
Dave Hitz -
Sam Schorr -
tkaczma＠gryf.net