I have done Snapshots with EVMS (Enterprise Volume Management Software) under LINUX. It does basic snapshots and only has a command line interface (you do your own crontab). It is slower then NetApp snapshots (even more so the more snapshots you get), but it does work. You still have to figure out how to quiesce your filesystem during the snap (umount/mount works, but can be a challenge).
The only thing close to SnapMirror I have found is to do RAID-1 via iSCSI or FCIP. Big investment in switches, though. Just have one local disk and more remote disk striped together under the "md" driver.
The RAID striping isn't done in the shelf, the enclosure has a SES/CEMI card that gives the Filer information on temperatures, voltages, what power supplies are working, what the health of the local drive bus is, etc. This is either a SCSI target or a Fibre Channel port/WWN that the Filer talks to. The shelves are just JBODs and you can hook them up to any host you want and use them. It's just that when you hook them into the filer, the filer expects certain behaviors.
You are on the right track on the IDE (I prefer the term ATA) drives, but the following issues present themselves:
1. LINUX has a 2TB file system limit. Solaris has a 1TB limit. NetApp has something like a 6TB limit. 96Tbytes is the most number of disk shelves (24) that can be hooked into that filer. I would much rather have 16 file systems with NetApp than 48 file systems with LINUX.
2. Getting LOTS of drives working all at once is a real challenge and starts to cost $$$ for more hardware (enclosures, SCSI/Fibre Channel interfaces, etc). The 7TB R200 is high on a $/MB basis ($.01 per MB), but a fully loaded R200 LISTS for half that per MB ($.006 per MB) and any reseller will give you a discount (maybe 20%) off that. The old R100/R150 have been discontinued (real pain to get those SCSI drive shelfs installed in the field). The R200 is easier to install/manage than the old R1x0. Plus MAXTOR doesn't make the old drive shelves anymore.
I'm not saying that NetApp is always better, but they have their place in the overall scheme. Small NAS boxes are getting really good and NetApp is responding to the market that they choose to, which I would characterize as the "more fully featured solution" that has a higher return to NetApp.
You might also want to look at http://www.openfiler.org. I have tried it and it has promise, but it is still a challenge. I still stick with RedHat 8 (yes, 8, not 9, cause 8 doesn't ever crash/hang) and just add in SAMBA 3.x for a really cheap NAS box.
-----Original Message----- From: owner-toasters@mathworks.com on behalf of Joe Schmoe Sent: Mon 6/14/2004 12:26 PM To: toasters@mathworks.com Subject: Piecing together a netapp on the cheap - PART II
Thanks for your help so far. Let me give some more details and answer some of the points you all have raised.
First, I am looking for cheap and large - and am willing to sacrifice speed to get it.
Second, I am very intrigued by SnapMirror (replication over WAN) and SnapShot copies (point in time backups). The reason I don't just build my own little unix fileservers with linux or FreeBSD is that I don't see how GNU tools and scripting can replicate the functionality of snapmirror and snapshot copies. Maybe I'm wrong ?
So, I see that my initial plan is ill advised - you have pointed out that the disk shelves that NetApp ships are not just JABODs, but have intelligence in them - I did not know that the RAID striping was done in the shelf - I thought all RAID and logic was housed in the head unit and it just accessed big stacks of disks.
So at this point, my questions are as follows:
1. Is there any reasonable implementation of snapmirror / snapshot copies in free UNIX OSs ? That is, can I get netapp like features out of a normal NFS fileserver ?
2. When I see netapp quote a certain capacity for its filers, like 96TB for the NearStore R200, does that simply mean that they can get to 96TB using the largest currently available IDE drives, and that once larger drives are available that number will naturally go up ? OR does it mean there is some logical limitation in place on the unit that will keep it from going above 96TB ?
3. What is the absolute cheapest IDE based NAS that netapp has put out ? Is it the NearStore R100 ? What if I bought the head unit and shelves from netapp (or netapp branded from ebay), but used my own 400gig hitachi IDE drives ? - I see a 7TB R100 on ebay right now for $78,500 - which is completely absurd. My 4TB freeBSD fileserver at home cost about $5k ...
Again, many thanks.
--------------------------------- Do you Yahoo!? Friends. Fun. Try the all-new Yahoo! Messenger
Timothy Sesow tsesow@nsllc.com wrote: 1. LINUX has a 2TB file system limit. Solaris has a 1TB limit. NetApp has something like a 6TB limit. 96Tbytes is the most number of disk shelves (24) that can be hooked into that filer. I would much rather have 16 file systems with NetApp than 48 file systems with LINUX.
Ok, but I still don't understand - the 96TB @ 24 shelves is based on a certain size disk drive - if disk drives double in size, does that mean you can do 192TB @ 24 shelves, or is there some restriction in the filer code that does not let you exceed the published maximum ?
BTW ... FreeBSD has an 8TB limit with a 2k blocksize, and 16TB limit with 8K blocks. This information is as of 3.0-CURRENT (very very old), so I assume it has improved/stabilized since then.
I'm not saying that NetApp is always better, but they have their place in the overall scheme. Small NAS boxes are getting really good and NetApp is responding to the market that they choose to, which I would characterize as the "more fully featured solution" that has a higher return to NetApp.
Well, I am all for making my own FreeBSD based NAS system, but again, I really want (need ?) the nice features like snapmirror and snapshot copies, and unmounting the volume every time I want to take a snapshot seems unreasonable. On the other hand, I do not have enough money to buy a fully loaded R200, even if it does bring the cost per megabyte down to .006.
I think I am going to need to look at competing, smaller hungrier companies that make IDE NAS devices that can do things like snapmirror and snapshot copies. I guess NetApp is just not for me (or my budget).
--------------------------------- Do you Yahoo!? Friends. Fun. Try the all-new Yahoo! Messenger
1. LINUX has a 2TB file system limit. Solaris has a 1TB limit. NetApp has something like a 6TB limit. 96Tbytes is the most number of disk shelves (24) that can be hooked into that filer. I would much rather have 16 file systems with NetApp than 48 file systems with LINUX. ReiserFS has a 16TB size limit on the filesystem itself and some very nice features. See the FAQ at http://www.namesys.com/ http://www.namesys.com/ Ok, but I still don't understand - the 96TB @ 24 shelves is based on a certain size disk drive - if disk drives double in size, does that mean you can do 192TB @ 24 shelves, or is there some restriction in the filer code that does not let you exceed the published maximum ? Joe: I can't say for sure (probably have to be a NetApp employee to do that) but there are benchmarks on this list with the 144GB drives. My R200 has the 300GB Maxtor drives whittled down to about 250GB each. The R200 requires 6.5.x so you might be onto something with the filer code, but I'm sure there's also a hardware dependency on that too. Well, I am all for making my own FreeBSD based NAS system, but again, I really want (need ?) the nice features like snapmirror and snapshot copies, and unmounting the volume every time I want to take a snapshot seems unreasonable. On the other hand, I do not have enough money to buy a fully loaded R200, even if it does bring the cost per megabyte down to .006. I think I am going to need to look at competing, smaller hungrier companies that make IDE NAS devices that can do things like snapmirror and snapshot copies. I guess NetApp is just not for me (or my budget). FreeBSD reports 64T-1 on their FAQ. Here's a summary of the links mentioned recently on the list:
http://www.snapappliance.com/ http://www.rsnapshot.org/ http://www.openfiler.org/ http://www.namesys.com http://www.namesys.com/ (reiserFS) http://www.freebsd.org/ http://lists.freebsd.org/pipermail/freebsd-current/2003-April/001444.htm l
It looks like the open source community has an appetite for producing something emulating a subset of filer features for cheap or free....looks like Joe Schmoe might be able to do some nice spiffy stuff at home like disk-to-disk backup and the like...
JKB
by the way,
snapmirror can be achieve with rsync or unison also w/ ndmpcopy (ndmp server can be setted up w/ linux) a raid 1 over network (md raid + NBD = DRBD) realise the syncmirror feature of a metrocluster (writes are sync) for the metrocluster cluster feature, you can try heartbeat
you can do almost all what the netapp box do but you can't open a case when something goes wrong, or really wrong and keep in mind that setting up a heartbeat cluster w/ metrocluster feature is a real challenge
ref : heartbeat http://www.linux-ha.org drbd http://www.drbd.org nbd http://nbd.sourceforge.net enbd http://www.it.uc3m.es/~ptb/nbd ultramonkey http://www.ultramonkey.org and more... only check this for fun, if you buy netapp, you don't look after need-60-days-to-be-set-up solutions
James Brigman wrote:
- LINUX has a 2TB file system limit. Solaris has a 1TB limit. NetApp
has something like a 6TB limit. 96Tbytes is the most number of disk shelves (24) that can be hooked into that filer. I would much rather have 16 file systems with NetApp than 48 file systems with LINUX.
ReiserFS has a 16TB size limit on the filesystem itself and some very nice features. See the FAQ at http://www.namesys.com/
Ok, but I still don't understand - the 96TB @ 24 shelves is based on a certain size disk drive - if disk drives double in size, does that mean you can do 192TB @ 24 shelves, or is there some restriction in the filer code that does not let you exceed the published maximum ?
Joe: I can't say for sure (probably have to be a NetApp employee to do that) but there are benchmarks on this list with the 144GB drives. My R200 has the 300GB Maxtor drives whittled down to about 250GB each. The R200 requires 6.5.x so you might be onto something with the filer code, but I'm sure there's also a hardware dependency on that too.
Well, I am all for making my own FreeBSD based NAS system, but again, I really want (need ?) the nice features like snapmirror and snapshot copies, and unmounting the volume every time I want to take a snapshot seems unreasonable. On the other hand, I do not have enough money to buy a fully loaded R200, even if it does bring the cost per megabyte down to .006.
I think I am going to need to look at competing, smaller hungrier companies that make IDE NAS devices that can do things like snapmirror and snapshot copies. I guess NetApp is just not for me (or my budget).
FreeBSD reports 64T-1 on their FAQ. Here's a summary of the links mentioned recently on the list:
http://www.namesys.com http://www.namesys.com/ (reiserFS)
http://lists.freebsd.org/pipermail/freebsd-current/2003-April/001444.html
It looks like the open source community has an appetite for producing something emulating a subset of filer features for cheap or free....looks like Joe Schmoe might be able to do some nice spiffy stuff at home like disk-to-disk backup and the like...
JKB
Joe Schmoe non_secure@yahoo.com writes:
Well, I am all for making my own FreeBSD based NAS system, but again, I really want (need ?) the nice features like snapmirror and snapshot copies, and unmounting the volume every time I want to take a snapshot seems unreasonable.
FreeBSD-5.x supports filesystem snapshots. I haven't played with them too much but they seem very quick to create, about a second or two, and I didn't have to take the filesystem offline or anything. See mksnap_ffs(8) and
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/snapshots.html
--On Friday, June 25, 2004 11:01 AM -0400 Chris Shenton Chris.Shenton@hq.nasa.gov wrote:
FreeBSD-5.x supports filesystem snapshots. I haven't played with them too much but they seem very quick to create, about a second or two, and I didn't have to take the filesystem offline or anything. See mksnap_ffs(8) and
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/snapshots.html
According to the README (http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/ufs/ffs/README.snapshot?rev=1.4) snapshots suspend filesystem activity for 5 seconds on an 8Gb filesystem, and may affect performance if you're removing many small files that belong to a snapshot. These might be concerns depending how you're using snapshots and what the system is doing.
YMMV, do your own testing, I've only used snapshots when dump(8)ing live filesystems, which creates a snapshot, dumps it, then removes the snapshot (same as on many large NAS systems I've used). Seems to work nicely for that purpose.
Bryan
tsesow@nsllc.com (Timothy Sesow) wrote:
- LINUX has a 2TB file system limit. Solaris has a 1TB limit. NetApp
has something like a 6TB limit. 96Tbytes is the most number of disk shelves (24) that can be hooked into that filer. I would much rather have 16 file systems with NetApp than 48 file systems with LINUX.
One shouldn't read too much into these various limits, as everyone is scrambling to increase them. For Solaris, in particular, ufs filing systems can now be up to 8TB if created with the mtb=y option (giving up fragments), and then there's zfs ... dfs ... well, whatever Sun marketing are calling it this week ... on the horizon.
Chris Thompson Email: cet1@cam.ac.uk
-
Bryan Fullerton -
Chris Shenton -
Chris Thompson -
James Brigman -
Joe Schmoe -
Stephane Bentebba -
Timothy Sesow