We have a similar situation on our F740. We want to keep the volume size small because: a) Restores can take forever as the data size gets bigger. b) 10-14 disks is a good compromise on reliability. Having 2 out of 14 disks go bad at one time is much rarer than 2 out of 51 disks. c) If you upgrade disks/shelves in the future, you will likely do it a volume at a time. We did this with a volume with 180GB and the volcopy took 6-7 hours to complete. It is NOT very fast. With 400GB+ in a volume, that should be 2-2.5x longer.
John
-----Original Message----- From: Moshe Linzer [mailto:Moshel.Linzer@nsc.com] Sent: Monday, September 18, 2000 4:31 PM To: toasters@mathworks.com Subject: Optimal volume size
We are moving from a 630 with a single 350GB volume (51x9GB SCSI) to a new 760 with 14 36GB disks. Our Netapp support recommends creating a 14-disk raid group, with a single raid group per volume (giving us a maximum volume size of around 400GB). I understand why 14 disks is a good number for a raid group (balance chances of disk failure vs efficient use of disks), but what is the reasoning behind keeping volume size down to around 400GB? Is it just dependant on restore time from tape? In which case, as faster tapes become available (LTO?), this number should increase. Does it also lessen chances of file system corruption on the volume, or is it just a warm fuzzy feeling that all your eggs (so to speak) are not sitting in one 1.2 terabyte volume?
If my max volume size will remain the single raid group, I would like to divide my data into 2 smaller volumes from the start, so I don't start off with a maxed-out volume. This means using a time-consuming ndmpcopy or rsync process to move the data, instead of the faster volcopy or snapmirror.
Thanks for your insights.
Moshe
Fortunately for me, someone else here takes care of backups, but...
We have a similar situation on our F740. We want to keep the volume size small because: a) Restores can take forever as the data size gets bigger.
Can you not back up with qtree granularity instead of per volume?
b) 10-14 disks is a good compromise on reliability. Having 2 out of 14 disks go bad at one time is much rarer than 2 out of 51 disks.
You can beat this by using several raid groups per volume. Of course, you need a parity drive for each raid group.
c) If you upgrade disks/shelves in the future, you will likely do it a volume at a time. We did this with a volume with 180GB and the volcopy took 6-7 hours to complete. It is NOT very fast. With 400GB+ in a volume, that should be 2-2.5x longer.
Hmm. My data-shuffling tends to be in smaller pieces. Wouldn't it be nice to have a qtreecopy? 7 hours for 180 GB sounds like about 7 MB/second. If this is on 100 MB, it's not all that bad.
--
Louis
Hi Louis!! =)
Our biggest volumes are 500GB. That is NetApp's recommendation for largest volume size in DOT 6.0 which supports up to 1.5 TB volumes. The problem is that the smaller the volume, the more administrative overhead.
Its nice to have one volume on smaller capacity machines because NFS exports and CIFS shares all point at "/some_data" rather than "/vol/vol0/some_data". This also causes difficulties with naming schemes since we like to see the qtree == cifs share (i.e. /some_project == \filer\some_project). If you have multiple volumes than which of /vol/vol0/some_project and /vol/vol1/some_project is the one pointed at by CIFS share "\filer\some_project".
This statements are nit-picky on my part, but hey, we like to use the KISS principal and multi-volume is just another thing to think about.
More on this stuff...
On Tue, Sep 19, 2000 at 07:47:17AM -0700, Louis Brune wrote:
We have a similar situation on our F740. We want to keep the volume size small because: a) Restores can take forever as the data size gets bigger.Can you not back up with qtree granularity instead of per volume?
I agree. Backup per qtree. Per volume will take forever on restores, especially for small sets of data. There are one or two drawbacks to per-qtree backups, but at least one of those is fixed in DOT 6.0. =)
b) 10-14 disks is a good compromise on reliability. Having 2 out of 14 disks go bad at one time is much rarer than 2 out of 51 disks.You can beat this by using several raid groups per volume. Of course, you need a parity drive for each raid group.
Again, Louis is on track here. With DOT 5.0 and above, they support multiple RAID groups per volume. Each RAID group can be n-1 on disks and still run fine which achieves the data protection you're looking for.
c) If you upgrade disks/shelves in the future, you will likely do it a volume at a time. We did this with a volume with 180GB and the volcopy took 6-7 hours to complete. It is NOT very fast. With 400GB+ in a volume, that should be 2-2.5x longer.
How did you run the volcopy? Over the network or local inside a single filer? What kind of head(s)?
We've found volcopy to be extremely fast, especially for upgrading disks and/or migrating data. In fact, we used volcopy to relocate and replicate a huge chunk of data when Louis' company and ours split and it was a life saver. We were using F760's, but it took about 1 hr. 20 minutes to volcopy ~180GB. That was all in one head that had both a FC-AL adapter and dual SCSI adapters.
Hmm. My data-shuffling tends to be in smaller pieces. Wouldn't it be nice to have a qtreecopy? 7 hours for 180 GB sounds like about 7 MB/second. If this is on 100 MB, it's not all that bad.
Yes! They do have ndmpcopy which will do filer-to-filer data copying, however, there have been bugs posted to toasters regarding it. Something that performed filer-to-filer, per-qtree data moves would be fantastic!
-- Jeff
-- ---------------------------------------------------------------------------- Jeff Krueger E-Mail: jeff@qualcomm.com Senior Engineer Phone: 858-651-6709 NetApp Filers / UNIX Infrastructure Fax: 858-651-6627 QUALCOMM, Inc. IT Engineering Web: www.qualcomm.com
Hi,
BTW, you can create a soft link on the filer called QTREE1 pointing to /vol/vol1/QTREE1 and mount using the link name.
Also regarding ndmpcopy, I've used it many times without any problems and also for moving qtrees or even lower lever directories.
Regards, Eli
----- Original Message ----- From: Jeffrey Krueger jkrueger@qualcomm.com To: Louis Brune lbrune@sd.us.am.ericsson.se Cc: toasters@mathworks.com Sent: Tuesday, September 19, 2000 7:35 PM Subject: Re: Optimal volume size
Hi Louis!! =)
Our biggest volumes are 500GB. That is NetApp's recommendation for
largest
volume size in DOT 6.0 which supports up to 1.5 TB volumes. The problem
is
that the smaller the volume, the more administrative overhead.
Its nice to have one volume on smaller capacity machines because NFS exports and CIFS shares all point at "/some_data" rather than "/vol/vol0/some_data". This also causes difficulties with naming schemes since we like to see the qtree == cifs share (i.e. /some_project == \filer\some_project). If you have multiple volumes than which of /vol/vol0/some_project and /vol/vol1/some_project is the one pointed at by CIFS share "\filer\some_project".
This statements are nit-picky on my part, but hey, we like to use the KISS principal and multi-volume is just another thing to think about.
More on this stuff...
On Tue, Sep 19, 2000 at 07:47:17AM -0700, Louis Brune wrote:
We have a similar situation on our F740. We want to keep the volume
size
small because: a) Restores can take forever as the data size gets bigger.Can you not back up with qtree granularity instead of per volume?
I agree. Backup per qtree. Per volume will take forever on restores, especially for small sets of data. There are one or two drawbacks to per-qtree backups, but at least one of those is fixed in DOT 6.0. =)
b) 10-14 disks is a good compromise on reliability. Having 2 out of
14
disks go bad at one time is much rarer than 2 out of 51 disks.You can beat this by using several raid groups per volume. Of course, you need a parity drive for each raid group.
Again, Louis is on track here. With DOT 5.0 and above, they support multiple RAID groups per volume. Each RAID group can be n-1 on disks and still run fine which achieves the data protection you're looking for.
c) If you upgrade disks/shelves in the future, you will likely do it
a
volume at a time. We did this with a volume with 180GB and the
volcopy took
6-7 hours to complete. It is NOT very fast. With 400GB+ in a
volume, that
should be 2-2.5x longer.How did you run the volcopy? Over the network or local inside a single filer? What kind of head(s)?
We've found volcopy to be extremely fast, especially for upgrading disks and/or migrating data. In fact, we used volcopy to relocate and replicate a huge chunk of data when Louis' company and ours split and it was a life saver. We were using F760's, but it took about 1 hr. 20 minutes to volcopy ~180GB. That was all in one head that had both a FC-AL adapter
and
dual SCSI adapters.
Hmm. My data-shuffling tends to be in smaller pieces. Wouldn't it be nice to have a qtreecopy? 7 hours for 180 GB sounds like about 7 MB/second. If this is on 100 MB, it's not all that bad.
Yes! They do have ndmpcopy which will do filer-to-filer data copying, however, there have been bugs posted to toasters regarding it. Something that performed filer-to-filer, per-qtree data moves would be fantastic!
-- Jeff
--
--
Jeff Krueger E-Mail: jeff@qualcomm.com Senior Engineer Phone: 858-651-6709 NetApp Filers / UNIX Infrastructure Fax: 858-651-6627 QUALCOMM, Inc. IT Engineering Web: www.qualcomm.com
On Wed, Sep 20, 2000 at 09:23:04AM +0200, Eli Lopez wrote:
Hi,
BTW, you can create a soft link on the filer called QTREE1 pointing to /vol/vol1/QTREE1 and mount using the link name.
Cool! I didn't know the NetApp could internally resolve symlinks for NFS/CIFS exports/shares and still give us the "right stuff" from rquotad and what not.
Also regarding ndmpcopy, I've used it many times without any problems and also for moving qtrees or even lower lever directories.
We've used ndmpcopy *extensively*. We've also modified the source to fix some frustrating UI "features". While it works nicely in most instances, last time we tried to do incremental ndmpcopy's to facilitate data migration (like successive rsync's), it hosed a bunch of data on the destination. What's more is that there wasn't any indication in its output that anything went wrong, so we cut-over about 300GB of mission critical data that had an undetermined number of problems throughout it. So all 300GB was suspected of being corrupt.
After the arduous weekend we spent initialy copying this data, we got the opportunity to spend the next weekend re-do'ing the whole effort from tape. Lucky us, huh? =)
I'd like to see a "-i" option in ndmpcopy like there is in jndmpcopy that allows an infinite number of incremental restores. Additionally I'd like ndmpcopy to be a documented and bundled tool with new filers, rather than a "supported tool". This would raise our confidence in it, and to be honest, migrating data is one of the things that NetApp doesn't provide much of an easy solution for. Maybe a robust version of the CLI java jndmpcopy that runs on UNIX and NT with another java program that acts as a GUI front end. That would satisfy both the UNIX and NT folks and be as platform in-specific. Something like the Veritas Storage Administrator...
-- Jeff
-- ---------------------------------------------------------------------------- Jeff Krueger E-Mail: jeff@qualcomm.com Senior Engineer Phone: 858-651-6709 NetApp Filers / UNIX Infrastructure Fax: 858-651-6627 QUALCOMM, Inc. IT Engineering Web: www.qualcomm.com
Jeffrey Krueger wrote:
On Wed, Sep 20, 2000 at 09:23:04AM +0200, Eli Lopez wrote:
Hi,
BTW, you can create a soft link on the filer called QTREE1 pointing to /vol/vol1/QTREE1 and mount using the link name.
Cool! I didn't know the NetApp could internally resolve symlinks for NFS/CIFS exports/shares and still give us the "right stuff" from rquotad and what not.
Yeah, the same trick allows us to share our CIFS homedirs under a single root, without actually moving the user's home directory.
Moshe
-- ----------------------------------------------------------------------------- Moshe Linzer | On the Internet, Unix Systems Manager | National Semiconductor, Israel | nobody knows you're a moron. Phone: 972-9-970-2247 | Fax: 972-9-970-2001 | - Network Magazine Email: moshel@nsc.com | -----------------------------------------------------------------------------
jn Tue, 26 Sep 2000, Moshe Linzer wrote:
Jeffrey Krueger wrote:
On Wed, Sep 20, 2000 at 09:23:04AM +0200, Eli Lopez wrote:
BTW, you can create a soft link on the filer called QTREE1 pointing to /vol/vol1/QTREE1 and mount using the link name.
Cool! I didn't know the NetApp could internally resolve symlinks for NFS/CIFS exports/shares and still give us the "right stuff" from rquotad and what not.
Yeah, the same trick allows us to share our CIFS homedirs under a single root, without actually moving the user's home directory.
Whoa. What's this? I knew symlinks if resolved on the same filesystem on a filer work correctly under CIFS, but this seems different. You can symlink across *volumes* on the same filer and it is transparent to NFS and CIFS clients???
Until next time...
The Mathworks, Inc. 508-647-7000 x7792 3 Apple Hill Drive, Natick, MA 01760-2098 508-647-7001 FAX tmerrill@mathworks.com http://www.mathworks.com ---
On Thu, 28 Sep 2000, Todd C. Merrill wrote:
On Tue, 26 Sep 2000, Moshe Linzer wrote:
Jeffrey Krueger wrote:
On Wed, Sep 20, 2000 at 09:23:04AM +0200, Eli Lopez wrote:
Aieee! EATTRIB2DEEP!
BTW, you can create a soft link on the filer called QTREE1 pointing to /vol/vol1/QTREE1 and mount using the link name.
Cool! I didn't know the NetApp could internally resolve symlinks for NFS/CIFS exports/shares and still give us the "right stuff" from rquotad and what not.
Yeah, the same trick allows us to share our CIFS homedirs under a single root, without actually moving the user's home directory.
Whoa. What's this? I knew symlinks if resolved on the same filesystem on a filer work correctly under CIFS, but this seems different. You can symlink across *volumes* on the same filer and it is transparent to NFS and CIFS clients???
<Scratches chin, tries to think back, way back, to the late 1900's...>
Y'know, I can't recall exactly now, but I thought we tried that trick on a filer running an early 5.x release... when we went from a pure, pristine NFS-only shop to the bag of rabid weasels that is a mixed CIFS/NFS environment, we had to figure out how to map a flat /home/users/* namespace into CIFS shares. Only our /home/users was really populated with symlinks to about a dozen separate subdirectories of a users qtree, which segregated them according to groups, so that NFS exports of those groups could be divided into netgroups of machines that were owned by said groups. (Phew! Say that five times fast.) I.e., we could restrict exports to some machines so that they could only "see" home directories for users in that group, but on shared central servers we could mount the parent of the users tree and see everything. Given the (political) circumstances that necessitated this somehwat bizarre arrangement, this was Very clean. Nee, even elegant in its own way.
And, of course, Windows choked on it like month-old leftovers.
Because cifs.home_dir only accepted one directory argument to do its magic, we were stuck. We had a bunch of separate roots, not just one, so now typing "cifs shares" on said filer produces something resembling the Portland Metro Area phone book. My, how we scroll! (I'm sure the folks that read the autosupport mail have probably seen worse, though. :-)
So I put in an RFE a couple of years ago to expand the cifs.home_dir option to accept multiple paths, rather than just a single directory. Like setting $PATH to a colon-separated list in any Unix shell:
/vol/vol0/users/dirA:/vol/vol0/users/dirB:/vol/vol1/users/dirC
I think someone else had already submitted that one too, around the same time.
"So, how is any of this relevant to the discussion at hand?" I hear you cry.
In glancing over the 6.0 release notes, I *thought* I read that this very RFE has finally been included in the release. So, rather than worry about directories of symlinks or the vagaries of resolving them across volumes in a mixed NFS/CIFS environment, you can simply set cifs.home_dir to a collection of paths and be done with it. Hooray! Of course, I'm far too tired to log into NOW and look for it and see if this is, indeed, the case, so don't take my word for it. I've had enough caffeine today to incapacitate a large barnyard animal.
OKAY, so sleep is overrated. I went and looked. Bug ID 13856 is the RFE that caught my eye, but it doesn't quite describe the thing I requested, although it does appear to add some interesting functionality, although it's full of Windows goop and not what I was describing with the path idea. Bzzzt. Sorry to get your hopes up. Please play again soon.
It's funny is that I really can't remember if we tried setting cifs.home_dir to the directory full of symlinks or not. We must have. Really. I've probably blocked out the whole sordid affair. If that had worked in 5.2.x and we could have avoided defining and maintaining all those separate shares, by hand, for every bloomin' user... but heck, that's what we have NT admins for. :-)
Nowadays I'm unencumbered by such restraints and I have one nice flat /home, and cifs.home_dir works famously. Still, even though I don't have a need for it anymore, I think having cifs.home_dir take a $PATH-style argument would be a spiffy feature. Don't you agree, Toastadminarians?
Cheers!
-- Chris
-- Chris Lamb, Unix Guy MeasureCast, Inc. 503-241-1469 x247 skeezics@measurecast.com
"Todd C. Merrill" wrote:
jn Tue, 26 Sep 2000, Moshe Linzer wrote:
Jeffrey Krueger wrote:
On Wed, Sep 20, 2000 at 09:23:04AM +0200, Eli Lopez wrote:
BTW, you can create a soft link on the filer called QTREE1 pointing to /vol/vol1/QTREE1 and mount using the link name.
Cool! I didn't know the NetApp could internally resolve symlinks for NFS/CIFS exports/shares and still give us the "right stuff" from rquotad and what not.
Yeah, the same trick allows us to share our CIFS homedirs under a single root, without actually moving the user's home directory.
Whoa. What's this? I knew symlinks if resolved on the samefilesystem on a filer work correctly under CIFS, but this seems different. You can symlink across *volumes* on the same filer and it is transparent to NFS and CIFS clients???
Yes, the difference is that you are SHARING THE SYMLINK, not putting a symlink inside a share. So without quoting all of Chris' email, you can set your cifs homedir to a directory full of symlinks (userid -> /path/to/homedir), and each user will get a share to his own homedir upon login! Cool undocumented feature.
And it appears to be true that some 6.X release will have multiple homedir paths, but this is still more versatile!
Moshe
-
Chris Lamb -
Clarke, John L. (CIS) -
Eli Lopez -
Jeffrey Krueger -
Louis Brune -
Moshe Linzer -
Moshe Linzer -
Todd C. Merrill