toasters

toasters@lists.teaparty.net

2 participants
13529 discussions

inode limitation?
by Magnus Thorne 16 Jan '01

16 Jan '01

Greetings, I'm planning on using a NetApp 760 to store *many* small files. However, I'm getting vague answers from NetApp for my problem, so I figure I'd ask here. NetApp has told me that they don't recommend that I have more than 250,000 inodes per volume. Is this a real limitation, because my NetApps currently have 6.1 million inodes available? The online documentation says I only need to add more hard drives to get more inodes (1 per 32K). Has anyone used the netapp for small files? Small means millions of < 60K. Also, I need all these files available on one share. Also, what should I look out for? I know to disable the read ahead cache. I plan to not use snapshots. thanx, -Mag

4 3

FW: /home layout with many filers and NIS automount
by Rainchik, Aleksandr (MED, Non GE) 15 Jan '01

15 Jan '01

Basically, if it's indirect map, as auto_home, you don't need to kick "automount" on every client, but if it's direct map - you have to. -----Original Message----- From: Jeffrey Krueger [mailto:jkrueger@qualcomm.com] Sent: Monday, January 15, 2001 12:38 PM To: Rainchik, Aleksandr (MED, Non GE) Cc: 'Jeffrey Krueger '; 'toasters(a)mathworks.com ' Subject: Re: /home layout with many filers and NIS automount On Sat, Jan 13, 2001 at 03:16:20PM -0600, Rainchik, Aleksandr (MED, Non GE) wrote: > > There is another thing against > > #auto_home NIS map > > user1 filer2:/vol/vol1/& > design. We have a lot of development servers. So any time you > add/delete/modify a single user you should go trough _all_ the > NIS/automount clients and run "automount" to let automountd know > that passwd map has been just updated. Am I wrong again? This is correct, but I simple cron job which runs periodically on all NIS clients should make short work of this task. =) -- Jeff

1 0

Re: inode limitation?
by Ketil Kirkerud Elgethun 15 Jan '01

15 Jan '01

Magnus Thorne <Magnus.Thorne(a)evoice.com> writes: > NetApp has told me that they don't recommend that I have more than 250,000 > inodes per volume. Are you sure you're not confusing this with "per directory" ? Even though WAFL is fairly efficient when it comes to large directories, there are limits to what you would want to experience.. > Is this a real limitation, because my NetApps currently > have 6.1 million inodes available? Neo, it's not. This is a 760: df Filesystem kbytes used avail capacity Mounted on /vol/vol0/ 343184120 265805356 77378764 77% /vol/vol0/ df -i Filesystem iused ifree %iused Mounted on /vol/vol0/ 13987164 6012814 70% /vol/vol0/ And it's running just fine (mailspool with one file per message) > The online documentation says I only > need to add more hard drives to get more inodes (1 per 32K). You can have 1 inode per 4K, IIRC. > Has anyone > used the netapp for small files? Small means millions of < 60K. Also, I > need all these files available on one share. As you can see: yes, no problems. > > Also, what should I look out for? I know to disable the read ahead cache. I'm not sure that would help a lot with small files - this is more of a win wit large files (or possible very slow clients) > I plan to not use snapshots. Whether or not to use snapshots is more a question of the change rate in your data set than the number of files - the 760 shown above is running with readahead set, and snapshots. No problems whatsoever. Regards, -- ---Ketil Kirkerud Elgethun, Infostream ASP

1 0

RE: inode limitation?
by Toal, Dave 15 Jan '01

15 Jan '01

Jim, The website I fix uses very small files to handle details of customer events so they can be billed. One file per event. I'm not saying this is a good design. <chuckle> smith> maxfiles spool Volume spool: maximum number of files is currently 8258829 (1428275 used). smith> df -i /vol/spool Filesystem iused ifree %iused Mounted on /vol/spool/ 1428275 6830554 17% /vol/spool/ smith> df /vol/spool Filesystem kbytes used avail capacity Mounted on /vol/spool/ 46944924 14936040 32008884 32% /vol/spool/ /vol/spool/.snapshot 0 7834048 0 ---% /vol/spool/.snapshot smith> How come it's not "maxfiles /vol/spool" -- different coders? Dave -----Original Message----- From: Jim VandeVegt To: Magnus Thorne; 'toasters(a)mathworks.com' Sent: 1/15/01 2:38 PM Subject: Re: inode limitation? --- Magnus Thorne <Magnus.Thorne(a)evoice.com> wrote: > NetApp has told me that they don't recommend that I have more than > 250,000 > inodes per volume. Is this a real limitation, because my NetApps > currently > have 6.1 million inodes available? The online documentation says I > only > need to add more hard drives to get more inodes (1 per 32K). Has > anyone > used the netapp for small files? Small means millions of < 60K. > Also, I > need all these files available on one share. The training I received from NetApp personnel indicated that a particular strength of WAFL is the ability to add more inodes on the fly. There wasn't any mention of any particular recommended limit. A quick check of my system shows that even my smallest volumes currently have "Maximum Numer of Files" set over 1 million. On a 760, one volume with 330 GB of data space, 13 million on MAX files, 550 thousand currently. Another with 150 GB of data space, 6 million on MAX files, 414 thousand currently. As far as I know, no one at our site purposely expanded any of the MAX settings. ===== Jim VandeVegt vandevegt(a)yahoo.com vandevegt(a)avaya.com http://members.xoom.com/vandevegt Toilet paper should come off the front! __________________________________________________ Do You Yahoo!? Get email at your own domain with Yahoo! Mail. http://personal.mail.yahoo.com/

1 0

solaris 2.6 nfs client tuning?
by Toal, Dave 15 Jan '01

15 Jan '01

Hey, all. This lots-of-nfs-mounts topic inspires me to ask yet another sanity-check question: Currently I'm fixing a process which moves lots of data from one vol to another. Can't simply vol copy because there are things I can't over-write on the destination. This data is in about 40 directories, with maybe 2G in each. The method I'm using is to set up many pairs of mounts, one for each directory, all from the same two vols. This is because the nfs client, a sun 450 running solaris 2.6, shows huge lag for ls, cp, whatever, in a mounted directory where a cp is already running. iostat shows the nfsd devices at 100% busy -- for destination and source -- when a cp runs between a mount-point pair. So: filer1:/vol/A/d1 /mnt/A_01 filer2:/vol/B/d1 /mnt/B_01 filer1:/vol/A/d2 /mnt/A_02 filer2:/vol/B/d2 /mnt/B_02 filer1:/vol/A/d3 /mnt/A_03 filer2:/vol/B/d3 /mnt/B_03 ad nauseum... and if I cp /mnt/A_01/fatfile to /mnt/B_01 then there's no i/o lag whatsoever in /mnt/A_02. These filers are 760's, by the way. Then I discovered a serious "sweet spot" at a very low number. "max" is the number of simultaneous copies between mount point pairs. "k/sec" are my visual estimates of _each_ cp rate, from watching iostat for about a minute; rates jump around by +- 100 k/sec. Column on the right is total throughput for all cp. max=3 ~ 2300 k/sec |= 6900 max=4 ~ 1800 k/sec |= 7200 max=5 ~ 1500 k/sec |= 7500 max=6 ~ 1200 k/sec |= 7200 max=8 ~ 800 k/sec |= 6400 max=9 ~ 800 k/sec |= 7200 max=10 ~ 600 k/sec |= 6000 I've seen much higher data rates from the 760's. So I'm thinking this means I need to learn nfs client tuning for solaris. Yes? Dave -- dave toal systems optimist Thomson & Thomson N. Quincy, MA -----Original Message----- From: Jeffrey Krueger To: Rainchik, Aleksandr (MED, Non GE) Cc: 'Jeffrey Krueger '; 'toasters(a)mathworks.com ' Sent: 1/15/01 1:37 PM Subject: Re: /home layout with many filers and NIS automount On Sat, Jan 13, 2001 at 03:16:20PM -0600, Rainchik, Aleksandr (MED, Non GE) wrote: > > There is another thing against > > #auto_home NIS map > > user1 filer2:/vol/vol1/& > design. We have a lot of development servers. So any time you > add/delete/modify a single user you should go trough _all_ the > NIS/automount clients and run "automount" to let automountd know > that passwd map has been just updated. Am I wrong again? This is correct, but I simple cron job which runs periodically on all NIS clients should make short work of this task. =) -- Jeff

1 0

Re: inode limitation?
by Jim VandeVegt 15 Jan '01

15 Jan '01

--- Magnus Thorne <Magnus.Thorne(a)evoice.com> wrote: > NetApp has told me that they don't recommend that I have more than > 250,000 > inodes per volume. Is this a real limitation, because my NetApps > currently > have 6.1 million inodes available? The online documentation says I > only > need to add more hard drives to get more inodes (1 per 32K). Has > anyone > used the netapp for small files? Small means millions of < 60K. > Also, I > need all these files available on one share. The training I received from NetApp personnel indicated that a particular strength of WAFL is the ability to add more inodes on the fly. There wasn't any mention of any particular recommended limit. A quick check of my system shows that even my smallest volumes currently have "Maximum Numer of Files" set over 1 million. On a 760, one volume with 330 GB of data space, 13 million on MAX files, 550 thousand currently. Another with 150 GB of data space, 6 million on MAX files, 414 thousand currently. As far as I know, no one at our site purposely expanded any of the MAX settings. ===== Jim VandeVegt vandevegt(a)yahoo.com vandevegt(a)avaya.com http://members.xoom.com/vandevegt Toilet paper should come off the front! __________________________________________________ Do You Yahoo!? Get email at your own domain with Yahoo! Mail. http://personal.mail.yahoo.com/

1 0

Re: /home layout with many filers and NIS automount
by Jim VandeVegt 15 Jan '01

15 Jan '01

--- Andrew Steingruebl <steingra(a)pprd.abbott.com> wrote: > "Rainchik, Aleksandr (MED, Non GE)" said: > > > > >Speaking about this case: > >> > >> #auto_home NIS map > >> user1 filer2:/vol/vol1/& > >> .... > >> user9999 filer3:/vol/vol4/& > > > >What I'm afraid of, is that if you have 200 users > >logged in your box, you will have 200 mountpoints, > >one mountpoint per each holy user!!! I really feel > >bad about it. Am I wrong? In the Solaris /etc/auto_home file you can specify an additional colon in the directory path to indicate the highest exported directory, and the automounter will use some sort of symbolic link such that only one connection per exported file system will result. A 'df- k', however, might still show you all the connections, I haven't tested with a large number as of yet. The syntax for /etc/auto_home for the above example would then be: user0001 filer2:/vol/vol1:& user9999 filer3:/vol/vol4:& Assuming you have /vol/vol? exported on the Filers. ===== Jim VandeVegt vandevegt(a)yahoo.com vandevegt(a)avaya.com http://members.xoom.com/vandevegt Toilet paper should come off the front! __________________________________________________ Do You Yahoo!? Get email at your own domain with Yahoo! Mail. http://personal.mail.yahoo.com/

1 0

RE: /home layout with many filers and NIS automount
by Rainchik, Aleksandr (MED, Non GE) 15 Jan '01

15 Jan '01

Speaking about this case: > > #auto_home NIS map > user1 filer2:/vol/vol1/& > .... > user9999 filer3:/vol/vol4/& What I'm afraid of, is that if you have 200 users logged in your box, you will have 200 mountpoints, one mountpoint per each holy user!!! I really feel bad about it. Am I wrong? -----Original Message----- From: Jeffrey Krueger [mailto:jkrueger@qualcomm.com] Sent: Friday, January 12, 2001 2:31 PM To: Rainchik, Aleksandr (MED, Non GE) Cc: toasters(a)mathworks.com Subject: Re: /home layout with many filers and NIS automount On Fri, Jan 12, 2001 at 10:47:54AM -0600, Rainchik, Aleksandr (MED, Non GE) wrote: > Hello, > > I just need a word of advice on how to structure /home if you > use automount, have 5-10 home directory servers (NetApp filers) > and around 1000 users. Point of clarification: 5-10 filers for 1,000 users? What model filers are they and I hope they are serving other data besides homedirs! At one point we had about 15,000 home directories mostly on two NetApps - a F330 and a F540 - and they were pretty bored filers. One way to simplify your design would be to reduce the number of filers serving homedirs, especially for 1,000 users. Two good reasons to spread out homedirs are for bandwidth constraints and to spread risk/reduce points of failure. But if you have consistently high bandwidth between all users and filers than 2-4 homedir servers should be more than plenty mitigation of risk considering the cost of additional filers versus the additional benefit (considering NetApps impressive reliability). > I'm personally is looking for /home/homeserver-vol#/user-name > scheme, so my automount maps will be nice, small and manageble: > > #auto_home NIS map > filer1-0 filer1:/vol/vol0 > filer1-1 filer1:/vol/vol1 > ... > filer5-5 filer5:/vol/vol5 This is a very restrictive design because it makes homedir moves extremely conspicuous. It also flies a bit in the face of a large selling point of the automounter - eliminating host-based network paths. If a user is moved from one filer to another, or even from one volume to another, they will have a whole new path to their home directory. Worse yet, any scripts or application configurations which use the full path to their home directory will need to be re-coded or re-configured - YUK! Being able to "hide" the location of a home directory (or any other remote filesystem) is extremely useful. > But it seems like users prefer /home/username scheme and it leads > to 1000+ NIS automount map for /home, like > > #auto_home NIS map > user1 filer2:/vol/vol1/& > .... > user9999 filer3:/vol/vol4/& This is the way we do it, and the way I'd venture to guess most/many places do it. From the user perspective, it is very simple. From the admin perspective, it is easy to manage. We've found it to give maximum flexibility with minimum hassle. > Do you know if there is more elegant solution for this problem? Can you > tell me, what is your preference for /home layout? What do you mean by elegant? The above is pretty darned elegant, and for 1,000 users it should be easily manageable too. We currently have 10,223 home directories and we've got one big home directory map like above. It works great! One recommendation is you may want to use a script or database to help generate that automount map. The easiest way would be to write a shell/perl script that scanned all your "homedir" directories on each filer, ensured username uniqueness and then build the automount map. For added flexibility, we use a small database. Each row in the homedir table has the username (mountpoint under /home), the filer, and the path to their homedir. At some interval a script comes thru and builds the map. Hope this info helps! -- Jeff -- ------------------------------------------------------------------------ ---- Jeff Krueger, NetApp CA E-Mail: jeff(a)qualcomm.com Senior Engineer Phone: 858-651-6709 NetApp Filers / UNIX Infrastructure Fax: 858-651-6627 QUALCOMM, Inc. IT Engineering Web: www.qualcomm.com

3 2

Can volumes span FCAL controllers?
by Matt Phelps 13 Jan '01

13 Jan '01

Hi, We're probably going to ba adding another FCAL controller to our 840 soon. Can volumes include disks that are on both controllers? -- Matt Phelps System Administrator, Computation Facility Harvard - Smithsonian Center for Astrophysics mphelps(a)cfa.harvard.edu, http://cfa-www.harvard.edu/~mphelps

6 9

/home layout with many filers and NIS automount
by Rainchik, Aleksandr (MED, Non GE) 13 Jan '01

13 Jan '01

Hello, I just need a word of advice on how to structure /home if you use automount, have 5-10 home directory servers (NetApp filers) and around 1000 users. I'm personally is looking for /home/homeserver-vol#/user-name scheme, so my automount maps will be nice, small and manageble: #auto_home NIS map filer1-0 filer1:/vol/vol0 filer1-1 filer1:/vol/vol1 ... filer5-5 filer5:/vol/vol5 But it seems like users prefer /home/username scheme and it leads to 1000+ NIS automount map for /home, like #auto_home NIS map user1 filer2:/vol/vol1/& .... user9999 filer3:/vol/vol4/& Do you know if there is more elegant solution for this problem? Can you tell me, what is your preference for /home layout? Thank you. -- Aleksandr Rainchik, SUN CSA

6 5

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters