We are mounting storage for a (brain dead) application which stores files generated daily on a Filer Volume. The dir has 125,000 files and growing.
The mount point is on a Solaris 2.8 box. What is the limit on number of files in a folder in this context? Anybody know?
Thanks
-Rob
We are mounting storage for a (brain dead) application which stores files generated daily on a Filer Volume. The dir has 125,000 files and growing.
The mount point is on a Solaris 2.8 box. What is the limit on number of files in a folder in this context? Anybody know?
I don't know any limit here (either for UFS or NFS). Obviously you start reducing performance on access because there's no file index in the directory, but the lack of an index also means that there's no static table to overflow here.
You might have to increase maxdir size on the filer at some point. I don't know how many files the standard limit will allow.
IMHO, Solaris and NFS aren't your concern. It is the number of inodes left available on the volume on the filer. Solaris 2.8 native file systems would have the same limitation. I believe NFSv2 couldn't handle files >2GB but as far as I know experienced nothing more than performance degradation because of file quantity. I would recommend sorting the files by month and/or year if possible. I know of once customer who has exceeded the inode capacity of his filer but we are talking 10s of millions of filers in a single directory structure.
Maintain adequate free blocks and free inodes If the percentage of free blocks or free inodes falls to less than 10% on any volume, the performance of writes and creates can suffer. Check free blocks and inodes using the df command and df -i command, respectively. If the percentage of used blocks is greater than 90%, increase blocks by adding disks or deleting Snapshots. If the percentage of free inodes is less than 10%, increase inodes by deleting files or using the maxfiles command.
NAME na_maxfiles - increase the number of files the volume can hold
SYNOPSIS maxfiles [ vol_name [ max ] ]
DESCRIPTION maxfiles increases the number of files that a volume can hold to max. Once increased, the value of max can never be lowered, so the new value must be larger than the cur- rent value. If no argument is specified, maxfiles dis- plays the current value of max for all volumes in the sys- tem. If just the vol_name argument is given, the current value of max for the specified volume is displayed.
See "man maxfiles" for more information.
netapp1> maxfiles Volume vol0: maximum number of files is currently 194533 (10281 used). Volume vol1: maximum number of files is currently 182363 (1533 used). netapp1> df -i Filesystem iused ifree %iused Mounted on /vol/vol0/ 10281 184252 5% /vol/vol0/ /vol/vol1/ 1533 180830 1% /vol/vol1/
E.
On Fri, 6 May 2005, Darren Dunham wrote:
We are mounting storage for a (brain dead) application which stores files generated daily on a Filer Volume. The dir has 125,000 files and growing.
The mount point is on a Solaris 2.8 box. What is the limit on number of files in a folder in this context? Anybody know?
I don't know any limit here (either for UFS or NFS). Obviously you start reducing performance on access because there's no file index in the directory, but the lack of an index also means that there's no static table to overflow here.
On UFS2, the maximum number of files is 1,000,000. UFS2 includes FFS which provides indexing of the directory. Accessing files in a directory which that many files is was not too bad in our experience using a single drive under FreeBSD, it did it in less than 1 second. What was very slow was to delete files, about 16 seconds per file.
I worked out that if your average file size is greater than 8KB you will run out of disk space before you run out of inodes going by the default inode count set by newfs.
default fragment size is 2048bytes, by default it creates one inode for every 4 fragsize. hence the 8KB i mention earlier.
One thing you can do is set the number of inodes with newfs -i, the default being 8196. I guess you could go down to 2048, I am not sure though what would happen if you go below that figure.
So first work out your average file size... and then decide how to newfs the drive. If you are not using a netapp and you are using a raid controller that allows you to set the stripe size on a volume you will get better performance if you also match the stripe size to the average file size.
A word of caution though. If you think that the average file size is likely to increase in the near future or you are close to the boundary between tripe sizes, use next stripe size. Eg, your average file size is 30KB a strip of 32KB maybe best at present. If the average file size increases over 32KB chose 64KB... of course makes sure that the average is based on the majority of the files, make your average is not pushed over the boundary because you have 1 file which is 4GB or something.
Anyway this is all pretty simplistic compared to how the netapp manages WAFL and stripe sizes. I can see that an admin can increase the inode on a file system. I guess that then re-structures the file system on the fly. have a read. http://www.netapp.com/tech_library/3002.html
Regards, Maren.
You might have to increase maxdir size on the filer at some point. I don't know how many files the standard limit will allow.
-- Darren Dunham ddunham@taos.com Senior Technical Consultant TAOS http://www.taos.com/ Got some Dr Pepper? San Francisco, CA bay area < This line left intentionally blank to confuse you. >
-------------------------------------------------------------- HKdotCOM Ltd Tel: 852 2865-4865 ext 888 Fax: 852 2865-4100 leizaola@hk.com AIM: MarenHKdotCOM ICQ: 39905706 MSN:MarenHKdotCOM -------------------------------------------------------------- Get your @hk.com email address on http://www.hk.com
-
Darren Dunham -
Eric Kimminau -
Maren S. Leizaola -
rob-7704＠austin.rr.com