UFS (Solaris9/Solaris10) will cut it, but probly isn't the best without tuning, and definately not on Linux. If your using Linux, carefully consider JFS and XFS, both can do what you want but have up and downsides. Do NOT run ReiserFS in production, too many people have been bitten by it. EXT2/3 are just not even contenders, and who wants to pay for VxFS. For your requirements my first attempt (on Linux anyway) would be JFS.
And, ZFS, btw, would handle this all nicely, but won't be introduced to Solaris10 untill Update1, sometime in the summer.
benr.
-----Original Message----- From: owner-toasters@mathworks.com on behalf of Maren S. Leizaola Sent: Fri 2/11/2005 2:35 AM To: toasters@mathworks.com Cc: Subject: high performance file systems, non Netapp.
Hi, We have an application where we need to store between 4 to 20 million small files on the on a large drive or a 3 drive raid system.
We are finding that file systems like UFS just don't cut it never mind Linux extFS... These file systems can handle what we are trying to do but they tend to slow down as they get full and in some cases we will have to be able to delete millions of files in one go.
In this type of configuration unfortunately we can't bundle at netapp as the technical overhead and cost would make the system unfeasable.
I have so far seen the ReiserFS can do what we are looking for but we need a fs that will work on Linux, FreeBSD, Solaris etc... We must have a path to scale up or deploy a heavy load system and migrate to a big sun box if required.
Anyone knows how well Solaris 10's ZFS would work for this kind of application?
What do you guys suggest?
Maren.
-------------------------------------------------------------- HKdotCOM Ltd Tel: 852 2865-4865 ext 888 Fax: 852 2865-4100 leizaola@hk.com AIM: MarenHKdotCOM ICQ: 39905706 MSN:MarenHKdotCOM -------------------------------------------------------------- Get your @hk.com email address on http://www.hk.com
Hi, We have an application where we need to store between 4 to 20 million small files on the on a large drive or a 3 drive raid system.
Hi Maren
A) How small is "small"? All current filesystems have the ability to store very small data files without using any additional datablocks by storing the filecontent in the file-inode itself. This sizelimit varies on the different filesystems... Therefore we need to know the size of your "small" files. Less than 64 Bytes? Less than x Bytes?
B) I now assume that your database produces files with "ugly" appr. 100 bytes/file ...
Filesystems are hierarchical databases - to usualy store big(!) amounts of data - using the filenames (incl. the path) as primary keys to relocate the stored data. Another focus is the multiuser management. Therefore I dare to declare, that any filesystem you might choose will be the "wrong" kind of database for the type of data (small data) your application produces.
=> Do you have the possibility to change the application?
If yes: 1) Is it possible to collect the data of multiple files to single files?
2) How about changing the application to use a "real" dedicated database for managing all these small data records? There are many databases that can handle even many small entries with lower response times than hierarchical filesystems. And these database files could be stored on a NetApp Filer... :-)
If no: 3) For Linux or IRIX, I would choose XFS. It's solid like a rock and flexible like a rubber band incl. a very dynamic flexible inode-management. As long a there is some space in the filesystem left, it will automatically create new inodes if required. But there is no inbuild version control like WAFL offers. :-(
Assuming the worst case 100 bytes/file scenario, my personal choice would be (2). "Back to the filers." ;-)
Smile & regards Dirk
On Sat, 12 Feb 2005, Dirk Schmiedt wrote:
Hi, We have an application where we need to store between 4 to 20 million small files on the on a large drive or a 3 drive raid system.
Hi Maren
A) How small is "small"?
about 8KB to 64KB's.
All current filesystems have the ability to store very small data files without using any additional datablocks by storing the filecontent in the file-inode itself. This sizelimit varies on the different filesystems...
well we initially just set a 4gb partition for the job and at 800,000 files we ran out of inodes... bigger drives do solve this issue..
Therefore we need to know the size of your "small" files. Less than 64 Bytes? Less than x Bytes?
less than 64KB.
B) I now assume that your database produces files with "ugly" appr. 100 bytes/file ...
Filesystems are hierarchical databases - to usualy store big(!) amounts of data - using the filenames (incl. the path) as primary keys to relocate the stored data. Another focus is the multiuser management. Therefore I dare to declare, that any filesystem you might choose will be the "wrong" kind of database for the type of data (small data) your application produces.
very true.
=> Do you have the possibility to change the application?
If yes:
- Is it possible to collect the data of multiple files to single files?
well the data is several databases will millions of tables... using mysql, still i wonder if any of the other databases have created their own file system or monster database into which its stores data efficiently.
- How about changing the application to use a "real" dedicated
database for managing all these small data records?
ermm.... that is what we are doing.... all the records could have been put into a single table but then the table would have a few billion records... so we have moved the goal post and have a problem at file system by spliting the data into many tables.
There are many databases that can handle even many small entries with lower response times than hierarchical filesystems. And these database files could be stored on a NetApp Filer... :-)
aye, i wish we could just bundle a netapp... in a way it would just complicate things so much, licensing on going support etc...
Before going that path it is simpler to get a commercial database for the job that has some custom file system. Any suggestions?
If no: 3) For Linux or IRIX, I would choose XFS. It's solid like a rock and flexible like a rubber band incl. a very dynamic flexible inode-management. As long a there is some space in the filesystem left, it will automatically create new inodes if required. But there is no inbuild version control like WAFL offers. :-(
Linux is starting to starting to appear to be the choice if we don't fine a database that has a custom file/storage system.
Assuming the worst case 100 bytes/file scenario, my personal choice would be (2). "Back to the filers." ;-)
i guess my "small files" are pretty big.
thanks for get great replies to all of you.
Maren.
-------------------------------------------------------------- HKdotCOM Ltd Tel: 852 2865-4865 ext 888 Fax: 852 2865-4100 leizaola@hk.com AIM: MarenHKdotCOM ICQ: 39905706 MSN:MarenHKdotCOM -------------------------------------------------------------- Get your @hk.com email address on http://www.hk.com
"Maren" == Maren S Leizaola leizaola@hk.com writes:
Maren> well the data is several databases will millions of tables... using Maren> mysql, still i wonder if any of the other databases have created Maren> their own file system or monster database into which its stores data Maren> efficiently.
To me, this tells me that you're doing something really really wrong with your DB design. You should have lots of ROWS or COLUMNS in a much much smaller number of tables. I think you need to really explain what you are trying to do here, or get some DB design help.
I personally can't imagine why you need so many tables. It would be easier to just add an index to each set of data to say how it should be grouped. Presto! No need for multiple tables, just more columns in the existing tables.
Once you fix your data design problems, you won't see the disk issues you are currently seeing, where you try to delete many rows (or tables) of data.
John John Stoffel - Senior Staff Systems Administrator - System LSI Group Toshiba America Electronic Components, Inc. - http://www.toshiba.com/taec john.stoffel@taec.toshiba.com - 508-486-1087
On Fri, 11 Feb 2005, Ben Rockwood wrote:
UFS (Solaris9/Solaris10) will cut it, but probly isn't the best without tuning, and definately not on Linux.
we are using UFS2 in a single directory and it is ok for reading and writing files but deleting them takes for ever.
From what I can see UFS2 is not much better than UFS for this application.
If your using Linux, carefully consider JFS and XFS,
The dynamic allocation of inodes is interesting....
We don't need journaling the data in the database is not critical..
I just found this... which is interesting... http://en.wikipedia.org/wiki/Comparison_of_file_systems
both can do what you want but have up and downsides. Do NOT run ReiserFS in production, too many people have been bitten by it. EXT2/3 are just not even contenders, and who wants to pay for VxFS. For your requirements my first attempt (on Linux anyway) would be JFS.
And, ZFS, btw, would handle this all nicely, but won't be introduced to Solaris10 untill Update1, sometime in the summer.
what a scam... they announced it was part of solaris 10... and it is not included??
maren.
benr.
-----Original Message----- From: owner-toasters@mathworks.com on behalf of Maren S. Leizaola Sent: Fri 2/11/2005 2:35 AM To: toasters@mathworks.com Cc: Subject: high performance file systems, non Netapp.
Hi, We have an application where we need to store between 4 to 20 million small files on the on a large drive or a 3 drive raid system.
We are finding that file systems like UFS just don't cut it never mind Linux extFS... These file systems can handle what we are trying to do but they tend to slow down as they get full and in some cases we will have to be able to delete millions of files in one go.
In this type of configuration unfortunately we can't bundle at netapp as the technical overhead and cost would make the system unfeasable.
I have so far seen the ReiserFS can do what we are looking for but we need a fs that will work on Linux, FreeBSD, Solaris etc... We must have a path to scale up or deploy a heavy load system and migrate to a big sun box if required.
Anyone knows how well Solaris 10's ZFS would work for this kind of application?
What do you guys suggest?
Maren.
HKdotCOM Ltd Tel: 852 2865-4865 ext 888 Fax: 852 2865-4100 leizaola@hk.com AIM: MarenHKdotCOM ICQ: 39905706 MSN:MarenHKdotCOM
Get your @hk.com email address on http://www.hk.com
-------------------------------------------------------------- HKdotCOM Ltd Tel: 852 2865-4865 ext 888 Fax: 852 2865-4100 leizaola@hk.com AIM: MarenHKdotCOM ICQ: 39905706 MSN:MarenHKdotCOM -------------------------------------------------------------- Get your @hk.com email address on http://www.hk.com
-
Ben Rockwood -
Dirk Schmiedt -
John Stoffel -
Maren S. Leizaola