Since there has been a lot of discussion about namespace virtualization lately, I thought I'd throw in a question.
Nightly, an application creates a fairly large number of files, makes a new directory on an R200, and then dumps the data to it. For a number of reasons, the volume this data is on is 5TB in size and it's currently half full (or half empty, depending on your way of looking at things). The R200 is running 6.5.3P4 and has plenty of space for additional volumes.
From the perspective of the application generating the data as well as from
the perspective of the users accessing it, the volume is expected to be accessible under one particular name. The obvious question is: What do I do when /vol/foo runs out of space and I need /vol/foo2? Usually, I'd play automounter tricks since the directories under the volumes are all uniquely named, but I'd rather not. This is NFS-only and the clients are all sorts of different UNIXish systems.
Any suggestions?
Thanks!
Alex.
Alex> Nightly, an application creates a fairly large number of files, Alex> makes a new directory on an R200, and then dumps the data to Alex> it.
I assume it actually works the other way? Creates a directory on the R200 and dumps the files into there? Something like /vol/foo/nightly.<date>/... ?
Alex> For a number of reasons, the volume this data is on is 5TB in Alex> size and it's currently half full (or half empty, depending on Alex> your way of looking at things). The R200 is running 6.5.3P4 and Alex> has plenty of space for additional volumes.
Or you can grow the existing volume... do you need to keep all the older data you've written online?
Alex> From the perspective of the application generating the data as Alex> well as from the perspective of the users accessing it, the Alex> volume is expected to be accessible under one particular Alex> name. The obvious question is: What do I do when /vol/foo runs Alex> out of space and I need /vol/foo2? Usually, I'd play automounter Alex> tricks since the directories under the volumes are all uniquely Alex> named, but I'd rather not. This is NFS-only and the clients are Alex> all sorts of different UNIXish systems.
Softlinks? Can you change the application and how it creates the base directory? What happens if the directory already exists when it runs? If nothing bad, then softlinks should do the trick, otherwise you'll have to migrate data.
It all depends on how fast it's filling up and how much control you have over the apps/clients.
John John Stoffel - Senior Staff Systems Administrator - System LSI Group Toshiba America Electronic Components, Inc. - http://www.toshiba.com/taec john.stoffel@taec.toshiba.com - 508-486-1087
John,
Thanks for the reply! Comments inlined.
--On Friday, May 20, 2005 9:26 AM -0400 John Stoffel john.stoffel@taec.toshiba.com wrote:
Alex> Nightly, an application creates a fairly large number of files, Alex> makes a new directory on an R200, and then dumps the data to Alex> it.
I assume it actually works the other way? Creates a directory on the R200 and dumps the files into there? Something like /vol/foo/nightly.<date>/... ?
It actually first creates the data locally on a server (multiple servers create their own data in fact) and only after finishing copies the data over. It's probably fairly irrelevant though. The only reason it's created locally first is because it's faster and the added redundancy makes us more comfortable.
Or you can grow the existing volume... do you need to keep all the older data you've written online?
Yes, I need to keep all the old data.
Softlinks? Can you change the application and how it creates the base directory? What happens if the directory already exists when it runs? If nothing bad, then softlinks should do the trick, otherwise you'll have to migrate data.
Yes, I can change the way the app works since the part of the process that copies the data is a separate script. Are you suggesting to make /vol/foo essentially a collection of links to foo1, foo2, etc.? That's doable, I guess.
It all depends on how fast it's filling up and how much control you have over the apps/clients.
Right now, it's about 250GB per week, but it's slowly accelerating. As I mentioned above, the app that copies the data over is just a script that I can modify. The clients don't really care either as long as they can get to the data through /vol/foo.
Thanks!
Alex.
-
Alex Bochannek -
John Stoffel