On 11/12/97 09:36:10 you wrote:
CIFS is new to the NetApp boxen and it seems to work pretty well, but NFS was your first baby and you have not done anything yet to help us in this world who need one item - soft quotas.
There may be some confusion as to *exactly* what you mean by soft quotas, and how they work. Someone once asked me that the next time someone mentioned this, to ask them to PLEASE explain what you expect the soft quotas to do, how they work, and how they currently work on your other (say, Sun) machines.
Bruce
There may be some confusion as to *exactly* what you mean by soft quotas, and how they work. Someone once asked me that the next time someone
mentioned
this, to ask them to PLEASE explain what you expect the soft quotas to do, how they work, and how they currently work on your other (say, Sun) machines.
My understanding of soft quotas is that, say, you give a user a soft quota of 20 MB and a hard quota of 50 MB.
If they are over their soft quota, then when they log in, and every time they try to write out a file, it succeeds, but also gives them a message saying, "You're over quota. Fix it."
If they're over their hard quota, it just doesn't succeed.
Does anyone else have a different impression?
And how are soft quotas implemented over NFS?
Amy
Amy Chused penned:
My understanding of soft quotas is that, say, you give a user a soft quota of 20 MB and a hard quota of 50 MB. If they are over their soft quota, then when they log in, and every time they try to write out a file, it succeeds, but also gives them a message saying, "You're over quota. Fix it." If they're over their hard quota, it just doesn't succeed. Does anyone else have a different impression?
That is how I understand and use it.
And how are soft quotas implemented over NFS?
There is a daemon rquotad which runs on the NFS server. At least on Suns.
Regards,
David K. Drum david@more.net
My understanding of soft quotas is that, say, you give a user a soft quota of 20 MB and a hard quota of 50 MB.
If they are over their soft quota, then when they log in, and every time they try to write out a file, it succeeds, but also gives them a message saying, "You're over quota. Fix it."
If they're over their hard quota, it just doesn't succeed.
Does anyone else have a different impression?
Yes - I do.
I have the impression that, over NFS, attempts to write to a file that put you over your soft quota give *NO MESSAGE WHATSOEVER*.
I have that impression because I've seen the NFS client code in, at least, SunOS 4.x (back when I worked at Sun and then at Auspex), and because I know that the NFS protocol has no mechanism to return a "this write succeeded but it went over your soft quota" indication; only the server knows that a write went over the soft quota, and it has no mechanism to communicate that to the client.
Do not be confused by the fact that you get a warning if you go over the soft quota on a *local* file system - what counts here is whether you get when when you go over it on an *NFS-mounted* file system, and, at best, what you get there is a message from the "quota" command, which some UNIXes run as part of the login process.
And how are soft quotas implemented over NFS?
Most of quotas are implemented in the server's file system; the only way NFS is involved is
1) there's a "this went over your *HARD* quota" error indication that can be returned by an NFS operation;
2) there's an ONC RPC protocol that a client can use to query the state of some particular user's quota on some particular server (that's what the "quota" command does in SunOS, for example).
Soft quotas are implemented by the server's file system; 1) is completely unaffected by soft quotas (except to the extent that if you're over your soft quota, when the "grace period" expires attempts to use additional resources get a "you're over your hard quota" error), and 2) is affected only in that the information you get back about a user's quota include the user's soft limits as well as their hard limits.
On Wed, Nov 12, 1997 at 01:33:48PM -0800, Guy Harris had written:
My understanding of soft quotas is that, say, you give a user a soft quota of 20 MB and a hard quota of 50 MB.
If they are over their soft quota, then when they log in, and every time they try to write out a file, it succeeds, but also gives them a message saying, "You're over quota. Fix it."
If they're over their hard quota, it just doesn't succeed.
Does anyone else have a different impression?
Yes - I do.
It is good to have opinions.
I have the impression that, over NFS, attempts to write to a file that put you over your soft quota give *NO MESSAGE WHATSOEVER*.
I have that impression because I've seen the NFS client code in, at least, SunOS 4.x (back when I worked at Sun and then at Auspex), and because I know that the NFS protocol has no mechanism to return a "this write succeeded but it went over your soft quota" indication; only the server knows that a write went over the soft quota, and it has no mechanism to communicate that to the client.
It might be time to check out something later than SunOS 4.1.1 then.
4.1.3 and above, for sure, return a warning that you have gone over your soft quota, and the write succeeds. Solaris 2.x also works as expected.
You are also given a grace period, by default it is 7 days, to bring your usage back below the soft quota, or it turns into a hard quota at the end of the period.
What I don't understand is that this kind of item has been asked about for years and nothing has been done about it, yet CIFS is implemented quickly and even ACLs are added before this long-sought-after feature.
I don't care if NFS by spec doesn't support it, most every UNIX system does.
Do not be confused by the fact that you get a warning if you go over the soft quota on a *local* file system - what counts here is whether you get when when you go over it on an *NFS-mounted* file system, and, at best, what you get there is a message from the "quota" command, which some UNIXes run as part of the login process.
Thanks, I am now an idiot because I can't tell the difference between a locally mounted filesystem and a filesystem mounted from a remote host.
You are full of yourself and what you think might be right.
Have you even tried this yourself in the last 10 years?
Soft quotas work, whether it is on a local filesystem, or over NFS, between SunOS and Solaris systems.
I am sorry you don't understand this.
4.1.3 and above, for sure, return a warning that you have gone over your soft quota, and the write succeeds. Solaris 2.x also works as expected.
I.e., you don't expect Solaris 2.x to print a warning?
diddy$ uname -sr SunOS 5.6 diddy$ df . Filesystem kbytes used avail capacity Mounted on /dev/dsk/c0t0d0s1 245911 4634 241032 2% /var diddy$ mount | egrep /var /var on /dev/dsk/c0t0d0s1 read/write/setuid/quota/largefiles on Wed Nov 12 15:55:16 1997 diddy$ quota -v | head -3 Disk quotas for guy (uid 75): Filesystem usage quota limit timeleft files quota limit timeleft /var 0 16 32 0 4 8
diddy$ dd if=/dev/zero of=file1 bs=1024 count=16 16+0 records in 16+0 records out diddy$ quota -v | head -3 Disk quotas for guy (uid 75): Filesystem usage quota limit timeleft files quota limit timeleft /var 16 16 32 7.0 days 1 4 8
No warning message printed when I went over the soft quota - even on a *local* file system.
I know 4.x prints one for local file systems - it got that from BSD. However, 4.1.4 doesn't print one for NFS-mounted file systems:
nova% uname -sr SunOS 4.1.3 nova% df . Filesystem kbytes used avail capacity Mounted on diddy:/var/tmp 245911 4635 241031 2% /mnt nova% dd if=/dev/zero of=file1 bs=1024 count=16
and "quota -v" on "diddy" reports:
diddy$ quota -v Disk quotas for guy (uid 75): Filesystem usage quota limit timeleft files quota limit timeleft /var 16 16 32 7.0 days 1 4 8
so I *did* go over the soft quota - I just didn't get told about it until I ran "quota" on "diddy".
Yes, we can do soft quotas.
We can't, however, do them the way they work on a *locally-mounted file system* - just as is the case with other NFS servers, the only indication you will get when you go over a soft quota is the indication you get when you run "quota".
If that's acceptable to everyone, we can put that on the schedule; there's no technical problem with providing soft quotas compatible with those you get from a SunOS 4.x or 5.x *NFS server* when a *client* goes over the soft quota.
If it's *not* acceptable, we can see what kind of notification we can provide. As per the above, we can't provide the same kind of notification you get on locally-mounted file systems, because the SunOS 4.x and 5.x kernels don't provide that (heck, the 5.6 kernel, at least, doesn't even provide it for *local* file systems); however, we could have it send mail, or something such as that.
No warning message printed when I went over the soft quota - even on a *local* file system.
Note that "warning message" here doesn't count any message put in "/var/adm/messages" - we can log a message in our "/etc/messages", too, but it doesn't inform the person who went over the quota that they just went over the quota, unless they're in the habit of monitoring the messages file on the NFS server in question.
On Wed, 12 Nov 1997, Guy Harris wrote:
If that's acceptable to everyone, we can put that on the schedule; there's no technical problem with providing soft quotas compatible with those you get from a SunOS 4.x or 5.x *NFS server* when a *client* goes over the soft quota.
I think we need to at least get some of the basic server-side requirements first:
1. Two limits for each quota type (including qtrees) 2. A grace period after which a soft quota becomes a hard quota 3. Mechanism to report quota information to a user-level process on an NFS client (e.g., rpc.rquotad)
Notifying the user in real time, as writes are performed to exceed a soft quota, is probably not worth pursuing at this time. Client-side processes can be developed to handle the reporting and enforcing of local policy. The filer shouldn't go out of its way to actively warn the user... if your users don't read their .login warning messages or their warning e-mail, that's their problem. Having the server somehow outputting a human-readable message to the associated tty seems rather messy and icky.
The filer will have to track each individual quota setting, incrementing and resetting grace timers as appropriate. When a grace period expires, the soft quota becomes a hard quota, and the NFS client will get EDQUOT's on writes. That's the only real-time notification I'm looking for.
Soft quotas on a quota tree could have some interesting effects. Doing a 'df' on an NFS filesystem would show the hard quota as the total fs size, while the "Used" and "Available" numbers would add up to the soft quota size. The difference between the soft and hard quotas can then be treated like the minfree threshold of a UFS filesystem, except you don't have to be root to exceed the soft quota.
Brian Tao penned:
I think we need to at least get some of the basic server-side requirements first:
I concur with everything you have detailed here.
Regards,
David K. Drum david@more.net
On Wed, Nov 12, 1997 at 11:47:42AM -0600, sirbruce@ix.netcom.com had written:
On 11/12/97 09:36:10 you wrote:
CIFS is new to the NetApp boxen and it seems to work pretty well, but NFS was your first baby and you have not done anything yet to help us in this world who need one item - soft quotas.
There may be some confusion as to *exactly* what you mean by soft quotas, and how they work. Someone once asked me that the next time someone mentioned this, to ask them to PLEASE explain what you expect the soft quotas to do, how they work, and how they currently work on your other (say, Sun) machines.
I have answered this question many times, and I am sick of it.
Go onto your own SunOS or Solaris system and try it yourself, soft quotas have been around for a very long time and so far, I have not heard a single technical reason why it should not work.
FreeBSD does it, shoddy Linux does it, hell, I can't name a unix system off the top of my head that doesn't.
Is it so much to ask?
I have been asking this question since late 1994 of my local sales rep, and I have talked with NetApp engineers.
The effects of soft quotas are relatively easy to implement. We've got a system with many thousands of 'soft' quotas running off of netapps servers. Remember that with the 'quota report' command, you can find the level of usage in real time for any quota ID. You can put a wrapper around this to check a quota very rapidly -- not rapidly enough to plug directly into realtime applications, but fast enough for use in, say, a .login message.
Since you need to be root on a hosts.equiv machine to do an rsh quota report on the netapps, you'll probably need a setuid root binary wrapper to actually do the rsh, but that's fairly simple to accomplish. Then set your physical quotas in the /etc/quotas file to be your _hard_ limit, and establish another file (which the netapps doesn't see) containing soft limits to compare to the real values.
We do this all through our database engines for thousands of accounts, poll the server on a very frequent basis, and everything works extremely well. There may be a couple of bugs where the numbers returned are slightly off (and in at least one case, negative), but these have been very rare and quite transitory, so I haven't pinned down whether there's really a bug or just a pookah.
--David Schairer Director, Core Technology Concentric Network Corp.
On Wed, Nov 12, 1997 at 11:47:42AM -0600, sirbruce@ix.netcom.com had written:
On 11/12/97 09:36:10 you wrote:
CIFS is new to the NetApp boxen and it seems to work pretty well, but NFS was your first baby and you have not done anything yet to help us in this world who need one item - soft quotas.
There may be some confusion as to *exactly* what you mean by soft quotas, and how they work. Someone once asked me that the next time someone mentioned this, to ask them to PLEASE explain what you expect the soft quotas to do, how they work, and how they currently work on your other (say, Sun) machines.
I have answered this question many times, and I am sick of it.
Go onto your own SunOS or Solaris system and try it yourself, soft quotas have been around for a very long time and so far, I have not heard a single technical reason why it should not work.
FreeBSD does it, shoddy Linux does it, hell, I can't name a unix system off the top of my head that doesn't.
Is it so much to ask?
I have been asking this question since late 1994 of my local sales rep, and I have talked with NetApp engineers.
-- Mike Horwath IRC: Drechsau LIFE: Lover drechsau@yuck.net Visi: info@visi.com drechsau@Geeks.ORG Twin Cities area Internet Access: 612-288-0880 for more info The founding member of Minn. Coalition for Internet Accessibility
David Schairer penned:
The effects of soft quotas are relatively easy to implement.
Assuming one has hard quotas available.
We've got a system with many thousands of 'soft' quotas running off of netapps servers. Remember that with the 'quota report' command, you can find the level of usage in real time for any quota ID. You can put a wrapper around this to check a quota very rapidly -- not rapidly enough to plug directly into realtime applications, but fast enough for use in, say, a .login message.
However, if one's site's policy is to allow users as much disk space as they need temporarily, in which case hard quotas are not set, this does not work.
Regards,
David K. Drum david@more.net
Actually, it works great for us, but I control all the applications through other OS-level implants, so I can throttle apps at a hard limit as appropriate.
If you have no hard quotas, just set the hard quotas in /etc/quotas to some amount that will never realistically be reached, such as the size of the filesystem :)
--DRS
David Schairer penned:
The effects of soft quotas are relatively easy to implement.
Assuming one has hard quotas available.
We've got a system with many thousands of 'soft' quotas running off of netapps servers. Remember that with the 'quota report' command, you can find the level of usage in real time for any quota ID. You can put a wrapper around this to check a quota very rapidly -- not rapidly enough to plug directly into realtime applications, but fast enough for use in, say, a .login message.
However, if one's site's policy is to allow users as much disk space as they need temporarily, in which case hard quotas are not set, this does not work.
Regards,
David K. Drum david@more.net -- It's hard to be bored when you're as stupid as a line. [1] Reality has a tendency to be so uncomfortably real. [2] You can only measure the size of your head from the inside. [3] When you proceed deliberately, mistakes don't cascade, they instruct. [4] [1] Vernor Vinge [2] Neil Peart [3] Larry Wall [4] Stewart Brand
David Schairer penned:
If you have no hard quotas, just set the hard quotas in /etc/quotas to some amount that will never realistically be reached, such as the size of the filesystem :)
Yeah, an oft-suggested hack. I'd just as soon have the real thing.
Regards,
David K. Drum david@more.net
On Wed, Nov 12, 1997 at 05:43:04PM -0600, David Drum had written:
David Schairer penned:
We've got a system with many thousands of 'soft' quotas running off of netapps servers. Remember that with the 'quota report' command, you can find the level of usage in real time for any quota ID. You can put a wrapper around this to check a quota very rapidly -- not rapidly enough to plug directly into realtime applications, but fast enough for use in, say, a .login message.
However, if one's site's policy is to allow users as much disk space as they need temporarily, in which case hard quotas are not set, this does not work.
Right, which is the one reason we want it here.
However, if one's site's policy is to allow users as much disk space as they need temporarily, in which case hard quotas are not set, this does not work.
Right, which is the one reason we want it here.
Yeah, I have to agree with that too.
We use quotas primarily to keep track of disk space so developers don't go wild and it helps us track where our usage is going. However, we invariably have someone who exceeds his quota smack in the middle of a deadline build and then I get paged late at night to increase quotas.
- mz
-- matthew zeier -- mrz@3com.com -- 3Com EWD Engineering -- 408/764-8420 ...................................................................... "Y el mundo se mueve, mas rapido y mejor." - Fey
On Wed, Nov 12, 1997 at 06:21:49PM -0500, David Schairer wrote: -> -> The effects of soft quotas are relatively easy to implement. We've got -> a system with many thousands of 'soft' quotas running off of netapps -> servers. Remember that with the 'quota report' command, you can -> find the level of usage in real time for any quota ID. You can put -> a wrapper around this to check a quota very rapidly -- not rapidly -> enough to plug directly into realtime applications, but fast enough -> for use in, say, a .login message. -> -> Since you need to be root on a hosts.equiv machine to do an rsh quota report -> on the netapps, you'll probably need a setuid root binary wrapper to -> actually do the rsh, but that's fairly simple to accomplish. Then -> set your physical quotas in the /etc/quotas file to be your _hard_ limit, -> and establish another file (which the netapps doesn't see) containing -> soft limits to compare to the real values. -> -> We do this all through our database engines for thousands of accounts, poll -> the server on a very frequent basis, and everything works extremely well. -> There may be a couple of bugs where the numbers returned are slightly off -> (and in at least one case, negative), but these have been very rare and quite -> transitory, so I haven't pinned down whether there's really a bug or just -> a pookah.
We did something a little different...
I wrote a script which checks to see if you are within a certain percentage, such as 15%, of your hard limit. This runs each day and sends email to anyone who exceeds that limit.
Granted this doesn't work in real time....
- dlb
-
Amy Chused -
Brian Tao -
Dave Bartlett -
David Drum -
David Schairer -
guy＠netapp.com -
matthew zeier -
Mike Horwath -
sirbruce＠ix.netcom.com