Roger,
You are correct, your filer is getting hammered. Look at the disk IO first. Normally, your filer will flush writes every 10 seconds (unless NVRAM fills up). You're writing all the time so you know people are trying to write more than 16MB in a 10-second period (Ontap splits the 32MB NVRAM into two parts; when one half fills up, it makes the other half available to users while it flushes the first half to disk).
Cache age of 1 means your read cache is being refreshed with new data every minute. Ideally, this number would be 60 meaning everything is in your read cache and your not reading from disk that often.
Try going into rc_toggle_basic mode and running the statit command. Run statit -b to begin collection. Wait 30 seconds or more and then run statit -e. This will give you details on what the CPU is doing. It also mentions CP or checkpoint which is the operation of flushing NVRAM to disk.
The other command you can use to gather meaninful data is wafl_susp. Run wafl_susp -z to reset the stats. Wait 30 seconds or more and then run wafl_susp -w (you may want to do this via rsh and save the output to a file).
In the output, there is a field called "cp_from_cp". If this is called at all, your NVRAM is overflowing and denying write requests to users. cp_from_cp means you're in the middle of checkpointing the first half of NVRAM and yet the second half already filled up and needs to checkpoint as well. This is VERY bad.
Recommendations: Upgrade the head. The F8XX filers have 128MB NVRAM and come with 1GB - 3GB RAM. This will fix your problem (if you have the budget). The other thing would be to find another filer in your environment and move some of the data off the busy one.
The output from statit will tell you which physical disks are most utilized. You may be able to use this to narrow down which volume is the culprit (assuming you have many volumes).
Hope this helps.
/Brian/
CPU NFS CIFS HTTP Net kB/s Disk kB/s Tape kB/s Cache 51% 4074 1 0 2854 3059 2040 908 0 0 1 100% 4146 0 0 4803 5471 6737 10593 0 0 1 67% 3228 1 0 4511 5770 3899 3587 0 0 1 81% 1894 0 0 3431 3363 8564 10631 0 0 1 71% 2308 0 0 5999 6482 3671 2843 0 0 1 57% 2108 1 0 2645 3359 4482 7641 0 0 1 65% 2882 1 0 5364 8072 3736 0 0 0 1 99% 2466 0 0 8339 7281 5808 6465 0 0 1 96% 2264 2 0 2869 3556 10404 12876 0 0 1 84% 4252 0 0 4003 2174 4090 12652 0 0 1 70% 3063 0 0 2939 4694 4820 5616 0 0 1 67% 3098 1 0 3097 4024 2878 10606 0 0 1 66% 4958 0 0 5352 2693 2137 0 0 0 1