Has it made any difference?
Yes, now the cpu usage is nearly 50% most of the time, with peaks at 60% opposed to the 100% that it used to. But this 90% occupation rule really lowers the filer disk space rentability.
I understand your point and concern. It is one of those situations where you have "good, fast and cheap, pick 2". The 90% is not a hard-fast rule. The nutshell is that as the filesystem fills, there are fewer open spots to write and thus there is potentially more work to do to find a spot and write.
I have not charted the performance hit you take as the filer fills. Might be an interesting graph :)
Ideally you should not be operating at 90% if you can avoid it, on any system. I tend to go with 85% as a comfort level.
Ok, but this a decision at volume creation, we couldn't reconfigure the raidsize at this stage without data migration, am I right?
Correct. And you probably don't want to backup 400GB, destroy the volumes and then restore. That would be a bit much!
To your cost question: you have different knobs to tweak on the filer: raid group sizes, number of spares, disk sizes, etc. They all have some impact on the performace but you need to make the technical and business tradeoffs: do you need more space than you do performance? Do you want to do as much as possible to protect your data vs optimize your disks (smaller vs larger raid groups), etc.
alexei