I'd put money on the client-side options in /etc/system (or ndd equivs in scripts) not being raised high enough. We've discussed these Solaris tunables to death before now. A good document on them can be found here:
http://www.rvs.uni-hannover.de/people/voeckler/tune/EN/tune.html
I also have a question stemming from this discussion; first, some background:
Oracle, and other db's I presume, keep a lot of data in memory. They arrange to have parts of tables in memory according to their use patterns, if at all possible. Solaris, as with most modern Unixen, has a filesystem buffer-cache that keeps small writes (and probably also large writes) and reads in memory that's not part of the user-process space, and isn't AFAIK shared with the user-process space.
It's possible in Sol2.6-7 to use priority-paging to alleviate the buffer- cache competition for process memory. Solaris 8 has a different memory model. Either way the problem doesn't go away, it just bites less hard and fast.
Now (I'm getting there!) VxFS, UFS (since Sol2.6), and raw-disk devices (or Disksuite metadevices or VxVM volumes) all allow for DirectIO or raw IO. The goal being *not* to double-cache database files.
The DBMS is better at cacheing database data and more system RAM can be used constructively by the DBMS if there's less buffer-cache competing. It also saves lots of double-copies between the buffer-cache and user process space. As an aside VxFS allows 'discovered' DirectIO which means Oracle's small writes can still cause a problem.
What I want to know, and what I have yet to have clarified for me, is:
1) are NFS files buffer-cached? (I believe they are) and,
2) is there anything that can be done to moderate the buffer-cache competition (other than priority-paging), and/or alleviate the double-copying that this implies.
Also, given that on Solaris I'm informed that nothing short of an obscure fcntl() call can ensure data is flushed from a file back to the server, just how do databases maintain their ACID transaction semantics? Do you need log sections on local disk regardless?
I feel I should know this stuff, being an old hand now on the list, but it occurred to me that I still don't have all the answers. Sorry if this was a little incoherent, not entirely sober and alert right now, I might have missed something obvious.