we already encounter a similar prb :
over CIFS in a large hierarchical subdirectorie structure with over 100.000 files in a directory with an application running some kind of find on these files (ie : looking for a specific string inside files)
the filer was blocking the creation of files over CIFS, there is a limitation concerning max number of files in a directory (around 130.00) you can adjust this but the client also modify his application to create less files per dir
also the costumer explain that the processing was longer as there were more files in dir
i think the most impact on your issue was the readattrbute. i make the parrallel between your readattrb and the find function of the above application this is CPU consuming as the half of the job has to be done on the filer side as there is lot of files (near 100.000), you saw the cpu get hight and experience longer response
F8XX are around 2 times much powerfull (CPU and mem concern) than the F760 so i think upgrade your Filer could perhaps let you resolve the prb definitely (despite your cpu is not just near the 100% limit but between 80 and 90%) upgrading Ontapp to a newer version perhaps could help, but I can't tell you 613r2 is a stable version in my opinion, but perhaps not optimized for your kind of job
bye
Chris Miles wrote:
Hi, we have over 20 webservers running a web application and serving all the data off a Netapp NFS mount. Netapp CPU normally sits at around 20-40% and NFS response is good.
Today we hit what appears to have been a bug of some sort. All of a sudden, with no apparent increase in client connections, the F760 CPU went to 80-90% and the load on all the webservers rose sharply (from approx 0.5-1.5 up to 15-20). The site response went down the drain (20+ seconds for a page that normally takes under 1 second).
It appeared to be caused by the application doing readdirs (along with other operations: read/write/getattr) on a specific directory, which at that time held about 70,000 files. We fixed the problem by disabling the readdirs within the application and also reducing the number of files in that directory down to about 45,000.
We don't know exactly which fix (stopping readdirs or removing the files) did the trick, but after that Netapp CPU dropped back to normal and the webservers were happy and the site responsive again.
It appeared that the combination of all the operations on the large directory were causing the NFS clients to hang and the Netapp CPU to max out. Although at the same time, other NFS operations were still performing at a reasonable speed (ie, the whole Netapp was not locked up).
Any ideas on a bug or limitation on either the Linux or the Netapp side with regards to large (70,000+ files) directories?
Info: Netapp F760C ONTAP 6.1.2R3
Linux 2.4.20 (Redhat) Mount options: rw,hard,intr,vers=3,proto=tcp,rsize=32768,wsize=32768
Cheers, Chris