It might also be helpful to know how many disks are in the raid group. 2? 10? Apologies if it was mentioned earlier. How much NVRAM?
Jason
-----Original Message----- From: Kumar, Rahul [mailto:Rahul.Kumar@sdrc.com] Sent: Wednesday, February 06, 2002 12:48 PM To: Chris Thompson; Mike Ball; Edward Hibbert Cc: toasters@mathworks.com Subject: RE: Slow write performance
Hi All, I think in this case if he could use
sysstat -x 1 he would get a output kinda like this which might help diaganosing further... filer1> sysstat -x 1 CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk in out read write read write age hit time ty util 2% 55 12 0 67 19 21 24 0 0 0 >60 99% 0% - 2% 8% 150 9 0 159 31 33 112 1309 0 0 >60 100% 15% T 15% 1% 21 4 0 25 28 30 0 0 0 0 >60 100% 0% - 0% 3% 132 0 0 132 1 1 0 0 0 0 >60 100% 0% - 0% 4% 97 71 0 168 26 30 0 0 0 0 >60 100% 0% - 0% 2% 91 0 0 91 36 21 0 0 0 0 >60 100% 0% - 0% 2% 57 20 0 77 14 17 32 0 0 0 >60 96% 0% - 4% 4% 172 10 0 182 31 33 0 0 0 0 >60 100% 0% - 0% 1% 18 4 0 22 31 33 0 0 0 0 >60 100% 0% - 0% filer1> ----------- Rahul Kumar EDS PLM Solutions - EIT (Pune-India) Rahul.Kumar@sdrc.com Work: 91-20-2930812
-----Original Message----- From: Chris Thompson [mailto:cet1@cus.cam.ac.uk] Sent: Wednesday, February 06, 2002 9:26 PM To: Mike Ball; Edward Hibbert Cc: toasters@mathworks.com Subject: Re: Slow write performance
MBall@datalink.com (Mike Ball) writes
I not sure I would be concerned about the 0's on disk writes you
are
seeing. As long as Net KB keeps coming in/out. The Netapp may be recalculating parity and holding the writes in NVRAM before it flushes
to
disk, etc..
He should be worried about there being too *few* 0's! As anyone who has done a "sysstat 1" (or just watched the pretty lights on the discs :-]) knows, ONTAP does disk writes in bursts (CPs). It will do a CP about once every 11 seconds unless NVRAM gets half-full (or some other exceptional condition happens) first.
Edward's CP's seem to be taking about 5-6 seconds each: that's definitely not normal. If he could do "sysstat -u" (presumably the reason he can't is that he is still on ONTAP 5.x), then he would see something like 50% in the "CP time" column. The cache age being stuck down at 1 minute all the time doesn't look healthy either.
It may be that the disc read traffic is what is making the CPs take so long. That's an effect I have observed during dumps when obviously there is a lot of read traffic, but the CP-expansion is not usually as extreme as in this case.
Chris Thompson University of Cambridge Computing Service, Email: cet1@ucs.cam.ac.uk New Museums Site, Cambridge CB2 3QH, Phone: +44 1223 334715 United Kingdom.
Seems like lots of writes compared to incoming data. If you have lots of tiny files being accessed constantly, you could turn off the access time update with an option on the filer. I've been told this can improve performance a little in cases where this is acceptable.
At 1:46 PM -0600 2/6/02, Knight, Jason wrote:
It might also be helpful to know how many disks are in the raid group. 2? 10? Apologies if it was mentioned earlier. How much NVRAM?
Jason
-----Original Message----- From: Kumar, Rahul [mailto:Rahul.Kumar@sdrc.com] Sent: Wednesday, February 06, 2002 12:48 PM To: Chris Thompson; Mike Ball; Edward Hibbert Cc: toasters@mathworks.com Subject: RE: Slow write performance
Hi All, I think in this case if he could use
sysstat -x 1 he would get a output kinda like this which might help diaganosing further... filer1> sysstat -x 1 CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk in out read write read write age hit time ty util 2% 55 12 0 67 19 21 24 0 0 0 >60 99% 0% - 2% 8% 150 9 0 159 31 33 112 1309 0 0 >60 100% 15% T 15% 1% 21 4 0 25 28 30 0 0 0 0 >60 100% 0% - 0% 3% 132 0 0 132 1 1 0 0 0 0 >60 100% 0% - 0% 4% 97 71 0 168 26 30 0 0 0 0 >60 100% 0% - 0% 2% 91 0 0 91 36 21 0 0 0 0 >60 100% 0% - 0% 2% 57 20 0 77 14 17 32 0 0 0 >60 96% 0% - 4% 4% 172 10 0 182 31 33 0 0 0 0 >60 100% 0% - 0% 1% 18 4 0 22 31 33 0 0 0 0 >60 100% 0% - 0% filer1>
Rahul Kumar EDS PLM Solutions - EIT (Pune-India) Rahul.Kumar@sdrc.com Work: 91-20-2930812
-----Original Message----- From: Chris Thompson [mailto:cet1@cus.cam.ac.uk] Sent: Wednesday, February 06, 2002 9:26 PM To: Mike Ball; Edward Hibbert Cc: toasters@mathworks.com Subject: Re: Slow write performance
MBall@datalink.com (Mike Ball) writes
I not sure I would be concerned about the 0's on disk writes you
are
seeing. As long as Net KB keeps coming in/out. The Netapp may be recalculating parity and holding the writes in NVRAM before it flushes
to
disk, etc..
He should be worried about there being too *few* 0's! As anyone who has done a "sysstat 1" (or just watched the pretty lights on the discs :-]) knows, ONTAP does disk writes in bursts (CPs). It will do a CP about once every 11 seconds unless NVRAM gets half-full (or some other exceptional condition happens) first.
Edward's CP's seem to be taking about 5-6 seconds each: that's definitely not normal. If he could do "sysstat -u" (presumably the reason he can't is that he is still on ONTAP 5.x), then he would see something like 50% in the "CP time" column. The cache age being stuck down at 1 minute all the time doesn't look healthy either.
It may be that the disc read traffic is what is making the CPs take so long. That's an effect I have observed during dumps when obviously there is a lot of read traffic, but the CP-expansion is not usually as extreme as in this case.
Chris Thompson University of Cambridge Computing Service, Email: cet1@ucs.cam.ac.uk New Museums Site, Cambridge CB2 3QH, Phone: +44 1223 334715 United Kingdom.
-
Jim Harm -
Knight, Jason