Hi Toasters,
we have a database on nfs space and i wonder about the output of nfs_hist.
v3 read: 35280016 (blocking requests) - millisecond units 0 1 2 3 4 5 6 7 17347681 2028005 655821 757932 1061542 1463375 1943351 1721450 <16 <24 <32 <40 <48 <56 <64 UNUSED 6417253 1334072 338371 111465 43542 18991 10063 0 <128 <192 <256 <320 <384 <448 <512 UNUSED 20806 2364 1282 617 526 280 266 0 <1024 <1536 <2048 <2560 <3072 <3584 <4096 UNUSED 723 158 77 3 0 0 0 0 <8192 <12288 <16384 <20480 <24576 <28672 <32768 UNUSED 0 0 0 0 0 0 0 0 <65536 <98304 <131072 <163840 <196608 <229376 <262144 >262144 0 0 0 0 0 0 0 0
Is the amount of read-calls with a RTT greater than 7ms ok, or is it something i have to wonder about?? I mean, the calls between 7-16ms are about 18%. But on the other hand nfs_hist shows only the blocking requests so the percentage of total calls will be even less than 18%.
Does my nfs_hist look like a healthy system or do i have a problem?
Regards and thanks for every answer :D
Jochen
When was the last time nfs_hist -z was run?
nfs_hist is good for spotting the one or two out of a couple billion read calls, but it has to be zeroed to make it more targeted.
Also there's stats.
stats show -i 1 -n 10 -r volume:oraclevol:nfs_read_latency
or if you are more interested in nfs
stats show nfsv3:nfs:nfsv3_nonblockop_latency nfsv3:nfs:nfsv3_blockop_latency nfsv3:nfs:nfsv3_op_latency
Its a long list, but easily parsed and popped into a rrd graph for historical monitoring, as opposed to nfs_hist.
Hope that helps, -Blake
On 3/14/07, Willeke, Jochen Jochen.Willeke@wincor-nixdorf.com wrote:
Hi Toasters,
we have a database on nfs space and i wonder about the output of nfs_hist.
v3 read: 35280016 (blocking requests) - millisecond units 0 1 2 3 4 5 6 7 17347681 2028005 655821 757932 1061542 1463375 1943351 1721450 <16 <24 <32 <40 <48 <56 <64 UNUSED 6417253 1334072 338371 111465 43542 18991 10063 0 <128 <192 <256 <320 <384 <448 <512 UNUSED 20806 2364 1282 617 526 280 266 0 <1024 <1536 <2048 <2560 <3072 <3584 <4096 UNUSED 723 158 77 3 0 0 0 0 <8192 <12288 <16384 <20480 <24576 <28672 <32768 UNUSED 0 0 0 0 0 0 0 0 <65536 <98304 <131072 <163840 <196608 <229376 <262144 >262144 0 0 0 0 0 0 0 0
Is the amount of read-calls with a RTT greater than 7ms ok, or is it something i have to wonder about?? I mean, the calls between 7-16ms are about 18%. But on the other hand nfs_hist shows only the blocking requests so the percentage of total calls will be even less than 18%.
Does my nfs_hist look like a healthy system or do i have a problem?
Regards and thanks for every answer :D
Jochen
Hi Blake,
the nfs_hist -z was run only about 2 hours before.
I know the stats show but it shows the average about a second and this is always ok and shows about 3ms as read_latency for a volume.
Best Regards
Jochen Willeke
-----Original Message----- From: Blake Golliher [mailto:thelastman@gmail.com] Sent: Wednesday, March 14, 2007 5:49 PM To: Willeke, Jochen Cc: toasters@mathworks.com Subject: Re: avg rtt is okay but maximum is high
When was the last time nfs_hist -z was run?
nfs_hist is good for spotting the one or two out of a couple billion read calls, but it has to be zeroed to make it more targeted.
Also there's stats.
stats show -i 1 -n 10 -r volume:oraclevol:nfs_read_latency
or if you are more interested in nfs
stats show nfsv3:nfs:nfsv3_nonblockop_latency nfsv3:nfs:nfsv3_blockop_latency nfsv3:nfs:nfsv3_op_latency
Its a long list, but easily parsed and popped into a rrd graph for historical monitoring, as opposed to nfs_hist.
Hope that helps, -Blake
On 3/14/07, Willeke, Jochen Jochen.Willeke@wincor-nixdorf.com wrote:
Hi Toasters,
we have a database on nfs space and i wonder about the output of nfs_hist.
v3 read: 35280016 (blocking requests) - millisecond units 0 1 2 3 4 5 6
7
17347681 2028005 655821 757932 1061542 1463375 1943351
1721450
<16 <24 <32 <40 <48 <56 <64
UNUSED
6417253 1334072 338371 111465 43542 18991 10063
0
<128 <192 <256 <320 <384 <448 <512
UNUSED
20806 2364 1282 617 526 280 266
0
<1024 <1536 <2048 <2560 <3072 <3584 <4096
UNUSED
723 158 77 3 0 0 0
0
<8192 <12288 <16384 <20480 <24576 <28672 <32768
UNUSED
0 0 0 0 0 0 0
0
<65536 <98304 <131072 <163840 <196608 <229376 <262144 262144 0 0 0 0 0 0 0
0
Is the amount of read-calls with a RTT greater than 7ms ok, or is it something i have to wonder about?? I mean, the calls between 7-16ms are about 18%. But on the other hand nfs_hist shows only the blocking requests so the percentage of total calls will be even less than 18%.
Does my nfs_hist look like a healthy system or do i have a problem?
Regards and thanks for every answer :D
Jochen
When I used nfs_hist I used to capture it in 30 sec loops to try and get a good average. I'd try to capture that, and see if it lines up with what stats pack (or awr) is showing for block read latencys. They should match up pretty closely.
-Blake
On 3/15/07, Willeke, Jochen Jochen.Willeke@wincor-nixdorf.com wrote:
Hi Blake,
the nfs_hist -z was run only about 2 hours before.
I know the stats show but it shows the average about a second and this is always ok and shows about 3ms as read_latency for a volume.
Best Regards
Jochen Willeke
-----Original Message----- From: Blake Golliher [mailto:thelastman@gmail.com] Sent: Wednesday, March 14, 2007 5:49 PM To: Willeke, Jochen Cc: toasters@mathworks.com Subject: Re: avg rtt is okay but maximum is high
When was the last time nfs_hist -z was run?
nfs_hist is good for spotting the one or two out of a couple billion read calls, but it has to be zeroed to make it more targeted.
Also there's stats.
stats show -i 1 -n 10 -r volume:oraclevol:nfs_read_latency
or if you are more interested in nfs
stats show nfsv3:nfs:nfsv3_nonblockop_latency nfsv3:nfs:nfsv3_blockop_latency nfsv3:nfs:nfsv3_op_latency
Its a long list, but easily parsed and popped into a rrd graph for historical monitoring, as opposed to nfs_hist.
Hope that helps, -Blake
On 3/14/07, Willeke, Jochen Jochen.Willeke@wincor-nixdorf.com wrote:
Hi Toasters,
we have a database on nfs space and i wonder about the output of nfs_hist.
v3 read: 35280016 (blocking requests) - millisecond units 0 1 2 3 4 5 6
7
17347681 2028005 655821 757932 1061542 1463375 1943351
1721450
<16 <24 <32 <40 <48 <56 <64UNUSED
6417253 1334072 338371 111465 43542 18991 10063
0
<128 <192 <256 <320 <384 <448 <512UNUSED
20806 2364 1282 617 526 280 2660
<1024 <1536 <2048 <2560 <3072 <3584 <4096UNUSED
723 158 77 3 0 0 00
<8192 <12288 <16384 <20480 <24576 <28672 <32768UNUSED
0 0 0 0 0 0 00
<65536 <98304 <131072 <163840 <196608 <229376 <262144 262144 0 0 0 0 0 0 0
0
Is the amount of read-calls with a RTT greater than 7ms ok, or is it something i have to wonder about?? I mean, the calls between 7-16ms are about 18%. But on the other hand nfs_hist shows only the blocking requests so the percentage of total calls will be even less than 18%.
Does my nfs_hist look like a healthy system or do i have a problem?
Regards and thanks for every answer :D
Jochen
Hi Blake,
i followed your instructions and after some mathematics i got a RRT for blocking_read_requests of ~3,9ms which is very close to what tetherreal and stats show tell but far away from what statspack shows.
In statspack i see something like this: Avg Total Wait wait Waits Event Waits Timeouts Time (s) (ms) /txn ---------------------------- ------------ ---------- ---------- ------ -------- db file sequential read 157,810 0 1,429 9 6.2 db file scattered read 33,951 0 728 21 1.3
Perhaps this time it is not the filer but the database which has a problem....
Regards
Jochen
-----Original Message----- From: Blake Golliher [mailto:thelastman@gmail.com] Sent: Thursday, March 15, 2007 10:08 AM To: Willeke, Jochen Cc: toasters@mathworks.com Subject: Re: avg rtt is okay but maximum is high
When I used nfs_hist I used to capture it in 30 sec loops to try and get a good average. I'd try to capture that, and see if it lines up with what stats pack (or awr) is showing for block read latencys. They should match up pretty closely.
-Blake
On 3/15/07, Willeke, Jochen Jochen.Willeke@wincor-nixdorf.com wrote:
Hi Blake,
the nfs_hist -z was run only about 2 hours before.
I know the stats show but it shows the average about a second and this is always ok and shows about 3ms as read_latency for a volume.
Best Regards
Jochen Willeke
-----Original Message----- From: Blake Golliher [mailto:thelastman@gmail.com] Sent: Wednesday, March 14, 2007 5:49 PM To: Willeke, Jochen Cc: toasters@mathworks.com Subject: Re: avg rtt is okay but maximum is high
When was the last time nfs_hist -z was run?
nfs_hist is good for spotting the one or two out of a couple billion read calls, but it has to be zeroed to make it more targeted.
Also there's stats.
stats show -i 1 -n 10 -r volume:oraclevol:nfs_read_latency
or if you are more interested in nfs
stats show nfsv3:nfs:nfsv3_nonblockop_latency nfsv3:nfs:nfsv3_blockop_latency nfsv3:nfs:nfsv3_op_latency
Its a long list, but easily parsed and popped into a rrd graph for historical monitoring, as opposed to nfs_hist.
Hope that helps, -Blake
On 3/14/07, Willeke, Jochen Jochen.Willeke@wincor-nixdorf.com wrote:
Hi Toasters,
we have a database on nfs space and i wonder about the output of nfs_hist.
v3 read: 35280016 (blocking requests) - millisecond units 0 1 2 3 4 5 6
7
17347681 2028005 655821 757932 1061542 1463375 1943351
1721450
<16 <24 <32 <40 <48 <56 <64UNUSED
6417253 1334072 338371 111465 43542 18991 10063
0
<128 <192 <256 <320 <384 <448 <512UNUSED
20806 2364 1282 617 526 280 2660
<1024 <1536 <2048 <2560 <3072 <3584 <4096UNUSED
723 158 77 3 0 0 00
<8192 <12288 <16384 <20480 <24576 <28672 <32768UNUSED
0 0 0 0 0 0 00
<65536 <98304 <131072 <163840 <196608 <229376 <262144 262144 0 0 0 0 0 0 0
0
Is the amount of read-calls with a RTT greater than 7ms ok, or is it something i have to wonder about?? I mean, the calls between 7-16ms are about 18%. But on the other
hand
nfs_hist shows only the blocking requests so the percentage of total calls will be even less than 18%.
Does my nfs_hist look like a healthy system or do i have a problem?
Regards and thanks for every answer :D
Jochen
-
Blake Golliher -
Willeke, Jochen