What I meant by "full reallocate" is to perform reallocate using the –f & -p flag on all the volumes on this aggr
see https://library.netapp.com/ecmdocs/ECMP1136573/html/GUID-E1789834-11E5-4748-...
I'm aware that running reallocate on the aggregate won't optimize the layout
Thanks for the heads up though
Alon Z
From: Jeff Mohler [mailto:speedtoys.racing@gmail.com] Sent: יום ג 22 אוקטובר 2013 20:46 To: Jordan Slingerland Cc: Alon Zeltser; toasters@teaparty.net Subject: Re: unexplained read operation
Indeed. :) It's still worth asking first.
On Tue, Oct 22, 2013 at 10:42 AM, Jordan Slingerland Jordan.Slingerland@independenthealth.com wrote:
I assume he means run a reallocation scan on all volumes within the aggregate, but NOT! on the aggregate.
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Jeff Mohler Sent: Tuesday, October 22, 2013 1:35 PM To: Alon Zeltser Cc: toasters@teaparty.net
Subject: Re: unexplained read operation
What do you mean "full reallocate".
There is a good way, and a really really -bad- way.
On Tue, Oct 22, 2013 at 1:23 AM, Alon Zeltser alonz@emet.co.il wrote:
Looks like you nailed it
After looking at statit we can see the read operation indeed come from cp reads due to partial stripes
1590.51 stripes written
1422.93 partial stripes
167.58 full stripes
disk ut% xfers ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs /aggr0/plex0/rg0:
0d.01.0 46 115.81 0.35 1.00 2500 53.57 29.94 263 61.89 23.12 232 0.00 .... . 0.00 .... .
0d.01.1 50 117.89 0.35 1.00 4000 55.65 28.89 294 61.89 23.12 246 0.00 .... . 0.00 .... .
0d.01.2 47 106.45 10.23 1.02 11383 50.80 27.06 338 45.42 5.62 1193 0.00 .... . 0.00 .... .
0d.01.3 47 108.53 12.66 1.05 12416 49.41 26.11 378 46.46 6.90 896 0.00 .... . 0.00 .... .
0d.01.4 46 108.88 12.66 1.15 11179 48.72 26.20 402 47.50 7.11 838 0.00 .... . 0.00 .... .
0d.01.5 48 109.92 12.14 1.21 10082 49.58 26.30 399 48.20 6.51 1081 0.00 .... . 0.00 .... .
0d.01.6 47 100.03 8.84 1.00 10235 49.41 26.08 377 41.78 7.76 774 0.00 .... . 0.00 .... .
0d.01.7 46 103.68 10.92 1.08 10441 48.54 26.80 357 44.21 6.75 963 0.00 .... . 0.00 .... .
0d.01.8 48 106.80 11.10 1.09 10314 49.06 26.54 390 46.64 6.61 949 0.00 .... . 0.00 .... .
0d.01.9 46 110.44 12.66 1.05 11429 46.98 26.48 370 50.80 7.31 820 0.00 .... . 0.00 .... .
0d.01.10 47 106.28 9.36 1.06 12105 49.24 25.46 373 47.68 7.53 807 0.00 .... . 0.00 .... .
0d.01.11 46 105.58 9.19 1.02 10463 49.06 26.77 360 47.33 5.90 1078 0.00 .... . 0.00 .... .
0d.01.12 42 108.18 11.44 1.09 11264 48.89 25.60 335 47.85 7.18 902 0.00 .... . 0.00 .... .
0d.01.13 43 103.85 11.10 1.06 11559 49.06 26.45 319 43.69 7.40 757 0.00 .... . 0.00 .... .
0d.01.14 42 105.93 10.92 1.08 11941 49.24 25.27 331 45.77 7.77 757 0.00 .... . 0.00 .... .
0d.01.15 45 117.02 12.14 1.04 12877 48.89 24.43 372 56.00 7.70 802 0.00 .... . 0.00 .... .
0d.01.16 44 102.64 10.40 1.03 11694 49.93 25.18 351 42.30 8.15 684 0.00 .... . 0.00 .... .
0d.01.17 44 104.72 11.10 1.13 10750 48.72 26.62 342 44.90 7.14 830 0.00 .... . 0.00 .... .
0d.01.18 43 104.89 10.06 1.03 12350 49.41 26.71 319 45.42 6.37 987 0.00 .... . 0.00 .... .
0d.01.19 44 109.57 10.23 1.05 10629 48.37 26.15 334 50.97 7.29 810 0.00 .... . 0.00 .... .
0d.01.20 47 111.13 12.31 1.10 11372 49.06 25.81 378 49.76 6.92 912 0.00 .... . 0.00 .... .
0d.01.21 44 110.26 11.79 1.06 13569 49.76 25.79 371 48.72 6.82 828 0.00 .... . 0.00 .... .
0d.01.22 44 106.97 11.62 1.06 10042 49.58 26.88 331 45.77 6.07 1030 0.00 .... . 0.00 .... .
Even though the aggregate is at 80% utilization it seems we have fragmentation issue
I'll try to do full reallocate and check again
Thanks for the Help
Alon Z
From: Sebastian Goetze [mailto:spgoetze@gmail.com] Sent: יום ב 21 אוקטובר 2013 19:59 To: Alon Zeltser; toasters@teaparty.net Subject: Re: unexplained read operation
I'm with Jeff... Looks like CP-Reads. (Need to read from disk to be able to calculate parity. Happens when unable to write 'full stripes', e.g. full aggregate)
Do statit and check your disk statistics:
uread = User Reads, writes = User writes, cpread = CP Read as described above, gread+gwrite = guaranteed writes, like for disk rebuild
Tell us what you find :-)
Sebastian
On 10/21/2013 4:28 PM, Alon Zeltser wrote:
Hello toasters
I encounter rather strange situation
I have a system FAS2040 with 1 aggr of 23x450g raid group ontap version 8.1.3p2
Every write operation being generated on this aggregate automatically caused a read operation
When I try to create 100% sequential write operation using tools such as sio_ntap or filersio
Instead of the expected 150Mb throughput I get 100MB write and 50MB read
On another FAS2040 system using the same commands I get 100% write and no reads
As you can see
filersio asyncio_active 0 -r 0 64k 0 15g 60 1 /vol/test/testfile
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s
93% 0 0 0 5 26 1 24242 118671 0 0 0s 99% 72% Fn 43% 0 0 5 0 0 24 0
90% 0 0 0 4 4 1 40959 102501 0 0 19 100% 89% Fn 43% 1 0 3 0 0 2 0
86% 0 0 0 3 3 1 23126 125022 0 0 0s 99% 69% : 59% 0 0 3 0 0 2 0
93% 0 0 0 7 1 1 28200 135740 0 0 0s 99% 97% Ff 48% 7 0 0 0 0 0 0
92% 0 0 0 6 15 1 48764 115324 0 0 0s 99% 91% Ff 52% 0 0 6 0 0 14 0
90% 0 0 0 6 16 1 36148 156492 0 0 0s 99% 83% Ff 56% 0 0 6 0 0 14 0
95% 0 0 0 0 4 4 32784 114555 0 0 0s 99% 77% Fs 48% 0 0 0 0 0 0 0
92% 0 0 0 18 90 12 34931 104059 0 0 0s 99% 88% Fn 42% 0 0 18 0 0 79 0
88% 0 0 0 8 3 1 20463 124800 0 0 0s 100% 83% Fn 45% 5 0 3 0 0 2 0
89% 0 0 0 2 2 1 34050 112166 0 0 0s 99% 87% Fn 44% 0 0 2 0 0 1 0
92% 0 0 0 3 7 1 33025 112816 0 0 0s 99% 88% Fn 46% 0 0 3 0 0 5 0
91% 0 0 0 3 3 1 30929 130833 0 0 0s 100% 73% : 50% 0 0 3 0 0 2 0
92% 0 0 0 7 24 2 36484 134260 0 0 0s 100% 98% Ff 50% 0 0 7 0 0 22 0
86% 0 0 0 6 1 1 38608 125140 0 0 0s 99% 90% Ff 44% 5 0 1 0 0 1 0
93% 0 0 0 4 3 1 28748 104544 0 0 0s 100% 75% Ff 43% 0 0 4 0 0 2 0
87% 0 0 0 5 18 1 28128 158608 0 0 0s 100% 90% Fs 55% 0 0 5 0 0 17 0
93% 0 0 0 2 3 34 33115 104980 0 0 0s 99% 85% Fn 41% 0 0 2 0 0 1 32
84% 0 0 0 5 28 7 37185 109425 0 0 0s 100% 78% Fn 48% 0 0 5 0 0 21 0
91% 0 0 0 9 3 1 30224 118322 0 0 0s 99% 97% Fn 47% 7 0 2 0 0 1 0
89% 0 0 0 2 3 4 27649 118335 0 0 0s 99% 88% : 45% 0 0 2 0 0 1 0
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s
in out read write read write age hit time ty util in out in out
88% 0 0 0 14 57 39 43284 133480 0 0 0s 100% 98% Ff 50% 0 0 14 0 0 52 0
92% 0 0 0 2 6 16 47936 109432 0 0 0s 99% 86% Ff 46% 0 0 2 0 0 1 0
88% 0 0 0 23 86 3 45340 141014 0 0 0s 99% 73% Ff 55% 3 0 20 0 0 82 0
92% 0 0 0 10 5 4 21787 123980 0 0 0s 99% 80% Fn 44% 7 0 3 0 0 1 0
95% 0 0 0 2 4 3 34751 98158 0 0 0s 99% 82% Fn 44% 0 0 2 0 0 1 0
91% 0 0 0 1 3 1 35022 118549 0 0 0s 100% 60% Fn 48% 0 0 1 0 0 0 0
92% 0 0 0 87 3 1 25455 123066 0 0 0s 100% 80% Fn 47% 84 0 3 0 0 1 0
83% 0 0 0 8 32 2 36956 121303 0 0 0s 100% 88% : 46% 0 0 8 0 0 30 0
85% 0 0 0 5 2 1 62064 140088 0 0 0s 100% 92% Ff 59% 5 0 0 0 0 0 0
95% 0 0 0 3 3 1 39352 117364 0 0 0s 99% 71% Ff 48% 0 0 3 0 0 2 0
94% 0 0 0 3 3 1 42008 136943 0 0 0s 99% 94% Fs 53% 0 0 3 0 0 2 0
92% 0 0 0 0 2 7 26147 111286 0 0 0s 99% 58% Fn 44% 0 0 0 0 0 0 0
90% 0 0 0 3 3 1 29567 110665 0 0 20 99% 87% Fn 44% 0 0 3 0 0 1 0
90% 0 0 0 9 4 1 40031 111273 0 0 16 99% 86% Fn 43% 5 0 4 0 0 2 0
84% 0 0 0 0 1 1 36760 105709 0 0 0s 100% 84% Fn 43% 0 0 0 0 0 0 0
92% 0 0 0 3 3 1 31033 137891 0 0 0s 100% 99% : 53% 0 0 3 0 0 2 0
83% 0 0 0 4 6 1 32991 140148 0 0 0s 100% 89% F 51% 0 0 4 0 0 5 0
Any thoughts why this could happens ?
Nothing else is running on this netapp except for this test
Thanks
Alon Z
_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters