RE: unexplained read operation

I assume he means run a reallocation scan on all volumes within the aggregate, but NOT! on the aggregate.

From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Jeff Mohler Sent: Tuesday, October 22, 2013 1:35 PM To: Alon Zeltser Cc: toasters@teaparty.net Subject: Re: unexplained read operation

What do you mean "full reallocate".

There is a good way, and a really really -bad- way.

On Tue, Oct 22, 2013 at 1:23 AM, Alon Zeltser <alonz@emet.co.ilmailto:alonz@emet.co.il> wrote: Looks like you nailed it After looking at statit we can see the read operation indeed come from cp reads due to partial stripes

1590.51 stripes written 1422.93 partial stripes 167.58 full stripes

disk ut% xfers ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs /aggr0/plex0/rg0: 0d.01.0 46 115.81 0.35 1.00 2500 53.57 29.94 263 61.89 23.12 232 0.00 .... . 0.00 .... . 0d.01.1 50 117.89 0.35 1.00 4000 55.65 28.89 294 61.89 23.12 246 0.00 .... . 0.00 .... . 0d.01.2 47 106.45 10.23 1.02 11383 50.80 27.06 338 45.42 5.62 1193 0.00 .... . 0.00 .... . 0d.01.3 47 108.53 12.66 1.05 12416 49.41 26.11 378 46.46 6.90 896 0.00 .... . 0.00 .... . 0d.01.4 46 108.88 12.66 1.15 11179 48.72 26.20 402 47.50 7.11 838 0.00 .... . 0.00 .... . 0d.01.5 48 109.92 12.14 1.21 10082 49.58 26.30 399 48.20 6.51 1081 0.00 .... . 0.00 .... . 0d.01.6 47 100.03 8.84 1.00 10235 49.41 26.08 377 41.78 7.76 774 0.00 .... . 0.00 .... . 0d.01.7 46 103.68 10.92 1.08 10441 48.54 26.80 357 44.21 6.75 963 0.00 .... . 0.00 .... . 0d.01.8 48 106.80 11.10 1.09 10314 49.06 26.54 390 46.64 6.61 949 0.00 .... . 0.00 .... . 0d.01.9 46 110.44 12.66 1.05 11429 46.98 26.48 370 50.80 7.31 820 0.00 .... . 0.00 .... . 0d.01.10 47 106.28 9.36 1.06 12105 49.24 25.46 373 47.68 7.53 807 0.00 .... . 0.00 .... . 0d.01.11 46 105.58 9.19 1.02 10463 49.06 26.77 360 47.33 5.90 1078 0.00 .... . 0.00 .... . 0d.01.12 42 108.18 11.44 1.09 11264 48.89 25.60 335 47.85 7.18 902 0.00 .... . 0.00 .... . 0d.01.13 43 103.85 11.10 1.06 11559 49.06 26.45 319 43.69 7.40 757 0.00 .... . 0.00 .... . 0d.01.14 42 105.93 10.92 1.08 11941 49.24 25.27 331 45.77 7.77 757 0.00 .... . 0.00 .... . 0d.01.15 45 117.02 12.14 1.04 12877 48.89 24.43 372 56.00 7.70 802 0.00 .... . 0.00 .... . 0d.01.16 44 102.64 10.40 1.03 11694 49.93 25.18 351 42.30 8.15 684 0.00 .... . 0.00 .... . 0d.01.17 44 104.72 11.10 1.13 10750 48.72 26.62 342 44.90 7.14 830 0.00 .... . 0.00 .... . 0d.01.18 43 104.89 10.06 1.03 12350 49.41 26.71 319 45.42 6.37 987 0.00 .... . 0.00 .... . 0d.01.19 44 109.57 10.23 1.05 10629 48.37 26.15 334 50.97 7.29 810 0.00 .... . 0.00 .... . 0d.01.20 47 111.13 12.31 1.10 11372 49.06 25.81 378 49.76 6.92 912 0.00 .... . 0.00 .... . 0d.01.21 44 110.26 11.79 1.06 13569 49.76 25.79 371 48.72 6.82 828 0.00 .... . 0.00 .... . 0d.01.22 44 106.97 11.62 1.06 10042 49.58 26.88 331 45.77 6.07 1030 0.00 .... . 0.00 .... .

Even though the aggregate is at 80% utilization it seems we have fragmentation issue I'll try to do full reallocate and check again Thanks for the Help Alon Z

From: Sebastian Goetze [mailto:spgoetze@gmail.commailto:spgoetze@gmail.com] Sent: יום ב 21 אוקטובר 2013 19:59 To: Alon Zeltser; toasters@teaparty.netmailto:toasters@teaparty.net Subject: Re: unexplained read operation

I'm with Jeff... Looks like CP-Reads. (Need to read from disk to be able to calculate parity. Happens when unable to write 'full stripes', e.g. full aggregate)

Do statit and check your disk statistics: uread = User Reads, writes = User writes, cpread = CP Read as described above, gread+gwrite = guaranteed writes, like for disk rebuild Tell us what you find :-)

Sebastian On 10/21/2013 4:28 PM, Alon Zeltser wrote: Hello toasters I encounter rather strange situation I have a system FAS2040 with 1 aggr of 23x450g raid group ontap version 8.1.3p2 Every write operation being generated on this aggregate automatically caused a read operation When I try to create 100% sequential write operation using tools such as sio_ntap or filersio Instead of the expected 150Mb throughput I get 100MB write and 50MB read On another FAS2040 system using the same commands I get 100% write and no reads As you can see

filersio asyncio_active 0 -r 0 64k 0 15g 60 1 /vol/test/testfile

CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s 93% 0 0 0 5 26 1 24242 118671 0 0 0s 99% 72% Fn 43% 0 0 5 0 0 24 0 90% 0 0 0 4 4 1 40959 102501 0 0 19 100% 89% Fn 43% 1 0 3 0 0 2 0 86% 0 0 0 3 3 1 23126 125022 0 0 0s 99% 69% : 59% 0 0 3 0 0 2 0 93% 0 0 0 7 1 1 28200 135740 0 0 0s 99% 97% Ff 48% 7 0 0 0 0 0 0 92% 0 0 0 6 15 1 48764 115324 0 0 0s 99% 91% Ff 52% 0 0 6 0 0 14 0 90% 0 0 0 6 16 1 36148 156492 0 0 0s 99% 83% Ff 56% 0 0 6 0 0 14 0 95% 0 0 0 0 4 4 32784 114555 0 0 0s 99% 77% Fs 48% 0 0 0 0 0 0 0 92% 0 0 0 18 90 12 34931 104059 0 0 0s 99% 88% Fn 42% 0 0 18 0 0 79 0 88% 0 0 0 8 3 1 20463 124800 0 0 0s 100% 83% Fn 45% 5 0 3 0 0 2 0 89% 0 0 0 2 2 1 34050 112166 0 0 0s 99% 87% Fn 44% 0 0 2 0 0 1 0 92% 0 0 0 3 7 1 33025 112816 0 0 0s 99% 88% Fn 46% 0 0 3 0 0 5 0 91% 0 0 0 3 3 1 30929 130833 0 0 0s 100% 73% : 50% 0 0 3 0 0 2 0 92% 0 0 0 7 24 2 36484 134260 0 0 0s 100% 98% Ff 50% 0 0 7 0 0 22 0 86% 0 0 0 6 1 1 38608 125140 0 0 0s 99% 90% Ff 44% 5 0 1 0 0 1 0 93% 0 0 0 4 3 1 28748 104544 0 0 0s 100% 75% Ff 43% 0 0 4 0 0 2 0 87% 0 0 0 5 18 1 28128 158608 0 0 0s 100% 90% Fs 55% 0 0 5 0 0 17 0 93% 0 0 0 2 3 34 33115 104980 0 0 0s 99% 85% Fn 41% 0 0 2 0 0 1 32 84% 0 0 0 5 28 7 37185 109425 0 0 0s 100% 78% Fn 48% 0 0 5 0 0 21 0 91% 0 0 0 9 3 1 30224 118322 0 0 0s 99% 97% Fn 47% 7 0 2 0 0 1 0 89% 0 0 0 2 3 4 27649 118335 0 0 0s 99% 88% : 45% 0 0 2 0 0 1 0 CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out 88% 0 0 0 14 57 39 43284 13348tel:43284%20133480 0 0 0s 100% 98% Ff 50% 0 0 14 0 0 52 0 92% 0 0 0 2 6 16 47936 109432 0 0 0s 99% 86% Ff 46% 0 0 2 0 0 1 0 88% 0 0 0 23 86 3 45340 141014 0 0 0s 99% 73% Ff 55% 3 0 20 0 0 82 0 92% 0 0 0 10 5 4 21787 123980 0 0 0s 99% 80% Fn 44% 7 0 3 0 0 1 0 95% 0 0 0 2 4 3 34751 98158tel:34751%C2%A0%2098158 0 0 0s 99% 82% Fn 44% 0 0 2 0 0 1 0 91% 0 0 0 1 3 1 35022 118549 0 0 0s 100% 60% Fn 48% 0 0 1 0 0 0 0 92% 0 0 0 87 3 1 25455 123066 0 0 0s 100% 80% Fn 47% 84 0 3 0 0 1 0 83% 0 0 0 8 32 2 36956 121303 0 0 0s 100% 88% : 46% 0 0 8 0 0 30 0 85% 0 0 0 5 2 1 62064 140088 0 0 0s 100% 92% Ff 59% 5 0 0 0 0 0 0 95% 0 0 0 3 3 1 39352 117364 0 0 0s 99% 71% Ff 48% 0 0 3 0 0 2 0 94% 0 0 0 3 3 1 42008 136943 0 0 0s 99% 94% Fs 53% 0 0 3 0 0 2 0 92% 0 0 0 0 2 7 26147 111286 0 0 0s 99% 58% Fn 44% 0 0 0 0 0 0 0 90% 0 0 0 3 3 1 29567 110665 0 0 20 99% 87% Fn 44% 0 0 3 0 0 1 0 90% 0 0 0 9 4 1 40031 111273 0 0 16 99% 86% Fn 43% 5 0 4 0 0 2 0 84% 0 0 0 0 1 1 36760 105709 0 0 0s 100% 84% Fn 43% 0 0 0 0 0 0 0 92% 0 0 0 3 3 1 31033 137891 0 0 0s 100% 99% : 53% 0 0 3 0 0 2 0 83% 0 0 0 4 6 1 32991 140148 0 0 0s 100% 89% F 51% 0 0 4 0 0 5 0

Any thoughts why this could happens ? Nothing else is running on this netapp except for this test

Thanks Alon Z

_______________________________________________

Toasters mailing list

Toasters@teaparty.netmailto:Toasters@teaparty.net

http://www.teaparty.net/mailman/listinfo/toasters

_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

-- --- Gustatus Similis Pullus