:)
Not only wont optimize, but makes some things much worse.
Glad you had that handy.
I consider vol reallocates at least a quarterly best practice.
Some people will disagree, as it invites the idea of a 'weakness'. I call it "Duh, its storage". :)
On Wed, Oct 23, 2013 at 12:01 AM, Alon Zeltser alonz@emet.co.il wrote:
What I meant by "full reallocate" is to perform reallocate using the –f & -p flag on all the volumes on this aggr****
see https://library.netapp.com/ecmdocs/ECMP1136573/html/GUID-E1789834-11E5-4748-...
I'm aware that running reallocate on the aggregate won't optimize the layout ****
Thanks for the heads up though ****
*Alon Z* ****
*From:* Jeff Mohler [mailto:speedtoys.racing@gmail.com] *Sent:* יום ג 22 אוקטובר 2013 20:46 *To:* Jordan Slingerland *Cc:* Alon Zeltser; toasters@teaparty.net
*Subject:* Re: unexplained read operation****
Indeed. :) It's still worth asking first.****
On Tue, Oct 22, 2013 at 10:42 AM, Jordan Slingerland < Jordan.Slingerland@independenthealth.com> wrote:****
I assume he means run a reallocation scan on all volumes within the aggregate, but NOT! on the aggregate. ****
*From:* toasters-bounces@teaparty.net [mailto: toasters-bounces@teaparty.net] *On Behalf Of *Jeff Mohler *Sent:* Tuesday, October 22, 2013 1:35 PM *To:* Alon Zeltser *Cc:* toasters@teaparty.net****
*Subject:* Re: unexplained read operation****
What do you mean "full reallocate".****
There is a good way, and a really really -bad- way.****
On Tue, Oct 22, 2013 at 1:23 AM, Alon Zeltser alonz@emet.co.il wrote:***
Looks like you nailed it ****
After looking at statit we can see the read operation indeed come from cp reads due to partial stripes****
1590.51 stripes written ****
1422.93 partial stripes ****
167.58 full stripes****
disk ut% xfers ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs /aggr0/plex0/rg0:****
0d.01.0 46 115.81 0.35 1.00 2500 53.57 29.94 263 *61.89 23*.12 232 0.00 .... . 0.00 .... .****
0d.01.1 50 117.89 0.35 1.00 4000 55.65 28.89 294 *61.89 23*.12 246 0.00 .... . 0.00 .... .****
0d.01.2 47 106.45 10.23 1.02 11383 50.80 27.06 338* 45.42 * 5.62 1193 0.00 .... . 0.00 .... .****
0d.01.3 47 108.53 12.66 1.05 12416 49.41 26.11 378* 46.46 * 6.90 896 0.00 .... . 0.00 .... .****
0d.01.4 46 108.88 12.66 1.15 11179 48.72 26.20 402* 47.50 * 7.11 838 0.00 .... . 0.00 .... .****
0d.01.5 48 109.92 12.14 1.21 10082 49.58 26.30 399* 48.20 * 6.51 1081 0.00 .... . 0.00 .... .****
0d.01.6 47 100.03 8.84 1.00 10235 49.41 26.08 377* 41.78 7*.76 774 0.00 .... . 0.00 .... .****
0d.01.7 46 103.68 10.92 1.08 10441 48.54 26.80 357* 44.21
- 6.75 963 0.00 .... . 0.00 .... .****
0d.01.8 48 106.80 11.10 1.09 10314 49.06 26.54 390* 46.64
- 6.61 949 0.00 .... . 0.00 .... .****
0d.01.9 46 110.44 12.66 1.05 11429 46.98 26.48 370* 50.80
- 7.31 820 0.00 .... . 0.00 .... .****
0d.01.10 47 106.28 9.36 1.06 12105 49.24 25.46 373* 47.68
- 7.53 807 0.00 .... . 0.00 .... .****
0d.01.11 46 105.58 9.19 1.02 10463 49.06 26.77 360* 47.33
- 5.90 1078 0.00 .... . 0.00 .... .****
0d.01.12 42 108.18 11.44 1.09 11264 48.89 25.60 335* 47.85 * 7.18 902 0.00 .... . 0.00 .... .****
0d.01.13 43 103.85 11.10 1.06 11559 49.06 26.45 319* 43.69 * 7.40 757 0.00 .... . 0.00 .... .****
0d.01.14 42 105.93 10.92 1.08 11941 49.24 25.27 331* 45.77 * 7.77 757 0.00 .... . 0.00 .... .****
0d.01.15 45 117.02 12.14 1.04 12877 48.89 24.43 372* 56.00 * 7.70 802 0.00 .... . 0.00 .... .****
0d.01.16 44 102.64 10.40 1.03 11694 49.93 25.18 351* 42.30 * 8.15 684 0.00 .... . 0.00 .... .****
0d.01.17 44 104.72 11.10 1.13 10750 48.72 26.62 342* 44.90 * 7.14 830 0.00 .... . 0.00 .... .****
0d.01.18 43 104.89 10.06 1.03 12350 49.41 26.71 319* 45.42 * 6.37 987 0.00 .... . 0.00 .... .****
0d.01.19 44 109.57 10.23 1.05 10629 48.37 26.15 334* 50.97 * 7.29 810 0.00 .... . 0.00 .... .****
0d.01.20 47 111.13 12.31 1.10 11372 49.06 25.81 378* 49.76 * 6.92 912 0.00 .... . 0.00 .... .****
0d.01.21 44 110.26 11.79 1.06 13569 49.76 25.79 371* 48.72 * 6.82 828 0.00 .... . 0.00 .... .****
0d.01.22 44 106.97 11.62 1.06 10042 49.58 26.88 331* 45.77 * 6.07 1030 0.00 .... . 0.00 .... .****
Even though the aggregate is at 80% utilization it seems we have fragmentation issue****
I'll try to do full reallocate and check again ****
Thanks for the Help****
*Alon Z*****
*From:* Sebastian Goetze [mailto:spgoetze@gmail.com] *Sent:* יום ב 21 אוקטובר 2013 19:59 *To:* Alon Zeltser; toasters@teaparty.net *Subject:* Re: unexplained read operation****
I'm with Jeff... Looks like CP-Reads. (Need to read from disk to be able to calculate parity. Happens when unable to write 'full stripes', e.g. full aggregate)
Do statit and check your disk statistics:****
uread = User Reads, writes = User writes, cpread = CP Read as described above, gread+gwrite = guaranteed writes, like for disk rebuild****
Tell us what you find :-)
Sebastian****
On 10/21/2013 4:28 PM, Alon Zeltser wrote:****
Hello toasters****
I encounter rather strange situation ****
I have a system FAS2040 with 1 aggr of 23x450g raid group ontap version 8.1.3p2****
Every write operation being generated on this aggregate automatically caused a read operation ****
When I try to create 100% sequential write operation using tools such as sio_ntap or filersio ****
Instead of the expected 150Mb throughput I get 100MB write and 50MB read *
On another FAS2040 system using the same commands I get 100% write and no reads****
As you can see****
filersio asyncio_active 0 -r 0 64k 0 15g 60 1 /vol/test/testfile****
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s****
93% 0 0 0 5 26 1 *24242 118671 * 0 0 0s 99% 72% Fn 43% 0 0 5 0 0 24 0****
90% 0 0 0 4 4 1 *40959 102501 * 0 0 19 100% 89% Fn 43% 1 0 3 0 0 2 0****
86% 0 0 0 3 3 1 *23126 125022 * 0 0 0s 99% 69% : 59% 0 0 3 0 0 2 0****
93% 0 0 0 7 1 1 *28200 135740 * 0 0 0s 99% 97% Ff 48% 7 0 0 0 0 0 0****
92% 0 0 0 6 15 1 * 48764 115324 * 0 0 0s 99% 91% Ff 52% 0 0 6 0 0 14 0****
90% 0 0 0 6 16 1 * 36148 156492 * 0 0 0s 99% 83% Ff 56% 0 0 6 0 0 14 0****
95% 0 0 0 0 4 4 *32784 114555 * 0 0 0s 99% 77% Fs 48% 0 0 0 0 0 0 0****
92% 0 0 0 18 90 1*2 34931 104059 * 0 0 0s 99% 88% Fn 42% 0 0 18 0 0 79 0****
88% 0 0 0 8 3 1 *20463 124800 * 0 0 0s 100% 83% Fn 45% 5 0 3 0 0 2 0****
89% 0 0 0 2 2 1 *34050 112166 * 0 0 0s 99% 87% Fn 44% 0 0 2 0 0 1 0****
92% 0 0 0 3 7 1 *33025 112816 * 0 0 0s 99% 88% Fn 46% 0 0 3 0 0 5 0****
91% 0 0 0 3 3 1 *30929 130833 * 0 0 0s 100% 73% : 50% 0 0 3 0 0 2 0****
92% 0 0 0 7 24 2 * 36484 134260 * 0 0 0s 100% 98% Ff 50% 0 0 7 0 0 22 0****
86% 0 0 0 6 1 1 *38608 125140 * 0 0 0s 99% 90% Ff 44% 5 0 1 0 0 1 0****
93% 0 0 0 4 3 1 *28748 104544 * 0 0 0s 100% 75% Ff 43% 0 0 4 0 0 2 0****
87% 0 0 0 5 18 1 * 28128 158608 * 0 0 0s 100% 90% Fs 55% 0 0 5 0 0 17 0****
93% 0 0 0 2 3 34 * 33115 104980 * 0 0 0s 99% 85% Fn 41% 0 0 2 0 0 1 32****
84% 0 0 0 5 28 7 * 37185 109425 * 0 0 0s 100% 78% Fn 48% 0 0 5 0 0 21 0****
91% 0 0 0 9 3 1 *30224 118322 * 0 0 0s 99% 97% Fn 47% 7 0 2 0 0 1 0****
89% 0 0 0 2 3 4 *27649 118335 * 0 0 0s 99% 88% : 45% 0 0 2 0 0 1 0****
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s****
in out read write readwrite age hit time ty util in out in out****
88% 0 0 0 14 57 39 * 43284 13348*0 0 0 0s 100% 98% Ff 50% 0 0 14 0 0 52 0****
92% 0 0 0 2 6 16 *47936 109432* 0 0 0s 99% 86% Ff 46% 0 0 2 0 0 1 0****
88% 0 0 0 23 86 3 * 45340 141014* 0 0 0s 99% 73% Ff 55% 3 0 20 0 0 82 0****
92% 0 0 0 10 5 4 *21787 123980* 0 0 0s 99% 80% Fn 44% 7 0 3 0 0 1 0****
95% 0 0 0 2 4 3 *34751 98158 * 0 0 0s 99% 82% Fn 44% 0 0 2 0 0 1 0****
91% 0 0 0 1 3 1 *35022 118549 * 0 0 0s 100% 60% Fn 48% 0 0 1 0 0 0 0****
92% 0 0 0 87 3 1 *25455 123066* 0 0 0s 100% 80% Fn 47% 84 0 3 0 0 1 0****
83% 0 0 0 8 32 2 *36956 121303* 0 0 0s 100% 88% : 46% 0 0 8 0 0 30 0****
85% 0 0 0 5 2 1 *62064 140088 * 0 0 0s 100% 92% Ff 59% 5 0 0 0 0 0 0****
95% 0 0 0 3 3 1 *39352 117364 * 0 0 0s 99% 71% Ff 48% 0 0 3 0 0 2 0****
94% 0 0 0 3 3 1 *42008 136943 * 0 0 0s 99% 94% Fs 53% 0 0 3 0 0 2 0****
92% 0 0 0 0 2 7 *26147 111286 * 0 0 0s 99% 58% Fn 44% 0 0 0 0 0 0 0****
90% 0 0 0 3 3 1 *29567 110665 * 0 0 20 99% 87% Fn 44% 0 0 3 0 0 1 0****
90% 0 0 0 9 4 1 *40031 111273 * 0 0 16 99% 86% Fn 43% 5 0 4 0 0 2 0****
84% 0 0 0 0 1 1 *36760 105709 * 0 0 0s 100% 84% Fn 43% 0 0 0 0 0 0 0****
92% 0 0 0 3 3 1 *31033 137891 * 0 0 0s 100% 99% : 53% 0 0 3 0 0 2 0****
83% 0 0 0 4 6 1 *32991 140148 * 0 0 0s 100% 89% F 51% 0 0 4 0 0 5 0****
Any thoughts why this could happens ?****
Nothing else is running on this netapp except for this test ****
Thanks****
Alon Z****
_______________________________________________****
Toasters mailing list****
Toasters@teaparty.net****
http://www.teaparty.net/mailman/listinfo/toasters****
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters****
--
Gustatus Similis Pullus ****
--
Gustatus Similis Pullus ****