Hello toasters
I encounter rather strange situation
I have a system FAS2040 with 1 aggr of 23x450g raid group ontap version 8.1.3p2
Every write operation being generated on this aggregate automatically caused a read operation
When I try to create 100% sequential write operation using tools such as sio_ntap or filersio
Instead of the expected 150Mb throughput I get 100MB write and 50MB read
On another FAS2040 system using the same commands I get 100% write and no reads
As you can see
filersio asyncio_active 0 -r 0 64k 0 15g 60 1 /vol/test/testfile
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s
93% 0 0 0 5 26 1 24242 118671 0 0 0s 99% 72% Fn 43% 0 0 5 0 0 24 0
90% 0 0 0 4 4 1 40959 102501 0 0 19 100% 89% Fn 43% 1 0 3 0 0 2 0
86% 0 0 0 3 3 1 23126 125022 0 0 0s 99% 69% : 59% 0 0 3 0 0 2 0
93% 0 0 0 7 1 1 28200 135740 0 0 0s 99% 97% Ff 48% 7 0 0 0 0 0 0
92% 0 0 0 6 15 1 48764 115324 0 0 0s 99% 91% Ff 52% 0 0 6 0 0 14 0
90% 0 0 0 6 16 1 36148 156492 0 0 0s 99% 83% Ff 56% 0 0 6 0 0 14 0
95% 0 0 0 0 4 4 32784 114555 0 0 0s 99% 77% Fs 48% 0 0 0 0 0 0 0
92% 0 0 0 18 90 12 34931 104059 0 0 0s 99% 88% Fn 42% 0 0 18 0 0 79 0
88% 0 0 0 8 3 1 20463 124800 0 0 0s 100% 83% Fn 45% 5 0 3 0 0 2 0
89% 0 0 0 2 2 1 34050 112166 0 0 0s 99% 87% Fn 44% 0 0 2 0 0 1 0
92% 0 0 0 3 7 1 33025 112816 0 0 0s 99% 88% Fn 46% 0 0 3 0 0 5 0
91% 0 0 0 3 3 1 30929 130833 0 0 0s 100% 73% : 50% 0 0 3 0 0 2 0
92% 0 0 0 7 24 2 36484 134260 0 0 0s 100% 98% Ff 50% 0 0 7 0 0 22 0
86% 0 0 0 6 1 1 38608 125140 0 0 0s 99% 90% Ff 44% 5 0 1 0 0 1 0
93% 0 0 0 4 3 1 28748 104544 0 0 0s 100% 75% Ff 43% 0 0 4 0 0 2 0
87% 0 0 0 5 18 1 28128 158608 0 0 0s 100% 90% Fs 55% 0 0 5 0 0 17 0
93% 0 0 0 2 3 34 33115 104980 0 0 0s 99% 85% Fn 41% 0 0 2 0 0 1 32
84% 0 0 0 5 28 7 37185 109425 0 0 0s 100% 78% Fn 48% 0 0 5 0 0 21 0
91% 0 0 0 9 3 1 30224 118322 0 0 0s 99% 97% Fn 47% 7 0 2 0 0 1 0
89% 0 0 0 2 3 4 27649 118335 0 0 0s 99% 88% : 45% 0 0 2 0 0 1 0
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s
in out read write read write age hit time ty util in out in out
88% 0 0 0 14 57 39 43284 133480 0 0 0s 100% 98% Ff 50% 0 0 14 0 0 52 0
92% 0 0 0 2 6 16 47936 109432 0 0 0s 99% 86% Ff 46% 0 0 2 0 0 1 0
88% 0 0 0 23 86 3 45340 141014 0 0 0s 99% 73% Ff 55% 3 0 20 0 0 82 0
92% 0 0 0 10 5 4 21787 123980 0 0 0s 99% 80% Fn 44% 7 0 3 0 0 1 0
95% 0 0 0 2 4 3 34751 98158 0 0 0s 99% 82% Fn 44% 0 0 2 0 0 1 0
91% 0 0 0 1 3 1 35022 118549 0 0 0s 100% 60% Fn 48% 0 0 1 0 0 0 0
92% 0 0 0 87 3 1 25455 123066 0 0 0s 100% 80% Fn 47% 84 0 3 0 0 1 0
83% 0 0 0 8 32 2 36956 121303 0 0 0s 100% 88% : 46% 0 0 8 0 0 30 0
85% 0 0 0 5 2 1 62064 140088 0 0 0s 100% 92% Ff 59% 5 0 0 0 0 0 0
95% 0 0 0 3 3 1 39352 117364 0 0 0s 99% 71% Ff 48% 0 0 3 0 0 2 0
94% 0 0 0 3 3 1 42008 136943 0 0 0s 99% 94% Fs 53% 0 0 3 0 0 2 0
92% 0 0 0 0 2 7 26147 111286 0 0 0s 99% 58% Fn 44% 0 0 0 0 0 0 0
90% 0 0 0 3 3 1 29567 110665 0 0 20 99% 87% Fn 44% 0 0 3 0 0 1 0
90% 0 0 0 9 4 1 40031 111273 0 0 16 99% 86% Fn 43% 5 0 4 0 0 2 0
84% 0 0 0 0 1 1 36760 105709 0 0 0s 100% 84% Fn 43% 0 0 0 0 0 0 0
92% 0 0 0 3 3 1 31033 137891 0 0 0s 100% 99% : 53% 0 0 3 0 0 2 0
83% 0 0 0 4 6 1 32991 140148 0 0 0s 100% 89% F 51% 0 0 4 0 0 5 0
Any thoughts why this could happens ?
Nothing else is running on this netapp except for this test
Thanks
Alon Z
Hmm, interesting question. I am curious to see what the results are.
It sounds like you ruled out that it could be reads from some other operation like such as: Snapmirror/snapvault Sync /semi-sync snapmirror Raid scrub Wafl scan Reallocation
Another thought, not sure if fragmentation could cause that behavior, but do you have a very full aggregate?
-JMS
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Alon Zeltser Sent: Monday, October 21, 2013 10:28 AM To: toasters@teaparty.net Subject: unexplained read operation
Hello toasters I encounter rather strange situation I have a system FAS2040 with 1 aggr of 23x450g raid group ontap version 8.1.3p2 Every write operation being generated on this aggregate automatically caused a read operation When I try to create 100% sequential write operation using tools such as sio_ntap or filersio Instead of the expected 150Mb throughput I get 100MB write and 50MB read On another FAS2040 system using the same commands I get 100% write and no reads As you can see
filersio asyncio_active 0 -r 0 64k 0 15g 60 1 /vol/test/testfile
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s 93% 0 0 0 5 26 1 24242 118671 0 0 0s 99% 72% Fn 43% 0 0 5 0 0 24 0 90% 0 0 0 4 4 1 40959 102501 0 0 19 100% 89% Fn 43% 1 0 3 0 0 2 0 86% 0 0 0 3 3 1 23126 125022 0 0 0s 99% 69% : 59% 0 0 3 0 0 2 0 93% 0 0 0 7 1 1 28200 135740 0 0 0s 99% 97% Ff 48% 7 0 0 0 0 0 0 92% 0 0 0 6 15 1 48764 115324 0 0 0s 99% 91% Ff 52% 0 0 6 0 0 14 0 90% 0 0 0 6 16 1 36148 156492 0 0 0s 99% 83% Ff 56% 0 0 6 0 0 14 0 95% 0 0 0 0 4 4 32784 114555 0 0 0s 99% 77% Fs 48% 0 0 0 0 0 0 0 92% 0 0 0 18 90 12 34931 104059 0 0 0s 99% 88% Fn 42% 0 0 18 0 0 79 0 88% 0 0 0 8 3 1 20463 124800 0 0 0s 100% 83% Fn 45% 5 0 3 0 0 2 0 89% 0 0 0 2 2 1 34050 112166 0 0 0s 99% 87% Fn 44% 0 0 2 0 0 1 0 92% 0 0 0 3 7 1 33025 112816 0 0 0s 99% 88% Fn 46% 0 0 3 0 0 5 0 91% 0 0 0 3 3 1 30929 130833 0 0 0s 100% 73% : 50% 0 0 3 0 0 2 0 92% 0 0 0 7 24 2 36484 134260 0 0 0s 100% 98% Ff 50% 0 0 7 0 0 22 0 86% 0 0 0 6 1 1 38608 125140 0 0 0s 99% 90% Ff 44% 5 0 1 0 0 1 0 93% 0 0 0 4 3 1 28748 104544 0 0 0s 100% 75% Ff 43% 0 0 4 0 0 2 0 87% 0 0 0 5 18 1 28128 158608 0 0 0s 100% 90% Fs 55% 0 0 5 0 0 17 0 93% 0 0 0 2 3 34 33115 104980 0 0 0s 99% 85% Fn 41% 0 0 2 0 0 1 32 84% 0 0 0 5 28 7 37185 109425 0 0 0s 100% 78% Fn 48% 0 0 5 0 0 21 0 91% 0 0 0 9 3 1 30224 118322 0 0 0s 99% 97% Fn 47% 7 0 2 0 0 1 0 89% 0 0 0 2 3 4 27649 118335 0 0 0s 99% 88% : 45% 0 0 2 0 0 1 0 CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out 88% 0 0 0 14 57 39 43284 133480 0 0 0s 100% 98% Ff 50% 0 0 14 0 0 52 0 92% 0 0 0 2 6 16 47936 109432 0 0 0s 99% 86% Ff 46% 0 0 2 0 0 1 0 88% 0 0 0 23 86 3 45340 141014 0 0 0s 99% 73% Ff 55% 3 0 20 0 0 82 0 92% 0 0 0 10 5 4 21787 123980 0 0 0s 99% 80% Fn 44% 7 0 3 0 0 1 0 95% 0 0 0 2 4 3 34751 98158 0 0 0s 99% 82% Fn 44% 0 0 2 0 0 1 0 91% 0 0 0 1 3 1 35022 118549 0 0 0s 100% 60% Fn 48% 0 0 1 0 0 0 0 92% 0 0 0 87 3 1 25455 123066 0 0 0s 100% 80% Fn 47% 84 0 3 0 0 1 0 83% 0 0 0 8 32 2 36956 121303 0 0 0s 100% 88% : 46% 0 0 8 0 0 30 0 85% 0 0 0 5 2 1 62064 140088 0 0 0s 100% 92% Ff 59% 5 0 0 0 0 0 0 95% 0 0 0 3 3 1 39352 117364 0 0 0s 99% 71% Ff 48% 0 0 3 0 0 2 0 94% 0 0 0 3 3 1 42008 136943 0 0 0s 99% 94% Fs 53% 0 0 3 0 0 2 0 92% 0 0 0 0 2 7 26147 111286 0 0 0s 99% 58% Fn 44% 0 0 0 0 0 0 0 90% 0 0 0 3 3 1 29567 110665 0 0 20 99% 87% Fn 44% 0 0 3 0 0 1 0 90% 0 0 0 9 4 1 40031 111273 0 0 16 99% 86% Fn 43% 5 0 4 0 0 2 0 84% 0 0 0 0 1 1 36760 105709 0 0 0s 100% 84% Fn 43% 0 0 0 0 0 0 0 92% 0 0 0 3 3 1 31033 137891 0 0 0s 100% 99% : 53% 0 0 3 0 0 2 0 83% 0 0 0 4 6 1 32991 140148 0 0 0s 100% 89% F 51% 0 0 4 0 0 5 0
Any thoughts why this could happens ? Nothing else is running on this netapp except for this test
Thanks Alon Z
Grab the output of statit for 30sec during this write test.
I have an idea, but need statit data to nail it down.
On Mon, Oct 21, 2013 at 7:44 AM, Jordan Slingerland < Jordan.Slingerland@independenthealth.com> wrote:
Hmm, interesting question. I am curious to see what the results are.****
It sounds like you ruled out that it could be reads from some other operation like such as:****
Snapmirror/snapvault****
Sync /semi-sync snapmirror****
Raid scrub****
Wafl scan****
Reallocation****
Another thought, not sure if fragmentation could cause that behavior, but do you have a very full aggregate? ****
-JMS****
*From:* toasters-bounces@teaparty.net [mailto: toasters-bounces@teaparty.net] *On Behalf Of *Alon Zeltser *Sent:* Monday, October 21, 2013 10:28 AM *To:* toasters@teaparty.net *Subject:* unexplained read operation****
Hello toasters****
I encounter rather strange situation ****
I have a system FAS2040 with 1 aggr of 23x450g raid group ontap version 8.1.3p2****
Every write operation being generated on this aggregate automatically caused a read operation ****
When I try to create 100% sequential write operation using tools such as sio_ntap or filersio ****
Instead of the expected 150Mb throughput I get 100MB write and 50MB read *
On another FAS2040 system using the same commands I get 100% write and no reads****
As you can see****
filersio asyncio_active 0 -r 0 64k 0 15g 60 1 /vol/test/testfile****
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s****
93% 0 0 0 5 26 1 *24242 118671 * 0 0 0s 99% 72% Fn 43% 0 0 5 0 0 24 0****
90% 0 0 0 4 4 1 *40959 102501 * 0 0 19 100% 89% Fn 43% 1 0 3 0 0 2 0****
86% 0 0 0 3 3 1 *23126 125022 * 0 0 0s 99% 69% : 59% 0 0 3 0 0 2 0****
93% 0 0 0 7 1 1 *28200 135740 * 0 0 0s 99% 97% Ff 48% 7 0 0 0 0 0 0****
92% 0 0 0 6 15 1 * 48764 115324 * 0 0 0s 99% 91% Ff 52% 0 0 6 0 0 14 0****
90% 0 0 0 6 16 1 * 36148 156492 * 0 0 0s 99% 83% Ff 56% 0 0 6 0 0 14 0****
95% 0 0 0 0 4 4 *32784 114555 * 0 0 0s 99% 77% Fs 48% 0 0 0 0 0 0 0****
92% 0 0 0 18 90 1*2 34931 104059 * 0 0 0s 99% 88% Fn 42% 0 0 18 0 0 79 0****
88% 0 0 0 8 3 1 *20463 124800 * 0 0 0s 100% 83% Fn 45% 5 0 3 0 0 2 0****
89% 0 0 0 2 2 1 *34050 112166 * 0 0 0s 99% 87% Fn 44% 0 0 2 0 0 1 0****
92% 0 0 0 3 7 1 *33025 112816 * 0 0 0s 99% 88% Fn 46% 0 0 3 0 0 5 0****
91% 0 0 0 3 3 1 *30929 130833 * 0 0 0s 100% 73% : 50% 0 0 3 0 0 2 0****
92% 0 0 0 7 24 2 * 36484 134260 * 0 0 0s 100% 98% Ff 50% 0 0 7 0 0 22 0****
86% 0 0 0 6 1 1 *38608 125140 * 0 0 0s 99% 90% Ff 44% 5 0 1 0 0 1 0****
93% 0 0 0 4 3 1 *28748 104544 * 0 0 0s 100% 75% Ff 43% 0 0 4 0 0 2 0****
87% 0 0 0 5 18 1 * 28128 158608 * 0 0 0s 100% 90% Fs 55% 0 0 5 0 0 17 0****
93% 0 0 0 2 3 34 * 33115 104980 * 0 0 0s 99% 85% Fn 41% 0 0 2 0 0 1 32****
84% 0 0 0 5 28 7 * 37185 109425 * 0 0 0s 100% 78% Fn 48% 0 0 5 0 0 21 0****
91% 0 0 0 9 3 1 *30224 118322 * 0 0 0s 99% 97% Fn 47% 7 0 2 0 0 1 0****
89% 0 0 0 2 3 4 *27649 118335 * 0 0 0s 99% 88% : 45% 0 0 2 0 0 1 0****
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s****
in out read write readwrite age hit time ty util in out in out****
88% 0 0 0 14 57 39 * 43284 13348*0 0 0 0s 100% 98% Ff 50% 0 0 14 0 0 52 0****
92% 0 0 0 2 6 16 *47936 109432* 0 0 0s 99% 86% Ff 46% 0 0 2 0 0 1 0****
88% 0 0 0 23 86 3 * 45340 141014* 0 0 0s 99% 73% Ff 55% 3 0 20 0 0 82 0****
92% 0 0 0 10 5 4 *21787 123980* 0 0 0s 99% 80% Fn 44% 7 0 3 0 0 1 0****
95% 0 0 0 2 4 3 *34751 98158 * 0 0 0s 99% 82% Fn 44% 0 0 2 0 0 1 0****
91% 0 0 0 1 3 1 *35022 118549 * 0 0 0s 100% 60% Fn 48% 0 0 1 0 0 0 0****
92% 0 0 0 87 3 1 *25455 123066* 0 0 0s 100% 80% Fn 47% 84 0 3 0 0 1 0****
83% 0 0 0 8 32 2 *36956 121303* 0 0 0s 100% 88% : 46% 0 0 8 0 0 30 0****
85% 0 0 0 5 2 1 *62064 140088 * 0 0 0s 100% 92% Ff 59% 5 0 0 0 0 0 0****
95% 0 0 0 3 3 1 *39352 117364 * 0 0 0s 99% 71% Ff 48% 0 0 3 0 0 2 0****
94% 0 0 0 3 3 1 *42008 136943 * 0 0 0s 99% 94% Fs 53% 0 0 3 0 0 2 0****
92% 0 0 0 0 2 7 *26147 111286 * 0 0 0s 99% 58% Fn 44% 0 0 0 0 0 0 0****
90% 0 0 0 3 3 1 *29567 110665 * 0 0 20 99% 87% Fn 44% 0 0 3 0 0 1 0****
90% 0 0 0 9 4 1 *40031 111273 * 0 0 16 99% 86% Fn 43% 5 0 4 0 0 2 0****
84% 0 0 0 0 1 1 *36760 105709 * 0 0 0s 100% 84% Fn 43% 0 0 0 0 0 0 0****
92% 0 0 0 3 3 1 *31033 137891 * 0 0 0s 100% 99% : 53% 0 0 3 0 0 2 0****
83% 0 0 0 4 6 1 *32991 140148 * 0 0 0s 100% 89% F 51% 0 0 4 0 0 5 0****
Any thoughts why this could happens ?****
Nothing else is running on this netapp except for this test ****
Thanks****
Alon Z****
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
I'm with Jeff... Looks like CP-Reads. (Need to read from disk to be able to calculate parity. Happens when unable to write 'full stripes', e.g. full aggregate)
Do statit and check your disk statistics:
uread = User Reads, writes = User writes, cpread = CP Read as described above, gread+gwrite = guaranteed writes, like for disk rebuild
Tell us what you find :-)
Sebastian
On 10/21/2013 4:28 PM, Alon Zeltser wrote:
Hello toasters
I encounter rather strange situation
I have a system FAS2040 with 1 aggr of 23x450g raid group ontap version 8.1.3p2
Every write operation being generated on this aggregate automatically caused a read operation
When I try to create 100% sequential write operation using tools such as sio_ntap or filersio
Instead of the expected 150Mb throughput I get 100MB write and 50MB read
On another FAS2040 system using the same commands I get 100% write and no reads
As you can see
filersio asyncio_active 0 -r 0 64k 0 15g 60 1 /vol/test/testfile
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s
93% 0 0 0 5 26 1 *24242 118671 * 0 0 0s 99% 72% Fn 43% 0 0 5 0 0 24 0
90% 0 0 0 4 4 1 *40959 102501
- 0 0 19 100% 89% Fn 43% 1 0 3
0 0 2 0
86% 0 0 0 3 3 1 *23126 125022
- 0 0 0s 99% 69% : 59% 0 0 3
0 0 2 0
93% 0 0 0 7 1 1 *28200 135740
- 0 0 0s 99% 97% Ff 48% 7 0 0
0 0 0 0
92% 0 0 0 6 15 1 * 48764 115324
- 0 0 0s 99% 91% Ff 52% 0 0 6
0 0 14 0
90% 0 0 0 6 16 1 * 36148 156492
- 0 0 0s 99% 83% Ff 56% 0 0 6
0 0 14 0
95% 0 0 0 0 4 4 *32784 114555
- 0 0 0s 99% 77% Fs 48% 0 0 0 0 0 0 0
92% 0 0 0 18 90 1*2 34931 104059
0 0 0s 99% 88% Fn 42% 0 018 0 0 79 0
88% 0 0 0 8 3 1 *20463 124800
- 0 0 0s 100% 83% Fn 45% 5 0 3
0 0 2 0
89% 0 0 0 2 2 1 *34050 112166
- 0 0 0s 99% 87% Fn 44% 0 0 2
0 0 1 0
92% 0 0 0 3 7 1 *33025 112816
- 0 0 0s 99% 88% Fn 46% 0 0 3
0 0 5 0
91% 0 0 0 3 3 1 *30929 130833
- 0 0 0s 100% 73% : 50% 0 0 3
0 0 2 0
92% 0 0 0 7 24 2 * 36484 134260
- 0 0 0s 100% 98% Ff 50% 0 0 7
0 0 22 0
86% 0 0 0 6 1 1 *38608 125140
- 0 0 0s 99% 90% Ff 44% 5 0 1
0 0 1 0
93% 0 0 0 4 3 1 *28748 104544
- 0 0 0s 100% 75% Ff 43% 0 0 4
0 0 2 0
87% 0 0 0 5 18 1 * 28128 158608
- 0 0 0s 100% 90% Fs 55% 0 0 5
0 0 17 0
93% 0 0 0 2 3 34 * 33115 104980
- 0 0 0s 99% 85% Fn 41% 0 0 2
0 0 1 32
84% 0 0 0 5 28 7 * 37185 109425
- 0 0 0s 100% 78% Fn 48% 0 0 5
0 0 21 0
91% 0 0 0 9 3 1 *30224 118322
- 0 0 0s 99% 97% Fn 47% 7 0 2
0 0 1 0
89% 0 0 0 2 3 4 *27649 118335
- 0 0 0s 99% 88% : 45% 0 0 2
0 0 1 0
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s
in out read write read write age hit time ty util in out in out
88% 0 0 0 14 57 39 * 43284 13348*0 0 0 0s 100% 98% Ff 50% 0 0 14 0 0 52 0
92% 0 0 0 2 6 16 *47936 109432* 0 0 0s 99% 86% Ff 46% 0 0 2 0 0 1 0
88% 0 0 0 23 86 3 * 45340 141014* 0 0 0s 99% 73% Ff 55% 3 0 20 0 0 82 0
92% 0 0 0 10 5 4 *21787 123980* 0 0 0s 99% 80% Fn 44% 7 0 3 0 0 1 0
95% 0 0 0 2 4 3 *34751 98158
- 0 0 0s 99% 82% Fn 44% 0 0 2
0 0 1 0
91% 0 0 0 1 3 1 *35022 118549
0 0 0s 100% 60% Fn 48% 0 0 10 0 0 0
92% 0 0 0 87 3 1 *25455 123066* 0 0 0s 100% 80% Fn 47% 84 0 3 0 0 1 0
83% 0 0 0 8 32 2 *36956 121303* 0 0 0s 100% 88% : 46% 0 0 8 0 0 30 0
85% 0 0 0 5 2 1 *62064 140088
0 0 0s 100% 92% Ff 59% 5 0 00 0 0 0
95% 0 0 0 3 3 1 *39352 117364
0 0 0s 99% 71% Ff 48% 0 0 30 0 2 0
94% 0 0 0 3 3 1 *42008 136943
0 0 0s 99% 94% Fs 53% 0 0 30 0 2 0
92% 0 0 0 0 2 7 *26147 111286
0 0 0s 99% 58% Fn 44% 0 0 00 0 0 0
90% 0 0 0 3 3 1 *29567 110665
0 0 20 99% 87% Fn 44% 0 0 30 0 1 0
90% 0 0 0 9 4 1 *40031 111273
0 0 16 99% 86% Fn 43% 5 0 40 0 2 0
84% 0 0 0 0 1 1 *36760 105709
0 0 0s 100% 84% Fn 43% 0 0 00 0 0 0
92% 0 0 0 3 3 1 *31033 137891
0 0 0s 100% 99% : 53% 0 0 30 0 2 0
83% 0 0 0 4 6 1 *32991 140148
0 0 0s 100% 89% F 51% 0 0 40 0 5 0
Any thoughts why this could happens ?
Nothing else is running on this netapp except for this test
Thanks
Alon Z
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Looks like you nailed it
After looking at statit we can see the read operation indeed come from cp reads due to partial stripes
1590.51 stripes written
1422.93 partial stripes
167.58 full stripes
disk ut% xfers ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs /aggr0/plex0/rg0:
0d.01.0 46 115.81 0.35 1.00 2500 53.57 29.94 263 61.89 23.12 232 0.00 .... . 0.00 .... .
0d.01.1 50 117.89 0.35 1.00 4000 55.65 28.89 294 61.89 23.12 246 0.00 .... . 0.00 .... .
0d.01.2 47 106.45 10.23 1.02 11383 50.80 27.06 338 45.42 5.62 1193 0.00 .... . 0.00 .... .
0d.01.3 47 108.53 12.66 1.05 12416 49.41 26.11 378 46.46 6.90 896 0.00 .... . 0.00 .... .
0d.01.4 46 108.88 12.66 1.15 11179 48.72 26.20 402 47.50 7.11 838 0.00 .... . 0.00 .... .
0d.01.5 48 109.92 12.14 1.21 10082 49.58 26.30 399 48.20 6.51 1081 0.00 .... . 0.00 .... .
0d.01.6 47 100.03 8.84 1.00 10235 49.41 26.08 377 41.78 7.76 774 0.00 .... . 0.00 .... .
0d.01.7 46 103.68 10.92 1.08 10441 48.54 26.80 357 44.21 6.75 963 0.00 .... . 0.00 .... .
0d.01.8 48 106.80 11.10 1.09 10314 49.06 26.54 390 46.64 6.61 949 0.00 .... . 0.00 .... .
0d.01.9 46 110.44 12.66 1.05 11429 46.98 26.48 370 50.80 7.31 820 0.00 .... . 0.00 .... .
0d.01.10 47 106.28 9.36 1.06 12105 49.24 25.46 373 47.68 7.53 807 0.00 .... . 0.00 .... .
0d.01.11 46 105.58 9.19 1.02 10463 49.06 26.77 360 47.33 5.90 1078 0.00 .... . 0.00 .... .
0d.01.12 42 108.18 11.44 1.09 11264 48.89 25.60 335 47.85 7.18 902 0.00 .... . 0.00 .... .
0d.01.13 43 103.85 11.10 1.06 11559 49.06 26.45 319 43.69 7.40 757 0.00 .... . 0.00 .... .
0d.01.14 42 105.93 10.92 1.08 11941 49.24 25.27 331 45.77 7.77 757 0.00 .... . 0.00 .... .
0d.01.15 45 117.02 12.14 1.04 12877 48.89 24.43 372 56.00 7.70 802 0.00 .... . 0.00 .... .
0d.01.16 44 102.64 10.40 1.03 11694 49.93 25.18 351 42.30 8.15 684 0.00 .... . 0.00 .... .
0d.01.17 44 104.72 11.10 1.13 10750 48.72 26.62 342 44.90 7.14 830 0.00 .... . 0.00 .... .
0d.01.18 43 104.89 10.06 1.03 12350 49.41 26.71 319 45.42 6.37 987 0.00 .... . 0.00 .... .
0d.01.19 44 109.57 10.23 1.05 10629 48.37 26.15 334 50.97 7.29 810 0.00 .... . 0.00 .... .
0d.01.20 47 111.13 12.31 1.10 11372 49.06 25.81 378 49.76 6.92 912 0.00 .... . 0.00 .... .
0d.01.21 44 110.26 11.79 1.06 13569 49.76 25.79 371 48.72 6.82 828 0.00 .... . 0.00 .... .
0d.01.22 44 106.97 11.62 1.06 10042 49.58 26.88 331 45.77 6.07 1030 0.00 .... . 0.00 .... .
Even though the aggregate is at 80% utilization it seems we have fragmentation issue
I'll try to do full reallocate and check again
Thanks for the Help
Alon Z
From: Sebastian Goetze [mailto:spgoetze@gmail.com] Sent: יום ב 21 אוקטובר 2013 19:59 To: Alon Zeltser; toasters@teaparty.net Subject: Re: unexplained read operation
I'm with Jeff... Looks like CP-Reads. (Need to read from disk to be able to calculate parity. Happens when unable to write 'full stripes', e.g. full aggregate)
Do statit and check your disk statistics:
uread = User Reads, writes = User writes, cpread = CP Read as described above, gread+gwrite = guaranteed writes, like for disk rebuild
Tell us what you find :-)
Sebastian
On 10/21/2013 4:28 PM, Alon Zeltser wrote:
Hello toasters
I encounter rather strange situation
I have a system FAS2040 with 1 aggr of 23x450g raid group ontap version 8.1.3p2
Every write operation being generated on this aggregate automatically caused a read operation
When I try to create 100% sequential write operation using tools such as sio_ntap or filersio
Instead of the expected 150Mb throughput I get 100MB write and 50MB read
On another FAS2040 system using the same commands I get 100% write and no reads
As you can see
filersio asyncio_active 0 -r 0 64k 0 15g 60 1 /vol/test/testfile
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s
93% 0 0 0 5 26 1 24242 118671 0 0 0s 99% 72% Fn 43% 0 0 5 0 0 24 0
90% 0 0 0 4 4 1 40959 102501 0 0 19 100% 89% Fn 43% 1 0 3 0 0 2 0
86% 0 0 0 3 3 1 23126 125022 0 0 0s 99% 69% : 59% 0 0 3 0 0 2 0
93% 0 0 0 7 1 1 28200 135740 0 0 0s 99% 97% Ff 48% 7 0 0 0 0 0 0
92% 0 0 0 6 15 1 48764 115324 0 0 0s 99% 91% Ff 52% 0 0 6 0 0 14 0
90% 0 0 0 6 16 1 36148 156492 0 0 0s 99% 83% Ff 56% 0 0 6 0 0 14 0
95% 0 0 0 0 4 4 32784 114555 0 0 0s 99% 77% Fs 48% 0 0 0 0 0 0 0
92% 0 0 0 18 90 12 34931 104059 0 0 0s 99% 88% Fn 42% 0 0 18 0 0 79 0
88% 0 0 0 8 3 1 20463 124800 0 0 0s 100% 83% Fn 45% 5 0 3 0 0 2 0
89% 0 0 0 2 2 1 34050 112166 0 0 0s 99% 87% Fn 44% 0 0 2 0 0 1 0
92% 0 0 0 3 7 1 33025 112816 0 0 0s 99% 88% Fn 46% 0 0 3 0 0 5 0
91% 0 0 0 3 3 1 30929 130833 0 0 0s 100% 73% : 50% 0 0 3 0 0 2 0
92% 0 0 0 7 24 2 36484 134260 0 0 0s 100% 98% Ff 50% 0 0 7 0 0 22 0
86% 0 0 0 6 1 1 38608 125140 0 0 0s 99% 90% Ff 44% 5 0 1 0 0 1 0
93% 0 0 0 4 3 1 28748 104544 0 0 0s 100% 75% Ff 43% 0 0 4 0 0 2 0
87% 0 0 0 5 18 1 28128 158608 0 0 0s 100% 90% Fs 55% 0 0 5 0 0 17 0
93% 0 0 0 2 3 34 33115 104980 0 0 0s 99% 85% Fn 41% 0 0 2 0 0 1 32
84% 0 0 0 5 28 7 37185 109425 0 0 0s 100% 78% Fn 48% 0 0 5 0 0 21 0
91% 0 0 0 9 3 1 30224 118322 0 0 0s 99% 97% Fn 47% 7 0 2 0 0 1 0
89% 0 0 0 2 3 4 27649 118335 0 0 0s 99% 88% : 45% 0 0 2 0 0 1 0
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s
in out read write read write age hit time ty util in out in out
88% 0 0 0 14 57 39 43284 133480 0 0 0s 100% 98% Ff 50% 0 0 14 0 0 52 0
92% 0 0 0 2 6 16 47936 109432 0 0 0s 99% 86% Ff 46% 0 0 2 0 0 1 0
88% 0 0 0 23 86 3 45340 141014 0 0 0s 99% 73% Ff 55% 3 0 20 0 0 82 0
92% 0 0 0 10 5 4 21787 123980 0 0 0s 99% 80% Fn 44% 7 0 3 0 0 1 0
95% 0 0 0 2 4 3 34751 98158 0 0 0s 99% 82% Fn 44% 0 0 2 0 0 1 0
91% 0 0 0 1 3 1 35022 118549 0 0 0s 100% 60% Fn 48% 0 0 1 0 0 0 0
92% 0 0 0 87 3 1 25455 123066 0 0 0s 100% 80% Fn 47% 84 0 3 0 0 1 0
83% 0 0 0 8 32 2 36956 121303 0 0 0s 100% 88% : 46% 0 0 8 0 0 30 0
85% 0 0 0 5 2 1 62064 140088 0 0 0s 100% 92% Ff 59% 5 0 0 0 0 0 0
95% 0 0 0 3 3 1 39352 117364 0 0 0s 99% 71% Ff 48% 0 0 3 0 0 2 0
94% 0 0 0 3 3 1 42008 136943 0 0 0s 99% 94% Fs 53% 0 0 3 0 0 2 0
92% 0 0 0 0 2 7 26147 111286 0 0 0s 99% 58% Fn 44% 0 0 0 0 0 0 0
90% 0 0 0 3 3 1 29567 110665 0 0 20 99% 87% Fn 44% 0 0 3 0 0 1 0
90% 0 0 0 9 4 1 40031 111273 0 0 16 99% 86% Fn 43% 5 0 4 0 0 2 0
84% 0 0 0 0 1 1 36760 105709 0 0 0s 100% 84% Fn 43% 0 0 0 0 0 0 0
92% 0 0 0 3 3 1 31033 137891 0 0 0s 100% 99% : 53% 0 0 3 0 0 2 0
83% 0 0 0 4 6 1 32991 140148 0 0 0s 100% 89% F 51% 0 0 4 0 0 5 0
Any thoughts why this could happens ?
Nothing else is running on this netapp except for this test
Thanks
Alon Z
_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
What do you mean "full reallocate".
There is a good way, and a really really -bad- way.
On Tue, Oct 22, 2013 at 1:23 AM, Alon Zeltser alonz@emet.co.il wrote:
Looks like you nailed it ****
After looking at statit we can see the read operation indeed come from cp reads due to partial stripes****
1590.51 stripes written ****
1422.93 partial stripes ****
167.58 full stripes****
disk ut% xfers ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs /aggr0/plex0/rg0:****
0d.01.0 46 115.81 0.35 1.00 2500 53.57 29.94 263 *61.89 23*.12 232 0.00 .... . 0.00 .... .****
0d.01.1 50 117.89 0.35 1.00 4000 55.65 28.89 294 *61.89 23*.12 246 0.00 .... . 0.00 .... .****
0d.01.2 47 106.45 10.23 1.02 11383 50.80 27.06 338* 45.42 * 5.62 1193 0.00 .... . 0.00 .... .****
0d.01.3 47 108.53 12.66 1.05 12416 49.41 26.11 378* 46.46 * 6.90 896 0.00 .... . 0.00 .... .****
0d.01.4 46 108.88 12.66 1.15 11179 48.72 26.20 402* 47.50 * 7.11 838 0.00 .... . 0.00 .... .****
0d.01.5 48 109.92 12.14 1.21 10082 49.58 26.30 399* 48.20 * 6.51 1081 0.00 .... . 0.00 .... .****
0d.01.6 47 100.03 8.84 1.00 10235 49.41 26.08 377* 41.78 7*.76 774 0.00 .... . 0.00 .... .****
0d.01.7 46 103.68 10.92 1.08 10441 48.54 26.80 357* 44.21
- 6.75 963 0.00 .... . 0.00 .... .****
0d.01.8 48 106.80 11.10 1.09 10314 49.06 26.54 390* 46.64
- 6.61 949 0.00 .... . 0.00 .... .****
0d.01.9 46 110.44 12.66 1.05 11429 46.98 26.48 370* 50.80
- 7.31 820 0.00 .... . 0.00 .... .****
0d.01.10 47 106.28 9.36 1.06 12105 49.24 25.46 373* 47.68
- 7.53 807 0.00 .... . 0.00 .... .****
0d.01.11 46 105.58 9.19 1.02 10463 49.06 26.77 360* 47.33
- 5.90 1078 0.00 .... . 0.00 .... .****
0d.01.12 42 108.18 11.44 1.09 11264 48.89 25.60 335* 47.85 * 7.18 902 0.00 .... . 0.00 .... .****
0d.01.13 43 103.85 11.10 1.06 11559 49.06 26.45 319* 43.69 * 7.40 757 0.00 .... . 0.00 .... .****
0d.01.14 42 105.93 10.92 1.08 11941 49.24 25.27 331* 45.77 * 7.77 757 0.00 .... . 0.00 .... .****
0d.01.15 45 117.02 12.14 1.04 12877 48.89 24.43 372* 56.00 * 7.70 802 0.00 .... . 0.00 .... .****
0d.01.16 44 102.64 10.40 1.03 11694 49.93 25.18 351* 42.30 * 8.15 684 0.00 .... . 0.00 .... .****
0d.01.17 44 104.72 11.10 1.13 10750 48.72 26.62 342* 44.90 * 7.14 830 0.00 .... . 0.00 .... .****
0d.01.18 43 104.89 10.06 1.03 12350 49.41 26.71 319* 45.42 * 6.37 987 0.00 .... . 0.00 .... .****
0d.01.19 44 109.57 10.23 1.05 10629 48.37 26.15 334* 50.97 * 7.29 810 0.00 .... . 0.00 .... .****
0d.01.20 47 111.13 12.31 1.10 11372 49.06 25.81 378* 49.76 * 6.92 912 0.00 .... . 0.00 .... .****
0d.01.21 44 110.26 11.79 1.06 13569 49.76 25.79 371* 48.72 * 6.82 828 0.00 .... . 0.00 .... .****
0d.01.22 44 106.97 11.62 1.06 10042 49.58 26.88 331* 45.77 * 6.07 1030 0.00 .... . 0.00 .... .****
Even though the aggregate is at 80% utilization it seems we have fragmentation issue****
I'll try to do full reallocate and check again ****
Thanks for the Help****
*Alon Z*****
*From:* Sebastian Goetze [mailto:spgoetze@gmail.com] *Sent:* יום ב 21 אוקטובר 2013 19:59 *To:* Alon Zeltser; toasters@teaparty.net *Subject:* Re: unexplained read operation****
I'm with Jeff... Looks like CP-Reads. (Need to read from disk to be able to calculate parity. Happens when unable to write 'full stripes', e.g. full aggregate)
Do statit and check your disk statistics:****
uread = User Reads, writes = User writes, cpread = CP Read as described above, gread+gwrite = guaranteed writes, like for disk rebuild****
Tell us what you find :-)
Sebastian
On 10/21/2013 4:28 PM, Alon Zeltser wrote:****
Hello toasters****
I encounter rather strange situation ****
I have a system FAS2040 with 1 aggr of 23x450g raid group ontap version 8.1.3p2****
Every write operation being generated on this aggregate automatically caused a read operation ****
When I try to create 100% sequential write operation using tools such as sio_ntap or filersio ****
Instead of the expected 150Mb throughput I get 100MB write and 50MB read *
On another FAS2040 system using the same commands I get 100% write and no reads****
As you can see****
filersio asyncio_active 0 -r 0 64k 0 15g 60 1 /vol/test/testfile****
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s****
93% 0 0 0 5 26 1 *24242 118671 * 0 0 0s 99% 72% Fn 43% 0 0 5 0 0 24 0****
90% 0 0 0 4 4 1 *40959 102501 * 0 0 19 100% 89% Fn 43% 1 0 3 0 0 2 0****
86% 0 0 0 3 3 1 *23126 125022 * 0 0 0s 99% 69% : 59% 0 0 3 0 0 2 0****
93% 0 0 0 7 1 1 *28200 135740 * 0 0 0s 99% 97% Ff 48% 7 0 0 0 0 0 0****
92% 0 0 0 6 15 1 * 48764 115324 * 0 0 0s 99% 91% Ff 52% 0 0 6 0 0 14 0****
90% 0 0 0 6 16 1 * 36148 156492 * 0 0 0s 99% 83% Ff 56% 0 0 6 0 0 14 0****
95% 0 0 0 0 4 4 *32784 114555 * 0 0 0s 99% 77% Fs 48% 0 0 0 0 0 0 0****
92% 0 0 0 18 90 1*2 34931 104059 * 0 0 0s 99% 88% Fn 42% 0 0 18 0 0 79 0****
88% 0 0 0 8 3 1 *20463 124800 * 0 0 0s 100% 83% Fn 45% 5 0 3 0 0 2 0****
89% 0 0 0 2 2 1 *34050 112166 * 0 0 0s 99% 87% Fn 44% 0 0 2 0 0 1 0****
92% 0 0 0 3 7 1 *33025 112816 * 0 0 0s 99% 88% Fn 46% 0 0 3 0 0 5 0****
91% 0 0 0 3 3 1 *30929 130833 * 0 0 0s 100% 73% : 50% 0 0 3 0 0 2 0****
92% 0 0 0 7 24 2 * 36484 134260 * 0 0 0s 100% 98% Ff 50% 0 0 7 0 0 22 0****
86% 0 0 0 6 1 1 *38608 125140 * 0 0 0s 99% 90% Ff 44% 5 0 1 0 0 1 0****
93% 0 0 0 4 3 1 *28748 104544 * 0 0 0s 100% 75% Ff 43% 0 0 4 0 0 2 0****
87% 0 0 0 5 18 1 * 28128 158608 * 0 0 0s 100% 90% Fs 55% 0 0 5 0 0 17 0****
93% 0 0 0 2 3 34 * 33115 104980 * 0 0 0s 99% 85% Fn 41% 0 0 2 0 0 1 32****
84% 0 0 0 5 28 7 * 37185 109425 * 0 0 0s 100% 78% Fn 48% 0 0 5 0 0 21 0****
91% 0 0 0 9 3 1 *30224 118322 * 0 0 0s 99% 97% Fn 47% 7 0 2 0 0 1 0****
89% 0 0 0 2 3 4 *27649 118335 * 0 0 0s 99% 88% : 45% 0 0 2 0 0 1 0****
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s****
in out read write readwrite age hit time ty util in out in out****
88% 0 0 0 14 57 39 * 43284 13348*0 0 0 0s 100% 98% Ff 50% 0 0 14 0 0 52 0****
92% 0 0 0 2 6 16 *47936 109432* 0 0 0s 99% 86% Ff 46% 0 0 2 0 0 1 0****
88% 0 0 0 23 86 3 * 45340 141014* 0 0 0s 99% 73% Ff 55% 3 0 20 0 0 82 0****
92% 0 0 0 10 5 4 *21787 123980* 0 0 0s 99% 80% Fn 44% 7 0 3 0 0 1 0****
95% 0 0 0 2 4 3 *34751 98158 * 0 0 0s 99% 82% Fn 44% 0 0 2 0 0 1 0****
91% 0 0 0 1 3 1 *35022 118549 * 0 0 0s 100% 60% Fn 48% 0 0 1 0 0 0 0****
92% 0 0 0 87 3 1 *25455 123066* 0 0 0s 100% 80% Fn 47% 84 0 3 0 0 1 0****
83% 0 0 0 8 32 2 *36956 121303* 0 0 0s 100% 88% : 46% 0 0 8 0 0 30 0****
85% 0 0 0 5 2 1 *62064 140088 * 0 0 0s 100% 92% Ff 59% 5 0 0 0 0 0 0****
95% 0 0 0 3 3 1 *39352 117364 * 0 0 0s 99% 71% Ff 48% 0 0 3 0 0 2 0****
94% 0 0 0 3 3 1 *42008 136943 * 0 0 0s 99% 94% Fs 53% 0 0 3 0 0 2 0****
92% 0 0 0 0 2 7 *26147 111286 * 0 0 0s 99% 58% Fn 44% 0 0 0 0 0 0 0****
90% 0 0 0 3 3 1 *29567 110665 * 0 0 20 99% 87% Fn 44% 0 0 3 0 0 1 0****
90% 0 0 0 9 4 1 *40031 111273 * 0 0 16 99% 86% Fn 43% 5 0 4 0 0 2 0****
84% 0 0 0 0 1 1 *36760 105709 * 0 0 0s 100% 84% Fn 43% 0 0 0 0 0 0 0****
92% 0 0 0 3 3 1 *31033 137891 * 0 0 0s 100% 99% : 53% 0 0 3 0 0 2 0****
83% 0 0 0 4 6 1 *32991 140148 * 0 0 0s 100% 89% F 51% 0 0 4 0 0 5 0****
Any thoughts why this could happens ?****
Nothing else is running on this netapp except for this test ****
Thanks****
Alon Z****
_______________________________________________****
Toasters mailing list****
Toasters@teaparty.net****
http://www.teaparty.net/mailman/listinfo/toasters****
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
I assume he means run a reallocation scan on all volumes within the aggregate, but NOT! on the aggregate.
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Jeff Mohler Sent: Tuesday, October 22, 2013 1:35 PM To: Alon Zeltser Cc: toasters@teaparty.net Subject: Re: unexplained read operation
What do you mean "full reallocate".
There is a good way, and a really really -bad- way.
On Tue, Oct 22, 2013 at 1:23 AM, Alon Zeltser <alonz@emet.co.ilmailto:alonz@emet.co.il> wrote: Looks like you nailed it After looking at statit we can see the read operation indeed come from cp reads due to partial stripes
1590.51 stripes written 1422.93 partial stripes 167.58 full stripes
disk ut% xfers ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs /aggr0/plex0/rg0: 0d.01.0 46 115.81 0.35 1.00 2500 53.57 29.94 263 61.89 23.12 232 0.00 .... . 0.00 .... . 0d.01.1 50 117.89 0.35 1.00 4000 55.65 28.89 294 61.89 23.12 246 0.00 .... . 0.00 .... . 0d.01.2 47 106.45 10.23 1.02 11383 50.80 27.06 338 45.42 5.62 1193 0.00 .... . 0.00 .... . 0d.01.3 47 108.53 12.66 1.05 12416 49.41 26.11 378 46.46 6.90 896 0.00 .... . 0.00 .... . 0d.01.4 46 108.88 12.66 1.15 11179 48.72 26.20 402 47.50 7.11 838 0.00 .... . 0.00 .... . 0d.01.5 48 109.92 12.14 1.21 10082 49.58 26.30 399 48.20 6.51 1081 0.00 .... . 0.00 .... . 0d.01.6 47 100.03 8.84 1.00 10235 49.41 26.08 377 41.78 7.76 774 0.00 .... . 0.00 .... . 0d.01.7 46 103.68 10.92 1.08 10441 48.54 26.80 357 44.21 6.75 963 0.00 .... . 0.00 .... . 0d.01.8 48 106.80 11.10 1.09 10314 49.06 26.54 390 46.64 6.61 949 0.00 .... . 0.00 .... . 0d.01.9 46 110.44 12.66 1.05 11429 46.98 26.48 370 50.80 7.31 820 0.00 .... . 0.00 .... . 0d.01.10 47 106.28 9.36 1.06 12105 49.24 25.46 373 47.68 7.53 807 0.00 .... . 0.00 .... . 0d.01.11 46 105.58 9.19 1.02 10463 49.06 26.77 360 47.33 5.90 1078 0.00 .... . 0.00 .... . 0d.01.12 42 108.18 11.44 1.09 11264 48.89 25.60 335 47.85 7.18 902 0.00 .... . 0.00 .... . 0d.01.13 43 103.85 11.10 1.06 11559 49.06 26.45 319 43.69 7.40 757 0.00 .... . 0.00 .... . 0d.01.14 42 105.93 10.92 1.08 11941 49.24 25.27 331 45.77 7.77 757 0.00 .... . 0.00 .... . 0d.01.15 45 117.02 12.14 1.04 12877 48.89 24.43 372 56.00 7.70 802 0.00 .... . 0.00 .... . 0d.01.16 44 102.64 10.40 1.03 11694 49.93 25.18 351 42.30 8.15 684 0.00 .... . 0.00 .... . 0d.01.17 44 104.72 11.10 1.13 10750 48.72 26.62 342 44.90 7.14 830 0.00 .... . 0.00 .... . 0d.01.18 43 104.89 10.06 1.03 12350 49.41 26.71 319 45.42 6.37 987 0.00 .... . 0.00 .... . 0d.01.19 44 109.57 10.23 1.05 10629 48.37 26.15 334 50.97 7.29 810 0.00 .... . 0.00 .... . 0d.01.20 47 111.13 12.31 1.10 11372 49.06 25.81 378 49.76 6.92 912 0.00 .... . 0.00 .... . 0d.01.21 44 110.26 11.79 1.06 13569 49.76 25.79 371 48.72 6.82 828 0.00 .... . 0.00 .... . 0d.01.22 44 106.97 11.62 1.06 10042 49.58 26.88 331 45.77 6.07 1030 0.00 .... . 0.00 .... .
Even though the aggregate is at 80% utilization it seems we have fragmentation issue I'll try to do full reallocate and check again Thanks for the Help Alon Z
From: Sebastian Goetze [mailto:spgoetze@gmail.commailto:spgoetze@gmail.com] Sent: יום ב 21 אוקטובר 2013 19:59 To: Alon Zeltser; toasters@teaparty.netmailto:toasters@teaparty.net Subject: Re: unexplained read operation
I'm with Jeff... Looks like CP-Reads. (Need to read from disk to be able to calculate parity. Happens when unable to write 'full stripes', e.g. full aggregate)
Do statit and check your disk statistics: uread = User Reads, writes = User writes, cpread = CP Read as described above, gread+gwrite = guaranteed writes, like for disk rebuild Tell us what you find :-)
Sebastian On 10/21/2013 4:28 PM, Alon Zeltser wrote: Hello toasters I encounter rather strange situation I have a system FAS2040 with 1 aggr of 23x450g raid group ontap version 8.1.3p2 Every write operation being generated on this aggregate automatically caused a read operation When I try to create 100% sequential write operation using tools such as sio_ntap or filersio Instead of the expected 150Mb throughput I get 100MB write and 50MB read On another FAS2040 system using the same commands I get 100% write and no reads As you can see
filersio asyncio_active 0 -r 0 64k 0 15g 60 1 /vol/test/testfile
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s 93% 0 0 0 5 26 1 24242 118671 0 0 0s 99% 72% Fn 43% 0 0 5 0 0 24 0 90% 0 0 0 4 4 1 40959 102501 0 0 19 100% 89% Fn 43% 1 0 3 0 0 2 0 86% 0 0 0 3 3 1 23126 125022 0 0 0s 99% 69% : 59% 0 0 3 0 0 2 0 93% 0 0 0 7 1 1 28200 135740 0 0 0s 99% 97% Ff 48% 7 0 0 0 0 0 0 92% 0 0 0 6 15 1 48764 115324 0 0 0s 99% 91% Ff 52% 0 0 6 0 0 14 0 90% 0 0 0 6 16 1 36148 156492 0 0 0s 99% 83% Ff 56% 0 0 6 0 0 14 0 95% 0 0 0 0 4 4 32784 114555 0 0 0s 99% 77% Fs 48% 0 0 0 0 0 0 0 92% 0 0 0 18 90 12 34931 104059 0 0 0s 99% 88% Fn 42% 0 0 18 0 0 79 0 88% 0 0 0 8 3 1 20463 124800 0 0 0s 100% 83% Fn 45% 5 0 3 0 0 2 0 89% 0 0 0 2 2 1 34050 112166 0 0 0s 99% 87% Fn 44% 0 0 2 0 0 1 0 92% 0 0 0 3 7 1 33025 112816 0 0 0s 99% 88% Fn 46% 0 0 3 0 0 5 0 91% 0 0 0 3 3 1 30929 130833 0 0 0s 100% 73% : 50% 0 0 3 0 0 2 0 92% 0 0 0 7 24 2 36484 134260 0 0 0s 100% 98% Ff 50% 0 0 7 0 0 22 0 86% 0 0 0 6 1 1 38608 125140 0 0 0s 99% 90% Ff 44% 5 0 1 0 0 1 0 93% 0 0 0 4 3 1 28748 104544 0 0 0s 100% 75% Ff 43% 0 0 4 0 0 2 0 87% 0 0 0 5 18 1 28128 158608 0 0 0s 100% 90% Fs 55% 0 0 5 0 0 17 0 93% 0 0 0 2 3 34 33115 104980 0 0 0s 99% 85% Fn 41% 0 0 2 0 0 1 32 84% 0 0 0 5 28 7 37185 109425 0 0 0s 100% 78% Fn 48% 0 0 5 0 0 21 0 91% 0 0 0 9 3 1 30224 118322 0 0 0s 99% 97% Fn 47% 7 0 2 0 0 1 0 89% 0 0 0 2 3 4 27649 118335 0 0 0s 99% 88% : 45% 0 0 2 0 0 1 0 CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out 88% 0 0 0 14 57 39 43284 13348tel:43284%20133480 0 0 0s 100% 98% Ff 50% 0 0 14 0 0 52 0 92% 0 0 0 2 6 16 47936 109432 0 0 0s 99% 86% Ff 46% 0 0 2 0 0 1 0 88% 0 0 0 23 86 3 45340 141014 0 0 0s 99% 73% Ff 55% 3 0 20 0 0 82 0 92% 0 0 0 10 5 4 21787 123980 0 0 0s 99% 80% Fn 44% 7 0 3 0 0 1 0 95% 0 0 0 2 4 3 34751 98158tel:34751%C2%A0%2098158 0 0 0s 99% 82% Fn 44% 0 0 2 0 0 1 0 91% 0 0 0 1 3 1 35022 118549 0 0 0s 100% 60% Fn 48% 0 0 1 0 0 0 0 92% 0 0 0 87 3 1 25455 123066 0 0 0s 100% 80% Fn 47% 84 0 3 0 0 1 0 83% 0 0 0 8 32 2 36956 121303 0 0 0s 100% 88% : 46% 0 0 8 0 0 30 0 85% 0 0 0 5 2 1 62064 140088 0 0 0s 100% 92% Ff 59% 5 0 0 0 0 0 0 95% 0 0 0 3 3 1 39352 117364 0 0 0s 99% 71% Ff 48% 0 0 3 0 0 2 0 94% 0 0 0 3 3 1 42008 136943 0 0 0s 99% 94% Fs 53% 0 0 3 0 0 2 0 92% 0 0 0 0 2 7 26147 111286 0 0 0s 99% 58% Fn 44% 0 0 0 0 0 0 0 90% 0 0 0 3 3 1 29567 110665 0 0 20 99% 87% Fn 44% 0 0 3 0 0 1 0 90% 0 0 0 9 4 1 40031 111273 0 0 16 99% 86% Fn 43% 5 0 4 0 0 2 0 84% 0 0 0 0 1 1 36760 105709 0 0 0s 100% 84% Fn 43% 0 0 0 0 0 0 0 92% 0 0 0 3 3 1 31033 137891 0 0 0s 100% 99% : 53% 0 0 3 0 0 2 0 83% 0 0 0 4 6 1 32991 140148 0 0 0s 100% 89% F 51% 0 0 4 0 0 5 0
Any thoughts why this could happens ? Nothing else is running on this netapp except for this test
Thanks Alon Z
_______________________________________________
Toasters mailing list
Toasters@teaparty.netmailto:Toasters@teaparty.net
http://www.teaparty.net/mailman/listinfo/toasters
_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
-- --- Gustatus Similis Pullus
Indeed. :) It's still worth asking first.
On Tue, Oct 22, 2013 at 10:42 AM, Jordan Slingerland < Jordan.Slingerland@independenthealth.com> wrote:
I assume he means run a reallocation scan on all volumes within the aggregate, but NOT! on the aggregate. ****
*From:* toasters-bounces@teaparty.net [mailto: toasters-bounces@teaparty.net] *On Behalf Of *Jeff Mohler *Sent:* Tuesday, October 22, 2013 1:35 PM *To:* Alon Zeltser *Cc:* toasters@teaparty.net
*Subject:* Re: unexplained read operation****
What do you mean "full reallocate".
There is a good way, and a really really -bad- way.
On Tue, Oct 22, 2013 at 1:23 AM, Alon Zeltser alonz@emet.co.il wrote:***
Looks like you nailed it ****
After looking at statit we can see the read operation indeed come from cp reads due to partial stripes****
1590.51 stripes written ****
1422.93 partial stripes ****
167.58 full stripes****
disk ut% xfers ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs /aggr0/plex0/rg0:****
0d.01.0 46 115.81 0.35 1.00 2500 53.57 29.94 263 *61.89 23*.12 232 0.00 .... . 0.00 .... .****
0d.01.1 50 117.89 0.35 1.00 4000 55.65 28.89 294 *61.89 23*.12 246 0.00 .... . 0.00 .... .****
0d.01.2 47 106.45 10.23 1.02 11383 50.80 27.06 338* 45.42 * 5.62 1193 0.00 .... . 0.00 .... .****
0d.01.3 47 108.53 12.66 1.05 12416 49.41 26.11 378* 46.46 * 6.90 896 0.00 .... . 0.00 .... .****
0d.01.4 46 108.88 12.66 1.15 11179 48.72 26.20 402* 47.50 * 7.11 838 0.00 .... . 0.00 .... .****
0d.01.5 48 109.92 12.14 1.21 10082 49.58 26.30 399* 48.20 * 6.51 1081 0.00 .... . 0.00 .... .****
0d.01.6 47 100.03 8.84 1.00 10235 49.41 26.08 377* 41.78 7*.76 774 0.00 .... . 0.00 .... .****
0d.01.7 46 103.68 10.92 1.08 10441 48.54 26.80 357* 44.21
- 6.75 963 0.00 .... . 0.00 .... .****
0d.01.8 48 106.80 11.10 1.09 10314 49.06 26.54 390* 46.64
- 6.61 949 0.00 .... . 0.00 .... .****
0d.01.9 46 110.44 12.66 1.05 11429 46.98 26.48 370* 50.80
- 7.31 820 0.00 .... . 0.00 .... .****
0d.01.10 47 106.28 9.36 1.06 12105 49.24 25.46 373* 47.68
- 7.53 807 0.00 .... . 0.00 .... .****
0d.01.11 46 105.58 9.19 1.02 10463 49.06 26.77 360* 47.33
- 5.90 1078 0.00 .... . 0.00 .... .****
0d.01.12 42 108.18 11.44 1.09 11264 48.89 25.60 335* 47.85 * 7.18 902 0.00 .... . 0.00 .... .****
0d.01.13 43 103.85 11.10 1.06 11559 49.06 26.45 319* 43.69 * 7.40 757 0.00 .... . 0.00 .... .****
0d.01.14 42 105.93 10.92 1.08 11941 49.24 25.27 331* 45.77 * 7.77 757 0.00 .... . 0.00 .... .****
0d.01.15 45 117.02 12.14 1.04 12877 48.89 24.43 372* 56.00 * 7.70 802 0.00 .... . 0.00 .... .****
0d.01.16 44 102.64 10.40 1.03 11694 49.93 25.18 351* 42.30 * 8.15 684 0.00 .... . 0.00 .... .****
0d.01.17 44 104.72 11.10 1.13 10750 48.72 26.62 342* 44.90 * 7.14 830 0.00 .... . 0.00 .... .****
0d.01.18 43 104.89 10.06 1.03 12350 49.41 26.71 319* 45.42 * 6.37 987 0.00 .... . 0.00 .... .****
0d.01.19 44 109.57 10.23 1.05 10629 48.37 26.15 334* 50.97 * 7.29 810 0.00 .... . 0.00 .... .****
0d.01.20 47 111.13 12.31 1.10 11372 49.06 25.81 378* 49.76 * 6.92 912 0.00 .... . 0.00 .... .****
0d.01.21 44 110.26 11.79 1.06 13569 49.76 25.79 371* 48.72 * 6.82 828 0.00 .... . 0.00 .... .****
0d.01.22 44 106.97 11.62 1.06 10042 49.58 26.88 331* 45.77 * 6.07 1030 0.00 .... . 0.00 .... .****
Even though the aggregate is at 80% utilization it seems we have fragmentation issue****
I'll try to do full reallocate and check again ****
Thanks for the Help****
*Alon Z*****
*From:* Sebastian Goetze [mailto:spgoetze@gmail.com] *Sent:* יום ב 21 אוקטובר 2013 19:59 *To:* Alon Zeltser; toasters@teaparty.net *Subject:* Re: unexplained read operation****
I'm with Jeff... Looks like CP-Reads. (Need to read from disk to be able to calculate parity. Happens when unable to write 'full stripes', e.g. full aggregate)
Do statit and check your disk statistics:****
uread = User Reads, writes = User writes, cpread = CP Read as described above, gread+gwrite = guaranteed writes, like for disk rebuild****
Tell us what you find :-)
Sebastian****
On 10/21/2013 4:28 PM, Alon Zeltser wrote:****
Hello toasters****
I encounter rather strange situation ****
I have a system FAS2040 with 1 aggr of 23x450g raid group ontap version 8.1.3p2****
Every write operation being generated on this aggregate automatically caused a read operation ****
When I try to create 100% sequential write operation using tools such as sio_ntap or filersio ****
Instead of the expected 150Mb throughput I get 100MB write and 50MB read *
On another FAS2040 system using the same commands I get 100% write and no reads****
As you can see****
filersio asyncio_active 0 -r 0 64k 0 15g 60 1 /vol/test/testfile****
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s****
93% 0 0 0 5 26 1 *24242 118671 * 0 0 0s 99% 72% Fn 43% 0 0 5 0 0 24 0****
90% 0 0 0 4 4 1 *40959 102501 * 0 0 19 100% 89% Fn 43% 1 0 3 0 0 2 0****
86% 0 0 0 3 3 1 *23126 125022 * 0 0 0s 99% 69% : 59% 0 0 3 0 0 2 0****
93% 0 0 0 7 1 1 *28200 135740 * 0 0 0s 99% 97% Ff 48% 7 0 0 0 0 0 0****
92% 0 0 0 6 15 1 * 48764 115324 * 0 0 0s 99% 91% Ff 52% 0 0 6 0 0 14 0****
90% 0 0 0 6 16 1 * 36148 156492 * 0 0 0s 99% 83% Ff 56% 0 0 6 0 0 14 0****
95% 0 0 0 0 4 4 *32784 114555 * 0 0 0s 99% 77% Fs 48% 0 0 0 0 0 0 0****
92% 0 0 0 18 90 1*2 34931 104059 * 0 0 0s 99% 88% Fn 42% 0 0 18 0 0 79 0****
88% 0 0 0 8 3 1 *20463 124800 * 0 0 0s 100% 83% Fn 45% 5 0 3 0 0 2 0****
89% 0 0 0 2 2 1 *34050 112166 * 0 0 0s 99% 87% Fn 44% 0 0 2 0 0 1 0****
92% 0 0 0 3 7 1 *33025 112816 * 0 0 0s 99% 88% Fn 46% 0 0 3 0 0 5 0****
91% 0 0 0 3 3 1 *30929 130833 * 0 0 0s 100% 73% : 50% 0 0 3 0 0 2 0****
92% 0 0 0 7 24 2 * 36484 134260 * 0 0 0s 100% 98% Ff 50% 0 0 7 0 0 22 0****
86% 0 0 0 6 1 1 *38608 125140 * 0 0 0s 99% 90% Ff 44% 5 0 1 0 0 1 0****
93% 0 0 0 4 3 1 *28748 104544 * 0 0 0s 100% 75% Ff 43% 0 0 4 0 0 2 0****
87% 0 0 0 5 18 1 * 28128 158608 * 0 0 0s 100% 90% Fs 55% 0 0 5 0 0 17 0****
93% 0 0 0 2 3 34 * 33115 104980 * 0 0 0s 99% 85% Fn 41% 0 0 2 0 0 1 32****
84% 0 0 0 5 28 7 * 37185 109425 * 0 0 0s 100% 78% Fn 48% 0 0 5 0 0 21 0****
91% 0 0 0 9 3 1 *30224 118322 * 0 0 0s 99% 97% Fn 47% 7 0 2 0 0 1 0****
89% 0 0 0 2 3 4 *27649 118335 * 0 0 0s 99% 88% : 45% 0 0 2 0 0 1 0****
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s****
in out read write readwrite age hit time ty util in out in out****
88% 0 0 0 14 57 39 * 43284 13348*0 0 0 0s 100% 98% Ff 50% 0 0 14 0 0 52 0****
92% 0 0 0 2 6 16 *47936 109432* 0 0 0s 99% 86% Ff 46% 0 0 2 0 0 1 0****
88% 0 0 0 23 86 3 * 45340 141014* 0 0 0s 99% 73% Ff 55% 3 0 20 0 0 82 0****
92% 0 0 0 10 5 4 *21787 123980* 0 0 0s 99% 80% Fn 44% 7 0 3 0 0 1 0****
95% 0 0 0 2 4 3 *34751 98158 * 0 0 0s 99% 82% Fn 44% 0 0 2 0 0 1 0****
91% 0 0 0 1 3 1 *35022 118549 * 0 0 0s 100% 60% Fn 48% 0 0 1 0 0 0 0****
92% 0 0 0 87 3 1 *25455 123066* 0 0 0s 100% 80% Fn 47% 84 0 3 0 0 1 0****
83% 0 0 0 8 32 2 *36956 121303* 0 0 0s 100% 88% : 46% 0 0 8 0 0 30 0****
85% 0 0 0 5 2 1 *62064 140088 * 0 0 0s 100% 92% Ff 59% 5 0 0 0 0 0 0****
95% 0 0 0 3 3 1 *39352 117364 * 0 0 0s 99% 71% Ff 48% 0 0 3 0 0 2 0****
94% 0 0 0 3 3 1 *42008 136943 * 0 0 0s 99% 94% Fs 53% 0 0 3 0 0 2 0****
92% 0 0 0 0 2 7 *26147 111286 * 0 0 0s 99% 58% Fn 44% 0 0 0 0 0 0 0****
90% 0 0 0 3 3 1 *29567 110665 * 0 0 20 99% 87% Fn 44% 0 0 3 0 0 1 0****
90% 0 0 0 9 4 1 *40031 111273 * 0 0 16 99% 86% Fn 43% 5 0 4 0 0 2 0****
84% 0 0 0 0 1 1 *36760 105709 * 0 0 0s 100% 84% Fn 43% 0 0 0 0 0 0 0****
92% 0 0 0 3 3 1 *31033 137891 * 0 0 0s 100% 99% : 53% 0 0 3 0 0 2 0****
83% 0 0 0 4 6 1 *32991 140148 * 0 0 0s 100% 89% F 51% 0 0 4 0 0 5 0****
Any thoughts why this could happens ?****
Nothing else is running on this netapp except for this test ****
Thanks****
Alon Z****
_______________________________________________****
Toasters mailing list****
Toasters@teaparty.net****
http://www.teaparty.net/mailman/listinfo/toasters****
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters****
--
Gustatus Similis Pullus ****
What I meant by "full reallocate" is to perform reallocate using the –f & -p flag on all the volumes on this aggr
see https://library.netapp.com/ecmdocs/ECMP1136573/html/GUID-E1789834-11E5-4748-...
I'm aware that running reallocate on the aggregate won't optimize the layout
Thanks for the heads up though
Alon Z
From: Jeff Mohler [mailto:speedtoys.racing@gmail.com] Sent: יום ג 22 אוקטובר 2013 20:46 To: Jordan Slingerland Cc: Alon Zeltser; toasters@teaparty.net Subject: Re: unexplained read operation
Indeed. :) It's still worth asking first.
On Tue, Oct 22, 2013 at 10:42 AM, Jordan Slingerland Jordan.Slingerland@independenthealth.com wrote:
I assume he means run a reallocation scan on all volumes within the aggregate, but NOT! on the aggregate.
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Jeff Mohler Sent: Tuesday, October 22, 2013 1:35 PM To: Alon Zeltser Cc: toasters@teaparty.net
Subject: Re: unexplained read operation
What do you mean "full reallocate".
There is a good way, and a really really -bad- way.
On Tue, Oct 22, 2013 at 1:23 AM, Alon Zeltser alonz@emet.co.il wrote:
Looks like you nailed it
After looking at statit we can see the read operation indeed come from cp reads due to partial stripes
1590.51 stripes written
1422.93 partial stripes
167.58 full stripes
disk ut% xfers ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs /aggr0/plex0/rg0:
0d.01.0 46 115.81 0.35 1.00 2500 53.57 29.94 263 61.89 23.12 232 0.00 .... . 0.00 .... .
0d.01.1 50 117.89 0.35 1.00 4000 55.65 28.89 294 61.89 23.12 246 0.00 .... . 0.00 .... .
0d.01.2 47 106.45 10.23 1.02 11383 50.80 27.06 338 45.42 5.62 1193 0.00 .... . 0.00 .... .
0d.01.3 47 108.53 12.66 1.05 12416 49.41 26.11 378 46.46 6.90 896 0.00 .... . 0.00 .... .
0d.01.4 46 108.88 12.66 1.15 11179 48.72 26.20 402 47.50 7.11 838 0.00 .... . 0.00 .... .
0d.01.5 48 109.92 12.14 1.21 10082 49.58 26.30 399 48.20 6.51 1081 0.00 .... . 0.00 .... .
0d.01.6 47 100.03 8.84 1.00 10235 49.41 26.08 377 41.78 7.76 774 0.00 .... . 0.00 .... .
0d.01.7 46 103.68 10.92 1.08 10441 48.54 26.80 357 44.21 6.75 963 0.00 .... . 0.00 .... .
0d.01.8 48 106.80 11.10 1.09 10314 49.06 26.54 390 46.64 6.61 949 0.00 .... . 0.00 .... .
0d.01.9 46 110.44 12.66 1.05 11429 46.98 26.48 370 50.80 7.31 820 0.00 .... . 0.00 .... .
0d.01.10 47 106.28 9.36 1.06 12105 49.24 25.46 373 47.68 7.53 807 0.00 .... . 0.00 .... .
0d.01.11 46 105.58 9.19 1.02 10463 49.06 26.77 360 47.33 5.90 1078 0.00 .... . 0.00 .... .
0d.01.12 42 108.18 11.44 1.09 11264 48.89 25.60 335 47.85 7.18 902 0.00 .... . 0.00 .... .
0d.01.13 43 103.85 11.10 1.06 11559 49.06 26.45 319 43.69 7.40 757 0.00 .... . 0.00 .... .
0d.01.14 42 105.93 10.92 1.08 11941 49.24 25.27 331 45.77 7.77 757 0.00 .... . 0.00 .... .
0d.01.15 45 117.02 12.14 1.04 12877 48.89 24.43 372 56.00 7.70 802 0.00 .... . 0.00 .... .
0d.01.16 44 102.64 10.40 1.03 11694 49.93 25.18 351 42.30 8.15 684 0.00 .... . 0.00 .... .
0d.01.17 44 104.72 11.10 1.13 10750 48.72 26.62 342 44.90 7.14 830 0.00 .... . 0.00 .... .
0d.01.18 43 104.89 10.06 1.03 12350 49.41 26.71 319 45.42 6.37 987 0.00 .... . 0.00 .... .
0d.01.19 44 109.57 10.23 1.05 10629 48.37 26.15 334 50.97 7.29 810 0.00 .... . 0.00 .... .
0d.01.20 47 111.13 12.31 1.10 11372 49.06 25.81 378 49.76 6.92 912 0.00 .... . 0.00 .... .
0d.01.21 44 110.26 11.79 1.06 13569 49.76 25.79 371 48.72 6.82 828 0.00 .... . 0.00 .... .
0d.01.22 44 106.97 11.62 1.06 10042 49.58 26.88 331 45.77 6.07 1030 0.00 .... . 0.00 .... .
Even though the aggregate is at 80% utilization it seems we have fragmentation issue
I'll try to do full reallocate and check again
Thanks for the Help
Alon Z
From: Sebastian Goetze [mailto:spgoetze@gmail.com] Sent: יום ב 21 אוקטובר 2013 19:59 To: Alon Zeltser; toasters@teaparty.net Subject: Re: unexplained read operation
I'm with Jeff... Looks like CP-Reads. (Need to read from disk to be able to calculate parity. Happens when unable to write 'full stripes', e.g. full aggregate)
Do statit and check your disk statistics:
uread = User Reads, writes = User writes, cpread = CP Read as described above, gread+gwrite = guaranteed writes, like for disk rebuild
Tell us what you find :-)
Sebastian
On 10/21/2013 4:28 PM, Alon Zeltser wrote:
Hello toasters
I encounter rather strange situation
I have a system FAS2040 with 1 aggr of 23x450g raid group ontap version 8.1.3p2
Every write operation being generated on this aggregate automatically caused a read operation
When I try to create 100% sequential write operation using tools such as sio_ntap or filersio
Instead of the expected 150Mb throughput I get 100MB write and 50MB read
On another FAS2040 system using the same commands I get 100% write and no reads
As you can see
filersio asyncio_active 0 -r 0 64k 0 15g 60 1 /vol/test/testfile
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s
93% 0 0 0 5 26 1 24242 118671 0 0 0s 99% 72% Fn 43% 0 0 5 0 0 24 0
90% 0 0 0 4 4 1 40959 102501 0 0 19 100% 89% Fn 43% 1 0 3 0 0 2 0
86% 0 0 0 3 3 1 23126 125022 0 0 0s 99% 69% : 59% 0 0 3 0 0 2 0
93% 0 0 0 7 1 1 28200 135740 0 0 0s 99% 97% Ff 48% 7 0 0 0 0 0 0
92% 0 0 0 6 15 1 48764 115324 0 0 0s 99% 91% Ff 52% 0 0 6 0 0 14 0
90% 0 0 0 6 16 1 36148 156492 0 0 0s 99% 83% Ff 56% 0 0 6 0 0 14 0
95% 0 0 0 0 4 4 32784 114555 0 0 0s 99% 77% Fs 48% 0 0 0 0 0 0 0
92% 0 0 0 18 90 12 34931 104059 0 0 0s 99% 88% Fn 42% 0 0 18 0 0 79 0
88% 0 0 0 8 3 1 20463 124800 0 0 0s 100% 83% Fn 45% 5 0 3 0 0 2 0
89% 0 0 0 2 2 1 34050 112166 0 0 0s 99% 87% Fn 44% 0 0 2 0 0 1 0
92% 0 0 0 3 7 1 33025 112816 0 0 0s 99% 88% Fn 46% 0 0 3 0 0 5 0
91% 0 0 0 3 3 1 30929 130833 0 0 0s 100% 73% : 50% 0 0 3 0 0 2 0
92% 0 0 0 7 24 2 36484 134260 0 0 0s 100% 98% Ff 50% 0 0 7 0 0 22 0
86% 0 0 0 6 1 1 38608 125140 0 0 0s 99% 90% Ff 44% 5 0 1 0 0 1 0
93% 0 0 0 4 3 1 28748 104544 0 0 0s 100% 75% Ff 43% 0 0 4 0 0 2 0
87% 0 0 0 5 18 1 28128 158608 0 0 0s 100% 90% Fs 55% 0 0 5 0 0 17 0
93% 0 0 0 2 3 34 33115 104980 0 0 0s 99% 85% Fn 41% 0 0 2 0 0 1 32
84% 0 0 0 5 28 7 37185 109425 0 0 0s 100% 78% Fn 48% 0 0 5 0 0 21 0
91% 0 0 0 9 3 1 30224 118322 0 0 0s 99% 97% Fn 47% 7 0 2 0 0 1 0
89% 0 0 0 2 3 4 27649 118335 0 0 0s 99% 88% : 45% 0 0 2 0 0 1 0
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s
in out read write read write age hit time ty util in out in out
88% 0 0 0 14 57 39 43284 133480 0 0 0s 100% 98% Ff 50% 0 0 14 0 0 52 0
92% 0 0 0 2 6 16 47936 109432 0 0 0s 99% 86% Ff 46% 0 0 2 0 0 1 0
88% 0 0 0 23 86 3 45340 141014 0 0 0s 99% 73% Ff 55% 3 0 20 0 0 82 0
92% 0 0 0 10 5 4 21787 123980 0 0 0s 99% 80% Fn 44% 7 0 3 0 0 1 0
95% 0 0 0 2 4 3 34751 98158 0 0 0s 99% 82% Fn 44% 0 0 2 0 0 1 0
91% 0 0 0 1 3 1 35022 118549 0 0 0s 100% 60% Fn 48% 0 0 1 0 0 0 0
92% 0 0 0 87 3 1 25455 123066 0 0 0s 100% 80% Fn 47% 84 0 3 0 0 1 0
83% 0 0 0 8 32 2 36956 121303 0 0 0s 100% 88% : 46% 0 0 8 0 0 30 0
85% 0 0 0 5 2 1 62064 140088 0 0 0s 100% 92% Ff 59% 5 0 0 0 0 0 0
95% 0 0 0 3 3 1 39352 117364 0 0 0s 99% 71% Ff 48% 0 0 3 0 0 2 0
94% 0 0 0 3 3 1 42008 136943 0 0 0s 99% 94% Fs 53% 0 0 3 0 0 2 0
92% 0 0 0 0 2 7 26147 111286 0 0 0s 99% 58% Fn 44% 0 0 0 0 0 0 0
90% 0 0 0 3 3 1 29567 110665 0 0 20 99% 87% Fn 44% 0 0 3 0 0 1 0
90% 0 0 0 9 4 1 40031 111273 0 0 16 99% 86% Fn 43% 5 0 4 0 0 2 0
84% 0 0 0 0 1 1 36760 105709 0 0 0s 100% 84% Fn 43% 0 0 0 0 0 0 0
92% 0 0 0 3 3 1 31033 137891 0 0 0s 100% 99% : 53% 0 0 3 0 0 2 0
83% 0 0 0 4 6 1 32991 140148 0 0 0s 100% 89% F 51% 0 0 4 0 0 5 0
Any thoughts why this could happens ?
Nothing else is running on this netapp except for this test
Thanks
Alon Z
_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Reallocate -A would improve *write* performance, though... It improves the 'free space layout'. But I'd switch that one on at the aggregate options level, if your system, isn't too burdened already.
aggr options aggr_name free_space_realloc on
Sebastian
P.S. For read performance, you could also think about: vol options vol1 read_realloc space_optimized (e.g. for a volume with dedupe on)
On 10/23/2013 9:01 AM, Alon Zeltser wrote:
What I meant by "full reallocate" is to perform reallocate using the --f & -p flag on all the volumes on this aggr
see https://library.netapp.com/ecmdocs/ECMP1136573/html/GUID-E1789834-11E5-4748-...
I'm aware that running reallocate on the aggregate won't optimize the layout
Thanks for the heads up though
**
*Alon Z*
*From:*Jeff Mohler [mailto:speedtoys.racing@gmail.com] *Sent:* ??? ? 22 ??????? 2013 20:46 *To:* Jordan Slingerland *Cc:* Alon Zeltser; toasters@teaparty.net *Subject:* Re: unexplained read operation
Indeed. :) It's still worth asking first.
On Tue, Oct 22, 2013 at 10:42 AM, Jordan Slingerland <Jordan.Slingerland@independenthealth.com mailto:Jordan.Slingerland@independenthealth.com> wrote:
I assume he means run a reallocation scan on all volumes within the aggregate, but NOT! on the aggregate.
*From:*toasters-bounces@teaparty.net mailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net mailto:toasters-bounces@teaparty.net] *On Behalf Of *Jeff Mohler *Sent:* Tuesday, October 22, 2013 1:35 PM *To:* Alon Zeltser *Cc:* toasters@teaparty.net mailto:toasters@teaparty.net
*Subject:* Re: unexplained read operation
What do you mean "full reallocate".
There is a good way, and a really really -bad- way.
On Tue, Oct 22, 2013 at 1:23 AM, Alon Zeltser <alonz@emet.co.il mailto:alonz@emet.co.il> wrote:
Looks like you nailed it
After looking at statit we can see the read operation indeed come from cp reads due to partial stripes
1590.51 stripes written
1422.93 partial stripes
167.58 full stripes
disk ut% xfers ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs /aggr0/plex0/rg0:
0d.01.0 46 115.81 0.35 1.00 2500 53.57 29.94 263 *61.89 23*.12 232 0.00 .... . 0.00 .... .
0d.01.1 50 117.89 0.35 1.00 4000 55.65 28.89 294 *61.89 23*.12 246 0.00 .... . 0.00 .... .
0d.01.2 47 106.45 10.23 1.02 11383 50.80 27.06 338* 45.42
- 5.62 1193 0.00 .... . 0.00 .... .
0d.01.3 47 108.53 12.66 1.05 12416 49.41 26.11 378* 46.46
- 6.90 896 0.00 .... . 0.00 .... .
0d.01.4 46 108.88 12.66 1.15 11179 48.72 26.20 402* 47.50
- 7.11 838 0.00 .... . 0.00 .... .
0d.01.5 48 109.92 12.14 1.21 10082 49.58 26.30 399* 48.20
- 6.51 1081 0.00 .... . 0.00 .... .
0d.01.6 47 100.03 8.84 1.00 10235 49.41 26.08 377* 41.78 7*.76 774 0.00 .... . 0.00 .... .
0d.01.7 46 103.68 10.92 1.08 10441 48.54 26.80 357* 44.21
- 6.75 963 0.00 .... . 0.00 .... .
0d.01.8 48 106.80 11.10 1.09 10314 49.06 26.54 390* 46.64
- 6.61 949 0.00 .... . 0.00 .... .
0d.01.9 46 110.44 12.66 1.05 11429 46.98 26.48 370* 50.80
- 7.31 820 0.00 .... . 0.00 .... .
0d.01.10 47 106.28 9.36 1.06 12105 49.24 25.46 373* 47.68
- 7.53 807 0.00 .... . 0.00 .... .
0d.01.11 46 105.58 9.19 1.02 10463 49.06 26.77 360* 47.33
- 5.90 1078 0.00 .... . 0.00 .... .
0d.01.12 42 108.18 11.44 1.09 11264 48.89 25.60 335* 47.85
- 7.18 902 0.00 .... . 0.00 .... .
0d.01.13 43 103.85 11.10 1.06 11559 49.06 26.45 319* 43.69
- 7.40 757 0.00 .... . 0.00 .... .
0d.01.14 42 105.93 10.92 1.08 11941 49.24 25.27 331* 45.77
- 7.77 757 0.00 .... . 0.00 .... .
0d.01.15 45 117.02 12.14 1.04 12877 48.89 24.43 372* 56.00
- 7.70 802 0.00 .... . 0.00 .... .
0d.01.16 44 102.64 10.40 1.03 11694 49.93 25.18 351* 42.30 * 8.15 684 0.00 .... . 0.00 .... .
0d.01.17 44 104.72 11.10 1.13 10750 48.72 26.62 342* 44.90
- 7.14 830 0.00 .... . 0.00 .... .
0d.01.18 43 104.89 10.06 1.03 12350 49.41 26.71 319* 45.42
- 6.37 987 0.00 .... . 0.00 .... .
0d.01.19 44 109.57 10.23 1.05 10629 48.37 26.15 334* 50.97
- 7.29 810 0.00 .... . 0.00 .... .
0d.01.20 47 111.13 12.31 1.10 11372 49.06 25.81 378* 49.76
- 6.92 912 0.00 .... . 0.00 .... .
0d.01.21 44 110.26 11.79 1.06 13569 49.76 25.79 371* 48.72
- 6.82 828 0.00 .... . 0.00 .... .
0d.01.22 44 106.97 11.62 1.06 10042 49.58 26.88 331* 45.77
- 6.07 1030 0.00 .... . 0.00 .... .
Even though the aggregate is at 80% utilization it seems we have fragmentation issue
I'll try to do full reallocate and check again
Thanks for the Help
*Alon Z*
*From:*Sebastian Goetze [mailto:spgoetze@gmail.com mailto:spgoetze@gmail.com] *Sent:* ???? 21 ??????? 2013 19:59 *To:* Alon Zeltser; toasters@teaparty.net mailto:toasters@teaparty.net *Subject:* Re: unexplained read operation
I'm with Jeff... Looks like CP-Reads. (Need to read from disk to be able to calculate parity. Happens when unable to write 'full stripes', e.g. full aggregate)
Do statit and check your disk statistics:
uread = User Reads, writes = User writes, cpread = CP Read as described above, gread+gwrite = guaranteed writes, like for disk rebuild
Tell us what you find :-)
Sebastian
On 10/21/2013 4:28 PM, Alon Zeltser wrote:
Hello toasters I encounter rather strange situation I have a system FAS2040 with 1 aggr of 23x450g raid group ontap version 8.1.3p2 Every write operation being generated on this aggregate automatically caused a read operation When I try to create 100% sequential write operation using tools such as sio_ntap or filersio Instead of the expected 150Mb throughput I get 100MB write and 50MB read On another FAS2040 system using the same commands I get 100% write and no reads As you can see filersio asyncio_active 0 -r 0 64k 0 15g 60 1 /vol/test/testfile CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s 93% 0 0 0 5 26 1 *24242 118671 * 0 0 0s 99% 72% Fn 43% 0 0 5 0 0 24 0 90% 0 0 0 4 4 1 *40959 102501 * 0 0 19 100% 89% Fn 43% 1 0 3 0 0 2 0 86% 0 0 0 3 3 1 *23126 125022 * 0 0 0s 99% 69% : 59% 0 0 3 0 0 2 0 93% 0 0 0 7 1 1 *28200 135740 * 0 0 0s 99% 97% Ff 48% 7 0 0 0 0 0 0 92% 0 0 0 6 15 1 * 48764 115324 * 0 0 0s 99% 91% Ff 52% 0 0 6 0 0 14 0 90% 0 0 0 6 16 1 * 36148 156492 * 0 0 0s 99% 83% Ff 56% 0 0 6 0 0 14 0 95% 0 0 0 0 4 4 *32784 114555 * 0 0 0s 99% 77% Fs 48% 0 0 0 0 0 0 0 92% 0 0 0 18 90 1*2 34931 104059 * 0 0 0s 99% 88% Fn 42% 0 0 18 0 0 79 0 88% 0 0 0 8 3 1 *20463 124800 * 0 0 0s 100% 83% Fn 45% 5 0 3 0 0 2 0 89% 0 0 0 2 2 1 *34050 112166 * 0 0 0s 99% 87% Fn 44% 0 0 2 0 0 1 0 92% 0 0 0 3 7 1 *33025 112816 * 0 0 0s 99% 88% Fn 46% 0 0 3 0 0 5 0 91% 0 0 0 3 3 1 *30929 130833 * 0 0 0s 100% 73% : 50% 0 0 3 0 0 2 0 92% 0 0 0 7 24 2 * 36484 134260 * 0 0 0s 100% 98% Ff 50% 0 0 7 0 0 22 0 86% 0 0 0 6 1 1 *38608 125140 * 0 0 0s 99% 90% Ff 44% 5 0 1 0 0 1 0 93% 0 0 0 4 3 1 *28748 104544 * 0 0 0s 100% 75% Ff 43% 0 0 4 0 0 2 0 87% 0 0 0 5 18 1 * 28128 158608 * 0 0 0s 100% 90% Fs 55% 0 0 5 0 0 17 0 93% 0 0 0 2 3 34 * 33115 104980 * 0 0 0s 99% 85% Fn 41% 0 0 2 0 0 1 32 84% 0 0 0 5 28 7 * 37185 109425 * 0 0 0s 100% 78% Fn 48% 0 0 5 0 0 21 0 91% 0 0 0 9 3 1 *30224 118322 * 0 0 0s 99% 97% Fn 47% 7 0 2 0 0 1 0 89% 0 0 0 2 3 4 *27649 118335 * 0 0 0s 99% 88% : 45% 0 0 2 0 0 1 0 CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out 88% 0 0 0 14 57 39 *43284 13348 <tel:43284%2013348>*0 0 0 0s 100% 98% Ff 50% 0 0 14 0 0 52 0 92% 0 0 0 2 6 16 *47936 109432* 0 0 0s 99% 86% Ff 46% 0 0 2 0 0 1 0 88% 0 0 0 23 86 3 * 45340 141014* 0 0 0s 99% 73% Ff 55% 3 0 20 0 0 82 0 92% 0 0 0 10 5 4 *21787 123980* 0 0 0s 99% 80% Fn 44% 7 0 3 0 0 1 0 95% 0 0 0 2 4 3 *34751 98158 <tel:34751%C2%A0%2098158> * 0 0 0s 99% 82% Fn 44% 0 0 2 0 0 1 0 91% 0 0 0 1 3 1 *35022 118549 * 0 0 0s 100% 60% Fn 48% 0 0 1 0 0 0 0 92% 0 0 0 87 3 1 *25455 123066* 0 0 0s 100% 80% Fn 47% 84 0 3 0 0 1 0 83% 0 0 0 8 32 2 *36956 121303* 0 0 0s 100% 88% : 46% 0 0 8 0 0 30 0 85% 0 0 0 5 2 1 *62064 140088 * 0 0 0s 100% 92% Ff 59% 5 0 0 0 0 0 0 95% 0 0 0 3 3 1 *39352 117364 * 0 0 0s 99% 71% Ff 48% 0 0 3 0 0 2 0 94% 0 0 0 3 3 1 *42008 136943 * 0 0 0s 99% 94% Fs 53% 0 0 3 0 0 2 0 92% 0 0 0 0 2 7 *26147 111286 * 0 0 0s 99% 58% Fn 44% 0 0 0 0 0 0 0 90% 0 0 0 3 3 1 *29567 110665 * 0 0 20 99% 87% Fn 44% 0 0 3 0 0 1 0 90% 0 0 0 9 4 1 *40031 111273 * 0 0 16 99% 86% Fn 43% 5 0 4 0 0 2 0 84% 0 0 0 0 1 1 *36760 105709 * 0 0 0s 100% 84% Fn 43% 0 0 0 0 0 0 0 92% 0 0 0 3 3 1 *31033 137891 * 0 0 0s 100% 99% : 53% 0 0 3 0 0 2 0 83% 0 0 0 4 6 1 *32991 140148 * 0 0 0s 100% 89% F 51% 0 0 4 0 0 5 0 Any thoughts why this could happens ? Nothing else is running on this netapp except for this test Thanks Alon Z _______________________________________________ Toasters mailing list Toasters@teaparty.net <mailto:Toasters@teaparty.net> http://www.teaparty.net/mailman/listinfo/toasters
Toasters mailing list Toasters@teaparty.net mailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
--
Gustatus Similis Pullus
--
Gustatus Similis Pullus
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
..that too. Realoc and dedupe were at times, not best friends...or..never looked each other in the eye.
On Wed, Oct 23, 2013 at 12:09 AM, Sebastian Goetze spgoetze@gmail.comwrote:
Reallocate -A would improve *write* performance, though... It improves the 'free space layout'. But I'd switch that one on at the aggregate options level, if your system, isn't too burdened already.
aggr options aggr_name free_space_realloc on
Sebastian
P.S. For read performance, you could also think about: vol options vol1 read_realloc space_optimized (e.g. for a volume with dedupe on)
On 10/23/2013 9:01 AM, Alon Zeltser wrote:
What I meant by "full reallocate" is to perform reallocate using the –f & -p flag on all the volumes on this aggr****
see https://library.netapp.com/ecmdocs/ECMP1136573/html/GUID-E1789834-11E5-4748-...
I'm aware that running reallocate on the aggregate won't optimize the layout ****
Thanks for the heads up though ****
*Alon Z* ****
*From:* Jeff Mohler [mailto:speedtoys.racing@gmail.comspeedtoys.racing@gmail.com]
*Sent:* יום ג 22 אוקטובר 2013 20:46 *To:* Jordan Slingerland *Cc:* Alon Zeltser; toasters@teaparty.net *Subject:* Re: unexplained read operation****
Indeed. :) It's still worth asking first.****
On Tue, Oct 22, 2013 at 10:42 AM, Jordan Slingerland < Jordan.Slingerland@independenthealth.com> wrote:****
I assume he means run a reallocation scan on all volumes within the aggregate, but NOT! on the aggregate. ****
*From:* toasters-bounces@teaparty.net [mailto: toasters-bounces@teaparty.net] *On Behalf Of *Jeff Mohler *Sent:* Tuesday, October 22, 2013 1:35 PM *To:* Alon Zeltser *Cc:* toasters@teaparty.net****
*Subject:* Re: unexplained read operation****
What do you mean "full reallocate".****
There is a good way, and a really really -bad- way.****
On Tue, Oct 22, 2013 at 1:23 AM, Alon Zeltser alonz@emet.co.il wrote:***
Looks like you nailed it ****
After looking at statit we can see the read operation indeed come from cp reads due to partial stripes****
1590.51 stripes written ****
1422.93 partial stripes ****
167.58 full stripes****
disk ut% xfers ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs /aggr0/plex0/rg0:****
0d.01.0 46 115.81 0.35 1.00 2500 53.57 29.94 263 *61.89 23*.12 232 0.00 .... . 0.00 .... .****
0d.01.1 50 117.89 0.35 1.00 4000 55.65 28.89 294 *61.89 23*.12 246 0.00 .... . 0.00 .... .****
0d.01.2 47 106.45 10.23 1.02 11383 50.80 27.06 338* 45.42 * 5.62 1193 0.00 .... . 0.00 .... .****
0d.01.3 47 108.53 12.66 1.05 12416 49.41 26.11 378* 46.46 * 6.90 896 0.00 .... . 0.00 .... .****
0d.01.4 46 108.88 12.66 1.15 11179 48.72 26.20 402* 47.50 * 7.11 838 0.00 .... . 0.00 .... .****
0d.01.5 48 109.92 12.14 1.21 10082 49.58 26.30 399* 48.20 * 6.51 1081 0.00 .... . 0.00 .... .****
0d.01.6 47 100.03 8.84 1.00 10235 49.41 26.08 377* 41.78 7*.76 774 0.00 .... . 0.00 .... .****
0d.01.7 46 103.68 10.92 1.08 10441 48.54 26.80 357* 44.21
- 6.75 963 0.00 .... . 0.00 .... .****
0d.01.8 48 106.80 11.10 1.09 10314 49.06 26.54 390* 46.64
- 6.61 949 0.00 .... . 0.00 .... .****
0d.01.9 46 110.44 12.66 1.05 11429 46.98 26.48 370* 50.80
- 7.31 820 0.00 .... . 0.00 .... .****
0d.01.10 47 106.28 9.36 1.06 12105 49.24 25.46 373* 47.68
- 7.53 807 0.00 .... . 0.00 .... .****
0d.01.11 46 105.58 9.19 1.02 10463 49.06 26.77 360* 47.33
- 5.90 1078 0.00 .... . 0.00 .... .****
0d.01.12 42 108.18 11.44 1.09 11264 48.89 25.60 335* 47.85 * 7.18 902 0.00 .... . 0.00 .... .****
0d.01.13 43 103.85 11.10 1.06 11559 49.06 26.45 319* 43.69 * 7.40 757 0.00 .... . 0.00 .... .****
0d.01.14 42 105.93 10.92 1.08 11941 49.24 25.27 331* 45.77 * 7.77 757 0.00 .... . 0.00 .... .****
0d.01.15 45 117.02 12.14 1.04 12877 48.89 24.43 372* 56.00 * 7.70 802 0.00 .... . 0.00 .... .****
0d.01.16 44 102.64 10.40 1.03 11694 49.93 25.18 351* 42.30 * 8.15 684 0.00 .... . 0.00 .... .****
0d.01.17 44 104.72 11.10 1.13 10750 48.72 26.62 342* 44.90 * 7.14 830 0.00 .... . 0.00 .... .****
0d.01.18 43 104.89 10.06 1.03 12350 49.41 26.71 319* 45.42 * 6.37 987 0.00 .... . 0.00 .... .****
0d.01.19 44 109.57 10.23 1.05 10629 48.37 26.15 334* 50.97 * 7.29 810 0.00 .... . 0.00 .... .****
0d.01.20 47 111.13 12.31 1.10 11372 49.06 25.81 378* 49.76 * 6.92 912 0.00 .... . 0.00 .... .****
0d.01.21 44 110.26 11.79 1.06 13569 49.76 25.79 371* 48.72 * 6.82 828 0.00 .... . 0.00 .... .****
0d.01.22 44 106.97 11.62 1.06 10042 49.58 26.88 331* 45.77 * 6.07 1030 0.00 .... . 0.00 .... .****
Even though the aggregate is at 80% utilization it seems we have fragmentation issue****
I'll try to do full reallocate and check again ****
Thanks for the Help****
*Alon Z*****
*From:* Sebastian Goetze [mailto:spgoetze@gmail.com] *Sent:* יום ב 21 אוקטובר 2013 19:59 *To:* Alon Zeltser; toasters@teaparty.net *Subject:* Re: unexplained read operation****
I'm with Jeff... Looks like CP-Reads. (Need to read from disk to be able to calculate parity. Happens when unable to write 'full stripes', e.g. full aggregate)
Do statit and check your disk statistics:****
uread = User Reads, writes = User writes, cpread = CP Read as described above, gread+gwrite = guaranteed writes, like for disk rebuild****
Tell us what you find :-)
Sebastian****
On 10/21/2013 4:28 PM, Alon Zeltser wrote:****
Hello toasters****
I encounter rather strange situation ****
I have a system FAS2040 with 1 aggr of 23x450g raid group ontap version 8.1.3p2****
Every write operation being generated on this aggregate automatically caused a read operation ****
When I try to create 100% sequential write operation using tools such as sio_ntap or filersio ****
Instead of the expected 150Mb throughput I get 100MB write and 50MB read *
On another FAS2040 system using the same commands I get 100% write and no reads****
As you can see****
filersio asyncio_active 0 -r 0 64k 0 15g 60 1 /vol/test/testfile****
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s****
93% 0 0 0 5 26 1 *24242 118671 * 0 0 0s 99% 72% Fn 43% 0 0 5 0 0 24 0****
90% 0 0 0 4 4 1 *40959 102501 * 0 0 19 100% 89% Fn 43% 1 0 3 0 0 2 0****
86% 0 0 0 3 3 1 *23126 125022 * 0 0 0s 99% 69% : 59% 0 0 3 0 0 2 0****
93% 0 0 0 7 1 1 *28200 135740 * 0 0 0s 99% 97% Ff 48% 7 0 0 0 0 0 0****
92% 0 0 0 6 15 1 * 48764 115324 * 0 0 0s 99% 91% Ff 52% 0 0 6 0 0 14 0****
90% 0 0 0 6 16 1 * 36148 156492 * 0 0 0s 99% 83% Ff 56% 0 0 6 0 0 14 0****
95% 0 0 0 0 4 4 *32784 114555 * 0 0 0s 99% 77% Fs 48% 0 0 0 0 0 0 0****
92% 0 0 0 18 90 1*2 34931 104059 * 0 0 0s 99% 88% Fn 42% 0 0 18 0 0 79 0****
88% 0 0 0 8 3 1 *20463 124800 * 0 0 0s 100% 83% Fn 45% 5 0 3 0 0 2 0****
89% 0 0 0 2 2 1 *34050 112166 * 0 0 0s 99% 87% Fn 44% 0 0 2 0 0 1 0****
92% 0 0 0 3 7 1 *33025 112816 * 0 0 0s 99% 88% Fn 46% 0 0 3 0 0 5 0****
91% 0 0 0 3 3 1 *30929 130833 * 0 0 0s 100% 73% : 50% 0 0 3 0 0 2 0****
92% 0 0 0 7 24 2 * 36484 134260 * 0 0 0s 100% 98% Ff 50% 0 0 7 0 0 22 0****
86% 0 0 0 6 1 1 *38608 125140 * 0 0 0s 99% 90% Ff 44% 5 0 1 0 0 1 0****
93% 0 0 0 4 3 1 *28748 104544 * 0 0 0s 100% 75% Ff 43% 0 0 4 0 0 2 0****
87% 0 0 0 5 18 1 * 28128 158608 * 0 0 0s 100% 90% Fs 55% 0 0 5 0 0 17 0****
93% 0 0 0 2 3 34 * 33115 104980 * 0 0 0s 99% 85% Fn 41% 0 0 2 0 0 1 32****
84% 0 0 0 5 28 7 * 37185 109425 * 0 0 0s 100% 78% Fn 48% 0 0 5 0 0 21 0****
91% 0 0 0 9 3 1 *30224 118322 * 0 0 0s 99% 97% Fn 47% 7 0 2 0 0 1 0****
89% 0 0 0 2 3 4 *27649 118335 * 0 0 0s 99% 88% : 45% 0 0 2 0 0 1 0****
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s****
in out read write readwrite age hit time ty util in out in out****
88% 0 0 0 14 57 39 * 43284 13348*0 0 0 0s 100% 98% Ff 50% 0 0 14 0 0 52 0****
92% 0 0 0 2 6 16 *47936 109432* 0 0 0s 99% 86% Ff 46% 0 0 2 0 0 1 0****
88% 0 0 0 23 86 3 * 45340 141014* 0 0 0s 99% 73% Ff 55% 3 0 20 0 0 82 0****
92% 0 0 0 10 5 4 *21787 123980* 0 0 0s 99% 80% Fn 44% 7 0 3 0 0 1 0****
95% 0 0 0 2 4 3 *34751 98158 * 0 0 0s 99% 82% Fn 44% 0 0 2 0 0 1 0****
91% 0 0 0 1 3 1 *35022 118549 * 0 0 0s 100% 60% Fn 48% 0 0 1 0 0 0 0****
92% 0 0 0 87 3 1 *25455 123066* 0 0 0s 100% 80% Fn 47% 84 0 3 0 0 1 0****
83% 0 0 0 8 32 2 *36956 121303* 0 0 0s 100% 88% : 46% 0 0 8 0 0 30 0****
85% 0 0 0 5 2 1 *62064 140088 * 0 0 0s 100% 92% Ff 59% 5 0 0 0 0 0 0****
95% 0 0 0 3 3 1 *39352 117364 * 0 0 0s 99% 71% Ff 48% 0 0 3 0 0 2 0****
94% 0 0 0 3 3 1 *42008 136943 * 0 0 0s 99% 94% Fs 53% 0 0 3 0 0 2 0****
92% 0 0 0 0 2 7 *26147 111286 * 0 0 0s 99% 58% Fn 44% 0 0 0 0 0 0 0****
90% 0 0 0 3 3 1 *29567 110665 * 0 0 20 99% 87% Fn 44% 0 0 3 0 0 1 0****
90% 0 0 0 9 4 1 *40031 111273 * 0 0 16 99% 86% Fn 43% 5 0 4 0 0 2 0****
84% 0 0 0 0 1 1 *36760 105709 * 0 0 0s 100% 84% Fn 43% 0 0 0 0 0 0 0****
92% 0 0 0 3 3 1 *31033 137891 * 0 0 0s 100% 99% : 53% 0 0 3 0 0 2 0****
83% 0 0 0 4 6 1 *32991 140148 * 0 0 0s 100% 89% F 51% 0 0 4 0 0 5 0****
Any thoughts why this could happens ?****
Nothing else is running on this netapp except for this test ****
Thanks****
Alon Z****
_______________________________________________****
Toasters mailing list****
Toasters@teaparty.net****
http://www.teaparty.net/mailman/listinfo/toasters****
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters****
--
Gustatus Similis Pullus ****
--
Gustatus Similis Pullus ****
Toasters mailing listToasters@teaparty.nethttp://www.teaparty.net/mailman/listinfo/toasters
Depending on what your system is doing (how it's used by the applications) it can be argued that making the writes to an Aggregate optimal is 'A Good Thing(TM)'. ONTAP prioritises writes anyway and if those bottleneck, everything stops with B2B CP for long periods of time in the worst case. In that kind of situation it makes no difference whatsoever if the layout for read is very good on all the FlexVols
So optimising for the Free Space Allocation Algorithm is not a bad idea per se no matter what workload you have. It won't help reads if FlexVols are fragmented, that's true as most(?) ppl are aware
Note that the idea is that you first do reallocate -A on the Aggregate in Q. Once that's finished (can take a long time, >1 week if it's big and has many inodes in it) you do
aggr options aggr_name free_space_realloc on
which will then try to keep the free space (for future writes) as optimal as it can. But it does take both back-end IOPS and CPU processing from the system. And it might not even be able to keep up with the pace of your workload very well since it's a background task. If your workload contains stuff like creating, writing and then reading plus deleting millions of files over NFS, many TB a day like we have here, it is kind of hard for the automatic free_space_realloc to keep up and it *will* cause more READs from disk which tend to increase NFS READ ops latency.
As with many other clever optimisation things NetApp has come up with (a_sis, compression...), under some workloads it is a two edged sword:
Jeff Mohler wrote:
Not only wont optimize, but makes some things much worse.
/M
Sebastian Goetze wrote:
Reallocate -A would improve *write* performance, though... It improves the 'free space layout'. But I'd switch that one on at the aggregate options level, if your system, isn't too burdened already.
aggr options aggr_name free_space_realloc on
Sebastian
P.S. For read performance, you could also think about: vol options vol1 read_realloc space_optimized (e.g. for a volume with dedupe on)
On 10/23/2013 9:01 AM, Alon Zeltser wrote:
What I meant by "full reallocate" is to perform reallocate using the –f & -p flag on all the volumes on this aggr
see https://library.netapp.com/ecmdocs/ECMP1136573/html/GUID-E1789834-11E5-4748-...
I'm aware that running reallocate on the aggregate won't optimize the layout
Thanks for the heads up though
:)
Not only wont optimize, but makes some things much worse.
Glad you had that handy.
I consider vol reallocates at least a quarterly best practice.
Some people will disagree, as it invites the idea of a 'weakness'. I call it "Duh, its storage". :)
On Wed, Oct 23, 2013 at 12:01 AM, Alon Zeltser alonz@emet.co.il wrote:
What I meant by "full reallocate" is to perform reallocate using the –f & -p flag on all the volumes on this aggr****
see https://library.netapp.com/ecmdocs/ECMP1136573/html/GUID-E1789834-11E5-4748-...
I'm aware that running reallocate on the aggregate won't optimize the layout ****
Thanks for the heads up though ****
*Alon Z* ****
*From:* Jeff Mohler [mailto:speedtoys.racing@gmail.com] *Sent:* יום ג 22 אוקטובר 2013 20:46 *To:* Jordan Slingerland *Cc:* Alon Zeltser; toasters@teaparty.net
*Subject:* Re: unexplained read operation****
Indeed. :) It's still worth asking first.****
On Tue, Oct 22, 2013 at 10:42 AM, Jordan Slingerland < Jordan.Slingerland@independenthealth.com> wrote:****
I assume he means run a reallocation scan on all volumes within the aggregate, but NOT! on the aggregate. ****
*From:* toasters-bounces@teaparty.net [mailto: toasters-bounces@teaparty.net] *On Behalf Of *Jeff Mohler *Sent:* Tuesday, October 22, 2013 1:35 PM *To:* Alon Zeltser *Cc:* toasters@teaparty.net****
*Subject:* Re: unexplained read operation****
What do you mean "full reallocate".****
There is a good way, and a really really -bad- way.****
On Tue, Oct 22, 2013 at 1:23 AM, Alon Zeltser alonz@emet.co.il wrote:***
Looks like you nailed it ****
After looking at statit we can see the read operation indeed come from cp reads due to partial stripes****
1590.51 stripes written ****
1422.93 partial stripes ****
167.58 full stripes****
disk ut% xfers ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs /aggr0/plex0/rg0:****
0d.01.0 46 115.81 0.35 1.00 2500 53.57 29.94 263 *61.89 23*.12 232 0.00 .... . 0.00 .... .****
0d.01.1 50 117.89 0.35 1.00 4000 55.65 28.89 294 *61.89 23*.12 246 0.00 .... . 0.00 .... .****
0d.01.2 47 106.45 10.23 1.02 11383 50.80 27.06 338* 45.42 * 5.62 1193 0.00 .... . 0.00 .... .****
0d.01.3 47 108.53 12.66 1.05 12416 49.41 26.11 378* 46.46 * 6.90 896 0.00 .... . 0.00 .... .****
0d.01.4 46 108.88 12.66 1.15 11179 48.72 26.20 402* 47.50 * 7.11 838 0.00 .... . 0.00 .... .****
0d.01.5 48 109.92 12.14 1.21 10082 49.58 26.30 399* 48.20 * 6.51 1081 0.00 .... . 0.00 .... .****
0d.01.6 47 100.03 8.84 1.00 10235 49.41 26.08 377* 41.78 7*.76 774 0.00 .... . 0.00 .... .****
0d.01.7 46 103.68 10.92 1.08 10441 48.54 26.80 357* 44.21
- 6.75 963 0.00 .... . 0.00 .... .****
0d.01.8 48 106.80 11.10 1.09 10314 49.06 26.54 390* 46.64
- 6.61 949 0.00 .... . 0.00 .... .****
0d.01.9 46 110.44 12.66 1.05 11429 46.98 26.48 370* 50.80
- 7.31 820 0.00 .... . 0.00 .... .****
0d.01.10 47 106.28 9.36 1.06 12105 49.24 25.46 373* 47.68
- 7.53 807 0.00 .... . 0.00 .... .****
0d.01.11 46 105.58 9.19 1.02 10463 49.06 26.77 360* 47.33
- 5.90 1078 0.00 .... . 0.00 .... .****
0d.01.12 42 108.18 11.44 1.09 11264 48.89 25.60 335* 47.85 * 7.18 902 0.00 .... . 0.00 .... .****
0d.01.13 43 103.85 11.10 1.06 11559 49.06 26.45 319* 43.69 * 7.40 757 0.00 .... . 0.00 .... .****
0d.01.14 42 105.93 10.92 1.08 11941 49.24 25.27 331* 45.77 * 7.77 757 0.00 .... . 0.00 .... .****
0d.01.15 45 117.02 12.14 1.04 12877 48.89 24.43 372* 56.00 * 7.70 802 0.00 .... . 0.00 .... .****
0d.01.16 44 102.64 10.40 1.03 11694 49.93 25.18 351* 42.30 * 8.15 684 0.00 .... . 0.00 .... .****
0d.01.17 44 104.72 11.10 1.13 10750 48.72 26.62 342* 44.90 * 7.14 830 0.00 .... . 0.00 .... .****
0d.01.18 43 104.89 10.06 1.03 12350 49.41 26.71 319* 45.42 * 6.37 987 0.00 .... . 0.00 .... .****
0d.01.19 44 109.57 10.23 1.05 10629 48.37 26.15 334* 50.97 * 7.29 810 0.00 .... . 0.00 .... .****
0d.01.20 47 111.13 12.31 1.10 11372 49.06 25.81 378* 49.76 * 6.92 912 0.00 .... . 0.00 .... .****
0d.01.21 44 110.26 11.79 1.06 13569 49.76 25.79 371* 48.72 * 6.82 828 0.00 .... . 0.00 .... .****
0d.01.22 44 106.97 11.62 1.06 10042 49.58 26.88 331* 45.77 * 6.07 1030 0.00 .... . 0.00 .... .****
Even though the aggregate is at 80% utilization it seems we have fragmentation issue****
I'll try to do full reallocate and check again ****
Thanks for the Help****
*Alon Z*****
*From:* Sebastian Goetze [mailto:spgoetze@gmail.com] *Sent:* יום ב 21 אוקטובר 2013 19:59 *To:* Alon Zeltser; toasters@teaparty.net *Subject:* Re: unexplained read operation****
I'm with Jeff... Looks like CP-Reads. (Need to read from disk to be able to calculate parity. Happens when unable to write 'full stripes', e.g. full aggregate)
Do statit and check your disk statistics:****
uread = User Reads, writes = User writes, cpread = CP Read as described above, gread+gwrite = guaranteed writes, like for disk rebuild****
Tell us what you find :-)
Sebastian****
On 10/21/2013 4:28 PM, Alon Zeltser wrote:****
Hello toasters****
I encounter rather strange situation ****
I have a system FAS2040 with 1 aggr of 23x450g raid group ontap version 8.1.3p2****
Every write operation being generated on this aggregate automatically caused a read operation ****
When I try to create 100% sequential write operation using tools such as sio_ntap or filersio ****
Instead of the expected 150Mb throughput I get 100MB write and 50MB read *
On another FAS2040 system using the same commands I get 100% write and no reads****
As you can see****
filersio asyncio_active 0 -r 0 64k 0 15g 60 1 /vol/test/testfile****
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s****
93% 0 0 0 5 26 1 *24242 118671 * 0 0 0s 99% 72% Fn 43% 0 0 5 0 0 24 0****
90% 0 0 0 4 4 1 *40959 102501 * 0 0 19 100% 89% Fn 43% 1 0 3 0 0 2 0****
86% 0 0 0 3 3 1 *23126 125022 * 0 0 0s 99% 69% : 59% 0 0 3 0 0 2 0****
93% 0 0 0 7 1 1 *28200 135740 * 0 0 0s 99% 97% Ff 48% 7 0 0 0 0 0 0****
92% 0 0 0 6 15 1 * 48764 115324 * 0 0 0s 99% 91% Ff 52% 0 0 6 0 0 14 0****
90% 0 0 0 6 16 1 * 36148 156492 * 0 0 0s 99% 83% Ff 56% 0 0 6 0 0 14 0****
95% 0 0 0 0 4 4 *32784 114555 * 0 0 0s 99% 77% Fs 48% 0 0 0 0 0 0 0****
92% 0 0 0 18 90 1*2 34931 104059 * 0 0 0s 99% 88% Fn 42% 0 0 18 0 0 79 0****
88% 0 0 0 8 3 1 *20463 124800 * 0 0 0s 100% 83% Fn 45% 5 0 3 0 0 2 0****
89% 0 0 0 2 2 1 *34050 112166 * 0 0 0s 99% 87% Fn 44% 0 0 2 0 0 1 0****
92% 0 0 0 3 7 1 *33025 112816 * 0 0 0s 99% 88% Fn 46% 0 0 3 0 0 5 0****
91% 0 0 0 3 3 1 *30929 130833 * 0 0 0s 100% 73% : 50% 0 0 3 0 0 2 0****
92% 0 0 0 7 24 2 * 36484 134260 * 0 0 0s 100% 98% Ff 50% 0 0 7 0 0 22 0****
86% 0 0 0 6 1 1 *38608 125140 * 0 0 0s 99% 90% Ff 44% 5 0 1 0 0 1 0****
93% 0 0 0 4 3 1 *28748 104544 * 0 0 0s 100% 75% Ff 43% 0 0 4 0 0 2 0****
87% 0 0 0 5 18 1 * 28128 158608 * 0 0 0s 100% 90% Fs 55% 0 0 5 0 0 17 0****
93% 0 0 0 2 3 34 * 33115 104980 * 0 0 0s 99% 85% Fn 41% 0 0 2 0 0 1 32****
84% 0 0 0 5 28 7 * 37185 109425 * 0 0 0s 100% 78% Fn 48% 0 0 5 0 0 21 0****
91% 0 0 0 9 3 1 *30224 118322 * 0 0 0s 99% 97% Fn 47% 7 0 2 0 0 1 0****
89% 0 0 0 2 3 4 *27649 118335 * 0 0 0s 99% 88% : 45% 0 0 2 0 0 1 0****
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s****
in out read write readwrite age hit time ty util in out in out****
88% 0 0 0 14 57 39 * 43284 13348*0 0 0 0s 100% 98% Ff 50% 0 0 14 0 0 52 0****
92% 0 0 0 2 6 16 *47936 109432* 0 0 0s 99% 86% Ff 46% 0 0 2 0 0 1 0****
88% 0 0 0 23 86 3 * 45340 141014* 0 0 0s 99% 73% Ff 55% 3 0 20 0 0 82 0****
92% 0 0 0 10 5 4 *21787 123980* 0 0 0s 99% 80% Fn 44% 7 0 3 0 0 1 0****
95% 0 0 0 2 4 3 *34751 98158 * 0 0 0s 99% 82% Fn 44% 0 0 2 0 0 1 0****
91% 0 0 0 1 3 1 *35022 118549 * 0 0 0s 100% 60% Fn 48% 0 0 1 0 0 0 0****
92% 0 0 0 87 3 1 *25455 123066* 0 0 0s 100% 80% Fn 47% 84 0 3 0 0 1 0****
83% 0 0 0 8 32 2 *36956 121303* 0 0 0s 100% 88% : 46% 0 0 8 0 0 30 0****
85% 0 0 0 5 2 1 *62064 140088 * 0 0 0s 100% 92% Ff 59% 5 0 0 0 0 0 0****
95% 0 0 0 3 3 1 *39352 117364 * 0 0 0s 99% 71% Ff 48% 0 0 3 0 0 2 0****
94% 0 0 0 3 3 1 *42008 136943 * 0 0 0s 99% 94% Fs 53% 0 0 3 0 0 2 0****
92% 0 0 0 0 2 7 *26147 111286 * 0 0 0s 99% 58% Fn 44% 0 0 0 0 0 0 0****
90% 0 0 0 3 3 1 *29567 110665 * 0 0 20 99% 87% Fn 44% 0 0 3 0 0 1 0****
90% 0 0 0 9 4 1 *40031 111273 * 0 0 16 99% 86% Fn 43% 5 0 4 0 0 2 0****
84% 0 0 0 0 1 1 *36760 105709 * 0 0 0s 100% 84% Fn 43% 0 0 0 0 0 0 0****
92% 0 0 0 3 3 1 *31033 137891 * 0 0 0s 100% 99% : 53% 0 0 3 0 0 2 0****
83% 0 0 0 4 6 1 *32991 140148 * 0 0 0s 100% 89% F 51% 0 0 4 0 0 5 0****
Any thoughts why this could happens ?****
Nothing else is running on this netapp except for this test ****
Thanks****
Alon Z****
_______________________________________________****
Toasters mailing list****
Toasters@teaparty.net****
http://www.teaparty.net/mailman/listinfo/toasters****
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters****
--
Gustatus Similis Pullus ****
--
Gustatus Similis Pullus ****
-
Alon Zeltser -
Jeff Mohler -
Jordan Slingerland -
Michael Bergman -
Sebastian Goetze