On 5/19/12 5:32 PM, "Dan Burkland" <
dburklan@NMDP.ORG> wrote:
>I unmounted the NFS share and rebooted the box before running the same
>"iozone" command again. This time I let "iozone" run through all of its
>test (including the read-based ones)
>
>
>Run began: Sat May 19 16:46:27 2012
>
> File size set to 5242880 KB
> Record Size 1024 KB
> Excel chart generation enabled
> Command line used: iozone -s 5g -r 1m -t 16 -R -b
>/root/iozone_mn4s31063_2012-05-d.csv -F tf1 t1 t2 t3 t4 t5 t6 t7 t8 t9 t10
>t11 t12 t13 t14 t15 t16 t17 t18
> Output is in Kbytes/sec
> Time Resolution = 0.000001 seconds.
> Processor cache size set to 1024 Kbytes.
> Processor cache line size set to 32 bytes.
> File stride size set to 17 * record size.
> Throughput test with 16 processes
> Each process writes a 5242880 Kbyte file in 1024 Kbyte records
>
> Children see throughput for 16 initial writers = 349500.55
>KB/sec
> Parent sees throughput for 16 initial writers = 173837.26
>KB/sec
> Min throughput per process = 21147.24
>KB/sec
> Max throughput per process = 22701.06
>KB/sec
> Avg throughput per process = 21843.78
>KB/sec
> Min xfer = 4884480.00 KB
>
> Children see throughput for 16 rewriters = 372333.90
>KB/sec
> Parent sees throughput for 16 rewriters = 179256.38
>KB/sec
> Min throughput per process = 22495.20
>KB/sec
> Max throughput per process = 24418.89
>KB/sec
> Avg throughput per process = 23270.87
>KB/sec
> Min xfer = 4830208.00 KB
>
> Children see throughput for 16 readers = 440115.98
>KB/sec
> Parent sees throughput for 16 readers = 439993.44
>KB/sec
> Min throughput per process = 26406.17
>KB/sec
> Max throughput per process = 28724.05
>KB/sec
> Avg throughput per process = 27507.25
>KB/sec
> Min xfer = 4819968.00 KB
>
> Children see throughput for 16 re-readers = 8953522.06
>KB/sec
> Parent sees throughput for 16 re-readers = 8930475.33
>KB/sec
> Min throughput per process = 408033.34
>KB/sec
> Max throughput per process = 671821.62
>KB/sec
> Avg throughput per process = 559595.13
>KB/sec
> Min xfer = 3186688.00 KB
>
> Children see throughput for 16 reverse readers = 5543829.37
>KB/sec
> Parent sees throughput for 16 reverse readers = 5425986.47
>KB/sec
> Min throughput per process = 15684.29
>KB/sec
> Max throughput per process = 2261884.25
>KB/sec
> Avg throughput per process = 346489.34
>KB/sec
> Min xfer = 36864.00 KB
>
> Children see throughput for 16 stride readers = 16532117.19
>KB/sec
> Parent sees throughput for 16 stride readers = 16272131.55
>KB/sec
> Min throughput per process = 257097.92
>KB/sec
> Max throughput per process = 2256125.75
>KB/sec
> Avg throughput per process = 1033257.32
>KB/sec
> Min xfer = 602112.00 KB
>
> Children see throughput for 16 random readers = 17297437.81
>KB/sec
> Parent sees throughput for 16 random readers = 16871312.92
>KB/sec
> Min throughput per process = 320909.25
>KB/sec
> Max throughput per process = 2083737.75
>KB/sec
> Avg throughput per process = 1081089.86
>KB/sec
> Min xfer = 826368.00 KB
>
> Children see throughput for 16 mixed workload = 10747970.97
>KB/sec
> Parent sees throughput for 16 mixed workload = 112898.07
>KB/sec
> Min throughput per process = 54960.62
>KB/sec
> Max throughput per process = 1991637.38
>KB/sec
> Avg throughput per process = 671748.19
>KB/sec
> Min xfer = 145408.00 KB
>
> Children see throughput for 16 random writers = 358103.29
>KB/sec
> Parent sees throughput for 16 random writers = 166805.09
>KB/sec
> Min throughput per process = 21263.60
>KB/sec
> Max throughput per process = 22942.70
>KB/sec
> Avg throughput per process = 22381.46
>KB/sec
> Min xfer = 4859904.00 KB
>
> Children see throughput for 16 pwrite writers = 325666.64
>KB/sec
> Parent sees throughput for 16 pwrite writers = 177771.50
>KB/sec
> Min throughput per process = 19902.90
>KB/sec
> Max throughput per process = 20863.29
>KB/sec
> Avg throughput per process = 20354.17
>KB/sec
> Min xfer = 5008384.00 KB
>
> Children see throughput for 16 pread readers = 445021.47
>KB/sec
> Parent sees throughput for 16 pread readers = 444618.25
>KB/sec
> Min throughput per process = 26932.47
>KB/sec
> Max throughput per process = 28361.61
>KB/sec
> Avg throughput per process = 27813.84
>KB/sec
> Min xfer = 4981760.00 KB
>
>
>
>"Throughput report Y-axis is type of test X-axis is number of processes"
>"Record size = 1024 Kbytes "
>"Output is in Kbytes/sec"
>
>" Initial write " 349500.55
>
>" Rewrite " 372333.90
>
>" Read " 440115.98
>
>" Re-read " 8953522.06
>
>" Reverse Read " 5543829.37
>
>" Stride read " 16532117.19
>
>" Random read " 17297437.81
>
>" Mixed workload " 10747970.97
>
>" Random write " 358103.29
>
>" Pwrite " 325666.64
>
>" Pread " 445021.47
>
>
>Regards,
>
>Dan
>
>
>On 5/19/12 4:48 PM, "Jeff Mother" <
speedtoys.racing@gmail.com> wrote:
>
>>You're now approaching storage write saturation for your box on writes at
>>that rate.
>>
>>Pull reads now.
>>
>>
>>
>>Sent from my iPhone
>>
>>On May 19, 2012, at 2:43 PM, Dan Burkland <
dburklan@NMDP.ORG> wrote:
>>
>>> Here are the IOZone results:
>>>
>>> Run began: Sat May 19 16:22:46 2012
>>>
>>> File size set to 5242880 KB
>>> Record Size 1024 KB
>>> Excel chart generation enabled
>>> Command line used: iozone -s 5g -r 1m -t 16 -R -b
>>> /root/iozone_mn4s31063_2012-05-d.csv -F tf1 t1 t2 t3 t4 t5 t6 t7 t8 t9
>>>t10
>>> t11 t12 t13 t14 t15 t16 t17 t18
>>> Output is in Kbytes/sec
>>> Time Resolution = 0.000001 seconds.
>>> Processor cache size set to 1024 Kbytes.
>>> Processor cache line size set to 32 bytes.
>>> File stride size set to 17 * record size.
>>> Throughput test with 16 processes
>>> Each process writes a 5242880 Kbyte file in 1024 Kbyte records
>>>
>>> Children see throughput for 16 initial writers = 371306.91
>>>KB/sec
>>> Parent sees throughput for 16 initial writers = 167971.82 KB/sec
>>> Min throughput per process = 21901.84 KB/sec
>>> Max throughput per process = 25333.62 KB/sec
>>> Avg throughput per process = 23206.68 KB/sec
>>> Min xfer = 4533248.00 KB
>>>
>>> Children see throughput for 16 rewriters = 350486.11 KB/sec
>>> Parent sees throughput for 16 rewriters = 176947.47 KB/sec
>>> Min throughput per process = 21154.26 KB/sec
>>> Max throughput per process = 23011.69 KB/sec
>>> Avg throughput per process = 21905.38 KB/sec
>>> Min xfer = 4819968.00 KB
>>>
>>> 362MB/s looks quite a bit higher however can somebody validate that I
>>>am
>>> reading these results correctly? Should I also run "iozone" with the -a
>>> (auto) option for good measure?
>>>
>>> Thanks again for all of your responses, I greatly appreciate it!
>>>
>>>
>>> Dan
>>>
>>>
>>> On 5/19/12 4:36 PM, "Dan Burkland" <
dburklan@NMDP.ORG> wrote:
>>>
>>>> Jeff Mother - Which specific setting are you referring to?
>>>>
>>>> I installed iozone on my test machine and am currently running the
>>>> following iozone command on it:
>>>>
>>>> iozone -s 5g -r 1m -t 16 -R -b /root/iozone_testserver_2012-05-d.csv
>>>>-F
>>>> tf1 t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 t12 t13 t14 t15 t16 t17 t18
>>>>
>>>> I'll post the results once it is finished
>>>>
>>>> Dan
>>>>
>>>>
>>>>
>>>> On 5/19/12 2:44 PM, "Jeff Mother" <
speedtoys.racing@gmail.com> wrote:
>>>>
>>>>> Easy one.
>>>>>
>>>>> If it went down in half, adjust your kernel tcp slot count.
>>>>>
>>>>>
>>>>>
>>>>> Sent from my iPhone
>>>>>
>>>>> On May 19, 2012, at 11:46 AM, Dan Burkland <
dburklan@NMDP.ORG> wrote:
>>>>>
>>>>>> I know dd isn't the best tool since it is a single threaded
>>>>>>application
>>>>>> and in no way represents the workload that Oracle will impose.
>>>>>>However,
>>>>>> I
>>>>>> thought it would still give me a decent ballpark figure regarding
>>>>>> throughput. I tried a block size of 64k, 128k, and 1M (just to see)
>>>>>>and
>>>>>> got a bit more promising results:
>>>>>>
>>>>>> # dd if=/dev/zero of=/mnt/testfile bs=1M count=5120
>>>>>> 5120+0 records in
>>>>>> 5120+0 records out
>>>>>> 5368709120 bytes (5.4 GB) copied, 26.6878 seconds, 201 MB/s
>>>>>>
>>>>>> If I run two of these dd sessions at once the throughput figure
>>>>>>above
>>>>>> gets
>>>>>> cut in half (each dd session reports it creates the file at around
>>>>>> 100MB/s).
>>>>>>
>>>>>> As far as the switch goes, I have not checked it yet however I did
>>>>>> notice
>>>>>> that flow control is set to full on the 6080 10GbE interfaces. We
>>>>>>are
>>>>>> also
>>>>>> running Jumbo Frames on all of the involved equipment.
>>>>>>
>>>>>> As far as the RHEL OS tweaks go, here are the settings that I have
>>>>>> changed
>>>>>> on the system:
>>>>>>
>>>>>> ###
>>>>>> /etc/sysctl.conf:
>>>>>>
>>>>>> # 10GbE Kernel Parameters
>>>>>> net.core.rmem_default = 262144
>>>>>> net.core.rmem_max = 16777216
>>>>>> net.core.wmem_default = 262144
>>>>>> net.core.wmem_max = 16777216
>>>>>> net.ipv4.tcp_rmem =
4096 262144 16777216
>>>>>> net.ipv4.tcp_wmem =
4096 262144 16777216
>>>>>> net.ipv4.tcp_window_scaling = 1
>>>>>> net.ipv4.tcp_syncookies = 0
>>>>>> net.ipv4.tcp_timestamps = 0
>>>>>> net.ipv4.tcp_sack = 0
>>>>>> #
>>>>>>
>>>>>> ###
>>>>>>
>>>>>> ###
>>>>>> /etc/modprobe.d/sunrpc.conf:
>>>>>>
>>>>>>
>>>>>> options sunrpc tcp_slot_table_entries=128
>>>>>>
>>>>>> ###
>>>>>>
>>>>>>
>>>>>> ###
>>>>>> Mount options for the NetApp test NFS share:
>>>>>>
>>>>>>
>>>>>>
>>>>>>rw,vers=3,rsize=65536,wsize=65536,hard,proto=tcp,timeo=600,retrans=2,
>>>>>>s
>>>>>>ec
>>>>>> =
>>>>>> sy
>>>>>> s
>>>>>>
>>>>>> ###
>>>>>>
>>>>>> Thanks again for all of your quick and detailed responses!
>>>>>>
>>>>>>
>>>>>> Dan
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 5/19/12 1:08 PM, "Robert McDermott" <
rmcdermo@fhcrc.org> wrote:
>>>>>>
>>>>>>> Your block size is only 1K; try increasing the block size and the
>>>>>>> throughput will increase. 1K IOs would generate a lot of IOPs with
>>>>>>> very
>>>>>>> little throughput.
>>>>>>>
>>>>>>> -Robert
>>>>>>>
>>>>>>> Sent from my iPhone
>>>>>>>
>>>>>>> On May 19, 2012, at 10:48, Dan Burkland <
dburklan@NMDP.ORG> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> My company just bought some Intel x520 10GbE cards which I
>>>>>>>>recently
>>>>>>>> installed into our Oracle EBS database servers (IBM 3850 X5s
>>>>>>>>running
>>>>>>>> RHEL
>>>>>>>> 5.8). As the "linux guy" I have been tasked with getting these
>>>>>>>> servers
>>>>>>>> to
>>>>>>>> communicate with our NetApp 6080s via NFS over the new 10GbE
>>>>>>>>links. I
>>>>>>>> have
>>>>>>>> got everything working however ever after tuning the RHEL kernel I
>>>>>>>>am
>>>>>>>> only
>>>>>>>> getting 160MB/s writes using the "dd if=/dev/zero of=/mnt/testfile
>>>>>>>> bs=1024
>>>>>>>> count=5242880" command. For you folks that run 10GbE to your
>>>>>>>> toasters,
>>>>>>>> what write speeds are you seeing from your 10GbE connected
>>>>>>>>servers?
>>>>>>>> Did
>>>>>>>> you have to do any tuning in order to get the best results
>>>>>>>>possible?
>>>>>>>> If
>>>>>>>> so
>>>>>>>> what did you change?
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> Dan
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Toasters mailing list
>>>>>>>>
Toasters@teaparty.net
>>>>>>>>
http://www.teaparty.net/mailman/listinfo/toasters
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Toasters mailing list
>>>>>>
Toasters@teaparty.net
>>>>>>
http://www.teaparty.net/mailman/listinfo/toasters
>>>>
>>>
>>>
>>> _______________________________________________
>>> Toasters mailing list
>>>
Toasters@teaparty.net
>>>
http://www.teaparty.net/mailman/listinfo/toasters
>
>
>_______________________________________________
>Toasters mailing list
>
Toasters@teaparty.net
>
http://www.teaparty.net/mailman/listinfo/toasters
_______________________________________________
Toasters mailing list
Toasters@teaparty.net
http://www.teaparty.net/mailman/listinfo/toasters