Re-read is from:
Host system cache
Netapp system cache (or pam)
Direct will bypass host caching..yup.
On Sun, May 20, 2012 at 2:12 PM, Dan Burkland <
dburklan@nmdp.org> wrote:
In regards to the latest "iozone" results, are these more in the ball park
of what I should be seeing? Also why is the re-read throughput value
roughly 20x that of the initial read speed? Would this be caching on the
NFS client side or some sort of caching done by the PAM card on the 6080?
(Should I be running these tests with the "-I" or "Direct IO" argument to
bypass any possible local caching mechanisms?"
Thanks again!
Dan
On 5/19/12 5:32 PM, "Dan Burkland" <
dburklan@NMDP.ORG> wrote:
>I unmounted the NFS share and rebooted the box before running the same
>"iozone" command again. This time I let "iozone" run through all of its
>test (including the read-based ones)
>
>
>Run began: Sat May 19 16:46:27 2012
>
> File size set to 5242880 KB
> Record Size 1024 KB
> Excel chart generation enabled
> Command line used: iozone -s 5g -r 1m -t 16 -R -b
>/root/iozone_mn4s31063_2012-05-d.csv -F tf1 t1 t2 t3 t4 t5 t6 t7 t8 t9 t10
>t11 t12 t13 t14 t15 t16 t17 t18
> Output is in Kbytes/sec
> Time Resolution = 0.000001 seconds.
> Processor cache size set to 1024 Kbytes.
> Processor cache line size set to 32 bytes.
> File stride size set to 17 * record size.
> Throughput test with 16 processes
> Each process writes a 5242880 Kbyte file in 1024 Kbyte records
>
> Children see throughput for 16 initial writers = 349500.55
>KB/sec
> Parent sees throughput for 16 initial writers = 173837.26
>KB/sec
> Min throughput per process = 21147.24
>KB/sec
> Max throughput per process = 22701.06
>KB/sec
> Avg throughput per process = 21843.78
>KB/sec
> Min xfer = 4884480.00 KB
>
> Children see throughput for 16 rewriters = 372333.90
>KB/sec
> Parent sees throughput for 16 rewriters = 179256.38
>KB/sec
> Min throughput per process = 22495.20
>KB/sec
> Max throughput per process = 24418.89
>KB/sec
> Avg throughput per process = 23270.87
>KB/sec
> Min xfer = 4830208.00 KB
>
> Children see throughput for 16 readers = 440115.98
>KB/sec
> Parent sees throughput for 16 readers = 439993.44
>KB/sec
> Min throughput per process = 26406.17
>KB/sec
> Max throughput per process = 28724.05
>KB/sec
> Avg throughput per process = 27507.25
>KB/sec
> Min xfer = 4819968.00 KB
>
> Children see throughput for 16 re-readers = 8953522.06
>KB/sec
> Parent sees throughput for 16 re-readers = 8930475.33
>KB/sec
> Min throughput per process = 408033.34
>KB/sec
> Max throughput per process = 671821.62
>KB/sec
> Avg throughput per process = 559595.13
>KB/sec
> Min xfer = 3186688.00 KB
>
> Children see throughput for 16 reverse readers = 5543829.37
>KB/sec
> Parent sees throughput for 16 reverse readers = 5425986.47
>KB/sec
> Min throughput per process = 15684.29
>KB/sec
> Max throughput per process = 2261884.25
>KB/sec
> Avg throughput per process = 346489.34
>KB/sec
> Min xfer = 36864.00 KB
>
> Children see throughput for 16 stride readers = 16532117.19
>KB/sec
> Parent sees throughput for 16 stride readers = 16272131.55
>KB/sec
> Min throughput per process = 257097.92
>KB/sec
> Max throughput per process = 2256125.75
>KB/sec
> Avg throughput per process = 1033257.32
>KB/sec
> Min xfer = 602112.00 KB
>
> Children see throughput for 16 random readers = 17297437.81
>KB/sec
> Parent sees throughput for 16 random readers = 16871312.92
>KB/sec
> Min throughput per process = 320909.25
>KB/sec
> Max throughput per process = 2083737.75
>KB/sec
> Avg throughput per process = 1081089.86
>KB/sec
> Min xfer = 826368.00 KB
>
> Children see throughput for 16 mixed workload = 10747970.97
>KB/sec
> Parent sees throughput for 16 mixed workload = 112898.07
>KB/sec
> Min throughput per process = 54960.62
>KB/sec
> Max throughput per process = 1991637.38
>KB/sec
> Avg throughput per process = 671748.19
>KB/sec
> Min xfer = 145408.00 KB
>
> Children see throughput for 16 random writers = 358103.29
>KB/sec
> Parent sees throughput for 16 random writers = 166805.09
>KB/sec
> Min throughput per process = 21263.60
>KB/sec
> Max throughput per process = 22942.70
>KB/sec
> Avg throughput per process = 22381.46
>KB/sec
> Min xfer = 4859904.00 KB
>
> Children see throughput for 16 pwrite writers = 325666.64
>KB/sec
> Parent sees throughput for 16 pwrite writers = 177771.50
>KB/sec
> Min throughput per process = 19902.90
>KB/sec
> Max throughput per process = 20863.29
>KB/sec
> Avg throughput per process = 20354.17
>KB/sec
> Min xfer = 5008384.00 KB
>
> Children see throughput for 16 pread readers = 445021.47
>KB/sec
> Parent sees throughput for 16 pread readers = 444618.25
>KB/sec
> Min throughput per process = 26932.47
>KB/sec
> Max throughput per process = 28361.61
>KB/sec
> Avg throughput per process = 27813.84
>KB/sec
> Min xfer = 4981760.00 KB
>
>
>
>"Throughput report Y-axis is type of test X-axis is number of processes"
>"Record size = 1024 Kbytes "
>"Output is in Kbytes/sec"
>
>" Initial write " 349500.55
>
>" Rewrite " 372333.90
>
>" Read " 440115.98
>
>" Re-read " 8953522.06
>
>" Reverse Read " 5543829.37
>
>" Stride read " 16532117.19
>
>" Random read " 17297437.81
>
>" Mixed workload " 10747970.97
>
>" Random write " 358103.29
>
>" Pwrite " 325666.64
>
>" Pread " 445021.47
>
>
>Regards,
>
>Dan
>
>
>On 5/19/12 4:48 PM, "Jeff Mother" <
speedtoys.racing@gmail.com> wrote:
>
>>You're now approaching storage write saturation for your box on writes at
>>that rate.
>>
>>Pull reads now.
>>
>>
>>
>>Sent from my iPhone
>>
>>On May 19, 2012, at 2:43 PM, Dan Burkland <
dburklan@NMDP.ORG> wrote:
>>
>>> Here are the IOZone results:
>>>
>>> Run began: Sat May 19 16:22:46 2012
>>>
>>> File size set to 5242880 KB
>>> Record Size 1024 KB
>>> Excel chart generation enabled
>>> Command line used: iozone -s 5g -r 1m -t 16 -R -b
>>> /root/iozone_mn4s31063_2012-05-d.csv -F tf1 t1 t2 t3 t4 t5 t6 t7 t8 t9
>>>t10
>>> t11 t12 t13 t14 t15 t16 t17 t18
>>> Output is in Kbytes/sec
>>> Time Resolution = 0.000001 seconds.
>>> Processor cache size set to 1024 Kbytes.
>>> Processor cache line size set to 32 bytes.
>>> File stride size set to 17 * record size.
>>> Throughput test with 16 processes
>>> Each process writes a 5242880 Kbyte file in 1024 Kbyte records
>>>
>>> Children see throughput for 16 initial writers = 371306.91
>>>KB/sec
>>> Parent sees throughput for 16 initial writers = 167971.82 KB/sec
>>> Min throughput per process = 21901.84 KB/sec
>>> Max throughput per process = 25333.62 KB/sec
>>> Avg throughput per process = 23206.68 KB/sec
>>> Min xfer = 4533248.00 KB
>>>
>>> Children see throughput for 16 rewriters = 350486.11 KB/sec
>>> Parent sees throughput for 16 rewriters = 176947.47 KB/sec
>>> Min throughput per process = 21154.26 KB/sec
>>> Max throughput per process = 23011.69 KB/sec
>>> Avg throughput per process = 21905.38 KB/sec
>>> Min xfer = 4819968.00 KB
>>>
>>> 362MB/s looks quite a bit higher however can somebody validate that I
>>>am
>>> reading these results correctly? Should I also run "iozone" with the -a
>>> (auto) option for good measure?
>>>
>>> Thanks again for all of your responses, I greatly appreciate it!
>>>
>>>
>>> Dan
>>>
>>>
>>> On 5/19/12 4:36 PM, "Dan Burkland" <
dburklan@NMDP.ORG> wrote:
>>>
>>>> Jeff Mother - Which specific setting are you referring to?
>>>>
>>>> I installed iozone on my test machine and am currently running the
>>>> following iozone command on it:
>>>>
>>>> iozone -s 5g -r 1m -t 16 -R -b /root/iozone_testserver_2012-05-d.csv
>>>>-F
>>>> tf1 t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 t12 t13 t14 t15 t16 t17 t18
>>>>
>>>> I'll post the results once it is finished
>>>>
>>>> Dan
>>>>
>>>>
>>>>
>>>> On 5/19/12 2:44 PM, "Jeff Mother" <
speedtoys.racing@gmail.com> wrote:
>>>>
>>>>> Easy one.
>>>>>
>>>>> If it went down in half, adjust your kernel tcp slot count.
>>>>>
>>>>>
>>>>>
>>>>> Sent from my iPhone
>>>>>
>>>>> On May 19, 2012, at 11:46 AM, Dan Burkland <
dburklan@NMDP.ORG> wrote:
>>>>>
>>>>>> I know dd isn't the best tool since it is a single threaded
>>>>>>application
>>>>>> and in no way represents the workload that Oracle will impose.
>>>>>>However,
>>>>>> I
>>>>>> thought it would still give me a decent ballpark figure regarding
>>>>>> throughput. I tried a block size of 64k, 128k, and 1M (just to see)
>>>>>>and
>>>>>> got a bit more promising results:
>>>>>>
>>>>>> # dd if=/dev/zero of=/mnt/testfile bs=1M count=5120
>>>>>> 5120+0 records in
>>>>>> 5120+0 records out
>>>>>> 5368709120 bytes (5.4 GB) copied, 26.6878 seconds, 201 MB/s
>>>>>>
>>>>>> If I run two of these dd sessions at once the throughput figure
>>>>>>above
>>>>>> gets
>>>>>> cut in half (each dd session reports it creates the file at around
>>>>>> 100MB/s).
>>>>>>
>>>>>> As far as the switch goes, I have not checked it yet however I did
>>>>>> notice
>>>>>> that flow control is set to full on the 6080 10GbE interfaces. We
>>>>>>are
>>>>>> also
>>>>>> running Jumbo Frames on all of the involved equipment.
>>>>>>
>>>>>> As far as the RHEL OS tweaks go, here are the settings that I have
>>>>>> changed
>>>>>> on the system:
>>>>>>
>>>>>> ###
>>>>>> /etc/sysctl.conf:
>>>>>>
>>>>>> # 10GbE Kernel Parameters
>>>>>> net.core.rmem_default = 262144
>>>>>> net.core.rmem_max = 16777216
>>>>>> net.core.wmem_default = 262144
>>>>>> net.core.wmem_max = 16777216