Hello,
i'll try to improve our I/0 Perfomance. We have a raidgroup with 16 disks (SAS), 1 aggregate and 1 volume (dedup enabled). The volume is the storage for 100 VM's on the VMware Cluser (access via NFS). Does it make sense to increase the raidgroup? 90% Percent of the Disk I/O are Read Ops, so i'll buy a PAM Card for our FAS3240,too.
And what's your Experience with the Raidgroupsize of a Raidgroup with SATA Disks (now: 11+1)?
Thanks and greets !
Steffen
If you're considering getting a cache adapter I'd hold off on messing with the RAID config until afterwards - tuning for writes is different than tuning for IO in general since you're looking for free space and generally the more contiguous the better.
Have you done a reallocate measurement, are your VMDK files all aligned?
On 09/25/2012 04:19 AM, Steffen Knauf wrote:
Hello,
i'll try to improve our I/0 Perfomance. We have a raidgroup with 16 disks (SAS), 1 aggregate and 1 volume (dedup enabled). The volume is the storage for 100 VM's on the VMware Cluser (access via NFS). Does it make sense to increase the raidgroup? 90% Percent of the Disk I/O are Read Ops, so i'll buy a PAM Card for our FAS3240,too.
And what's your Experience with the Raidgroupsize of a Raidgroup with SATA Disks (now: 11+1)?
Thanks and greets !
Steffen
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Please be advised that this email may contain confidential information. If you are not the intended recipient, please notify us by email by replying to the sender and delete this message. The sender disclaims that the content of this email constitutes an offer to enter into, or the acceptance of, any agreement; provided that the foregoing does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment.
It ssems that we have a alignment problem, too:
nfsstat:
Misaligned Read request stats
BIN-0 BIN-1 BIN-2 BIN-3 BIN-4 BIN-5 BIN-6 BIN-7
1096332359 1984142147 16821356 3951254 469217191 6145276 94494489 245873608
Misaligned Write request stats
BIN-0 BIN-1 BIN-2 BIN-3 BIN-4 BIN-5 BIN-6 BIN-7
252471989 1287671452 52026962 32535853 278299197 40175418 38565555 692647941
I'll do a reallocate measurement tomorrow.
greets
Steffen
Von: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] Im Auftrag von Jeremy Page Gesendet: Dienstag, 25. September 2012 14:23 An: toasters@teaparty.net Betreff: Re: Raidgroupsize and I/O Performance
If you're considering getting a cache adapter I'd hold off on messing with the RAID config until afterwards - tuning for writes is different than tuning for IO in general since you're looking for free space and generally the more contiguous the better.
Have you done a reallocate measurement, are your VMDK files all aligned?
On 09/25/2012 04:19 AM, Steffen Knauf wrote:
Hello,
i'll try to improve our I/0 Perfomance. We have a raidgroup with 16 disks (SAS), 1 aggregate and 1 volume (dedup enabled). The volume is the storage for 100 VM's on the VMware Cluser (access via NFS). Does it make sense to increase the raidgroup? 90% Percent of the Disk I/O are Read Ops, so i'll buy a PAM Card for our FAS3240,too.
And what's your Experience with the Raidgroupsize of a Raidgroup with SATA Disks (now: 11+1)?
Thanks and greets !
Steffen
_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Please be advised that this email may contain confidential information. If you are not the intended recipient, please notify us by email by replying to the sender and delete this message. The sender disclaims that the content of this email constitutes an offer to enter into, or the acceptance of, any agreement; provided that the foregoing does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment.
What will that accomplish? Alignment is first order pain you need to resolve.
Sent from my iPhone
On Sep 26, 2012, at 8:16 AM, "Steffen Knauf" sknauf@chipxonio.de wrote:
It ssems that we have a alignment problem, too:
nfsstat: Misaligned Read request stats BIN-0 BIN-1 BIN-2 BIN-3 BIN-4 BIN-5 BIN-6 BIN-7 1096332359 1984142147 16821356 3951254 469217191 6145276 94494489 245873608 Misaligned Write request stats BIN-0 BIN-1 BIN-2 BIN-3 BIN-4 BIN-5 BIN-6 BIN-7 252471989 1287671452 52026962 32535853 278299197 40175418 38565555 692647941
I'll do a reallocate measurement tomorrow.
greets
Steffen
Von: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] Im Auftrag von Jeremy Page Gesendet: Dienstag, 25. September 2012 14:23 An: toasters@teaparty.net Betreff: Re: Raidgroupsize and I/O Performance
If you're considering getting a cache adapter I'd hold off on messing with the RAID config until afterwards - tuning for writes is different than tuning for IO in general since you're looking for free space and generally the more contiguous the better.
Have you done a reallocate measurement, are your VMDK files all aligned?
On 09/25/2012 04:19 AM, Steffen Knauf wrote: Hello,
i'll try to improve our I/0 Perfomance. We have a raidgroup with 16 disks (SAS), 1 aggregate and 1 volume (dedup enabled). The volume is the storage for 100 VM's on the VMware Cluser (access via NFS). Does it make sense to increase the raidgroup? 90% Percent of the Disk I/O are Read Ops, so i'll buy a PAM Card for our FAS3240,too.
And what's your Experience with the Raidgroupsize of a Raidgroup with SATA Disks (now: 11+1)?
Thanks and greets !
Steffen
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Please be advised that this email may contain confidential information. If you are not the intended recipient, please notify us by email by replying to the sender and delete this message. The sender disclaims that the content of this email constitutes an offer to enter into, or the acceptance of, any agreement; provided that the foregoing does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment. _______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Yes that's true. A strange thing is that VSC (4.0) and mbrscan show me different results:
/opt/ontap # ./mbrscan /vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk
mbrtools esxi version 1.0
--------------------
/vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk p1 (Linux) lba:64 offset:32768 aligned:Yes
/vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk p2 (swap) lba:401640 offset:205639680 aligned:Yes
/vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk p3 (Linux) lba:16771888 offset:8587206656 aligned:Yes
I think i can delete the backup file?:
[Counter=2891433], Filename=vol_vm1/openx-test01/openx-test01-flat.vmdk-mbralign-backup
I rescan the datastores via VSC and get the result that openx-test01 is misaligned
And some mbrscans are not possible:
./mbrscan /vmfs/volumes/chip4-vmware_data_nfs01/openx-test03/openx-test03-flat.vmdk
Failed to open /vmfs/volumes/chip4-vmware_data_nfs01/openx-test03/openx-test03-flat.vmdk - [Device or resource busy]
Thanks for your help and have a nice weekend. I'm on holiday for 1 Week, so my response could be delayed.....
greets
Steffen
Von: Jeff Mohler [mailto:speedtoys.racing@gmail.com] Gesendet: Mittwoch, 26. September 2012 17:58 An: Steffen Knauf Cc: Jeremy Page; toasters@teaparty.net Betreff: Re: AW: Raidgroupsize and I/O Performance
What will that accomplish? Alignment is first order pain you need to resolve.
Sent from my iPhone
On Sep 26, 2012, at 8:16 AM, "Steffen Knauf" sknauf@chipxonio.de wrote:
It ssems that we have a alignment problem, too:
nfsstat:
Misaligned Read request stats
BIN-0 BIN-1 BIN-2 BIN-3 BIN-4 BIN-5 BIN-6 BIN-7
1096332359 1984142147 16821356 3951254 469217191 6145276 94494489 245873608
Misaligned Write request stats
BIN-0 BIN-1 BIN-2 BIN-3 BIN-4 BIN-5 BIN-6 BIN-7
252471989 1287671452 52026962 32535853 278299197 40175418 38565555 692647941
I'll do a reallocate measurement tomorrow.
greets
Steffen
Von: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] Im Auftrag von Jeremy Page Gesendet: Dienstag, 25. September 2012 14:23 An: toasters@teaparty.net Betreff: Re: Raidgroupsize and I/O Performance
If you're considering getting a cache adapter I'd hold off on messing with the RAID config until afterwards - tuning for writes is different than tuning for IO in general since you're looking for free space and generally the more contiguous the better.
Have you done a reallocate measurement, are your VMDK files all aligned?
On 09/25/2012 04:19 AM, Steffen Knauf wrote:
Hello,
i'll try to improve our I/0 Perfomance. We have a raidgroup with 16 disks (SAS), 1 aggregate and 1 volume (dedup enabled). The volume is the storage for 100 VM's on the VMware Cluser (access via NFS). Does it make sense to increase the raidgroup? 90% Percent of the Disk I/O are Read Ops, so i'll buy a PAM Card for our FAS3240,too.
And what's your Experience with the Raidgroupsize of a Raidgroup with SATA Disks (now: 11+1)?
Thanks and greets !
Steffen
_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Please be advised that this email may contain confidential information. If you are not the intended recipient, please notify us by email by replying to the sender and delete this message. The sender disclaims that the content of this email constitutes an offer to enter into, or the acceptance of, any agreement; provided that the foregoing does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment.
_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Aligning your VMDK files is the first thing I'd do for certain. As was mentioned on other messages you should verify that the PAM cards will help so you can evaluate cost/benefit. For us they worked wonders but if you are write constrained then all the cache in the world is not going to help.
On 09/28/2012 11:05 AM, Steffen Knauf wrote:
Yes that's true. A strange thing is that VSC (4.0) and mbrscan show me different results:
/opt/ontap # ./mbrscan /vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk
mbrtools esxi version 1.0
/vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk p1 (Linux) lba:64 offset:32768 aligned:Yes
/vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk p2 (swap) lba:401640 offset:205639680 aligned:Yes
/vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk p3 (Linux) lba:16771888 offset:8587206656 aligned:Yes
I think i can delete the backup file?:
[Counter=2891433], Filename=vol_vm1/openx-test01/openx-test01-flat.vmdk-mbralign-backup
I rescan the datastores via VSC and get the result that openx-test01 is misaligned
And some mbrscans are not possible:
./mbrscan /vmfs/volumes/chip4-vmware_data_nfs01/openx-test03/openx-test03-flat.vmdk
Failed to open /vmfs/volumes/chip4-vmware_data_nfs01/openx-test03/openx-test03-flat.vmdk
- [Device or resource busy]
Thanks for your help and have a nice weekend. I'm on holiday for 1 Week, so my response could be delayed.....
greets
Steffen
*Von:*Jeff Mohler [mailto:speedtoys.racing@gmail.com] *Gesendet:* Mittwoch, 26. September 2012 17:58 *An:* Steffen Knauf *Cc:* Jeremy Page; toasters@teaparty.net *Betreff:* Re: AW: Raidgroupsize and I/O Performance
What will that accomplish? Alignment is first order pain you need to resolve.
Sent from my iPhone
On Sep 26, 2012, at 8:16 AM, "Steffen Knauf" <sknauf@chipxonio.de mailto:sknauf@chipxonio.de> wrote:
It ssems that we have a alignment problem, too: nfsstat: Misaligned Read request stats BIN-0 BIN-1 BIN-2 BIN-3 BIN-4 BIN-5 BIN-6 BIN-7 1096332359 1984142147 16821356 3951254 469217191 6145276 94494489 245873608 Misaligned Write request stats BIN-0 BIN-1 BIN-2 BIN-3 BIN-4 BIN-5 BIN-6 BIN-7 252471989 1287671452 52026962 32535853 278299197 40175418 38565555 692647941 I'll do a reallocate measurement tomorrow. greets Steffen *Von:*toasters-bounces@teaparty.net <mailto:toasters-bounces@teaparty.net> [mailto:toasters-bounces@teaparty.net] *Im Auftrag von *Jeremy Page *Gesendet:* Dienstag, 25. September 2012 14:23 *An:* toasters@teaparty.net <mailto:toasters@teaparty.net> *Betreff:* Re: Raidgroupsize and I/O Performance If you're considering getting a cache adapter I'd hold off on messing with the RAID config until afterwards - tuning for writes is different than tuning for IO in general since you're looking for free space and generally the more contiguous the better. Have you done a reallocate measurement, are your VMDK files all aligned? On 09/25/2012 04:19 AM, Steffen Knauf wrote: Hello, i'll try to improve our I/0 Perfomance. We have a raidgroup with 16 disks (SAS), 1 aggregate and 1 volume (dedup enabled). The volume is the storage for 100 VM's on the VMware Cluser (access via NFS). Does it make sense to increase the raidgroup? 90% Percent of the Disk I/O are Read Ops, so i'll buy a PAM Card for our FAS3240,too. And what's your Experience with the Raidgroupsize of a Raidgroup with SATA Disks (now: 11+1)? Thanks and greets ! Steffen _______________________________________________ Toasters mailing list Toasters@teaparty.net <mailto:Toasters@teaparty.net> http://www.teaparty.net/mailman/listinfo/toasters Please be advised that this email may contain confidential information. If you are not the intended recipient, please notify us by email by replying to the sender and delete this message. The sender disclaims that the content of this email constitutes an offer to enter into, or the acceptance of, any agreement; provided that the foregoing does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment. _______________________________________________ Toasters mailing list Toasters@teaparty.net <mailto:Toasters@teaparty.net> http://www.teaparty.net/mailman/listinfo/toasters
Please be advised that this email may contain confidential information. If you are not the intended recipient, please notify us by email by replying to the sender and delete this message. The sender disclaims that the content of this email constitutes an offer to enter into, or the acceptance of, any agreement; provided that the foregoing does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment.
Hi,
a small update. After aligning some VM's, a couple of them are slower (disk latency is higher than 200ms). The NFS Storage, where the *.vmdk's are stored has a different lateny than the disk. Strange........
./mbralign --sparse /vmfs/volumes/chip4-vmware_data_nfs01/web02/web02-flat.vmdk
Part Type old LBA New Start LBA New End LBA Length in KB
P1 83 63 64 401626 200781
P2 82 401625 401640 4482150 2040255
P3 83 4482135 4482160 314568790 155043315
Any Ideas?
greets
Steffen
Von: Jeremy Page [mailto:jeremy.page@gilbarco.com] Gesendet: Freitag, 28. September 2012 17:11 An: Steffen Knauf Cc: 'Jeff Mohler'; toasters@teaparty.net Betreff: Re: AW: AW: Raidgroupsize and I/O Performance
Aligning your VMDK files is the first thing I'd do for certain. As was mentioned on other messages you should verify that the PAM cards will help so you can evaluate cost/benefit. For us they worked wonders but if you are write constrained then all the cache in the world is not going to help.
On 09/28/2012 11:05 AM, Steffen Knauf wrote:
Yes that's true. A strange thing is that VSC (4.0) and mbrscan show me different results:
/opt/ontap # ./mbrscan /vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk
mbrtools esxi version 1.0
--------------------
/vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk p1 (Linux) lba:64 offset:32768 aligned:Yes
/vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk p2 (swap) lba:401640 offset:205639680 aligned:Yes
/vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk p3 (Linux) lba:16771888 offset:8587206656 aligned:Yes
I think i can delete the backup file?:
[Counter=2891433], Filename=vol_vm1/openx-test01/openx-test01-flat.vmdk-mbralign-backup
I rescan the datastores via VSC and get the result that openx-test01 is misaligned
And some mbrscans are not possible:
./mbrscan /vmfs/volumes/chip4-vmware_data_nfs01/openx-test03/openx-test03-flat.vmdk
Failed to open /vmfs/volumes/chip4-vmware_data_nfs01/openx-test03/openx-test03-flat.vmdk - [Device or resource busy]
Thanks for your help and have a nice weekend. I'm on holiday for 1 Week, so my response could be delayed.....
greets
Steffen
How are you mapping the latency? On the filer stats show -i5 nfsv3 will give you good info, on the VM itself are you using iostat or ?
It does not make sense, aligning should never cause increased latency unless something else odd is going on.
Is it write or read latency that's increased (or both)? Do you have atime updates disabled on your volume options (which can cause "other_ops" to increase, although it shouldn't change the latency).
*Jeremy Page* | Gilbarco Veeder-Root, A Danaher Company Senior Technical Architect | IS Infrastructure Systems | Yahoo IM: jeremypage Office:336-547-5399 | Cell: 336-601-7274 | 24x7 Emergency: 336-430-8151
On 10/17/2012 09:49 AM, Steffen Knauf wrote:
Hi,
a small update. After aligning some VM's, a couple of them are slower (disk latency is higher than 200ms). The NFS Storage, where the *.vmdk's are stored has a different lateny than the disk. Strange........
./mbralign --sparse /vmfs/volumes/chip4-vmware_data_nfs01/web02/web02-flat.vmdk
Part Type old LBA New Start LBA New End LBA Length in KB
P1 83 63 64 401626 200781
P2 82 401625 401640 4482150 2040255
P3 83 4482135 4482160 314568790 155043315
Any Ideas?
greets
Steffen
*Von:*Jeremy Page [mailto:jeremy.page@gilbarco.com] *Gesendet:* Freitag, 28. September 2012 17:11 *An:* Steffen Knauf *Cc:* 'Jeff Mohler'; toasters@teaparty.net *Betreff:* Re: AW: AW: Raidgroupsize and I/O Performance
Aligning your VMDK files is the first thing I'd do for certain. As was mentioned on other messages you should verify that the PAM cards will help so you can evaluate cost/benefit. For us they worked wonders but if you are write constrained then all the cache in the world is not going to help.
On 09/28/2012 11:05 AM, Steffen Knauf wrote:
Yes that's true. A strange thing is that VSC (4.0) and mbrscan show me different results: /opt/ontap # ./mbrscan /vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk mbrtools esxi version 1.0 -------------------- /vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk p1 (Linux) lba:64 offset:32768 aligned:Yes /vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk p2 (swap) lba:401640 offset:205639680 aligned:Yes /vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk p3 (Linux) lba:16771888 offset:8587206656 aligned:Yes I think i can delete the backup file?: [Counter=2891433], Filename=vol_vm1/openx-test01/openx-test01-flat.vmdk-mbralign-backup I rescan the datastores via VSC and get the result that openx-test01 is misaligned And some mbrscans are not possible: ./mbrscan /vmfs/volumes/chip4-vmware_data_nfs01/openx-test03/openx-test03-flat.vmdk Failed to open /vmfs/volumes/chip4-vmware_data_nfs01/openx-test03/openx-test03-flat.vmdk - [Device or resource busy] Thanks for your help and have a nice weekend. I'm on holiday for 1 Week, so my response could be delayed..... greets Steffen
Please be advised that this email may contain confidential information. If you are not the intended recipient, please notify us by email by replying to the sender and delete this message. The sender disclaims that the content of this email constitutes an offer to enter into, or the acceptance of, any agreement; provided that the foregoing does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment.
The latency on nfs storage seems to be fine :
Instance nfsv3_ops nfsv3_read_l nfsv3_read_o nfsv3_write_ nfsv3_write_
/s ms /s ms /s
nfs 1021 3.52 222 0.18 795
nfs 1189 3.03 210 0.29 976
nfs 1138 6.29 221 0.47 908
nfs 1447 2.80 220 0.11 1220
nfs 2226 6.19 207 0.33 2016
nfs 2180 3.64 191 0.41 1985
nfs 1706 6.04 199 5.47 1498
nfs 2388 3.31 236 0.23 2146
nfs 2366 7.47 287 3.35 2075
nfs 1623 5.79 194 1.26 1426
stats show -i3 volume:vol_vm1:read_latency volume:vol_vm1:write_latency
Instance read_latency write_latenc
us us
vol_vm1 9720.45 24395.98
vol_vm1 6743.96 932.14
vol_vm1 4996.47 183.15
vol_vm1 5898.92 333.17
vol_vm1 8366.29 320.44
vol_vm1 20432.34 1644.11
vol_vm1 8089.78 1071.03
vol_vm1 15534.60 13517.11
vol_vm1 8862.75 2014.93
vol_vm1 4092.92 358.95
vol_vm1 8605.04 1528.93
vol_vm1 11640.06 3673.86
vol_vm1 5593.18 467.98
vol_vm1 5007.53 211.68
vol_vm1 11465.33 681.64
vol_vm1 8117.90 714.84
vol_vm1 6328.100 416.68
vol_vm1 6368.60 687.42
vol_vm1 7068.68 1430.92
vol_vm1 16762.75 1071.98
vol_vm1 7867.80 915.64
vol_vm1 4484.89 336.83
I think a latency <25ms is ok? Sometimes i have some spikes:
vol_vm1 36878.74 13100.62
vol_vm1 22711.02 12180.19
vol_vm1 10190.43 14667.47
vol_vm1 72765.70 65223.42
vol_vm1 25004.14 7689.66
vol_vm1 23907.21 1882.91
After aligning a VM i took this VM to create a template. During the the OS Installation i got nfs timeouts (pxe) and the Installation is really slow (if the VM virtual disk latency graph is correct, it is more than 300 ms, but the ). If i don't take the template to create a new VM i don't have any trouble. After creating a new template of an other aligned VM i don't have any trouble. It looks like a VMware problem.............grrr ;(
Thanks for help!
greets
Steffen
Von: Jeremy Page [mailto:jeremy.page@gilbarco.com] Gesendet: Mittwoch, 17. Oktober 2012 15:54 An: Steffen Knauf Cc: toasters@teaparty.net Betreff: Re: AW: Raidgroupsize and I/O Performance
How are you mapping the latency? On the filer stats show -i5 nfsv3 will give you good info, on the VM itself are you using iostat or ?
It does not make sense, aligning should never cause increased latency unless something else odd is going on.
Is it write or read latency that's increased (or both)? Do you have atime updates disabled on your volume options (which can cause "other_ops" to increase, although it shouldn't change the latency).
Jeremy Page | Gilbarco Veeder-Root, A Danaher Company Senior Technical Architect | IS Infrastructure Systems | Yahoo IM: jeremypage Office:336-547-5399 | Cell: 336-601-7274 | 24x7 Emergency: 336-430-8151
On 10/17/2012 09:49 AM, Steffen Knauf wrote:
Hi,
a small update. After aligning some VM's, a couple of them are slower (disk latency is higher than 200ms). The NFS Storage, where the *.vmdk's are stored has a different lateny than the disk. Strange........
./mbralign --sparse /vmfs/volumes/chip4-vmware_data_nfs01/web02/web02-flat.vmdk
Part Type old LBA New Start LBA New End LBA Length in KB
P1 83 63 64 401626 200781
P2 82 401625 401640 4482150 2040255
P3 83 4482135 4482160 314568790 155043315
Any Ideas?
greets
Steffen
Von: Jeremy Page [mailto:jeremy.page@gilbarco.com] Gesendet: Freitag, 28. September 2012 17:11 An: Steffen Knauf Cc: 'Jeff Mohler'; toasters@teaparty.net Betreff: Re: AW: AW: Raidgroupsize and I/O Performance
Aligning your VMDK files is the first thing I'd do for certain. As was mentioned on other messages you should verify that the PAM cards will help so you can evaluate cost/benefit. For us they worked wonders but if you are write constrained then all the cache in the world is not going to help.
On 09/28/2012 11:05 AM, Steffen Knauf wrote:
Yes that's true. A strange thing is that VSC (4.0) and mbrscan show me different results:
/opt/ontap # ./mbrscan /vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk
mbrtools esxi version 1.0
--------------------
/vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk p1 (Linux) lba:64 offset:32768 aligned:Yes
/vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk p2 (swap) lba:401640 offset:205639680 aligned:Yes
/vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk p3 (Linux) lba:16771888 offset:8587206656 aligned:Yes
I think i can delete the backup file?:
[Counter=2891433], Filename=vol_vm1/openx-test01/openx-test01-flat.vmdk-mbralign-backup
I rescan the datastores via VSC and get the result that openx-test01 is misaligned
And some mbrscans are not possible:
./mbrscan /vmfs/volumes/chip4-vmware_data_nfs01/openx-test03/openx-test03-flat.vmdk
Failed to open /vmfs/volumes/chip4-vmware_data_nfs01/openx-test03/openx-test03-flat.vmdk - [Device or resource busy]
Thanks for your help and have a nice weekend. I'm on holiday for 1 Week, so my response could be delayed.....
greets
Steffen
Please be advised that this email may contain confidential information. If you are not the intended recipient, please notify us by email by replying to the sender and delete this message. The sender disclaims that the content of this email constitutes an offer to enter into, or the acceptance of, any agreement; provided that the foregoing does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment.
Hi Steffen,
are you _really_ sure the VMs are aligned well?
Best way to check IMO is *nfsstat -d*:
MISALIGNED I/O A NFS read or write is termed misaligned when its length is a multiple of 4K and it's starting offset is not aligned to a 4K boundary. A large number of misaligned reads and writes will have an adverse impact on the performance of the filer. NFS reads and writes are binned into eight bins (BIN-0 to BIN-7). All read and write requests from BIN-1 to BIN-7 are misaligned. *A list of filenames that caused** **the most misaligned I/O's is displayed with the -d option.*
If you're using *VSC 4*, you might also be able to see which VMs are mis-aligned. The *Optimization and Migration capability* lets you scan datastores and correct the alignment of certain misaligned virtual machines (VM) without having to power down the VM.
These two methods check the *actual *read/write patterns, so it's a lot safer, than to check the theory only. (E.g. I've seen people 'aligning' MBR VMDKs that were created using Windows _2008_...)
Greetings
Sebastian
On 17.10.2012 17:43, Steffen Knauf wrote:
The latency on nfs storage seems to be fine :
Instance nfsv3_ops nfsv3_read_l nfsv3_read_o nfsv3_write_ nfsv3_write_
/s ms /s ms /s
nfs 1021 3.52 222 0.18 795
nfs 1189 3.03 210 0.29 976
nfs 1138 6.29 221 0.47 908 nfs 1447 2.80 220 0.11 1220 nfs 2226 6.19 207 0.33 2016 nfs 2180 3.64 191 0.41 1985 nfs 1706 6.04 199 5.47 1498 nfs 2388 3.31 236 0.23 2146 nfs 2366 7.47 287 3.35 2075 nfs 1623 5.79 194 1.26 1426stats show -i3 volume:vol_vm1:read_latency volume:vol_vm1:write_latency
Instance read_latency write_latenc
us us
vol_vm1 9720.45 24395.98
vol_vm1 6743.96 932.14
vol_vm1 4996.47 183.15
vol_vm1 5898.92 333.17
vol_vm1 8366.29 320.44
vol_vm1 20432.34 1644.11
vol_vm1 8089.78 1071.03
vol_vm1 15534.60 13517.11
vol_vm1 8862.75 2014.93
vol_vm1 4092.92 358.95
vol_vm1 8605.04 1528.93
vol_vm1 11640.06 3673.86
vol_vm1 5593.18 467.98
vol_vm1 5007.53 211.68
vol_vm1 11465.33 681.64
vol_vm1 8117.90 714.84
vol_vm1 6328.100 416.68
vol_vm1 6368.60 687.42
vol_vm1 7068.68 1430.92
vol_vm1 16762.75 1071.98
vol_vm1 7867.80 915.64
vol_vm1 4484.89 336.83
I think a latency <25ms is ok? Sometimes i have some spikes:
vol_vm1 36878.74 13100.62
vol_vm1 22711.02 12180.19
vol_vm1 10190.43 14667.47
vol_vm1 72765.70 65223.42
vol_vm1 25004.14 7689.66
vol_vm1 23907.21 1882.91
After aligning a VM i took this VM to create a template. During the the OS Installation i got nfs timeouts (pxe) and the Installation is really slow (if the VM virtual disk latency graph is correct, it is more than 300 ms, but the ). If i don't take the template to create a new VM i don't have any trouble. After creating a new template of an other aligned VM i don't have any trouble. It looks like a VMware problem.............grrr ;(
Thanks for help!
greets
Steffen
*Von:*Jeremy Page [mailto:jeremy.page@gilbarco.com] *Gesendet:* Mittwoch, 17. Oktober 2012 15:54 *An:* Steffen Knauf *Cc:* toasters@teaparty.net *Betreff:* Re: AW: Raidgroupsize and I/O Performance
How are you mapping the latency? On the filer stats show -i5 nfsv3 will give you good info, on the VM itself are you using iostat or ?
It does not make sense, aligning should never cause increased latency unless something else odd is going on.
Is it write or read latency that's increased (or both)? Do you have atime updates disabled on your volume options (which can cause "other_ops" to increase, although it shouldn't change the latency).
*Jeremy Page*| Gilbarco Veeder-Root, A Danaher Company Senior Technical Architect | IS Infrastructure Systems | Yahoo IM: jeremypage Office:336-547-5399 | Cell: 336-601-7274 | 24x7 Emergency: 336-430-8151
On 10/17/2012 09:49 AM, Steffen Knauf wrote:
Hi, a small update. After aligning some VM's, a couple of them are slower (disk latency is higher than 200ms). The NFS Storage, where the *.vmdk's are stored has a different lateny than the disk. Strange........ ./mbralign --sparse /vmfs/volumes/chip4-vmware_data_nfs01/web02/web02-flat.vmdk Part Type old LBA New Start LBA New End LBA Length in KB P1 83 63 64 401626 200781 P2 82 401625 401640 4482150 2040255 P3 83 4482135 4482160 314568790 155043315 Any Ideas? greets Steffen *Von:*Jeremy Page [mailto:jeremy.page@gilbarco.com] *Gesendet:* Freitag, 28. September 2012 17:11 *An:* Steffen Knauf *Cc:* 'Jeff Mohler'; toasters@teaparty.net <mailto:toasters@teaparty.net> *Betreff:* Re: AW: AW: Raidgroupsize and I/O Performance Aligning your VMDK files is the first thing I'd do for certain. As was mentioned on other messages you should verify that the PAM cards will help so you can evaluate cost/benefit. For us they worked wonders but if you are write constrained then all the cache in the world is not going to help. On 09/28/2012 11:05 AM, Steffen Knauf wrote: Yes that's true. A strange thing is that VSC (4.0) and mbrscan show me different results: /opt/ontap # ./mbrscan /vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk mbrtools esxi version 1.0 -------------------- /vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk p1 (Linux) lba:64 offset:32768 aligned:Yes /vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk p2 (swap) lba:401640 offset:205639680 aligned:Yes /vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk p3 (Linux) lba:16771888 offset:8587206656 aligned:Yes I think i can delete the backup file?: [Counter=2891433], Filename=vol_vm1/openx-test01/openx-test01-flat.vmdk-mbralign-backup I rescan the datastores via VSC and get the result that openx-test01 is misaligned And some mbrscans are not possible: ./mbrscan /vmfs/volumes/chip4-vmware_data_nfs01/openx-test03/openx-test03-flat.vmdk Failed to open /vmfs/volumes/chip4-vmware_data_nfs01/openx-test03/openx-test03-flat.vmdk - [Device or resource busy] Thanks for your help and have a nice weekend. I'm on holiday for 1 Week, so my response could be delayed..... greets SteffenPlease be advised that this email may contain confidential information. If you are not the intended recipient, please notify us by email by replying to the sender and delete this message. The sender disclaims that the content of this email constitutes an offer to enter into, or the acceptance of, any agreement; provided that the foregoing does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment.
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Hi Sebastian,
i'm sure, because i used the "nfsstat -d" Option. But there is one strange thing. If i try to rescan an aligned VM, i get a "Failed to open XXX-flat.vmdk - [Device or resource busy]" . A Scan is only possible if the VM is powered off (VSC and cmd line). And there is another strange thing. I find a aligned VM in the list of "Files Causing Misaligned IO's" (nfsstat -d) again.
greets
Steffen
Von: Sebastian Goetze [mailto:spgoetze@gmail.com] Gesendet: Mittwoch, 17. Oktober 2012 19:47 An: Steffen Knauf Cc: 'Jeremy Page'; toasters@teaparty.net Betreff: Re: Raidgroupsize and I/O Performance
Hi Steffen,
are you really sure the VMs are aligned well?
Best way to check IMO is nfsstat -d:
MISALIGNED I/O A NFS read or write is termed misaligned when its length is a multiple of 4K and it's starting offset is not aligned to a 4K boundary. A large number of misaligned reads and writes will have an adverse impact on the performance of the filer. NFS reads and writes are binned into eight bins (BIN-0 to BIN-7). All read and write requests from BIN-1 to BIN-7 are misaligned. A list of filenames that caused the most misaligned I/O's is displayed with the -d option.
If you're using VSC 4, you might also be able to see which VMs are mis-aligned. The Optimization and Migration capability lets you scan datastores and correct the alignment of certain misaligned virtual machines (VM) without having to power down the VM.
These two methods check the actual read/write patterns, so it's a lot safer, than to check the theory only. (E.g. I've seen people 'aligning' MBR VMDKs that were created using Windows 2008...)
Greetings
Sebastian
On 17.10.2012 17:43, Steffen Knauf wrote:
The latency on nfs storage seems to be fine :
Instance nfsv3_ops nfsv3_read_l nfsv3_read_o nfsv3_write_ nfsv3_write_
/s ms /s ms /s
nfs 1021 3.52 222 0.18 795
nfs 1189 3.03 210 0.29 976
nfs 1138 6.29 221 0.47 908
nfs 1447 2.80 220 0.11 1220
nfs 2226 6.19 207 0.33 2016
nfs 2180 3.64 191 0.41 1985
nfs 1706 6.04 199 5.47 1498
nfs 2388 3.31 236 0.23 2146
nfs 2366 7.47 287 3.35 2075
nfs 1623 5.79 194 1.26 1426
stats show -i3 volume:vol_vm1:read_latency volume:vol_vm1:write_latency
Instance read_latency write_latenc
us us
vol_vm1 9720.45 24395.98
vol_vm1 6743.96 932.14
vol_vm1 4996.47 183.15
vol_vm1 5898.92 333.17
vol_vm1 8366.29 320.44
vol_vm1 20432.34 1644.11
vol_vm1 8089.78 1071.03
vol_vm1 15534.60 13517.11
vol_vm1 8862.75 2014.93
vol_vm1 4092.92 358.95
vol_vm1 8605.04 1528.93
vol_vm1 11640.06 3673.86
vol_vm1 5593.18 467.98
vol_vm1 5007.53 211.68
vol_vm1 11465.33 681.64
vol_vm1 8117.90 714.84
vol_vm1 6328.100 416.68
vol_vm1 6368.60 687.42
vol_vm1 7068.68 1430.92
vol_vm1 16762.75 1071.98
vol_vm1 7867.80 915.64
vol_vm1 4484.89 336.83
I think a latency <25ms is ok? Sometimes i have some spikes:
vol_vm1 36878.74 13100.62
vol_vm1 22711.02 12180.19
vol_vm1 10190.43 14667.47
vol_vm1 72765.70 65223.42
vol_vm1 25004.14 7689.66
vol_vm1 23907.21 1882.91
After aligning a VM i took this VM to create a template. During the the OS Installation i got nfs timeouts (pxe) and the Installation is really slow (if the VM virtual disk latency graph is correct, it is more than 300 ms, but the ). If i don't take the template to create a new VM i don't have any trouble. After creating a new template of an other aligned VM i don't have any trouble. It looks like a VMware problem.............grrr ;(
Thanks for help!
greets
Steffen
Von: Jeremy Page [mailto:jeremy.page@gilbarco.com] Gesendet: Mittwoch, 17. Oktober 2012 15:54 An: Steffen Knauf Cc: toasters@teaparty.net Betreff: Re: AW: Raidgroupsize and I/O Performance
How are you mapping the latency? On the filer stats show -i5 nfsv3 will give you good info, on the VM itself are you using iostat or ?
It does not make sense, aligning should never cause increased latency unless something else odd is going on.
Is it write or read latency that's increased (or both)? Do you have atime updates disabled on your volume options (which can cause "other_ops" to increase, although it shouldn't change the latency).
Jeremy Page | Gilbarco Veeder-Root, A Danaher Company Senior Technical Architect | IS Infrastructure Systems | Yahoo IM: jeremypage Office:336-547-5399 | Cell: 336-601-7274 | 24x7 Emergency: 336-430-8151
On 10/17/2012 09:49 AM, Steffen Knauf wrote:
Hi,
a small update. After aligning some VM's, a couple of them are slower (disk latency is higher than 200ms). The NFS Storage, where the *.vmdk's are stored has a different lateny than the disk. Strange........
./mbralign --sparse /vmfs/volumes/chip4-vmware_data_nfs01/web02/web02-flat.vmdk
Part Type old LBA New Start LBA New End LBA Length in KB
P1 83 63 64 401626 200781
P2 82 401625 401640 4482150 2040255
P3 83 4482135 4482160 314568790 155043315
Any Ideas?
greets
Steffen
Von: Jeremy Page [mailto:jeremy.page@gilbarco.com] Gesendet: Freitag, 28. September 2012 17:11 An: Steffen Knauf Cc: 'Jeff Mohler'; toasters@teaparty.net Betreff: Re: AW: AW: Raidgroupsize and I/O Performance
Aligning your VMDK files is the first thing I'd do for certain. As was mentioned on other messages you should verify that the PAM cards will help so you can evaluate cost/benefit. For us they worked wonders but if you are write constrained then all the cache in the world is not going to help.
On 09/28/2012 11:05 AM, Steffen Knauf wrote:
Yes that's true. A strange thing is that VSC (4.0) and mbrscan show me different results:
/opt/ontap # ./mbrscan /vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk
mbrtools esxi version 1.0
--------------------
/vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk p1 (Linux) lba:64 offset:32768 aligned:Yes
/vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk p2 (swap) lba:401640 offset:205639680 aligned:Yes
/vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk p3 (Linux) lba:16771888 offset:8587206656 aligned:Yes
I think i can delete the backup file?:
[Counter=2891433], Filename=vol_vm1/openx-test01/openx-test01-flat.vmdk-mbralign-backup
I rescan the datastores via VSC and get the result that openx-test01 is misaligned
And some mbrscans are not possible:
./mbrscan /vmfs/volumes/chip4-vmware_data_nfs01/openx-test03/openx-test03-flat.vmdk
Failed to open /vmfs/volumes/chip4-vmware_data_nfs01/openx-test03/openx-test03-flat.vmdk - [Device or resource busy]
Thanks for your help and have a nice weekend. I'm on holiday for 1 Week, so my response could be delayed.....
greets
Steffen
Please be advised that this email may contain confidential information. If you are not the intended recipient, please notify us by email by replying to the sender and delete this message. The sender disclaims that the content of this email constitutes an offer to enter into, or the acceptance of, any agreement; provided that the foregoing does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment.
_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
" I find a aligned VM in the list of "Files Causing Misaligned IO's" (nfsstat -d) again."
It's an imperfect tool..just guessing, does the "misaligned" aligned VM, have a database _log_ file being written to within it?
On Thu, Oct 18, 2012 at 1:40 AM, Steffen Knauf sknauf@chipxonio.de wrote:
Hi Sebastian,
i'm sure, because i used the "nfsstat -d" Option. But there is one strange thing. If i try to rescan an aligned VM, i get a "Failed to open XXX-flat.vmdk - [Device or resource busy]" . A Scan is only possible if the VM is powered off (VSC and cmd line). And there is another strange thing. I find a aligned VM in the list of "Files Causing Misaligned IO's" (nfsstat -d) again.
greets
Steffen
Von: Sebastian Goetze [mailto:spgoetze@gmail.com] Gesendet: Mittwoch, 17. Oktober 2012 19:47
An: Steffen Knauf Cc: 'Jeremy Page'; toasters@teaparty.net Betreff: Re: Raidgroupsize and I/O Performance
Hi Steffen,
are you really sure the VMs are aligned well?
Best way to check IMO is nfsstat -d:
MISALIGNED I/O A NFS read or write is termed misaligned when its length is a multiple of 4K and it’s starting offset is not aligned to a 4K boundary. A large number of misaligned reads and writes will have an adverse impact on the performance of the filer. NFS reads and writes are binned into eight bins (BIN-0 to BIN-7). All read and write requests from BIN-1 to BIN-7 are misaligned. A list of filenames that caused the most misaligned I/O’s is displayed with the -d option.
If you're using VSC 4, you might also be able to see which VMs are mis-aligned. The Optimization and Migration capability lets you scan datastores and correct the alignment of certain misaligned virtual machines (VM) without having to power down the VM.
These two methods check the actual read/write patterns, so it's a lot safer, than to check the theory only. (E.g. I've seen people 'aligning' MBR VMDKs that were created using Windows 2008...)
Greetings
Sebastian
On 17.10.2012 17:43, Steffen Knauf wrote:
The latency on nfs storage seems to be fine :
Instance nfsv3_ops nfsv3_read_l nfsv3_read_o nfsv3_write_ nfsv3_write_
/s ms /s ms /s nfs 1021 3.52 222 0.18 795 nfs 1189 3.03 210 0.29 976 nfs 1138 6.29 221 0.47 908 nfs 1447 2.80 220 0.11 1220 nfs 2226 6.19 207 0.33 2016 nfs 2180 3.64 191 0.41 1985 nfs 1706 6.04 199 5.47 1498 nfs 2388 3.31 236 0.23 2146 nfs 2366 7.47 287 3.35 2075 nfs 1623 5.79 194 1.26 1426stats show -i3 volume:vol_vm1:read_latency volume:vol_vm1:write_latency
Instance read_latency write_latenc
us usvol_vm1 9720.45 24395.98
vol_vm1 6743.96 932.14
vol_vm1 4996.47 183.15
vol_vm1 5898.92 333.17
vol_vm1 8366.29 320.44
vol_vm1 20432.34 1644.11
vol_vm1 8089.78 1071.03
vol_vm1 15534.60 13517.11
vol_vm1 8862.75 2014.93
vol_vm1 4092.92 358.95
vol_vm1 8605.04 1528.93
vol_vm1 11640.06 3673.86
vol_vm1 5593.18 467.98
vol_vm1 5007.53 211.68
vol_vm1 11465.33 681.64
vol_vm1 8117.90 714.84
vol_vm1 6328.100 416.68
vol_vm1 6368.60 687.42
vol_vm1 7068.68 1430.92
vol_vm1 16762.75 1071.98
vol_vm1 7867.80 915.64
vol_vm1 4484.89 336.83
I think a latency <25ms is ok? Sometimes i have some spikes:
vol_vm1 36878.74 13100.62
vol_vm1 22711.02 12180.19
vol_vm1 10190.43 14667.47
vol_vm1 72765.70 65223.42
vol_vm1 25004.14 7689.66
vol_vm1 23907.21 1882.91
After aligning a VM i took this VM to create a template. During the the OS Installation i got nfs timeouts (pxe) and the Installation is really slow (if the VM virtual disk latency graph is correct, it is more than 300 ms, but the ). If i don't take the template to create a new VM i don't have any trouble. After creating a new template of an other aligned VM i don't have any trouble. It looks like a VMware problem.............grrr ;(
Thanks for help!
greets
Steffen
Von: Jeremy Page [mailto:jeremy.page@gilbarco.com] Gesendet: Mittwoch, 17. Oktober 2012 15:54 An: Steffen Knauf Cc: toasters@teaparty.net Betreff: Re: AW: Raidgroupsize and I/O Performance
How are you mapping the latency? On the filer stats show -i5 nfsv3 will give you good info, on the VM itself are you using iostat or ?
It does not make sense, aligning should never cause increased latency unless something else odd is going on.
Is it write or read latency that's increased (or both)? Do you have atime updates disabled on your volume options (which can cause "other_ops" to increase, although it shouldn't change the latency).
Jeremy Page | Gilbarco Veeder-Root, A Danaher Company Senior Technical Architect | IS Infrastructure Systems | Yahoo IM: jeremypage Office:336-547-5399 | Cell: 336-601-7274 | 24x7 Emergency: 336-430-8151
On 10/17/2012 09:49 AM, Steffen Knauf wrote:
Hi,
a small update. After aligning some VM's, a couple of them are slower (disk latency is higher than 200ms). The NFS Storage, where the *.vmdk's are stored has a different lateny than the disk. Strange........
./mbralign --sparse /vmfs/volumes/chip4-vmware_data_nfs01/web02/web02-flat.vmdk
Part Type old LBA New Start LBA New End LBA Length in KB
P1 83 63 64 401626 200781
P2 82 401625 401640 4482150 2040255
P3 83 4482135 4482160 314568790 155043315
Any Ideas?
greets
Steffen
Von: Jeremy Page [mailto:jeremy.page@gilbarco.com] Gesendet: Freitag, 28. September 2012 17:11 An: Steffen Knauf Cc: 'Jeff Mohler'; toasters@teaparty.net Betreff: Re: AW: AW: Raidgroupsize and I/O Performance
Aligning your VMDK files is the first thing I'd do for certain. As was mentioned on other messages you should verify that the PAM cards will help so you can evaluate cost/benefit. For us they worked wonders but if you are write constrained then all the cache in the world is not going to help.
On 09/28/2012 11:05 AM, Steffen Knauf wrote:
Yes that's true. A strange thing is that VSC (4.0) and mbrscan show me different results:
/opt/ontap # ./mbrscan /vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk
mbrtools esxi version 1.0
/vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk p1 (Linux) lba:64 offset:32768 aligned:Yes
/vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk p2 (swap) lba:401640 offset:205639680 aligned:Yes
/vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk p3 (Linux) lba:16771888 offset:8587206656 aligned:Yes
I think i can delete the backup file?:
[Counter=2891433], Filename=vol_vm1/openx-test01/openx-test01-flat.vmdk-mbralign-backup
I rescan the datastores via VSC and get the result that openx-test01 is misaligned
And some mbrscans are not possible:
./mbrscan /vmfs/volumes/chip4-vmware_data_nfs01/openx-test03/openx-test03-flat.vmdk
Failed to open /vmfs/volumes/chip4-vmware_data_nfs01/openx-test03/openx-test03-flat.vmdk - [Device or resource busy]
Thanks for your help and have a nice weekend. I'm on holiday for 1 Week, so my response could be delayed.....
greets
Steffen
Please be advised that this email may contain confidential information. If you are not the intended recipient, please notify us by email by replying to the sender and delete this message. The sender disclaims that the content of this email constitutes an offer to enter into, or the acceptance of, any agreement; provided that the foregoing does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment.
Toasters mailing list
Toasters@teaparty.net
http://www.teaparty.net/mailman/listinfo/toasters
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Nfsstat is not an imperfect tool, nor is mbralign...
You should zero your counters for nfsstat after alignment (nfsstat -z) and let the aligned VM run for a while (24 hours) before checking again using nfsstat -d. You are still dragging the misaligned status in the counters from before the alignment on your filer.
Kind regards,
Peter.
-----Original Message----- From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Jeff Mohler Sent: donderdag 18 oktober 2012 10:46 To: Steffen Knauf Cc: toasters@teaparty.net Subject: Re: Raidgroupsize and I/O Performance
" I find a aligned VM in the list of "Files Causing Misaligned IO's" (nfsstat -d) again."
It's an imperfect tool..just guessing, does the "misaligned" aligned VM, have a database _log_ file being written to within it?
On Thu, Oct 18, 2012 at 1:40 AM, Steffen Knauf sknauf@chipxonio.de wrote:
Hi Sebastian,
i'm sure, because i used the "nfsstat -d" Option. But there is one strange thing. If i try to rescan an aligned VM, i get a "Failed to open XXX-flat.vmdk - [Device or resource busy]" . A Scan is only possible if the VM is powered off (VSC and cmd line). And there is another strange thing. I find a aligned VM in the list of "Files Causing Misaligned IO's" (nfsstat -d) again.
greets
Steffen
Von: Sebastian Goetze [mailto:spgoetze@gmail.com] Gesendet: Mittwoch, 17. Oktober 2012 19:47
An: Steffen Knauf Cc: 'Jeremy Page'; toasters@teaparty.net Betreff: Re: Raidgroupsize and I/O Performance
Hi Steffen,
are you really sure the VMs are aligned well?
Best way to check IMO is nfsstat -d:
MISALIGNED I/O A NFS read or write is termed misaligned when its length is a multiple of 4K and it's starting offset is not aligned to a 4K boundary. A large number of misaligned reads and writes will have an adverse impact on the performance of the filer. NFS reads and writes are binned into eight bins (BIN-0 to BIN-7). All read and write requests from BIN-1 to BIN-7 are misaligned. A list of filenames that caused the most misaligned I/O's is displayed with the -d option.
If you're using VSC 4, you might also be able to see which VMs are mis-aligned. The Optimization and Migration capability lets you scan datastores and correct the alignment of certain misaligned virtual machines (VM) without having to power down the VM.
These two methods check the actual read/write patterns, so it's a lot safer, than to check the theory only. (E.g. I've seen people 'aligning' MBR VMDKs that were created using Windows 2008...)
Greetings
Sebastian
On 17.10.2012 17:43, Steffen Knauf wrote:
The latency on nfs storage seems to be fine :
Instance nfsv3_ops nfsv3_read_l nfsv3_read_o nfsv3_write_ nfsv3_write_
/s ms /s ms /s nfs 1021 3.52 222 0.18 795 nfs 1189 3.03 210 0.29 976 nfs 1138 6.29 221 0.47 908 nfs 1447 2.80 220 0.11 1220 nfs 2226 6.19 207 0.33 2016 nfs 2180 3.64 191 0.41 1985 nfs 1706 6.04 199 5.47 1498 nfs 2388 3.31 236 0.23 2146 nfs 2366 7.47 287 3.35 2075 nfs 1623 5.79 194 1.26 1426stats show -i3 volume:vol_vm1:read_latency volume:vol_vm1:write_latency
Instance read_latency write_latenc
us usvol_vm1 9720.45 24395.98
vol_vm1 6743.96 932.14
vol_vm1 4996.47 183.15
vol_vm1 5898.92 333.17
vol_vm1 8366.29 320.44
vol_vm1 20432.34 1644.11
vol_vm1 8089.78 1071.03
vol_vm1 15534.60 13517.11
vol_vm1 8862.75 2014.93
vol_vm1 4092.92 358.95
vol_vm1 8605.04 1528.93
vol_vm1 11640.06 3673.86
vol_vm1 5593.18 467.98
vol_vm1 5007.53 211.68
vol_vm1 11465.33 681.64
vol_vm1 8117.90 714.84
vol_vm1 6328.100 416.68
vol_vm1 6368.60 687.42
vol_vm1 7068.68 1430.92
vol_vm1 16762.75 1071.98
vol_vm1 7867.80 915.64
vol_vm1 4484.89 336.83
I think a latency <25ms is ok? Sometimes i have some spikes:
vol_vm1 36878.74 13100.62
vol_vm1 22711.02 12180.19
vol_vm1 10190.43 14667.47
vol_vm1 72765.70 65223.42
vol_vm1 25004.14 7689.66
vol_vm1 23907.21 1882.91
After aligning a VM i took this VM to create a template. During the the OS Installation i got nfs timeouts (pxe) and the Installation is really slow (if the VM virtual disk latency graph is correct, it is more than 300 ms, but the ). If i don't take the template to create a new VM i don't have any trouble. After creating a new template of an other aligned VM i don't have any trouble. It looks like a VMware problem.............grrr ;(
Thanks for help!
greets
Steffen
Von: Jeremy Page [mailto:jeremy.page@gilbarco.com] Gesendet: Mittwoch, 17. Oktober 2012 15:54 An: Steffen Knauf Cc: toasters@teaparty.net Betreff: Re: AW: Raidgroupsize and I/O Performance
How are you mapping the latency? On the filer stats show -i5 nfsv3 will give you good info, on the VM itself are you using iostat or ?
It does not make sense, aligning should never cause increased latency unless something else odd is going on.
Is it write or read latency that's increased (or both)? Do you have atime updates disabled on your volume options (which can cause "other_ops" to increase, although it shouldn't change the latency).
Jeremy Page | Gilbarco Veeder-Root, A Danaher Company Senior Technical Architect | IS Infrastructure Systems | Yahoo IM: jeremypage Office:336-547-5399 | Cell: 336-601-7274 | 24x7 Emergency: 336-430-8151
On 10/17/2012 09:49 AM, Steffen Knauf wrote:
Hi,
a small update. After aligning some VM's, a couple of them are slower (disk latency is higher than 200ms). The NFS Storage, where the *.vmdk's are stored has a different lateny than the disk. Strange........
./mbralign --sparse /vmfs/volumes/chip4-vmware_data_nfs01/web02/web02-flat.vmdk
Part Type old LBA New Start LBA New End LBA Length in KB
P1 83 63 64 401626 200781
P2 82 401625 401640 4482150 2040255
P3 83 4482135 4482160 314568790 155043315
Any Ideas?
greets
Steffen
Von: Jeremy Page [mailto:jeremy.page@gilbarco.com] Gesendet: Freitag, 28. September 2012 17:11 An: Steffen Knauf Cc: 'Jeff Mohler'; toasters@teaparty.net Betreff: Re: AW: AW: Raidgroupsize and I/O Performance
Aligning your VMDK files is the first thing I'd do for certain. As was mentioned on other messages you should verify that the PAM cards will help so you can evaluate cost/benefit. For us they worked wonders but if you are write constrained then all the cache in the world is not going to help.
On 09/28/2012 11:05 AM, Steffen Knauf wrote:
Yes that's true. A strange thing is that VSC (4.0) and mbrscan show me different results:
/opt/ontap # ./mbrscan /vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.v mdk
mbrtools esxi version 1.0
/vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk p1 (Linux) lba:64 offset:32768 aligned:Yes
/vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk p2 (swap) lba:401640 offset:205639680 aligned:Yes
/vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk p3 (Linux) lba:16771888 offset:8587206656 aligned:Yes
I think i can delete the backup file?:
[Counter=2891433], Filename=vol_vm1/openx-test01/openx-test01-flat.vmdk-mbralign-backup
I rescan the datastores via VSC and get the result that openx-test01 is misaligned
And some mbrscans are not possible:
./mbrscan /vmfs/volumes/chip4-vmware_data_nfs01/openx-test03/openx-test03-flat.v mdk
Failed to open /vmfs/volumes/chip4-vmware_data_nfs01/openx-test03/openx-test03-flat.v mdk - [Device or resource busy]
Thanks for your help and have a nice weekend. I'm on holiday for 1 Week, so my response could be delayed.....
greets
Steffen
Please be advised that this email may contain confidential information. If you are not the intended recipient, please notify us by email by replying to the sender and delete this message. The sender disclaims that the content of this email constitutes an offer to enter into, or the acceptance of, any agreement; provided that the foregoing does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment.
Toasters mailing list
Toasters@teaparty.net
http://www.teaparty.net/mailman/listinfo/toasters
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
-- --- Gustatus Similis Pullus
_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
You can use stats show -p flexscale-pcs to see an estimate of what the flash cache would pick up.
Jeremy Page|Senior Technical Architect|Gilbarco Veeder-Root, A Danaher Company Office:336-547-5399|Cell:336-601-7274|24x7 Emergency:336-430-8151
________________________________________ From: Steffen Knauf [sknauf@chipxonio.de] Sent: Wednesday, September 26, 2012 11:16 AM To: Page, Jeremy Cc: toasters@teaparty.net Subject: AW: Raidgroupsize and I/O Performance
It ssems that we have a alignment problem, too:
nfsstat: Misaligned Read request stats BIN-0 BIN-1 BIN-2 BIN-3 BIN-4 BIN-5 BIN-6 BIN-7 1096332359 1984142147 16821356 3951254 469217191 6145276 94494489 245873608 Misaligned Write request stats BIN-0 BIN-1 BIN-2 BIN-3 BIN-4 BIN-5 BIN-6 BIN-7 252471989 1287671452 52026962 32535853 278299197 40175418 38565555 692647941
I'll do a reallocate measurement tomorrow.
greets
Steffen
Von: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] Im Auftrag von Jeremy Page Gesendet: Dienstag, 25. September 2012 14:23 An: toasters@teaparty.net Betreff: Re: Raidgroupsize and I/O Performance
If you're considering getting a cache adapter I'd hold off on messing with the RAID config until afterwards - tuning for writes is different than tuning for IO in general since you're looking for free space and generally the more contiguous the better.
Have you done a reallocate measurement, are your VMDK files all aligned? On 09/25/2012 04:19 AM, Steffen Knauf wrote: Hello,
i'll try to improve our I/0 Perfomance. We have a raidgroup with 16 disks (SAS), 1 aggregate and 1 volume (dedup enabled). The volume is the storage for 100 VM's on the VMware Cluser (access via NFS). Does it make sense to increase the raidgroup? 90% Percent of the Disk I/O are Read Ops, so i'll buy a PAM Card for our FAS3240,too.
And what's your Experience with the Raidgroupsize of a Raidgroup with SATA Disks (now: 11+1)?
Thanks and greets !
Steffen
_______________________________________________
Toasters mailing list
Toasters@teaparty.netmailto:Toasters@teaparty.net
http://www.teaparty.net/mailman/listinfo/toasters
Please be advised that this email may contain confidential information. If you are not the intended recipient, please notify us by email by replying to the sender and delete this message. The sender disclaims that the content of this email constitutes an offer to enter into, or the acceptance of, any agreement; provided that the foregoing does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment. Please be advised that this email may contain confidential information. If you are not the intended recipient, please notify us by email by replying to the sender and delete this message. The sender disclaims that the content of this email constitutes an offer to enter into, or the acceptance of, any agreement; provided that the foregoing does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment.
Hi Jeremy,
it seems that a additional Flash Card make sense:
stats show -p flexscale-pcs (Default Mode)
Usage Hit Meta Miss Hit Evict Inval Insert Chain Blocks Chain Blocks Replaced % /s /s /s % /s /s /s /s /s /s /s /s 43 1919 79 1239 60 0 19 4559 0 0 0 0 503 43 2299 0 6238 26 0 0 4199 0 0 0 0 563 43 1979 179 5498 26 0 0 6418 0 0 0 0 593 43 2079 0 3659 36 19 0 5358 0 0 0 0 435 43 2859 0 4379 39 0 0 4839 0 0 0 0 621 43 2499 199 3019 45 19 19 7178 0 0 0 0 774 43 2299 0 3479 39 0 0 5218 0 0 0 0 478 43 2179 159 1119 66 0 0 1419 0 0 0 0 756 43 1899 19 1319 59 0 0 0 0 0 0 0 446 43 2279 0 1979 53 0 0 0 0 0 0 0 490 43 1879 39 1659 53 19 19 4679 0 0 0 0 498 43 2239 0 919 70 0 0 0 0 0 0 0 346 43 3039 99 1379 68 0 0 4139 0 0 0 0 726
greets
Steffen
-----Ursprüngliche Nachricht----- Von: Page, Jeremy [mailto:jeremy.page@gilbarco.com] Gesendet: Mittwoch, 26. September 2012 18:31 An: Steffen Knauf Cc: toasters@teaparty.net Betreff: RE: Raidgroupsize and I/O Performance
You can use stats show -p flexscale-pcs to see an estimate of what the flash cache would pick up.
Jeremy Page|Senior Technical Architect|Gilbarco Veeder-Root, A Danaher Company Office:336-547-5399|Cell:336-601-7274|24x7 Emergency:336-430-8151
________________________________________
Take a look at http://media.netapp.com/documents/tr-3801.pdf first before you add the PAM card to see if it will make a difference in your environment.
On Tue, Sep 25, 2012 at 1:19 AM, Steffen Knauf sknauf@chipxonio.de wrote:
Hello,****
i'll try to improve our I/0 Perfomance. We have a raidgroup with 16 disks (SAS), 1 aggregate and 1 volume (dedup enabled). The volume is the storage for 100 VM's on the VMware Cluser (access via NFS). Does it make sense to increase the raidgroup? 90% Percent of the Disk I/O are Read Ops, so i'll buy a PAM Card for our FAS3240,too.****
And what's your Experience with the Raidgroupsize of a Raidgroup with SATA Disks (now: 11+1)?****
Thanks and greets !****
Steffen****
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
hi,
in the last month we have the following average Values of the 16 Disks (DFM Report) :
Read Ops/sec: 90
write Ops/sec: 4
Throughput Blocks/sec: 410
Busy (%): 30%
We don't have any intensive I/O VM's, but we'll buy a second shelf, too. After starting a the backup process (via NFS), it looks like this:
chip4> sysstat -c 10 -u 1
CPU Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk
ops/s in out read write read write age hit time ty util
13% 2069 10553 54047 54364 0 0 0 3s 89% 0% - 73%
15% 2153 11929 57947 59072 0 0 0 3s 90% 0% - 78%
30% 3717 55428 53787 54260 32 0 0 7s 89% 0% - 80%
14% 1958 8163 54683 54284 0 0 0 6s 92% 0% - 62%
10% 1748 4542 59106 56584 0 0 0 6s 92% 0% - 58%
20% 1120 2840 39477 57432 61508 0 0 2 98% 61% T 87%
18% 1367 4342 51629 52008 57672 0 0 2 98% 100% : 77%
14% 1686 6793 57079 55528 29748 0 0 2 92% 64% : 73%
10% 1390 2445 45892 44252 24 0 0 2 91% 0% - 60%
9% 1522 3233 57545 54080 0 0 0 2 92% 0% - 52%
chip4> sysstat -c 10 1
CPU NFS CIFS HTTP Net kB/s Disk kB/s Tape kB/s Cache
in out read write read write age
11% 2296 0 0 2973 76892 54128 32 0 0 6s
18% 4346 0 0 4197 71055 77452 0 0 0 6s
15% 4373 0 0 2398 77533 65772 0 0 0 6s
20% 2104 0 0 2507 46679 47012 35220 0 0 4
12% 2464 0 0 2348 57280 46576 20636 0 0 4
11% 2699 0 0 3298 60688 43872 8 0 0 4
11% 2317 0 0 3760 55468 48212 24 0 0 4
13% 2940 0 0 3645 66395 50112 0 0 0 6s
12% 3124 0 0 2774 70399 46052 0 0 0 6s
11% 2602 0 0 2888 60654 37088 24 0 0 6s
greets
Steffen
Von: Sto RageC [mailto:netbacker@gmail.com] Gesendet: Mittwoch, 26. September 2012 01:34 An: Steffen Knauf Cc: toasters@teaparty.net Betreff: Re: Raidgroupsize and I/O Performance
Take a look at http://media.netapp.com/documents/tr-3801.pdf first before you add the PAM card to see if it will make a difference in your environment.
On Tue, Sep 25, 2012 at 1:19 AM, Steffen Knauf sknauf@chipxonio.de wrote:
Hello,
i'll try to improve our I/0 Perfomance. We have a raidgroup with 16 disks (SAS), 1 aggregate and 1 volume (dedup enabled). The volume is the storage for 100 VM's on the VMware Cluser (access via NFS). Does it make sense to increase the raidgroup? 90% Percent of the Disk I/O are Read Ops, so i'll buy a PAM Card for our FAS3240,too.
And what's your Experience with the Raidgroupsize of a Raidgroup with SATA Disks (now: 11+1)?
Thanks and greets !
Steffen
_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
-
Jeff Mohler -
Jeremy Page -
Page, Jeremy
-
Sebastian Goetze -
Steffen Knauf -
Sto Rage© -
Tas Peter