Hi Steffen,
are you _really_ sure the VMs are aligned well?
Best way to check IMO is *nfsstat -d*:
MISALIGNED I/O A NFS read or write is termed misaligned when its length is a multiple of 4K and it's starting offset is not aligned to a 4K boundary. A large number of misaligned reads and writes will have an adverse impact on the performance of the filer. NFS reads and writes are binned into eight bins (BIN-0 to BIN-7). All read and write requests from BIN-1 to BIN-7 are misaligned. *A list of filenames that caused** **the most misaligned I/O's is displayed with the -d option.*
If you're using *VSC 4*, you might also be able to see which VMs are mis-aligned. The *Optimization and Migration capability* lets you scan datastores and correct the alignment of certain misaligned virtual machines (VM) without having to power down the VM.
These two methods check the *actual *read/write patterns, so it's a lot safer, than to check the theory only. (E.g. I've seen people 'aligning' MBR VMDKs that were created using Windows _2008_...)
Greetings
Sebastian
On 17.10.2012 17:43, Steffen Knauf wrote:
The latency on nfs storage seems to be fine :
Instance nfsv3_ops nfsv3_read_l nfsv3_read_o nfsv3_write_ nfsv3_write_
/s ms /s ms /s
nfs 1021 3.52 222 0.18 795
nfs 1189 3.03 210 0.29 976
nfs 1138 6.29 221 0.47 908 nfs 1447 2.80 220 0.11 1220 nfs 2226 6.19 207 0.33 2016 nfs 2180 3.64 191 0.41 1985 nfs 1706 6.04 199 5.47 1498 nfs 2388 3.31 236 0.23 2146 nfs 2366 7.47 287 3.35 2075 nfs 1623 5.79 194 1.26 1426stats show -i3 volume:vol_vm1:read_latency volume:vol_vm1:write_latency
Instance read_latency write_latenc
us us
vol_vm1 9720.45 24395.98
vol_vm1 6743.96 932.14
vol_vm1 4996.47 183.15
vol_vm1 5898.92 333.17
vol_vm1 8366.29 320.44
vol_vm1 20432.34 1644.11
vol_vm1 8089.78 1071.03
vol_vm1 15534.60 13517.11
vol_vm1 8862.75 2014.93
vol_vm1 4092.92 358.95
vol_vm1 8605.04 1528.93
vol_vm1 11640.06 3673.86
vol_vm1 5593.18 467.98
vol_vm1 5007.53 211.68
vol_vm1 11465.33 681.64
vol_vm1 8117.90 714.84
vol_vm1 6328.100 416.68
vol_vm1 6368.60 687.42
vol_vm1 7068.68 1430.92
vol_vm1 16762.75 1071.98
vol_vm1 7867.80 915.64
vol_vm1 4484.89 336.83
I think a latency <25ms is ok? Sometimes i have some spikes:
vol_vm1 36878.74 13100.62
vol_vm1 22711.02 12180.19
vol_vm1 10190.43 14667.47
vol_vm1 72765.70 65223.42
vol_vm1 25004.14 7689.66
vol_vm1 23907.21 1882.91
After aligning a VM i took this VM to create a template. During the the OS Installation i got nfs timeouts (pxe) and the Installation is really slow (if the VM virtual disk latency graph is correct, it is more than 300 ms, but the ). If i don't take the template to create a new VM i don't have any trouble. After creating a new template of an other aligned VM i don't have any trouble. It looks like a VMware problem.............grrr ;(
Thanks for help!
greets
Steffen
*Von:*Jeremy Page [mailto:jeremy.page@gilbarco.com] *Gesendet:* Mittwoch, 17. Oktober 2012 15:54 *An:* Steffen Knauf *Cc:* toasters@teaparty.net *Betreff:* Re: AW: Raidgroupsize and I/O Performance
How are you mapping the latency? On the filer stats show -i5 nfsv3 will give you good info, on the VM itself are you using iostat or ?
It does not make sense, aligning should never cause increased latency unless something else odd is going on.
Is it write or read latency that's increased (or both)? Do you have atime updates disabled on your volume options (which can cause "other_ops" to increase, although it shouldn't change the latency).
*Jeremy Page*| Gilbarco Veeder-Root, A Danaher Company Senior Technical Architect | IS Infrastructure Systems | Yahoo IM: jeremypage Office:336-547-5399 | Cell: 336-601-7274 | 24x7 Emergency: 336-430-8151
On 10/17/2012 09:49 AM, Steffen Knauf wrote:
Hi, a small update. After aligning some VM's, a couple of them are slower (disk latency is higher than 200ms). The NFS Storage, where the *.vmdk's are stored has a different lateny than the disk. Strange........ ./mbralign --sparse /vmfs/volumes/chip4-vmware_data_nfs01/web02/web02-flat.vmdk Part Type old LBA New Start LBA New End LBA Length in KB P1 83 63 64 401626 200781 P2 82 401625 401640 4482150 2040255 P3 83 4482135 4482160 314568790 155043315 Any Ideas? greets Steffen *Von:*Jeremy Page [mailto:jeremy.page@gilbarco.com] *Gesendet:* Freitag, 28. September 2012 17:11 *An:* Steffen Knauf *Cc:* 'Jeff Mohler'; toasters@teaparty.net <mailto:toasters@teaparty.net> *Betreff:* Re: AW: AW: Raidgroupsize and I/O Performance Aligning your VMDK files is the first thing I'd do for certain. As was mentioned on other messages you should verify that the PAM cards will help so you can evaluate cost/benefit. For us they worked wonders but if you are write constrained then all the cache in the world is not going to help. On 09/28/2012 11:05 AM, Steffen Knauf wrote: Yes that's true. A strange thing is that VSC (4.0) and mbrscan show me different results: /opt/ontap # ./mbrscan /vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk mbrtools esxi version 1.0 -------------------- /vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk p1 (Linux) lba:64 offset:32768 aligned:Yes /vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk p2 (swap) lba:401640 offset:205639680 aligned:Yes /vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk p3 (Linux) lba:16771888 offset:8587206656 aligned:Yes I think i can delete the backup file?: [Counter=2891433], Filename=vol_vm1/openx-test01/openx-test01-flat.vmdk-mbralign-backup I rescan the datastores via VSC and get the result that openx-test01 is misaligned And some mbrscans are not possible: ./mbrscan /vmfs/volumes/chip4-vmware_data_nfs01/openx-test03/openx-test03-flat.vmdk Failed to open /vmfs/volumes/chip4-vmware_data_nfs01/openx-test03/openx-test03-flat.vmdk - [Device or resource busy] Thanks for your help and have a nice weekend. I'm on holiday for 1 Week, so my response could be delayed..... greets SteffenPlease be advised that this email may contain confidential information. If you are not the intended recipient, please notify us by email by replying to the sender and delete this message. The sender disclaims that the content of this email constitutes an offer to enter into, or the acceptance of, any agreement; provided that the foregoing does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment.
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters