Nfsstat is not an imperfect tool, nor is mbralign...
You should zero your counters for nfsstat after alignment (nfsstat -z) and let the aligned VM run for a while (24 hours) before checking again using nfsstat -d. You are still dragging the misaligned status in the counters from before the alignment on your filer.
Kind regards,
Peter.
-----Original Message----- From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Jeff Mohler Sent: donderdag 18 oktober 2012 10:46 To: Steffen Knauf Cc: toasters@teaparty.net Subject: Re: Raidgroupsize and I/O Performance
" I find a aligned VM in the list of "Files Causing Misaligned IO's" (nfsstat -d) again."
It's an imperfect tool..just guessing, does the "misaligned" aligned VM, have a database _log_ file being written to within it?
On Thu, Oct 18, 2012 at 1:40 AM, Steffen Knauf sknauf@chipxonio.de wrote:
Hi Sebastian,
i'm sure, because i used the "nfsstat -d" Option. But there is one strange thing. If i try to rescan an aligned VM, i get a "Failed to open XXX-flat.vmdk - [Device or resource busy]" . A Scan is only possible if the VM is powered off (VSC and cmd line). And there is another strange thing. I find a aligned VM in the list of "Files Causing Misaligned IO's" (nfsstat -d) again.
greets
Steffen
Von: Sebastian Goetze [mailto:spgoetze@gmail.com] Gesendet: Mittwoch, 17. Oktober 2012 19:47
An: Steffen Knauf Cc: 'Jeremy Page'; toasters@teaparty.net Betreff: Re: Raidgroupsize and I/O Performance
Hi Steffen,
are you really sure the VMs are aligned well?
Best way to check IMO is nfsstat -d:
MISALIGNED I/O A NFS read or write is termed misaligned when its length is a multiple of 4K and it's starting offset is not aligned to a 4K boundary. A large number of misaligned reads and writes will have an adverse impact on the performance of the filer. NFS reads and writes are binned into eight bins (BIN-0 to BIN-7). All read and write requests from BIN-1 to BIN-7 are misaligned. A list of filenames that caused the most misaligned I/O's is displayed with the -d option.
If you're using VSC 4, you might also be able to see which VMs are mis-aligned. The Optimization and Migration capability lets you scan datastores and correct the alignment of certain misaligned virtual machines (VM) without having to power down the VM.
These two methods check the actual read/write patterns, so it's a lot safer, than to check the theory only. (E.g. I've seen people 'aligning' MBR VMDKs that were created using Windows 2008...)
Greetings
Sebastian
On 17.10.2012 17:43, Steffen Knauf wrote:
The latency on nfs storage seems to be fine :
Instance nfsv3_ops nfsv3_read_l nfsv3_read_o nfsv3_write_ nfsv3_write_
/s ms /s ms /s nfs 1021 3.52 222 0.18 795 nfs 1189 3.03 210 0.29 976 nfs 1138 6.29 221 0.47 908 nfs 1447 2.80 220 0.11 1220 nfs 2226 6.19 207 0.33 2016 nfs 2180 3.64 191 0.41 1985 nfs 1706 6.04 199 5.47 1498 nfs 2388 3.31 236 0.23 2146 nfs 2366 7.47 287 3.35 2075 nfs 1623 5.79 194 1.26 1426stats show -i3 volume:vol_vm1:read_latency volume:vol_vm1:write_latency
Instance read_latency write_latenc
us usvol_vm1 9720.45 24395.98
vol_vm1 6743.96 932.14
vol_vm1 4996.47 183.15
vol_vm1 5898.92 333.17
vol_vm1 8366.29 320.44
vol_vm1 20432.34 1644.11
vol_vm1 8089.78 1071.03
vol_vm1 15534.60 13517.11
vol_vm1 8862.75 2014.93
vol_vm1 4092.92 358.95
vol_vm1 8605.04 1528.93
vol_vm1 11640.06 3673.86
vol_vm1 5593.18 467.98
vol_vm1 5007.53 211.68
vol_vm1 11465.33 681.64
vol_vm1 8117.90 714.84
vol_vm1 6328.100 416.68
vol_vm1 6368.60 687.42
vol_vm1 7068.68 1430.92
vol_vm1 16762.75 1071.98
vol_vm1 7867.80 915.64
vol_vm1 4484.89 336.83
I think a latency <25ms is ok? Sometimes i have some spikes:
vol_vm1 36878.74 13100.62
vol_vm1 22711.02 12180.19
vol_vm1 10190.43 14667.47
vol_vm1 72765.70 65223.42
vol_vm1 25004.14 7689.66
vol_vm1 23907.21 1882.91
After aligning a VM i took this VM to create a template. During the the OS Installation i got nfs timeouts (pxe) and the Installation is really slow (if the VM virtual disk latency graph is correct, it is more than 300 ms, but the ). If i don't take the template to create a new VM i don't have any trouble. After creating a new template of an other aligned VM i don't have any trouble. It looks like a VMware problem.............grrr ;(
Thanks for help!
greets
Steffen
Von: Jeremy Page [mailto:jeremy.page@gilbarco.com] Gesendet: Mittwoch, 17. Oktober 2012 15:54 An: Steffen Knauf Cc: toasters@teaparty.net Betreff: Re: AW: Raidgroupsize and I/O Performance
How are you mapping the latency? On the filer stats show -i5 nfsv3 will give you good info, on the VM itself are you using iostat or ?
It does not make sense, aligning should never cause increased latency unless something else odd is going on.
Is it write or read latency that's increased (or both)? Do you have atime updates disabled on your volume options (which can cause "other_ops" to increase, although it shouldn't change the latency).
Jeremy Page | Gilbarco Veeder-Root, A Danaher Company Senior Technical Architect | IS Infrastructure Systems | Yahoo IM: jeremypage Office:336-547-5399 | Cell: 336-601-7274 | 24x7 Emergency: 336-430-8151
On 10/17/2012 09:49 AM, Steffen Knauf wrote:
Hi,
a small update. After aligning some VM's, a couple of them are slower (disk latency is higher than 200ms). The NFS Storage, where the *.vmdk's are stored has a different lateny than the disk. Strange........
./mbralign --sparse /vmfs/volumes/chip4-vmware_data_nfs01/web02/web02-flat.vmdk
Part Type old LBA New Start LBA New End LBA Length in KB
P1 83 63 64 401626 200781
P2 82 401625 401640 4482150 2040255
P3 83 4482135 4482160 314568790 155043315
Any Ideas?
greets
Steffen
Von: Jeremy Page [mailto:jeremy.page@gilbarco.com] Gesendet: Freitag, 28. September 2012 17:11 An: Steffen Knauf Cc: 'Jeff Mohler'; toasters@teaparty.net Betreff: Re: AW: AW: Raidgroupsize and I/O Performance
Aligning your VMDK files is the first thing I'd do for certain. As was mentioned on other messages you should verify that the PAM cards will help so you can evaluate cost/benefit. For us they worked wonders but if you are write constrained then all the cache in the world is not going to help.
On 09/28/2012 11:05 AM, Steffen Knauf wrote:
Yes that's true. A strange thing is that VSC (4.0) and mbrscan show me different results:
/opt/ontap # ./mbrscan /vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.v mdk
mbrtools esxi version 1.0
/vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk p1 (Linux) lba:64 offset:32768 aligned:Yes
/vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk p2 (swap) lba:401640 offset:205639680 aligned:Yes
/vmfs/volumes/chip4-vmware_data_nfs01/openx-test01/openx-test01-flat.vmdk p3 (Linux) lba:16771888 offset:8587206656 aligned:Yes
I think i can delete the backup file?:
[Counter=2891433], Filename=vol_vm1/openx-test01/openx-test01-flat.vmdk-mbralign-backup
I rescan the datastores via VSC and get the result that openx-test01 is misaligned
And some mbrscans are not possible:
./mbrscan /vmfs/volumes/chip4-vmware_data_nfs01/openx-test03/openx-test03-flat.v mdk
Failed to open /vmfs/volumes/chip4-vmware_data_nfs01/openx-test03/openx-test03-flat.v mdk - [Device or resource busy]
Thanks for your help and have a nice weekend. I'm on holiday for 1 Week, so my response could be delayed.....
greets
Steffen
Please be advised that this email may contain confidential information. If you are not the intended recipient, please notify us by email by replying to the sender and delete this message. The sender disclaims that the content of this email constitutes an offer to enter into, or the acceptance of, any agreement; provided that the foregoing does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment.
Toasters mailing list
Toasters@teaparty.net
http://www.teaparty.net/mailman/listinfo/toasters
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
-- --- Gustatus Similis Pullus
_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters