Hi,
We have a small metrocluster with 2TB SATA disks in DS14mk2 trays, mostly used for VMware. We desperately need more IOPs, and are tossing up another 14x2TB SATA tray, which would have the benefit of just expanding the aggregate, vs 12 of 600GB 15krpm SAS (half a DS4243), which would give more IOPs but be more admin overhead since we'd have to move VMs amongst the shelves.
Capacity isn't a problem, and when we built the metrocluster the ATTO 6500N fibre bridge wasn't available, so SATA was our only option at the time. We're getting quotes for both options, but does anyone have an opinion on expanding the SATA aggregate vs setting up a new SAS one?
Thanks,
Welcome to the dollars for IOPS game ! Assuming a 20ms response time threshold - 15K RPM SAS can deliver 220 IOPS vs SATAs 44 IOPS (about 5x) So SAS will get you the IOPS your VMs need.
You should also check out where your IOPs are being spent - are you doing any snap mirror operations? de-dup?
http://www.vmadmin.info/2010/07/vmware-and-netapp-deconstructing.html
Also which VMs are hogging the IOPs? You can download a free trial of vCenter Operations and see via the VM IO heat maps which VMs might be moved to local disk on the ESXi hosts (to offload the netapp)
good luck - would like to hear how it turns out!
On May 21, 2012, at 10:12 PM, James Andrewartha wrote:
Hi,
We have a small metrocluster with 2TB SATA disks in DS14mk2 trays, mostly used for VMware. We desperately need more IOPs, and are tossing up another 14x2TB SATA tray, which would have the benefit of just expanding the aggregate, vs 12 of 600GB 15krpm SAS (half a DS4243), which would give more IOPs but be more admin overhead since we'd have to move VMs amongst the shelves.
Capacity isn't a problem, and when we built the metrocluster the ATTO 6500N fibre bridge wasn't available, so SATA was our only option at the time. We're getting quotes for both options, but does anyone have an opinion on expanding the SATA aggregate vs setting up a new SAS one?
Thanks,
-- James Andrewartha Network & Projects Engineer Christ Church Grammar School Claremont, Western Australia Ph. (08) 9442 1757 Mob. 0424 160 877 _______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
On 22/05/12 13:44, Fletcher Cocquyt wrote:
Welcome to the dollars for IOPS game ! Assuming a 20ms response time threshold - 15K RPM SAS can deliver 220 IOPS vs SATAs 44 IOPS (about 5x) So SAS will get you the IOPS your VMs need.
We've got a 256GB flashcache, and you can tell it's working, as the second operation on data is quick.
Here's the report from my performance case:
Perfstat iteration 4 demonstrates the highest NFS latency - read latency is around 20-40ms and write latency around 15-25ms. That is pretty high.
Aggr1 plex0 is the only aggregate showing activity. It is extremely busy reporting about 250 operations per disks, peaking at 507!
That is extremely high activity for7.2K RPM disks.
In iteration 4, 16% of the writes are misaligned. Fixing this will make disk activity more efficient and we should see fewer operations per disk.
I've moved the worst of the misaligned data off to a CIFS volume on the partner.
You should also check out where your IOPs are being spent - are you doing any snap mirror operations? de-dup?
No snapmirror, but OnCommand Host VM backups every two hours, with snapvault overnight. Dedup is run at midnight and saving about 30%.
http://www.vmadmin.info/2010/07/vmware-and-netapp-deconstructing.html
Looking at our top 5 disks by disk_busy, they're sitting above 75% a lot of the time, peaking at 95%. There's 11 disks in the raid group, a background noise of 100 IOPs from Exchange (iSCSI), and 250 from VMware (NFS v3), so that's soaking up most of the 400 available from our disks. During VM snapshotting there's a few thousand.
Also which VMs are hogging the IOPs? You can download a free trial of vCenter Operations and see via the VM IO heat maps which VMs might be moved to local disk on the ESXi hosts (to offload the netapp)
I do have a vCOPs standard license, and it was a bit useful but also a bit hard to understand. I've moved some of our internal VMs with high IO to the other side of the metrocluster. but we have a new SQL application coming into production soon and it must perform well.
good luck - would like to hear how it turns out!
I'm yet to get pricing, but I'm thinking SAS is the way to go.
On 2012-05-22 08:49, James Andrewartha wrote:
Aggr1 plex0 is the only aggregate showing activity.
Please make sure to set the option raid.read_plex_pref to 'alternate'. That should give you a boost in IOPS already by spreading the load over all the disks in a syncmirror, not only half of them.
Regards, Oliver
On 22/05/12 13:44, Fletcher Cocquyt wrote:
Welcome to the dollars for IOPS game ! Assuming a 20ms response time threshold - 15K RPM SAS can deliver 220 IOPS vs SATAs 44 IOPS (about 5x) So SAS will get you the IOPS your VMs need.
It turned out that two DS14mk4 trays of 600GB FC was not much more than two DS14mk2 trays of 2TB SATA. SAS turned out to be way cheaper than 12x600GB 15krpm SAS disks, I think mostly due to the cost of the ATTO 6500N FC-SAS bridges required for a metrocluster. It's not actually installed yet, so I can't give you actual performance comparisons.
Speaking of metrocluster and SyncMirror, I'm going to reassign the old SATA disks to the other head and add them to its SATA aggregate. What's the best way to reallocate volumes that are being SyncMirrored?
Also which VMs are hogging the IOPs? You can download a free trial of vCenter Operations and see via the VM IO heat maps which VMs might be moved to local disk on the ESXi hosts (to offload the netapp)
While waiting for the disk to arrive, we put 6x 256GB OCZ Vertex 4 SSDs into a RAID 5 and put the most important SQL server on that, which is performing adequately ;) Since we don't use SMSQL I'm just using OSSV to backup the native SQL backups.
Having used vCOps for a while, I'm don't think I'd pay retail price for it.
Hi, Adding SAS Disk is ok but expensive :(. Another way is using PAM Card, it's cheaper.
Terimakasih, EKO HARYONO
-----Original Message----- From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of James Andrewartha Sent: Tuesday, May 22, 2012 12:12 PM To: toasters@teaparty.net Subject: Expand SATA aggregate or buy SAS disks
Hi,
We have a small metrocluster with 2TB SATA disks in DS14mk2 trays, mostly used for VMware. We desperately need more IOPs, and are tossing up another 14x2TB SATA tray, which would have the benefit of just expanding the aggregate, vs 12 of 600GB 15krpm SAS (half a DS4243), which would give more IOPs but be more admin overhead since we'd have to move VMs amongst the shelves.
Capacity isn't a problem, and when we built the metrocluster the ATTO 6500N fibre bridge wasn't available, so SATA was our only option at the time. We're getting quotes for both options, but does anyone have an opinion on expanding the SATA aggregate vs setting up a new SAS one?
Thanks,
-
Fletcher Cocquyt -
IT.SVC.DC.g.Eko Haryono -
James Andrewartha -
Oliver Brakmann