Hi there
We have an A300 connected to two FC fabrics with 2 x 8G FC per controller node. We are doing tests on an “older” HP C7000 with VirtualConnect modules and with blades that have 2 x 8G FC mezzanine. The VCs are connected into the fabric with two 8G FC per fabric. So the throughput to a given LUN would be 16Gbps. As the other two links are standby. The blade is running ESXi 7.0 and on top of this we are running a virtual Windows 2016 server from where the tests are done. There are no one else on the A300. We hav created a standard LUN in a volume, presented it to ESXi from where we created a VMFS and the VM is then presented with a VMDK-based disk.
It seems that no matter what we adjust we have a limit at about 70.000 IOPs on the disks. We are able to run other workloads while we are testing with 70.000 IOPs, and we are able to load the system even more, so the NetApp does not seem to be the limiting factor… we can also not see that much load on the system from the NetApp Harvest/Grafana we have running… We of cause make sure that the tests we do is not done in cache on the host… The specific test is with 64k block size.
Questions:
* Can there be a limit of IOPs in the C7000 setup? PCI-bridge stuff? * Does it make sense to add more paths, in either the NetApp and/or the C7000? (each blade can only have two ports) * Would it make sense to upgrade the FC links to 16G? * We have adjusted the queue depth as per NetApp’s recommendations, as far as I know there is no way to adjust QD on the ONTAP node itself, is this correct? * The shelf attached is IOM6 based, would it make sense to upgrade this to IOM12? * Would it make sense to cable the IOM12 in a Quad-Path setup?
Any suggestions are welcome… also if you have any insights of how many IOPs one should expect from an A300 with 24 x 3.7T SSDs.
/Heino --
Heino,
It might be your 8Gb FC links that are the bottleneck. You'd have to check the switch/host stats to verify. By my math, 70,000 IOPS at 64kB IO size is 4,480,000 kB/s throughput. Converted to Gbit/s, that's 34.18Gbit/s, which is more than your 2x 8Gb links at the host. You did not mention the read/write ratio of your tests. Is it 50/50? If you had 35k read and 35k write IOPS, that's more like 17Gb/s in each direction, which is still over the max of your 2x 8Gb links. If the IO sizes are mostly smaller than 64kB, then this would be functioning.
Check the performance on the switches and host posts and see how saturated they are during the test. You might also be able to add more paths of 8Gb, if that's all you have. Increase the number of connections on the host to 4 and 8 connections to the filer.
Regards,
Brian Jones
On Thu, Oct 29, 2020 at 7:26 AM Heino Walther hw@beardmann.dk wrote:
Hi there
We have an A300 connected to two FC fabrics with 2 x 8G FC per controller node.
We are doing tests on an “older” HP C7000 with VirtualConnect modules and with blades that have 2 x 8G FC mezzanine.
The VCs are connected into the fabric with two 8G FC per fabric.
So the throughput to a given LUN would be 16Gbps. As the other two links are standby.
The blade is running ESXi 7.0 and on top of this we are running a virtual Windows 2016 server from where the tests are done.
There are no one else on the A300. We hav created a standard LUN in a volume, presented it to ESXi from where we created a VMFS and the VM is then presented with a VMDK-based disk.
It seems that no matter what we adjust we have a limit at about 70.000 IOPs on the disks.
We are able to run other workloads while we are testing with 70.000 IOPs, and we are able to load the system even more, so the NetApp does not seem to be the limiting factor… we can also not see that much load on the system from the NetApp Harvest/Grafana we have running…
We of cause make sure that the tests we do is not done in cache on the host…
The specific test is with 64k block size.
Questions:
- Can there be a limit of IOPs in the C7000 setup? PCI-bridge stuff?
- Does it make sense to add more paths, in either the NetApp and/or
the C7000? (each blade can only have two ports)
- Would it make sense to upgrade the FC links to 16G?
- We have adjusted the queue depth as per NetApp’s recommendations, as
far as I know there is no way to adjust QD on the ONTAP node itself, is this correct?
- The shelf attached is IOM6 based, would it make sense to upgrade
this to IOM12?
- Would it make sense to cable the IOM12 in a Quad-Path setup?
Any suggestions are welcome… also if you have any insights of how many IOPs one should expect from an A300 with 24 x 3.7T SSDs.
/Heino
--
Toasters mailing list Toasters@teaparty.net https://www.teaparty.net/mailman/listinfo/toasters
Hi Brian
Even with 4k blocks 100% read we only get the 70.000 IOPs… As this blade is the only one using the NetApp and it only have 2 x 8G FC HBAs (which cannot be expanded), it might not make much sense adding more or faster paths to the NetApp…
/Heino
Fra: Toasters toasters-bounces@teaparty.net Dato: torsdag, 29. oktober 2020 kl. 14.32 Til: Toasters toasters@teaparty.net Emne: Re: IOPs limit? Heino,
It might be your 8Gb FC links that are the bottleneck. You'd have to check the switch/host stats to verify. By my math, 70,000 IOPS at 64kB IO size is 4,480,000 kB/s throughput. Converted to Gbit/s, that's 34.18Gbit/s, which is more than your 2x 8Gb links at the host. You did not mention the read/write ratio of your tests. Is it 50/50? If you had 35k read and 35k write IOPS, that's more like 17Gb/s in each direction, which is still over the max of your 2x 8Gb links. If the IO sizes are mostly smaller than 64kB, then this would be functioning.
Check the performance on the switches and host posts and see how saturated they are during the test. You might also be able to add more paths of 8Gb, if that's all you have. Increase the number of connections on the host to 4 and 8 connections to the filer.
Regards, Brian Jones
On Thu, Oct 29, 2020 at 7:26 AM Heino Walther <hw@beardmann.dkmailto:hw@beardmann.dk> wrote: Hi there
We have an A300 connected to two FC fabrics with 2 x 8G FC per controller node. We are doing tests on an “older” HP C7000 with VirtualConnect modules and with blades that have 2 x 8G FC mezzanine. The VCs are connected into the fabric with two 8G FC per fabric. So the throughput to a given LUN would be 16Gbps. As the other two links are standby. The blade is running ESXi 7.0 and on top of this we are running a virtual Windows 2016 server from where the tests are done. There are no one else on the A300. We hav created a standard LUN in a volume, presented it to ESXi from where we created a VMFS and the VM is then presented with a VMDK-based disk.
It seems that no matter what we adjust we have a limit at about 70.000 IOPs on the disks. We are able to run other workloads while we are testing with 70.000 IOPs, and we are able to load the system even more, so the NetApp does not seem to be the limiting factor… we can also not see that much load on the system from the NetApp Harvest/Grafana we have running… We of cause make sure that the tests we do is not done in cache on the host… The specific test is with 64k block size.
Questions:
* Can there be a limit of IOPs in the C7000 setup? PCI-bridge stuff? * Does it make sense to add more paths, in either the NetApp and/or the C7000? (each blade can only have two ports) * Would it make sense to upgrade the FC links to 16G? * We have adjusted the queue depth as per NetApp’s recommendations, as far as I know there is no way to adjust QD on the ONTAP node itself, is this correct? * The shelf attached is IOM6 based, would it make sense to upgrade this to IOM12? * Would it make sense to cable the IOM12 in a Quad-Path setup?
Any suggestions are welcome… also if you have any insights of how many IOPs one should expect from an A300 with 24 x 3.7T SSDs.
/Heino --
_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net https://www.teaparty.net/mailman/listinfo/toasters
Where do you measure the 70k iops? Inside of the virtual machine? Directly on the storage?
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.comhttp://www.anexia-it.com/
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Von: Toasters toasters-bounces@teaparty.net Im Auftrag von Heino Walther Gesendet: Donnerstag, 29. Oktober 2020 14:41 An: Brian Jones bjones@aggiejones.com; Toasters toasters@teaparty.net Betreff: SV: IOPs limit?
Hi Brian
Even with 4k blocks 100% read we only get the 70.000 IOPs... As this blade is the only one using the NetApp and it only have 2 x 8G FC HBAs (which cannot be expanded), it might not make much sense adding more or faster paths to the NetApp...
/Heino
Fra: Toasters <toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net> Dato: torsdag, 29. oktober 2020 kl. 14.32 Til: Toasters <toasters@teaparty.netmailto:toasters@teaparty.net> Emne: Re: IOPs limit? Heino,
It might be your 8Gb FC links that are the bottleneck. You'd have to check the switch/host stats to verify. By my math, 70,000 IOPS at 64kB IO size is 4,480,000 kB/s throughput. Converted to Gbit/s, that's 34.18Gbit/s, which is more than your 2x 8Gb links at the host. You did not mention the read/write ratio of your tests. Is it 50/50? If you had 35k read and 35k write IOPS, that's more like 17Gb/s in each direction, which is still over the max of your 2x 8Gb links. If the IO sizes are mostly smaller than 64kB, then this would be functioning.
Check the performance on the switches and host posts and see how saturated they are during the test. You might also be able to add more paths of 8Gb, if that's all you have. Increase the number of connections on the host to 4 and 8 connections to the filer.
Regards, Brian Jones
On Thu, Oct 29, 2020 at 7:26 AM Heino Walther <hw@beardmann.dkmailto:hw@beardmann.dk> wrote: Hi there
We have an A300 connected to two FC fabrics with 2 x 8G FC per controller node. We are doing tests on an "older" HP C7000 with VirtualConnect modules and with blades that have 2 x 8G FC mezzanine. The VCs are connected into the fabric with two 8G FC per fabric. So the throughput to a given LUN would be 16Gbps. As the other two links are standby. The blade is running ESXi 7.0 and on top of this we are running a virtual Windows 2016 server from where the tests are done. There are no one else on the A300. We hav created a standard LUN in a volume, presented it to ESXi from where we created a VMFS and the VM is then presented with a VMDK-based disk.
It seems that no matter what we adjust we have a limit at about 70.000 IOPs on the disks. We are able to run other workloads while we are testing with 70.000 IOPs, and we are able to load the system even more, so the NetApp does not seem to be the limiting factor... we can also not see that much load on the system from the NetApp Harvest/Grafana we have running... We of cause make sure that the tests we do is not done in cache on the host... The specific test is with 64k block size.
Questions:
* Can there be a limit of IOPs in the C7000 setup? PCI-bridge stuff? * Does it make sense to add more paths, in either the NetApp and/or the C7000? (each blade can only have two ports) * Would it make sense to upgrade the FC links to 16G? * We have adjusted the queue depth as per NetApp's recommendations, as far as I know there is no way to adjust QD on the ONTAP node itself, is this correct? * The shelf attached is IOM6 based, would it make sense to upgrade this to IOM12? * Would it make sense to cable the IOM12 in a Quad-Path setup?
Any suggestions are welcome... also if you have any insights of how many IOPs one should expect from an A300 with 24 x 3.7T SSDs.
/Heino --
_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net https://www.teaparty.net/mailman/listinfo/toasters
We see it in ”lun stat show…” on the NetApp..
/Heino
Fra: Alexander Griesser AGriesser@anexia-it.com Dato: torsdag, 29. oktober 2020 kl. 14.41 Til: Heino Walther hw@beardmann.dk, Brian Jones bjones@aggiejones.com, Toasters toasters@teaparty.net Emne: AW: IOPs limit? Where do you measure the 70k iops? Inside of the virtual machine? Directly on the storage?
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.comhttp://www.anexia-it.com/
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Von: Toasters toasters-bounces@teaparty.net Im Auftrag von Heino Walther Gesendet: Donnerstag, 29. Oktober 2020 14:41 An: Brian Jones bjones@aggiejones.com; Toasters toasters@teaparty.net Betreff: SV: IOPs limit?
Hi Brian
Even with 4k blocks 100% read we only get the 70.000 IOPs… As this blade is the only one using the NetApp and it only have 2 x 8G FC HBAs (which cannot be expanded), it might not make much sense adding more or faster paths to the NetApp…
/Heino
Fra: Toasters <toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net> Dato: torsdag, 29. oktober 2020 kl. 14.32 Til: Toasters <toasters@teaparty.netmailto:toasters@teaparty.net> Emne: Re: IOPs limit? Heino,
It might be your 8Gb FC links that are the bottleneck. You'd have to check the switch/host stats to verify. By my math, 70,000 IOPS at 64kB IO size is 4,480,000 kB/s throughput. Converted to Gbit/s, that's 34.18Gbit/s, which is more than your 2x 8Gb links at the host. You did not mention the read/write ratio of your tests. Is it 50/50? If you had 35k read and 35k write IOPS, that's more like 17Gb/s in each direction, which is still over the max of your 2x 8Gb links. If the IO sizes are mostly smaller than 64kB, then this would be functioning.
Check the performance on the switches and host posts and see how saturated they are during the test. You might also be able to add more paths of 8Gb, if that's all you have. Increase the number of connections on the host to 4 and 8 connections to the filer.
Regards, Brian Jones
On Thu, Oct 29, 2020 at 7:26 AM Heino Walther <hw@beardmann.dkmailto:hw@beardmann.dk> wrote: Hi there
We have an A300 connected to two FC fabrics with 2 x 8G FC per controller node. We are doing tests on an “older” HP C7000 with VirtualConnect modules and with blades that have 2 x 8G FC mezzanine. The VCs are connected into the fabric with two 8G FC per fabric. So the throughput to a given LUN would be 16Gbps. As the other two links are standby. The blade is running ESXi 7.0 and on top of this we are running a virtual Windows 2016 server from where the tests are done. There are no one else on the A300. We hav created a standard LUN in a volume, presented it to ESXi from where we created a VMFS and the VM is then presented with a VMDK-based disk.
It seems that no matter what we adjust we have a limit at about 70.000 IOPs on the disks. We are able to run other workloads while we are testing with 70.000 IOPs, and we are able to load the system even more, so the NetApp does not seem to be the limiting factor… we can also not see that much load on the system from the NetApp Harvest/Grafana we have running… We of cause make sure that the tests we do is not done in cache on the host… The specific test is with 64k block size.
Questions:
* Can there be a limit of IOPs in the C7000 setup? PCI-bridge stuff? * Does it make sense to add more paths, in either the NetApp and/or the C7000? (each blade can only have two ports) * Would it make sense to upgrade the FC links to 16G? * We have adjusted the queue depth as per NetApp’s recommendations, as far as I know there is no way to adjust QD on the ONTAP node itself, is this correct? * The shelf attached is IOM6 based, would it make sense to upgrade this to IOM12? * Would it make sense to cable the IOM12 in a Quad-Path setup?
Any suggestions are welcome… also if you have any insights of how many IOPs one should expect from an A300 with 24 x 3.7T SSDs.
/Heino --
_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net https://www.teaparty.net/mailman/listinfo/toasters
Of cause it is “statistics lun show”…
Fra: Toasters toasters-bounces@teaparty.net Dato: torsdag, 29. oktober 2020 kl. 14.47 Til: Alexander Griesser AGriesser@anexia-it.com, Brian Jones bjones@aggiejones.com, Toasters toasters@teaparty.net Emne: SV: IOPs limit? We see it in ”lun stat show…” on the NetApp..
/Heino
-
Alexander Griesser -
Brian Jones -
Heino Walther