I need help, I cannot figure out how to troubleshoot this problem further.
We have a rebranded 3070A with ~200 VMs running on it. Maybe 30 on a batch of FC disks and the rest on 2 aggrs of 33 SATA drives each. None of the VMs are disk IO intensive. When I try and copy a VM to a new volume I get EXTREMELY slow speeds, something on the order of 3 gig an hour. I am also seeing incredibly slow Snapvault transfers and initializations. I think that in part we are hitting the sequential read bug, but this occurs even on volumes that do not have SIS enabled. For instance I have a snapvault job running that's been going for 4 days (!) and only xfered 260 gig. And it's an update so should always be hitting data (and thus not having the bug).
Statit shows the drives all under 15% utilization. It's over dual 10gigE connections with the preferred adapter set (there are also 2 1 gig VIFs on this box for user and admin access).
I'm at a loss. IBM says we need to upgrade (we are at 7.2.4) but I am concerned that it's not going to fix our problem and downtime windows are difficult to get so I want to make sure it's resolved with as little interruption as possible. Any suggestions on troubleshooting?
Please be advised that this email may contain confidential information. If you are not the intended recipient, please do not read, copy or re-transmit this email. If you have received this email in error, please notify us by email by replying to the sender and by telephone (call us collect at +1 202-828-0850) and delete this message and any attachments. Thank you in advance for your cooperation and assistance.
In addition, Danaher and its subsidiaries disclaim that the content of this email constitutes an offer to enter into, or the acceptance of, any contract or agreement or any amendment thereto; provided that the foregoing disclaimer does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment to this email.
Jeremy,
I'd agree with IBM's analysis (and it's not often I say that).
We had performance issues on a filer with dedup enabled (and not always on the dedup volumes) that seem to have been fixed by upgrading to 7.2.5.1 which is now the Netapp recommended minimum if you're using dedup.
I'm told 7.2.6.1 has been released which will fix the sequential read bug too.
Darren.
________________________________
From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Page, Jeremy Sent: 21 November 2008 15:33 To: toasters@mathworks.com Subject: Very slow large file operations
I need help, I cannot figure out how to troubleshoot this problem further.
We have a rebranded 3070A with ~200 VMs running on it. Maybe 30 on a batch of FC disks and the rest on 2 aggrs of 33 SATA drives each. None of the VMs are disk IO intensive. When I try and copy a VM to a new volume I get EXTREMELY slow speeds, something on the order of 3 gig an hour. I am also seeing incredibly slow Snapvault transfers and initializations. I think that in part we are hitting the sequential read bug, but this occurs even on volumes that do not have SIS enabled. For instance I have a snapvault job running that's been going for 4 days (!) and only xfered 260 gig. And it's an update so should always be hitting data (and thus not having the bug).
Statit shows the drives all under 15% utilization. It's over dual 10gigE connections with the preferred adapter set (there are also 2 1 gig VIFs on this box for user and admin access).
I'm at a loss. IBM says we need to upgrade (we are at 7.2.4) but I am concerned that it's not going to fix our problem and downtime windows are difficult to get so I want to make sure it's resolved with as little interruption as possible. Any suggestions on troubleshooting?
To report this email as spam click here https://www.mailcontrol.com/sr/zWPh+74n8KTTndxI!oX7UpDBHGT3onM6072OTIrs vVit97YGlgZZEsRBLbU6zIXsS4gKezR7S7W1yubdBaCdqw== .
Please be advised that this email may contain confidential information. If you are not the intended recipient, please do not read, copy or re-transmit this email. If you have received this email in error, please notify us by email by replying to the sender and by telephone (call us collect at +1 202-828-0850) and delete this message and any attachments. Thank you in advance for your cooperation and assistance.
In addition, Danaher and its subsidiaries disclaim that the content of this email constitutes an offer to enter into, or the acceptance of, any contract or agreement or any amendment thereto; provided that the foregoing disclaimer does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment to this email.
You could check to see if minra is set to off as well. We've had some slow backups of VMWare volumes fixed with setting this to 'off' (30MB/s --> 100MB/s).
Glenn
________________________________
From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Darren Sykes Sent: Friday, November 21, 2008 11:20 AM To: Page, Jeremy; toasters@mathworks.com Subject: RE: Very slow large file operations
Jeremy,
I'd agree with IBM's analysis (and it's not often I say that).
We had performance issues on a filer with dedup enabled (and not always on the dedup volumes) that seem to have been fixed by upgrading to 7.2.5.1 which is now the Netapp recommended minimum if you're using dedup.
I'm told 7.2.6.1 has been released which will fix the sequential read bug too.
Darren.
________________________________
From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Page, Jeremy Sent: 21 November 2008 15:33 To: toasters@mathworks.com Subject: Very slow large file operations
I need help, I cannot figure out how to troubleshoot this problem further.
We have a rebranded 3070A with ~200 VMs running on it. Maybe 30 on a batch of FC disks and the rest on 2 aggrs of 33 SATA drives each. None of the VMs are disk IO intensive. When I try and copy a VM to a new volume I get EXTREMELY slow speeds, something on the order of 3 gig an hour. I am also seeing incredibly slow Snapvault transfers and initializations. I think that in part we are hitting the sequential read bug, but this occurs even on volumes that do not have SIS enabled. For instance I have a snapvault job running that's been going for 4 days (!) and only xfered 260 gig. And it's an update so should always be hitting data (and thus not having the bug).
Statit shows the drives all under 15% utilization. It's over dual 10gigE connections with the preferred adapter set (there are also 2 1 gig VIFs on this box for user and admin access).
I'm at a loss. IBM says we need to upgrade (we are at 7.2.4) but I am concerned that it's not going to fix our problem and downtime windows are difficult to get so I want to make sure it's resolved with as little interruption as possible. Any suggestions on troubleshooting?
To report this email as spam click here https://www.mailcontrol.com/sr/zWPh+74n8KTTndxI!oX7UpDBHGT3onM6072OTIrs vVit97YGlgZZEsRBLbU6zIXsS4gKezR7S7W1yubdBaCdqw== .
Please be advised that this email may contain confidential information. If you are not the intended recipient, please do not read, copy or re-transmit this email. If you have received this email in error, please notify us by email by replying to the sender and by telephone (call us collect at +1 202-828-0850) and delete this message and any attachments. Thank you in advance for your cooperation and assistance.
In addition, Danaher and its subsidiaries disclaim that the content of this email constitutes an offer to enter into, or the acceptance of, any contract or agreement or any amendment thereto; provided that the foregoing disclaimer does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment to this email.
-
Darren Sykes -
Glenn Walker -
Page, Jeremy