Hi all,
We recently purchased a NetApp FAS-2554 and I'm getting what I feel is less than stellar performance. I have a case open with support. But I'm trying to 'calibrate my expectations' a bit, so I thought I'd send this out to the list. Any feedback would be appreciated.
Here's a some info about the configuration cDOT, 8.3.1RC1 2-node switchless cluster 22 SATA drives (3TB) in two RAID-DP aggregates + 2 spares 22 10K SAS drives (900GB) in two RAID-DP aggregates + 2 spares No flash (yet?) 10Gb cluster interconnect 10Gb to server(s) with jumbo frames 2 vFilers - the one under test has a volume on the sas_data_1 aggregate which is serving NFS v3 (TCP)
I'm currently testing with a Cisco UCS C220 M4S running CentOS 6.6 as the NFS client. We noticed that performance (on a database load) seemed poor, so I started some benchmark testing with "iozone". The test I'm looking at in particular is sequential writes with an 8K record size. The write throughput was 2MB/S! There are no other workloads on the filer or the server.
NetApp suggested a workaround for the bug described here: http://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=876563 and that actually improved things - my sequential write throughput test can now do 15MB/S. Definitely better, but that still seems slow to me.
So with all that background here is my question - does anyone have a similar configuration out there, and what sort of write speeds are you seeing? I'm just looking for a sanity check. Is 15MB/S reasonable?
Thanks!
Roy McMorran
What do you see for sequential read?
Double check all your network transition points for jumbo settings?
Try backing off of jumbo as a test?
.
On Jul 17, 2015, at 9:17 AM, Roy McMorran <mcmorran@mdibl.orgmailto:mcmorran@mdibl.org> wrote:
Hi all,
We recently purchased a NetApp FAS-2554 and I'm getting what I feel is less than stellar performance. I have a case open with support. But I'm trying to 'calibrate my expectations' a bit, so I thought I'd send this out to the list. Any feedback would be appreciated.
Here's a some info about the configuration cDOT, 8.3.1RC1 2-node switchless cluster 22 SATA drives (3TB) in two RAID-DP aggregates + 2 spares 22 10K SAS drives (900GB) in two RAID-DP aggregates + 2 spares No flash (yet?) 10Gb cluster interconnect 10Gb to server(s) with jumbo frames 2 vFilers - the one under test has a volume on the sas_data_1 aggregate which is serving NFS v3 (TCP)
I'm currently testing with a Cisco UCS C220 M4S running CentOS 6.6 as the NFS client. We noticed that performance (on a database load) seemed poor, so I started some benchmark testing with "iozone". The test I'm looking at in particular is sequential writes with an 8K record size. The write throughput was 2MB/S! There are no other workloads on the filer or the server.
NetApp suggested a workaround for the bug described here: http://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=876563 and that actually improved things - my sequential write throughput test can now do 15MB/S. Definitely better, but that still seems slow to me.
So with all that background here is my question - does anyone have a similar configuration out there, and what sort of write speeds are you seeing? I'm just looking for a sanity check. Is 15MB/S reasonable?
Thanks!
Roy McMorran
_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Hi Francis, thanks for the reply.
On 7/17/2015 1:48 PM, Francis Kim wrote:
What do you see for sequential read?
About 42 MB/S
Double check all your network transition points for jumbo settings?
Yes, already have. NetApp and server are set to 9000, switch at 9216.
Try backing off of jumbo as a test?
I'll give that a try.
Thanks! Roy
On Jul 17, 2015, at 9:17 AM, Roy McMorran <mcmorran@mdibl.org mailto:mcmorran@mdibl.org> wrote:
Hi all,
We recently purchased a NetApp FAS-2554 and I'm getting what I feel is less than stellar performance. I have a case open with support. But I'm trying to 'calibrate my expectations' a bit, so I thought I'd send this out to the list. Any feedback would be appreciated.
Here's a some info about the configuration cDOT, 8.3.1RC1 2-node switchless cluster 22 SATA drives (3TB) in two RAID-DP aggregates + 2 spares 22 10K SAS drives (900GB) in two RAID-DP aggregates + 2 spares No flash (yet?) 10Gb cluster interconnect 10Gb to server(s) with jumbo frames 2 vFilers - the one under test has a volume on the sas_data_1 aggregate which is serving NFS v3 (TCP)
I'm currently testing with a Cisco UCS C220 M4S running CentOS 6.6 as the NFS client. We noticed that performance (on a database load) seemed poor, so I started some benchmark testing with "iozone". The test I'm looking at in particular is sequential writes with an 8K record size. The write throughput was 2MB/S! There are no other workloads on the filer or the server.
NetApp suggested a workaround for the bug described here: http://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=876563 and that actually improved things - my sequential write throughput test can now do 15MB/S. Definitely better, but that still seems slow to me.
So with all that background here is my question - does anyone have a similar configuration out there, and what sort of write speeds are you seeing? I'm just looking for a sanity check. Is 15MB/S reasonable?
Thanks!
Roy McMorran
Toasters mailing list Toasters@teaparty.net mailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Are your aggregates full?
Is there another workload? Check out sysstat -x 1 and see what the CP time column looks like and also check out disk busy. Rebuild in progress? Snapmirrors in progress? Deduplication in progress?
Those are some small aggregates, but I would still expect more than 2Mb. Each aggregate is 1 raid group, right?
Personally, with that few disks, I would have assigned each disk type to 1 node and have them all in 1 aggregate, but it sounds like it is 2 late for that.
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Roy McMorran Sent: Friday, July 17, 2015 2:03 PM To: Francis Kim Cc: toasters@teaparty.net Subject: Re: FAS-2554 cDOT sanity check requested
Hi Francis, thanks for the reply.
On 7/17/2015 1:48 PM, Francis Kim wrote: What do you see for sequential read?
About 42 MB/S
Double check all your network transition points for jumbo settings?
Yes, already have. NetApp and server are set to 9000, switch at 9216.
Try backing off of jumbo as a test? I'll give that a try.
Thanks! Roy
On Jul 17, 2015, at 9:17 AM, Roy McMorran <mcmorran@mdibl.orgmailto:mcmorran@mdibl.org> wrote: Hi all,
We recently purchased a NetApp FAS-2554 and I'm getting what I feel is less than stellar performance. I have a case open with support. But I'm trying to 'calibrate my expectations' a bit, so I thought I'd send this out to the list. Any feedback would be appreciated.
Here's a some info about the configuration cDOT, 8.3.1RC1 2-node switchless cluster 22 SATA drives (3TB) in two RAID-DP aggregates + 2 spares 22 10K SAS drives (900GB) in two RAID-DP aggregates + 2 spares No flash (yet?) 10Gb cluster interconnect 10Gb to server(s) with jumbo frames 2 vFilers - the one under test has a volume on the sas_data_1 aggregate which is serving NFS v3 (TCP)
I'm currently testing with a Cisco UCS C220 M4S running CentOS 6.6 as the NFS client. We noticed that performance (on a database load) seemed poor, so I started some benchmark testing with "iozone". The test I'm looking at in particular is sequential writes with an 8K record size. The write throughput was 2MB/S! There are no other workloads on the filer or the server.
NetApp suggested a workaround for the bug described here: http://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=876563 and that actually improved things - my sequential write throughput test can now do 15MB/S. Definitely better, but that still seems slow to me.
So with all that background here is my question - does anyone have a similar configuration out there, and what sort of write speeds are you seeing? I'm just looking for a sanity check. Is 15MB/S reasonable?
Thanks!
Roy McMorran
_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Hi Jordan, thanks for your reply.
On 7/17/2015 2:32 PM, Jordan Slingerland wrote:
Are your aggregates full?
Nope, brand new box, hardly using anything at all.
Is there another workload? Check out sysstat –x 1 and see what the CP time column looks like and also check out disk busy.
My test server is the only thing talking to it at this point.
Rebuild in progress?
Nope
Snapmirrors in progress?
None exist
Deduplication in progress?
Not enabled on any volumes on this vfiler. Idle on the other vfiler.
Those are some small aggregates, but I would still expect more than 2Mb. Each aggregate is 1 raid group, right?
Yes. And 2MB went to 15MB after the 'setenv wafl_c2c_xfer_timeout 0' workaround. That's really the root of my question - is this as good as it'll get?
Personally, with that few disks, I would have assigned each disk type to 1 node and have them all in 1 aggregate, but it sounds like it is 2 late for that.
Maybe not, and thanks for the input. That's not something NetApp has suggested yet, and I don't want to change anything right now while we're working the case, but it's not in production and I could nuke it and start over at some point.
Thanks, Roy
Disclaimer: Still on 7-mode so these commands may not apply to cluster mode. NetApp Technical Support should have gone through this with you so this may not be of any help. This information would be in a perfstat as well.
During the test, how does disk utilization look with the following commands. CPU?
# sysstat -u 1 # stats show disk:*:disk_busy -- look for hot spots in this output. # sysstat -m 1
Check out nfs statistics. You may need to explore the options available with nfsstat
# nfsstat
Check on any tcp errors.
# netstat -s
Check for any increasing error statistics on the ethernet interface
# ifstat -a
Are you using link aggregation on your NetApp's 10G nics?
Perhaps try a larger block size with IOZONE?
Did you add all your disks to the aggregate RG's at once? By chance, did you create the aggregate with the minimum number of disks, create the volume then add more disks to the aggregate later? If yes, look into reallocating the volume as you may have a few hot spots.
On Fri, Jul 17, 2015 at 1:51 PM, Roy McMorran mcmorran@mdibl.org wrote:
Hi Jordan, thanks for your reply.
On 7/17/2015 2:32 PM, Jordan Slingerland wrote:
Are your aggregates full?
Nope, brand new box, hardly using anything at all.
Is there another workload? Check out sysstat –x 1 and see what the CP time column looks like and also check out disk busy.
My test server is the only thing talking to it at this point.
Rebuild in progress?
Nope
Snapmirrors in progress?
None exist
Deduplication in progress?
Not enabled on any volumes on this vfiler. Idle on the other vfiler.
Those are some small aggregates, but I would still expect more than 2Mb. Each aggregate is 1 raid group, right?
Yes. And 2MB went to 15MB after the 'setenv wafl_c2c_xfer_timeout 0' workaround. That's really the root of my question - is this as good as it'll get?
Personally, with that few disks, I would have assigned each disk type to 1 node and have them all in 1 aggregate, but it sounds like it is 2 late for that.
Maybe not, and thanks for the input. That's not something NetApp has suggested yet, and I don't want to change anything right now while we're working the case, but it's not in production and I could nuke it and start over at some point.
Thanks, Roy
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Please remove me from this list
-----Original Message----- From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Philbert Rupkins Sent: Tuesday, July 21, 2015 10:55 PM To: Roy McMorran Cc: toasters@teaparty.net Subject: Re: FAS-2554 cDOT sanity check requested
Disclaimer: Still on 7-mode so these commands may not apply to cluster mode. NetApp Technical Support should have gone through this with you so this may not be of any help. This information would be in a perfstat as well.
During the test, how does disk utilization look with the following commands. CPU?
# sysstat -u 1 # stats show disk:*:disk_busy -- look for hot spots in this output. # sysstat -m 1
Check out nfs statistics. You may need to explore the options available with nfsstat
# nfsstat
Check on any tcp errors.
# netstat -s
Check for any increasing error statistics on the ethernet interface
# ifstat -a
Are you using link aggregation on your NetApp's 10G nics?
Perhaps try a larger block size with IOZONE?
Did you add all your disks to the aggregate RG's at once? By chance, did you create the aggregate with the minimum number of disks, create the volume then add more disks to the aggregate later? If yes, look into reallocating the volume as you may have a few hot spots.
On Fri, Jul 17, 2015 at 1:51 PM, Roy McMorran <mcmorran@mdibl.org mailto:mcmorran@mdibl.org > wrote:
Hi Jordan, thanks for your reply. On 7/17/2015 2:32 PM, Jordan Slingerland wrote:
Are your aggregates full?
Nope, brand new box, hardly using anything at all.
Is there another workload? Check out sysstat –x 1 and see what the CP time column looks like and also check out disk busy.
My test server is the only thing talking to it at this point.
Rebuild in progress?
Nope
Snapmirrors in progress?
None exist
Deduplication in progress?
Not enabled on any volumes on this vfiler. Idle on the other vfiler.
Those are some small aggregates, but I would still expect more than 2Mb. Each aggregate is 1 raid group, right?
Yes. And 2MB went to 15MB after the 'setenv wafl_c2c_xfer_timeout 0' workaround. That's really the root of my question - is this as good as it'll get?
Personally, with that few disks, I would have assigned each disk type to 1 node and have them all in 1 aggregate, but it sounds like it is 2 late for that.
Maybe not, and thanks for the input. That's not something NetApp has suggested yet, and I don't want to change anything right now while we're working the case, but it's not in production and I could nuke it and start over at some point. Thanks, Roy
_______________________________________________ Toasters mailing list Toasters@teaparty.net mailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
You wouldn't be here unless you subscribed, and unsubscribing is simple: http://teaparty.net/mailman/listinfo/toasters
I don't know if the admins read the list either, so emailing them directly is probably a better option than emailing the rest of us.
On Wed, Jul 22, 2015 at 8:45 AM, Schwartz, Sheila D CIV USA NETCOM/9TH SC A Sheila.Schwartz@usma.edu wrote:
Please remove me from this list
-----Original Message----- From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Philbert Rupkins Sent: Tuesday, July 21, 2015 10:55 PM To: Roy McMorran Cc: toasters@teaparty.net Subject: Re: FAS-2554 cDOT sanity check requested
Disclaimer: Still on 7-mode so these commands may not apply to cluster mode. NetApp Technical Support should have gone through this with you so this may not be of any help. This information would be in a perfstat as well.
During the test, how does disk utilization look with the following commands. CPU?
# sysstat -u 1 # stats show disk:*:disk_busy -- look for hot spots in this output. # sysstat -m 1
Check out nfs statistics. You may need to explore the options available with nfsstat
# nfsstat
Check on any tcp errors.
# netstat -s
Check for any increasing error statistics on the ethernet interface
# ifstat -a
Are you using link aggregation on your NetApp's 10G nics?
Perhaps try a larger block size with IOZONE?
Did you add all your disks to the aggregate RG's at once? By chance, did you create the aggregate with the minimum number of disks, create the volume then add more disks to the aggregate later? If yes, look into reallocating the volume as you may have a few hot spots.
On Fri, Jul 17, 2015 at 1:51 PM, Roy McMorran <mcmorran@mdibl.org mailto: mcmorran@mdibl.org > wrote:
Hi Jordan, thanks for your reply. On 7/17/2015 2:32 PM, Jordan Slingerland wrote: Are your aggregates full? Nope, brand new box, hardly using anything at all. Is there another workload? Check out sysstat –x 1 and seewhat the CP time column looks like and also check out disk busy.
My test server is the only thing talking to it at this point. Rebuild in progress? Nope Snapmirrors in progress? None exist Deduplication in progress? Not enabled on any volumes on this vfiler. Idle on the othervfiler.
Those are some small aggregates, but I would still expectmore than 2Mb. Each aggregate is 1 raid group, right?
Yes. And 2MB went to 15MB after the 'setenv wafl_c2c_xfer_timeout0' workaround. That's really the root of my question - is this as good as it'll get?
Personally, with that few disks, I would have assignedeach disk type to 1 node and have them all in 1 aggregate, but it sounds like it is 2 late for that.
Maybe not, and thanks for the input. That's not something NetApphas suggested yet, and I don't want to change anything right now while we're working the case, but it's not in production and I could nuke it and start over at some point.
Thanks, Roy _______________________________________________ Toasters mailing list Toasters@teaparty.net <mailto:Toasters@teaparty.net> http://www.teaparty.net/mailman/listinfo/toasters
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Hi Philbert, thanks for the reply!
I chose the 8K blocksize because that's what our database will be using, so it seemed like a close approximation of our data load (which is where the problem first became evident). I am planning to try a few different sizes as an experiment, as well as different NFS mount options.
Yes I am using link aggregation. My switch tells me that all this traffic is using only one of the two physical links, but the NFS throughput I'm seeing would barely saturate a 100Mb link so that doesn't seem like a concern.
Zero network errors on the switch, the server and the NetApp, at any layer.
Yes we've collected several perfstats in the course of working the case. Their analysis hasn't pointed out any issues thus far.
One interesting data point - iSCSI performs MUCH better than NFS. I'm about to post a followup about that experiment.
Thanks everyone for the help so far!
Roy
On 7/21/15 10:54 PM, Philbert Rupkins wrote:
Disclaimer: Still on 7-mode so these commands may not apply to cluster mode. NetApp Technical Support should have gone through this with you so this may not be of any help. This information would be in a perfstat as well.
During the test, how does disk utilization look with the following commands. CPU?
# sysstat -u 1 # stats show disk:*:disk_busy -- look for hot spots in this output. # sysstat -m 1
Check out nfs statistics. You may need to explore the options available with nfsstat
# nfsstat
Check on any tcp errors.
# netstat -s
Check for any increasing error statistics on the ethernet interface
# ifstat -a
Are you using link aggregation on your NetApp's 10G nics?
Perhaps try a larger block size with IOZONE?
Did you add all your disks to the aggregate RG's at once? By chance, did you create the aggregate with the minimum number of disks, create the volume then add more disks to the aggregate later? If yes, look into reallocating the volume as you may have a few hot spots.
rsize and wsize set to 32k on your nfs v3 client?
Francis Kim | Engineer 510-644-1599 x334 | fkim@berkcom.commailto:fkim@berkcom.com
BerkCom | www.berkcom.comhttp://www.berkcom.com/ NetApp | Cisco | Supermicro | Brocade | VMware
On Jul 22, 2015, at 6:53 AM, Roy McMorran <mcmorran@mdibl.orgmailto:mcmorran@mdibl.org> wrote:
Hi Philbert, thanks for the reply!
I chose the 8K blocksize because that's what our database will be using, so it seemed like a close approximation of our data load (which is where the problem first became evident). I am planning to try a few different sizes as an experiment, as well as different NFS mount options.
Yes I am using link aggregation. My switch tells me that all this traffic is using only one of the two physical links, but the NFS throughput I'm seeing would barely saturate a 100Mb link so that doesn't seem like a concern.
Zero network errors on the switch, the server and the NetApp, at any layer.
Yes we've collected several perfstats in the course of working the case. Their analysis hasn't pointed out any issues thus far.
One interesting data point - iSCSI performs MUCH better than NFS. I'm about to post a followup about that experiment.
Thanks everyone for the help so far!
Roy
On 7/21/15 10:54 PM, Philbert Rupkins wrote: Disclaimer: Still on 7-mode so these commands may not apply to cluster mode. NetApp Technical Support should have gone through this with you so this may not be of any help. This information would be in a perfstat as well.
During the test, how does disk utilization look with the following commands. CPU?
# sysstat -u 1 # stats show disk:*:disk_busy -- look for hot spots in this output. # sysstat -m 1
Check out nfs statistics. You may need to explore the options available with nfsstat
# nfsstat
Check on any tcp errors.
# netstat -s
Check for any increasing error statistics on the ethernet interface
# ifstat -a
Are you using link aggregation on your NetApp's 10G nics?
Perhaps try a larger block size with IOZONE?
Did you add all your disks to the aggregate RG's at once? By chance, did you create the aggregate with the minimum number of disks, create the volume then add more disks to the aggregate later? If yes, look into reallocating the volume as you may have a few hot spots.
_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
There is really no good reason to use "soft" when using RW. Just asking, begging for data corruption.
I remember a post many years ago from Guy Harris (you know, one of the guys who actually help invent NFS!) and it was basically this"
"Never ever ever ever ever ever use soft...."
I know there are case over a WAN where it might help, or maybe if you are using ReadOnly.
--tmac
*Tim McCarthy* *Principal Consultant*
Clustered ONTAP Clustered ONTAP NCDA ID: XK7R3GEKC1QQ2LVD RHCE6 110-107-141 https://www.redhat.com/wapps/training/certification/verify.html?certNumber=110-107-141&isSearch=False&verify=Verify NCSIE ID: C14QPHE21FR4YWD4 Expires: 27 October 2016 Current until Aug 02, 2016 Expires: 29 October 2016
On Wed, Jul 22, 2015 at 10:03 AM, Roy McMorran mcmorran@mdibl.org wrote:
On 7/22/15 9:59 AM, Francis Kim wrote:
rsize and wsize set to 32k on your nfs v3 client?
Yes.
rsize=32768,wsize=32768,intr,rw,soft,bg,tcp,vers=3
NetApp has suggested I try fiddling with this a bit but I haven't gotten 'round to it yet.
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
TR-4067 covers general mount best practices and “hard” is one of those best practices. ;)
On page 97: http://www.netapp.com/us/media/tr-4067.pdf
hard or soft hard or soft specifies whether the program using a file using NFS should stop and wait (hard) for the server to come back online if the NFS server is unavailable or if it should report an error (soft). If hard is specified, processes directed to an NFS mount that is unavailable cannot be terminated unless the intr option is also specified. If soft is specified, the timeo= option can be specified, where is the number of seconds before an error is reported.
For business-critical NFS exports, NetApp recommends using hard mounts. NetApp strongly discourages the use of soft mounts.
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of tmac Sent: Wednesday, July 22, 2015 10:10 AM To: Roy McMorran Cc: toasters@teaparty.net Subject: Re: FAS-2554 cDOT sanity check requested
There is really no good reason to use "soft" when using RW. Just asking, begging for data corruption.
I remember a post many years ago from Guy Harris (you know, one of the guys who actually help invent NFS!) and it was basically this"
"Never ever ever ever ever ever use soft...."
I know there are case over a WAN where it might help, or maybe if you are using ReadOnly.
--tmac
Tim McCarthy Principal Consultant
[http://dl.dropbox.com/u/6874230/na_cert_dma_2c.jpg] [http://dl.dropbox.com/u/6874230/rhce.jpeg] [http://dl.dropbox.com/u/6874230/na_cert_ie-san_2c.jpg]
Clustered ONTAP Clustered ONTAP NCDA ID: XK7R3GEKC1QQ2LVD RHCE6 110-107-141https://www.redhat.com/wapps/training/certification/verify.html?certNumber=110-107-141&isSearch=False&verify=Verify NCSIE ID: C14QPHE21FR4YWD4 Expires: 27 October 2016 Current until Aug 02, 2016 Expires: 29 October 2016
On Wed, Jul 22, 2015 at 10:03 AM, Roy McMorran <mcmorran@mdibl.orgmailto:mcmorran@mdibl.org> wrote: On 7/22/15 9:59 AM, Francis Kim wrote: rsize and wsize set to 32k on your nfs v3 client? Yes.
rsize=32768,wsize=32768,intr,rw,soft,bg,tcp,vers=3
NetApp has suggested I try fiddling with this a bit but I haven't gotten 'round to it yet.
_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
OK, noted, I will make that change. Thanks for that Tim and Justin.
On 7/22/15 10:13 AM, Parisi, Justin wrote:
TR-4067 covers general mount best practices and “hard” is one of those best practices. ;)
On page 97: http://www.netapp.com/us/media/tr-4067.pdf
hard or soft
hard or soft specifies whether the program using a file using NFS should stop and wait (hard) for the server to come back online if the NFS server is unavailable or if it should report an error (soft). If hard is specified, processes directed to an NFS mount that is unavailable cannot be terminated unless the intr option is also specified. If soft is specified, the timeo= option can be specified, where is the number of seconds before an error is reported.
For business-critical NFS exports, NetApp recommends using hard mounts. NetApp strongly discourages the use of soft mounts.
*From:*toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] *On Behalf Of *tmac *Sent:* Wednesday, July 22, 2015 10:10 AM *To:* Roy McMorran *Cc:* toasters@teaparty.net *Subject:* Re: FAS-2554 cDOT sanity check requested
There is really no good reason to use "soft" when using RW.
Just asking, begging for data corruption.
I remember a post many years ago from Guy Harris (you know, one of the guys who actually help invent NFS!)
and it was basically this"
"Never ever ever ever ever ever use soft...."
I know there are case over a WAN where it might help, or maybe if you are using ReadOnly.
--tmac
*Tim McCarthy*
/Principal Consultant/
You can try something like:
rsize=65536,wsize=65535,intr,rw,hard,tcp,vers=3
Check out page 15 of Justin's Doc: RPC Slots Increased in RHEL 6.3 and Later -> maybe work that in your NFS client too.
--tmac
*Tim McCarthy*
On Wed, Jul 22, 2015 at 10:18 AM, Roy McMorran mcmorran@mdibl.org wrote:
OK, noted, I will make that change. Thanks for that Tim and Justin.
On 7/22/15 10:13 AM, Parisi, Justin wrote:
TR-4067 covers general mount best practices and “hard” is one of those best practices. ;)
On page 97: http://www.netapp.com/us/media/tr-4067.pdf
hard or soft
hard or soft specifies whether the program using a file using NFS should stop and wait (hard) for the server to come back online if the NFS server is unavailable or if it should report an error (soft). If hard is specified, processes directed to an NFS mount that is unavailable cannot be terminated unless the intr option is also specified. If soft is specified, the timeo= option can be specified, where is the number of seconds before an error is reported.
For business-critical NFS exports, NetApp recommends using hard mounts. NetApp strongly discourages the use of soft mounts.
*From:* toasters-bounces@teaparty.net [ mailto:toasters-bounces@teaparty.net toasters-bounces@teaparty.net] *On Behalf Of *tmac *Sent:* Wednesday, July 22, 2015 10:10 AM *To:* Roy McMorran *Cc:* toasters@teaparty.net *Subject:* Re: FAS-2554 cDOT sanity check requested
There is really no good reason to use "soft" when using RW.
Just asking, begging for data corruption.
I remember a post many years ago from Guy Harris (you know, one of the guys who actually help invent NFS!)
and it was basically this"
"Never ever ever ever ever ever use soft...."
I know there are case over a WAN where it might help, or maybe if you are using ReadOnly.
--tmac
*Tim McCarthy*
*Principal Consultant*
Try udp? Try another nfs client, a different OS or at least a different version?
Francis Kim | Engineer 510-644-1599 x334 | fkim@berkcom.commailto:fkim@berkcom.com
BerkCom | www.berkcom.comhttp://www.berkcom.com/ NetApp | Cisco | Supermicro | Brocade | VMware
On Jul 22, 2015, at 7:03 AM, Roy McMorran <mcmorran@mdibl.orgmailto:mcmorran@mdibl.org> wrote:
On 7/22/15 9:59 AM, Francis Kim wrote: rsize and wsize set to 32k on your nfs v3 client?
Yes.
rsize=32768,wsize=32768,intr,rw,soft,bg,tcp,vers=3
NetApp has suggested I try fiddling with this a bit but I haven't gotten 'round to it yet.
Just wanted to post a followup with the results of an experiment we tried.
I created an iSCSI LUN (on the exact same aggregate, volume, and vfiler) and exposed it to a Windows server. Also using 10Gb, jumbo frames, same switch.
Ran iometer on the Windows box (using similar test parameters) and got 275MB/S for sequential writes to that LUN. As compared to the 15MB/S for NFSv3 from the Linux system.
I'd expect iSCSI to be faster than NFS, but this difference seems extreme. Any thoughts?
Thanks, Roy
On 7/17/15 12:16 PM, Roy McMorran wrote:
Hi all,
We recently purchased a NetApp FAS-2554 and I'm getting what I feel is less than stellar performance. I have a case open with support. But I'm trying to 'calibrate my expectations' a bit, so I thought I'd send this out to the list. Any feedback would be appreciated.
Here's a some info about the configuration cDOT, 8.3.1RC1 2-node switchless cluster 22 SATA drives (3TB) in two RAID-DP aggregates + 2 spares 22 10K SAS drives (900GB) in two RAID-DP aggregates + 2 spares No flash (yet?) 10Gb cluster interconnect 10Gb to server(s) with jumbo frames 2 vFilers - the one under test has a volume on the sas_data_1 aggregate which is serving NFS v3 (TCP)
I'm currently testing with a Cisco UCS C220 M4S running CentOS 6.6 as the NFS client. We noticed that performance (on a database load) seemed poor, so I started some benchmark testing with "iozone". The test I'm looking at in particular is sequential writes with an 8K record size. The write throughput was 2MB/S! There are no other workloads on the filer or the server.
NetApp suggested a workaround for the bug described here: http://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=876563 and that actually improved things - my sequential write throughput test can now do 15MB/S. Definitely better, but that still seems slow to me.
So with all that background here is my question - does anyone have a similar configuration out there, and what sort of write speeds are you seeing? I'm just looking for a sanity check. Is 15MB/S reasonable?
Thanks!
Roy McMorran
Try CIFS to the same windows server?
Francis Kim | Engineer 510-644-1599 x334 | fkim@berkcom.commailto:fkim@berkcom.com
BerkCom | www.berkcom.comhttp://www.berkcom.com/ NetApp | Cisco | Supermicro | Brocade | VMware
On Jul 22, 2015, at 7:14 AM, Roy McMorran <mcmorran@mdibl.orgmailto:mcmorran@mdibl.org> wrote:
Just wanted to post a followup with the results of an experiment we tried.
I created an iSCSI LUN (on the exact same aggregate, volume, and vfiler) and exposed it to a Windows server. Also using 10Gb, jumbo frames, same switch.
Ran iometer on the Windows box (using similar test parameters) and got 275MB/S for sequential writes to that LUN. As compared to the 15MB/S for NFSv3 from the Linux system.
I'd expect iSCSI to be faster than NFS, but this difference seems extreme. Any thoughts?
Thanks, Roy
On 7/17/15 12:16 PM, Roy McMorran wrote: Hi all,
We recently purchased a NetApp FAS-2554 and I'm getting what I feel is less than stellar performance. I have a case open with support. But I'm trying to 'calibrate my expectations' a bit, so I thought I'd send this out to the list. Any feedback would be appreciated.
Here's a some info about the configuration cDOT, 8.3.1RC1 2-node switchless cluster 22 SATA drives (3TB) in two RAID-DP aggregates + 2 spares 22 10K SAS drives (900GB) in two RAID-DP aggregates + 2 spares No flash (yet?) 10Gb cluster interconnect 10Gb to server(s) with jumbo frames 2 vFilers - the one under test has a volume on the sas_data_1 aggregate which is serving NFS v3 (TCP)
I'm currently testing with a Cisco UCS C220 M4S running CentOS 6.6 as the NFS client. We noticed that performance (on a database load) seemed poor, so I started some benchmark testing with "iozone". The test I'm looking at in particular is sequential writes with an 8K record size. The write throughput was 2MB/S! There are no other workloads on the filer or the server.
NetApp suggested a workaround for the bug described here: http://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=876563 and that actually improved things - my sequential write throughput test can now do 15MB/S. Definitely better, but that still seems slow to me.
So with all that background here is my question - does anyone have a similar configuration out there, and what sort of write speeds are you seeing? I'm just looking for a sanity check. Is 15MB/S reasonable?
Thanks!
Roy McMorran
_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
What iometer/iozone configuration were you using?
From: <toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net> on behalf of Roy McMorran <mcmorran@mdibl.orgmailto:mcmorran@mdibl.org> Date: Wednesday, July 22, 2015 at 10:14 AM To: "toasters@teaparty.netmailto:toasters@teaparty.net" <toasters@teaparty.netmailto:toasters@teaparty.net> Subject: Re: FAS-2554 cDOT sanity check requested
Just wanted to post a followup with the results of an experiment we tried.
I created an iSCSI LUN (on the exact same aggregate, volume, and vfiler) and exposed it to a Windows server. Also using 10Gb, jumbo frames, same switch.
Ran iometer on the Windows box (using similar test parameters) and got 275MB/S for sequential writes to that LUN. As compared to the 15MB/S for NFSv3 from the Linux system.
I'd expect iSCSI to be faster than NFS, but this difference seems extreme. Any thoughts?
Thanks, Roy
On 7/17/15 12:16 PM, Roy McMorran wrote: Hi all,
We recently purchased a NetApp FAS-2554 and I'm getting what I feel is less than stellar performance. I have a case open with support. But I'm trying to 'calibrate my expectations' a bit, so I thought I'd send this out to the list. Any feedback would be appreciated.
Here's a some info about the configuration cDOT, 8.3.1RC1 2-node switchless cluster 22 SATA drives (3TB) in two RAID-DP aggregates + 2 spares 22 10K SAS drives (900GB) in two RAID-DP aggregates + 2 spares No flash (yet?) 10Gb cluster interconnect 10Gb to server(s) with jumbo frames 2 vFilers - the one under test has a volume on the sas_data_1 aggregate which is serving NFS v3 (TCP)
I'm currently testing with a Cisco UCS C220 M4S running CentOS 6.6 as the NFS client. We noticed that performance (on a database load) seemed poor, so I started some benchmark testing with "iozone". The test I'm looking at in particular is sequential writes with an 8K record size. The write throughput was 2MB/S! There are no other workloads on the filer or the server.
NetApp suggested a workaround for the bug described here: http://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=876563 and that actually improved things - my sequential write throughput test can now do 15MB/S. Definitely better, but that still seems slow to me.
So with all that background here is my question - does anyone have a similar configuration out there, and what sort of write speeds are you seeing? I'm just looking for a sanity check. Is 15MB/S reasonable?
Thanks!
Roy McMorran
-
Basil -
Francis Kim -
Jordan Slingerland -
Parisi, Justin -
Philbert Rupkins -
Roy McMorran -
Schwartz, Sheila D CIV USA NETCOM/9TH SC A -
tmac -
Waltham, Christopher