Does anyone have a very rough rule of thumb as to IOPs you might expect out of a controller?
I've got a FAS6280 that's starting to 'be a bit sluggish'. I'm inclined to shrug and say 'yes, it's doing 20K IOPs'.
I know there's very much an 'it depends' in there - but 'very rough' is good enough for me here. (Disk wise - it's not particularly high on disk utilisation, nor is the network bandwidth particularly full).
It depends mostly on the type of IO. File IO takes lots of processor, but block IO tends to bottleneck on the disks. If you have high latency, I'd start off by looking at the processor. If that correlates with the latency, that's your bottleneck. If it doesn't, open a case with Netapp, capture a performance dump during high latency, and have them tell you.
On Wed, Feb 11, 2015 at 11:54 AM, Edward Rolison ed.rolison@gmail.com wrote:
Does anyone have a very rough rule of thumb as to IOPs you might expect out of a controller?
I've got a FAS6280 that's starting to 'be a bit sluggish'. I'm inclined to shrug and say 'yes, it's doing 20K IOPs'.
I know there's very much an 'it depends' in there - but 'very rough' is good enough for me here. (Disk wise - it's not particularly high on disk utilisation, nor is the network bandwidth particularly full).
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
I suppose, somewhere between 1 and 100,000 would be about right.
I would call your sales machine, and ask for a PS Performance Review.
Not that Support cant say what its doing, but PS will tell you where it's been...and where it's going.
On Wed, Feb 11, 2015 at 5:54 PM, Edward Rolison ed.rolison@gmail.com wrote:
Does anyone have a very rough rule of thumb as to IOPs you might expect out of a controller?
I've got a FAS6280 that's starting to 'be a bit sluggish'. I'm inclined to shrug and say 'yes, it's doing 20K IOPs'.
I know there's very much an 'it depends' in there - but 'very rough' is good enough for me here. (Disk wise - it's not particularly high on disk utilisation, nor is the network bandwidth particularly full).
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
I agree with Jeff... Could be a bunch of things.. I know in the past if you added storage and didn't do a reallocate, could be the cause.. Could be lots of things, but a perfstat is a good 1st pass. Look at performance advisor if you have it.. Maybe you have some hot disks.. A little more info such as "what's slow" would be appreciated to assist further.
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Jeff Mohler Sent: Wednesday, February 11, 2015 5:00 PM To: Edward Rolison Cc: toasters@teaparty.net Subject: Re: IOPs from a controller
I suppose, somewhere between 1 and 100,000 would be about right.
I would call your sales machine, and ask for a PS Performance Review.
Not that Support cant say what its doing, but PS will tell you where it's been...and where it's going.
On Wed, Feb 11, 2015 at 5:54 PM, Edward Rolison <ed.rolison@gmail.commailto:ed.rolison@gmail.com> wrote: Does anyone have a very rough rule of thumb as to IOPs you might expect out of a controller?
I've got a FAS6280 that's starting to 'be a bit sluggish'. I'm inclined to shrug and say 'yes, it's doing 20K IOPs'.
I know there's very much an 'it depends' in there - but 'very rough' is good enough for me here. (Disk wise - it's not particularly high on disk utilisation, nor is the network bandwidth particularly full).
_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
-- --- Gustatus Similis Pullus
Hi Edward,
Just to put it in perspective: we had a 3210, that did 38.000+ IOPs in our VMware environment. I also heard about a 3250, that did more than 1 Mio real world IOPs with the help of a batch job pre-warming the FlashCache every morning (98% hit rate, they happened to interview him on the NetApp podcast). So it really depends on what you're doing... (e.g. big sequential read/write: expect low IOPs)
As to the 'hot disk', sysstat gives you 'the disk with the highest utilization'. So if the number sysstat gives as disk utilization is 'low', it's unlikely, that you have a hot disk problem.
To prevent allocation/fragmentation problems I usually advise to turn on automatic reallocate options, like
* free_space_realloc on (for the aggregates) * read_realloc space_optimized (for volumes that contain e.g. databases)
my 2c...
Sebastian
On 2/12/2015 2:11 AM, Klise, Steve wrote:
I agree with Jeff… Could be a bunch of things.. I know in the past if you added storage and didn’t do a reallocate, could be the cause.. Could be lots of things, but a perfstat is a good 1^st pass. Look at performance advisor if you have it.. Maybe you have some hot disks.. A little more info such as “what’s slow” would be appreciated to assist further.
*From:*toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] *On Behalf Of *Jeff Mohler *Sent:* Wednesday, February 11, 2015 5:00 PM *To:* Edward Rolison *Cc:* toasters@teaparty.net *Subject:* Re: IOPs from a controller
I suppose, somewhere between 1 and 100,000 would be about right.
I would call your sales machine, and ask for a PS Performance Review.
Not that Support cant say what its doing, but PS will tell you where it's been...and where it's going.
On Wed, Feb 11, 2015 at 5:54 PM, Edward Rolison <ed.rolison@gmail.com mailto:ed.rolison@gmail.com> wrote:
Does anyone have a very rough rule of thumb as to IOPs you might expect out of a controller?
I've got a FAS6280 that's starting to 'be a bit sluggish'. I'm inclined to shrug and say 'yes, it's doing 20K IOPs'.
I know there's very much an 'it depends' in there - but 'very rough' is good enough for me here. (Disk wise - it's not particularly high on disk utilisation, nor is the network bandwidth particularly full).
Toasters mailing list Toasters@teaparty.net mailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
--
Gustatus Similis Pullus
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Thanks for all the answers thus far. They've been helpful.
I'm bottle necked on CPU primarily. I've checked the drives, and there's no particular hotspots there - I have nice big aggregates with ~200 spindles, and they're comfortably less than 30%. Nor network - trunked 10G cards, and none are particularly 'hot'. (Nor anywhere else I'm looking).
But the 'sysstat -x' cpu load is in the mid-high 90s, and the 'sysstat -M' is giving a sum of around 850% (12CPUs).
I do have particular volumes 'running hot' that correlates with latency spikes on another volume (and is getting complaints from another user group). By 'running hot' I mean '10K IOPs' and 200MB/sec read, 200MB/sec write. Whilst it's doing that, if my 'other customer' tries to use their share, they get pretty persistent 20ms+ latency (from 'stats show volume') at about 1000 iops/20MB read/sec.
I'm pretty sure this is my root cause, but my 'gut feeling' is that it shouldn't be. Hence the question.
On 12 February 2015 at 01:11, Klise, Steve Steve.Klise@wwt.com wrote:
I agree with Jeff… Could be a bunch of things.. I know in the past if you added storage and didn’t do a reallocate, could be the cause.. Could be lots of things, but a perfstat is a good 1st pass. Look at performance advisor if you have it.. Maybe you have some hot disks.. A little more info such as “what’s slow” would be appreciated to assist further.
*From:* toasters-bounces@teaparty.net [mailto: toasters-bounces@teaparty.net] *On Behalf Of *Jeff Mohler *Sent:* Wednesday, February 11, 2015 5:00 PM *To:* Edward Rolison *Cc:* toasters@teaparty.net *Subject:* Re: IOPs from a controller
I suppose, somewhere between 1 and 100,000 would be about right.
I would call your sales machine, and ask for a PS Performance Review.
Not that Support cant say what its doing, but PS will tell you where it's been...and where it's going.
On Wed, Feb 11, 2015 at 5:54 PM, Edward Rolison ed.rolison@gmail.com wrote:
Does anyone have a very rough rule of thumb as to IOPs you might expect out of a controller?
I've got a FAS6280 that's starting to 'be a bit sluggish'. I'm inclined to shrug and say 'yes, it's doing 20K IOPs'.
I know there's very much an 'it depends' in there - but 'very rough' is good enough for me here. (Disk wise - it's not particularly high on disk utilisation, nor is the network bandwidth particularly full).
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
--
Gustatus Similis Pullus
This is typical of NAS- it really requires a lot more processing than SAN.
On Thu, Feb 12, 2015 at 4:58 AM, Edward Rolison ed.rolison@gmail.com wrote:
Thanks for all the answers thus far. They've been helpful.
I'm bottle necked on CPU primarily. I've checked the drives, and there's no particular hotspots there - I have nice big aggregates with ~200 spindles, and they're comfortably less than 30%. Nor network - trunked 10G cards, and none are particularly 'hot'. (Nor anywhere else I'm looking).
But the 'sysstat -x' cpu load is in the mid-high 90s, and the 'sysstat -M' is giving a sum of around 850% (12CPUs).
I do have particular volumes 'running hot' that correlates with latency spikes on another volume (and is getting complaints from another user group). By 'running hot' I mean '10K IOPs' and 200MB/sec read, 200MB/sec write. Whilst it's doing that, if my 'other customer' tries to use their share, they get pretty persistent 20ms+ latency (from 'stats show volume') at about 1000 iops/20MB read/sec.
I'm pretty sure this is my root cause, but my 'gut feeling' is that it shouldn't be. Hence the question.
On 12 February 2015 at 01:11, Klise, Steve Steve.Klise@wwt.com wrote:
I agree with Jeff… Could be a bunch of things.. I know in the past if you added storage and didn’t do a reallocate, could be the cause.. Could be lots of things, but a perfstat is a good 1st pass. Look at performance advisor if you have it.. Maybe you have some hot disks.. A little more info such as “what’s slow” would be appreciated to assist further.
*From:* toasters-bounces@teaparty.net [mailto: toasters-bounces@teaparty.net] *On Behalf Of *Jeff Mohler *Sent:* Wednesday, February 11, 2015 5:00 PM *To:* Edward Rolison *Cc:* toasters@teaparty.net *Subject:* Re: IOPs from a controller
I suppose, somewhere between 1 and 100,000 would be about right.
I would call your sales machine, and ask for a PS Performance Review.
Not that Support cant say what its doing, but PS will tell you where it's been...and where it's going.
On Wed, Feb 11, 2015 at 5:54 PM, Edward Rolison ed.rolison@gmail.com wrote:
Does anyone have a very rough rule of thumb as to IOPs you might expect out of a controller?
I've got a FAS6280 that's starting to 'be a bit sluggish'. I'm inclined to shrug and say 'yes, it's doing 20K IOPs'.
I know there's very much an 'it depends' in there - but 'very rough' is good enough for me here. (Disk wise - it's not particularly high on disk utilisation, nor is the network bandwidth particularly full).
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
--
Gustatus Similis Pullus
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Other items to check:
Do you have overlapping/frequent snap mirror jobs? Are you running dedup jobs? Do you have misaligned IO ? http://www.vmadmin.info/2010/07/quantifying-vmdk-misalignment.html http://www.vmadmin.info/2010/07/quantifying-vmdk-misalignment.html Try deconstructing your IOPS to determine source of latency spikes: http://www.vmadmin.info/2010/07/vmware-and-netapp-deconstructing.html http://www.vmadmin.info/2010/07/vmware-and-netapp-deconstructing.html
On Feb 12, 2015, at 1:58 AM, Edward Rolison ed.rolison@gmail.com wrote:
Thanks for all the answers thus far. They've been helpful.
I'm bottle necked on CPU primarily. I've checked the drives, and there's no particular hotspots there - I have nice big aggregates with ~200 spindles, and they're comfortably less than 30%. Nor network - trunked 10G cards, and none are particularly 'hot'. (Nor anywhere else I'm looking).
But the 'sysstat -x' cpu load is in the mid-high 90s, and the 'sysstat -M' is giving a sum of around 850% (12CPUs).
I do have particular volumes 'running hot' that correlates with latency spikes on another volume (and is getting complaints from another user group). By 'running hot' I mean '10K IOPs' and 200MB/sec read, 200MB/sec write. Whilst it's doing that, if my 'other customer' tries to use their share, they get pretty persistent 20ms+ latency (from 'stats show volume') at about 1000 iops/20MB read/sec.
I'm pretty sure this is my root cause, but my 'gut feeling' is that it shouldn't be. Hence the question.
On 12 February 2015 at 01:11, Klise, Steve <Steve.Klise@wwt.com mailto:Steve.Klise@wwt.com> wrote: I agree with Jeff… Could be a bunch of things.. I know in the past if you added storage and didn’t do a reallocate, could be the cause.. Could be lots of things, but a perfstat is a good 1st pass. Look at performance advisor if you have it.. Maybe you have some hot disks.. A little more info such as “what’s slow” would be appreciated to assist further.
From: toasters-bounces@teaparty.net mailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net mailto:toasters-bounces@teaparty.net] On Behalf Of Jeff Mohler Sent: Wednesday, February 11, 2015 5:00 PM To: Edward Rolison Cc: toasters@teaparty.net mailto:toasters@teaparty.net Subject: Re: IOPs from a controller
I suppose, somewhere between 1 and 100,000 would be about right.
I would call your sales machine, and ask for a PS Performance Review.
Not that Support cant say what its doing, but PS will tell you where it's been...and where it's going.
On Wed, Feb 11, 2015 at 5:54 PM, Edward Rolison <ed.rolison@gmail.com mailto:ed.rolison@gmail.com> wrote:
Does anyone have a very rough rule of thumb as to IOPs you might expect out of a controller?
I've got a FAS6280 that's starting to 'be a bit sluggish'. I'm inclined to shrug and say 'yes, it's doing 20K IOPs'.
I know there's very much an 'it depends' in there - but 'very rough' is good enough for me here. (Disk wise - it's not particularly high on disk utilisation, nor is the network bandwidth particularly full).
Toasters mailing list Toasters@teaparty.net mailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters http://www.teaparty.net/mailman/listinfo/toasters
--
Gustatus Similis Pullus
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Useful links, thank you. One (idle) snapmirror, no misaligned IO (but knowing _that_ will be very useful to check in future).
We do dedupe, and have been running it on 'auto'. However I've disabled that, on the offchance it would help - and it hasn't.
I've somewhat differing symptoms in that my 'disk busy' is low. Nothing above 30%.
On 12 February 2015 at 17:19, Fletcher Cocquyt fcocquyt@stanford.edu wrote:
Other items to check:
Do you have overlapping/frequent snap mirror jobs? Are you running dedup jobs? Do you have misaligned IO ? http://www.vmadmin.info/2010/07/quantifying-vmdk-misalignment.html Try deconstructing your IOPS to determine source of latency spikes: http://www.vmadmin.info/2010/07/vmware-and-netapp-deconstructing.html
On Feb 12, 2015, at 1:58 AM, Edward Rolison ed.rolison@gmail.com wrote:
Thanks for all the answers thus far. They've been helpful.
I'm bottle necked on CPU primarily. I've checked the drives, and there's no particular hotspots there - I have nice big aggregates with ~200 spindles, and they're comfortably less than 30%. Nor network - trunked 10G cards, and none are particularly 'hot'. (Nor anywhere else I'm looking).
But the 'sysstat -x' cpu load is in the mid-high 90s, and the 'sysstat -M' is giving a sum of around 850% (12CPUs).
I do have particular volumes 'running hot' that correlates with latency spikes on another volume (and is getting complaints from another user group). By 'running hot' I mean '10K IOPs' and 200MB/sec read, 200MB/sec write. Whilst it's doing that, if my 'other customer' tries to use their share, they get pretty persistent 20ms+ latency (from 'stats show volume') at about 1000 iops/20MB read/sec.
I'm pretty sure this is my root cause, but my 'gut feeling' is that it shouldn't be. Hence the question.
On 12 February 2015 at 01:11, Klise, Steve Steve.Klise@wwt.com wrote:
I agree with Jeff… Could be a bunch of things.. I know in the past if you added storage and didn’t do a reallocate, could be the cause.. Could be lots of things, but a perfstat is a good 1st pass. Look at performance advisor if you have it.. Maybe you have some hot disks.. A little more info such as “what’s slow” would be appreciated to assist further.
*From:* toasters-bounces@teaparty.net [mailto: toasters-bounces@teaparty.net] *On Behalf Of *Jeff Mohler *Sent:* Wednesday, February 11, 2015 5:00 PM *To:* Edward Rolison *Cc:* toasters@teaparty.net *Subject:* Re: IOPs from a controller
I suppose, somewhere between 1 and 100,000 would be about right.
I would call your sales machine, and ask for a PS Performance Review.
Not that Support cant say what its doing, but PS will tell you where it's been...and where it's going.
On Wed, Feb 11, 2015 at 5:54 PM, Edward Rolison ed.rolison@gmail.com wrote:
Does anyone have a very rough rule of thumb as to IOPs you might expect out of a controller?
I've got a FAS6280 that's starting to 'be a bit sluggish'. I'm inclined to shrug and say 'yes, it's doing 20K IOPs'.
I know there's very much an 'it depends' in there - but 'very rough' is good enough for me here. (Disk wise - it's not particularly high on disk utilisation, nor is the network bandwidth particularly full).
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
--
Gustatus Similis Pullus
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Edward Rolison wrote:
Thanks for all the answers thus far. They've been helpful.
I'm bottle necked on CPU primarily. I've checked the drives, and there's
Sorry, but I have to ask: which ONTAP version is this on your 6280? Before or after 8.2.1? 7-mode I presume. When you say your're at 90% "CPU" you're referring to the *first* column of the output of sysstat -x, under a File Sharing ("NAS") type workload, yes?
Your're basically stating that your're at 850/1200 = 71% CPU. It's a respectful load on a head, you'd better be below ~30% on the partner head or else your s**t will stop short at an HA failover
The interesting part here is what the whole output from systat -M looks like in detail, rolling stats with say 2.5 m interval:
'priv set -q diag; sysstat -M 150'
no particular hotspots there - I have nice big aggregates with ~200 spindles, and they're comfortably less than 30%. Nor network - trunked 10G cards, and none are particularly 'hot'. (Nor anywhere else I'm looking).
Very good -- there's some processing bottleneck you're hitting it sounds like that to me. I have quite some experiance from this kind of stuff, on several large 6290s rather simliar to yours it feels like. Same "class" of File Sharing (NFS dominated) workload as well. You might be badly fragmented for free space, especially of the W latencu is bad but R is AOK and metadata (GETATTR, LOOKUP & ACCES) are all good
BTW the answer to your Q at the start of this trehad is, of course, "it depends". It's always that answer. Always. Always. (Not possible to give any rought rule of thumb even)
/M
But the 'sysstat -x' cpu load is in the mid-high 90s, and the 'sysstat -M' is giving a sum of around 850% (12CPUs).
I do have particular volumes 'running hot' that correlates with latency spikes on another volume (and is getting complaints from another user group). By 'running hot' I mean '10K IOPs' and 200MB/sec read, 200MB/sec write. Whilst it's doing that, if my 'other customer' tries to use their share, they get pretty persistent 20ms+ latency (from 'stats show volume') at about 1000 iops/20MB read/sec.
I'm pretty sure this is my root cause, but my 'gut feeling' is that it shouldn't be. Hence the question.
-
Basil -
Edward Rolison -
Fletcher Cocquyt -
Jeff Mohler -
Klise, Steve -
Michael Bergman -
Sebastian Goetze