Jeff is right, let it run if you can. The option to disable it is so tempting, but you're better off just adding another shelf and running a reallocate.

On Jun 23, 2015, at 11:55 PM, Jeffrey Mohler <jmohler@yahoo-inc.commailto:jmohler@yahoo-inc.com> wrote:

This is normal, and -required- to...to simplify this, comb out the new physical to virtual block map required after a snapmirror update.

It will eventually finish, but if it never finishes, run a less intensive snapmirror schedule and let it catch up.

_________________________________ Jeff Mohlermailto:jmohler@yahoo-inc.com Tech Yahoo, Storage Architect, Principal (831)454-6712 TW: @PrincipalYahoo YM: Supra89ta

On Tuesday, June 23, 2015 11:51 PM, Mike Michalakis <mike@directit.com.cymailto:mike@directit.com.cy> wrote:

Hi I have a storage system that is under stress, and as luck has it I need to added more stress by loading it with a large cifs related volume, this is a transition from a 7-Mode ontap system

Every time we complete the transition, the system spikes disks 100%, cpu 100%, etc. On the perfstat we see deswizzling running. This is a normal operation every time a snapmirror break-off occurs, that is the deswizzle part of the equation.. When we take this volume offline, operations go back to normal.

My question, what is the result of disabling deswizzle is the short term, do I look functionality, e.g. vol move?

To turn off deswizzle: Nodeshell*> options wafl.deswizzle.enable off To turn on deswizzle: Nodeshell*> options wafl.deswizzle.enable on

Backround: FAS3210, C-Dot 8.2.3, SAS disks

_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

Sebastian et al are correct. You do not need to let deswizzling finish, but it is recommended that you do. Without deswizzling the destination volume, reads at the destination volume will be slightly slower (and slightly more expensive) as we do the PVBN -> VVBN lookups.

As Jeff said, the wafl_scan_speed setting is responsible for quite a few things -- not just deswizzling -- so it's best to use that at the behest of NGS.

Chris

From: <toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net> on behalf of Sebastian Goetze <spgoetze@gmail.commailto:spgoetze@gmail.com> Date: Wednesday, June 24, 2015 at 5:05 AM To: Mike Michalakis <mike@directit.com.cymailto:mike@directit.com.cy>, Jeffrey Mohler <jmohler@yahoo-inc.commailto:jmohler@yahoo-inc.com> Cc: "toasters@teaparty.netmailto:toasters@teaparty.net" <toasters@teaparty.netmailto:toasters@teaparty.net> Subject: Re: Deswizzle

Hi Mike,

to simplify, deswizzling turns a two-way-lookup (-> logical block in volume -> physical block in aggregate) into a one-way-lookup. Read access to the destination volume will be slower (and more work for the controller) until deswizzling is done. If you don't access the destination volume, I don't foresee problems...

Sebastian

On 6/24/2015 10:36 AM, Tony Bar wrote: Jeff is right, let it run if you can. The option to disable it is so tempting, but you're better off just adding another shelf and running a reallocate.

On Jun 23, 2015, at 11:55 PM, Jeffrey Mohler <mailto:jmohler@yahoo-inc.comjmohler@yahoo-inc.commailto:jmohler@yahoo-inc.com> wrote:

This is normal, and -required- to...to simplify this, comb out the new physical to virtual block map required after a snapmirror update.

It will eventually finish, but if it never finishes, run a less intensive snapmirror schedule and let it catch up.

_________________________________ Jeff Mohlermailto:jmohler@yahoo-inc.com Tech Yahoo, Storage Architect, Principal (831)454-6712 TW: @PrincipalYahoo YM: Supra89ta

On Tuesday, June 23, 2015 11:51 PM, Mike Michalakis <mailto:mike@directit.com.cymike@directit.com.cymailto:mike@directit.com.cy> wrote:

Hi I have a storage system that is under stress, and as luck has it I need to added more stress by loading it with a large cifs related volume, this is a transition from a 7-Mode ontap system

Every time we complete the transition, the system spikes disks 100%, cpu 100%, etc. On the perfstat we see deswizzling running. This is a normal operation every time a snapmirror break-off occurs, that is the deswizzle part of the equation.. When we take this volume offline, operations go back to normal.

My question, what is the result of disabling deswizzle is the short term, do I look functionality, e.g. vol move?

To turn off deswizzle: Nodeshell*> options wafl.deswizzle.enable off To turn on deswizzle: Nodeshell*> options wafl.deswizzle.enable on

Backround: FAS3210, C-Dot 8.2.3, SAS disks

_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.nethttp://www.teaparty.net/mailman/listinfo/toasters

Tony Bar wrote:

The reallocate commands are hidden in CDOT, but you can find a reference in this KB if you need them:

Hidden...? I don't see that but I may be missing something. I noticed that the man pages online at the Support site Documentation section seem to lack an entry for reallocate. Curious.

Aggregate reallocate (freespace) cannot be done from Cluster Lvl AFAICS, it has to be done inside the node that owns the Aggr in Q.

$ xxxx1 'set -priv diag -conf off; run xxxx1-08 "reallocate ?"'

Invalid subcommand '?' usage: reallocate on | off reallocate start [-t threshold] [-p] [-u] [-o] [-n] [-i interval] <path> | /vol/<volname> reallocate start -f [-p] [-u] <path> | /vol/<volname> reallocate start -A [-o] [-i interval] <aggr_name> NOTE: -A is for aggregate (freespace) reallocation. Do NOT use -A after growing an aggregate if you wish to optimize the layout of existing data; instead use reallocate start -f /vol/<volname> for each volume in the aggregate. reallocate status [-v] [<path> | <aggr_name>] reallocate stop <path> | <aggr_name> reallocate quiesce <path> | <aggr_name> reallocate restart [-i] <path> | <aggr_name> reallocate schedule {-d | -s <schedule>} <path> | <aggr_name> reallocate measure [-l logfile] [-t threshold] [-o] [-i interval] <path> | /vol/<volname>

$ xxxx1 version NetApp Release 8.3P1D4: Wed May 13 16:24:01 PDT 2015

$ xxxx1 reallocate ? measure Start reallocate measure job off Disable reallocate jobs on Enable reallocate jobs quiesce Quiesce reallocate job restart Restart reallocate job schedule Modify schedule of reallocate job show Show reallocate job status start Start reallocate job stop Stop reallocate job

$ xxxx1 reallocate start ? -vserver <vserver name> Vserver [-path] <text> Path { [[-interval|-i] <text>] Interval Schedule | [ -once|-o [true] ] Once | [ -force|-f [true] ] } Force { [ -space-optimized|-p [true] ] Space Optimized | [ -unshare|-u [true] ] } Unshare Deduplicated Blocks { [ -threshold|-t {3..10} ] Threshold | [ -no-check|-n [true] ] } No Threshold Check

Sebastian -

I believe it might have been "hidden" behind priv advanced in an earlier release 8.2, although I haven't got access to an 8.2 VSIM any longer to try it with.

Thanks for the command options! It still does help to rebalance after adding disks to an aggregate.

Anthony Bar | Director of Engineering 650.207.5368 | tbar@berkcom.com   Berkeley Communications | www.berkcom.com NetApp | Cisco | VMware | SuperMicro | Big Data & Analytics | HPC

-----Original Message----- From: Sebastian Goetze [mailto:spgoetze@gmail.com] Sent: Friday, June 26, 2015 11:15 AM To: Michael Bergman; Toasters; Tony Bar Subject: Re: Deswizzle -> Reallocate

Hi Tony, hi Michael,

not sure how we got here from deswizzle...

First: reallocate isn't exactly hidden:

c83-PC::reallocate> ?

measure Start reallocate measure job

off Disable reallocate jobs

on Enable reallocate jobs

quiesce Quiesce reallocate job

restart Restart reallocate job

schedule Modify schedule of reallocate job

show Show reallocate job status

start Start reallocate job

stop Stop reallocate job

c83-PC::reallocate> version

NetApp Release 8.3RC1: Fri Oct 31 20:01:30 PDT 2014

But I prefer the options: * -read-realloc space-optimized for volumes * -free-space-realloc {on|off|no_redirect} for aggregates also both available from the cluster CLI (vol modify & aggr modify).

That way, there's a constant optimization running for data that's actually in use...

my 2c

Sebastian

On 6/26/2015 5:20 PM, Michael Bergman wrote:

Tony Bar wrote:

...

The reallocate commands are hidden in CDOT, but you can find a reference in this KB if you need them:

Hidden...? I don't see that but I may be missing something. I noticed that the man pages online at the Support site Documentation section seem to lack an entry for reallocate. Curious.

Aggregate reallocate (freespace) cannot be done from Cluster Lvl AFAICS, it has to be done inside the node that owns the Aggr in Q.

$ xxxx1 'set -priv diag -conf off; run xxxx1-08 "reallocate ?"'

Invalid subcommand '?' usage: reallocate on | off reallocate start [-t threshold] [-p] [-u] [-o] [-n] [-i interval] <path> | /vol/<volname> reallocate start -f [-p] [-u] <path> | /vol/<volname> reallocate start -A [-o] [-i interval] <aggr_name> NOTE: -A is for aggregate (freespace) reallocation. Do NOT use -A after growing an aggregate if you wish to optimize the layout of existing data; instead use reallocate start -f /vol/<volname> for each volume in the aggregate. reallocate status [-v] [<path> | <aggr_name>] reallocate stop <path> | <aggr_name> reallocate quiesce <path> | <aggr_name> reallocate restart [-i] <path> | <aggr_name> reallocate schedule {-d | -s <schedule>} <path> | <aggr_name> reallocate measure [-l logfile] [-t threshold] [-o] [-i interval] <path> | /vol/<volname>

$ xxxx1 version NetApp Release 8.3P1D4: Wed May 13 16:24:01 PDT 2015

$ xxxx1 reallocate ? measure Start reallocate measure job off Disable reallocate jobs on Enable reallocate jobs quiesce Quiesce reallocate job restart Restart reallocate job schedule Modify schedule of reallocate job show Show reallocate job status start Start reallocate job stop Stop reallocate job

$ xxxx1 reallocate start ? -vserver <vserver name> Vserver [-path] <text> Path { [[-interval|-i] <text>] Interval Schedule | [ -once|-o [true] ] Once | [ -force|-f [true] ] } Force { [ -space-optimized|-p [true] ] Space Optimized | [ -unshare|-u [true] ] } Unshare Deduplicated Blocks { [ -threshold|-t {3..10} ] Threshold | [ -no-check|-n [true] ] } No Threshold Check

---

Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

John Clear wrote:

As others have mentioned, you want to let deswizzling finish. You can tune the priority of the deswizzling process with 'wafl scan speed' to limit the impact of the scan.

Actually wafl scan speed is a global cap type of knob.

wafl scan speed is auto tuning (by setting it to 0 'zero') and turning it down lower than the 1300 or so that the auto tuner does anyway, probably isn't what you want in the general case. YMMV, depending on the size (model) of FAS machine this is.

There's a longer discussion here, but I believe it might be possible to fine tune the prio of the deswizzler scanner -- I'm not sure though I'm afraid. There are lots of "secret" knobs that are in theory turnable for various types of scanners, but it's not documented. Basically it's the teams that author or maintain the code that know what those knobs might do, but they are usually not experienced w.r.t. field workloads and hence cannot really give good advice on what way to turn the knobs

Now, disabling the deswizzler is easy, it's a global (hidden) option, once set it's visible:

"options wafl.deswizzle.enable off" will terminate any running deswizzlers so manually aborting them should not be necessary. "options wafl.deswizzle.enable on" will start the deswizzler on any volume that has swizzled blocks.

As long as this option is "off" the deswizzling will be done inside WAFL, on-the-fly, when data is read basically. This will of course slow access down somehwhat, at least on the first access. Depending on the power of your controller (the size of it) and the workload, this may be better for you, it depends.

I can say that when we Vol SnapMirrored billions of files from 100s of FlexVols 7mode -> C.DOT, the target system (all FAS8080) had wafl.deswizzle.enable off by design from day one. They still have. We have heavy workload, random R & W it's a sandstorm. Bogging the controllers down in s-Kahuna (before 8.3) with a bunch of basically never ending deswizzler scanners, is of no added value for us with the amount of cache and FlashPool we have

In 8.3 and onwards the deswizzler is in Waffinity, the VOL Affinity to be exact. So it's much more parallelised in 8.3 than before. OTOH it can make VOL become busier than you'd like... sure it's bg type work but it's there and it can disturb things w.r.t. user workload even if your're good on CPU cycles

Hope this helps some ppl at least!

/M

Thank you all for the comments. The result will be to leave the settings are is and the to change the snapmirror schedule to ondemand, so that process can complete, an each time reduce the snapmirror ondemand trigger

Again thanks and how someone else found this helpful in there struggles. /Mike

-----Original Message----- From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Michael Bergman Sent: Wednesday, June 24, 2015 6:58 PM To: Toasters Subject: Re: Deswizzle

John Clear wrote:

As others have mentioned, you want to let deswizzling finish. You can tune the priority of the deswizzling process with 'wafl scan speed' to limit the impact of the scan.

Actually wafl scan speed is a global cap type of knob.

wafl scan speed is auto tuning (by setting it to 0 'zero') and turning it down lower than the 1300 or so that the auto tuner does anyway, probably isn't what you want in the general case. YMMV, depending on the size (model) of FAS machine this is.

There's a longer discussion here, but I believe it might be possible to fine tune the prio of the deswizzler scanner -- I'm not sure though I'm afraid. There are lots of "secret" knobs that are in theory turnable for various types of scanners, but it's not documented. Basically it's the teams that author or maintain the code that know what those knobs might do, but they are usually not experienced w.r.t. field workloads and hence cannot really give good advice on what way to turn the knobs

Now, disabling the deswizzler is easy, it's a global (hidden) option, once set it's visible:

"options wafl.deswizzle.enable off" will terminate any running deswizzlers so manually aborting them should not be necessary. "options wafl.deswizzle.enable on" will start the deswizzler on any volume that has swizzled blocks.

As long as this option is "off" the deswizzling will be done inside WAFL, on-the-fly, when data is read basically. This will of course slow access down somehwhat, at least on the first access. Depending on the power of your controller (the size of it) and the workload, this may be better for you, it depends.

I can say that when we Vol SnapMirrored billions of files from 100s of FlexVols 7mode -> C.DOT, the target system (all FAS8080) had wafl.deswizzle.enable off by design from day one. They still have. We have heavy workload, random R & W it's a sandstorm. Bogging the controllers down in s-Kahuna (before 8.3) with a bunch of basically never ending deswizzler scanners, is of no added value for us with the amount of cache and FlashPool we have

In 8.3 and onwards the deswizzler is in Waffinity, the VOL Affinity to be exact. So it's much more parallelised in 8.3 than before. OTOH it can make VOL become busier than you'd like... sure it's bg type work but it's there and it can disturb things w.r.t. user workload even if your're good on CPU cycles

Hope this helps some ppl at least!

/M -- Michael Bergman Sr Systems Analyst / Storage Architect michael.bergman at ericsson dot com -- This communication is confidential. We only send and receive email on the basis of the terms set out at www.ericsson.com/email_disclaimer _______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

Recap and result(as promised) about the deswizzler scanner

I wrote:

The da** thing walks inodes, and in all snapshots dirs as well *sigh*. It can also easily swamp the disks because it reads data fairly inefficiently. [...]

I'll try and find out if a reallocate -A has an effect on what the deswizzler needs to do, and share my findings here later (after summer).

1. There is no connection between reallocate -A and what the deswizzler is doing. I'd recommend turning off deswizzler before doing reallocate -A, just to save some IOPS. The disks are hammered very hard by reallocate -A, and W latency will shoot up. A lot.

2. The deswizzler scanner will do, in the bg, what a certain WAFL message also does. But it does it on-demand when data is read. Meaning that the 1st time a block is read, it will be "fixed" if there's an extra pointer there.

In a HFC (high file count) environment it's pointless to have deswizzler on. It won't do anything that is fruitful for the performance, rather on the contrary. Expecially with the bigger FAS models (80x0) my recommendation is to always set

options wafl.deswizzle.enable off

for good.

/M

Hi,

On Mon, Aug 24, 2015 at 8:28 PM, Michael Bergman < michael.bergman@ericsson.com> wrote:

In a HFC (high file count) environment it's pointless to have deswizzler on. It won't do anything that is fruitful for the performance, rather on the contrary. Expecially with the bigger FAS models (80x0) my recommendation is to always set

options wafl.deswizzle.enable off

for good.

How can it be turned off in Clustered Data ONTAP ?

Kind regards, Filip