Not sure if this bug applies to you, but it’s fixed in 9.3P5 and there appears to be a workaround as well:

https://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=1144006

At least worth checking …

Anthony Bar

Berkeley Communications

www.berkcom.com

From: toasters-bounces@teaparty.net <toasters-bounces@teaparty.net> On Behalf Of jordan slingerland
Sent: Thursday, June 14, 2018 7:49 AM
To: Toasters <toasters@teaparty.net>
Subject: FAS8040 cpu pegged for over 1 month 24/7

I have an 8040 is at 100% cpu all the time on all cores. It has been like this for at least a month, my stats do not go back further than this. I expect the cpu was not this high before upgradin to 9.3P2. I feel like i should be able to get more than 10k ops out of a 150TB hybrid aggregate with 222 disks in it. Any help or feedback on performance expectations will be appreciated. Let me know if any stats would be usefull, i stripped my email down as my first try was rejected for being too big.

HOSTNAME::> node run -node HOSTNAME-0

HOSTNAME-01 HOSTNAME-02

HOSTNAME::> node run -node HOSTNAME-02

Type 'exit' or 'Ctrl-D' to return to the CLI

HOSTNAME-02> sysstat prif set diag priv set diag

Warning: These diagnostic commands are for use by NetApp

personnel only.

HOSTNAME-02*> sysstat -M 1

ANY1+ ANY2+ ANY3+ ANY4+ ANY5+ ANY6+ ANY7+ ANY8+ AVG CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 Nwk_Excl Nwk_Lg Nwk_Exmpt Protocol Storage Raid Raid_Ex Xor_Ex Target Kahuna WAFL_Ex(Kahu) WAFL_MPClean SM_Exempt Exempt SSAN_Ex Intr Host Ops/s CP

100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 0% 32% 0% 0% 0% 30% 10% 0% 0% 253%( 36%) 0% 0% 99% 18% 3% 353% 3748 100%

100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 0% 33% 0% 0% 0% 29% 10% 0% 0% 235%( 33%) 0% 0% 70% 19% 3% 398% 3959 100%

100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 0% 34% 0% 0% 0% 30% 9% 0% 0% 245%( 35%) 0% 0% 74% 19% 3% 385% 3802 100%

100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 0% 30% 0% 1% 0% 28% 8% 0% 0% 236%( 33%) 0% 0% 76% 17% 3% 399% 3367 100%

100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 1% 30% 0% 2% 0% 29% 8% 0% 0% 217%( 31%) 0% 0% 71% 16% 3% 422% 3427 100%

100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 1% 36% 0% 1% 1% 33% 9% 0% 0% 242%( 34%) 0% 0% 91% 19% 3% 362% 3754 100%

100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 1% 33% 0% 0% 0% 26% 4% 0% 0% 272%( 38%) 18% 0% 102% 18% 4% 320% 3496 60%

100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 1% 34% 0% 1% 0% 24% 4% 0% 0% 258%( 36%) 3% 0% 87% 20% 3% 364% 4001 0%

100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 1% 41% 0% 0% 0% 32% 5% 0% 0% 276%( 39%) 2% 0% 101% 24% 4% 312% 4752 0%

100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 1% 44% 0% 0% 0% 27% 4% 0% 0% 268%( 38%) 1% 0% 101% 25% 4% 323% 5077 28%

100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 1% 36% 0% 0% 0% 29% 2% 0% 0% 256%( 36%) 3% 0% 96% 21% 4% 352% 4105 2%

100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 1% 40% 0% 2% 0% 22% 1% 0% 0% 252%( 36%) 2% 0% 114% 23% 4% 339% 4801 0%

100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 0% 33% 0% 3% 0% 17% 1% 0% 0% 236%( 33%) 2% 0% 73% 20% 3% 410% 4280 0%

100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 0% 33% 0% 1% 0% 16% 2% 0% 0% 232%( 33%) 1% 0% 67% 20% 3% 424% 4219 0%

100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 0% 29% 0% 0% 0% 15% 1% 0% 0% 230%( 32%) 2% 0% 71% 18% 3% 429% 3719 0%

100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 1% 31% 0% 0% 0% 24% 2% 0% 0% 241%( 34%) 3% 0% 75% 19% 3% 400% 4206 0%

100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 1% 28% 0% 0% 0% 46% 11% 0% 0% 263%( 37%) 67% 0% 82% 17% 3% 280% 3496 59%

100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 0% 25% 0% 0% 1% 38% 7% 0% 0% 324%( 46%) 52% 0% 107% 16% 3% 226% 3127 100%

100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 0% 30% 0% 0% 1% 35% 7% 0% 1% 285%( 40%) 49% 0% 86% 18% 3% 285% 3633 100%

0% 0% 94% 20% 4% 341% 4300 100%

HOSTNAME-02*> sysstat -x 1

CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP_Ty CP_Ph Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s

in out read write read write age hit time [T--H--F--N--B--O--#--:] [n--v--p--f] util in out in out

100% 6 0 0 3907 100046 124797 278121 20862 0 0 3s 94% 0% 0--0--0--0--0--0--0--0 0--0--0--0 25% 3 0 3898 0 0 99115 127259

100% 3 0 0 3869 60123 134198 411571 20704 0 0 4s 94% 0% 0--0--0--0--0--0--0--0 0--0--0--0 33% 5 0 3861 0 0 59346 120058

100% 0 0 0 4089 27375 116197 225213 18145 0 0 4s 93% 0% 0--0--0--0--0--0--0--0 0--0--0--0 23% 0 0 4089 0 0 26622 115872

100% 13 0 0 4175 35817 111147 260996 28849 0 0 2s 94% 0% 0--0--0--0--0--0--0--0 0--0--0--0 25% 0 0 4162 0 0 34874 125888

100% 2 0 0 4143 56386 99541 401056 23882 0 0 0s 93% 0% 0--0--0--0--0--0--0--0 0--0--0--0 37% 17 0 4124 0 0 55471 105104

100% 11 0 0 3750 25706 149389 364912 154655 0 0 2s 93% 43% 1--0--0--0--0--0--0--0 0--1--0--0 29% 0 0 3739 0 0 24904 119412

100% 4 0 0 3395 36225 45390 280507 211869 0 0 3s 92% 100% 0--0--0--0--0--0--0--1 0--0--1--0 40% 0 0 3391 0 0 35744 81843

100% 25 0 0 4713 46662 149078 307272 93644 0 0 4s 95% 100% 0--0--0--0--0--0--0--1 0--0--0--1 42% 184 0 4504 0 0 45774 118418

100% 2 0 0 3930 51697 110975 221814 105300 0 0 3s 93% 100% 0--0--0--0--0--0--0--1 0--0--0--1 26% 0 0 3928 0 0 50821 108484

100% 5 0 0 4148 54985 137949 267436 208711 0 0 3s 93% 100% 1--0--0--0--0--0--0--1 1--0--0--1 31% 9 0 4134 0 0 53645 142356

100% 16 0 0 4883 81959 108174 364367 666466 0 0 4s 94% 100% 0--0--0--0--0--0--0--2 0--0--0--2 40% 1 0 4866 0 0 81554 95623

100% 5 0 0 6111 74315 122018 268918 79372 0 0 3s 95% 100% 0--0--0--0--0--0--0--1 0--0--0--1 26% 4 0 6102 0 0 72956 126959

100% 3 0 0 4658 50514 119345 218663 157863 0 0 3s 94% 100% 0--0--0--0--0--0--0--1 0--0--0--1 22% 0 0 4655 0 0 50021 112484

100% 2 0 0 4125 61813 137044 244725 223996 0 0 3s 93% 100% 0--0--0--0--0--0--0--1 0--0--0--1 23% 0 0 4123 0 0 59479 135345

100% 6 0 0 4005 48098 158865 283920 54318 0 0 4s 95% 15% 0--0--0--0--0--0--0--0 0--0--0--0 26% 14 0 3985 0 0 47165 162826

100% 24 0 0 4490 45946 155717 266290 96976 0 0 2s 93% 0% 0--0--0--0--0--0--0--0 0--0--0--0 28% 0 0 4466 0 0 44575 146650

100% 7 0 0 4935 52081 155766 331777 11753 0 0 4s 93% 0% 0--0--0--0--0--0--0--0 0--0--0--0 32% 2 0 4926 0 0 51171 165626

100% 0 0 0 6518 39526 179763 350349 24130 0 0 1s 92% 0% 0--0--0--0--0--0--0--0 0--0--0--0 29% 4 0 6514 0 0 38678 177139

HOSTNAME-02*> qos exit

logout

HOSTNAME::> qos statistics characteristics show -iterations 0 -rows 10

Policy Group IOPS Throughput Request size Read Concurrency Is Adaptive?

-------------------- -------- --------------- ------------ ---- ----------- ------------

-total- 8094 321.31MB/s 41627B 41% 17 -

data02_PROD_2 1626 101.69MB/s 65578B 80% 7 false

DEV_DATA_DEV_2 1528 147.53MB/s 101219B 57% 5 false

_System-Work 1243 4.40KB/s 3B 1% 0 false

DEV_DATA_DEV 749 6.51MB/s 9121B 93% 1 false

data02_PROD 667 9.48MB/s 14911B 0% 1 false

DEV_OS_DEV_2 470 28.78MB/s 64214B 35% 2 false

data04_PROD 256 4.35MB/s 17812B 8% 0 false

data03_PROD 191 3.64MB/s 19971B 0% 0 false

os03_PROD_2 191 1.55MB/s 8496B 0% 0 false

shares02_PROD 183 1382.41KB/s 7721B 84% 0 false

-total- 9542 347.25MB/s 38158B 42% 21 -

DEV_DATA_DEV_2 1675 83.11MB/s 52028B 71% 4 false

data02_PROD_2 1547 103.42MB/s 70112B 86% 7 false

_System-Work 1191 6.58KB/s 5B 0% 0 false

DEV_OS_DEV_2 860 30.42MB/s 37095B 37% 2 false

DEV_DATA_DEV 667 5.22MB/s 8208B 96% 0 false

data03_PROD_2 627 11.51MB/s 19255B 2% 3 false

data02_PROD 528 7.24MB/s 14370B 0% 0 false

shares02_PROD 526 80.62MB/s 160604B 66% 3 false

data01_PROD 411 4.87MB/s 12431B 0% 0 false

data03_PROD 377 4.95MB/s 13746B 0% 0 false

Policy Group IOPS Throughput Request size Read Concurrency Is Adaptive?

-------------------- -------- --------------- ------------ ---- ----------- ------------

-total- 11004 218.56MB/s 20827B 36% 15 -

_System-Work 3750 24.82KB/s 6B 0% 0 false

DEV_DATA_DEV_2 1572 25.67MB/s 17129B 78% 2 false

data02_PROD_2 1568 91.24MB/s 61028B 99% 7 false

DEV_DATA_DEV 793 6.19MB/s 8185B 96% 0 false

os03_PROD 652 5.18MB/s 8322B 4% 1 false

DEV_OS_DEV_2 549 31.44MB/s 60082B 44% 2 false

data01_PROD 411 6.96MB/s 17753B 0% 0 false

shares02_PROD 304 34.42MB/s 118717B 42% 2 false

data02_PROD 261 4.56MB/s 18303B 0% 0 false

data03_PROD 237 3.59MB/s 15904B 0% 0 false