Hello

I attach a wafltop output run for 10 minutes. We are experiencing a fair amount of CPU contention and latency. Does anyone have any input on the below?

I also show a wafl scan status which shows there is a lot of activity. Is that normal?

wafltop stop

CPU Utilization Percent Application Total STRIPE VOL_LOG VOL_VBN VBN VOL AGGR_VBN AGGR SERIAL XCleaner ----------- -------- -------- -------- -------- -------- -------- -------- -------- -------- ------- aggrsata2:SV_L_LUN_VOL0199:other:_WAFL_BACKDOOR: 15 0 0 0 0 0 0 0 15 0 other:other:other:FREE_CACHE_REFILL: 8 5 0 3 0 0 0 0 0 0 aggrsata2:SV_L_LUN_VOL0199:scanner:_WAFL_BACKDOOR: 5 0 0 0 0 0 0 0 5 0 other:other:other:RAIDIO_READ_DONE: 5 4 0 1 0 0 0 0 0 0 aggrvm:L_NFS_VOL0178:nfsv3:WAFL_WRITE: 4 4 0 0 0 0 0 0 0 0 other:other:other:WAFL_WRITE_DONE: 4 4 0 0 0 0 0 0 0 0 aggrvm:L_NFS_VOL0176:nfsv3:WAFL_READ: 3 3 0 0 0 0 0 0 0 0 aggrvm:L_NFS_VOL0183:nfsv3:WAFL_WRITE: 3 3 0 0 0 0 0 0 0 0 aggrsata1:L_LUN_VOL0198:iscsi:WAFL_READ: 3 3 0 0 0 0 0 0 0 0 aggrvm:L_NFS_VOL0176:nfsv3:WAFL_WRITE: 3 3 0 0 0 0 0 0 0 0 aggrvm:L_NFS_VOL0177:nfsv3:WAFL_WRITE: 3 3 0 0 0 0 0 0 0 0 aggrvm:L_NFS_VOL0179:nfsv3:WAFL_READ: 2 2 0 0 0 0 0 0 0 0 aggrsata1:L_LUN_VOL0198:iscsi:WAFL_WRITE: 2 2 0 0 0 0 0 0 0 0 aggrsata2:SV_L_LUN_VOL0198:scanner:_WAFL_BACKDOOR: 2 0 0 0 0 0 0 0 2 0 aggrsata2:L_LUN_VOL0199:iscsi:WAFL_READ: 2 2 0 0 0 0 0 0 0 0 aggrvm:L_NFS_VOL0179:nfsv3:WAFL_WRITE: 2 2 0 0 0 0 0 0 0 0 aggrvm:L_NFS_VOL0161:nfsv3:WAFL_WRITE: 1 1 0 0 0 0 0 0 0 0 aggrvm:L_NFS_VOL0174:nfsv3:WAFL_WRITE: 1 1 0 0 0 0 0 0 0 0 aggrvm::walloc:WAFL_BLKFREE_PVOL_CO: 1 0 0 0 0 0 1 0 0 0 other:other:other:WAFFINITY_NVLOG_COMP: 1 1 0 0 0 0 0 0 0 0

CPU Time us Application Total STRIPE VOL_LOG VOL_VBN VBN VOL AGGR_VBN AGGR SERIAL XCleaner ----------- -------- -------- -------- -------- -------- -------- -------- -------- -------- ------- aggrsata2:L_LUN_VOL0199:file i/o:WAFL_LOOKUP: 46164 0 0 0 0 0 0 0 46164 0 aggr0:vol0:file i/o:WAFL_LOOKUP: 33469 0 0 0 0 0 0 0 33469 0 aggr0:vol0:file i/o:WAFL_READDIR: 31339 0 0 0 0 0 0 0 31339 0 aggrsata1::file i/o:WAFL_LOOKUP: 4725 0 0 0 0 0 0 0 4725 0 aggrvm:L_NFS_VOL0179:nfsv3:WAFL_READ: 4479 4349 130 0 0 0 0 0 0 0 aggrsata2:SV_L_LUN_VOL0198:file i/o:WAFL_GETATTR: 3639 3639 0 0 0 0 0 0 0 0 aggrsata2:SV_L_LUN_VOL0198:file i/o:WAFL_READ: 2397 1635 0 0 0 0 0 0 762 0 aggrsata1:L_LUN_VOL0201:iscsi:WAFL_READ: 2252 2191 55 0 0 0 0 0 6 0 aggrvm:L_NFS_VOL0183:nfsv3:WAFL_READDIR: 1650 1628 22 0 0 0 0 0 0 0 aggrvm:L_NFS_VOL0174:nfsv3:WAFL_READ: 1542 1514 28 0 0 0 0 0 0 0 aggrvm:P_NFS_WPF0160:nfsv3:WAFL_DOACCESS: 1525 1525 0 0 0 0 0 0 0 0 aggrvm:L_NFS_VOL0161:nfsv3:WAFL_READ: 1520 1497 23 0 0 0 0 0 0 0 aggrsata2:L_LUN_VOL0199:iscsi:WAFL_READ: 1480 1441 39 0 0 0 0 0 0 0 aggrvm:L_NFS_VOL0158:nfsv3:WAFL_READ: 1378 1355 23 0 0 0 0 0 0 0 aggr0:vol0:spinvfs:WAFL_READ: 1296 1289 7 0 0 0 0 0 0 0 aggrsata2:L_LUN_VOL0199:file i/o:WAFL_READ: 1283 900 0 0 0 357 0 0 26 0 aggrvm:L_NFS_VOL0181:nfsv3:WAFL_READ: 1270 1263 7 0 0 0 0 0 0 0 aggrvm:L_LUN_VOL0152:iscsi:WAFL_READ: 1234 1213 21 0 0 0 0 0 0 0 aggr0:vol0:file i/o:WAFL_READ: 1198 1189 1 0 0 0 0 0 8 0 aggrsata2:SV_L_LUN_VOL0199:file i/o:WAFL_READ: 1195 791 0 0 0 0 0 0 404 0

Latency Application Latency ms ----------- ---------- aggrvm:L_NFS_VOL0180:nfsv3:WAFL_SETATTR: 936.000 aggrvm:L_NFS_VOL0179:nfsv3:WAFL_READ: 13.501 aggrvm:L_NFS_VOL0183:nfsv3:WAFL_READ: 10.962 aggrvm:P_NFS_WPF0160:nfsv3:WAFL_READ: 9.148 aggrvm:L_NFS_VOL0180:nfsv3:WAFL_READ: 8.713 aggrvm:L_NFS_VOL0181:nfsv3:WAFL_READ: 8.205 aggrsata1:L_LUN_VOL0198:iscsi:WAFL_READ: 8.082 aggrvm:L_NFS_VOL0161:nfsv3:WAFL_READ: 7.525 aggrvm:L_NFS_VOL0179:nfsv3:WAFL_READDIR: 6.000 aggrvm:L_NFS_VOL0177:nfsv3:WAFL_READ: 5.825 aggrvm:L_NFS_VOL0180:nfsv3:WAFL_LOOKUP: 5.000 aggrvm:L_NFS_VOL0183:nfsv3:WAFL_READDIR: 5.000 aggrvm:L_NFS_VOL0158:nfsv3:WAFL_READ: 4.926 aggrvm:L_NFS_VOL0178:nfsv3:WAFL_READ: 4.775 aggrvm:L_NFS_VOL0174:nfsv3:WAFL_GETATTR: 4.666 aggr0:vol0:spinvfs:WAFL_READ: 3.250 aggrsata1:L_LUN_VOL0201:iscsi:WAFL_READ: 3.111 aggrvm:P_NFS_WPF0160:nfsv3:WAFL_STATFS: 3.000 aggrsata1:L_LUN_VOL0200:iscsi:WAFL_WRITE: 2.435 aggrsata2:L_LUN_VOL0199:iscsi:WAFL_READ: 2.289

Application System Latency ms ----------- ----------------- aggrsata2:SV_L_LUN_VOL0198:file i/o:WAFL_WRITE: 21.786 aggrsata2:SV_L_LUN_VOL0198:file i/o:WAFL_READ: 21.000 aggrsata2:SV_L_LUN_VOL0198:file i/o:WAFL_GETATTR: 8.000 aggrsata1:L_LUN_VOL0198:file i/o:WAFL_READ: 4.454 aggrvm:L_NFS_VOL0158:file i/o:WAFL_WRITE: 4.000 aggrsata2:L_LUN_VOL0199:file i/o:WAFL_READ: 3.333 aggrvm:L_NFS_VOL0180:file i/o:WAFL_WRITE: 3.261 aggrsata1:L_LUN_VOL0198:file i/o:WAFL_VDISK_SETTYPE: 3.000 aggrvm:L_NFS_VOL0224:file i/o:WAFL_WRITE: 3.000 aggrvm:L_NFS_VOL0177:file i/o:WAFL_WRITE: 2.245 aggrvm:L_NFS_VOL0183:file i/o:WAFL_WRITE: 2.220 aggrvm:L_NFS_VOL0176:file i/o:WAFL_WRITE: 2.201 aggrvm:L_NFS_VOL0179:file i/o:WAFL_WRITE: 2.136 aggrvm:L_NFS_VOL0181:file i/o:WAFL_WRITE: 2.050 aggrsata2:SV_L_LUN_VOL0199:file i/o:WAFL_GETATTR: 1.750 aggrsata2:L_LUN_VOL0199:file i/o:WAFL_LOOKUP: 1.500 aggrsata2:L_LUN_VOL0199:file i/o:WAFL_VDISK_SETTYPE: 1.444 aggrsata1:L_LUN_VOL0198:file i/o:WAFL_STATFS: 1.000 aggrsata1::file i/o:WAFL_LOOKUP: 1.000 aggrvm:L_NFS_VOL0178:file i/o:WAFL_STATFS: 1.000

I/O utilization ---------MB Read---------- ---------MB Write--------- --------IOs Read---------- --------IOs Write--------- Application MB Total Standard PAM Hybrid Standard PAM Hybrid Standard PAM Hybrid Standard PAM Hybrid ----------- -------- -------- -------- -------- -------- -------- -------- -------- -------- -------- -------- -------- -------- aggrsata1:L_LUN_VOL0198:iscsi:WAFL_READ: 17124 15503 1621 0 0 0 0 412113 88557 0 0 0 0 other:other:other:FREE_CACHE_REFILL: 13732 0 0 0 0 13732 0 0 0 0 0 54917 0 aggrvm:L_NFS_VOL0179:nfsv3:WAFL_READ: 10348 9956 392 0 0 0 0 483841 58184 0 0 0 0 aggrvm:L_NFS_VOL0176:nfsv3:WAFL_READ: 7954 6989 965 0 0 0 0 407363 124931 0 0 0 0 aggrsata1:L_LUN_VOL0198:snapmirror:WAFL_SNAP_DIFF: 7523 6052 1471 0 0 0 0 249428 54191 0 0 0 0 other:other:other:WAFFINITY_RA_STRIPE_: 7432 7339 93 0 0 0 0 159291 4544 0 0 0 0 aggrvm:L_NFS_VOL0176:nfsv3:WAFL_WRITE: 7288 384 152 0 6752 0 0 98462 38971 0 0 0 0 aggrsata2:SV_L_LUN_VOL0198:file i/o:WAFL_WRITE: 6759 148 40 0 6571 0 0 37781 10251 0 0 0 0 aggrsata1:L_LUN_VOL0198:iscsi:WAFL_WRITE: 6726 65 112 0 6549 0 0 16507 28706 0 0 0 0 aggrvm:L_NFS_VOL0177:nfsv3:WAFL_WRITE: 5538 426 102 0 5010 0 0 109134 26076 0 0 0 0 aggrvm:L_NFS_VOL0178:nfsv3:WAFL_WRITE: 5444 442 171 0 4831 0 0 113018 43797 0 0 0 0 aggrvm:L_NFS_VOL0183:nfsv3:WAFL_WRITE: 5298 419 125 0 4754 0 0 107294 32047 0 0 0 0 aggrvm:L_NFS_VOL0179:nfsv3:WAFL_WRITE: 4153 381 92 0 3680 0 0 97639 23545 0 0 0 0 aggrvm:L_NFS_VOL0174:nfsv3:WAFL_WRITE: 3748 84 32 0 3632 0 0 21436 8233 0 0 0 0 aggrsata2:L_LUN_VOL0199:iscsi:WAFL_READ: 3692 3482 210 0 0 0 0 90262 10488 0 0 0 0 aggrvm:L_NFS_VOL0180:nfsv3:WAFL_WRITE: 3367 205 85 0 3077 0 0 52391 21642 0 0 0 0 other:other:other:WAFL_WRITE_DONE: 3325 0 0 0 0 3325 0 0 0 0 0 13309 0 aggrsata2:L_LUN_VOL0199:iscsi:WAFL_WRITE: 2738 25 15 0 2698 0 0 6364 3849 0 0 0 0 aggrsata1:L_LUN_VOL0198:snapmirror:WAFL_SNAP_READ: 2340 1838 502 0 0 0 0 356929 96469 0 0 0 0 aggrsata2:SV_L_LUN_VOL0199:other:_WAFL_BACKDOOR: 2265 12 2241 0 12 0 0 3029 536488 0 0 0 0

NVLog Utilization Application NVLog in KB/s ----------- ------------------ aggrsata2:SV_L_LUN_VOL0198:file i/o:WAFL_WRITE: 10036 aggrsata1:L_LUN_VOL0198:iscsi:WAFL_WRITE: 9692 aggrvm:L_NFS_VOL0176:nfsv3:WAFL_WRITE: 9452 aggrvm:L_NFS_VOL0177:nfsv3:WAFL_WRITE: 7462 aggrvm:L_NFS_VOL0183:nfsv3:WAFL_WRITE: 7159 aggrvm:L_NFS_VOL0178:nfsv3:WAFL_WRITE: 7140 aggrsata2:L_LUN_VOL0199:iscsi:WAFL_WRITE: 7083 aggrvm:L_NFS_VOL0174:nfsv3:WAFL_WRITE: 5422 aggrvm:L_NFS_VOL0179:nfsv3:WAFL_WRITE: 5358 aggrvm:L_NFS_VOL0180:nfsv3:WAFL_WRITE: 4222 aggrvm:L_NFS_VOL0182:nfsv3:WAFL_WRITE: 2698 aggrvm:L_NFS_VOL0161:nfsv3:WAFL_WRITE: 1944 aggrvm:L_NFS_VOL0181:nfsv3:WAFL_WRITE: 1647 aggrvm:P_NFS_WPF0160:nfsv3:WAFL_WRITE: 842 aggrvm:L_NFS_VOL0158:nfsv3:WAFL_WRITE: 351 aggrvm:L_NFS_VOL0176:file i/o:WAFL_WRITE: 70 aggrvm:L_NFS_VOL0177:file i/o:WAFL_WRITE: 53 aggrvm:L_NFS_VOL0183:file i/o:WAFL_WRITE: 50 aggrvm:L_NFS_VOL0179:file i/o:WAFL_WRITE: 42 aggrvm:L_NFS_VOL0180:file i/o:WAFL_WRITE: 31

Application NVLog_b2b in KB/s ----------- ----------------------