Hello
I attach a wafltop output run for 10 minutes. We are experiencing a fair amount of CPU contention and latency. Does anyone have any input on the below?
I also show a wafl scan status which shows there is a lot of activity. Is that normal?
wafltop stop
CPU Utilization Percent
Application Total STRIPE VOL_LOG VOL_VBN VBN VOL AGGR_VBN AGGR SERIAL XCleaner
----------- -------- -------- -------- -------- -------- -------- -------- -------- -------- -------
aggrsata2:SV_L_LUN_VOL0199:other:_WAFL_BACKDOOR: 15 0 0 0 0 0 0 0 15 0
other:other:other:FREE_CACHE_REFILL: 8 5 0 3 0 0 0 0 0 0
aggrsata2:SV_L_LUN_VOL0199:scanner:_WAFL_BACKDOOR: 5 0 0 0 0 0 0 0 5 0
other:other:other:RAIDIO_READ_DONE: 5 4 0 1 0 0 0 0 0 0
aggrvm:L_NFS_VOL0178:nfsv3:WAFL_WRITE: 4 4 0 0 0 0 0 0 0 0
other:other:other:WAFL_WRITE_DONE: 4 4 0 0 0 0 0 0 0 0
aggrvm:L_NFS_VOL0176:nfsv3:WAFL_READ: 3 3 0 0 0 0 0 0 0 0
aggrvm:L_NFS_VOL0183:nfsv3:WAFL_WRITE: 3 3 0 0 0 0 0 0 0 0
aggrsata1:L_LUN_VOL0198:iscsi:WAFL_READ: 3 3 0 0 0 0 0 0 0 0
aggrvm:L_NFS_VOL0176:nfsv3:WAFL_WRITE: 3 3 0 0 0 0 0 0 0 0
aggrvm:L_NFS_VOL0177:nfsv3:WAFL_WRITE: 3 3 0 0 0 0 0 0 0 0
aggrvm:L_NFS_VOL0179:nfsv3:WAFL_READ: 2 2 0 0 0 0 0 0 0 0
aggrsata1:L_LUN_VOL0198:iscsi:WAFL_WRITE: 2 2 0 0 0 0 0 0 0 0
aggrsata2:SV_L_LUN_VOL0198:scanner:_WAFL_BACKDOOR: 2 0 0 0 0 0 0 0 2 0
aggrsata2:L_LUN_VOL0199:iscsi:WAFL_READ: 2 2 0 0 0 0 0 0 0 0
aggrvm:L_NFS_VOL0179:nfsv3:WAFL_WRITE: 2 2 0 0 0 0 0 0 0 0
aggrvm:L_NFS_VOL0161:nfsv3:WAFL_WRITE: 1 1 0 0 0 0 0 0 0 0
aggrvm:L_NFS_VOL0174:nfsv3:WAFL_WRITE: 1 1 0 0 0 0 0 0 0 0
aggrvm::walloc:WAFL_BLKFREE_PVOL_CO: 1 0 0 0 0 0 1 0 0 0
other:other:other:WAFFINITY_NVLOG_COMP: 1 1 0 0 0 0 0 0 0 0
CPU Time us
Application Total STRIPE VOL_LOG VOL_VBN VBN VOL AGGR_VBN AGGR SERIAL XCleaner
----------- -------- -------- -------- -------- -------- -------- -------- -------- -------- -------
aggrsata2:L_LUN_VOL0199:file i/o:WAFL_LOOKUP: 46164 0 0 0 0 0 0 0 46164 0
aggr0:vol0:file i/o:WAFL_LOOKUP: 33469 0 0 0 0 0 0 0 33469 0
aggr0:vol0:file i/o:WAFL_READDIR: 31339 0 0 0 0 0 0 0 31339 0
aggrsata1::file i/o:WAFL_LOOKUP: 4725 0 0 0 0 0 0 0 4725 0
aggrvm:L_NFS_VOL0179:nfsv3:WAFL_READ: 4479 4349 130 0 0 0 0 0 0 0
aggrsata2:SV_L_LUN_VOL0198:file i/o:WAFL_GETATTR: 3639 3639 0 0 0 0 0 0 0 0
aggrsata2:SV_L_LUN_VOL0198:file i/o:WAFL_READ: 2397 1635 0 0 0 0 0 0 762 0
aggrsata1:L_LUN_VOL0201:iscsi:WAFL_READ: 2252 2191 55 0 0 0 0 0 6 0
aggrvm:L_NFS_VOL0183:nfsv3:WAFL_READDIR: 1650 1628 22 0 0 0 0 0 0 0
aggrvm:L_NFS_VOL0174:nfsv3:WAFL_READ: 1542 1514 28 0 0 0 0 0 0 0
aggrvm:P_NFS_WPF0160:nfsv3:WAFL_DOACCESS: 1525 1525 0 0 0 0 0 0 0 0
aggrvm:L_NFS_VOL0161:nfsv3:WAFL_READ: 1520 1497 23 0 0 0 0 0 0 0
aggrsata2:L_LUN_VOL0199:iscsi:WAFL_READ: 1480 1441 39 0 0 0 0 0 0 0
aggrvm:L_NFS_VOL0158:nfsv3:WAFL_READ: 1378 1355 23 0 0 0 0 0 0 0
aggr0:vol0:spinvfs:WAFL_READ: 1296 1289 7 0 0 0 0 0 0 0
aggrsata2:L_LUN_VOL0199:file i/o:WAFL_READ: 1283 900 0 0 0 357 0 0 26 0
aggrvm:L_NFS_VOL0181:nfsv3:WAFL_READ: 1270 1263 7 0 0 0 0 0 0 0
aggrvm:L_LUN_VOL0152:iscsi:WAFL_READ: 1234 1213 21 0 0 0 0 0 0 0
aggr0:vol0:file i/o:WAFL_READ: 1198 1189 1 0 0 0 0 0 8 0
aggrsata2:SV_L_LUN_VOL0199:file i/o:WAFL_READ: 1195 791 0 0 0 0 0 0 404 0
Latency
Application Latency ms
----------- ----------
aggrvm:L_NFS_VOL0180:nfsv3:WAFL_SETATTR: 936.000
aggrvm:L_NFS_VOL0179:nfsv3:WAFL_READ: 13.501
aggrvm:L_NFS_VOL0183:nfsv3:WAFL_READ: 10.962
aggrvm:P_NFS_WPF0160:nfsv3:WAFL_READ: 9.148
aggrvm:L_NFS_VOL0180:nfsv3:WAFL_READ: 8.713
aggrvm:L_NFS_VOL0181:nfsv3:WAFL_READ: 8.205
aggrsata1:L_LUN_VOL0198:iscsi:WAFL_READ: 8.082
aggrvm:L_NFS_VOL0161:nfsv3:WAFL_READ: 7.525
aggrvm:L_NFS_VOL0179:nfsv3:WAFL_READDIR: 6.000
aggrvm:L_NFS_VOL0177:nfsv3:WAFL_READ: 5.825
aggrvm:L_NFS_VOL0180:nfsv3:WAFL_LOOKUP: 5.000
aggrvm:L_NFS_VOL0183:nfsv3:WAFL_READDIR: 5.000
aggrvm:L_NFS_VOL0158:nfsv3:WAFL_READ: 4.926
aggrvm:L_NFS_VOL0178:nfsv3:WAFL_READ: 4.775
aggrvm:L_NFS_VOL0174:nfsv3:WAFL_GETATTR: 4.666
aggr0:vol0:spinvfs:WAFL_READ: 3.250
aggrsata1:L_LUN_VOL0201:iscsi:WAFL_READ: 3.111
aggrvm:P_NFS_WPF0160:nfsv3:WAFL_STATFS: 3.000
aggrsata1:L_LUN_VOL0200:iscsi:WAFL_WRITE: 2.435
aggrsata2:L_LUN_VOL0199:iscsi:WAFL_READ: 2.289
Application System Latency ms
----------- -----------------
aggrsata2:SV_L_LUN_VOL0198:file i/o:WAFL_WRITE: 21.786
aggrsata2:SV_L_LUN_VOL0198:file i/o:WAFL_READ: 21.000
aggrsata2:SV_L_LUN_VOL0198:file i/o:WAFL_GETATTR: 8.000
aggrsata1:L_LUN_VOL0198:file i/o:WAFL_READ: 4.454
aggrvm:L_NFS_VOL0158:file i/o:WAFL_WRITE: 4.000
aggrsata2:L_LUN_VOL0199:file i/o:WAFL_READ: 3.333
aggrvm:L_NFS_VOL0180:file i/o:WAFL_WRITE: 3.261
aggrsata1:L_LUN_VOL0198:file i/o:WAFL_VDISK_SETTYPE: 3.000
aggrvm:L_NFS_VOL0224:file i/o:WAFL_WRITE: 3.000
aggrvm:L_NFS_VOL0177:file i/o:WAFL_WRITE: 2.245
aggrvm:L_NFS_VOL0183:file i/o:WAFL_WRITE: 2.220
aggrvm:L_NFS_VOL0176:file i/o:WAFL_WRITE: 2.201
aggrvm:L_NFS_VOL0179:file i/o:WAFL_WRITE: 2.136
aggrvm:L_NFS_VOL0181:file i/o:WAFL_WRITE: 2.050
aggrsata2:SV_L_LUN_VOL0199:file i/o:WAFL_GETATTR: 1.750
aggrsata2:L_LUN_VOL0199:file i/o:WAFL_LOOKUP: 1.500
aggrsata2:L_LUN_VOL0199:file i/o:WAFL_VDISK_SETTYPE: 1.444
aggrsata1:L_LUN_VOL0198:file i/o:WAFL_STATFS: 1.000
aggrsata1::file i/o:WAFL_LOOKUP: 1.000
aggrvm:L_NFS_VOL0178:file i/o:WAFL_STATFS: 1.000
I/O utilization
---------MB Read---------- ---------MB Write--------- --------IOs Read---------- --------IOs Write---------
Application MB Total Standard PAM Hybrid Standard PAM Hybrid Standard PAM Hybrid Standard PAM Hybrid
----------- -------- -------- -------- -------- -------- -------- -------- -------- -------- -------- -------- -------- --------
aggrsata1:L_LUN_VOL0198:iscsi:WAFL_READ: 17124 15503 1621 0 0 0 0 412113 88557 0 0 0 0
other:other:other:FREE_CACHE_REFILL: 13732 0 0 0 0 13732 0 0 0 0 0 54917 0
aggrvm:L_NFS_VOL0179:nfsv3:WAFL_READ: 10348 9956 392 0 0 0 0 483841 58184 0 0 0 0
aggrvm:L_NFS_VOL0176:nfsv3:WAFL_READ: 7954 6989 965 0 0 0 0 407363 124931 0 0 0 0
aggrsata1:L_LUN_VOL0198:snapmirror:WAFL_SNAP_DIFF: 7523 6052 1471 0 0 0 0 249428 54191 0 0 0 0
other:other:other:WAFFINITY_RA_STRIPE_: 7432 7339 93 0 0 0 0 159291 4544 0 0 0 0
aggrvm:L_NFS_VOL0176:nfsv3:WAFL_WRITE: 7288 384 152 0 6752 0 0 98462 38971 0 0 0 0
aggrsata2:SV_L_LUN_VOL0198:file i/o:WAFL_WRITE: 6759 148 40 0 6571 0 0 37781 10251 0 0 0 0
aggrsata1:L_LUN_VOL0198:iscsi:WAFL_WRITE: 6726 65 112 0 6549 0 0 16507 28706 0 0 0 0
aggrvm:L_NFS_VOL0177:nfsv3:WAFL_WRITE: 5538 426 102 0 5010 0 0 109134 26076 0 0 0 0
aggrvm:L_NFS_VOL0178:nfsv3:WAFL_WRITE: 5444 442 171 0 4831 0 0 113018 43797 0 0 0 0
aggrvm:L_NFS_VOL0183:nfsv3:WAFL_WRITE: 5298 419 125 0 4754 0 0 107294 32047 0 0 0 0
aggrvm:L_NFS_VOL0179:nfsv3:WAFL_WRITE: 4153 381 92 0 3680 0 0 97639 23545 0 0 0 0
aggrvm:L_NFS_VOL0174:nfsv3:WAFL_WRITE: 3748 84 32 0 3632 0 0 21436 8233 0 0 0 0
aggrsata2:L_LUN_VOL0199:iscsi:WAFL_READ: 3692 3482 210 0 0 0 0 90262 10488 0 0 0 0
aggrvm:L_NFS_VOL0180:nfsv3:WAFL_WRITE: 3367 205 85 0 3077 0 0 52391 21642 0 0 0 0
other:other:other:WAFL_WRITE_DONE: 3325 0 0 0 0 3325 0 0 0 0 0 13309 0
aggrsata2:L_LUN_VOL0199:iscsi:WAFL_WRITE: 2738 25 15 0 2698 0 0 6364 3849 0 0 0 0
aggrsata1:L_LUN_VOL0198:snapmirror:WAFL_SNAP_READ: 2340 1838 502 0 0 0 0 356929 96469 0 0 0 0
aggrsata2:SV_L_LUN_VOL0199:other:_WAFL_BACKDOOR: 2265 12 2241 0 12 0 0 3029 536488 0 0 0 0
NVLog Utilization
Application NVLog in KB/s
----------- ------------------
aggrsata2:SV_L_LUN_VOL0198:file i/o:WAFL_WRITE: 10036
aggrsata1:L_LUN_VOL0198:iscsi:WAFL_WRITE: 9692
aggrvm:L_NFS_VOL0176:nfsv3:WAFL_WRITE: 9452
aggrvm:L_NFS_VOL0177:nfsv3:WAFL_WRITE: 7462
aggrvm:L_NFS_VOL0183:nfsv3:WAFL_WRITE: 7159
aggrvm:L_NFS_VOL0178:nfsv3:WAFL_WRITE: 7140
aggrsata2:L_LUN_VOL0199:iscsi:WAFL_WRITE: 7083
aggrvm:L_NFS_VOL0174:nfsv3:WAFL_WRITE: 5422
aggrvm:L_NFS_VOL0179:nfsv3:WAFL_WRITE: 5358
aggrvm:L_NFS_VOL0180:nfsv3:WAFL_WRITE: 4222
aggrvm:L_NFS_VOL0182:nfsv3:WAFL_WRITE: 2698
aggrvm:L_NFS_VOL0161:nfsv3:WAFL_WRITE: 1944
aggrvm:L_NFS_VOL0181:nfsv3:WAFL_WRITE: 1647
aggrvm:P_NFS_WPF0160:nfsv3:WAFL_WRITE: 842
aggrvm:L_NFS_VOL0158:nfsv3:WAFL_WRITE: 351
aggrvm:L_NFS_VOL0176:file i/o:WAFL_WRITE: 70
aggrvm:L_NFS_VOL0177:file i/o:WAFL_WRITE: 53
aggrvm:L_NFS_VOL0183:file i/o:WAFL_WRITE: 50
aggrvm:L_NFS_VOL0179:file i/o:WAFL_WRITE: 42
aggrvm:L_NFS_VOL0180:file i/o:WAFL_WRITE: 31
Application NVLog_b2b in KB/s
----------- ----------------------