Hello
I attach a wafltop output run for 10 minutes. We are experiencing a fair amount of CPU contention and latency. Does anyone have any input on the below?
I also show a wafl scan status which shows there is a lot of activity. Is that normal?
wafltop stop
CPU Utilization Percent
Application Total STRIPE VOL_LOG VOL_VBN VBN VOL AGGR_VBN AGGR SERIAL XCleaner
----------- -------- -------- -------- -------- -------- -------- -------- -------- -------- -------
aggrsata2:SV_L_LUN_VOL0199:other:_WAFL_BACKDOOR: 15 0 0 0 0 0 0 0 15 0
other:other:other:FREE_CACHE_REFILL: 8 5 0 3 0 0 0 0 0 0
aggrsata2:SV_L_LUN_VOL0199:scanner:_WAFL_BACKDOOR: 5 0 0 0 0 0 0 0 5 0
other:other:other:RAIDIO_READ_DONE: 5 4 0 1 0 0 0 0 0 0
aggrvm:L_NFS_VOL0178:nfsv3:WAFL_WRITE: 4 4 0 0 0 0 0 0 0 0
other:other:other:WAFL_WRITE_DONE: 4 4 0 0 0 0 0 0 0 0
aggrvm:L_NFS_VOL0176:nfsv3:WAFL_READ: 3 3 0 0 0 0 0 0 0 0
aggrvm:L_NFS_VOL0183:nfsv3:WAFL_WRITE: 3 3 0 0 0 0 0 0 0 0
aggrsata1:L_LUN_VOL0198:iscsi:WAFL_READ: 3 3 0 0 0 0 0 0 0 0
aggrvm:L_NFS_VOL0176:nfsv3:WAFL_WRITE: 3 3 0 0 0 0 0 0 0 0
aggrvm:L_NFS_VOL0177:nfsv3:WAFL_WRITE: 3 3 0 0 0 0 0 0 0 0
aggrvm:L_NFS_VOL0179:nfsv3:WAFL_READ: 2 2 0 0 0 0 0 0 0 0
aggrsata1:L_LUN_VOL0198:iscsi:WAFL_WRITE: 2 2 0 0 0 0 0 0 0 0
aggrsata2:SV_L_LUN_VOL0198:scanner:_WAFL_BACKDOOR: 2 0 0 0 0 0 0 0 2 0
aggrsata2:L_LUN_VOL0199:iscsi:WAFL_READ: 2 2 0 0 0 0 0 0 0 0
aggrvm:L_NFS_VOL0179:nfsv3:WAFL_WRITE: 2 2 0 0 0 0 0 0 0 0
aggrvm:L_NFS_VOL0161:nfsv3:WAFL_WRITE: 1 1 0 0 0 0 0 0 0 0
aggrvm:L_NFS_VOL0174:nfsv3:WAFL_WRITE: 1 1 0 0 0 0 0 0 0 0
aggrvm::walloc:WAFL_BLKFREE_PVOL_CO: 1 0 0 0 0 0 1 0 0 0
other:other:other:WAFFINITY_NVLOG_COMP: 1 1 0 0 0 0 0 0 0 0
CPU Time us
Application Total STRIPE VOL_LOG VOL_VBN VBN VOL AGGR_VBN AGGR SERIAL XCleaner
----------- -------- -------- -------- -------- -------- -------- -------- -------- -------- -------
aggrsata2:L_LUN_VOL0199:file i/o:WAFL_LOOKUP: 46164 0 0 0 0 0 0 0 46164 0
aggr0:vol0:file i/o:WAFL_LOOKUP: 33469 0 0 0 0 0 0 0 33469 0
aggr0:vol0:file i/o:WAFL_READDIR: 31339 0 0 0 0 0 0 0 31339 0
aggrsata1::file i/o:WAFL_LOOKUP: 4725 0 0 0 0 0 0 0 4725 0
aggrvm:L_NFS_VOL0179:nfsv3:WAFL_READ: 4479 4349 130 0 0 0 0 0 0 0
aggrsata2:SV_L_LUN_VOL0198:file i/o:WAFL_GETATTR: 3639 3639 0 0 0 0 0 0 0 0
aggrsata2:SV_L_LUN_VOL0198:file i/o:WAFL_READ: 2397 1635 0 0 0 0 0 0 762 0
aggrsata1:L_LUN_VOL0201:iscsi:WAFL_READ: 2252 2191 55 0 0 0 0 0 6 0
aggrvm:L_NFS_VOL0183:nfsv3:WAFL_READDIR: 1650 1628 22 0 0 0 0 0 0 0
aggrvm:L_NFS_VOL0174:nfsv3:WAFL_READ: 1542 1514 28 0 0 0 0 0 0 0
aggrvm:P_NFS_WPF0160:nfsv3:WAFL_DOACCESS: 1525 1525 0 0 0 0 0 0 0 0
aggrvm:L_NFS_VOL0161:nfsv3:WAFL_READ: 1520 1497 23 0 0 0 0 0 0 0
aggrsata2:L_LUN_VOL0199:iscsi:WAFL_READ: 1480 1441 39 0 0 0 0 0 0 0
aggrvm:L_NFS_VOL0158:nfsv3:WAFL_READ: 1378 1355 23 0 0 0 0 0 0 0
aggr0:vol0:spinvfs:WAFL_READ: 1296 1289 7 0 0 0 0 0 0 0
aggrsata2:L_LUN_VOL0199:file i/o:WAFL_READ: 1283 900 0 0 0 357 0 0 26 0
aggrvm:L_NFS_VOL0181:nfsv3:WAFL_READ: 1270 1263 7 0 0 0 0 0 0 0
aggrvm:L_LUN_VOL0152:iscsi:WAFL_READ: 1234 1213 21 0 0 0 0 0 0 0
aggr0:vol0:file i/o:WAFL_READ: 1198 1189 1 0 0 0 0 0 8 0
aggrsata2:SV_L_LUN_VOL0199:file i/o:WAFL_READ: 1195 791 0 0 0 0 0 0 404 0
Latency
Application Latency ms
----------- ----------
aggrvm:L_NFS_VOL0180:nfsv3:WAFL_SETATTR: 936.000
aggrvm:L_NFS_VOL0179:nfsv3:WAFL_READ: 13.501
aggrvm:L_NFS_VOL0183:nfsv3:WAFL_READ: 10.962
aggrvm:P_NFS_WPF0160:nfsv3:WAFL_READ: 9.148
aggrvm:L_NFS_VOL0180:nfsv3:WAFL_READ: 8.713
aggrvm:L_NFS_VOL0181:nfsv3:WAFL_READ: 8.205
aggrsata1:L_LUN_VOL0198:iscsi:WAFL_READ: 8.082
aggrvm:L_NFS_VOL0161:nfsv3:WAFL_READ: 7.525
aggrvm:L_NFS_VOL0179:nfsv3:WAFL_READDIR: 6.000
aggrvm:L_NFS_VOL0177:nfsv3:WAFL_READ: 5.825
aggrvm:L_NFS_VOL0180:nfsv3:WAFL_LOOKUP: 5.000
aggrvm:L_NFS_VOL0183:nfsv3:WAFL_READDIR: 5.000
aggrvm:L_NFS_VOL0158:nfsv3:WAFL_READ: 4.926
aggrvm:L_NFS_VOL0178:nfsv3:WAFL_READ: 4.775
aggrvm:L_NFS_VOL0174:nfsv3:WAFL_GETATTR: 4.666
aggr0:vol0:spinvfs:WAFL_READ: 3.250
aggrsata1:L_LUN_VOL0201:iscsi:WAFL_READ: 3.111
aggrvm:P_NFS_WPF0160:nfsv3:WAFL_STATFS: 3.000
aggrsata1:L_LUN_VOL0200:iscsi:WAFL_WRITE: 2.435
aggrsata2:L_LUN_VOL0199:iscsi:WAFL_READ: 2.289
Application System Latency ms
----------- -----------------
aggrsata2:SV_L_LUN_VOL0198:file i/o:WAFL_WRITE: 21.786
aggrsata2:SV_L_LUN_VOL0198:file i/o:WAFL_READ: 21.000
aggrsata2:SV_L_LUN_VOL0198:file i/o:WAFL_GETATTR: 8.000
aggrsata1:L_LUN_VOL0198:file i/o:WAFL_READ: 4.454
aggrvm:L_NFS_VOL0158:file i/o:WAFL_WRITE: 4.000
aggrsata2:L_LUN_VOL0199:file i/o:WAFL_READ: 3.333
aggrvm:L_NFS_VOL0180:file i/o:WAFL_WRITE: 3.261
aggrsata1:L_LUN_VOL0198:file i/o:WAFL_VDISK_SETTYPE: 3.000
aggrvm:L_NFS_VOL0224:file i/o:WAFL_WRITE: 3.000
aggrvm:L_NFS_VOL0177:file i/o:WAFL_WRITE: 2.245
aggrvm:L_NFS_VOL0183:file i/o:WAFL_WRITE: 2.220
aggrvm:L_NFS_VOL0176:file i/o:WAFL_WRITE: 2.201
aggrvm:L_NFS_VOL0179:file i/o:WAFL_WRITE: 2.136
aggrvm:L_NFS_VOL0181:file i/o:WAFL_WRITE: 2.050
aggrsata2:SV_L_LUN_VOL0199:file i/o:WAFL_GETATTR: 1.750
aggrsata2:L_LUN_VOL0199:file i/o:WAFL_LOOKUP: 1.500
aggrsata2:L_LUN_VOL0199:file i/o:WAFL_VDISK_SETTYPE: 1.444
aggrsata1:L_LUN_VOL0198:file i/o:WAFL_STATFS: 1.000
aggrsata1::file i/o:WAFL_LOOKUP: 1.000
aggrvm:L_NFS_VOL0178:file i/o:WAFL_STATFS: 1.000
I/O utilization
---------MB Read---------- ---------MB Write--------- --------IOs Read---------- --------IOs Write---------
Application MB Total Standard PAM Hybrid Standard PAM Hybrid Standard PAM Hybrid Standard PAM Hybrid
----------- -------- -------- -------- -------- -------- -------- -------- -------- -------- -------- -------- -------- --------
aggrsata1:L_LUN_VOL0198:iscsi:WAFL_READ: 17124 15503 1621 0 0 0 0 412113 88557 0 0 0 0
other:other:other:FREE_CACHE_REFILL: 13732 0 0 0 0 13732 0 0 0 0 0 54917 0
aggrvm:L_NFS_VOL0179:nfsv3:WAFL_READ: 10348 9956 392 0 0 0 0 483841 58184 0 0 0 0
aggrvm:L_NFS_VOL0176:nfsv3:WAFL_READ: 7954 6989 965 0 0 0 0 407363 124931 0 0 0 0
aggrsata1:L_LUN_VOL0198:snapmirror:WAFL_SNAP_DIFF: 7523 6052 1471 0 0 0 0 249428 54191 0 0 0 0
other:other:other:WAFFINITY_RA_STRIPE_: 7432 7339 93 0 0 0 0 159291 4544 0 0 0 0
aggrvm:L_NFS_VOL0176:nfsv3:WAFL_WRITE: 7288 384 152 0 6752 0 0 98462 38971 0 0 0 0
aggrsata2:SV_L_LUN_VOL0198:file i/o:WAFL_WRITE: 6759 148 40 0 6571 0 0 37781 10251 0 0 0 0
aggrsata1:L_LUN_VOL0198:iscsi:WAFL_WRITE: 6726 65 112 0 6549 0 0 16507 28706 0 0 0 0
aggrvm:L_NFS_VOL0177:nfsv3:WAFL_WRITE: 5538 426 102 0 5010 0 0 109134 26076 0 0 0 0
aggrvm:L_NFS_VOL0178:nfsv3:WAFL_WRITE: 5444 442 171 0 4831 0 0 113018 43797 0 0 0 0
aggrvm:L_NFS_VOL0183:nfsv3:WAFL_WRITE: 5298 419 125 0 4754 0 0 107294 32047 0 0 0 0
aggrvm:L_NFS_VOL0179:nfsv3:WAFL_WRITE: 4153 381 92 0 3680 0 0 97639 23545 0 0 0 0
aggrvm:L_NFS_VOL0174:nfsv3:WAFL_WRITE: 3748 84 32 0 3632 0 0 21436 8233 0 0 0 0
aggrsata2:L_LUN_VOL0199:iscsi:WAFL_READ: 3692 3482 210 0 0 0 0 90262 10488 0 0 0 0
aggrvm:L_NFS_VOL0180:nfsv3:WAFL_WRITE: 3367 205 85 0 3077 0 0 52391 21642 0 0 0 0
other:other:other:WAFL_WRITE_DONE: 3325 0 0 0 0 3325 0 0 0 0 0 13309 0
aggrsata2:L_LUN_VOL0199:iscsi:WAFL_WRITE: 2738 25 15 0 2698 0 0 6364 3849 0 0 0 0
aggrsata1:L_LUN_VOL0198:snapmirror:WAFL_SNAP_READ: 2340 1838 502 0 0 0 0 356929 96469 0 0 0 0
aggrsata2:SV_L_LUN_VOL0199:other:_WAFL_BACKDOOR: 2265 12 2241 0 12 0 0 3029 536488 0 0 0 0
NVLog Utilization
Application NVLog in KB/s
----------- ------------------
aggrsata2:SV_L_LUN_VOL0198:file i/o:WAFL_WRITE: 10036
aggrsata1:L_LUN_VOL0198:iscsi:WAFL_WRITE: 9692
aggrvm:L_NFS_VOL0176:nfsv3:WAFL_WRITE: 9452
aggrvm:L_NFS_VOL0177:nfsv3:WAFL_WRITE: 7462
aggrvm:L_NFS_VOL0183:nfsv3:WAFL_WRITE: 7159
aggrvm:L_NFS_VOL0178:nfsv3:WAFL_WRITE: 7140
aggrsata2:L_LUN_VOL0199:iscsi:WAFL_WRITE: 7083
aggrvm:L_NFS_VOL0174:nfsv3:WAFL_WRITE: 5422
aggrvm:L_NFS_VOL0179:nfsv3:WAFL_WRITE: 5358
aggrvm:L_NFS_VOL0180:nfsv3:WAFL_WRITE: 4222
aggrvm:L_NFS_VOL0182:nfsv3:WAFL_WRITE: 2698
aggrvm:L_NFS_VOL0161:nfsv3:WAFL_WRITE: 1944
aggrvm:L_NFS_VOL0181:nfsv3:WAFL_WRITE: 1647
aggrvm:P_NFS_WPF0160:nfsv3:WAFL_WRITE: 842
aggrvm:L_NFS_VOL0158:nfsv3:WAFL_WRITE: 351
aggrvm:L_NFS_VOL0176:file i/o:WAFL_WRITE: 70
aggrvm:L_NFS_VOL0177:file i/o:WAFL_WRITE: 53
aggrvm:L_NFS_VOL0183:file i/o:WAFL_WRITE: 50
aggrvm:L_NFS_VOL0179:file i/o:WAFL_WRITE: 42
aggrvm:L_NFS_VOL0180:file i/o:WAFL_WRITE: 31
Application NVLog_b2b in KB/s
----------- ----------------------
wafl scan status
Volume vol0:
Scan id Type of scan progress
11 active bitmap rearrangement fbn 3263 of 3971 w/ max_chain_len 3
Volume L_NFS_VOL0224:
Scan id Type of scan progress
12 active bitmap rearrangement fbn 5565 of 6853 w/ max_chain_len 3
Volume L_LUN_VOL0152:
Scan id Type of scan progress
13 active bitmap rearrangement fbn 1283 of 5577 w/ max_chain_len 3
35 container block reclamation block 147 of 5578 (fbn 147)
Volume L_NFS_VOL0158:
Scan id Type of scan progress
14 active bitmap rearrangement fbn 43778 of 45689 w/ max_chain_len 3
Volume P_NFS_VSW0159:
Scan id Type of scan progress
15 active bitmap rearrangement fbn 23963 of 36551 w/ max_chain_len 3
Volume P_NFS_WPF0160:
Scan id Type of scan progress
16 active bitmap rearrangement fbn 5242 of 36551 w/ max_chain_len 3
Volume L_NFS_VOL0161:
Scan id Type of scan progress
17 active bitmap rearrangement fbn 44716 of 45689 w/ max_chain_len 3
3431 snap create summary update block 7992 of 45690 (fbn 29116)
3434 blocks used summary update block 4284 of 45690 (fbn 14381)
Volume L_NFS_VOL0172:
Scan id Type of scan progress
18 active bitmap rearrangement fbn 16466 of 27413 w/ max_chain_len 3
Volume L_NFS_VOL0174:
Scan id Type of scan progress
19 active bitmap rearrangement fbn 22931 of 25129 w/ max_chain_len 3
Volume L_NFS_VOL0176:
Scan id Type of scan progress
20 active bitmap rearrangement fbn 29215 of 54827 w/ max_chain_len 3
Volume L_NFS_VOL0177:
Scan id Type of scan progress
21 active bitmap rearrangement fbn 32845 of 54827 w/ max_chain_len 3
Volume L_NFS_VOL0178:
Scan id Type of scan progress
22 active bitmap rearrangement fbn 28871 of 54827 w/ max_chain_len 3
Volume L_NFS_VOL0179:
Scan id Type of scan progress
23 active bitmap rearrangement fbn 12076 of 36551 w/ max_chain_len 3
Volume L_NFS_VOL0180:
Scan id Type of scan progress
24 active bitmap rearrangement fbn 13106 of 36551 w/ max_chain_len 3
Volume L_NFS_VOL0181:
Scan id Type of scan progress
25 active bitmap rearrangement fbn 12588 of 36551 w/ max_chain_len 3
Volume L_NFS_VOL0182:
Scan id Type of scan progress
26 active bitmap rearrangement fbn 2419 of 4461 w/ max_chain_len 3
Volume L_NFS_VOL0183:
Scan id Type of scan progress
27 active bitmap rearrangement fbn 63137 of 63965 w/ max_chain_len 3
Volume SV_L_NFS_VOL0178:
Scan id Type of scan progress
28 active bitmap rearrangement fbn 39988 of 54827 w/ max_chain_len 3
Volume L_LUN_VOL0198:
Scan id Type of scan progress
32 active bitmap rearrangement fbn 60526 of 68534 w/ max_chain_len 3
3443 snap create summary update block 4897 of 68535 (fbn 4437)
Volume L_LUN_VOL0200:
Scan id Type of scan progress
34 active bitmap rearrangement fbn 3607 of 54827 w/ max_chain_len 3
Volume L_LUN_VOL0201:
Scan id Type of scan progress
52 active bitmap rearrangement fbn 12935 of 54827 w/ max_chain_len 3
Volume SV_L_NFS_VOL0161:
Scan id Type of scan progress
53 active bitmap rearrangement fbn 44350 of 63965 w/ max_chain_len 3
Volume SV_L_NFS_VOL0177:
Scan id Type of scan progress
56 active bitmap rearrangement fbn 7859 of 61680 w/ max_chain_len 3
Volume SV_L_NFS_VOL0181:
Scan id Type of scan progress
57 active bitmap rearrangement fbn 1550 of 45689 w/ max_chain_len 3
Volume SV_L_NFS_VOL0202:
Scan id Type of scan progress
60 active bitmap rearrangement fbn 13923 of 31982 w/ max_chain_len 3
Volume SV_L_NFS_VOL0180:
Scan id Type of scan progress
62 active bitmap rearrangement fbn 35314 of 50258 w/ max_chain_len 3
Volume SV_L_NFS_VOL0172:
Scan id Type of scan progress
63 active bitmap rearrangement fbn 763 of 2230 w/ max_chain_len 3
Volume L_NFS_VOL0202:
Scan id Type of scan progress
65 active bitmap rearrangement fbn 9076 of 20560 w/ max_chain_len 3
Volume SV_L_NFS_VOL0158:
Scan id Type of scan progress
66 active bitmap rearrangement fbn 33221 of 73103 w/ max_chain_len 3
Volume SV_L_LUN_VOL0198:
Scan id Type of scan progress
68 active bitmap rearrangement fbn 200707 of 319142 w/ max_chain_len 3
3421 blocks used summary update block 120530 of 319143 (fbn 226694)
3422 snap create summary update block 126523 of 319143 (fbn 266270)
Volume SV_L_LUN_VOL0199:
Scan id Type of scan progress
70 active bitmap rearrangement fbn 244335 of 319142 w/ max_chain_len 3
3408 snap create summary update block 125303 of 319143 (fbn 220403)
3423 blocks used summary update block 120064 of 319143 (fbn 220612)
Volume L_LUN_VOL0199:
Scan id Type of scan progress
3325 active bitmap rearrangement fbn 32437 of 54827 w/ max_chain_len 3
3444 snap create summary update block 283 of 54828 (fbn 4437)