Hello
I attach a wafltop output run for 10 minutes. We are experiencing a fair amount of CPU contention and latency. Does anyone have any input on the below?
I also show a wafl scan status which shows there is a lot of activity. Is that normal?
wafltop stop
CPU Utilization Percent Application Total STRIPE VOL_LOG VOL_VBN VBN VOL AGGR_VBN AGGR SERIAL XCleaner ----------- -------- -------- -------- -------- -------- -------- -------- -------- -------- ------- aggrsata2:SV_L_LUN_VOL0199:other:_WAFL_BACKDOOR: 15 0 0 0 0 0 0 0 15 0 other:other:other:FREE_CACHE_REFILL: 8 5 0 3 0 0 0 0 0 0 aggrsata2:SV_L_LUN_VOL0199:scanner:_WAFL_BACKDOOR: 5 0 0 0 0 0 0 0 5 0 other:other:other:RAIDIO_READ_DONE: 5 4 0 1 0 0 0 0 0 0 aggrvm:L_NFS_VOL0178:nfsv3:WAFL_WRITE: 4 4 0 0 0 0 0 0 0 0 other:other:other:WAFL_WRITE_DONE: 4 4 0 0 0 0 0 0 0 0 aggrvm:L_NFS_VOL0176:nfsv3:WAFL_READ: 3 3 0 0 0 0 0 0 0 0 aggrvm:L_NFS_VOL0183:nfsv3:WAFL_WRITE: 3 3 0 0 0 0 0 0 0 0 aggrsata1:L_LUN_VOL0198:iscsi:WAFL_READ: 3 3 0 0 0 0 0 0 0 0 aggrvm:L_NFS_VOL0176:nfsv3:WAFL_WRITE: 3 3 0 0 0 0 0 0 0 0 aggrvm:L_NFS_VOL0177:nfsv3:WAFL_WRITE: 3 3 0 0 0 0 0 0 0 0 aggrvm:L_NFS_VOL0179:nfsv3:WAFL_READ: 2 2 0 0 0 0 0 0 0 0 aggrsata1:L_LUN_VOL0198:iscsi:WAFL_WRITE: 2 2 0 0 0 0 0 0 0 0 aggrsata2:SV_L_LUN_VOL0198:scanner:_WAFL_BACKDOOR: 2 0 0 0 0 0 0 0 2 0 aggrsata2:L_LUN_VOL0199:iscsi:WAFL_READ: 2 2 0 0 0 0 0 0 0 0 aggrvm:L_NFS_VOL0179:nfsv3:WAFL_WRITE: 2 2 0 0 0 0 0 0 0 0 aggrvm:L_NFS_VOL0161:nfsv3:WAFL_WRITE: 1 1 0 0 0 0 0 0 0 0 aggrvm:L_NFS_VOL0174:nfsv3:WAFL_WRITE: 1 1 0 0 0 0 0 0 0 0 aggrvm::walloc:WAFL_BLKFREE_PVOL_CO: 1 0 0 0 0 0 1 0 0 0 other:other:other:WAFFINITY_NVLOG_COMP: 1 1 0 0 0 0 0 0 0 0
CPU Time us Application Total STRIPE VOL_LOG VOL_VBN VBN VOL AGGR_VBN AGGR SERIAL XCleaner ----------- -------- -------- -------- -------- -------- -------- -------- -------- -------- ------- aggrsata2:L_LUN_VOL0199:file i/o:WAFL_LOOKUP: 46164 0 0 0 0 0 0 0 46164 0 aggr0:vol0:file i/o:WAFL_LOOKUP: 33469 0 0 0 0 0 0 0 33469 0 aggr0:vol0:file i/o:WAFL_READDIR: 31339 0 0 0 0 0 0 0 31339 0 aggrsata1::file i/o:WAFL_LOOKUP: 4725 0 0 0 0 0 0 0 4725 0 aggrvm:L_NFS_VOL0179:nfsv3:WAFL_READ: 4479 4349 130 0 0 0 0 0 0 0 aggrsata2:SV_L_LUN_VOL0198:file i/o:WAFL_GETATTR: 3639 3639 0 0 0 0 0 0 0 0 aggrsata2:SV_L_LUN_VOL0198:file i/o:WAFL_READ: 2397 1635 0 0 0 0 0 0 762 0 aggrsata1:L_LUN_VOL0201:iscsi:WAFL_READ: 2252 2191 55 0 0 0 0 0 6 0 aggrvm:L_NFS_VOL0183:nfsv3:WAFL_READDIR: 1650 1628 22 0 0 0 0 0 0 0 aggrvm:L_NFS_VOL0174:nfsv3:WAFL_READ: 1542 1514 28 0 0 0 0 0 0 0 aggrvm:P_NFS_WPF0160:nfsv3:WAFL_DOACCESS: 1525 1525 0 0 0 0 0 0 0 0 aggrvm:L_NFS_VOL0161:nfsv3:WAFL_READ: 1520 1497 23 0 0 0 0 0 0 0 aggrsata2:L_LUN_VOL0199:iscsi:WAFL_READ: 1480 1441 39 0 0 0 0 0 0 0 aggrvm:L_NFS_VOL0158:nfsv3:WAFL_READ: 1378 1355 23 0 0 0 0 0 0 0 aggr0:vol0:spinvfs:WAFL_READ: 1296 1289 7 0 0 0 0 0 0 0 aggrsata2:L_LUN_VOL0199:file i/o:WAFL_READ: 1283 900 0 0 0 357 0 0 26 0 aggrvm:L_NFS_VOL0181:nfsv3:WAFL_READ: 1270 1263 7 0 0 0 0 0 0 0 aggrvm:L_LUN_VOL0152:iscsi:WAFL_READ: 1234 1213 21 0 0 0 0 0 0 0 aggr0:vol0:file i/o:WAFL_READ: 1198 1189 1 0 0 0 0 0 8 0 aggrsata2:SV_L_LUN_VOL0199:file i/o:WAFL_READ: 1195 791 0 0 0 0 0 0 404 0
Latency Application Latency ms ----------- ---------- aggrvm:L_NFS_VOL0180:nfsv3:WAFL_SETATTR: 936.000 aggrvm:L_NFS_VOL0179:nfsv3:WAFL_READ: 13.501 aggrvm:L_NFS_VOL0183:nfsv3:WAFL_READ: 10.962 aggrvm:P_NFS_WPF0160:nfsv3:WAFL_READ: 9.148 aggrvm:L_NFS_VOL0180:nfsv3:WAFL_READ: 8.713 aggrvm:L_NFS_VOL0181:nfsv3:WAFL_READ: 8.205 aggrsata1:L_LUN_VOL0198:iscsi:WAFL_READ: 8.082 aggrvm:L_NFS_VOL0161:nfsv3:WAFL_READ: 7.525 aggrvm:L_NFS_VOL0179:nfsv3:WAFL_READDIR: 6.000 aggrvm:L_NFS_VOL0177:nfsv3:WAFL_READ: 5.825 aggrvm:L_NFS_VOL0180:nfsv3:WAFL_LOOKUP: 5.000 aggrvm:L_NFS_VOL0183:nfsv3:WAFL_READDIR: 5.000 aggrvm:L_NFS_VOL0158:nfsv3:WAFL_READ: 4.926 aggrvm:L_NFS_VOL0178:nfsv3:WAFL_READ: 4.775 aggrvm:L_NFS_VOL0174:nfsv3:WAFL_GETATTR: 4.666 aggr0:vol0:spinvfs:WAFL_READ: 3.250 aggrsata1:L_LUN_VOL0201:iscsi:WAFL_READ: 3.111 aggrvm:P_NFS_WPF0160:nfsv3:WAFL_STATFS: 3.000 aggrsata1:L_LUN_VOL0200:iscsi:WAFL_WRITE: 2.435 aggrsata2:L_LUN_VOL0199:iscsi:WAFL_READ: 2.289
Application System Latency ms ----------- ----------------- aggrsata2:SV_L_LUN_VOL0198:file i/o:WAFL_WRITE: 21.786 aggrsata2:SV_L_LUN_VOL0198:file i/o:WAFL_READ: 21.000 aggrsata2:SV_L_LUN_VOL0198:file i/o:WAFL_GETATTR: 8.000 aggrsata1:L_LUN_VOL0198:file i/o:WAFL_READ: 4.454 aggrvm:L_NFS_VOL0158:file i/o:WAFL_WRITE: 4.000 aggrsata2:L_LUN_VOL0199:file i/o:WAFL_READ: 3.333 aggrvm:L_NFS_VOL0180:file i/o:WAFL_WRITE: 3.261 aggrsata1:L_LUN_VOL0198:file i/o:WAFL_VDISK_SETTYPE: 3.000 aggrvm:L_NFS_VOL0224:file i/o:WAFL_WRITE: 3.000 aggrvm:L_NFS_VOL0177:file i/o:WAFL_WRITE: 2.245 aggrvm:L_NFS_VOL0183:file i/o:WAFL_WRITE: 2.220 aggrvm:L_NFS_VOL0176:file i/o:WAFL_WRITE: 2.201 aggrvm:L_NFS_VOL0179:file i/o:WAFL_WRITE: 2.136 aggrvm:L_NFS_VOL0181:file i/o:WAFL_WRITE: 2.050 aggrsata2:SV_L_LUN_VOL0199:file i/o:WAFL_GETATTR: 1.750 aggrsata2:L_LUN_VOL0199:file i/o:WAFL_LOOKUP: 1.500 aggrsata2:L_LUN_VOL0199:file i/o:WAFL_VDISK_SETTYPE: 1.444 aggrsata1:L_LUN_VOL0198:file i/o:WAFL_STATFS: 1.000 aggrsata1::file i/o:WAFL_LOOKUP: 1.000 aggrvm:L_NFS_VOL0178:file i/o:WAFL_STATFS: 1.000
I/O utilization ---------MB Read---------- ---------MB Write--------- --------IOs Read---------- --------IOs Write--------- Application MB Total Standard PAM Hybrid Standard PAM Hybrid Standard PAM Hybrid Standard PAM Hybrid ----------- -------- -------- -------- -------- -------- -------- -------- -------- -------- -------- -------- -------- -------- aggrsata1:L_LUN_VOL0198:iscsi:WAFL_READ: 17124 15503 1621 0 0 0 0 412113 88557 0 0 0 0 other:other:other:FREE_CACHE_REFILL: 13732 0 0 0 0 13732 0 0 0 0 0 54917 0 aggrvm:L_NFS_VOL0179:nfsv3:WAFL_READ: 10348 9956 392 0 0 0 0 483841 58184 0 0 0 0 aggrvm:L_NFS_VOL0176:nfsv3:WAFL_READ: 7954 6989 965 0 0 0 0 407363 124931 0 0 0 0 aggrsata1:L_LUN_VOL0198:snapmirror:WAFL_SNAP_DIFF: 7523 6052 1471 0 0 0 0 249428 54191 0 0 0 0 other:other:other:WAFFINITY_RA_STRIPE_: 7432 7339 93 0 0 0 0 159291 4544 0 0 0 0 aggrvm:L_NFS_VOL0176:nfsv3:WAFL_WRITE: 7288 384 152 0 6752 0 0 98462 38971 0 0 0 0 aggrsata2:SV_L_LUN_VOL0198:file i/o:WAFL_WRITE: 6759 148 40 0 6571 0 0 37781 10251 0 0 0 0 aggrsata1:L_LUN_VOL0198:iscsi:WAFL_WRITE: 6726 65 112 0 6549 0 0 16507 28706 0 0 0 0 aggrvm:L_NFS_VOL0177:nfsv3:WAFL_WRITE: 5538 426 102 0 5010 0 0 109134 26076 0 0 0 0 aggrvm:L_NFS_VOL0178:nfsv3:WAFL_WRITE: 5444 442 171 0 4831 0 0 113018 43797 0 0 0 0 aggrvm:L_NFS_VOL0183:nfsv3:WAFL_WRITE: 5298 419 125 0 4754 0 0 107294 32047 0 0 0 0 aggrvm:L_NFS_VOL0179:nfsv3:WAFL_WRITE: 4153 381 92 0 3680 0 0 97639 23545 0 0 0 0 aggrvm:L_NFS_VOL0174:nfsv3:WAFL_WRITE: 3748 84 32 0 3632 0 0 21436 8233 0 0 0 0 aggrsata2:L_LUN_VOL0199:iscsi:WAFL_READ: 3692 3482 210 0 0 0 0 90262 10488 0 0 0 0 aggrvm:L_NFS_VOL0180:nfsv3:WAFL_WRITE: 3367 205 85 0 3077 0 0 52391 21642 0 0 0 0 other:other:other:WAFL_WRITE_DONE: 3325 0 0 0 0 3325 0 0 0 0 0 13309 0 aggrsata2:L_LUN_VOL0199:iscsi:WAFL_WRITE: 2738 25 15 0 2698 0 0 6364 3849 0 0 0 0 aggrsata1:L_LUN_VOL0198:snapmirror:WAFL_SNAP_READ: 2340 1838 502 0 0 0 0 356929 96469 0 0 0 0 aggrsata2:SV_L_LUN_VOL0199:other:_WAFL_BACKDOOR: 2265 12 2241 0 12 0 0 3029 536488 0 0 0 0
NVLog Utilization Application NVLog in KB/s ----------- ------------------ aggrsata2:SV_L_LUN_VOL0198:file i/o:WAFL_WRITE: 10036 aggrsata1:L_LUN_VOL0198:iscsi:WAFL_WRITE: 9692 aggrvm:L_NFS_VOL0176:nfsv3:WAFL_WRITE: 9452 aggrvm:L_NFS_VOL0177:nfsv3:WAFL_WRITE: 7462 aggrvm:L_NFS_VOL0183:nfsv3:WAFL_WRITE: 7159 aggrvm:L_NFS_VOL0178:nfsv3:WAFL_WRITE: 7140 aggrsata2:L_LUN_VOL0199:iscsi:WAFL_WRITE: 7083 aggrvm:L_NFS_VOL0174:nfsv3:WAFL_WRITE: 5422 aggrvm:L_NFS_VOL0179:nfsv3:WAFL_WRITE: 5358 aggrvm:L_NFS_VOL0180:nfsv3:WAFL_WRITE: 4222 aggrvm:L_NFS_VOL0182:nfsv3:WAFL_WRITE: 2698 aggrvm:L_NFS_VOL0161:nfsv3:WAFL_WRITE: 1944 aggrvm:L_NFS_VOL0181:nfsv3:WAFL_WRITE: 1647 aggrvm:P_NFS_WPF0160:nfsv3:WAFL_WRITE: 842 aggrvm:L_NFS_VOL0158:nfsv3:WAFL_WRITE: 351 aggrvm:L_NFS_VOL0176:file i/o:WAFL_WRITE: 70 aggrvm:L_NFS_VOL0177:file i/o:WAFL_WRITE: 53 aggrvm:L_NFS_VOL0183:file i/o:WAFL_WRITE: 50 aggrvm:L_NFS_VOL0179:file i/o:WAFL_WRITE: 42 aggrvm:L_NFS_VOL0180:file i/o:WAFL_WRITE: 31
Application NVLog_b2b in KB/s ----------- ----------------------
wafl scan status Volume vol0: Scan id Type of scan progress 11 active bitmap rearrangement fbn 3263 of 3971 w/ max_chain_len 3 Volume L_NFS_VOL0224: Scan id Type of scan progress 12 active bitmap rearrangement fbn 5565 of 6853 w/ max_chain_len 3 Volume L_LUN_VOL0152: Scan id Type of scan progress 13 active bitmap rearrangement fbn 1283 of 5577 w/ max_chain_len 3 35 container block reclamation block 147 of 5578 (fbn 147) Volume L_NFS_VOL0158: Scan id Type of scan progress 14 active bitmap rearrangement fbn 43778 of 45689 w/ max_chain_len 3 Volume P_NFS_VSW0159: Scan id Type of scan progress 15 active bitmap rearrangement fbn 23963 of 36551 w/ max_chain_len 3 Volume P_NFS_WPF0160: Scan id Type of scan progress 16 active bitmap rearrangement fbn 5242 of 36551 w/ max_chain_len 3 Volume L_NFS_VOL0161: Scan id Type of scan progress 17 active bitmap rearrangement fbn 44716 of 45689 w/ max_chain_len 3 3431 snap create summary update block 7992 of 45690 (fbn 29116) 3434 blocks used summary update block 4284 of 45690 (fbn 14381) Volume L_NFS_VOL0172: Scan id Type of scan progress 18 active bitmap rearrangement fbn 16466 of 27413 w/ max_chain_len 3 Volume L_NFS_VOL0174: Scan id Type of scan progress 19 active bitmap rearrangement fbn 22931 of 25129 w/ max_chain_len 3 Volume L_NFS_VOL0176: Scan id Type of scan progress 20 active bitmap rearrangement fbn 29215 of 54827 w/ max_chain_len 3 Volume L_NFS_VOL0177: Scan id Type of scan progress 21 active bitmap rearrangement fbn 32845 of 54827 w/ max_chain_len 3 Volume L_NFS_VOL0178: Scan id Type of scan progress 22 active bitmap rearrangement fbn 28871 of 54827 w/ max_chain_len 3 Volume L_NFS_VOL0179: Scan id Type of scan progress 23 active bitmap rearrangement fbn 12076 of 36551 w/ max_chain_len 3 Volume L_NFS_VOL0180: Scan id Type of scan progress 24 active bitmap rearrangement fbn 13106 of 36551 w/ max_chain_len 3 Volume L_NFS_VOL0181: Scan id Type of scan progress 25 active bitmap rearrangement fbn 12588 of 36551 w/ max_chain_len 3 Volume L_NFS_VOL0182: Scan id Type of scan progress 26 active bitmap rearrangement fbn 2419 of 4461 w/ max_chain_len 3 Volume L_NFS_VOL0183: Scan id Type of scan progress 27 active bitmap rearrangement fbn 63137 of 63965 w/ max_chain_len 3 Volume SV_L_NFS_VOL0178: Scan id Type of scan progress 28 active bitmap rearrangement fbn 39988 of 54827 w/ max_chain_len 3 Volume L_LUN_VOL0198: Scan id Type of scan progress 32 active bitmap rearrangement fbn 60526 of 68534 w/ max_chain_len 3 3443 snap create summary update block 4897 of 68535 (fbn 4437) Volume L_LUN_VOL0200: Scan id Type of scan progress 34 active bitmap rearrangement fbn 3607 of 54827 w/ max_chain_len 3 Volume L_LUN_VOL0201: Scan id Type of scan progress 52 active bitmap rearrangement fbn 12935 of 54827 w/ max_chain_len 3 Volume SV_L_NFS_VOL0161: Scan id Type of scan progress 53 active bitmap rearrangement fbn 44350 of 63965 w/ max_chain_len 3 Volume SV_L_NFS_VOL0177: Scan id Type of scan progress 56 active bitmap rearrangement fbn 7859 of 61680 w/ max_chain_len 3 Volume SV_L_NFS_VOL0181: Scan id Type of scan progress 57 active bitmap rearrangement fbn 1550 of 45689 w/ max_chain_len 3 Volume SV_L_NFS_VOL0202: Scan id Type of scan progress 60 active bitmap rearrangement fbn 13923 of 31982 w/ max_chain_len 3 Volume SV_L_NFS_VOL0180: Scan id Type of scan progress 62 active bitmap rearrangement fbn 35314 of 50258 w/ max_chain_len 3 Volume SV_L_NFS_VOL0172: Scan id Type of scan progress 63 active bitmap rearrangement fbn 763 of 2230 w/ max_chain_len 3 Volume L_NFS_VOL0202: Scan id Type of scan progress 65 active bitmap rearrangement fbn 9076 of 20560 w/ max_chain_len 3 Volume SV_L_NFS_VOL0158: Scan id Type of scan progress 66 active bitmap rearrangement fbn 33221 of 73103 w/ max_chain_len 3 Volume SV_L_LUN_VOL0198: Scan id Type of scan progress 68 active bitmap rearrangement fbn 200707 of 319142 w/ max_chain_len 3 3421 blocks used summary update block 120530 of 319143 (fbn 226694) 3422 snap create summary update block 126523 of 319143 (fbn 266270) Volume SV_L_LUN_VOL0199: Scan id Type of scan progress 70 active bitmap rearrangement fbn 244335 of 319142 w/ max_chain_len 3 3408 snap create summary update block 125303 of 319143 (fbn 220403) 3423 blocks used summary update block 120064 of 319143 (fbn 220612) Volume L_LUN_VOL0199: Scan id Type of scan progress 3325 active bitmap rearrangement fbn 32437 of 54827 w/ max_chain_len 3 3444 snap create summary update block 283 of 54828 (fbn 4437)