Hello,
We have a series of old IBM N3220 (NetApp 2240-2) running Data ONTAP Release 8.1.2P1 7-Mode. We are actively migrating off these systems and plan to be off of them by early next month.
One of the systems is for DR. It's a snapmirror target as well as a snapvault source. We snapmirror vols from a prod system to this DR system, and then snapvault the vols to another system for a 2nd copy at the DR site. So all this DR system does (normally) is receive snapmirrors and is the source for snapvaults.
PROD pair --sm--> DR pair --sv--> snapvault target system
Over the past days the snapvaults have been running REAL slow. A snapvault that used to run in a few minutes is now running up to an hour or more.
When NO snapmirror (target) or snapvault (source) is running, there is constant disk READ activity that is hitting the disks at 100% util. The sysstats below are taken when no snapmirror or snapvault activity is happening. The system is idle as far as any work goes.
sdnapr0666a> sysstat -u -i 2 CPU Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk ops/s in out read write read write age hit time ty util 14% 0 1 1 9886 726 0 0 1 96% 61% T 100% 14% 0 2 2 9792 12 0 0 1 96% 100% : 100% 43% 0 2 33 12144 106 0 0 1 97% 100% : 100% 34% 0 5 75 10560 142 0 0 1 96% 100% : 100% 15% 0 0 0 10434 24 0 0 1 96% 57% : 100% 14% 0 23 558 10200 4 0 0 1 96% 0% - 100% 16% 0 1 0 10618 12 0 0 1 96% 0% - 100% 14% 0 0 0 10548 16 0 0 1 96% 0% - 100%
sdnapr0666a> sysstat -s -i 2 CPU NFS CIFS iSCSI Net kB/s Disk kB/s iSCSI kB/s Cache in out read write in out age 15% 0 0 0 0 0 11372 4 0 0 1 15% 0 0 0 23 558 9912 62 0 0 1 14% 0 0 0 1 0 10692 18 0 0 1 15% 0 0 0 0 0 11240 0 0 0 1 15% 0 0 0 0 0 10624 12 0 0 1 16% 0 0 0 23 558 10384 16 0 0 1 15% 0 0 0 0 0 10902 548 0 0 1 14% 0 0 0 0 0 10484 16 0 0 1
The disks (a shelf of SATA disks) are getting pounded. There are no hardware problem like a failed disk. The system looks healthy.
For curiosity I performed a takeover/giveback, but this activity followed to the partner head and back.
Below is a statit for 60 seconds. Again, with no activity happening.
Any thoughts on what's happening is welcome!
Rick
==> statit for 60 seconds
issuing statit -b cmd Warning: These advanced commands are potentially dangerous; use them only when directed to do so by IBM personnel. sleeping for 60 issuing statit -e cmd
Hostname: sdnapr0666a ID: 1892873400 Memory: 4342 MB Data ONTAP Release 8.1.2P1 7-Mode: Mon Jan 7 15:55:05 PST 2013 <1O> Start time: Thu Mar 8 07:46:41 EST 2018
CPU Statistics 60.364027 time (seconds) 100 % 17.647400 system time 29 % 0.705493 rupt time 1 % (617560 rupts x 1 usec/rupt) 16.941907 non-rupt system time 28 % 223.808704 idle time 371 %
28.281968 time in CP 47 % 100 % 0.330279 rupt time in CP 1 % (289024 rupts x 1 usec/rupt)
Multiprocessor Statistics (per second) cpu0 cpu1 cpu2 cpu3 total sk switches 25405.59 23819.82 20634.33 23665.75 93525.49 hard switches 3671.11 11061.21 11051.23 2377.44 28160.99 domain switches 320.97 117.44 103.52 61.97 603.90 CP rupts 1968.90 936.90 945.31 936.90 4788.02 nonCP rupts 2242.33 1062.79 1074.68 1062.79 5442.58 IPI rupts 0.00 0.00 0.00 0.00 0.00 grab kahuna 0.00 0.00 0.00 0.00 0.00
grab kahuna usec 0.00 0.00 0.00 0.00 0.00
CP rupt usec 3466.57 378.19 399.89 1226.79 5471.45 nonCP rupt usec 3943.76 427.32 456.17 1388.59 6215.84 idle 954903.67 928083.Warning: These advanced commands are potentially dangerous; use them only when directed to do so by IBM personnel. 11 895281.16 929382.40 3707650.35 kahuna 0.00 0.00 0.00 25821.64 25821.64 storage 58.00 40654.71 43.47 0.00 40756.19 exempt 7709.69 3161.45 8699.97 6.79 19577.94 raid 79.98 54.75 45300.54 0.00 45435.29 target 9.26 7.16 8.85 0.00 25.28 dnscache 0.00 0.00 0.00 0.00 0.00 cifs 32.64 41.61 37.31 0.00 111.57 wafl_exempt 23660.32 22689.72 44831.88 42173.68 133355.62 wafl_xcleaner 29.42 17.94 37.75 0.00 85.13 sm_exempt 838.96 673.91 1093.71 0.00 2606.60 cluster 0.00 0.00 0.00 0.00 0.00 protocol 30.80 37.95 27.80 0.00 96.56 nwk_exclusive 218.04 150.37 92.84 0.00 461.27 nwk_exempt 624.54 629.35 587.63 0.08 1841.63 nwk_legacy 1534.08 21.32 14.23 0.00 1569.64 hostOS 2860.18 2970.99 3086.67 0.00 8917.88
12.265273 seconds with one or more CPUs active ( 20%) 3.098531 seconds with 2 or more CPUs active ( 5%) 1.011320 seconds with 3 or more CPUs active ( 2%)
9.166741 seconds with one CPU active ( 15%) 2.087210 seconds with 2 CPUs active ( 3%) 0.929439 seconds with 3 CPUs active ( 2%) 0.081881 seconds with all CPUs active ( 0%)
Domain Utilization of Shared Domains (per second) 0.00 idle 125450.41 kahuna 0.00 storage 0.00 exempt 0.00 raid 0.00 target 0.00 dnscache 0.00 cifs 0.00 wafl_exempt 0.00 wafl_xcleaner 0.00 sm_exempt 0.00 cluster 0.00 protocol 2482.49 nwk_exclusive 0.00 nwk_exempt 0.00 nwk_legacy 0.00 hostOS
Miscellaneous Statistics (per second) 28160.99 hard context switches 0.00 NFS operations 0.00 CIFS operations 0.00 HTTP operations 4.08 network KB received 90.32 network KB transmitted 10664.89 disk KB read 149.10 disk KB written 2.93 NVRAM KB written 0.00 nolog KB written 0.00 WAFL bufs given to clients 0.00 checksum cache hits ( 0%) 0.00 no checksum - partial buffer 0.00 FCP operations 0.00 iSCSI operations
WAFL Statistics (per second) 11.48 name cache hits ( 98%) 0.22 name cache misses ( 2%) 36472.78 buf hash hits ( 83%) 7309.54 buf hash misses ( 17%) 1319.00 inode cache hits ( 87%) 193.03 inode cache misses ( 13%) 11030.66 buf cache hits ( 96%) 507.36 buf cache misses ( 4%) 195.58 blocks read 2452.11 blocks read-ahead 418.10 chains read-ahead 4.85 dummy reads 0.00 blocks speculative read-ahead 17.59 blocks written 0.33 stripes written 0.00 blocks page flipped 0.00 blocks over-written 0.07 wafl_timer generated CP 0.00 snapshot generated CP 0.00 wafl_avail_bufs generated CP 0.00 dirty_blk_cnt generated CP 0.00 full NV-log generated CP 0.00 back-to-back CP 0.00 flush generated CP 0.00 sync generated CP 0.00 deferred back-to-back CP 0.00 container-indirect-pin CP 0.00 low mbufs generated CP 0.00 low datavecs generated CP 3327.41 non-restart messages 53.28 IOWAIT suspends 0.00 next nvlog nearly full msecs 0.00 dirty buffer susp msecs 0.00 nvlog full susp msecs 762408 buffers
RAID Statistics (per second) 9.34 xors 0.00 long dispatches [0] 0.00 long consumed [0] 0.00 long consumed hipri [0] 0.00 long low priority [0] 0.00 long high priority [0] 0.00 long monitor tics [0] 0.00 long monitor clears [0] 0.00 long dispatches [1] 0.00 long consumed [1] 0.00 long consumed hipri [1] 0.00 long low priority [1] 0.00 long high priority [1] 0.00 long monitor tics [1] 0.00 long monitor clears [1] 15 max batch 0.78 blocked mode xor 1.61 timed mode xor 0.40 fast adjustments 0.27 slow adjustments 0 avg batch start 0 avg stripe/msec 2628.35 checksum dispatches 13837.23 checksum consumed 0.56 tetrises written 0.00 master tetrises 0.00 slave tetrises 8.27 stripes written 1.08 partial stripes 7.19 full stripes 17.83 blocks written 3.99 blocks read 6.00 1 blocks per stripe size 1 0.35 1 blocks per stripe size 7 0.13 2 blocks per stripe size 7 0.08 3 blocks per stripe size 7 0.18 4 blocks per stripe size 7 0.10 5 blocks per stripe size 7 0.23 6 blocks per stripe size 7 1.19 7 blocks per stripe size 7
Network Interface Statistics (per second) iface side bytes packets multicasts errors collisions pkt drops e0a recv 4042.26 56.92 0.00 0.00 0.00 xmit 92475.23 17.92 0.25 0.00 0.00 e0b recv 135.28 1.94 0.00 0.00 0.00 xmit 6.96 0.17 0.17 0.00 0.00 e0c recv 0.00 0.00 0.00 0.00 0.00 xmit 0.00 0.00 0.00 0.00 0.00 e0d recv 0.00 0.00 0.00 0.00 0.00 xmit 0.00 0.00 0.00 0.00 0.00 e0M recv 1.06 0.02 0.02 0.00 0.00 xmit 10.14 0.08 0.08 0.00 0.00 e0P recv 0.00 0.00 0.00 0.00 0.00 xmit 0.00 0.00 0.00 0.00 0.00 vh recv 0.00 0.00 0.00 0.00 0.00 xmit 0.00 0.00 0.00 0.00 0.00 vif1 recv 4172.14 58.78 3.79 0.15 0.00 xmit 92481.29 18.07 0.41 0.00 0.00
Disk Statistics (per second) ut% is the percent of time the disk was busy. xfers is the number of data-transfer commands issued per second. xfers = ureads + writes + cpreads + greads + gwrites chain is the average number of 4K blocks per command. usecs is the average disk round-trip time per 4K block.
disk ut% xfers ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs /aggrcifs/plex0/rg0: 0a.02.0 0 0.48 0.00 .... . 0.18 6.91 816 0.30 3.11 1482 0.00 .... . 0.00 .... . 0a.02.10 1 0.58 0.00 .... . 0.31 4.84 1391 0.27 3.56 1263 0.00 .... . 0.00 .... . 0a.02.12 99 205.04 204.63 1.01 37088 0.18 4.91 26056 0.23 1.64 42043 0.00 .... . 0.00 .... . 0b.02.1 99 206.75 206.53 1.00 36560 0.08 10.40 6962 0.13 5.00 6850 0.00 .... . 0.00 .... . 0a.02.2 98 202.02 201.83 1.01 36183 0.08 10.00 10120 0.12 3.14 12455 0.00 .... . 0.00 .... . 0b.02.3 99 207.97 207.81 1.02 35700 0.08 8.40 11262 0.08 5.80 10207 0.00 .... . 0.00 .... . 0a.02.4 100 207.00 206.81 1.01 36822 0.08 10.20 8039 0.10 4.00 8542 0.00 .... . 0.00 .... . 0b.02.5 98 201.66 201.46 1.02 36275 0.08 11.40 2263 0.12 4.57 6188 0.00 .... . 0.00 .... . 0a.02.16 54 77.97 77.74 1.02 22560 0.08 10.60 3698 0.15 3.11 7964 0.00 .... . 0.00 .... . /aggrcifs/plex0/rg1: 0a.02.6 0 0.25 0.00 .... . 0.08 12.20 852 0.17 4.30 419 0.00 .... . 0.00 .... . 0b.02.7 0 0.25 0.00 .... . 0.08 12.20 754 0.17 4.30 581 0.00 .... . 0.00 .... . 0a.02.8 99 209.70 209.53 1.01 36216 0.08 10.60 7113 0.08 5.60 7036 0.00 .... . 0.00 .... . 0b.02.9 99 208.67 208.45 1.01 36722 0.08 9.40 8383 0.13 3.13 20640 0.00 .... . 0.00 .... . 0b.02.11 99 205.98 205.77 1.01 36561 0.08 10.80 3907 0.13 3.88 7194 0.00 .... . 0.00 .... . 0b.02.13 100 208.45 208.22 1.01 36938 0.08 9.80 12204 0.15 3.11 7714 0.00 .... . 0.00 .... . 0a.02.14 99 208.02 207.89 1.01 36361 0.08 10.20 9588 0.05 6.33 5316 0.00 .... . 0.00 .... . 0b.02.15 99 208.82 208.64 1.00 36455 0.08 10.40 10077 0.10 3.67 9455 0.00 .... . 0.00 .... . 0b.02.17 53 73.52 73.32 1.02 23057 0.08 9.80 2510 0.12 3.57 4840 0.00 .... . 0.00 .... . /aggr0/plex0/rg0: 0a.00.0 1 1.82 0.33 1.00 2150 1.49 4.82 316 0.00 .... . 0.00 .... . 0.00 .... . 0a.00.2 1 1.96 0.33 1.00 1600 1.62 4.59 336 0.00 .... . 0.00 .... . 0.00 .... . 0a.00.4 0 0.76 0.27 1.00 7125 0.30 20.11 262 0.20 13.75 473 0.00 .... . 0.00 .... .
Aggregate statistics: Minimum 0 0.25 0.00 0.08 0.00 0.00 0.00 Mean 62 125.59 125.21 0.25 0.13 0.00 0.00 Maximum 100 209.70 209.53 1.62 0.30 0.00 0.00
Spares and other disks: 0a.00.5 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0a.00.3 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0a.00.6 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0a.00.7 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.4 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.11 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0a.00.1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.6 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.18 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.3 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.0 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.5 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.12 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.8 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.9 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.10 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.20 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.02.19 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.7 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.23 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.21 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0a.02.18 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.17 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.22 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.19 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.15 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.13 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.14 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.16 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
FCP Statistics (per second) 0.00 FCP Bytes recv 0.00 FCP Bytes sent 0.00 FCP ops
iSCSI Statistics (per second) 0.00 iSCSI Bytes recv 0.00 iSCSI Bytes xmit 0.00 iSCSI ops
Interrupt Statistics (per second) 2209.15 int_0 4.62 int_1 0.20 int_2 0.20 int_3 14.68 int_6 2.10 int_7 0.08 Gigabit Ethernet (IRQ 8) 0.00 RTC 0.00 IPI 999.83 Msec Clock 3230.86 total
------------------------------------------------------------------------------
The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately, and delete the original message.