toasters

toasters@lists.teaparty.net

1 participants
13531 discussions

What's using up my disks?
by Rhodes, Richard L. 08 Mar '18

08 Mar '18

Hello, We have a series of old IBM N3220 (NetApp 2240-2) running Data ONTAP Release 8.1.2P1 7-Mode. We are actively migrating off these systems and plan to be off of them by early next month. One of the systems is for DR. It's a snapmirror target as well as a snapvault source. We snapmirror vols from a prod system to this DR system, and then snapvault the vols to another system for a 2nd copy at the DR site. So all this DR system does (normally) is receive snapmirrors and is the source for snapvaults. PROD pair --sm--> DR pair --sv--> snapvault target system Over the past days the snapvaults have been running REAL slow. A snapvault that used to run in a few minutes is now running up to an hour or more. When NO snapmirror (target) or snapvault (source) is running, there is constant disk READ activity that is hitting the disks at 100% util. The sysstats below are taken when no snapmirror or snapvault activity is happening. The system is idle as far as any work goes. sdnapr0666a> sysstat -u -i 2 CPU Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk ops/s in out read write read write age hit time ty util 14% 0 1 1 9886 726 0 0 1 96% 61% T 100% 14% 0 2 2 9792 12 0 0 1 96% 100% : 100% 43% 0 2 33 12144 106 0 0 1 97% 100% : 100% 34% 0 5 75 10560 142 0 0 1 96% 100% : 100% 15% 0 0 0 10434 24 0 0 1 96% 57% : 100% 14% 0 23 558 10200 4 0 0 1 96% 0% - 100% 16% 0 1 0 10618 12 0 0 1 96% 0% - 100% 14% 0 0 0 10548 16 0 0 1 96% 0% - 100% sdnapr0666a> sysstat -s -i 2 CPU NFS CIFS iSCSI Net kB/s Disk kB/s iSCSI kB/s Cache in out read write in out age 15% 0 0 0 0 0 11372 4 0 0 1 15% 0 0 0 23 558 9912 62 0 0 1 14% 0 0 0 1 0 10692 18 0 0 1 15% 0 0 0 0 0 11240 0 0 0 1 15% 0 0 0 0 0 10624 12 0 0 1 16% 0 0 0 23 558 10384 16 0 0 1 15% 0 0 0 0 0 10902 548 0 0 1 14% 0 0 0 0 0 10484 16 0 0 1 The disks (a shelf of SATA disks) are getting pounded. There are no hardware problem like a failed disk. The system looks healthy. For curiosity I performed a takeover/giveback, but this activity followed to the partner head and back. Below is a statit for 60 seconds. Again, with no activity happening. Any thoughts on what's happening is welcome! Rick ==> statit for 60 seconds issuing statit -b cmd Warning: These advanced commands are potentially dangerous; use them only when directed to do so by IBM personnel. sleeping for 60 issuing statit -e cmd Hostname: sdnapr0666a ID: 1892873400 Memory: 4342 MB Data ONTAP Release 8.1.2P1 7-Mode: Mon Jan 7 15:55:05 PST 2013 <1O> Start time: Thu Mar 8 07:46:41 EST 2018 CPU Statistics 60.364027 time (seconds) 100 % 17.647400 system time 29 % 0.705493 rupt time 1 % (617560 rupts x 1 usec/rupt) 16.941907 non-rupt system time 28 % 223.808704 idle time 371 % 28.281968 time in CP 47 % 100 % 0.330279 rupt time in CP 1 % (289024 rupts x 1 usec/rupt) Multiprocessor Statistics (per second) cpu0 cpu1 cpu2 cpu3 total sk switches 25405.59 23819.82 20634.33 23665.75 93525.49 hard switches 3671.11 11061.21 11051.23 2377.44 28160.99 domain switches 320.97 117.44 103.52 61.97 603.90 CP rupts 1968.90 936.90 945.31 936.90 4788.02 nonCP rupts 2242.33 1062.79 1074.68 1062.79 5442.58 IPI rupts 0.00 0.00 0.00 0.00 0.00 grab kahuna 0.00 0.00 0.00 0.00 0.00 grab kahuna usec 0.00 0.00 0.00 0.00 0.00 CP rupt usec 3466.57 378.19 399.89 1226.79 5471.45 nonCP rupt usec 3943.76 427.32 456.17 1388.59 6215.84 idle 954903.67 928083.Warning: These advanced commands are potentially dangerous; use them only when directed to do so by IBM personnel. 11 895281.16 929382.40 3707650.35 kahuna 0.00 0.00 0.00 25821.64 25821.64 storage 58.00 40654.71 43.47 0.00 40756.19 exempt 7709.69 3161.45 8699.97 6.79 19577.94 raid 79.98 54.75 45300.54 0.00 45435.29 target 9.26 7.16 8.85 0.00 25.28 dnscache 0.00 0.00 0.00 0.00 0.00 cifs 32.64 41.61 37.31 0.00 111.57 wafl_exempt 23660.32 22689.72 44831.88 42173.68 133355.62 wafl_xcleaner 29.42 17.94 37.75 0.00 85.13 sm_exempt 838.96 673.91 1093.71 0.00 2606.60 cluster 0.00 0.00 0.00 0.00 0.00 protocol 30.80 37.95 27.80 0.00 96.56 nwk_exclusive 218.04 150.37 92.84 0.00 461.27 nwk_exempt 624.54 629.35 587.63 0.08 1841.63 nwk_legacy 1534.08 21.32 14.23 0.00 1569.64 hostOS 2860.18 2970.99 3086.67 0.00 8917.88 12.265273 seconds with one or more CPUs active ( 20%) 3.098531 seconds with 2 or more CPUs active ( 5%) 1.011320 seconds with 3 or more CPUs active ( 2%) 9.166741 seconds with one CPU active ( 15%) 2.087210 seconds with 2 CPUs active ( 3%) 0.929439 seconds with 3 CPUs active ( 2%) 0.081881 seconds with all CPUs active ( 0%) Domain Utilization of Shared Domains (per second) 0.00 idle 125450.41 kahuna 0.00 storage 0.00 exempt 0.00 raid 0.00 target 0.00 dnscache 0.00 cifs 0.00 wafl_exempt 0.00 wafl_xcleaner 0.00 sm_exempt 0.00 cluster 0.00 protocol 2482.49 nwk_exclusive 0.00 nwk_exempt 0.00 nwk_legacy 0.00 hostOS Miscellaneous Statistics (per second) 28160.99 hard context switches 0.00 NFS operations 0.00 CIFS operations 0.00 HTTP operations 4.08 network KB received 90.32 network KB transmitted 10664.89 disk KB read 149.10 disk KB written 2.93 NVRAM KB written 0.00 nolog KB written 0.00 WAFL bufs given to clients 0.00 checksum cache hits ( 0%) 0.00 no checksum - partial buffer 0.00 FCP operations 0.00 iSCSI operations WAFL Statistics (per second) 11.48 name cache hits ( 98%) 0.22 name cache misses ( 2%) 36472.78 buf hash hits ( 83%) 7309.54 buf hash misses ( 17%) 1319.00 inode cache hits ( 87%) 193.03 inode cache misses ( 13%) 11030.66 buf cache hits ( 96%) 507.36 buf cache misses ( 4%) 195.58 blocks read 2452.11 blocks read-ahead 418.10 chains read-ahead 4.85 dummy reads 0.00 blocks speculative read-ahead 17.59 blocks written 0.33 stripes written 0.00 blocks page flipped 0.00 blocks over-written 0.07 wafl_timer generated CP 0.00 snapshot generated CP 0.00 wafl_avail_bufs generated CP 0.00 dirty_blk_cnt generated CP 0.00 full NV-log generated CP 0.00 back-to-back CP 0.00 flush generated CP 0.00 sync generated CP 0.00 deferred back-to-back CP 0.00 container-indirect-pin CP 0.00 low mbufs generated CP 0.00 low datavecs generated CP 3327.41 non-restart messages 53.28 IOWAIT suspends 0.00 next nvlog nearly full msecs 0.00 dirty buffer susp msecs 0.00 nvlog full susp msecs 762408 buffers RAID Statistics (per second) 9.34 xors 0.00 long dispatches [0] 0.00 long consumed [0] 0.00 long consumed hipri [0] 0.00 long low priority [0] 0.00 long high priority [0] 0.00 long monitor tics [0] 0.00 long monitor clears [0] 0.00 long dispatches [1] 0.00 long consumed [1] 0.00 long consumed hipri [1] 0.00 long low priority [1] 0.00 long high priority [1] 0.00 long monitor tics [1] 0.00 long monitor clears [1] 15 max batch 0.78 blocked mode xor 1.61 timed mode xor 0.40 fast adjustments 0.27 slow adjustments 0 avg batch start 0 avg stripe/msec 2628.35 checksum dispatches 13837.23 checksum consumed 0.56 tetrises written 0.00 master tetrises 0.00 slave tetrises 8.27 stripes written 1.08 partial stripes 7.19 full stripes 17.83 blocks written 3.99 blocks read 6.00 1 blocks per stripe size 1 0.35 1 blocks per stripe size 7 0.13 2 blocks per stripe size 7 0.08 3 blocks per stripe size 7 0.18 4 blocks per stripe size 7 0.10 5 blocks per stripe size 7 0.23 6 blocks per stripe size 7 1.19 7 blocks per stripe size 7 Network Interface Statistics (per second) iface side bytes packets multicasts errors collisions pkt drops e0a recv 4042.26 56.92 0.00 0.00 0.00 xmit 92475.23 17.92 0.25 0.00 0.00 e0b recv 135.28 1.94 0.00 0.00 0.00 xmit 6.96 0.17 0.17 0.00 0.00 e0c recv 0.00 0.00 0.00 0.00 0.00 xmit 0.00 0.00 0.00 0.00 0.00 e0d recv 0.00 0.00 0.00 0.00 0.00 xmit 0.00 0.00 0.00 0.00 0.00 e0M recv 1.06 0.02 0.02 0.00 0.00 xmit 10.14 0.08 0.08 0.00 0.00 e0P recv 0.00 0.00 0.00 0.00 0.00 xmit 0.00 0.00 0.00 0.00 0.00 vh recv 0.00 0.00 0.00 0.00 0.00 xmit 0.00 0.00 0.00 0.00 0.00 vif1 recv 4172.14 58.78 3.79 0.15 0.00 xmit 92481.29 18.07 0.41 0.00 0.00 Disk Statistics (per second) ut% is the percent of time the disk was busy. xfers is the number of data-transfer commands issued per second. xfers = ureads + writes + cpreads + greads + gwrites chain is the average number of 4K blocks per command. usecs is the average disk round-trip time per 4K block. disk ut% xfers ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs /aggrcifs/plex0/rg0: 0a.02.0 0 0.48 0.00 .... . 0.18 6.91 816 0.30 3.11 1482 0.00 .... . 0.00 .... . 0a.02.10 1 0.58 0.00 .... . 0.31 4.84 1391 0.27 3.56 1263 0.00 .... . 0.00 .... . 0a.02.12 99 205.04 204.63 1.01 37088 0.18 4.91 26056 0.23 1.64 42043 0.00 .... . 0.00 .... . 0b.02.1 99 206.75 206.53 1.00 36560 0.08 10.40 6962 0.13 5.00 6850 0.00 .... . 0.00 .... . 0a.02.2 98 202.02 201.83 1.01 36183 0.08 10.00 10120 0.12 3.14 12455 0.00 .... . 0.00 .... . 0b.02.3 99 207.97 207.81 1.02 35700 0.08 8.40 11262 0.08 5.80 10207 0.00 .... . 0.00 .... . 0a.02.4 100 207.00 206.81 1.01 36822 0.08 10.20 8039 0.10 4.00 8542 0.00 .... . 0.00 .... . 0b.02.5 98 201.66 201.46 1.02 36275 0.08 11.40 2263 0.12 4.57 6188 0.00 .... . 0.00 .... . 0a.02.16 54 77.97 77.74 1.02 22560 0.08 10.60 3698 0.15 3.11 7964 0.00 .... . 0.00 .... . /aggrcifs/plex0/rg1: 0a.02.6 0 0.25 0.00 .... . 0.08 12.20 852 0.17 4.30 419 0.00 .... . 0.00 .... . 0b.02.7 0 0.25 0.00 .... . 0.08 12.20 754 0.17 4.30 581 0.00 .... . 0.00 .... . 0a.02.8 99 209.70 209.53 1.01 36216 0.08 10.60 7113 0.08 5.60 7036 0.00 .... . 0.00 .... . 0b.02.9 99 208.67 208.45 1.01 36722 0.08 9.40 8383 0.13 3.13 20640 0.00 .... . 0.00 .... . 0b.02.11 99 205.98 205.77 1.01 36561 0.08 10.80 3907 0.13 3.88 7194 0.00 .... . 0.00 .... . 0b.02.13 100 208.45 208.22 1.01 36938 0.08 9.80 12204 0.15 3.11 7714 0.00 .... . 0.00 .... . 0a.02.14 99 208.02 207.89 1.01 36361 0.08 10.20 9588 0.05 6.33 5316 0.00 .... . 0.00 .... . 0b.02.15 99 208.82 208.64 1.00 36455 0.08 10.40 10077 0.10 3.67 9455 0.00 .... . 0.00 .... . 0b.02.17 53 73.52 73.32 1.02 23057 0.08 9.80 2510 0.12 3.57 4840 0.00 .... . 0.00 .... . /aggr0/plex0/rg0: 0a.00.0 1 1.82 0.33 1.00 2150 1.49 4.82 316 0.00 .... . 0.00 .... . 0.00 .... . 0a.00.2 1 1.96 0.33 1.00 1600 1.62 4.59 336 0.00 .... . 0.00 .... . 0.00 .... . 0a.00.4 0 0.76 0.27 1.00 7125 0.30 20.11 262 0.20 13.75 473 0.00 .... . 0.00 .... . Aggregate statistics: Minimum 0 0.25 0.00 0.08 0.00 0.00 0.00 Mean 62 125.59 125.21 0.25 0.13 0.00 0.00 Maximum 100 209.70 209.53 1.62 0.30 0.00 0.00 Spares and other disks: 0a.00.5 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0a.00.3 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0a.00.6 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0a.00.7 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.4 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.11 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0a.00.1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.6 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.18 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.3 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.0 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.5 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.12 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.8 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.9 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.10 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.20 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.02.19 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.7 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.23 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.21 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0a.02.18 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.17 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.22 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.19 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.15 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.13 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.14 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0b.01.16 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . FCP Statistics (per second) 0.00 FCP Bytes recv 0.00 FCP Bytes sent 0.00 FCP ops iSCSI Statistics (per second) 0.00 iSCSI Bytes recv 0.00 iSCSI Bytes xmit 0.00 iSCSI ops Interrupt Statistics (per second) 2209.15 int_0 4.62 int_1 0.20 int_2 0.20 int_3 14.68 int_6 2.10 int_7 0.08 Gigabit Ethernet (IRQ 8) 0.00 RTC 0.00 IPI 999.83 Msec Clock 3230.86 total ------------------------------------------------------------------------------ The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately, and delete the original message.

2 1

NetApp at SCaLE in Pasadena
by Lori Barfield 07 Mar '18

07 Mar '18

The Southern California Linux Expo is a an annual four-day open source conference that starts this Thursday, 3/8. The expo hall will be open Friday 2 p.m. through Sunday 2 p.m., and NetApp is in Booth #416. Here's the website: http://www.socallinuxexpo.org I am an organizer. Is anyone else here planning on being there? ...lori

1 0

its been a month I have not seen any posting, hope my subscription is still valid.
by Uddhav Regmi 04 Mar '18

04 Mar '18

thx

4 3

RE: NFS issue after upgrading filers to 9.2P2
by Steiner, Jeffrey 24 Jan '18

24 Jan '18

It takes a lot for an ONTAP system to flat-out be unable to respond. Unless the timeout parameters are exceedingly short, you shouldn't reach that point, especially with anything capable of running ONTAP 9.2. I'd open a support case on this one. In addition, if you want to trigger an autosupport and send me the serial numbers directly I can take a glance at a few stats to see if anything looks odd. From: Fenn, Michael [mailto:fennm@DEShawResearch.com] Sent: Tuesday, January 23, 2018 6:23 PM To: Steiner, Jeffrey <Jeffrey.Steiner(a)netapp.com>; Mark Saunders <Mark.Saunders(a)pcmsgroup.com>; toasters(a)teaparty.net Subject: Re: NFS issue after upgrading filers to 9.2P2 The messages are not necessarily indicative of a network problem. The kernel prints "nfs: server … not responding, still trying" when an operation times out (timeo deciseconds) for the configured (retrans) number of tries. Once the server responds, then it prints "nfs: server … OK". Networking problems are certainly one reason that an operation would time out, but not the only reason. An overloaded or down file server will cause the same effect. Thanks, Michael From: <toasters-bounces(a)teaparty.net<mailto:toasters-bounces@teaparty.net>> on behalf of "Steiner, Jeffrey" <Jeffrey.Steiner(a)netapp.com<mailto:Jeffrey.Steiner@netapp.com>> Date: Tuesday, January 23, 2018 at 10:38 AM To: Mark Saunders <Mark.Saunders(a)pcmsgroup.com<mailto:Mark.Saunders@pcmsgroup.com>>, "toasters(a)teaparty.net<mailto:toasters@teaparty.net>" <toasters(a)teaparty.net<mailto:toasters@teaparty.net>> Subject: RE: NFS issue after upgrading filers to 9.2P2 Those messages are indicative of a network problem. The packets are going through, then they succeed when the NFS client retries, then they pause again. I can't think why an ONTAP upgrade of this type would cause such a problem. If it was working before, it should be working now. If you had any kind of a locking, firewall, or general configuration problem you should have no access whatsoever. I've seen some weird NFS bug sin SUSE, but that RHEL version should be fine. What are the mount options used, and are you using DNFS? From: toasters-bounces(a)teaparty.net<mailto:toasters-bounces@teaparty.net> [mailto:toasters-bounces@teaparty.net] On Behalf Of Mark Saunders Sent: Tuesday, January 23, 2018 4:29 PM To: toasters(a)teaparty.net<mailto:toasters@teaparty.net> Subject: NFS issue after upgrading filers to 9.2P2 Hi gents today we have upgraded our Coventry cluster from 9.1P6 to 9.2P2 and we are about 99% working we just have a strange issue with SAP database servers NFS mounts. When the server is bounced the mounts are attached with no problems but after a few minutes a df –h starts to be very slow reporting the NFS mounted directories and if the databases are started up they hang and a df –h then also hangs. This sometimes recovers enough to then allow a df –h to work again but the databases are a lost cause right now. In the server messages we get lots of entries for the SVM Jan 23 07:01:27 jwukccsbci kernel: nfs: server JWUKCSVM01 not responding, still trying Jan 23 07:01:47 jwukccsbci last message repeated 5 times Jan 23 07:02:07 jwukccsbci kernel: nfs: server JWUKCSVM01 OK Jan 23 07:02:07 jwukccsbci last message repeated 5 times Jan 23 07:02:27 jwukccsbci kernel: nfs: server JWUKCSVM01 not responding, still trying Jan 23 07:02:47 jwukccsbci last message repeated 5 times Jan 23 07:02:48 jwukccsbci kernel: nfs: server JWUKCSVM01 OK Is there anything that would of changed in the upgrade to lock down NFS or changes options that we might need to change back. The redhat servers are an old kernel version 2.6.18-371.el5 that has some bugs but this was working fine before the filer upgrade was carried out. Regards Mark Data Centre Sysadmin Team Managed Services Phone:- 02476 694455 Ext 2567 The Sysadmin Team promoting PCMS Values ~Integrity~Respect~Commitment~ ~Continuous Improvement~ The information contained in this e-mail is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient of this e-mail, the use of this information or any disclosure, copying or distribution is prohibited and may be unlawful. If you received this in error, please contact the sender and delete the material from any computer. The views expressed in this e-mail may not necessarily be the views of the PCMS Group plc and should not be taken as authority to carry out any instruction contained. The PCMS Group reserves the right to monitor and examine the content of all e-mails. The PCMS Group plc is a company registered in England and Wales with company number 1459419 whose registered office is at PCMS House, Torwood Close, Westwood Business Park, Coventry CV4 8HX, United Kingdom. VAT No: GB 705338743

7 20

NFS issue after upgrading filers to 9.2P2
by Mark Saunders 23 Jan '18

23 Jan '18

Hi gents today we have upgraded our Coventry cluster from 9.1P6 to 9.2P2 and we are about 99% working we just have a strange issue with SAP database servers NFS mounts. When the server is bounced the mounts are attached with no problems but after a few minutes a df -h starts to be very slow reporting the NFS mounted directories and if the databases are started up they hang and a df -h then also hangs. This sometimes recovers enough to then allow a df -h to work again but the databases are a lost cause right now. In the server messages we get lots of entries for the SVM Jan 23 07:01:27 jwukccsbci kernel: nfs: server JWUKCSVM01 not responding, still trying Jan 23 07:01:47 jwukccsbci last message repeated 5 times Jan 23 07:02:07 jwukccsbci kernel: nfs: server JWUKCSVM01 OK Jan 23 07:02:07 jwukccsbci last message repeated 5 times Jan 23 07:02:27 jwukccsbci kernel: nfs: server JWUKCSVM01 not responding, still trying Jan 23 07:02:47 jwukccsbci last message repeated 5 times Jan 23 07:02:48 jwukccsbci kernel: nfs: server JWUKCSVM01 OK Is there anything that would of changed in the upgrade to lock down NFS or changes options that we might need to change back. The redhat servers are an old kernel version 2.6.18-371.el5 that has some bugs but this was working fine before the filer upgrade was carried out. Regards Mark Data Centre Sysadmin Team Managed Services Phone:- 02476 694455 Ext 2567 The Sysadmin Team promoting PCMS Values ~Integrity~Respect~Commitment~ ~Continuous Improvement~ The information contained in this e-mail is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient of this e-mail, the use of this information or any disclosure, copying or distribution is prohibited and may be unlawful. If you received this in error, please contact the sender and delete the material from any computer. The views expressed in this e-mail may not necessarily be the views of the PCMS Group plc and should not be taken as authority to carry out any instruction contained. The PCMS Group reserves the right to monitor and examine the content of all e-mails. The PCMS Group plc is a company registered in England and Wales with company number 1459419 whose registered office is at PCMS House, Torwood Close, Westwood Business Park, Coventry CV4 8HX, United Kingdom. VAT No: GB 705338743

2 1

Implementing NSE FDE on an existing system
by jordan slingerland 10 Jan '18

10 Jan '18

I have an 8040 currently running CM 9.1P1. All disks are SED but currently in protection-mode open mode. Is there a method to convert the existing disks/aggregates to protection-mode full? Is enabling encryption on spare disks and preforming a disk replace 1 or 2 disks at a time a viable option? That sounds time consuming but I don't have 100TB of swing space.

1 0

Node(s) Out of Quorum During Takeover
by Philbert Rupkins 06 Jan '18

06 Jan '18

Hello, Just inherited a new 4-node ONTAP 9.1P9 cluster. First order of business - testing manual failover functionality. # CLUSTER INITIAL STATE HA PAIR 1: node1 - epsilon, master node2 - secondary HA PAIR 2: node3 - secondary node4 - secondary The node I intended to takeover (node1) was both the master and held the Epsilon. I moved Epsilon to a node in an HA pair that would remain up (node4). However, per the "cluster ring" output, the node I planned to takeover (node1) was still the master. # CLUSTER NODE1 PRE-TAKEOVER STATE HA PAIR 1: node1 - master node2 - secondary HA PAIR 2: node3 - secondary node4 - epsilon, secondary I initiated the takeover of node1 and received "Node(s) Out of Quorum" alerts from the three other members of the cluster. The cluster seemed to be happy after the takeover/giveback was complete. The "cluster ring" output showed the master role had been assumed by another node. # CLUSTER POST NODE1 TAKEOVER/GIVEBACK STATE HA PAIR 1: node1 - secondary node2 - master HA PAIR 2: node3 - secondary node4 - epsilon, secondary Is it normal to receive "Node(s) Out of Quorum" alerts when performing a manual takeover of the master node? Are these alerts the result of an election being forced? Thanks, Phil

1 0

VMware SRM + NetApp SRA + NetApp WFA?
by Ehrenwald, Ian 04 Jan '18

04 Jan '18

This isn't 100% NetApp related, more like 60%, but I think it's still a relevant topic of discussion that others may eventually benefit from too. I've been tasked with setting up DR for a set of VMware VMs at our primary data center by replicating them via SnapMirror to our secondary data center. Normally I'd just use VMware SRM with the NetApp SRA, set up Array Based Replication within SRM, and be done with it. Let SRM and the SRA take care of test, failover, reprotect, etc. However, the sticking point is that a lot of the protected VMs use in-guest NFS mounts for things like database volumes or application binaries and SRM doesn't know what to do with those SnapMirror pair volumes because they're not VMware datastores. My initial thought was to use some PowerShell with the NetApp PowerShell Toolkit in the SRM Recovery Plan for each individual VM pre-boot to connect to the SnapMirror destination filer, break the SnapMirror(s) that the VM uses, mount the volume(s), set export policies, etc. Then figure out how to correctly do resync and reprotect without blowing things up. I got as far as the first part in PowerShell (connect to the destination filer, break the mirror) and thought to myself that this was stupid.. I'm re-inventing the wheel. Surely someone else out there must have done this before and there is a pre-existing solution? I spoke with a NetApp tech support engineer that I happened to have a case open with already regarding the NetApp SRA and he was able to come up with a suggestion from another engineer: Call WFA from SRM in each VMs recovery plan to break the SnapMirror for the volumes required for the VM to work, then also use WFA in the cleanup and reprotect plan to do that work. Has anyone else done something like this? I have a feeling that I'm not breaking new ground with this task. Implementing it wrong has pretty severe implications, though, so I'm not eager to mess it up. Ian Ehrenwald Senior Infrastructure Engineer Hachette Book Group, Inc. 1.617.263.1948 / ian.ehrenwald(a)hbgusa.com This may contain confidential material. If you are not an intended recipient, please notify the sender, delete immediately, and understand that no disclosure or reliance on the information herein is permitted. Hachette Book Group may monitor email to and from our network.

4 7

Panic on bootup (page fault supervisor read, page not present)
by Alexander Griesser 27 Dec '17

27 Dec '17

Hey toasters, I’m trying to repurpose one of my old FAS3240 systems and have booted it up today just to be greeted with the following panic on one of the two controllers: Starting AUTOBOOT press Ctrl-C to abort... Loading X86_64/freebsd/image2/kernel:0x100000/8450504 0x90f1c8/1278720 Entry at 0x80158900 Loading X86_64/freebsd/image2/platform.ko:0xa48000/655232 0xb8ea20/694680 0xae7f80/39656 0xc383b8/43152 0xaf1a68/86316 0xb06b94/63845 0xb16500/140640 0xc42c48/159120 0xb38a60/2024 0xc699d8/6072 0xb39248/304 0xc6b190/912 0xb39378/1680 0xc6b520/5040 0xb39a08/960 0xc6c8d0/2880 0xb39dc8/184 0xc6d410/552 0xb39e80/448 0xb65de0/12918 0xb8e933/237 0xb69058/84120 0xb7d8f0/69699 Starting program at 0x80158900 NetApp Data ONTAP 8.1.3P1 7-Mode Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 04 fault virtual address = 0x28 fault code = supervisor read data, page not present instruction pointer = 0x8:0xffffffff80315571 stack pointer = 0x10:0xffffffff80c73c90 frame pointer = 0x10:0xffffffff80c73cc0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 0 (swapper) trap number = 12 PANIC : page fault (supervisor read, page not present) on VA 0x28 cs:rip = 0x8:0xffffffff80315571 rflags = 0x10246 version: 8.1.3P1: Wed Jul 10 20:17:14 PDT 2013 conf : x86_64 cpuid = 0 Uptime: 1s coredump: primary dumper unavailable. coredump: trying secondary dumper... no tftp server ip specified netdump: no targets available to dump coredump: secondary dumper failed -1. The operating system has halted. Please press any key to reboot. System halting... cpu_reset called on cpu#0 Any insights on what might be the issue here? Needless to say, there’s no maintenance anymore on this system so opening a support case is not possible 😊 Thanks, Alexander Griesser Head of Systems Operations ANEXIA Internetdienstleistungs GmbH E-Mail: AGriesser(a)anexia-it.com<mailto:AGriesser@anexia-it.com> Web: http://www.anexia-it.com<http://www.anexia-it.com/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

4 8

Chicken/egg dillema doing a hardware upgrade on a FAS3250
by Jan-Pieter Cornet 20 Dec '17

20 Dec '17

We have a FAS3250 that's primarily backup storage. It hosts a lot of snapmirror targets, and a few mounted filesystems with backup data. As the hardware is getting old, we've purchased a shiny new FAS8200 with a couple of 8TB drive shelves. We assume that the FAS8200 can be initialized on 9.2 and above using the ADP on external drives feature, to partition those 8T disks, so we don't have to throw away about 50T of storage just on the root aggregates. However... the FAS3250 hardware can only run ontap 9.1. Newer versions of ontap are not available on that hardware. And to make the 8200 nodes join the existing cluster, it has to run the same sofware version (9.1), so... it cannot be initialized with ADP. And you can't initialize first and partition later, because repartitioning wipes all existing data. That's our dilemma. I've actually tried to "manually" partition the disks, by going into maintenance mode and using "disk partition" to force the 8200 nodes to see only partitioned disks, and then re-initializing them using boot menu "4 - clean config and reinitialize disks". That fails, and 9.1 doesn't want to write a root FS to partitioned disks (it does wipe them, though). Besides, it would likely be unsupported. I did learn some interesting things about the "disk partition" command, by the way, like the "-b" blocksize is in 4k blocks, and the '-i' option that numbers partitions starts at 1, which is a data partition, and number 2 is the root partition, at least in the 2-partition root/data setup. So we either have the option to throw away a large chunk of storage for root aggregates, initialize the new nodes on 9.1, join in cluster, and move the existing data using 'vol move' and all the cDOT goodness that comes with it. ... or build the new 8200 as a separate cluster, initialize it with 9.3, partition the disks via the boot menu, and then move the data over using snapmirror, and remounting the clients. That's doable because most of the data is snapmirror target anyway, and there's a limited number of mounted filesystems that would need a remount. It's a shame that cDOT doesn't have "snapmirror migrate" like 7mode did. Does anyone have any other options? All I could think is get swing gear and basically do the migration twice, first to hardware that supports ontap >= 9.2, then to our new 8200. But I'm not willing to spend a lot of money renting the swing gear and getting a lot of extra setup work basically because of a flaw in netapp software. Thanks, -- Jan-Pieter Cornet <johnpc(a)xs4all.nl> "Any sufficiently advanced incompetence is indistinguishable from malice." - Grey's Law

7 10

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters