toasters

toasters@lists.teaparty.net

13513 discussions

node too busy
by Peter D. Gray 20 Jun '18

20 Jun '18

Hi people, I saw a recent post from a list member asking about how to finsd out what is keeping a node so busy. No, I had a quick look at my systems and sure as eggs I have a node which is 99% bust pretty much all the time dispite not doing that mush I/O. Example: Node CPU_(%) *Total_Ops Total_(Bps) Latency_(us) UOW-DR-03 38 6507 178442598 4152 UOW-DR-01 40 6195 24945585 1142 UOW-DR-02 99 2618 79917069 3246 UOW-DR-04 2 0 0 0 This is just a 5 sec sample, but its pretty much like that all the time. How do a diagnose why? Ontap 9.3P2 Regards, pdg Peter Gray Ph (direct): +61 2 4221 3770 Information Management & Technology Services Ph (switch): +61 2 4221 3555 University of Wollongong Fax: +61 2 4229 1958 Wollongong NSW 2522 Email: pdg(a)uow.edu.au Australia URL: http://pdg.uow.edu.au

2 1

FAS8040 cpu pegged for over 1 month 24/7
by jordan slingerland 15 Jun '18

15 Jun '18

I have an 8040 is at 100% cpu all the time on all cores. It has been like this for at least a month, my stats do not go back further than this. I expect the cpu was not this high before upgradin to 9.3P2. I feel like i should be able to get more than 10k ops out of a 150TB hybrid aggregate with 222 disks in it. Any help or feedback on performance expectations will be appreciated. Let me know if any stats would be usefull, i stripped my email down as my first try was rejected for being too big. HOSTNAME::> node run -node HOSTNAME-0 HOSTNAME-01 HOSTNAME-02 HOSTNAME::> node run -node HOSTNAME-02 Type 'exit' or 'Ctrl-D' to return to the CLI HOSTNAME-02> sysstat prif set diag priv set diag Warning: These diagnostic commands are for use by NetApp personnel only. HOSTNAME-02*> sysstat -M 1 ANY1+ ANY2+ ANY3+ ANY4+ ANY5+ ANY6+ ANY7+ ANY8+ AVG CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 Nwk_Excl Nwk_Lg Nwk_Exmpt Protocol Storage Raid Raid_Ex Xor_Ex Target Kahuna WAFL_Ex(Kahu) WAFL_MPClean SM_Exempt Exempt SSAN_Ex Intr Host Ops/s CP 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 0% 32% 0% 0% 0% 30% 10% 0% 0% 253%( 36%) 0% 0% 99% 18% 3% 353% 3748 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 0% 33% 0% 0% 0% 29% 10% 0% 0% 235%( 33%) 0% 0% 70% 19% 3% 398% 3959 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 0% 34% 0% 0% 0% 30% 9% 0% 0% 245%( 35%) 0% 0% 74% 19% 3% 385% 3802 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 0% 30% 0% 1% 0% 28% 8% 0% 0% 236%( 33%) 0% 0% 76% 17% 3% 399% 3367 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 1% 30% 0% 2% 0% 29% 8% 0% 0% 217%( 31%) 0% 0% 71% 16% 3% 422% 3427 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 1% 36% 0% 1% 1% 33% 9% 0% 0% 242%( 34%) 0% 0% 91% 19% 3% 362% 3754 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 1% 33% 0% 0% 0% 26% 4% 0% 0% 272%( 38%) 18% 0% 102% 18% 4% 320% 3496 60% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 1% 34% 0% 1% 0% 24% 4% 0% 0% 258%( 36%) 3% 0% 87% 20% 3% 364% 4001 0% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 1% 41% 0% 0% 0% 32% 5% 0% 0% 276%( 39%) 2% 0% 101% 24% 4% 312% 4752 0% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 1% 44% 0% 0% 0% 27% 4% 0% 0% 268%( 38%) 1% 0% 101% 25% 4% 323% 5077 28% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 1% 36% 0% 0% 0% 29% 2% 0% 0% 256%( 36%) 3% 0% 96% 21% 4% 352% 4105 2% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 1% 40% 0% 2% 0% 22% 1% 0% 0% 252%( 36%) 2% 0% 114% 23% 4% 339% 4801 0% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 0% 33% 0% 3% 0% 17% 1% 0% 0% 236%( 33%) 2% 0% 73% 20% 3% 410% 4280 0% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 0% 33% 0% 1% 0% 16% 2% 0% 0% 232%( 33%) 1% 0% 67% 20% 3% 424% 4219 0% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 0% 29% 0% 0% 0% 15% 1% 0% 0% 230%( 32%) 2% 0% 71% 18% 3% 429% 3719 0% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 1% 31% 0% 0% 0% 24% 2% 0% 0% 241%( 34%) 3% 0% 75% 19% 3% 400% 4206 0% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 1% 28% 0% 0% 0% 46% 11% 0% 0% 263%( 37%) 67% 0% 82% 17% 3% 280% 3496 59% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 0% 25% 0% 0% 1% 38% 7% 0% 0% 324%( 46%) 52% 0% 107% 16% 3% 226% 3127 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 0% 30% 0% 0% 1% 35% 7% 0% 1% 285%( 40%) 49% 0% 86% 18% 3% 285% 3633 100% 0% 0% 94% 20% 4% 341% 4300 100% HOSTNAME-02*> sysstat -x 1 CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP_Ty CP_Ph Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time [T--H--F--N--B--O--#--:] [n--v--p--f] util in out in out 100% 6 0 0 3907 100046 124797 278121 20862 0 0 3s 94% 0% 0--0--0--0--0--0--0--0 0--0--0--0 25% 3 0 3898 0 0 99115 127259 100% 3 0 0 3869 60123 134198 411571 20704 0 0 4s 94% 0% 0--0--0--0--0--0--0--0 0--0--0--0 33% 5 0 3861 0 0 59346 120058 100% 0 0 0 4089 27375 116197 225213 18145 0 0 4s 93% 0% 0--0--0--0--0--0--0--0 0--0--0--0 23% 0 0 4089 0 0 26622 115872 100% 13 0 0 4175 35817 111147 260996 28849 0 0 2s 94% 0% 0--0--0--0--0--0--0--0 0--0--0--0 25% 0 0 4162 0 0 34874 125888 100% 2 0 0 4143 56386 99541 401056 23882 0 0 0s 93% 0% 0--0--0--0--0--0--0--0 0--0--0--0 37% 17 0 4124 0 0 55471 105104 100% 11 0 0 3750 25706 149389 364912 154655 0 0 2s 93% 43% 1--0--0--0--0--0--0--0 0--1--0--0 29% 0 0 3739 0 0 24904 119412 100% 4 0 0 3395 36225 45390 280507 211869 0 0 3s 92% 100% 0--0--0--0--0--0--0--1 0--0--1--0 40% 0 0 3391 0 0 35744 81843 100% 25 0 0 4713 46662 149078 307272 93644 0 0 4s 95% 100% 0--0--0--0--0--0--0--1 0--0--0--1 42% 184 0 4504 0 0 45774 118418 100% 2 0 0 3930 51697 110975 221814 105300 0 0 3s 93% 100% 0--0--0--0--0--0--0--1 0--0--0--1 26% 0 0 3928 0 0 50821 108484 100% 5 0 0 4148 54985 137949 267436 208711 0 0 3s 93% 100% 1--0--0--0--0--0--0--1 1--0--0--1 31% 9 0 4134 0 0 53645 142356 100% 16 0 0 4883 81959 108174 364367 666466 0 0 4s 94% 100% 0--0--0--0--0--0--0--2 0--0--0--2 40% 1 0 4866 0 0 81554 95623 100% 5 0 0 6111 74315 122018 268918 79372 0 0 3s 95% 100% 0--0--0--0--0--0--0--1 0--0--0--1 26% 4 0 6102 0 0 72956 126959 100% 3 0 0 4658 50514 119345 218663 157863 0 0 3s 94% 100% 0--0--0--0--0--0--0--1 0--0--0--1 22% 0 0 4655 0 0 50021 112484 100% 2 0 0 4125 61813 137044 244725 223996 0 0 3s 93% 100% 0--0--0--0--0--0--0--1 0--0--0--1 23% 0 0 4123 0 0 59479 135345 100% 6 0 0 4005 48098 158865 283920 54318 0 0 4s 95% 15% 0--0--0--0--0--0--0--0 0--0--0--0 26% 14 0 3985 0 0 47165 162826 100% 24 0 0 4490 45946 155717 266290 96976 0 0 2s 93% 0% 0--0--0--0--0--0--0--0 0--0--0--0 28% 0 0 4466 0 0 44575 146650 100% 7 0 0 4935 52081 155766 331777 11753 0 0 4s 93% 0% 0--0--0--0--0--0--0--0 0--0--0--0 32% 2 0 4926 0 0 51171 165626 100% 0 0 0 6518 39526 179763 350349 24130 0 0 1s 92% 0% 0--0--0--0--0--0--0--0 0--0--0--0 29% 4 0 6514 0 0 38678 177139 HOSTNAME-02*> qos exit logout HOSTNAME::> qos statistics characteristics show -iterations 0 -rows 10 Policy Group IOPS Throughput Request size Read Concurrency Is Adaptive? -------------------- -------- --------------- ------------ ---- ----------- ------------ -total- 8094 321.31MB/s 41627B 41% 17 - data02_PROD_2 1626 101.69MB/s 65578B 80% 7 false DEV_DATA_DEV_2 1528 147.53MB/s 101219B 57% 5 false _System-Work 1243 4.40KB/s 3B 1% 0 false DEV_DATA_DEV 749 6.51MB/s 9121B 93% 1 false data02_PROD 667 9.48MB/s 14911B 0% 1 false DEV_OS_DEV_2 470 28.78MB/s 64214B 35% 2 false data04_PROD 256 4.35MB/s 17812B 8% 0 false data03_PROD 191 3.64MB/s 19971B 0% 0 false os03_PROD_2 191 1.55MB/s 8496B 0% 0 false shares02_PROD 183 1382.41KB/s 7721B 84% 0 false -total- 9542 347.25MB/s 38158B 42% 21 - DEV_DATA_DEV_2 1675 83.11MB/s 52028B 71% 4 false data02_PROD_2 1547 103.42MB/s 70112B 86% 7 false _System-Work 1191 6.58KB/s 5B 0% 0 false DEV_OS_DEV_2 860 30.42MB/s 37095B 37% 2 false DEV_DATA_DEV 667 5.22MB/s 8208B 96% 0 false data03_PROD_2 627 11.51MB/s 19255B 2% 3 false data02_PROD 528 7.24MB/s 14370B 0% 0 false shares02_PROD 526 80.62MB/s 160604B 66% 3 false data01_PROD 411 4.87MB/s 12431B 0% 0 false data03_PROD 377 4.95MB/s 13746B 0% 0 false Policy Group IOPS Throughput Request size Read Concurrency Is Adaptive? -------------------- -------- --------------- ------------ ---- ----------- ------------ -total- 11004 218.56MB/s 20827B 36% 15 - _System-Work 3750 24.82KB/s 6B 0% 0 false DEV_DATA_DEV_2 1572 25.67MB/s 17129B 78% 2 false data02_PROD_2 1568 91.24MB/s 61028B 99% 7 false DEV_DATA_DEV 793 6.19MB/s 8185B 96% 0 false os03_PROD 652 5.18MB/s 8322B 4% 1 false DEV_OS_DEV_2 549 31.44MB/s 60082B 44% 2 false data01_PROD 411 6.96MB/s 17753B 0% 0 false shares02_PROD 304 34.42MB/s 118717B 42% 2 false data02_PROD 261 4.56MB/s 18303B 0% 0 false data03_PROD 237 3.59MB/s 15904B 0% 0 false

3 4

mgmtgwd.aggregate.used.rising
by Mike Thompson 14 Jun '18

14 Jun '18

Hey guys, In Ontap 8, we used to route this message to a group of operators, who would move volumes around when aggregates started getting full. I don't see the associated message in the Ontap 9 message catalog, or anything that looks similar. Any ideas?

4 6

can't auth against domain with group
by jordan slingerland 13 Jun '18

13 Jun '18

Hey guys, this one is racking my brain and I am dreading calling netapp with this issue. I have like 10 differnet files all authenticating against the same domain for admin access. On one of them only hostname::> security login show -vserver hostname User/Group Application Auth MEthod Role domain\Storage_Team ssh domain admin Does not work, yet domain\jordan ssh domain admin works just fine. I am using the exact same commands, dns servers, etc to configure them all. almost all are currently 9.3P2. I beleive I have them all configured exactly the same way for auth, but appearently I am missing somehting. Thanks, Jordan

1 0

Spare Disks
by Philbert Rupkins 30 May '18

30 May '18

Toasters, Inherited a few production 7-mode systems that are off maintenance, due to be retired and pretty low on spare disks. Migrations to new ONTAP 9 systems (A300's, yay!) are in progress but I'd like to purchase several spare disks for the 7-mode systems as a safety measure. Unfortunately, these disks are so old that NetApp no longer sells them. Is it generally safe to purchase refurbished (or new) NetApp disks from a 3rd party? Can anybody recommend a vendor? I already grabbed quotes from our NetApp account team for lager capacity disks that I'm told won't be a problem for existing RAID groups made up of smaller disks. Primary concern is keeping the data safe/available so, if purchasing 3rd party is a bad idea, we'll definitely to through NetApp. 3rd party options always come up in discussions and we've never gotten outside feedback so just curious if anybody has had good or bad experiences going 3rd party. Thank you. Phil

6 5

administrivia: GDPR
by Tom Yates 29 May '18

29 May '18

This list was originally run by majordomo, and then by mailman. No-one got on this list without asking to be there. You all get emailed every month telling you how to get off it, and on the odd occasions people can't work it out from there I remove them on request. I am not fannying about requiring everyone to re-opt in. The exception to this is people who are subscribed via exploders. I don't approve of exploders, since they give me a headache, but I don't actively hunt them down and remove them. They, and their GDPR implications if any, are the problem of those who run the exploders, not me. Me, my wife (who is co-sysadmin on the hosting box), and Kendall Libby (being co-list-admin) have access to the subscriber list. We won't be telling anyone who's on it. If anyone needs further clarification, email me. While I'm sending this out, I should mention that I have enabled automated message-wrapping from people whose ISPs have, in their infinite wisdom, decided to enable DMARC reject or quarantine policies. People from those domains who post will now get their message "wrapped" in a digest from the list itself. When replying to the senders of list messages, have care to check the sender's email address in case it turns out to be the whole list. I'd hoped DMARC would go away, but no such luck, so this seems least dreadful. ** If for some reason any of this turns out to be contentious, please start with me before discussing it on-list. ** -- Tom Yates - http://www.teaparty.net

1 0

Fwd: Spare Disks
by Sebastian P. Goetze 22 May '18

22 May '18

---------- Forwarded message --------- From: Scott Miller <scott.miller(a)dreamworks.com> Date: Tue, May 22, 2018, 06:49 Subject: Re: Spare Disks To: Sebastian P. Goetze <spgoetze(a)gmail.com> What size and form factor are you looking for? We’re doing s round of retirement and may have some. Also: try the guys at ZeroWait They deal in used and off-lease kit, And have a huge inventory. -skottie (via my thumbs) On May 21, 2018, at 9:40 PM, Sebastian P. Goetze <spgoetze(a)gmail.com> wrote: Hi Phil, hi Alex, Our Admin bought some pallets of old DS14 shelfs some time ago off of eBay for our labs. Like 80%+ of the disks we were able to use, but for some we had to go to maintenance mode and delete labels and stuff to do that... It was about 100€ for a shelf, if I remember correctly, so we didn't mind the 20% that were dead/unusable... A zeroed, unassigned disk should automatically be reformatted with the correct WAFL version, by the way. Up or down. So no worries there with disks from newer ONTAP versions... Sebastian NCI On Tue, May 22, 2018, 06:10 Alexander Griesser <AGriesser(a)anexia-it.com> wrote: > Hi Philbert, > > not sure where you're located but since you didn't mention it, I guess > you're based in the US. > I do not have any recommendations for third-party suppliers in the US, but > I've used some within Europe already for outdated filers of our customers > already and never had real issues with those refurbished or used hard > drives. > > JFTR, here are some suggestions for suppliers based in Germany: > https://www.servershop24.de/ > https://www.miller-anlagen.de/shop > https://www.fibrestore.de/ > > You might experience bad labels on the disks as well as firmware issues > which might need to be sorted out then; sometimes the disks are not zeroed > so when plugging them in, you will see some left over aggregates from the > previous filer they did belong to or the owner ids are still set; you can > work around every single issue that might come up here, as long as your > Ontap version is not too old on the outdated filers. > > Best, > > Alexander Griesser > Head of Systems Operations > > ANEXIA Internetdienstleistungs GmbH > > E-Mail: AGriesser(a)anexia-it.com > Web: http://www.anexia-it.com > > Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt > <https://maps.google.com/?q=Klagenfurt:+Feldkirchnerstra%C3%9Fe+140,+9020+Kl…> > Geschäftsführer: Alexander Windbichler > Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT > U63216601 > > -----Ursprüngliche Nachricht----- > Von: toasters-bounces(a)teaparty.net [mailto:toasters-bounces@teaparty.net] > Im Auftrag von Philbert Rupkins > Gesendet: Dienstag, 22. Mai 2018 05:58 > An: Toasters <toasters(a)teaparty.net> > Betreff: Spare Disks > > Toasters, > > Inherited a few production 7-mode systems that are off maintenance, > due to be retired and pretty low on spare disks. Migrations to new > ONTAP 9 systems (A300's, yay!) are in progress but I'd like to purchase > several spare disks for the 7-mode systems as a safety > measure. Unfortunately, these disks are so old that NetApp no longer > sells them. > > Is it generally safe to purchase refurbished (or new) NetApp disks > from a 3rd party? Can anybody recommend a vendor? > > I already grabbed quotes from our NetApp account team for lager capacity > disks that I'm told won't be a problem for existing RAID > groups made up of smaller disks. Primary concern is keeping the data > safe/available so, if purchasing 3rd party is a bad idea, we'll definitely > to through NetApp. > > 3rd party options always come up in discussions and we've never gotten > outside feedback so just curious if anybody has had good or bad experiences > going 3rd party. > > Thank you. > Phil > _______________________________________________ > Toasters mailing list > Toasters(a)teaparty.net > http://www.teaparty.net/mailman/listinfo/toasters > > _______________________________________________ > Toasters mailing list > Toasters(a)teaparty.net > http://www.teaparty.net/mailman/listinfo/toasters > -- sent from my mobile, spellcheck might have messed up... _______________________________________________ Toasters mailing list Toasters(a)teaparty.net http://www.teaparty.net/mailman/listinfo/toasters -- sent from my mobile, spellcheck might have messed up...

1 0

Restrict VSC to a Subset of SVM's
by Philbert Rupkins 22 May '18

22 May '18

Hello Toasters, Our ontap clusters contain a number of SVM's. For purposes of this post I'll classify our SVM's into two broad categories: * SVM's that host storage resources for our ESXi clusters * SVM's that do NOT host storage resources for our ESXi clusters We initially direct connected VSC to the SVM's hosting VMware resources. As documented by NetApp, this resulted in VSC provisioning volumes (NFS datastores) then mounting them via indirect paths (our SVMs have multiple lifs). We dont want datastores mounted via indirect paths, nor do we want to deal with the other limitations associated with direct connecting VSC to SVM's. Now, AFAIK, the only option we're left with is connecting VSC to the cluster management LIF. The catch is we only want to allow VSC privileges to manage the SVM's hosting VMware resources. VSC should not have privileges to the non-Vmware related SVMs. Is there a way to connect VSC to the cluster management LIF while only allowing VSC the ability to provision storage to and manage a subset of SVM's on the cluster? We're currently running VSC 6.2.1 and ONTAP 9.2P2. -Phil

4 4

Re: moving CIFS shares
by Peter D. Gray 08 May '18

08 May '18

On Tue, May 08, 2018 at 01:33:47AM +0000, Fenn, Michael wrote: > How many total shares do you have? We've had good luck with a 1:1 ratio of volumes to shares since the platform limit is 1000 volumes/node on most recent hardware. With that model, you can shuffle volumes (and thus shares) between aggregates online with vol move, and you get easy access to goodies like QoS. We have about 1300 shares at the moment. More volumes is not manageable IMHO. For example, an altavault restricts snapmirror to 100 volumes. We already have more volumes that that. Regards, pdg

2 1

moving CIFS shares
by Peter D. Gray 08 May '18

08 May '18

Hi people, Background: we have 20 odd volumes for fileshares. We limit the size of the volumes to 10T. When a volume gets near full, we need to shuffle shares to new volumes or volumes with plenty of space. Does anybody have a magical solution for moving fileshares between volumes? With minimal but not necessarily zero downtime? Obvious solutions are: 1) Down share, copy share to new volume, up share 2) Down share, clone volume, up share on new volume, split volume, cleanup mess. I was wondering if anybody has seen a system to proxy the share and move it in the background. I have seen this work with NFS shares but not with CIFS. Any magicians out there? Regards, pdg

4 4

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters