toasters October 2013

toasters@lists.teaparty.net

38 participants
17 discussions

[OT] General Purpose Storage Discussion
by Ray Van Dolson 29 Nov '13

29 Nov '13

This list is great -- but obviously is NetApp centric. I'm wondering if any of you can recommend a mailing list or forum, with a similar level of participation and expertise, that's a bit more "general purpose" but still storage-centric. Something similar to the SAGE/LISA mailing lists for Systems Administration. Perhaps SNIA offers such discussion groups to its members? Thanks, Ray

7 15

Netapp FAS610 8.1.3P1 stalling
by Martin 21 Nov '13

21 Nov '13

One of our 6210 Filers running 8.1.3P1 stalled I/O for approx 1 minute (11:08:27 -11:09:24) We saw this as latency of 5 seconds on the VMware hosts attached to the Filer via NFS and eventually "All Paths Down" messages on the ESX hosts. I also saw warnings in the messages file: Tue Oct 22 11:09:24 BST [TOASTER1: NwkThd_01:warning]: NFS response to client x.x.x.x for volume 0x5834a5c(vol004) was slow, op was v3 write, 69 > 60 (in seconds) >From looking in DFM Filer Summary view I see a "Z" shape in the graph for most of the counters on the Filer e.g. CPU, Network Throughput, All Protocol Ops. The counters dip low then rapidly increase and tail off again. (see attached JPG) <http://network-appliance-toasters.10978.n7.nabble.com/file/n25314/Filer-Z-s…> During this time it all of the ESX hosts saw timeouts to the NFS datastores. I checked the disk_busy on the only aggregate (90 x 15K SAS 450GB) on the Filer and it only shows the disks as 30-40% busy and the disk_busy drops during the time the Filer stalled. It seems odd as the overall load on the Filer didn't increase to precipitate this. <http://network-appliance-toasters.10978.n7.nabble.com/file/n25314/Filer-Z-s…> >From previous experience of performance cases with Netapp, we're usually asked to gather a perfstat next time it happens but this isn't possible as it's unpredictable and short lived. The thing that concerns me is from previous experience this is usually a precursor to the Filer stalling for much longer periods causing much more impact. In the past we've found the only option is to upgrade the Filer head. I would appreciate any pointers on how to identify the root cause of this. -- View this message in context: http://network-appliance-toasters.10978.n7.nabble.com/Netapp-FAS610-8-1-3P1… Sent from the Network Appliance - Toasters mailing list archive at Nabble.com.

3 9

ndmpcopy and vol copy via ssh?
by Jeff Cleverley 01 Nov '13

01 Nov '13

Greetings, Is it acceptable to use ssh from an admin host to run ndmpcopy and vol copy commands? It seems to work initially, but both seem to die after varying periods of time with no useful explanations in the log files. I found a KB about someone connecting to the filer and running ctrl-c thinking the console is idle, but that doesn't apply here unless an ssh command from a cron job or something has the same effect. ndmpd status and vol copy status verify that there are no copies running. All filers are 8.1.2P4, source is a 6080, destination is a 6290, 10G networking on both. I'm doing restore time tests of entire volumes using various methods and these 2 are part of the list. The vol copy ran for ~10 minutes out of the 466 estimated, then stopped. No errors on the command line. It does run, so all the host.equiv entries and permissions are good. I'm sure it did not do the 11TB in 10 minutes :-) The ndmpcopy command ran for ~80 minutes and silently quit. It restored 217G, almost all are in smaller directories. The larger directories didn't show up and there are no abort messages in the logs. They just quit logging. For the ndmpcopy I had a script do an ndmpcopy for each directory (9 total). It looked like multiple ones were running at the same time which seems OK. I was hoping to use parallel threads to speed things up. Any ideas on the silent failures? Thanks, Jeff -- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611

5 10

AW: Dedup killing performance (again)
by Alexander Griesser 27 Oct '13

27 Oct '13

Is it stable? I mean 8.1.3 is just in GA for now... Alexander Griesser System-Administrator ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-320 Telefax: +43-5-0556-500 E-Mail: ag(a)anexia.at<mailto:ag@anexia.at> Web: http://www.anexia.at<http://www.anexia.at/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 Von: Peter-Paul Witta [mailto:paul.witta@CUBiT.at] Gesendet: Samstag, 19. Oktober 2013 00:30 An: Alexander Griesser; Sebastian Goetze Cc: toasters(a)teaparty.net Betreff: Re: Dedup killing performance (again) I'd recommend 813p2 immediately... lg paul witta Am 19.10.13 00:19, schrieb Alexander Griesser: Yah, just found it while googling I guess, must be bug id #657692 :-/ Anyhow, I'm now stuck with this: > sis status Path State Status Progress /vol/CIFS_SHARES Enabled Active Fingerprint is being recovered from aggregate (6% complete) It's doing that since quite some time now, although it now actually jumped from 0% to 6%, so it at least is still alive. This bug is first fixed in 8.1.2P4 as it seems, so I might need to upgrade ASAP; for now, I'll check if it at least finishes this process now, my volume is filling up quickly, but there's still some aggregate space left so I'll just increase the volume size before it goes full. I'll go above the magic 80% fill of an aggregate then, but better that then having the volume go full. I'll see how it goes and if it troubles me too much, I hope there is a way to abort sis in that state. Alexander Griesser System-Administrator ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-320 Telefax: +43-5-0556-500 E-Mail: ag(a)anexia.at<mailto:ag@anexia.at> Web: http://www.anexia.at<http://www.anexia.at/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 Von: Sebastian Goetze [mailto:spgoetze@gmail.com] Gesendet: Samstag, 19. Oktober 2013 00:16 An: Alexander Griesser Betreff: Re: Dedup killing performance (again) Isn't that the Dedupe bug again, that was fixed in 8.1.3? (since like 8.1.2P3 or P4)... There was a thread about it some time ago... Anyway, I'm off to bed, just wanted to point you in the right direction. BFN On 10/18/2013 11:52 PM, Alexander Griesser wrote: Hi there, I am just trying to dedupe some volumes on one of our filers and the first volume (2.4TB data, CIFS share) deduplicates since about 4 hours now. Sysstat -x 1 shows: CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out 58% 115 0 0 116 2239 3635 281593 32 0 0 0s 98% 0% - 100% 0 0 1 0 0 0 0 63% 162 0 0 169 1682 4149 310829 0 0 0 0s 99% 0% - 100% 7 0 0 0 0 0 0 50% 381 0 0 384 11793 11325 182100 0 0 0 0s 96% 0% - 100% 0 0 3 0 0 0 0 55% 412 0 0 416 6487 6040 223036 24 0 0 0s 97% 0% - 100% 0 0 4 0 0 37 0 51% 19 0 0 19 185 9 239388 0 0 0 2 98% 0% - 100% 0 0 0 0 0 0 0 51% 77 0 0 80 348 906 273880 8 0 0 2 99% 0% - 100% 0 0 3 0 0 0 0 60% 48 0 0 49 174 410 196179 271 0 0 2 98% 35% Tn 100% 0 0 1 0 0 0 0 45% 102 0 0 110 337 337 144387 21222 0 0 2 96% 100% :f 100% 8 0 0 0 0 0 0 71% 143 0 0 149 258 2814 109728 24268 0 0 2 85% 100% :f 86% 0 0 6 0 0 25 0 53% 64 0 0 65 203 1368 236620 14152 0 0 2 98% 100% :f 100% 0 0 1 0 0 0 0 49% 96 0 0 96 319 1317 185552 19748 0 0 2 96% 100% :f 100% 0 0 0 0 0 0 0 35% 109 0 0 117 381 1962 62613 18010 0 0 2 88% 100% :f 100% 5 0 3 0 0 0 0 55% 38 0 0 39 159 1168 239408 1980 0 0 2 97% 100% :f 100% 0 0 1 0 0 0 0 57% 177 0 0 179 308 6692 136568 520 0 0 2 92% 82% : 99% 0 0 2 0 0 37 0 39% 256 0 0 271 403 12647 117284 0 0 0 2 93% 0% - 97% 12 0 3 0 0 0 0 40% 232 0 0 473 808 7506 162368 32 0 0 2 96% 0% - 100% 240 0 1 0 0 0 0 38% 86 0 0 86 246 3413 160092 0 0 0 2 96% 0% - 100% 0 0 0 0 0 0 0 47% 113 0 0 117 314 1738 199516 0 0 0 2 96% 0% - 100% 1 0 3 0 0 0 0 59% 116 0 0 120 476 2751 288048 24 0 0 0s 99% 0% - 100% 0 0 4 0 0 24 0 45% 230 0 0 230 2879 6547 185787 120 0 0 0s 98% 0% - 100% 0 0 0 0 0 0 0 > version NetApp Release 8.1.2P2 7-Mode: Wed Feb 20 19:58:27 PST 2013 I'm aware that deduplication _SHOULD_ be a background process hugging up as much resources as it can but when host i/o is needed, it will be preferred over deduplication. There are no other tasks running at the same time (snapvault, etc.), and the virtual machinse on this system are _VERY_ slow - f.ex. logging in via RDP on one of the virtual machines now took > 1 minute for the login window to appear. This is a FAS2240 with 24x600 SAS 10k and basically my question now is: Is there any way to tell a-sis to really run with lower priority or is this a bug in the OnTap version I'm running? I've used a-sis a lot in the past and never had performance issues like this when it was running... I'm thankful for any ideas how to track down this issue or improve performance while a-sis is running. Thanks, Alexander Griesser System-Administrator ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-320 Telefax: +43-5-0556-500 E-Mail: ag(a)anexia.at<mailto:ag@anexia.at> Web: http://www.anexia.at<http://www.anexia.at/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 _______________________________________________ Toasters mailing list Toasters(a)teaparty.net<mailto:Toasters@teaparty.net> http://www.teaparty.net/mailman/listinfo/toasters _______________________________________________ Toasters mailing list Toasters(a)teaparty.net<mailto:Toasters@teaparty.net> http://www.teaparty.net/mailman/listinfo/toasters -- kind regards, +43-664-4542287 Ing. Peter-Paul Witta +43-1-7189880-0 Chief Technology Officer www.CUBiT.at<http://www.CUBiT.at>

4 5

Thin and Over subscription
by Iluhes 25 Oct '13

25 Oct '13

Dear toasters, Did anyone seen any script that provides an oversubscription rate for a volume. Something like 1. Check for thin lun size allocated size 2. Compare that to "Occupied" size 3. Compare that to volume size And give a oversubscription rate Thanks in advance

1 0

cDOT CIFS setup
by Aaron Lewis 25 Oct '13

25 Oct '13

Has anyone seen the error listed below while setting up CIFS in cDOT? The filer is running 8.2P3. This has been a very straightforward process in the past for me, but with the newness of cDOT, I'm not finding a lot of information when I search. Thanks! netapp::vserver cifs> create -cifs-server "testcifs" -domain " XXXXXX.local" In order to create an Active Directory machine account for the CIFS server, you must supply the name and password of a Windows account with sufficient privileges to add computers to the "CN=Computers" container within the " XXXXXX.local" domain. Enter the user name: XXXXXX Enter the password: Error: CIFS server creation procedure failed [ 27] Loaded the preliminary configuration. [ 62] Created a machine account for the Cifs server in the domain [ 62] SID to name translations of Domain Users and Admins completed successfully **[ 23082] FAILURE: Kerberos password set for ** 'TESTCIFS$(a)XXXXXX.LOCAL' failed with Cannot contact any ** KDC for requested realm [ 23104] Deleted existing account 'CN=TESTCIFS,CN=Computers,DC=xxxxxxx,DC=local' Error: command failed: Failed to create CIFS server TESTCIFS. Reason: Kerberos Error: KDC Unreachable. netapp::vserver cifs> Regards, Aaron

7 7

CIFS Scanner
by Jordan Slingerland 23 Oct '13

23 Oct '13

Does anyone know of a good tool that would scan a cifs share for meta data and possibly put the information in a database that can be queried for Creator Owner Information. For example, the netapp "quota report" command shows a user owns 100GB and 5000 files. I would like to give said user a list of the 5000 files and their sizes. Alternatively, a less sophisticated too that accomplishes the same result that is better than a power shell script that could scan for this information for single user or as needed. In the past I have gathered this information to a text file with a power shell script and it was very messy and certainly less then optimal. --JMS For the fourth consecutive year, Independent Health is the highest ranked health insurance plan in the New York/New Jersey region by J.D. Power and Associates 2013 Member Health Plan StudySM. For the sixth consecutive year, Independent Health was named one of the best companies to work for in New York State. Go to www.independenthealth.com to view careers and available positions. We are an Equal Opportunity Employer. CONFIDENTIALITY NOTICE: This e-mail and its attachments (collectively referred to as "e-mail") may contain confidential information that is privileged and protected from disclosure by Federal and State confidentiality laws, rules or regulations. This e-mail is intended for the designated addressee only. If you are not the designated addressee, you are notified that any disclosure, copying or distribution of this e-mail may be unlawful and may subject you to legal consequences. If you have received this e-mail in error, please contact me immediately by telephone at (716) 631 - 3001 and delete the e-mail from your computer immediately. Thank you for your attention.

3 4

Deswizzling and Reallocate
by Martin 23 Oct '13

23 Oct '13

I've recently destroyed one aggregate and merged the spared disks it into another aggregate on the same Filer. I now need to snapmirror the volumes out of this aggregate into another on another Filer (musical volumes!). I didn't run a volume reallocation on the volumes after I added the disks from the recently destroyed aggregate as I'm going to be snapmirroring the data out of it. I'm not clear on whether I will need to run a volume level reallocate on the volumes once they are in the new aggregate? Will the deswizzle post snapmirror on the destination aggregate optimise the layout on the aggregate? I did a Google for "reallocate deswizzle" and looked on the NOW site but didn't find many hits and none were relevant. I can run a reallocate measure once the volumes are on the new Filer and aggregate but wonder how the volume reallocate and deswizzling are related. -- View this message in context: http://network-appliance-toasters.10978.n7.nabble.com/Deswizzling-and-Reall… Sent from the Network Appliance - Toasters mailing list archive at Nabble.com.

3 2

unexplained read operation
by Alon Zeltser 23 Oct '13

23 Oct '13

Hello toasters I encounter rather strange situation I have a system FAS2040 with 1 aggr of 23x450g raid group ontap version 8.1.3p2 Every write operation being generated on this aggregate automatically caused a read operation When I try to create 100% sequential write operation using tools such as sio_ntap or filersio Instead of the expected 150Mb throughput I get 100MB write and 50MB read On another FAS2040 system using the same commands I get 100% write and no reads As you can see filersio asyncio_active 0 -r 0 64k 0 15g 60 1 /vol/test/testfile CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s 93% 0 0 0 5 26 1 24242 118671 0 0 0s 99% 72% Fn 43% 0 0 5 0 0 24 0 90% 0 0 0 4 4 1 40959 102501 0 0 19 100% 89% Fn 43% 1 0 3 0 0 2 0 86% 0 0 0 3 3 1 23126 125022 0 0 0s 99% 69% : 59% 0 0 3 0 0 2 0 93% 0 0 0 7 1 1 28200 135740 0 0 0s 99% 97% Ff 48% 7 0 0 0 0 0 0 92% 0 0 0 6 15 1 48764 115324 0 0 0s 99% 91% Ff 52% 0 0 6 0 0 14 0 90% 0 0 0 6 16 1 36148 156492 0 0 0s 99% 83% Ff 56% 0 0 6 0 0 14 0 95% 0 0 0 0 4 4 32784 114555 0 0 0s 99% 77% Fs 48% 0 0 0 0 0 0 0 92% 0 0 0 18 90 12 34931 104059 0 0 0s 99% 88% Fn 42% 0 0 18 0 0 79 0 88% 0 0 0 8 3 1 20463 124800 0 0 0s 100% 83% Fn 45% 5 0 3 0 0 2 0 89% 0 0 0 2 2 1 34050 112166 0 0 0s 99% 87% Fn 44% 0 0 2 0 0 1 0 92% 0 0 0 3 7 1 33025 112816 0 0 0s 99% 88% Fn 46% 0 0 3 0 0 5 0 91% 0 0 0 3 3 1 30929 130833 0 0 0s 100% 73% : 50% 0 0 3 0 0 2 0 92% 0 0 0 7 24 2 36484 134260 0 0 0s 100% 98% Ff 50% 0 0 7 0 0 22 0 86% 0 0 0 6 1 1 38608 125140 0 0 0s 99% 90% Ff 44% 5 0 1 0 0 1 0 93% 0 0 0 4 3 1 28748 104544 0 0 0s 100% 75% Ff 43% 0 0 4 0 0 2 0 87% 0 0 0 5 18 1 28128 158608 0 0 0s 100% 90% Fs 55% 0 0 5 0 0 17 0 93% 0 0 0 2 3 34 33115 104980 0 0 0s 99% 85% Fn 41% 0 0 2 0 0 1 32 84% 0 0 0 5 28 7 37185 109425 0 0 0s 100% 78% Fn 48% 0 0 5 0 0 21 0 91% 0 0 0 9 3 1 30224 118322 0 0 0s 99% 97% Fn 47% 7 0 2 0 0 1 0 89% 0 0 0 2 3 4 27649 118335 0 0 0s 99% 88% : 45% 0 0 2 0 0 1 0 CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out 88% 0 0 0 14 57 39 43284 133480 0 0 0s 100% 98% Ff 50% 0 0 14 0 0 52 0 92% 0 0 0 2 6 16 47936 109432 0 0 0s 99% 86% Ff 46% 0 0 2 0 0 1 0 88% 0 0 0 23 86 3 45340 141014 0 0 0s 99% 73% Ff 55% 3 0 20 0 0 82 0 92% 0 0 0 10 5 4 21787 123980 0 0 0s 99% 80% Fn 44% 7 0 3 0 0 1 0 95% 0 0 0 2 4 3 34751 98158 0 0 0s 99% 82% Fn 44% 0 0 2 0 0 1 0 91% 0 0 0 1 3 1 35022 118549 0 0 0s 100% 60% Fn 48% 0 0 1 0 0 0 0 92% 0 0 0 87 3 1 25455 123066 0 0 0s 100% 80% Fn 47% 84 0 3 0 0 1 0 83% 0 0 0 8 32 2 36956 121303 0 0 0s 100% 88% : 46% 0 0 8 0 0 30 0 85% 0 0 0 5 2 1 62064 140088 0 0 0s 100% 92% Ff 59% 5 0 0 0 0 0 0 95% 0 0 0 3 3 1 39352 117364 0 0 0s 99% 71% Ff 48% 0 0 3 0 0 2 0 94% 0 0 0 3 3 1 42008 136943 0 0 0s 99% 94% Fs 53% 0 0 3 0 0 2 0 92% 0 0 0 0 2 7 26147 111286 0 0 0s 99% 58% Fn 44% 0 0 0 0 0 0 0 90% 0 0 0 3 3 1 29567 110665 0 0 20 99% 87% Fn 44% 0 0 3 0 0 1 0 90% 0 0 0 9 4 1 40031 111273 0 0 16 99% 86% Fn 43% 5 0 4 0 0 2 0 84% 0 0 0 0 1 1 36760 105709 0 0 0s 100% 84% Fn 43% 0 0 0 0 0 0 0 92% 0 0 0 3 3 1 31033 137891 0 0 0s 100% 99% : 53% 0 0 3 0 0 2 0 83% 0 0 0 4 6 1 32991 140148 0 0 0s 100% 89% F 51% 0 0 4 0 0 5 0 Any thoughts why this could happens ? Nothing else is running on this netapp except for this test Thanks Alon Z

5 12

Slow copy of a directory full of files via an NFS client across a WAN
by Arnold de Leon 21 Oct '13

21 Oct '13

I have an NFS client (CentOS 5) that is mounting a volume from filer across a WAN with about 20ms of latency (40 ms round trip). We have an application that makes a "backup" copy of a directory of data before modifying it. The directory can contain a lot tiny of files (for example 600 files, taking up less than 12MB). I'm using NFS V3 over TCP (also tried V3 over UDP). The NFS copy (using cp -a) is glacial. Looking at the WireShark packet captures shows the client and the server in a deadly lock step request/reply situation (typically GETATTR and SETATTR). As far as I can tell the client/server are not allowing any windowing/pipelining of requests. Is there a setting I am missing to enable more outstanding requests (on the client or the server)? All the searches I've done for tuning seem to be about fixing issues for large bulk transfers. Is the NFS protocol inherently limited in this way. I expected this kind of behavior in SMB 1 but not with NFS. Thanks. arnold

13 20

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters October 2013