toasters

toasters@lists.teaparty.net

13513 discussions

unexplained read operation
by Alon Zeltser 23 Oct '13

23 Oct '13

Hello toasters I encounter rather strange situation I have a system FAS2040 with 1 aggr of 23x450g raid group ontap version 8.1.3p2 Every write operation being generated on this aggregate automatically caused a read operation When I try to create 100% sequential write operation using tools such as sio_ntap or filersio Instead of the expected 150Mb throughput I get 100MB write and 50MB read On another FAS2040 system using the same commands I get 100% write and no reads As you can see filersio asyncio_active 0 -r 0 64k 0 15g 60 1 /vol/test/testfile CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s 93% 0 0 0 5 26 1 24242 118671 0 0 0s 99% 72% Fn 43% 0 0 5 0 0 24 0 90% 0 0 0 4 4 1 40959 102501 0 0 19 100% 89% Fn 43% 1 0 3 0 0 2 0 86% 0 0 0 3 3 1 23126 125022 0 0 0s 99% 69% : 59% 0 0 3 0 0 2 0 93% 0 0 0 7 1 1 28200 135740 0 0 0s 99% 97% Ff 48% 7 0 0 0 0 0 0 92% 0 0 0 6 15 1 48764 115324 0 0 0s 99% 91% Ff 52% 0 0 6 0 0 14 0 90% 0 0 0 6 16 1 36148 156492 0 0 0s 99% 83% Ff 56% 0 0 6 0 0 14 0 95% 0 0 0 0 4 4 32784 114555 0 0 0s 99% 77% Fs 48% 0 0 0 0 0 0 0 92% 0 0 0 18 90 12 34931 104059 0 0 0s 99% 88% Fn 42% 0 0 18 0 0 79 0 88% 0 0 0 8 3 1 20463 124800 0 0 0s 100% 83% Fn 45% 5 0 3 0 0 2 0 89% 0 0 0 2 2 1 34050 112166 0 0 0s 99% 87% Fn 44% 0 0 2 0 0 1 0 92% 0 0 0 3 7 1 33025 112816 0 0 0s 99% 88% Fn 46% 0 0 3 0 0 5 0 91% 0 0 0 3 3 1 30929 130833 0 0 0s 100% 73% : 50% 0 0 3 0 0 2 0 92% 0 0 0 7 24 2 36484 134260 0 0 0s 100% 98% Ff 50% 0 0 7 0 0 22 0 86% 0 0 0 6 1 1 38608 125140 0 0 0s 99% 90% Ff 44% 5 0 1 0 0 1 0 93% 0 0 0 4 3 1 28748 104544 0 0 0s 100% 75% Ff 43% 0 0 4 0 0 2 0 87% 0 0 0 5 18 1 28128 158608 0 0 0s 100% 90% Fs 55% 0 0 5 0 0 17 0 93% 0 0 0 2 3 34 33115 104980 0 0 0s 99% 85% Fn 41% 0 0 2 0 0 1 32 84% 0 0 0 5 28 7 37185 109425 0 0 0s 100% 78% Fn 48% 0 0 5 0 0 21 0 91% 0 0 0 9 3 1 30224 118322 0 0 0s 99% 97% Fn 47% 7 0 2 0 0 1 0 89% 0 0 0 2 3 4 27649 118335 0 0 0s 99% 88% : 45% 0 0 2 0 0 1 0 CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out 88% 0 0 0 14 57 39 43284 133480 0 0 0s 100% 98% Ff 50% 0 0 14 0 0 52 0 92% 0 0 0 2 6 16 47936 109432 0 0 0s 99% 86% Ff 46% 0 0 2 0 0 1 0 88% 0 0 0 23 86 3 45340 141014 0 0 0s 99% 73% Ff 55% 3 0 20 0 0 82 0 92% 0 0 0 10 5 4 21787 123980 0 0 0s 99% 80% Fn 44% 7 0 3 0 0 1 0 95% 0 0 0 2 4 3 34751 98158 0 0 0s 99% 82% Fn 44% 0 0 2 0 0 1 0 91% 0 0 0 1 3 1 35022 118549 0 0 0s 100% 60% Fn 48% 0 0 1 0 0 0 0 92% 0 0 0 87 3 1 25455 123066 0 0 0s 100% 80% Fn 47% 84 0 3 0 0 1 0 83% 0 0 0 8 32 2 36956 121303 0 0 0s 100% 88% : 46% 0 0 8 0 0 30 0 85% 0 0 0 5 2 1 62064 140088 0 0 0s 100% 92% Ff 59% 5 0 0 0 0 0 0 95% 0 0 0 3 3 1 39352 117364 0 0 0s 99% 71% Ff 48% 0 0 3 0 0 2 0 94% 0 0 0 3 3 1 42008 136943 0 0 0s 99% 94% Fs 53% 0 0 3 0 0 2 0 92% 0 0 0 0 2 7 26147 111286 0 0 0s 99% 58% Fn 44% 0 0 0 0 0 0 0 90% 0 0 0 3 3 1 29567 110665 0 0 20 99% 87% Fn 44% 0 0 3 0 0 1 0 90% 0 0 0 9 4 1 40031 111273 0 0 16 99% 86% Fn 43% 5 0 4 0 0 2 0 84% 0 0 0 0 1 1 36760 105709 0 0 0s 100% 84% Fn 43% 0 0 0 0 0 0 0 92% 0 0 0 3 3 1 31033 137891 0 0 0s 100% 99% : 53% 0 0 3 0 0 2 0 83% 0 0 0 4 6 1 32991 140148 0 0 0s 100% 89% F 51% 0 0 4 0 0 5 0 Any thoughts why this could happens ? Nothing else is running on this netapp except for this test Thanks Alon Z

5 12

Slow copy of a directory full of files via an NFS client across a WAN
by Arnold de Leon 21 Oct '13

21 Oct '13

I have an NFS client (CentOS 5) that is mounting a volume from filer across a WAN with about 20ms of latency (40 ms round trip). We have an application that makes a "backup" copy of a directory of data before modifying it. The directory can contain a lot tiny of files (for example 600 files, taking up less than 12MB). I'm using NFS V3 over TCP (also tried V3 over UDP). The NFS copy (using cp -a) is glacial. Looking at the WireShark packet captures shows the client and the server in a deadly lock step request/reply situation (typically GETATTR and SETATTR). As far as I can tell the client/server are not allowing any windowing/pipelining of requests. Is there a setting I am missing to enable more outstanding requests (on the client or the server)? All the searches I've done for tuning seem to be about fixing issues for large bulk transfers. Is the NFS protocol inherently limited in this way. I expected this kind of behavior in SMB 1 but not with NFS. Thanks. arnold

13 20

AW: Dedup killing performance (again)
by Alexander Griesser 18 Oct '13

18 Oct '13

Yah, just found it while googling I guess, must be bug id #657692 :-/ Anyhow, I'm now stuck with this: > sis status Path State Status Progress /vol/CIFS_SHARES Enabled Active Fingerprint is being recovered from aggregate (6% complete) It's doing that since quite some time now, although it now actually jumped from 0% to 6%, so it at least is still alive. This bug is first fixed in 8.1.2P4 as it seems, so I might need to upgrade ASAP; for now, I'll check if it at least finishes this process now, my volume is filling up quickly, but there's still some aggregate space left so I'll just increase the volume size before it goes full. I'll go above the magic 80% fill of an aggregate then, but better that then having the volume go full. I'll see how it goes and if it troubles me too much, I hope there is a way to abort sis in that state. Alexander Griesser System-Administrator ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-320 Telefax: +43-5-0556-500 E-Mail: ag(a)anexia.at<mailto:ag@anexia.at> Web: http://www.anexia.at<http://www.anexia.at/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 Von: Sebastian Goetze [mailto:spgoetze@gmail.com] Gesendet: Samstag, 19. Oktober 2013 00:16 An: Alexander Griesser Betreff: Re: Dedup killing performance (again) Isn't that the Dedupe bug again, that was fixed in 8.1.3? (since like 8.1.2P3 or P4)... There was a thread about it some time ago... Anyway, I'm off to bed, just wanted to point you in the right direction. BFN On 10/18/2013 11:52 PM, Alexander Griesser wrote: Hi there, I am just trying to dedupe some volumes on one of our filers and the first volume (2.4TB data, CIFS share) deduplicates since about 4 hours now. Sysstat -x 1 shows: CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out 58% 115 0 0 116 2239 3635 281593 32 0 0 0s 98% 0% - 100% 0 0 1 0 0 0 0 63% 162 0 0 169 1682 4149 310829 0 0 0 0s 99% 0% - 100% 7 0 0 0 0 0 0 50% 381 0 0 384 11793 11325 182100 0 0 0 0s 96% 0% - 100% 0 0 3 0 0 0 0 55% 412 0 0 416 6487 6040 223036 24 0 0 0s 97% 0% - 100% 0 0 4 0 0 37 0 51% 19 0 0 19 185 9 239388 0 0 0 2 98% 0% - 100% 0 0 0 0 0 0 0 51% 77 0 0 80 348 906 273880 8 0 0 2 99% 0% - 100% 0 0 3 0 0 0 0 60% 48 0 0 49 174 410 196179 271 0 0 2 98% 35% Tn 100% 0 0 1 0 0 0 0 45% 102 0 0 110 337 337 144387 21222 0 0 2 96% 100% :f 100% 8 0 0 0 0 0 0 71% 143 0 0 149 258 2814 109728 24268 0 0 2 85% 100% :f 86% 0 0 6 0 0 25 0 53% 64 0 0 65 203 1368 236620 14152 0 0 2 98% 100% :f 100% 0 0 1 0 0 0 0 49% 96 0 0 96 319 1317 185552 19748 0 0 2 96% 100% :f 100% 0 0 0 0 0 0 0 35% 109 0 0 117 381 1962 62613 18010 0 0 2 88% 100% :f 100% 5 0 3 0 0 0 0 55% 38 0 0 39 159 1168 239408 1980 0 0 2 97% 100% :f 100% 0 0 1 0 0 0 0 57% 177 0 0 179 308 6692 136568 520 0 0 2 92% 82% : 99% 0 0 2 0 0 37 0 39% 256 0 0 271 403 12647 117284 0 0 0 2 93% 0% - 97% 12 0 3 0 0 0 0 40% 232 0 0 473 808 7506 162368 32 0 0 2 96% 0% - 100% 240 0 1 0 0 0 0 38% 86 0 0 86 246 3413 160092 0 0 0 2 96% 0% - 100% 0 0 0 0 0 0 0 47% 113 0 0 117 314 1738 199516 0 0 0 2 96% 0% - 100% 1 0 3 0 0 0 0 59% 116 0 0 120 476 2751 288048 24 0 0 0s 99% 0% - 100% 0 0 4 0 0 24 0 45% 230 0 0 230 2879 6547 185787 120 0 0 0s 98% 0% - 100% 0 0 0 0 0 0 0 > version NetApp Release 8.1.2P2 7-Mode: Wed Feb 20 19:58:27 PST 2013 I'm aware that deduplication _SHOULD_ be a background process hugging up as much resources as it can but when host i/o is needed, it will be preferred over deduplication. There are no other tasks running at the same time (snapvault, etc.), and the virtual machinse on this system are _VERY_ slow - f.ex. logging in via RDP on one of the virtual machines now took > 1 minute for the login window to appear. This is a FAS2240 with 24x600 SAS 10k and basically my question now is: Is there any way to tell a-sis to really run with lower priority or is this a bug in the OnTap version I'm running? I've used a-sis a lot in the past and never had performance issues like this when it was running... I'm thankful for any ideas how to track down this issue or improve performance while a-sis is running. Thanks, Alexander Griesser System-Administrator ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-320 Telefax: +43-5-0556-500 E-Mail: ag(a)anexia.at<mailto:ag@anexia.at> Web: http://www.anexia.at<http://www.anexia.at/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 _______________________________________________ Toasters mailing list Toasters(a)teaparty.net<mailto:Toasters@teaparty.net> http://www.teaparty.net/mailman/listinfo/toasters

1 0

Dedup killing performance (again)
by Alexander Griesser 18 Oct '13

18 Oct '13

Hi there, I am just trying to dedupe some volumes on one of our filers and the first volume (2.4TB data, CIFS share) deduplicates since about 4 hours now. Sysstat -x 1 shows: CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out 58% 115 0 0 116 2239 3635 281593 32 0 0 0s 98% 0% - 100% 0 0 1 0 0 0 0 63% 162 0 0 169 1682 4149 310829 0 0 0 0s 99% 0% - 100% 7 0 0 0 0 0 0 50% 381 0 0 384 11793 11325 182100 0 0 0 0s 96% 0% - 100% 0 0 3 0 0 0 0 55% 412 0 0 416 6487 6040 223036 24 0 0 0s 97% 0% - 100% 0 0 4 0 0 37 0 51% 19 0 0 19 185 9 239388 0 0 0 2 98% 0% - 100% 0 0 0 0 0 0 0 51% 77 0 0 80 348 906 273880 8 0 0 2 99% 0% - 100% 0 0 3 0 0 0 0 60% 48 0 0 49 174 410 196179 271 0 0 2 98% 35% Tn 100% 0 0 1 0 0 0 0 45% 102 0 0 110 337 337 144387 21222 0 0 2 96% 100% :f 100% 8 0 0 0 0 0 0 71% 143 0 0 149 258 2814 109728 24268 0 0 2 85% 100% :f 86% 0 0 6 0 0 25 0 53% 64 0 0 65 203 1368 236620 14152 0 0 2 98% 100% :f 100% 0 0 1 0 0 0 0 49% 96 0 0 96 319 1317 185552 19748 0 0 2 96% 100% :f 100% 0 0 0 0 0 0 0 35% 109 0 0 117 381 1962 62613 18010 0 0 2 88% 100% :f 100% 5 0 3 0 0 0 0 55% 38 0 0 39 159 1168 239408 1980 0 0 2 97% 100% :f 100% 0 0 1 0 0 0 0 57% 177 0 0 179 308 6692 136568 520 0 0 2 92% 82% : 99% 0 0 2 0 0 37 0 39% 256 0 0 271 403 12647 117284 0 0 0 2 93% 0% - 97% 12 0 3 0 0 0 0 40% 232 0 0 473 808 7506 162368 32 0 0 2 96% 0% - 100% 240 0 1 0 0 0 0 38% 86 0 0 86 246 3413 160092 0 0 0 2 96% 0% - 100% 0 0 0 0 0 0 0 47% 113 0 0 117 314 1738 199516 0 0 0 2 96% 0% - 100% 1 0 3 0 0 0 0 59% 116 0 0 120 476 2751 288048 24 0 0 0s 99% 0% - 100% 0 0 4 0 0 24 0 45% 230 0 0 230 2879 6547 185787 120 0 0 0s 98% 0% - 100% 0 0 0 0 0 0 0 > version NetApp Release 8.1.2P2 7-Mode: Wed Feb 20 19:58:27 PST 2013 I'm aware that deduplication _SHOULD_ be a background process hugging up as much resources as it can but when host i/o is needed, it will be preferred over deduplication. There are no other tasks running at the same time (snapvault, etc.), and the virtual machinse on this system are _VERY_ slow - f.ex. logging in via RDP on one of the virtual machines now took > 1 minute for the login window to appear. This is a FAS2240 with 24x600 SAS 10k and basically my question now is: Is there any way to tell a-sis to really run with lower priority or is this a bug in the OnTap version I'm running? I've used a-sis a lot in the past and never had performance issues like this when it was running... I'm thankful for any ideas how to track down this issue or improve performance while a-sis is running. Thanks, Alexander Griesser System-Administrator ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-320 Telefax: +43-5-0556-500 E-Mail: ag(a)anexia.at<mailto:ag@anexia.at> Web: http://www.anexia.at<http://www.anexia.at/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

1 0

Aftermarket Hardware Support?
by Randy Rue 17 Oct '13

17 Oct '13

Hello All, This is not a NetApp question but a general storage question. We have an older system with two years left in it maximum. Its support contract is coming up. We're considering going with Sherlock Services for hardware support. There's potentially a savings over the next two years that significant enough to warrant the risk. Anyone gone this route? Anyone familiar with this provider? Hope to hear from you, Randy

2 2

Cluster mode root volume recovery
by Mike Thompson 17 Oct '13

17 Oct '13

Hey all, I've got a 8.0.2 c-mode cluster that recently had a single node joined to it, and a few empty aggregates created on it. We had a extended power outtage that required a lot of gear in the data center to get shut down, and since this node in the cluster didn't have any live data or VIFs on it, it got shut down. A few days later, and we are now powering it up, but get this upon login: "The contents of the root volume may have changed and the local management databases may be out of sync with the replicated databases due to corruption of NVLOG data during takeover. This node is not fully operational. Contact support personnel for the root volume recovery procedures." The node comes up fine, can see all it's aggregates, and the other nodes in the cluster can see it via the cluster network, but the node is indeed not fully functional and part of the cluster again. It's aggregates and other info are not visible from the other nodes in the cluster. Did a wafl_check of the root aggr and vol0 and that came back clean. I seem to recall having been through this before in the past, but can't find anything in my notes. This particular cluster is not under support, due to some genius decisions by management, so I'm on my own with this. There are a few empty aggregates on this node, no volumes other than the root vol. Maybe I can force unjoin it from the cluster and rebuild it? Would rather not try to do that. If there is a way to sync up the dbs on the rootvol so it will come back into the cluster, that would be ideal. Any ideas?

6 12

Script to analizyse over subscription rate
by Iluhes 14 Oct '13

14 Oct '13

Dear toasters, Did anyone seen any script that provides an oversubscription rate for a volume. Something like 1. Check for thin lun size allocated size 2. Compare that to "Occupied" size 3. Compare that to volume size And give a oversubscription rate Thanks in advance

1 0

Options reduce support renewal
by Fletcher Cocquyt 10 Oct '13

10 Oct '13

We have two 3270 clusters (active/standby) coming off 3 yr warranty support (10 trays of DS4243 + 5 trays of MK14 each) Our software ELA is also expiring - resulting in the Netapp support renewal quote being over 2 times last year's total (and budget) - they want ~$160K I'm looking for advice on support options. I've already got a quote from Zeriva for the HW support (we just need the parts, we can handle the replacement ourselves) What other creative ideas might help lower the support renewal? thanks, Fletcher

2 1

Re: Options reduce support renewal
by Mark Waterhouse 10 Oct '13

10 Oct '13

This probably isnt what you want to hear....but we have generally found that replacing the hardware works out cheaper than renewing support. This would give you a great excuse to move to Cluster Mode as well From: Fletcher Cocquyt Sent: Thursday, October 10, 2013 6:19 AM To: mailto:toasters@teaparty.net Subject: Options reduce support renewal We have two 3270 clusters (active/standby) coming off 3 yr warranty support (10 trays of DS4243 + 5 trays of MK14 each) Our software ELA is also expiring - resulting in the Netapp support renewal quote being over 2 times last year's total (and budget) - they want ~$160K I'm looking for advice on support options. I've already got a quote from Zeriva for the HW support (we just need the parts, we can handle the replacement ourselves) What other creative ideas might help lower the support renewal? thanks, Fletcher -- To report this message as spam, click here This message has been scanned for viruses and dangerous content by DFK Systems Limited, and is believed to be clean. ------------------------------------------------------------------------------ _______________________________________________ Toasters mailing list Toasters(a)teaparty.net http://www.teaparty.net/mailman/listinfo/toasters -- This message has been scanned for viruses and dangerous content by DFK Systems Limited, and is believed to be clean. To report this message as spam, please click on this link - https://secure.dfk-systems.com/quarantine/spam.php?srv=m2&id=r9ACE2Q4000933

3 2

Alternatives to DFM/Oncommand Core
by Martin 30 Sep '13

30 Sep '13

We've recently been told that vFiler DR is not currently support with Oncommand Core. As cDOT doesn't have the concept of vFilers and from what I have read 5.2 for 7-mode is the last release of Oncommand Core I doubt this functionality will ever be added. We currently rely on vFiler DR to allow fast DR recovery but can't use Oncommand Core due to this limitation. I'm looking at alternatives to Oncommand Core. From past experience there is: Solarwinds Logicmonitor Various Nagios based solutions Does anyone have any other suggestions for monitoring Netapp Filers? Thanks Martin -- View this message in context: http://network-appliance-toasters.10978.n7.nabble.com/Alternatives-to-DFM-O… Sent from the Network Appliance - Toasters mailing list archive at Nabble.com.

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters