toasters February 2013

toasters@lists.teaparty.net

44 participants
30 discussions

Aggregate Disk Busy 100% with volume IOPS low
by Fletcher Cocquyt 01 Apr '13

01 Apr '13

3270 cluster, OnTAP 8.1-7mode We are investigating a SATA aggregate showing repeated 5am disk 100% busy spikes without its volumes showing any corresponding IOPS spike as reported by Netapp Management Console (NMC). The 5am disk busy spikes correlate with very high latency on volumes on a different SAS aggregate. These volumes host VMs which then timeout, some needing reboots. Today when I heard from Netapp support after reviewing my perfstat the engineer reported this is expected since NVRAM buffers are shared btw aggregates. But when I dig further into the NMC stats I see the SATA aggregate disk busy actually corresponds to a DROP in IOPS on the 3 volumes hosted on the SATA aggregate - almost like some internal aggregate operations are starving out the external volume ops. I checked the snapshots (vol and aggr), snap mirror, dedup and none of the usual suspects were running. When I look at the NMC throughput graphs and switch on the legend - it shows a 5am READ blocks/sec spike corresponding perfectly to the disk busy. Where are these AGGR level READ operations coming from that are missing from the constituent volume IOPS, and in fact seem to be starving out volume level IO? I don't see much in the messages log, but will check the rest of the logs for internal type OPS thanks for any insight

13 24

wafl_cp_slovol_warning_1 with big latency spikes
by Fletcher Cocquyt 08 Mar '13

08 Mar '13

Yesterday morning one of the heads on our 3270 experienced large NFS latency spikes causing our VMware hosts and their VMs to log storage timeouts. This latency does not correlate to any external metrics like CPU, network, OPS etc. But in the logs do show CP events on the aggregate hosting the VMs: Jan 14 05:27:56 [n04:wafl.cp.slovol:warning]: aggregate aggr2 is holding up the CP. And the EMS log has CP events logged for the duration of the episode - what can we do to prevent these issues? <wafl_cp_toolong_warning_1 total_ms="117825" total_dbufs="32276" clean="4312" v_ino="3" v_bm="29" a_ino="0" a_bm="3428" flush="1209"/> </LR> <LR d="14Jan2013 05:19:38" n="irt-na04" t="1358169578" id="1335304168/148007" p="4" s="Ok" o="wafl_CP_proc" vf="" type="0" seq="633232" > <wafl_cp_slovol_warning_1 voltype="aggregate" volowner="" volname="aggr2" volident="" nt="35" nb="22045" clean="1346852" v_ino="0" v_bm="113" a_ino="0" a_bm="4" flush="0" rgid="2"/> Netapp support wants me to run perfstats, but the issue is not ongoing - things are idle thanks Fletcher Cocquyt Principal Engineer Information Resources and Technology (IRT) Stanford University School of Medicine Email: fcocquyt(a)stanford.edu Phone: (650) 724-7485

5 4

Re: wafl_cp_slovol_warning_1 with big latency spikes
by Brian Beaulieu 04 Mar '13

04 Mar '13

I had opened a case when this happened and let the TSE know that I'm not the only one having this issue.. asked for it to be escalated and treated as a possible bug. We'll see where it goes. Brian On Thu, Jan 17, 2013 at 4:16 PM, Scott Eno <s.eno(a)icloud.com> wrote: > We're on 8.1.1P1 across the environment. > > 7-mode, of course. > > > On Jan 17, 2013, at 3:35 PM, Brian Beaulieu <brian.beaulieu(a)gmail.com> > wrote: > > Scott, what version of OnTAP are you on? > We're on 8.1.1P1 > > I saw a BURT that sounded related to this but it was apparently fixed by > 8.1.1P1. > > Brian > > > On Thu, Jan 17, 2013 at 2:21 PM, Jeff Mohler <speedtoys.racing(a)gmail.com>wrote: > >> :s is wafl updating special files in the CP process. >> >> >> Going on _that_ long....??? A few seconds of special file updates in a >> CP sure, but that much? >> >> I'd be pretty pushy on getting an answer, id put that in the "its a big >> bug" bucket. That's not normal IO activity in a healthy system. >> >> >> On Thu, Jan 17, 2013 at 8:07 PM, Scott Eno <s.eno(a)me.com> wrote: >> >>> >>> Yes, this is what I see. ":s" and all the other protocols go to "0". >>> >>> There's been some correlation, when this happens, to cleanup of VMware >>> snapshots (not NetApp snaps on the volumes, but VMware snapshots of vm's >>> via vcenter). But it happens other times too. >>> >>> >>> >>> On Jan 17, 2013, at 1:47 PM, Brian Beaulieu <brian.beaulieu(a)gmail.com> >>> wrote: >>> >>> 3rd time is the charm. >>> >>> I've attached my sysstat from the other night when NFS/CIFS hung up... >>> is this what you've seen as well? >>> >>> During that issue, FCP was also slow.. had some MPIO failovers happening >>> on our AIX LPARs. But, AIX handles that just fine and at least has an >>> alternate path through the other filer. NFS isn't so lucky. >>> >>> I have a 3250+1TB PAM sitting on deck.. you'd think that the 3240+512GB >>> PAM would be sufficient for what we do. >>> While I do have SATA in use for VMWare, it's not heavy hitting VMs.. >>> it's the dormant stuff, mostly. >>> I'm moving a lot of it, though, to 6xDS4243x600GB-15k shelves ASAP. >>> >>> I'm drinking the PAM kool-aid too but do have some measurable results >>> primarily on our PeopleSoft DB2 databases. >>> I definitely wouldn't bet on SATA+PAM == FC/SAS performance. >>> >>> Brian >>> <sysstat - Copy.txt>_______________________________________________ >>> >>> Toasters mailing list >>> Toasters(a)teaparty.net >>> http://www.teaparty.net/mailman/listinfo/toasters >>> >>> >>> >>> _______________________________________________ >>> Toasters mailing list >>> Toasters(a)teaparty.net >>> http://www.teaparty.net/mailman/listinfo/toasters >>> >>> >> >> >> -- >> --- >> Gustatus Similis Pullus > > > _______________________________________________ > Toasters mailing list > Toasters(a)teaparty.net > http://www.teaparty.net/mailman/listinfo/toasters > > >

4 3

DS2246 shelves and drives questions.
by Jeff Cleverley 28 Feb '13

28 Feb '13

Greetings, We will be upgrading an existing 6080 cluster to a 6280 cluster. We also have a 6080 cluster we'll be adding disks to also. Both clusters are 7-mode. We'll probably end up at 8.1 on both clusters. My 2 questions stem from the disks and shelves being proposed for the upgrades (ds2246 with 450g SAS). 1. We currently have DS4243 shelves with 600gig SAS. I know I can't mix these in the same stack because of the 3/6gb connection differences. I have a free port on each SAS adapter in our 6080s. Can I put the 6gb shelves on the same adapter as long as they are on their own stack? I'm guessing it may not be a best practice, but we don't have any slots free in the 6080 for more cards. 2. I found a white paper that states the 10K drive performance can be up to 20% slower than the 15K SAS drives for random reads. That seems like a pretty big performance it. It also says mixing the disks in aggregates is not recommended. Other than the smaller footprint and better density, I'm not seeing an advantage over using 600g SAS in the DS4243 shelves. Are there any advantages other than the faster SAS backend to warrant going to these shelves in our environment? Thanks, Jeff -- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611

4 4

clustered ONTAP, junction paths, export policies, NFS mounts ?
by Steve Losen 28 Feb '13

28 Feb '13

Hi Folks, Can anyone point me to some good documentation for the above? I've looked on the Netapp Support site at the man pages and admin guides and they cover the command options, but I haven't found anything explaining how all this stuff works together. I understand that the purpose is to provide a unified namespace, but I would like more information about what is legal/illegal, recommended practices, etc. I there a white paper on this? Steve Losen scl(a)virginia.edu phone: 434-924-0640 University of Virginia ITC Unix Support

5 5

Re: NDMP speed question
by tmac 27 Feb '13

27 Feb '13

For everyone... http://www.symantec.com/business/support/index?page=content&id=TECH51967 Hope it helps. By the way, looks like up to 256 is ok now. --tmac *Tim McCarthy* *Principal Consultant* Clustered ONTAP Clustered ONTAP NCDA ID: XK7R3GEKC1QQ2LVD RHCE5 805007643429572 NCSIE ID: C14QPHE21FR4YWD4 Expires: 08 November 2014 Expires w/release of RHEL7 Expires: 08 November 2014 On Wed, Feb 27, 2013 at 3:35 AM, Jayanathan, David <djayan(a)qualcomm.com>wrote: > Hi Tim,**** > > ** ** > > This mail from you caught my eye because our backup environment heavily > relies on NDMP and we are always looking for ways to improve it. Recently > our backup team has started to change their architecture to do 3-way NDMP > over 10GbE to a master server which has drives attached to it via tape SAN > and has multiplexing/pooling enabled.**** > > ** ** > > Were you able to dig up what specific environment variable you were > thinking of to change the backup blocking factor?**** > > ** ** > > Thanks in advance J**** > > ** ** > > -David**** > > ** ** > > *From:* toasters-bounces(a)teaparty.net [mailto: > toasters-bounces(a)teaparty.net] *On Behalf Of *tmac > *Sent:* Wednesday, February 20, 2013 11:53 AM > *To:* Klise, Steve > > *Cc:* toasters(a)teaparty.net > *Subject:* Re: NDMP speed question**** > > ** ** > > You know, another way to speed up backup is to change the backup blocking > factor.**** > > ** ** > > I think the default is 63 and if you are going over the network, it is > certainly feasible to make it 126.**** > > ** ** > > If I recall, there is an environment variable that can be set/modified to > change the blocking factor.**** > > > **** > > --tmac**** > > ** ** > > *Tim McCarthy***** > > *Principal Consultant***** > > ** ** > > [image: Image removed by sender.] [image: Image removed by sender.] > [image: Image removed by sender.]**** > > ** ** > > Clustered ONTAP > Clustered ONTAP**** > > NCDA ID: XK7R3GEKC1QQ2LVD RHCE5 805007643429572 NCSIE ID: > C14QPHE21FR4YWD4**** > > Expires: 08 November 2014 Expires w/release of RHEL7 > Expires: 08 November 2014**** > > ** ** > > On Wed, Feb 20, 2013 at 1:38 PM, Klise, Steve <klises(a)sutterhealth.org> > wrote:**** > > Just a lame question here, but how many and type of spindles are behind > the volume you are dumping? If you only have a handful of drives, could be > the culprit.**** > > > -----Original Message----- > From: toasters-bounces(a)teaparty.net [mailto:toasters-bounces@teaparty.net] > On Behalf Of Scott Eno > Sent: Wednesday, February 20, 2013 10:36 AM > To: Patrick Giagnocavo > Cc: toasters(a)teaparty.net > Subject: Re: NDMP speed question > > **** > > Likely a slight boost, but I am at the mercy of the network guys. I do > what they tell me. > > > On Feb 20, 2013, at 1:17 PM, Patrick Giagnocavo <xemacs5(a)gmail.com> wrote: > > > Stupid question, but isn't 10Gbe a lot faster with e.g. 9000 byte MTU? > I thought I saw in the original post, a 1500 byte MTU: > > > > > > From original post: > > > > e1a: flags=0x5f4e867<UP,BROADCAST, > > RUNNING,MULTICAST,TCPCKSUM,NOWINS> mtu > > 1500 > > > > if the 1500 byte MTU can be changed to 9000 without interrupting service > (depends on switch and other configuration, most likely), you might see a > speed bump. > > > > Cheers > > > > Patrick > > _______________________________________________ > > Toasters mailing list > > Toasters(a)teaparty.net > > http://www.teaparty.net/mailman/listinfo/toasters > > > _______________________________________________ > Toasters mailing list > Toasters(a)teaparty.net > http://www.teaparty.net/mailman/listinfo/toasters > > _______________________________________________ > Toasters mailing list > Toasters(a)teaparty.net > http://www.teaparty.net/mailman/listinfo/toasters**** > > ** ** >

2 2

absurd numbers in sysstat
by Scott Eno 26 Feb '13

26 Feb '13

Ever come across this? That's a whole lot of reads and writes in one second! All other numbers seem accurate, but those Disk read and kB/s write numbers are absurd. CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out 56% 11041 843 0 11926 48613 42164 571755764 5350017024 0 0 1 94% 100% :v 100% 12 0 30 0 0 655 647 66% 5655 943 0 6612 38929 30760 39402 62888 0 0 1 98% 97% Hs 100% 3 0 11 0 0 53 375 52% 4508 419 0 4992 37034 23069 36347 92920 0 0 1 97% 100% :f 100% 19 0 46 0 0 842 800 CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out 33% 1716 1830 0 3805 5017 34051 26328 43388 0 0 53s 96% 100% :f 24% 0 0 259 0 0 36 7332 42% 9556 1372 0 11483 10188 18549 21440 23824 0 0 53s 96% 100% :v 35% 7 0 548 0 0 307 8364 40% 3067 1878 0 6167 11743 38513 36296 68 0 0 53s 95% 5% : 47% 3 0 1219 0 0 803 14610 35% 1764 1664 0 4289 16814 38170 48450 0 0 0 57s 94% 0% - 59% 6 0 855 0 0 392 11342 35% 3068 931 0 4669 15851 52406 6530898751 4766869982 0 0 55s 95% 0% - 58% 12 0 658 0 0 17 9101 27% 3008 928 0 4263 14336 54182 60996 0 0 0 55s 94% 0% - 64% 3 0 324 0 0 0 9834 53% 4781 3550 0 8356 37070 56790 57780 15816 0 0 1s 96% 40% : 93% 0 0 25 0 0 135 541 72% 6323 1761 0 8095 42569 22303 75804 100132 0 0 0s 96% 76% Hf 91% 0 0 11 0 0 163 16 56% 11134 3358 0 14525 35197 22982 6598065628 4808667416 0 0 0s 93% 100% :f 73% 20 0 13 0 0 70 0 79% 10695 2746 0 13449 112103 17847 74439 96096 0 0 0s 96% 78% Hq 89% 3 0 5 0 0 17 0 84% 9690 2643 0 12438 91183 26266 82004 162335 0 0 42s 96% 100% Hn 100% 11 0 94 0 0 979 726 77% 10334 5223 0 15659 92746 28877 51385 107409 0 0 42s 96% 99% : 100% 88 0 14 0 0 7 74 CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out 56% 7776 5374 0 13736 12413 20623 40795 70765 0 0 1 97% 100% :f 46% 7 0 579 0 0 258 4419 48% 9344 3753 0 13575 12647 50606 37204 35308 0 0 1 97% 61% : 46% 0 0 478 0 0 10 4063 38% 9098 2426 0 12015 11441 29765 17704 0 0 0 1 97% 0% - 74% 0 0 491 0 0 279 4137 48% 10616 2339 0 13448 13008 41186 6618900600 4809364036 0 0 1 97% 0% - 68% 15 0 478 0 0 49 3924 50% 13892 1838 0 15995 12338 50661 41572 24 0 0 1 97% 0% - 76% 3 0 262 0 0 123 2077

2 2

Netapp monitoring - OOIDs in 1.3.6.1.4.1.789.1.5 (filesys).
by Rafał Radecki 22 Feb '13

22 Feb '13

Hi All. I am using NetApp Release 8.0.2P4 7-Mode in active/active mode. I am monitoring it with snmp and Cacti. Yesterday I've performed takeover, sp reboot and giveback. During this process OIDS from 1.3.6.1.4.1.789.1.5 have changed for volumes. I have no list from before the change so I cannot compare but I see in the graphs that volumes are available under different OIDs. Have someone had such problems when restarting netapp/performing takeover giveback? Best regards, Rafal Radecki.

4 3

Adding a vlan but preserving base interface's IP
by Ray Van Dolson 22 Feb '13

22 Feb '13

ONTAP 7.3.6. This is a FAS2050C (IBM N3600 A20) that has a SAS expansion card in each controller. As such, we're limited to only the two onboard GigE ports. e0a and e0b both go to different physical switches and suck in two different networks, so we can't vif 'em together easily. I have a need to introduce a vlan interface to e0a and am wondering if it's possible to do so non-disruptively and preserve access to the e0a via the untagged traffic on the network. Both interfaces are configured with 'nfo'. I've found conflicting information on this.... some say you can mix untagged on your e0a interface and tagged traffic on e0a-vlan## just fine. Others say it's not doable and I'll need to get my network guys to tag all the traffic and move my e0a to a vlan interface. Not sure I'm seeing a way to do this without some downtime... even if I do a cluster failover, I don't see how a failback would work if I've mucked with the interfaces. Thanks, Ray

2 1

NDMP speed question
by Christopher S Eno 21 Feb '13

21 Feb '13

Hi, In testing NDMP dumps over 10GbE, I am seeing it max out at 1.5Gb/s. Some here are thinking that NDMP should be using more of that pipe. Anyone know about, or have numbers of NDMP transfer rates over 10GbE? Any caps or thresholds in OnTap that keep NDMP from gobbling up resources on the controller? I have run tests over the 10GbE connection from controller to controller, 10k drives to 10k drives, sata drives to sata drives, and always 1.5Gb/s is the ceiling. FTP'ing data from server to server across the same 10GbE network shows rates of 2.5Gb/s or higher. My NDMP options: ndmpd.access all ndmpd.authtype challenge ndmpd.connectlog.enabled off ndmpd.data_port_range all ndmpd.enable on ndmpd.fh_node_retry_interval 250 ndmpd.ignore_ctime.enabled off ndmpd.maxversion 4 ndmpd.offset_map.enable on ndmpd.password_length 16 ndmpd.preferred_interface e1a (value might be overwritten in takeover) ndmpd.tcpnodelay.enable on ndmpd.tcpwinsize 65534 IF e1a configuration: e1a: flags=0x5f4e867<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM,NOWINS> mtu 1500 inet x.x.x.x netmask 0xffffff00 broadcast x.x.x.x partner e1a (not in use) ether 00:07:43:08:98:ae (auto-10g_sr-fd-up) flowcontrol none

7 14

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters February 2013