toasters

toasters@lists.teaparty.net

1 participants
13515 discussions

Re: Aggregate Disk Busy 100% with volume IOPS low
by Fletcher Cocquyt 24 Jan '13

24 Jan '13

No NDMP in use for us - at a loss to explain this level of AGGR disk busy with no vol level IO Feels like an internal type operation hitting a bug thanks On Jan 23, 2013, at 10:21 PM, "Klise, Steve" <klises(a)sutterhealth.org> wrote: > A stab but what about ndmp jobs? > > From: Fletcher Cocquyt [mailto:fcocquyt@stanford.edu] > Sent: Wednesday, January 23, 2013 09:27 PM > To: toasters(a)teaparty.net Lists <toasters(a)teaparty.net> > Subject: Aggregate Disk Busy 100% with volume IOPS low > > 3270 cluster, OnTAP 8.1-7mode > > We are investigating a SATA aggregate showing repeated 5am disk 100% busy spikes without its volumes showing any corresponding IOPS spike as reported by Netapp Management Console (NMC). > The 5am disk busy spikes correlate with very high latency on volumes on a different SAS aggregate. These volumes host VMs which then timeout, some needing reboots. > Today when I heard from Netapp support after reviewing my perfstat the engineer reported this is expected since NVRAM buffers are shared btw aggregates. > > But when I dig further into the NMC stats I see the SATA aggregate disk busy actually corresponds to a DROP in IOPS on the 3 volumes hosted on the SATA aggregate - almost like some internal aggregate operations are starving out the external volume ops. > I checked the snapshots (vol and aggr), snap mirror, dedup and none of the usual suspects were running. > > When I look at the NMC throughput graphs and switch on the legend - it shows a 5am READ blocks/sec spike corresponding perfectly to the disk busy. > > Where are these AGGR level READ operations coming from that are missing from the constituent volume IOPS, and in fact seem to be starving out volume level IO? > > I don't see much in the messages log, but will check the rest of the logs for internal type OPS > > thanks for any insight > > > > > >

1 0

Vfiler DR activate
by Iluhes 22 Jan '13

22 Jan '13

Hello Toasters, After running VFILER DR Activate and failing over vFiler to DR site, none of the shares were available. The DC controller was not available in DR bubble, but we thought that local admin users on vfiler should have access to shares. We could not access shares. For certain vFiler DRs when the production side is a domain member, we have a need to have the filer available in a DR test situation in which the domain controller is not available. We have found in testing that in the absence of the domain controller, all share access is denied. Even when we try accessing with local accounts, which on a Windows file server, would have let us access the share in the absence of a domain controller we are denied access. Is that correct behavior? We would get: "There are currently no logon server available to service the logon request" Is there anyway to do vFiler DR where the vFiler is a domain member where one can access the shares in DR without the domain being present--using local accounts only. Thanks!!!

2 1

RE: Vfiler DR activate
by steve klise 22 Jan '13

22 Jan '13

Was there a delta in time between the filer and the clinet? Usually AD has to be within 5 minutes of source and destination for Kerberos. You would want an AD controller to be available, and usually the 1st box up for CIFS authentication... Sent from Windows Mail From: Iluhes Sent: ‎January‎ ‎21‎, ‎2013 ‎6‎:‎06‎ ‎PM To: toasters(a)teaparty.net Subject: Vfiler DR activate Hello Toasters, After running VFILER DR Activate and failing over vFiler to DR site, none of the shares were available. The DC controller was not available in DR bubble, but we thought that local admin users on vfiler should have access to shares. We could not access shares. For certain vFiler DRs when the production side is a domain member, we have a need to have the filer available in a DR test situation in which the domain controller is not available. We have found in testing that in the absence of the domain controller, all share access is denied. Even when we try accessing with local accounts, which on a Windows file server, would have let us access the share in the absence of a domain controller we are denied access. Is that correct behavior? We would get: "There are currently no logon server available to service the logon request" Is there anyway to do vFiler DR where the vFiler is a domain member where one can access the shares in DR without the domain being present--using local accounts only. Thanks!!!

1 0

Re: wafl_cp_slovol_warning_1 with big latency spikes
by Brian Beaulieu 19 Jan '13

19 Jan '13

What about other IO on the aggregate? Any other volume activity (snaps, resize) I feel your pain. I've had one single VM cause IO spikes and thus latency. I had this happen when I moved the VM from an NFS datastore to a VMFS datastore to get more IO potential. Well, that extra IO was too much of a piece of the pie. The other night I had all NFS/CIFS IO completely stop for a couple of minutes during a volume resize. I love NFS+VMWare+NetApp but sometimes I hate it as well. Brian

4 5

Reallocate a Deduplicated Volume?
by Brad Smith 18 Jan '13

18 Jan '13

Looking for help/direction on reallocating deduplicated volumes. Have a FAS2040 running 7.3.6 and adding an additional DS4243 shelf with SAS drives and want to extend current aggregate by adding a new raid group created with the new disk to the aggregate. On this aggregate I have volumes that are deduplicated and have snapshots. I have read several NetApp documents that seem to conflict on running reallocate on the volumes after adding the new disks to the existing aggregate. I have read you should reallocate but not on a volume that is deduplicated. I have also read that if you use the -p switch this will be OK, reallocate start -f -p /vol/VOLUMENAME. Will I have problems if I run reallocate on my deduplicated volumes? Thanks. Brad

6 6

Snapvault between 8.0.2P4 and 8.1.1
by Steven Kreuzer 17 Jan '13

17 Jan '13

Hello- I have two filers that are both running 8.0.2P4 and they are both snap vaulting a volume from one to the other. Rebooting one of the filers requires tons and tons of coordination between several different groups and the other filer I can pretty much reboot whenever I fancy. I would like to upgrade that filer to 8.1.1 so I can take advantage of all the 64 bit migration stuff and also test out the branchcache support. The only thing I am concerned about is that I may lose the ability to snapvault between the two. I recall it was not possible to snapvault from a filer running 8.x to one running 7.x, is it possible to still snap from a filer running 8.0 to one running 8.1? Thanks

2 1

Re: wafl_cp_slovol_warning_1 with big latency spikes
by Brian Beaulieu 17 Jan '13

17 Jan '13

3rd time is the charm. I've attached my sysstat from the other night when NFS/CIFS hung up... is this what you've seen as well? During that issue, FCP was also slow.. had some MPIO failovers happening on our AIX LPARs. But, AIX handles that just fine and at least has an alternate path through the other filer. NFS isn't so lucky. I have a 3250+1TB PAM sitting on deck.. you'd think that the 3240+512GB PAM would be sufficient for what we do. While I do have SATA in use for VMWare, it's not heavy hitting VMs.. it's the dormant stuff, mostly. I'm moving a lot of it, though, to 6xDS4243x600GB-15k shelves ASAP. I'm drinking the PAM kool-aid too but do have some measurable results primarily on our PeopleSoft DB2 databases. I definitely wouldn't bet on SATA+PAM == FC/SAS performance. Brian

3 4

wafl_cp_slovol_warning_1 with big latency spikes
by Fletcher Cocquyt 16 Jan '13

16 Jan '13

resending this without the 80kb chart Yesterday morning one of the heads on our 3270 experienced large NFS latency spikes causing our VMware hosts and their VMs to log storage timeouts. This latency does not correlate to any external metrics like CPU, network, OPS etc. But in the logs do show CP events on the aggregate hosting the VMs: Jan 14 05:27:56 [n04:wafl.cp.slovol:warning]: aggregate aggr2 is holding up the CP. And the EMS log has CP events logged for the duration of the episode - what can we do to prevent these issues? <wafl_cp_toolong_warning_1 total_ms="117825" total_dbufs="32276" clean="4312" v_ino="3" v_bm="29" a_ino="0" a_bm="3428" flush="1209"/> </LR> <LR d="14Jan2013 05:19:38" n="irt-na04" t="1358169578" id="1335304168/148007" p="4" s="Ok" o="wafl_CP_proc" vf="" type="0" seq="633232" > <wafl_cp_slovol_warning_1 voltype="aggregate" volowner="" volname="aggr2" volident="" nt="35" nb="22045" clean="1346852" v_ino="0" v_bm="113" a_ino="0" a_bm="4" flush="0" rgid="2"/> Netapp support wants me to run perfstats, but the issue is not ongoing - things are idle thanks

9 12

Re: BURT 519766 panic on production 3270
by Fletcher Cocquyt 16 Jan '13

16 Jan '13

Dec 25 04:19:26 na03.GoCardinal.EDU Dec 25 12:20:49 [na03:mgr.stack.string:notice]: Panic string: Uncorrectable Machine Check Error at CPU1. MC5 Error: STATUS<0xb200001080200e0f>(Val,UnCor,Enable,PCC,ErrCode(Gen,NTO,Gen,Gen,Gen)); PLX PCI-E switch on IO Exp thanks - yes the fact Netapp is immediately willing to replace most of our HW indicates they know its an issue with our current HW I'd feel better recommending this plan if they could point to the specific HW issue in the current HW and demonstrate how its fixed in newer HW revs Currently those details are not public/forthcoming... On Jan 2, 2013, at 3:02 PM, Doug Siggins <DSiggins(a)ma.maileig.com> wrote: > Fletcher, > What was the panic string? Did you get a core to netapp? Sometimes support is a bit reluctant to investigate further unless you press for a real answer. After 2-3 core dumps with the same type panic string, I start demanding a fix whether it be hardware or software. > > > Here are two forum posts: > > https://forums.netapp.com/thread/33616 > https://forums.netapp.com/thread/35456 > > I had a similar issue on an older filer. It panic'd 2-3 times. Luckily, over time has we prepare to retire the system the load has dropped significantly, and I haven't seen the NMI panic for 6+ months. I had suggested we replace the system immediately and migrate off. > > I guess the rule of thumb is that if you see the panic more than once, you should definitely think about hardware replacements. > > > > > > > > > > From: toasters-bounces(a)teaparty.net [toasters-bounces(a)teaparty.net] on behalf of Fletcher Cocquyt [fcocquyt(a)stanford.edu] > Sent: Wednesday, January 02, 2013 5:21 PM > To: toasters(a)teaparty.net Lists > Cc: netapp-users(a)mailman.stanford.edu > Subject: BURT 519766 panic on production 3270 > > Happy 2013! > > One of our production 3270 heads panic'ed and rebooted 3:30 am Dec 25 - lump of coal ? > > The good news is, when our system panic'ed and rebooted, the failover performed as expected so we had only a 2 second timeout logged on our ESXi hosts, Oracle - no downtime. > > There is scarce public info on this issue and Netapp is recommending options from "do nothing - (its rare and may never happen again)" to "replace motherboards and all cards" > Our 3270 clusters (we have 2 in Active:Standby mode) have been stable since we installed them in Feb 2011. We are on 8.1GA - Netapp support says the issue is independent of OnTAP version. > > Anyone else encountered this issue? > What was your action and outcome? > > thanks, > > Fletcher Cocquyt > Stanford University School of Medicine

8 11

PDU in netapp cabinet mysteriously switched off
by Steve Losen 15 Jan '13

15 Jan '13

Hi Folks, Last Sunday six shelves in a netapp cabinet each lost power to one PSU. The other PSUs still had power, so no downtime. I investigated and the PDU in the cabinet that the PSUs are plugged into was powered off, i.e., the switch was in the "off" position. I flipped the switch and everything went back to normal. The PDU is plugged into its own 240V circuit. The same thing happened to us several months ago, but I can't remember if it was the same PDU. I am very confident that when this happened, no one was anywhere near that cabinet (or even in the room). This was definitely "spontaneous". Has anyone heard of a netapp cabinet PSU switch flipping itself to "off" ? Does it behave like a breaker if there is a surge? I'm trying to remember if the "off" position is down. Could the switch be faulty and get flipped by gravity and vibration? Steve Losen scl(a)virginia.edu phone: 434-924-0640 University of Virginia ITC Unix Support

2 1

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters