toasters

toasters@lists.teaparty.net

1 participants
13528 discussions

wafl_cp_slovol_warning_1 with big latency spikes
by Fletcher Cocquyt 16 Jan '13

16 Jan '13

resending this without the 80kb chart Yesterday morning one of the heads on our 3270 experienced large NFS latency spikes causing our VMware hosts and their VMs to log storage timeouts. This latency does not correlate to any external metrics like CPU, network, OPS etc. But in the logs do show CP events on the aggregate hosting the VMs: Jan 14 05:27:56 [n04:wafl.cp.slovol:warning]: aggregate aggr2 is holding up the CP. And the EMS log has CP events logged for the duration of the episode - what can we do to prevent these issues? <wafl_cp_toolong_warning_1 total_ms="117825" total_dbufs="32276" clean="4312" v_ino="3" v_bm="29" a_ino="0" a_bm="3428" flush="1209"/> </LR> <LR d="14Jan2013 05:19:38" n="irt-na04" t="1358169578" id="1335304168/148007" p="4" s="Ok" o="wafl_CP_proc" vf="" type="0" seq="633232" > <wafl_cp_slovol_warning_1 voltype="aggregate" volowner="" volname="aggr2" volident="" nt="35" nb="22045" clean="1346852" v_ino="0" v_bm="113" a_ino="0" a_bm="4" flush="0" rgid="2"/> Netapp support wants me to run perfstats, but the issue is not ongoing - things are idle thanks

9 12

Re: BURT 519766 panic on production 3270
by Fletcher Cocquyt 16 Jan '13

16 Jan '13

Dec 25 04:19:26 na03.GoCardinal.EDU Dec 25 12:20:49 [na03:mgr.stack.string:notice]: Panic string: Uncorrectable Machine Check Error at CPU1. MC5 Error: STATUS<0xb200001080200e0f>(Val,UnCor,Enable,PCC,ErrCode(Gen,NTO,Gen,Gen,Gen)); PLX PCI-E switch on IO Exp thanks - yes the fact Netapp is immediately willing to replace most of our HW indicates they know its an issue with our current HW I'd feel better recommending this plan if they could point to the specific HW issue in the current HW and demonstrate how its fixed in newer HW revs Currently those details are not public/forthcoming... On Jan 2, 2013, at 3:02 PM, Doug Siggins <DSiggins(a)ma.maileig.com> wrote: > Fletcher, > What was the panic string? Did you get a core to netapp? Sometimes support is a bit reluctant to investigate further unless you press for a real answer. After 2-3 core dumps with the same type panic string, I start demanding a fix whether it be hardware or software. > > > Here are two forum posts: > > https://forums.netapp.com/thread/33616 > https://forums.netapp.com/thread/35456 > > I had a similar issue on an older filer. It panic'd 2-3 times. Luckily, over time has we prepare to retire the system the load has dropped significantly, and I haven't seen the NMI panic for 6+ months. I had suggested we replace the system immediately and migrate off. > > I guess the rule of thumb is that if you see the panic more than once, you should definitely think about hardware replacements. > > > > > > > > > > From: toasters-bounces(a)teaparty.net [toasters-bounces(a)teaparty.net] on behalf of Fletcher Cocquyt [fcocquyt(a)stanford.edu] > Sent: Wednesday, January 02, 2013 5:21 PM > To: toasters(a)teaparty.net Lists > Cc: netapp-users(a)mailman.stanford.edu > Subject: BURT 519766 panic on production 3270 > > Happy 2013! > > One of our production 3270 heads panic'ed and rebooted 3:30 am Dec 25 - lump of coal ? > > The good news is, when our system panic'ed and rebooted, the failover performed as expected so we had only a 2 second timeout logged on our ESXi hosts, Oracle - no downtime. > > There is scarce public info on this issue and Netapp is recommending options from "do nothing - (its rare and may never happen again)" to "replace motherboards and all cards" > Our 3270 clusters (we have 2 in Active:Standby mode) have been stable since we installed them in Feb 2011. We are on 8.1GA - Netapp support says the issue is independent of OnTAP version. > > Anyone else encountered this issue? > What was your action and outcome? > > thanks, > > Fletcher Cocquyt > Stanford University School of Medicine

8 11

PDU in netapp cabinet mysteriously switched off
by Steve Losen 15 Jan '13

15 Jan '13

Hi Folks, Last Sunday six shelves in a netapp cabinet each lost power to one PSU. The other PSUs still had power, so no downtime. I investigated and the PDU in the cabinet that the PSUs are plugged into was powered off, i.e., the switch was in the "off" position. I flipped the switch and everything went back to normal. The PDU is plugged into its own 240V circuit. The same thing happened to us several months ago, but I can't remember if it was the same PDU. I am very confident that when this happened, no one was anywhere near that cabinet (or even in the room). This was definitely "spontaneous". Has anyone heard of a netapp cabinet PSU switch flipping itself to "off" ? Does it behave like a breaker if there is a surge? I'm trying to remember if the "off" position is down. Could the switch be faulty and get flipped by gravity and vibration? Steve Losen scl(a)virginia.edu phone: 434-924-0640 University of Virginia ITC Unix Support

2 1

New Bundled DS4243 Shelves
by Brian Beaulieu 15 Jan '13

15 Jan '13

I received our order of a new FAS3250 along with a 7 shelves. I want to use the shelves on our FAS3240 for now. We'll be doing a head swap in the future. I got 5 of the shelves attached today and then realized they were all owned by the FAS3250 that we got. On top of that, it looks like all disks are in an aggregate. I suspect this because I forced disk ownership of drive 0 on one of the shelves and it imported aggr0 but then renamed it to aggr0(1) and put it offline. I tried the same thing with drive 0 of another shelf and that imported aggr0(2) and put it offline. So, what's the best way to handle this? disk reassign ? My fear is if I import 144 drives using 'disk assign <disk> -s <sysid>-f', I will end up with 144 aggr0(X) offline aggregates and then have to clean all of that up. Once the headswap is done, the FAS3240 along with the other 2 remaining shelves will be going to our DR site and I imagine I'll be doing a fresh install of OnTAP on there since there will be no root vol Thanks! Brian

2 2

When is 1Tb == 1000Gb == 1000^2Mb == 1000^3Kb == 1000^4bytes ?
by Fletcher Cocquyt 13 Jan '13

13 Jan '13

This came up when I was doing some work with our new backup vendor's API, pulling out their backup numbers all reported in bytes and converting to Gb or Tb for our cacti graphs. set bytes=7696581394432 echo $bytes bytes = `echo $bytes"/1024/1024/1024/1024" | bc` Tb 7696581394432 bytes = 7 Tb But the numbers on the vendors dashboard were different. I asked vendor support and the next day they confirmed (as I suspected) they were converting using powers of 10 But what I did not know, was this was on purpose due to the disk manufacturer's convention: http://en.wikipedia.org/wiki/Hard_disk_drive#Units For someone with a computer science background this was surprising to not have uniform definitions of units across media Ram manufacturers don't do this according to the article they use (do Flash or SSD makers?) So if I have full 1Tb RAM drive and try to copy it to a 1Tb HDD, it won't fit - I'll need 9.95% more space due to the different definitions of Tb Made me wonder what the Netapp reporting tools, OpsMgr, System Manager, DFM etc do when reporting the #bytes available on a HDD as Tb do they do binary or decimal conversions? What are some other implications of the unit definition difference? thanks

4 3

CIFS Proxy
by Steven Kreuzer 11 Jan '13

11 Jan '13

Greetings- I have a remote office that is a mix of Windows and Unix clients that have to access a filer far away in our main office. For the unix folks, we have deployed a flexcache and their quality of life has greatly improved. It looks like NetApp doesn't offer any type of flexcache type solution for CIFS so I am look at other options. Are there any flexcache like solutions from a 3rd party that people have deployed and is anyone using a tcp accelerator appliance such as Riverbed to speed to access to a volume via CIFS? At this point, I am open to just about any solution :) Thanks

6 5

Where does the "125L" come fromin diskname 0c.125L72 (when using ATTO FC-to-SAS bridges)
by Filip Sneppe 10 Jan '13

10 Jan '13

Hi, This is perhaps a bit of a silly question, but a customer of ours asked this , and I am not able to provide him with a full answer. On a FAS system with ATTO FC-to-SAS bridges, a disk is identified by a name like this 0c.125L72. On a V-series system, you may encounter disks names such as "fcswitchname:17.126L31". This link: https://library.netapp.com/ecmdocs/ECMP1147535/html/GUID-6F23FBA1-07DC-4CF1… explains the 125L and 126L are device IDs, but can anyone give me a little more information on what they represent, how they are set etc. Thanks in advance, Filip

1 0

Latency reporting
by Peta Thames 04 Jan '13

04 Jan '13

Hi and happy new year to you all! My first conundrum of 2013... Anyone got any ideas on how I get the amount of time in any given time period that overall FCP read/write latency has been above a certain threshold? The threshold is already set in perf advisor and has been for some time. I have the usual free opsmanager tools to hand, plus Balance. I know there are other performance metrics I can and do look at, however for this report I have very explicitly been asked to find something that can be expressed as a percentage, and so far this is the best I can come up with (% of time in a month spent over the threshold). Any other ideas would be welcome though. Thanks, Peta

3 2

BURT 519766 panic on production 3270
by Fletcher Cocquyt 02 Jan '13

02 Jan '13

Happy 2013! One of our production 3270 heads panic'ed and rebooted 3:30 am Dec 25 - lump of coal ? The good news is, when our system panic'ed and rebooted, the failover performed as expected so we had only a 2 second timeout logged on our ESXi hosts, Oracle - no downtime. There is scarce public info on this issue and Netapp is recommending options from "do nothing - (its rare and may never happen again)" to "replace motherboards and all cards" Our 3270 clusters (we have 2 in Active:Standby mode) have been stable since we installed them in Feb 2011. We are on 8.1GA - Netapp support says the issue is independent of OnTAP version. Anyone else encountered this issue? What was your action and outcome? thanks, Fletcher Cocquyt Stanford University School of Medicine

2 1

Aggregate Best Practices
by Ray Van Dolson 28 Dec '12

28 Dec '12

We're revisiting how we set up our aggregates and I want to see how others out there do it. Specifically, what strategies do you use for ensuring certain key applications or environments get the performance they need in a shared environment. Typcially we'll create large aggregates based on a homogeneous disk type. 15K SAS disks in one aggregate, SATA in another. In some cases when it's only a single type of disk, we'd have 60 15K disks in one aggregate and 60 in other (assigned to each controller respectively). The idea here is that more spindles gives us the most performance. However, some applications/workloads are more important than others, and some can be "bullies" impacting the important stuff. Ideally we'd try and keep our OLTP random workloads on one filer and heavy sequential workloads on another (maybe dedicated). We've also been discussing creating multiple, smaller aggregates that we then assign to specific workloads guaranteting those spindles for those workloads. Lower possible maximum performance, but better protection against "bullies"[1]. I also know ONTAP has some I/O QoS options. I'm less inclined to go that direction however. Our workloads tend to be ESX VM's using the filers as NFS datastores. We have the usual budgetary / purchasing cycle constraints, so trying to minimize pain for as long as possible until we can add resources. How do folks out there handle this? Thanks, Ray [1] Controller obviously still is shared.

6 8

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters