toasters May 2012

toasters@lists.teaparty.net

53 participants
34 discussions

Diagnosing outage on FAS6040 running 8.0.1 7-mode
by Steve Losen 25 May '12

25 May '12

Hello Toasters, We had an incident where a FAS6040 running 8.0.1 gave very slow response to a variety of requests. This resulted in a bunch of Linux VMs experiencing failed disk I/O requests, which resulted in corrupted local Linux filesystems. One Linux VM logged that it waited 180 sec. for a disk I/O to complete. The filer did not reboot, and recovered on its own, but we are very interested in figuring out what happened and if we can avoid it (known bug?, should we upgrade ONTAP?) This 6040 does a variety of jobs -- NFS server for a Communigate Pro email system, NFS volumes for VMWARE, and it even has a few FC SAN LUNs used by sharepoint. In general it does not appear to be overloaded. It's been up for over 300 days with no problems. The CF partner filer experienced no problems at that time and it also holds VMWARE volumes and mail volumes. At the time of the outage the /etc/messages file indicated a slow NFS response (93 sec) to one of the Communigate servers. It also indicated that a FC LUN was reset by the sharepoint server. I'm guessing due to delayed response. The mail and sharepoint volumes are in the same aggregate. I see three resets for the LUN in /etc/messages. I looked in /etc/log/ems at about the time of the outage (Tue May 22 17:30 EDT) I see that a raid scrub of a raid group in the mail/sharepoint aggregate completed with no errors. I also see wafl_cp_toolong_warning_1 and wafl_cp_slovol_warning_1 for a different aggregate (which contains VMs). I see several of these, which I presume are caused by slow completion of CPs. I don't know if these caused the problem or were caused by the problem. Anyone have any suggestions for further investigation or diagnosis? Any other logs to look at? Everything is fine now and has been running fine since the outage. Thanks, Steve Losen scl(a)virginia.edu phone: 434-924-0640 University of Virginia ITC Unix Support

5 5

Two semi-related questions - Adding a shelf, and FC config
by Justin Popa 24 May '12

24 May '12

I put in a FAS2020HA last year, and we're looking to do a little expansion. We currently just have the base unit, with 600GB SAS drives. We'd like to add a shelf for our testing environment. We asked for a quote from our reseller but it was astronomical, so we've looked into the used market. Other than the added equipment not being covered by my service & support contract, is there any issue buying a ds14mk4 and installing it? Assuming I do add it, I need some help figuring out my fiber connections. We're currently NFS, but I've recently procured some HBA's, as well as switches. To make things more interesting, we currently do our backup over fiber to an HP Autoloader connected directly to the first filer. Here are my concerns: 1) I've been told that you can't put a tape backup on a fiber switch with a netapp. True? If so, can I just leave it connected to filer A, and then only connect the second FC port to the switch? I know I'm looking at a point of failure there, but that would be temporary until we change our backup infrastructure. 2) Can you hang a shelf off a fiber switch as well? Is there any concern of doing that given question 1?

3 2

encrypted SMB traffic?
by Randy Rue 23 May '12

23 May '12

We've had a request to consider providing encrypted access to a CIFS filer. From the quick search I've done so far it appears that SMB itself doesn't provide any encryption options so instead we'd need to use IPSec at the filer and workstation ends. I've found NetApp man entries for the commands to manage this. Before I dig too deep into this, I'm hoping to get a feeling of whether anyone is using this in the real world. If so, how hard is it to set up? Is there a significant performance overhead? Any other gotchas? Hope to hear from you, rrue

3 2

Im Hiring!
by Jeff Mohler 22 May '12

22 May '12

If you are a high end, super skilled, deductive reasoning data and storage guy overflowing with awesome, I may have an interesting opportunity for you in Field Performance. SF Bay Area Location. Please do NOT contact me thru the list, but Pvt Email at jmohler(a)netapp.com So lang und Dank für alle Fische --- I drank what?

1 0

Cluster Notification from filer-a (SCHEDULED SNAPSHOT CREATE FAILED) CRITICAL
by Randy Rue 18 May '12

18 May '12

Hello All, Got this autosupport from our FAS3020 running 7.2.4. We use the "vol options VOLXX schedsnapname create_time" so that snapshot names reflect when the snapshot was taken, and the autosupport email contains the line: VOLXX - hourly.2012-05-16_1200 : Snapshot already exists. Trying to take a snapshot twice? Failing to roll off an old one? Just noise? I've STFW'd a little but I'm not finding anything like this. Hope to hear from you, Randy

2 2

LUN migration and consolidation
by Milazzo Giacomo 18 May '12

18 May '12

Hi all, I have the following situation. Source file system: an AIX volume created on hundreds of very small LUNs (HDS storage array, something like 15 GB each!) Destination file system: a "mirrored" one to online migrate using LVM with no disruption on services but at the same time consolidating on a single LUN on NetApp (as you know the geometry is completely different!) In the past I did it but on a 1:1 relationship between source and destination LUNs. Not there's somebody stating that on AIX this is not possible... ?!? But this is possibile on any distribution of Linux too!!! My idea in other words is to map the new space to the same AIX host that will use to create a sort of R1 between file systems (Veritas Volume Manager also doed this i.e) regards

3 3

Rate Limiting an NFS client
by Steven Kreuzer 18 May '12

18 May '12

Offhand, does anyone know if 8.1 has the ability to do rate limiting for nfs traffic? I have been asked to configure a filer so that nfs traffic from a certain ip addresses will not be able to exceed a maximum transfer rate while other clients will not be subjected to a maximum transfer rate I am aware that this can be done with ACLs on a switch, but management's would prefer to do it on the filer side if possible. Many Thanks -- Steven Kreuzer

1 1

Reallocate redux: Reallocation and de-dupe
by Peta Thames 17 May '12

17 May '12

Hi all, So I've been able to reallocate about 20% of our luns so far, and it's already made a huge improvement. I still see some huge latency spikes (up to 500ms) on some luns, mostly ones I haven't reallocated yet, but the spikes I originally described, where I had spikes on every volume in the aggregate, sometimes the entire filer, at the same time, have disappeared. Our VMware guy and the Citrix team are also reporting performance improvements. I must admit, I'm surprised by how quickly reallocate has improved this environment, to the point where users are already noticing it, and I'm nowhere near finished. If you've never checked your reallocation measurement, I highly recommend it! Thanks especially to Jeff Mohler and Fletcher Cocquyt (I found your blog posts through google before you'd emailed and thank you for writing them, they are really good). I was thinking over the weekend, as you do, about why I'd sometimes see spikes on multiple aggregates at once, and a common denominator there would be the backend loops. CPU has never been an issue. Does it make sense that the loops become a bottleneck on systems with severe fragmentation? Everything on the filer queues up there behind the IO going to one disk? Anyway, the question I really would like answered this time is: How best to balance reallocation and de-dupe. On Friday I reallocated a lun that's been de-duplicated, and the measurement changed from 6, hotspot 0, to 4, hotspot 28. Obviously the hotspot is due to the de-duplicated blocks. So, is it better to leave a higher reallocate measure (in this case, 6), with no hotspot, or better to reallocate these luns to lower the overall measure, even though it creates a horrible hotspot? Thanks, Peta On 30 April 2012 03:29, Milazzo Giacomo <G.Milazzo(a)sinergy.it> wrote: > > At last am answer with sense J > > I explain it better. I’m referring to what Fletcher wrote: “It was maddening to me back in 2010 how netapp support could blockade support cases with a blanket "must align VMs first" without a real quantification of the impact of misalignment – see” > > We had some critical case of this “performances” issues and in both cases the first thing that (support) asked to do: realign! > > Well. After a long and tedious process of realignment things were better…meanly 5% better, not more!!! And don’t let customer see graphs coming from the CMPG portal! Something like this attached…this could be pure terrorism :-D and a lot of nightmare for you trying to explain (and understand) the meaning of buckets J > > > Alignment is important…but NOT SO important. I concentrate investigation first on other levels. > > > > Reagards, > > > > > Da: toasters-bounces(a)teaparty.net [mailto:toasters-bounces@teaparty.net] Per conto di Fletcher Cocquyt > Inviato: venerdì 27 aprile 2012 08:46 > A: Peta Thames > Cc: Toasters(a)teaparty.net > Oggetto: Re: Reallocation and VMware > > > > Peta - we were dealing with this very issue (unexplained latency spikes Netapp blamed on VM misalignment) > > back in 2010 - I wrote up how we deconstructed the IOPs after many wasted perfstat iterations > > to solve it pretty much on our own: > > > > http://www.vmadmin.info/2010/07/vmware-and-netapp-deconstructing.html > > > > It was maddening to me back in 2010 how netapp support could blockade support cases with a > > blanket "must align VMs first" without a real quantification of the impact of misalignment - see > > > > http://www.vmadmin.info/2010/07/quantifying-vmdk-misalignment.html > > > > We ended up taking the downtime back then to align all VMs > > But now, I would be one to encourage your making the leap to 8.x - we are on 8.1GA and we are not looking back. > > The data motion of vFilers is allowing us to upgrade clusters with no downtime > > > > http://www.vmadmin.info/2012/04/meta-storage-vmotion-netapp-datamotion.html > > > > They have me almost believing in cluster mode for scale out... > > > > > > > > > > > > On Apr 25, 2012, at 11:40 PM, Peta Thames wrote: > > > > Hi Jack, > > You're right, and I should have mentioned it before. Large numbers of > the VMDKs are misaligned. I'd estimate about 33%, but I don't know > exactly how many as the shiny new VSC scanner got stuck halfway > through the scan I ran, leaving several VMs in a "being scanned" > state. I have a case open with Netapp to find out how to get those > VMs out of that state so I can a) continue the scan b) schedule fixing > the misaligned luns. > > Not all the luns that have large latency spikes are misaligned > however. Mind you, by the same token, not all of them are fragmented, > although so far (I'm still getting through measuring them all) there's > definitely a strong correlation. > > I also have to admit that I read the scale wrong in perf advisor, and > the numbers I'm seeing are in microseconds, not milliseconds. Still > way more than the 10ms I would like, but an order of magnitude better! > > Peta > > On 26 April 2012 15:52, Jack Lyons <jack1729(a)gmail.com> wrote: > > Have you checked the alignment of the VMDK's? > > > > Jack > > Sent from my Verizon Wireless BlackBerry > > > > -----Original Message----- > > From: Peta Thames <petathames(a)gmail.com> > > Sender: toasters-bounces(a)teaparty.net > > Date: Thu, 26 Apr 2012 14:49:43 > > To: <Toasters(a)teaparty.net> > > Subject: Reallocation and VMware > > > > Hi all, > > > > I'd like to pick your collective brains about your experiences with > > reallocate, specifically when reallocating luns under VMware. > > > > For background, we're running ONTAP 8.0.1 on a 3170 that's over three > > years old. I've been going through measuring reallocation, and most > > of the volumes are over 3. We have no snapshots, and only a > > relatively small number of volumes are de-duplicated. All our volumes > > and luns are thin-provisioned, and no aggregate is more than 76% full > > (most are ~65%). We regularly have huge latency spikes (worst I've > > seen so far is 5000000ms, and there are far too many to even track > > over 50000ms daily), and on one filer head, but not its partner, I > > regularly see disk utilisation go to 100% or more. I'm hoping > > reallocate will help here. > > > > I have a brief note from a NetApp support person who says "It’s very > > important that you complete the reallocation in the following order: > > 1:OS 2:LUN 3: Volume". > > > > I have two questions about this: > > - is it absolutely necessary to defrag the OS before you reallocate > > the lun? I'm sure I've run reallocate without defraging the OS and > > still seen performance improvements. I'm also assuming that this is > > only relevant to Windows VMs, not Linux (in our case, Red Hat/CentOS) > > ones. > > - if you only have one lun per volume, do you still need to run > > reallocate on both the lun and the volume? If only one, which is > > preferable? > > > > All advice appreciated. > > > > Thanks, > > Peta > > > > _______________________________________________ > > Toasters mailing list > > Toasters(a)teaparty.net > > http://www.teaparty.net/mailman/listinfo/toasters > > > _______________________________________________ > Toasters mailing list > Toasters(a)teaparty.net > http://www.teaparty.net/mailman/listinfo/toasters > >

3 2

upgrade impact from 7.3.5 to 8
by Randy Rue 16 May '12

16 May '12

Hello All, We're starting the planning for an upgrade of a v3170 from 7.3.5.1P5 to 8.0x or possibly 8.1GA 7-mode (we've already upgraded another filer to 8.1 without trouble). The system is backed by a 3Par T800 serving up LUNs as "disks" for the v3170 with all NetApp aggregates in essentially RAID0 (redundancy is provided at the SAN level). The v3170 is heavily used with a long list of protocols and features enabled including autohomes, de-dupe, vfilers serving CIFS, NFS, and iSCSI. NFS is v3 and v4, sometimes Kerberized, sometimes auth_sys. iSCSI is CHAP authenticated, sometimes using native initiators and sometimes SnapDrive and/or SnapManager for SQL. Backups are via NDMP to a CommVault system. Our enterpise SQL clusters are served (iSCSI) as well as our main VMware Farm (datastores are via NFS). I welcome feedback from anyone with experience in this type of effort. My sense is that the biggest dependencies will be on connected SnapDrive and SnapManager clients. For example, is there a way to identify all connected SnapDrive and SnapManager clients and their versions? Hope to hear from you, Randy Rue

1 0

Opinions on SnapProtect
by Fletcher Cocquyt 16 May '12

16 May '12

This looks like exactly the level of snapshot + backup integration we've been looking for - especially for our 100's of VMs on Netapp NFS volumes. http://www.netapp.com/us/products/protection-software/snapprotect.html Can anyone offer any opinions on the product - especially integration with vmware and existing tape backups (eg Legato) thanks, Fletcher

2 3

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters May 2012