toasters

toasters@lists.teaparty.net

2 participants
13514 discussions

Reallocate redux: Reallocation and de-dupe
by Peta Thames 17 May '12

17 May '12

Hi all, So I've been able to reallocate about 20% of our luns so far, and it's already made a huge improvement. I still see some huge latency spikes (up to 500ms) on some luns, mostly ones I haven't reallocated yet, but the spikes I originally described, where I had spikes on every volume in the aggregate, sometimes the entire filer, at the same time, have disappeared. Our VMware guy and the Citrix team are also reporting performance improvements. I must admit, I'm surprised by how quickly reallocate has improved this environment, to the point where users are already noticing it, and I'm nowhere near finished. If you've never checked your reallocation measurement, I highly recommend it! Thanks especially to Jeff Mohler and Fletcher Cocquyt (I found your blog posts through google before you'd emailed and thank you for writing them, they are really good). I was thinking over the weekend, as you do, about why I'd sometimes see spikes on multiple aggregates at once, and a common denominator there would be the backend loops. CPU has never been an issue. Does it make sense that the loops become a bottleneck on systems with severe fragmentation? Everything on the filer queues up there behind the IO going to one disk? Anyway, the question I really would like answered this time is: How best to balance reallocation and de-dupe. On Friday I reallocated a lun that's been de-duplicated, and the measurement changed from 6, hotspot 0, to 4, hotspot 28. Obviously the hotspot is due to the de-duplicated blocks. So, is it better to leave a higher reallocate measure (in this case, 6), with no hotspot, or better to reallocate these luns to lower the overall measure, even though it creates a horrible hotspot? Thanks, Peta On 30 April 2012 03:29, Milazzo Giacomo <G.Milazzo(a)sinergy.it> wrote: > > At last am answer with sense J > > I explain it better. I’m referring to what Fletcher wrote: “It was maddening to me back in 2010 how netapp support could blockade support cases with a blanket "must align VMs first" without a real quantification of the impact of misalignment – see” > > We had some critical case of this “performances” issues and in both cases the first thing that (support) asked to do: realign! > > Well. After a long and tedious process of realignment things were better…meanly 5% better, not more!!! And don’t let customer see graphs coming from the CMPG portal! Something like this attached…this could be pure terrorism :-D and a lot of nightmare for you trying to explain (and understand) the meaning of buckets J > > > Alignment is important…but NOT SO important. I concentrate investigation first on other levels. > > > > Reagards, > > > > > Da: toasters-bounces(a)teaparty.net [mailto:toasters-bounces@teaparty.net] Per conto di Fletcher Cocquyt > Inviato: venerdì 27 aprile 2012 08:46 > A: Peta Thames > Cc: Toasters(a)teaparty.net > Oggetto: Re: Reallocation and VMware > > > > Peta - we were dealing with this very issue (unexplained latency spikes Netapp blamed on VM misalignment) > > back in 2010 - I wrote up how we deconstructed the IOPs after many wasted perfstat iterations > > to solve it pretty much on our own: > > > > http://www.vmadmin.info/2010/07/vmware-and-netapp-deconstructing.html > > > > It was maddening to me back in 2010 how netapp support could blockade support cases with a > > blanket "must align VMs first" without a real quantification of the impact of misalignment - see > > > > http://www.vmadmin.info/2010/07/quantifying-vmdk-misalignment.html > > > > We ended up taking the downtime back then to align all VMs > > But now, I would be one to encourage your making the leap to 8.x - we are on 8.1GA and we are not looking back. > > The data motion of vFilers is allowing us to upgrade clusters with no downtime > > > > http://www.vmadmin.info/2012/04/meta-storage-vmotion-netapp-datamotion.html > > > > They have me almost believing in cluster mode for scale out... > > > > > > > > > > > > On Apr 25, 2012, at 11:40 PM, Peta Thames wrote: > > > > Hi Jack, > > You're right, and I should have mentioned it before. Large numbers of > the VMDKs are misaligned. I'd estimate about 33%, but I don't know > exactly how many as the shiny new VSC scanner got stuck halfway > through the scan I ran, leaving several VMs in a "being scanned" > state. I have a case open with Netapp to find out how to get those > VMs out of that state so I can a) continue the scan b) schedule fixing > the misaligned luns. > > Not all the luns that have large latency spikes are misaligned > however. Mind you, by the same token, not all of them are fragmented, > although so far (I'm still getting through measuring them all) there's > definitely a strong correlation. > > I also have to admit that I read the scale wrong in perf advisor, and > the numbers I'm seeing are in microseconds, not milliseconds. Still > way more than the 10ms I would like, but an order of magnitude better! > > Peta > > On 26 April 2012 15:52, Jack Lyons <jack1729(a)gmail.com> wrote: > > Have you checked the alignment of the VMDK's? > > > > Jack > > Sent from my Verizon Wireless BlackBerry > > > > -----Original Message----- > > From: Peta Thames <petathames(a)gmail.com> > > Sender: toasters-bounces(a)teaparty.net > > Date: Thu, 26 Apr 2012 14:49:43 > > To: <Toasters(a)teaparty.net> > > Subject: Reallocation and VMware > > > > Hi all, > > > > I'd like to pick your collective brains about your experiences with > > reallocate, specifically when reallocating luns under VMware. > > > > For background, we're running ONTAP 8.0.1 on a 3170 that's over three > > years old. I've been going through measuring reallocation, and most > > of the volumes are over 3. We have no snapshots, and only a > > relatively small number of volumes are de-duplicated. All our volumes > > and luns are thin-provisioned, and no aggregate is more than 76% full > > (most are ~65%). We regularly have huge latency spikes (worst I've > > seen so far is 5000000ms, and there are far too many to even track > > over 50000ms daily), and on one filer head, but not its partner, I > > regularly see disk utilisation go to 100% or more. I'm hoping > > reallocate will help here. > > > > I have a brief note from a NetApp support person who says "It’s very > > important that you complete the reallocation in the following order: > > 1:OS 2:LUN 3: Volume". > > > > I have two questions about this: > > - is it absolutely necessary to defrag the OS before you reallocate > > the lun? I'm sure I've run reallocate without defraging the OS and > > still seen performance improvements. I'm also assuming that this is > > only relevant to Windows VMs, not Linux (in our case, Red Hat/CentOS) > > ones. > > - if you only have one lun per volume, do you still need to run > > reallocate on both the lun and the volume? If only one, which is > > preferable? > > > > All advice appreciated. > > > > Thanks, > > Peta > > > > _______________________________________________ > > Toasters mailing list > > Toasters(a)teaparty.net > > http://www.teaparty.net/mailman/listinfo/toasters > > > _______________________________________________ > Toasters mailing list > Toasters(a)teaparty.net > http://www.teaparty.net/mailman/listinfo/toasters > >

3 2

upgrade impact from 7.3.5 to 8
by Randy Rue 16 May '12

16 May '12

Hello All, We're starting the planning for an upgrade of a v3170 from 7.3.5.1P5 to 8.0x or possibly 8.1GA 7-mode (we've already upgraded another filer to 8.1 without trouble). The system is backed by a 3Par T800 serving up LUNs as "disks" for the v3170 with all NetApp aggregates in essentially RAID0 (redundancy is provided at the SAN level). The v3170 is heavily used with a long list of protocols and features enabled including autohomes, de-dupe, vfilers serving CIFS, NFS, and iSCSI. NFS is v3 and v4, sometimes Kerberized, sometimes auth_sys. iSCSI is CHAP authenticated, sometimes using native initiators and sometimes SnapDrive and/or SnapManager for SQL. Backups are via NDMP to a CommVault system. Our enterpise SQL clusters are served (iSCSI) as well as our main VMware Farm (datastores are via NFS). I welcome feedback from anyone with experience in this type of effort. My sense is that the biggest dependencies will be on connected SnapDrive and SnapManager clients. For example, is there a way to identify all connected SnapDrive and SnapManager clients and their versions? Hope to hear from you, Randy Rue

1 0

Opinions on SnapProtect
by Fletcher Cocquyt 16 May '12

16 May '12

This looks like exactly the level of snapshot + backup integration we've been looking for - especially for our 100's of VMs on Netapp NFS volumes. http://www.netapp.com/us/products/protection-software/snapprotect.html Can anyone offer any opinions on the product - especially integration with vmware and existing tape backups (eg Legato) thanks, Fletcher

2 3

snapshot cleanup and performance
by Peter D. Gray 16 May '12

16 May '12

OK, just wondering if anybody can shed light on this. We just had a massive performance problem on one of our netapps. One aggregate was amazingly busy, with disk drives 100% busy all the time. IO latency went through the roof for all the volumes on the aggregate. We spent a bit of time on this and we are not novice netapp users. In the end we could not identify the problem. There appeared to be no relationship between the amountof I/O coming from clients and the amount of I/O on the aggregate. So, we called netapp support. It took them a while (a few days) but eventually they suggested changing the snapshot schedules and removing snapshots off the aggregates and also removing hourly snapshots off the volumes on the aggregate. We complied. It took a while, but after 5 hours or so, relatively suddenly the problem went away and seems to have stayed away for a day now. I am assuming the snapshot cleanup was the problem and the problem stopped when it finally caught up. So, I guess my question is "why is snapshot cleanup so expensive". Do blocks freed from snapshots need to be written, if so why? It seems snapshot creation is cheap, but deletion expensive, which makes the entire snapshot management cycle rather more trouble then you might hope. Regards, pdg

4 3

NFS Lock and Oracle
by Craig A. Falls 15 May '12

15 May '12

Hi, I was wondering if anyone had a script for viewing/removing NFS locks on a filer? We have just started to use Oracle on NFS and have had an issue where we have had the host crash and leave locks which then stop the DB being started again (on that node) when that node is recovered and will probably (guessing) eventually stall the DB when other nodes try to access the data that is being locked. I was planning on something like: - check the locks for a filer/vfiler for the specific host - remove the locks using lock status/lock break Any thoughts/help or advice would be appreciated. thanks c

3 2

Writing to the list via Gmane
by Oliver Brakmann 15 May '12

15 May '12

Hello all, just a heads-up: if this post reaches the list, it means that posting to the list via gmane (http://gmane.org) works again. It broke after the move to the new domain, but I contacted the Gmane maintainer recently and asked him to please fix the address, which has now happened. Regards, Oliver

1 0

What is Flash Cache caching?
by Fletcher Cocquyt 15 May '12

15 May '12

We setup a custom view in NMC to show us the hit % and Disk Reads replaced per the Flash Cache Best Practices doc TR-3832. Is it possible to get more granular information about what is being cached? These are NFS data stores and one of the volumes is 500Gb of web content. So the web team would be very interested to know the net benefit to them - eg how much are the apache instances benefitting. We also have oracle instances, VMware volumes and would like per volume stats if possible I guess I am looking for the ability to apply the custom view to different vFilers and see per volume hits and hit %, Disk Reads replaced We have the 1Tb card installed and see the Disk Reads Replaced topping out around 5000 - with one spike up to ~8500 Does this look like what others are seeing in a mixed workload environment like I described? thanks, Fletcher

3 5

Cleaning up snapvault relationships.
by Jeff Cleverley 14 May '12

14 May '12

Greetings, We retired a SV primary volume and I need to keep the destination SV volume around until the snapshots age out. The backups are done via command line/cron/snapvault snap sched setups. They are not managed by DFM/PM or any other package. I did a snapvault stop on the destination which seems to have worked fine. I verified that the destination qtree is gone. I also did a snapvault release on the primary and a snapvault snap unsched on both primary and secondary. I've even rebooted the destination filer which would have the effect of stopping and starting snapvault. The primary shows no trace of the former relationship. The secondary got rid of everything, but some of the destinations still show up in the snapvault status output. I need to figure out how to clean these up. Here is the output of a snapvault -l of one of the destination qtrees: Snapvault secondary is ON. Source: source:/vol/carmel2/test Destination: destination:/vol/carmel2_i/test Status: Idle Progress: - State: Unknown Lag: - Mirror Timestamp: - Base Snapshot: - Current Transfer Type: - Current Transfer Error: - Contents: - Last Transfer Type: Update Last Transfer Size: 8 KB Last Transfer Duration: 00:00:06 Last Transfer From: source:/vol/carmel2/test Thanks, Jeff -- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611

1 1

revision control for /etc/exports with post-commit exports -va API call?
by Fletcher Cocquyt 11 May '12

11 May '12

Is anyone doing revision control on /etc/exports with post-commit hooks to call exportfs -va? (including vfiler:/rootvol/etc/exports)? Or more generally what is the recommended best practice for these command line file updates in a team environment (multiple admins)? There used to be a Filerview UI for updating exports, but I can not see one in the new System Manager - is it there? In a team environment we'd like to avoid touching files directly with vi (and introducing errors) - but if we need to, we'd like revision control and automation with post-commit hooks which has worked well for our apache and puppet configs. thanks Fletcher

1 0

SMSQL question
by Klise, Steve 10 May '12

10 May '12

I just upgraded a SQL 2005 SQL cluster from snapdrive 5, to 6.3.1r1 and SMSQL 2.x to 5.0. I also upgraded the verification server (remote server). Now, after upgrading, on a set of drives (G:\data and L:\logs), the backup works, but DBCC fails. What is strange is there is another set of drives (H:\data M:\logs) DBCC works.. I checked for snapshots that would still be mounted, flex clones, etc. Nothing appears to have "busy snapshots". This is the error (names/ip's changed to protect the innocent). Even checked for LUNS that would be not assigned to an igroup.. nothing.. This is the event from the verification server: Event Type: Error Event Source: SnapDrive Event Category: Generic event Event ID: 331 Date: 5/4/2012 Time: 1:19:32 PM User: N/A Computer: VERIFICATIONSRV Description: Failed to connect FlexClone of Snapshot copy (sqlsnap__SQL__recent) as drive C:\Program Files\NetApp\SnapManager for SQL Server\SnapMgrMountPoint\MPDisk001\ on the computer (VERIFICATIONSRV). LUN Name = lun0.lun Storage Path = /vol/SQLSERVER/SQLSERVER/ Protocol Type = LUN Storage System Name = 10.10.10.10 Error code : ZAPI: An attempt to online LUN '/vol/sdw_cl_SQLSERVER_0/SQLSERVER/lun0.lun' failed on the storage system 10.10.10.10. Error description: Another LUN mapped at this number This is the snap list. Volume SQLSERVER working... %/used %/total date name ---------- ---------- ------------ -------- 0% ( 0%) 0% ( 0%) May 04 13:18 sqlsnap__SQLSERVER__recent 0% ( 0%) 0% ( 0%) May 04 11:33 sqlsnap__SQLSERVER_05-04-2012_11.33.12__weekly 0% ( 0%) 0% ( 0%) May 04 10:43 sqlsnap__SQLSERVER_05-04-2012_10.42.44__weekly 0% ( 0%) 0% ( 0%) May 04 10:03 panstor2(1573990916)_SQLSERVER_snap.65 (snapmirror) 0% ( 0%) 0% ( 0%) May 04 10:03 @snapmir@{EF1F6E8C-64BA-4C23-A8F1-1566E881A373} 0% ( 0%) 0% ( 0%) May 04 10:01 sqlsnap__SQLSERVER_05-04-2012_10.00.07__daily 0% ( 0%) 0% ( 0%) May 04 07:59 sqlsnap__SQLSERVER_05-04-2012_07.58.44__weekly 0% ( 0%) 0% ( 0%) May 04 07:20 sqlsnap__SQLSERVER_05-04-2012_07.19.32__weekly 0% ( 0%) 0% ( 0%) May 04 07:12 @snapmir@{5CF5E20A-6C44-4CE9-B212-0F4BAEB90636} 0% ( 0%) 0% ( 0%) May 04 07:11 sqlsnap__SQLSERVER_05-04-2012_07.10.46__daily 0% ( 0%) 0% ( 0%) May 03 20:40 sqlsnap__SQLSERVER_05-03-2012_20.39.16__daily 0% ( 0%) 0% ( 0%) May 03 19:00 sqlsnap__SQLSERVER_05-03-2012_19.00.00__weekly 0% ( 0%) 0% ( 0%) May 03 18:30 sqlsnap__SQLSERVER_05-03-2012_18.30.00__daily 0% ( 0%) 0% ( 0%) May 03 14:30 sqlsnap__SQLSERVER_05-03-2012_14.30.00__daily 0% ( 0%) 0% ( 0%) May 03 10:30 sqlsnap__SQLSERVER_05-03-2012_10.30.00__daily 0% ( 0%) 0% ( 0%) May 02 19:00 sqlsnap__SQLSERVER_05-02-2012_19.00.00__weekly 0% ( 0%) 0% ( 0%) May 01 19:00 sqlsnap__SQLSERVER_05-01-2012_19.00.00__weekly 0% ( 0%) 0% ( 0%) Apr 30 19:00 sqlsnap__SQLSERVER_04-30-2012_19.00.00__weekly Any help would be appreciated.

2 3

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters