toasters

toasters@lists.teaparty.net

1 participants
13528 discussions

instantaneous snap restore ?
by Avi Ben Emanuel (Wollman) 04 Feb '13

04 Feb '13

marketing continuously claim that snap restore is instantaneous. yet I have yet to succeeded doing so. "snap restore" of files to a location quite a while and is "copying the files" nothing close to instantaneous. i would be happy to understand what i am missing. the following is a example of a virtual disk that was restored, it took quite a while to "restore"=copy this 80gb file. snap restore -t file -s nightly.0 /vol/vmsata/vmfs1/lms-client/lms-clinet-flat.vmdk Avi -- ------------ Avi Ben Emanuel (Wollman) (אבי בן עמנואל (וולמן Systems Engineer | Unix Microsoft Vmware Jerusalem College of Technology מחלקת תקשוב בית הספר הגבוה לטכנולוגיה - מכון לב POB 16031 Jerusalem 91160 Israel הועד הלאומי 21 ירושלים Email: avi(a)jct.ac.il | Office: +972-2-6751036 | Mobile: +972-52-3809107 ------------

4 3

Re: Aggregate Disk Busy 100% with volume IOPS low
by Fletcher Cocquyt 27 Jan '13

27 Jan '13

Indeed, We are considering replacing our premium support with next day and using the savings to buy some professional services - we've heard other groups see better support ROI with this combination. On the wafl block reclamation - are you talking about options wafl.trunc.throttle.vol.max etc? we had to tune that back in 2010 7.3x days: http://www.vmadmin.info/2010/11/vfiler-migrate-netapp-lockup.html But not sure this is still a hidden option in 8.1.x? I read references to a tool called perfviewer - anyone still using that? thanks On Jan 26, 2013, at 1:45 PM, Isaiah <zoratu(a)gmail.com> wrote: > If I were you, I would purchase some incident-based support from Berkeley Communications ("Berkcom"). They're the only reseller of both used and new NetApp gear in the world. They know more about NetApp than NetApp. I've been a customer for nine years and they're the first number I call--because they're all experts. No escalation yadda yadda. Completely worth the modest fees to support gear not purchased through them. > > The last time your situation happened to me I ended up having to tune the wafl block reclamation aggressiveness. There were so many snapshots happening on the system with lots of gradual changes that the disk utilization was high just reclaiming blocks that used to be dirty. > -- > - Isaiah > > On Jan 26, 2013, at 10:15, Fletcher Cocquyt <fcocquyt(a)stanford.edu> wrote: > >> On Nick's advice I setup a job to log both wafltop and ps -c 1 once per minute - and we had a sustained sata0 disk busy from 5am-7am as reported by NMC. >> First question I have from wafltop show is - what is the first row (sata0::file i/o) reporting ? What could be the source of these 28907 non-volume specific Read IOs? >> >> Application MB Total MB Read(STD) MB Write(STD) Read IOs(STD) Write IOs(STD) >> ----------- -------- ------------ ------------- ------------- -------------- >> sata0::file i/o: 5860 5830 30 28907 0 >> sata0:backup:nfsv3: 608 0 608 31 0 >> >> I'm just starting to go through the data >> >> aggr status >> Aggr State Status Options >> sata0 online raid_dp, aggr nosnap=on, raidsize=12 >> 64-bit >> aggr2 online raid_dp, aggr nosnap=on, raidsize=19 >> 64-bit >> aggr1 online raid_dp, aggr root, nosnap=on, raidsize=14 >> 32-bit >> na04*> df -Ah >> Aggregate total used avail capacity >> aggr1 13TB 11TB 1431GB 89% >> aggr2 19TB 14TB 5305GB 74% >> sata0 27TB 19TB 8027GB 72% >> >> >> <sataIOPSJan26.jpeg> >> >> thanks >> >> >> On Jan 25, 2013, at 5:33 PM, Nicholas Bernstein <nick(a)nicholasbernstein.com> wrote: >> >>> Try doing a 'ps -c 1' or a wafltop show (double check the syntax) while you're getting the spike; those will probably help you narrow down the processes that are using your disks. Both are priv set advanced/diag commands. >>> >>> Nick >>> >> >> _______________________________________________ >> Toasters mailing list >> Toasters(a)teaparty.net >> http://www.teaparty.net/mailman/listinfo/toasters

1 0

Re: Aggregate Disk Busy 100% with volume IOPS low
by Fletcher Cocquyt 24 Jan '13

24 Jan '13

Hi Mike, this is a snap mirror SOURCE. The perfstats captured during the issue include wafl scan status - I'll mention it to the engineer when I show the disparity btw AGGR OPS and total volume OPS. Unfortunately its not easy to differentiate the source of IOPS with any existing tools in my experience - including perfstats and escalated Netapp support analysis. But the NMC AGGR vs volume stats need to be explained somehow. thanks On Jan 23, 2013, at 9:35 PM, Mike Nye <Mike.Nye(a)xpanse.com.au> wrote: > Hi Fletcher, > > Is this FAS a SnapMirror destination? This sounds very much like a volume deswizzle / CBR scan issue… > > You could try scheduling a “priv set advanced; wafl scan status” to run via RSH/SSH during the time of poor performance, to see what internal WAFL scans are taking place on the system. > > Thinks to look for are “Container Block Reclamation” and/or “Volume Deswizzle”. > > Kind regards, > Mike Nye > > > <image001.jpg> > <image002.jpg> > Mike Nye > Team Lead Systems Engineer > mike.nye(a)xpanse.com.au > www.xpanse.com.au > Mob: > Office: > Fax: > +61 407 772 465 > +61 8 9322 6767 > +61 8 9322 6077 > 18 Emerald Tce > West Perth > WA 6005 > <image003.jpg> > > > From: toasters-bounces(a)teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Fletcher Cocquyt > Sent: Thursday, 24 January 2013 1:28 PM > To: toasters(a)teaparty.net Lists > Subject: Aggregate Disk Busy 100% with volume IOPS low > > 3270 cluster, OnTAP 8.1-7mode > > We are investigating a SATA aggregate showing repeated 5am disk 100% busy spikes without its volumes showing any corresponding IOPS spike as reported by Netapp Management Console (NMC). > The 5am disk busy spikes correlate with very high latency on volumes on a different SAS aggregate. These volumes host VMs which then timeout, some needing reboots. > Today when I heard from Netapp support after reviewing my perfstat the engineer reported this is expected since NVRAM buffers are shared btw aggregates. > > But when I dig further into the NMC stats I see the SATA aggregate disk busy actually corresponds to a DROP in IOPS on the 3 volumes hosted on the SATA aggregate - almost like some internal aggregate operations are starving out the external volume ops. > I checked the snapshots (vol and aggr), snap mirror, dedup and none of the usual suspects were running. > > When I look at the NMC throughput graphs and switch on the legend - it shows a 5am READ blocks/sec spike corresponding perfectly to the disk busy. > > Where are these AGGR level READ operations coming from that are missing from the constituent volume IOPS, and in fact seem to be starving out volume level IO? > > I don't see much in the messages log, but will check the rest of the logs for internal type OPS > > thanks for any insight > > > > > >

2 1

Re: Aggregate Disk Busy 100% with volume IOPS low
by Fletcher Cocquyt 24 Jan '13

24 Jan '13

No NDMP in use for us - at a loss to explain this level of AGGR disk busy with no vol level IO Feels like an internal type operation hitting a bug thanks On Jan 23, 2013, at 10:21 PM, "Klise, Steve" <klises(a)sutterhealth.org> wrote: > A stab but what about ndmp jobs? > > From: Fletcher Cocquyt [mailto:fcocquyt@stanford.edu] > Sent: Wednesday, January 23, 2013 09:27 PM > To: toasters(a)teaparty.net Lists <toasters(a)teaparty.net> > Subject: Aggregate Disk Busy 100% with volume IOPS low > > 3270 cluster, OnTAP 8.1-7mode > > We are investigating a SATA aggregate showing repeated 5am disk 100% busy spikes without its volumes showing any corresponding IOPS spike as reported by Netapp Management Console (NMC). > The 5am disk busy spikes correlate with very high latency on volumes on a different SAS aggregate. These volumes host VMs which then timeout, some needing reboots. > Today when I heard from Netapp support after reviewing my perfstat the engineer reported this is expected since NVRAM buffers are shared btw aggregates. > > But when I dig further into the NMC stats I see the SATA aggregate disk busy actually corresponds to a DROP in IOPS on the 3 volumes hosted on the SATA aggregate - almost like some internal aggregate operations are starving out the external volume ops. > I checked the snapshots (vol and aggr), snap mirror, dedup and none of the usual suspects were running. > > When I look at the NMC throughput graphs and switch on the legend - it shows a 5am READ blocks/sec spike corresponding perfectly to the disk busy. > > Where are these AGGR level READ operations coming from that are missing from the constituent volume IOPS, and in fact seem to be starving out volume level IO? > > I don't see much in the messages log, but will check the rest of the logs for internal type OPS > > thanks for any insight > > > > > >

1 0

Vfiler DR activate
by Iluhes 22 Jan '13

22 Jan '13

Hello Toasters, After running VFILER DR Activate and failing over vFiler to DR site, none of the shares were available. The DC controller was not available in DR bubble, but we thought that local admin users on vfiler should have access to shares. We could not access shares. For certain vFiler DRs when the production side is a domain member, we have a need to have the filer available in a DR test situation in which the domain controller is not available. We have found in testing that in the absence of the domain controller, all share access is denied. Even when we try accessing with local accounts, which on a Windows file server, would have let us access the share in the absence of a domain controller we are denied access. Is that correct behavior? We would get: "There are currently no logon server available to service the logon request" Is there anyway to do vFiler DR where the vFiler is a domain member where one can access the shares in DR without the domain being present--using local accounts only. Thanks!!!

2 1

RE: Vfiler DR activate
by steve klise 22 Jan '13

22 Jan '13

Was there a delta in time between the filer and the clinet? Usually AD has to be within 5 minutes of source and destination for Kerberos. You would want an AD controller to be available, and usually the 1st box up for CIFS authentication... Sent from Windows Mail From: Iluhes Sent: ‎January‎ ‎21‎, ‎2013 ‎6‎:‎06‎ ‎PM To: toasters(a)teaparty.net Subject: Vfiler DR activate Hello Toasters, After running VFILER DR Activate and failing over vFiler to DR site, none of the shares were available. The DC controller was not available in DR bubble, but we thought that local admin users on vfiler should have access to shares. We could not access shares. For certain vFiler DRs when the production side is a domain member, we have a need to have the filer available in a DR test situation in which the domain controller is not available. We have found in testing that in the absence of the domain controller, all share access is denied. Even when we try accessing with local accounts, which on a Windows file server, would have let us access the share in the absence of a domain controller we are denied access. Is that correct behavior? We would get: "There are currently no logon server available to service the logon request" Is there anyway to do vFiler DR where the vFiler is a domain member where one can access the shares in DR without the domain being present--using local accounts only. Thanks!!!

1 0

Re: wafl_cp_slovol_warning_1 with big latency spikes
by Brian Beaulieu 19 Jan '13

19 Jan '13

What about other IO on the aggregate? Any other volume activity (snaps, resize) I feel your pain. I've had one single VM cause IO spikes and thus latency. I had this happen when I moved the VM from an NFS datastore to a VMFS datastore to get more IO potential. Well, that extra IO was too much of a piece of the pie. The other night I had all NFS/CIFS IO completely stop for a couple of minutes during a volume resize. I love NFS+VMWare+NetApp but sometimes I hate it as well. Brian

4 5

Reallocate a Deduplicated Volume?
by Brad Smith 18 Jan '13

18 Jan '13

Looking for help/direction on reallocating deduplicated volumes. Have a FAS2040 running 7.3.6 and adding an additional DS4243 shelf with SAS drives and want to extend current aggregate by adding a new raid group created with the new disk to the aggregate. On this aggregate I have volumes that are deduplicated and have snapshots. I have read several NetApp documents that seem to conflict on running reallocate on the volumes after adding the new disks to the existing aggregate. I have read you should reallocate but not on a volume that is deduplicated. I have also read that if you use the -p switch this will be OK, reallocate start -f -p /vol/VOLUMENAME. Will I have problems if I run reallocate on my deduplicated volumes? Thanks. Brad

6 6

Snapvault between 8.0.2P4 and 8.1.1
by Steven Kreuzer 17 Jan '13

17 Jan '13

Hello- I have two filers that are both running 8.0.2P4 and they are both snap vaulting a volume from one to the other. Rebooting one of the filers requires tons and tons of coordination between several different groups and the other filer I can pretty much reboot whenever I fancy. I would like to upgrade that filer to 8.1.1 so I can take advantage of all the 64 bit migration stuff and also test out the branchcache support. The only thing I am concerned about is that I may lose the ability to snapvault between the two. I recall it was not possible to snapvault from a filer running 8.x to one running 7.x, is it possible to still snap from a filer running 8.0 to one running 8.1? Thanks

2 1

Re: wafl_cp_slovol_warning_1 with big latency spikes
by Brian Beaulieu 17 Jan '13

17 Jan '13

3rd time is the charm. I've attached my sysstat from the other night when NFS/CIFS hung up... is this what you've seen as well? During that issue, FCP was also slow.. had some MPIO failovers happening on our AIX LPARs. But, AIX handles that just fine and at least has an alternate path through the other filer. NFS isn't so lucky. I have a 3250+1TB PAM sitting on deck.. you'd think that the 3240+512GB PAM would be sufficient for what we do. While I do have SATA in use for VMWare, it's not heavy hitting VMs.. it's the dormant stuff, mostly. I'm moving a lot of it, though, to 6xDS4243x600GB-15k shelves ASAP. I'm drinking the PAM kool-aid too but do have some measurable results primarily on our PeopleSoft DB2 databases. I definitely wouldn't bet on SATA+PAM == FC/SAS performance. Brian

3 4

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters