toasters December 2012

toasters@lists.teaparty.net

34 participants
16 discussions

Aggregate Best Practices
by Ray Van Dolson 28 Dec '12

28 Dec '12

We're revisiting how we set up our aggregates and I want to see how others out there do it. Specifically, what strategies do you use for ensuring certain key applications or environments get the performance they need in a shared environment. Typcially we'll create large aggregates based on a homogeneous disk type. 15K SAS disks in one aggregate, SATA in another. In some cases when it's only a single type of disk, we'd have 60 15K disks in one aggregate and 60 in other (assigned to each controller respectively). The idea here is that more spindles gives us the most performance. However, some applications/workloads are more important than others, and some can be "bullies" impacting the important stuff. Ideally we'd try and keep our OLTP random workloads on one filer and heavy sequential workloads on another (maybe dedicated). We've also been discussing creating multiple, smaller aggregates that we then assign to specific workloads guaranteting those spindles for those workloads. Lower possible maximum performance, but better protection against "bullies"[1]. I also know ONTAP has some I/O QoS options. I'm less inclined to go that direction however. Our workloads tend to be ESX VM's using the filers as NFS datastores. We have the usual budgetary / purchasing cycle constraints, so trying to minimize pain for as long as possible until we can add resources. How do folks out there handle this? Thanks, Ray [1] Controller obviously still is shared.

6 8

question about NDMP dumps
by Scott Eno 21 Dec '12

21 Dec '12

In investigating why an NDMP dump over 10GbE isn't going as fast as it seems it should, a question arose. Does the data being dumped via netbackup to a data domain device, 1.58TB deduped down to 813GB, have to get re-hydrated as the dump proceeds? And, if so, would that impact the speed of the dump?

3 4

Moving to 64bit aggrs
by Steven Kreuzer 21 Dec '12

21 Dec '12

Greetings- On our primary filer we are running 8.0.3p2 and I have a 11TB 32bit aggregate that is close to running out of space. I would like to move this volume, along with all its snapshots to a new 64bit aggregate. >From my research, it looks like if I was running 8.1 I could either upgrade the aggr in place, or do a vol move from a 32bit 64bit aggr. In 8.0.3, do I have any options like this? I would really like to avoid using rsync to copy over a snapshot and take a snapshot on the destination, wash rinse repeat. Many Thanks

4 3

8.1.1 RLW Performance Trouble, 8.1.2 issues?
by Randy Rue 12 Dec '12

12 Dec '12

Hello All, After a few months of planning and preparation we were just about to pull the trigger on an 8.1.1 upgrade to our v3170 currently running 7.3.5.1P5 with a 3Par T800 spinning 528 SATA spindles. We've crawled through release notes and upgrade notes, used the online "upgrade advisor" tools, checked this list and engaged our NetApp SE. Tested against a simulator , killed a chicken at midnight of the full moon, etc. Then last week one of our admins found this link and asked in passing if this issue was relevant to our situation: Data Ontap 8.1 upgrade - RLW_Upgrading process and other issues https://communities.netapp.com/thread/22676 Looks like upgrading to 8.1x implements a new layer of protection called RLW (RAID protection from Lost Writes). This requires the addition of some metadata to the disk system and after upgrade a background process "rlw_update" runs for some period of time. The trouble is, this process does not "nice" itself as it's meant to when other processes are running. Worse, if it runs at the same time as other "not so nice" processes like de-dupe there can be disastrous performance issues. This problem is exacerbated if disk utilization is high, or if slower disks are used, or if a lot of misaligned traffic is running. Users in the wild have reported the rlw_update process taking several weeks and horrible performance issues during its tenure. Yike, I'm just glad we found out about this now and not Monday morning. Our filer consistently runs ~90% disk utilization, is usually running two or three de-dupe processes, is running on SATA disks, and one node does nothing but serve up NFS datastores to our VMware farm which we suspect is running mostly misaligned VMDK files. So now we've postponed the upgrade. We'll need to retest against 8.1.2 (or later). We're averse to get too eager to upgrade to 8.1.2 until it's been out a while, especially as NetApp seems to be on a roll lately, releasing new versions to address performance issues that then seem have performance issues. We're also concerned about upgrading to any new version while the system is heavily loaded (we're also in the process of deploying a new block-level storage system that will offload more than half of the current performance load on the v3170). So. This post is intended first as a heads-up to anyone in a similar situation who's about upgrade to 8.1.1. Another takeaway might be to acknowledge the value of the official NetApp Communities pages: I've usually relied on this toasters list and/or our NetApp/VAR technical staff, but will also be following the "official" user community resources from now on. And I also welcome any feedback from anyone with experience or information to offer. Anyone been in this situation? Anyone running 8.1.2 yet? Anyone have advice on upgrading a significantly busy system? Hope to hear from you, Randy Rue Seattle

4 3

RE: snapmirror socket error :vfiler dr configure : 8.1
by Uddhav Regmi 09 Dec '12

09 Dec '12

PLEAES DISREGARD MY ALL PREVIOUS MESSAGES. I DO NOT WANT ANY FEEDBACK From: Uddhav Regmi [mailto:uregmi111@gmail.com] Sent: Sunday, December 09, 2012 12:04 PM To: 'Cruxrealm'; 'toasters(a)teaparty.net' Subject: RE: snapmirror socket error :vfiler dr configure : 8.1 Here is the file na021> rdfile /etc/snapmirror.conf #Regenerated by registry Thu Dec 6 18:44:27 GMT 2012 na002-sm:vfdoleprd02_db201 na021:vfdoleprd02_db201 - 0-59/1 * * * na002-sm:vfdoleprd02root na021:vfdoleprd02root - 0-59/3 * * * na021> snapmirror status Snapmirror is on. Source Destination State Lag S tatus na002-sm:vfdoleprd02_db201 na021:vfdoleprd02_db201 Uninitialized - I dle na002-sm:vfdoleprd02root na021:vfdoleprd02root Uninitialized - I dle na021> snapmirror status Snapmirror is on. Source Destination State Lag Status na002-sm:vfdoleprd02_db201 na021:vfdoleprd02_db201 Uninitialized - Idle na002-sm:vfdoleprd02root na021:vfdoleprd02root Uninitialized - Idle From: Cruxrealm [mailto:cruxrealm@aol.com] Sent: Sunday, December 09, 2012 11:55 AM To: Uddhav Regmi Subject: Re: snapmirror socket error :vfiler dr configure : 8.1 Try to do a test replication snapmirror from your source and destination. If that works, then check your conf file. Sent from my iPad On Dec 9, 2012, at 8:28 AM, "Uddhav Regmi" <uregmi111(a)gmail.com> wrote: snapmirror socket error na002 - > na009 ( snapmirror now happening ) source - > dr both are 8.1 ; mirror is happening every 30 min vfiler in na009 is in stopped condition. Aggregate got full here, so we decided to move the dr volume to another filer I did vfiler dr configure command and vol got moved to another box na021 all looks good, vfiler got created. snapmirror was in unitialized state when I start snapmirror : I'm getting this weired error na021> vol restrict vfdoleprd02_db201 Thu Dec 6 15:57:24 EST [na021:vFiler.storageUnit.off:warning]: vFiler vfdoleprd02: storage unit /vol/vfdoleprd02_db201 now offline. Volume 'vfdoleprd02_db201' is now restricted. na021> snapmirror initialize -S na002-sm:vfdoleprd02_db201 na021:vfdoleprd02_db201 Transfer aborted: could not read from socket. na021> Thu Dec 6 15:57:41 EST [na021:replication.dst.err:error]: SnapMirror: destination transfer from na002-sm:vfdoleprd02_db201 to vfdoleprd02_db201 : could not read from socket. na021> snapmirror statusThu Dec 6 15:58:00 EST [na021:replication.dst.err:error]: SnapMirror: destination transfer from na002-sm:vfdoleprd02_db201 to vfdoleprd02_db201 : volume is not online; cannot execute operation. Snapmirror is on. Source Destination State Lag Status na002-sm:vfdoleprd02_db201 na021:vfdoleprd02_db201 Unknown - Pending na002-sm:vfdoleprd02root na021:vfdoleprd02root Uninitialized - Idle na021> snapmirror status -l na021:vfdoleprd02_db201 Snapmirror is on. Source: na002-sm:vfdoleprd02_db201 Destination: na021:vfdoleprd02_db201 Status: Pending Progress: - State: Unknown Lag: - Mirror Timestamp: - Base Snapshot: - Current Transfer Type: Scheduled Current Transfer Error: volume is not online; cannot execute operation Contents: - Last Transfer Type: - Last Transfer Size: - Last Transfer Duration: - Last Transfer From: - na021> any idea what is going on here ???? snapmirror.access list is fine My thought is : do we need to break the original mirror ????? which was going from na002 to na009 _______________________________________________ Toasters mailing list Toasters(a)teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

1 0

RE: snapmirror socket error :vfiler dr configure : 8.1
by Uddhav Regmi 09 Dec '12

09 Dec '12

Here is the file na021> rdfile /etc/snapmirror.conf #Regenerated by registry Thu Dec 6 18:44:27 GMT 2012 na002-sm:vfdoleprd02_db201 na021:vfdoleprd02_db201 - 0-59/1 * * * na002-sm:vfdoleprd02root na021:vfdoleprd02root - 0-59/3 * * * na021> snapmirror status Snapmirror is on. Source Destination State Lag S tatus na002-sm:vfdoleprd02_db201 na021:vfdoleprd02_db201 Uninitialized - I dle na002-sm:vfdoleprd02root na021:vfdoleprd02root Uninitialized - I dle na021> snapmirror status Snapmirror is on. Source Destination State Lag Status na002-sm:vfdoleprd02_db201 na021:vfdoleprd02_db201 Uninitialized - Idle na002-sm:vfdoleprd02root na021:vfdoleprd02root Uninitialized - Idle From: Cruxrealm [mailto:cruxrealm@aol.com] Sent: Sunday, December 09, 2012 11:55 AM To: Uddhav Regmi Subject: Re: snapmirror socket error :vfiler dr configure : 8.1 Try to do a test replication snapmirror from your source and destination. If that works, then check your conf file. Sent from my iPad On Dec 9, 2012, at 8:28 AM, "Uddhav Regmi" <uregmi111(a)gmail.com> wrote: snapmirror socket error na002 - > na009 ( snapmirror now happening ) source - > dr both are 8.1 ; mirror is happening every 30 min vfiler in na009 is in stopped condition. Aggregate got full here, so we decided to move the dr volume to another filer I did vfiler dr configure command and vol got moved to another box na021 all looks good, vfiler got created. snapmirror was in unitialized state when I start snapmirror : I'm getting this weired error na021> vol restrict vfdoleprd02_db201 Thu Dec 6 15:57:24 EST [na021:vFiler.storageUnit.off:warning]: vFiler vfdoleprd02: storage unit /vol/vfdoleprd02_db201 now offline. Volume 'vfdoleprd02_db201' is now restricted. na021> snapmirror initialize -S na002-sm:vfdoleprd02_db201 na021:vfdoleprd02_db201 Transfer aborted: could not read from socket. na021> Thu Dec 6 15:57:41 EST [na021:replication.dst.err:error]: SnapMirror: destination transfer from na002-sm:vfdoleprd02_db201 to vfdoleprd02_db201 : could not read from socket. na021> snapmirror statusThu Dec 6 15:58:00 EST [na021:replication.dst.err:error]: SnapMirror: destination transfer from na002-sm:vfdoleprd02_db201 to vfdoleprd02_db201 : volume is not online; cannot execute operation. Snapmirror is on. Source Destination State Lag Status na002-sm:vfdoleprd02_db201 na021:vfdoleprd02_db201 Unknown - Pending na002-sm:vfdoleprd02root na021:vfdoleprd02root Uninitialized - Idle na021> snapmirror status -l na021:vfdoleprd02_db201 Snapmirror is on. Source: na002-sm:vfdoleprd02_db201 Destination: na021:vfdoleprd02_db201 Status: Pending Progress: - State: Unknown Lag: - Mirror Timestamp: - Base Snapshot: - Current Transfer Type: Scheduled Current Transfer Error: volume is not online; cannot execute operation Contents: - Last Transfer Type: - Last Transfer Size: - Last Transfer Duration: - Last Transfer From: - na021> any idea what is going on here ???? snapmirror.access list is fine My thought is : do we need to break the original mirror ????? which was going from na002 to na009 _______________________________________________ Toasters mailing list Toasters(a)teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

1 0

snapmirror socket error :vfiler dr configure : 8.1
by Uddhav Regmi 09 Dec '12

09 Dec '12

snapmirror socket error na002 - > na009 ( snapmirror now happening ) source - > dr both are 8.1 ; mirror is happening every 30 min vfiler in na009 is in stopped condition. Aggregate got full here, so we decided to move the dr volume to another filer I did vfiler dr configure command and vol got moved to another box na021 all looks good, vfiler got created. snapmirror was in unitialized state when I start snapmirror : I'm getting this weired error na021> vol restrict vfdoleprd02_db201 Thu Dec 6 15:57:24 EST [na021:vFiler.storageUnit.off:warning]: vFiler vfdoleprd02: storage unit /vol/vfdoleprd02_db201 now offline. Volume 'vfdoleprd02_db201' is now restricted. na021> snapmirror initialize -S na002-sm:vfdoleprd02_db201 na021:vfdoleprd02_db201 Transfer aborted: could not read from socket. na021> Thu Dec 6 15:57:41 EST [na021:replication.dst.err:error]: SnapMirror: destination transfer from na002-sm:vfdoleprd02_db201 to vfdoleprd02_db201 : could not read from socket. na021> snapmirror statusThu Dec 6 15:58:00 EST [na021:replication.dst.err:error]: SnapMirror: destination transfer from na002-sm:vfdoleprd02_db201 to vfdoleprd02_db201 : volume is not online; cannot execute operation. Snapmirror is on. Source Destination State Lag Status na002-sm:vfdoleprd02_db201 na021:vfdoleprd02_db201 Unknown - Pending na002-sm:vfdoleprd02root na021:vfdoleprd02root Uninitialized - Idle na021> snapmirror status -l na021:vfdoleprd02_db201 Snapmirror is on. Source: na002-sm:vfdoleprd02_db201 Destination: na021:vfdoleprd02_db201 Status: Pending Progress: - State: Unknown Lag: - Mirror Timestamp: - Base Snapshot: - Current Transfer Type: Scheduled Current Transfer Error: volume is not online; cannot execute operation Contents: - Last Transfer Type: - Last Transfer Size: - Last Transfer Duration: - Last Transfer From: - na021> any idea what is going on here ???? snapmirror.access list is fine My thought is : do we need to break the original mirror ????? which was going from na002 to na009

2 1

CIFS share hiding qtrees within it
by Andrew Leonard 08 Dec '12

08 Dec '12

Yesterday afternoon around 2p local time, a CIFS share of a FlexVol containing multiple qtree spontaneously began hiding the qtrees within the share (filer is running 8.0.2P3 7-Mode): At the root of the share, non-qtree directories are still visible; qtrees are no longer visible. However, if a user enters the path to the qtree manually, they can access a hidden qtree and files beneath it. By "spontaneously," I mean that /etc/log/auditlog and our homegrown API-based tool confirm that there were no filer configuration changes made. The FlexVol and qtrees are all ntfs mode. The "hidden qtree" behavior has been seen on SMB connections from Windows 7, w2k3 and OS X (at least 10.8, probably other versions). We also share this FlexVol to NFS clients, although usage is light. NFS clients can see and access all qtrees as usual. There were a couple of initial reports from OS X users that they could not connect to the share at all, although I could not confirm or replicate this myself. Share permissions are basic: Name Mount Point Description ---- ----------- ----------- General /vol/general General public share everyone / Change BUILTIN\Administrators / Full Control DAYJOB\Domain Admins / Full Control If we use "Previous versions" on Windows to access the 12p snapshot yesterday, the qtrees are all visible. Using "cacls" or the "Security" tab to view permissions on both hidden and visible directories shows nothing unexpected - and no changes in permissions relative to the 12p snapshot. A "cifs terminate"/"cifs restart" requested by NetApp support did not resolve the issue; they continue to work on it, but assert that they have never seen anything like it before. I'd be grateful for any ideas anyone here might have. Thanks, Andy

2 2

Fiber cards for 2240 ?
by chaim.rieger＠gmail.com 08 Dec '12

08 Dec '12

Trying to bring up two new clusters only to discover that the fiber cards were never ordered. Am in Denver Co. Any ideas ? -- Sent from my Android phone with K-9 Mail. Please excuse my brevity.

3 2

File size and age distribution stats per volume on Ontap8.1?
by Kawakubo, Ken 07 Dec '12

07 Dec '12

Dear List, I am trying to get file size and age distribution stats per volume on a Ontap8.1 system. Apparently, "filestats" command was removed on Ontap8.1 as "filestats" relies on java and java was removed on 8.1. Does anybody know of any quick ways to get this kind of stats on 8.1? Regards, Ken Kawakubo Seattle

8 8

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters December 2012