toasters March 2012

toasters@lists.teaparty.net

44 participants
23 discussions

V-series / WAFL performance / 20% free
by Stuart Kendrick 04 Apr '12

04 Apr '12

Hi folks, I'm familiar with the NetApp recommendation to keep ~20% free on all volumes ... this allows WAFL to maximize its performance (convert random IO to streaming IO ... i.e. the 'Write Anywhere ...' part of WAFL). But I'm skeptical that this recommendation applies to V-Series installations, i.e. situations in which the backend array was built by someone other than NetApp. Seems to me that with V-Series, WAFL may find what it thinks are 128K of contiguous blocks on storage and perform the WRITE ... but ... since the backend array has virtualized the blocks in some fashion, opaque to ONTAP, there is no knowing what really happens. In other words, the blocks which ONTAP specifies do /not/ map to physical blocks ... once the WRITE arrives at the array, the backend will do whatever it thinks best with them, perhaps perform a streaming write to contiguous blocks, perhaps not. Implementation-specific. -In fact, with a V-Series, I would imagine (I have a rich imagination) that ONTAP doesn't bother to even try to convert random IO into streaming IO -- I would imagine that it just hands the blocks to the backend in a jumble and says "Do whatever". So, I would like to claim that with V-Series, I can fill up my volumes without impacting write performance ... or, more precisely, that impact is dependent on the characteristics of the backend ... perhaps the backend needs free space in order to optimize its write performance, perhaps it does not. [In our particular case, we're using 3Par, and 'fullness' would not impact performance.] Would anyone see holes in my thinking? --sk Stuart Kendrick FHCRC

2 2

Operations Manager / Backup Manager.
by Jeff Cleverley 31 Mar '12

31 Mar '12

Greetings, We have a 3.8 version of OM/DFM that has been managing backups from multiple sites to a NearStore. We just moved the shelves to a different NearStore yesterday due to disk count limitations. Everything went well except OM will not recognize the new destination for the relationships. A snapvault status on the secondary shows all the relationships and the aggregates and volumes are online. We did a snapvault start -S of the relationships, a snapvault update, and also a snapvault release on the primaries. The primaries all show the correct relationships. OM can see the new destination filer and all the volumes, so it has access. It has also figured out the original NearStore volumes and relationships are not there because it removed them, and also emptied the schedules of any backup relationships. We tried a "Add new backup" which seemed to have worked except it did an Initialize of the relationship according to the snapmirror log file and seems to have done a baseline. We cannot re-baseline all of these relationships. Does anyone know how to get these relationships into OM without having to do a new baseline? Thanks, Jeff -- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611

3 5

Oracle (RHEL 5.7) on NFS (Ontap 8.1RC2) - complains of write
by Fletcher Cocquyt 30 Mar '12

30 Mar '12

Hi, I'm not seeing write performance issues on the OS (throughput tests on the command line are > 100Mb/sec) But Oracle is complaining about long commit times for the Netapp NFS mount with the following options: rw,noatime,bg,hard,nointr,tcp,nfsvers=3,timeo=600,rsize=32768,wsize=32768 (matches what I see in Oracle and Netapp docs) I'm working with the DBA to see if the Oracle tuning is optimal, but thought I'd poll this list for any nuggets of tuning Oracle on Netapp NFS And the disk busy on this aggregate is only 10% - and the CPU reads 40-50% thanks! Fletcher

7 10

Re: Oracle (RHEL 5.7) on NFS (Ontap 8.1RC2) - complains of write performance
by Fletcher Cocquyt 29 Mar '12

29 Mar '12

> Hi Dan and all - thanks for the great info! > > We'll check out dnfs and I've downloaded Netapp's SIO tool > > Will check > Flow Control (this is usually an issue if you see 10g/1g buffer overruns which could be the case since our 3270 is 10g and this client is 1g…) > test Jumbo Frames (this is challenging since it requires end to end Netapp<==switch==>client config and not all clients can/will do jumbo - bu maybe there is a hybrid where some clients are jumbo and others not?) > > Which nfsiostat tool are you referring to? > > http://sebastien.godard.pagesperso-orange.fr/man_nfsiostat.html ? Or > http://nfs-utils.sourcearchive.com/documentation/1.2.0/nfs-iostat_8py-sourc… ? > > I've also engaged Confio with their Ignite trial (which I've used to help refute server/storage vs app/oracle wait times - highly recommend it) > > > thanks again > > > > On Mar 28, 2012, at 8:03 PM, Daniel Brown wrote: > >> If this is oracle 11g, use dnfs. It will avoid mount option questions, as well as any weirdness at the kernel Nfs level. The noatime and actimeo=0 can make a big difference. Really simple to get dnfs setup. >> >> Double check mount options, looking for differences with rac vs single instance. You didn't say, and I know the docs differ for each, but aren't always very clear. go with the NetApp kb article settings. >> >> Check jumbo frames. You probably want them. >> >> Look at network Configs. We just worked on one where the issue was flow control at the network level. Getting that set right meant things worked lots better. There is a NetApp kb about it. >> >> Test with netapps sio tool. Note that oracle mostly uses direct io to bypass os level caching. Os level tools like dd or copy won't. NetApp sio offers it as an option. >> >> Install the nfsiostat tools to monitor what's going on. >> >> >> >> -- >> Dan Brown >> >> On Mar 28, 2012, at 7:42 PM, Fletcher Cocquyt <fcocquyt(a)stanford.edu> wrote: >> >>> Hi, I'm not seeing write performance issues on the OS (throughput tests on the command line are > 100Mb/sec) >>> >>> But Oracle is complaining about long commit times for the Netapp NFS mount with the following options: >>> >>> rw,noatime,bg,hard,nointr,tcp,nfsvers=3,timeo=600,rsize=32768,wsize=32768 (matches what I see in Oracle and Netapp docs) >>> >>> I'm working with the DBA to see if the Oracle tuning is optimal, but thought I'd poll this list for any nuggets of tuning Oracle on Netapp NFS >>> >>> And the disk busy on this aggregate is only 10% - and the CPU reads 40-50% >>> >>> thanks! >>> >>> Fletcher >>> >>> >>> >>> >>> >>> _______________________________________________ >>> Toasters mailing list >>> Toasters(a)teaparty.net >>> http://www.teaparty.net/mailman/listinfo/toasters >

1 0

Question regarding snapmirror when the source changes
by Michael Homa 22 Mar '12

22 Mar '12

Hi. Maybe someone can help with a question. I would have submitted it to netapp tech support but was unable to describe the question in 250 characters or less. I apologize for the length. We have three systems: o our production 270 (ONTAP 7.2.7) called netapp1 o a 2040, a relacement for the 270 (ONTAP 7.3.4), called netapp1-new o a 2040, used as a backup system, called netapp2 Currently, all of the production volumes (flexible, not Qtrees) on the 270 (netapp1) are being snapmirrored to both netapp1-new and netapp2 though it is not a cascade operation. Netapp1 is the source, separately, for the two 2040s. Running snapmirror destinations on netapp1 shows me the separate destinations. In the near future I will replace the 270 with the 2040 called netapp1-new. I will: o break the snapmirror relationship between the netapp1-based volumes on netapp1 and the corresponding snapmirrored volumes on netapp1-new, resulting in those netapp1-new volumes now being read/write and flexible, o shutdown netapp1, the 270, o run setup on netapp1-new, giving it the same EXACT configuration (IP, hostname, etc) as the now-shutdown 270, and o reboot the netapp1-new box, resulting it now being called netapp1 with the appropriate volumes. I've swapped out the 270 and swapped in the 2040. But, the 2040 looks for all intents and purposes like it was the 270. Same hostname, same IPs. Exports, hosts.equiv, and netgroup will all be taken care of. Here comes the issue. As I said, the netapp1 volumes, on the 270, were also being snapmirrored to netapp2, the second of the 2040s. Now that netapp1, the snapmirror source, is a 2040, can I continue to run snapmirror update from netapp2 on the netapp1-based volumes or is there no relationship anymore, forcing me to run snapmirror initialization from netapp2 on all the netapp1 volumes? Worse case situation (not a deal breaker) - it will take one or two days to re-initialize versus the two hours the updates take. Anyways, if someone could shed some light or point me to some info that would be great. I didn't find anything in the Snapmirror Overview and Best Practices Guide. Michael Homa Enterprise Systems and Development Group Academic Computing and Communication Center University of Illinois at Chicago email: mhoma(a)uic.edu

3 2

Snapmirror Sync
by Milazzo Giacomo 22 Mar '12

22 Mar '12

Hi all, what is your opinion and your experience, if any, on using Snapmirror Sync. I mean what's about media, bandwidth limits, special configuration, overhead on CPU and so on. Can this replication method used for environments where both RPO and RTO should be around zero? Regards, Dott. Giacomo Milazzo [Descrizione: cid:image001.jpg@01C75422.0D316270] Technical Account Manager Sinergy SpA Filiale di Roma * 00198. viale Regina Margherita, 269 ' (+39) 3406001045 0644245272 7 (+39) 0226922048 * Giacomo.Milazzo(a)Sinergy.it

1 0

least busy aggr report?
by Fletcher Cocquyt 22 Mar '12

22 Mar '12

While looking to locate the next Oracle VM I am wishing there was a least busy aggr report. I've been doing spot tests to decide, but it would be great if in addition to the "days to full" report there was an analogous IOPS capacity report. thanks, Fletcher http://vmadmin.info

1 0

re[4]: Physical reallocate for a deduped volume
by Mathew Kilham 21 Mar '12

21 Mar '12

1 0

Query Netapp for NFS client mount options?
by Fletcher Cocquyt 21 Mar '12

21 Mar '12

I'd like to know which NFS clients are mounted with the noac option. Can I query OnTap 8.1RC2 for this info? If so, how? thanks, Fletcher.

2 1

Physical reallocate for a deduped volume
by Mathew Kilham 20 Mar '12

20 Mar '12

Hi Toasters, I've posed the following question to NetApp support but wasn't too confident in the answer I received, so I was hoping the knowledgeable folks on this list could assist? Also, before I begin, please excuse any ignorance or NetApp faux pas on my behalf, I'm far from an expert at administering NetApp filers. --- We have a FAS2050C running DOT 7.3.4. We recently added an external disk shelf to the filer and performed a bit of reorganisation, which has left us with two unused disks in the internal shelf. We want to add these two disks to an existing 16-disk aggregate by expanding it's RAID group size from 16 to 18. The aggregate is currently ~80% full and contains 5 flexvols, most of which have de-dupe and snapshots enabled. I understand that after expanding the RG / aggregate we need to run a physical reallocate or else we will end up with a significant "hot spot" on the newly-added disks. I also understand that we need ensure DOT only does a physical redistribution of the blocks across all disks in the aggregate and does not try to otherwise rearrange them, otherwise we risk "un-deduping" our deduplicated data and/or greatly increasing the space consumed by our snapshots. First question: I believe the command we need to run to perform the above style of reallocation is "reallocate start -f -p /vol/<volname>" - is that correct? I noted, however, that there is a warning with regards to the "reallocate -p" command in the release notes of 7.3.x. Unfortunately I can't find a link anymore, but it reads: "A file reallocation scan using reallocate start or reallocate start -p does not rearrange blocks that are shared between files by deduplication on deduplicated volumes. Since a file reallocation scan does not predictably improve read performance when used on deduplicated volumes, performing file reallocation on deduplicated volumes is not recommended. Instead, for files to benefit from the reallocation scan, they should be stored on volumes that are not enabled for deduplication." I then queried with NetApp how we could perform a physical reallocate on the aggregate to avoid a hot spot, given the above statement that "reallocate start -p does not rearrange blocks that are shared between files by deduplication on deduplicated volumes", and almost all of our data is deduped. After a bit of to-ing and fro-ing with NetApp, the final suggestion was as follows: "Ive talked to two senior resources so far about this. The suggested steps are as follows>> #1: add your disks to the aggregate, and then grow the volume. #2: Turn off Deduplication, ( no, this will not revert to the 2.5 gigs from the one gig ratio it is now and suddenly cause you to be out of space) this will only stop deduplication from this point forward until it is re-enabled later. Everything that is currently deduped will stay the ratio it is. #3: Perform the reallocate p command as specified in the KBs sent earlier. #4: After all above steps are performed and reallocate is done, re-enable deduplication on said volumes." Second question: Does this sound kosher - will this actually result in a physical reallocate being performed, or will the deduped blocks still be skipped because they're already de-duplicated, regardless of whether dedupe is enabled for new data on the volume or not? Third question: If yes to the previous question, is there any risk that our deduped data get un-deduped by performing the suggested steps above? Fourth and final question: Is there any way to measure how data is spread across disks in a Raid Group / aggregate, so that I can check that we get our expected results from the reallocate? Thanks everyone, really appreciate any feedback / assistance. Cheers, Matt

3 4

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters March 2012