toasters

toasters@lists.teaparty.net

13514 discussions

Re: volume move and SFO
by Francis Kim 11 Aug '18

11 Aug '18

Here’s a repost, with a link to the TR rather than the actual pdf. https://www.netapp.com/us/media/tr-4075.pdf Francis Kim Cell: 415-606-2525 Direct: 510-644-1599 x334 fkim(a)berkcom.com<mailto:fkim@berkcom.com> www.berkcom.com On Aug 11, 2018, at 12:02 PM, Francis Kim <fkim(a)BERKCOM.com<mailto:fkim@BERKCOM.com>> wrote: Ian, <tr-4075 DataMotion for Volumes NetApp clustered Data ONTAP 8.2 and 8.3.pdf> With recurring controller and shelf refreshes, I’ve become a frequent vol mover (and a fan of it) over the last two years. NDO aspect of this task is obviously a big draw. Attached TR-4075, dated March 2015, has the most detailed explanation of vol moves I’ve been able to find, but its discussion is limited to 8.2 vs 8.3. Nothing about ONTAP 9 in this TR, so YMMV. In this TR much is made about the two phases of a vol move, the iterative (baseline and updates) and cutover, with respective to what other operations are (in)compatible with the progress of a vol move. During the iterative phase, a vol move in 8.2 would have to be restarted after a FO/GB while in 8.3 they would resume at its most recent checkpoint. However, once the cutover phase has been entered, a vol move in 8.2 would survive a FO/GB if it had crossed its “point of no return” checkpoint, while an 8.3 vol move is mutually exclusive with a FO/GB, suggesting a cutover would have to be reattempted afterward. I’ve not been able to find under 9.1 (even in diag mode) any information specific to these checkpoints. Not sure whether “Bytes sent” is a checkpoint. Since documentation on vol moves is generally skinny and this TR is now over three years and five releases old, a lab run is probably a good move if you have access to gear. Francis Kim Cell: 415-606-2525 Direct: 510-644-1599 x334 fkim(a)berkcom.com<mailto:fkim@berkcom.com> www.berkcom.com<http://www.berkcom.com> On Aug 10, 2018, at 2:55 AM, Ian Ehrenwald <Ian.Ehrenwald(a)hbgusa.com<mailto:Ian.Ehrenwald@hbgusa.com>> wrote: Based on a Justin Parisi article that Douglas pointed out earlier in the thread, I'm going with the assumption that the cluster would veto the giveback if a move was in progress. I actually just got a new FAS2720 HA pair for my lab this week, so when it's set up and I have a bunch of junk data and volumes on it for testing, I will see what happens if we were to do a move, takeover, and giveback in Real Life (tm). I can't make any promises on an ETA for answer, but as soon as I do it I'll send an update. ________________________________________ From: RUIS Henk <henk.ruis(a)axians.com<mailto:henk.ruis@axians.com>> Sent: Friday, August 10, 2018 4:28:30 AM To: Ian Ehrenwald; Mike Gossett Cc: Toasters Subject: RE: volume move and SFO Hi, Great, but now we still don't know if it's possible to move a volume during maintenance ;-( Met vriendelijke groet / Kind regards, Henk Ruis Technical Consultant -----Oorspronkelijk bericht----- Van: toasters-bounces(a)teaparty.net<mailto:toasters-bounces@teaparty.net> <toasters-bounces(a)teaparty.net<mailto:toasters-bounces@teaparty.net>> Namens Ian Ehrenwald Verzonden: donderdag 9 augustus 2018 18:13 Aan: Mike Gossett <cmgossett(a)gmail.com<mailto:cmgossett@gmail.com>> CC: Toasters <toasters(a)teaparty.net<mailto:toasters@teaparty.net>> Onderwerp: Re: volume move and SFO Good afternoon Update, as promised. The volume move completed early this morning so there will be no conflict with our maintenance tonight. Inline compaction is AWESOME - it has saved us over 16TB on this new SSD aggregate. Crazy stuff. ________________________________________ From: Mike Gossett <cmgossett(a)gmail.com<mailto:cmgossett@gmail.com>> Sent: Wednesday, August 8, 2018 2:37:35 PM To: Ian Ehrenwald Cc: Douglas Siggins; Toasters Subject: Re: volume move and SFO Hi Ian, The good news is that, forgetting about what it is estimating, we've seen that in 24 hours 21TB has been copied and. Hopefully another 30 hours or whatever is sufficient for the remaining 9.5TB - thanks for sharing. I'm interested to know the result of the SFO - but if was me I'd try to push the maintenance back to be safe. Or open a ticket with support and see if they can tell you what to expect On Aug 8, 2018, at 1:29 PM, Ian Ehrenwald <Ian.Ehrenwald(a)hbgusa.com<mailto:Ian.Ehrenwald@hbgusa.com>> wrote: Hi Mike Somewhat redacted output from a few minutes ago: MyCluster1::> volume move show -instance Vserver Name: mySvm Volume Name: aVeryLargeVolume Actual Completion Time: - Bytes Remaining: 9.44TB Destination Aggregate: aggr_ssd_3800g_c1n4 Detailed Status: Transferring data: 20.96TB sent. Estimated Time of Completion: Thu Aug 09 01:58:40 2018 Managing Node: Clus1-Node1 Percentage Complete: 68% Move Phase: replicating Estimated Remaining Duration: 11:45:25 Replication Throughput: 233.9MB/s Duration of Move: 23:47:43 Source Aggregate: aggr_sas_600g_c1n1 Start Time of Move: Tue Aug 07 14:25:38 2018 Move State: healthy Is Source Volume Encrypted: false Encryption Key ID of Source Volume: - Is Destination Volume Encrypted: false Encryption Key ID of Destination Volume: - MyCluster1::> Depending on source filer load, I've seen Replication Throughput anywhere from 25MB/s through 400MB/s and higher. The source aggregate is 192x600GB SAS on a filer with 2TB FlashCache and it sometimes sees periods of 30K IOPS. I guess the point is ETA has been anywhere from just about now all the way out to this coming Sunday. There's a good chance that the move will complete before tomorrow evening according to these numbers though. If it doesn't complete on time, I guess we'll find out what effect, if any, SFO has (unless I can easily reschedule this HW maintenance). ________________________________________ From: Mike Gossett <cmgossett(a)gmail.com<mailto:cmgossett@gmail.com>> Sent: Wednesday, August 8, 2018 13:59 To: Douglas Siggins Cc: Ian Ehrenwald; Toasters Subject: Re: volume move and SFO How long did it take to get to your 60% point that you reference? Some quick math says 30TB at 400MB/s should complete in about 22 hours. If you're 60 percent done, and you have a day and a half, then it should complete before tomorrow night, right? _______________________________________________ Toasters mailing list Toasters(a)teaparty.net<mailto:Toasters@teaparty.net> http://www.teaparty.net/mailman/listinfo/toasters<https://protect-us.mimecast.com/s/bF00ClYkMYUo703Os9r2z6?domain=teaparty.net> _______________________________________________ Toasters mailing list Toasters(a)teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

1 0

volume move and SFO
by Ian Ehrenwald 10 Aug '18

10 Aug '18

Good morning I have a four node cluster, nodes1/2 are SAS/SATA and nodes 3/4 are AFF. I have a long running volume move going from node 1 to node 4. Long running, like 30TB+, and it's about 60% done. I need to do some hardware maintenance on nodes 1 and 2 tomorrow evening (install additional FlashCache cards). Will a takeover of node 2 by node 1, then a takeover of node 1 by node 2, interrupt this volume move? I can't seem to find much in the way of documentation about what happens during a SFO and a volume move, but it's possible I'm just not looking hard enough. Thanks for any insights. Ian Ehrenwald Senior Infrastructure Engineer Hachette Book Group, Inc. 1.617.263.1948 / ian.ehrenwald(a)hbgusa.com

4 9

Retrieving lock status in cDOT (CIFS/NFSv4)
by Jacek 08 Aug '18

08 Aug '18

We implemented simple file lock monitoring on our 7mode filers: A script runs each 5 minutes retrieving output from lock status -n command from all vfilers. For example: # ssh [...] vfiler run sngnas100703 lock status -n CIFS: 0 share locks, 0 byte locks [...] NFSV4: 29 share locks, 5 byte locks [...] Total Locks: 29 share locks, 5 byte locks [...] We would like to implement equivalent solution for cDOT but there is no command returning lock status summary (all we need is to count share and byte locks per protocol: CIFS and NFSv4). cDOT documentation and NetApp support suggest using command: vserver locks show But: Command warns that it may affect system performance and we will query clusters each couple of minutes. The script will have to retrieve huge output, parse it and calculate required totals. What is the best and non-intrusive way to retrieve locks status? (other commands? API?) Best regards, Jacek Nogala

2 2

wafl.vol.walloc.rsv.failmount
by Mike Thompson 08 Aug '18

08 Aug '18

Looks like we filled up an aggregate that we dump backups into. Filer bounced itself, to take the aggregate offline, which is an interesting way to deal with the situation: Aug 6 01:39:46 filer1b [filer1b:wafl.vol.walloc.rsv.offline:error]: Volume backups free space reserve is critically low. Rebooting system to take volume offline. Please contact support for assistance. Anyway, the aggr is in restricted state now, and when I try to online it, event log says: 8/7/2018 08:43:46 filer1b WARNING wafl.vol.walloc.rsv.failmount: Failing mount of volume backups. Free space reserve is too low. Please contact support for assistance. I'd really like to not have to blow this aggr away and re-run all these backups. Presumably there's a way to get this back online. This is on cdot 8.2.4p4 Any ideas?

5 6

Re: SnapMirror Bi-directional possible?
by Michael Bergman 26 Jul '18

26 Jul '18

On 2018-07-24 23:35, Scott M Gelb via Toasters wrote: > For the same data set, bi-directional, you might want to POC Peer Software. > They PeerSync bi-directional and also have PeerLink which locks files on > either side while keeping in sync. FPOLICY is used for real time event > notification without file scans. Locks files... Well, if you can live with that, then *maybe* it's viable. But cumbersome and slow. Now, in many use cases w.r.t. NFS (which is what is used by the Applications to *do* something with that data right?) the problems which such locking incur can be devastating in practice. If it's just some ppl browsing files manually, then sure -- go ahead. But with many heavy application systems it's just not acceptable. So the whole things falls over in any case. This might not be detectable in a PoC; race conditions. It's a bogmire of ifs and buts. In a way normal one-directional SM also "locks" (or rather, makes not-available) files and dirs in the RO target volume. In a certain way. So if you run applications intensely on the RO copy (target FlexVol) via NFS there is always a short moment when inodes are shifted at the updates. This is absolutely unavoidable, but what will happen to the App is then 'Stale NFS file handle'. Ka-boom. So then you go into the complexity of trying to asynchronously time the app chain to not run (in certain ways at least) at the exact second(s) this shift inside the inode structure happens. And then you give up. Been there, done that. /M

5 4

mirror one volume read/write across snapmirror?
by John Stoffel 26 Jul '18

26 Jul '18

>>>>> "Scott" == Scott M Gelb via Toasters <toasters(a)teaparty.net> writes: Scott> From: Scott M Gelb <scottgelb(a)yahoo.com> Scott> To: "toasters(a)teaparty.net" <toasters(a)teaparty.net>, Eric Peng <epeng(a)esri.com> Scott> Date: Tue, 24 Jul 2018 21:34:46 +0000 (UTC) Scott> Subject: Re: SnapMirror Bi-directional possible? Scott> For the same data set, bi-directional, you might want to POC Peer Software. Scott> They PeerSync bi-directional and also have PeerLink which locks files on either Scott> side while keeping in sync. FPOLICY is used for real time event notification Scott> without file scans. And from my experience with fpolicy and CommVault for HSM (Archiving of old files) it's fragile as all hell and not reliable. When fpolicy goes down... all hell will break loose. And what happens when you have two different people editing the same file at seperate sites? Last edit wins? This way lies sanity... you need a cluster aware filesystems. Of course people will point out that you have the same problem with NFS if you edit the same file from two seperate clients at the same site. It just silently uses the last one to write is the answer. But for remote sites with fpolicy, I'm sure it will be even more fragile. John

1 0

Max volume count per node
by Alexander Griesser 26 Jul '18

26 Jul '18

Hey there, does anyone know what the real implication of the max volume count per node setting on the FAS systems is good for? Is there any chance that this limit gets raised on newer OnTap releases or is there any secret flag to raise it (knowing that support would not want to see that)? Best, Alexander Griesser Head of Systems Operations ANEXIA Internetdienstleistungs GmbH E-Mail: AGriesser(a)anexia-it.com<mailto:AGriesser@anexia-it.com> Web: http://www.anexia-it.com<http://www.anexia-it.com/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

3 4

Re: SnapMirror Bi-directional possible?
by Klise, Steve 25 Jul '18

25 Jul '18

You can also look at somthing like DFS or a global name space like panzura. I believe they can do multiple copies but have some hardware requirements. On July 24, 2018 at 4:40:09 PM PDT, Mike Gossett <cmgossett(a)gmail.com> wrote: You could look at metrocluster, potentially Sent from my mobile phone > On Jul 24, 2018, at 5:53 PM, Eric Peng <epeng(a)esri.com> wrote: > > Thanks all, for confirming that bi-directional SnapMirror on the *same* volume is not possible. We're still in the requirements gathering/planning part of the project, so we have some time to figure this out. This would be for keeping user profiles in sync between 2 different sites. I may take a look at that PeerSync software. > > Eric > > -----Original Message----- > From: toasters-bounces(a)teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Michael Bergman > Sent: Tuesday, July 24, 2018 3:07 PM > To: Toasters <toasters(a)teaparty.net> > Subject: Re: SnapMirror Bi-directional possible? > >> On 2018-07-24 23:35, Scott M Gelb via Toasters wrote: >> For the same data set, bi-directional, you might want to POC Peer Software. >> They PeerSync bi-directional and also have PeerLink which locks files >> on either side while keeping in sync. FPOLICY is used for real time >> event notification without file scans. > > Locks files... Well, if you can live with that, then *maybe* it's viable. > > But cumbersome and slow. Now, in many use cases w.r.t. NFS (which is what is used by the Applications to *do* something with that data right?) the problems which such locking incur can be devastating in practice. If it's just some ppl browsing files manually, then sure -- go ahead. > > But with many heavy application systems it's just not acceptable. So the whole things falls over in any case. This might not be detectable in a PoC; race conditions. It's a bogmire of ifs and buts. > > In a way normal one-directional SM also "locks" (or rather, makes > not-available) files and dirs in the RO target volume. In a certain way. So if you run applications intensely on the RO copy (target FlexVol) via NFS there is always a short moment when inodes are shifted at the updates. This is absolutely unavoidable, but what will happen to the App is then 'Stale NFS file handle'. Ka-boom. So then you go into the complexity of trying to asynchronously time the app chain to not run (in certain ways at least) at the exact second(s) this shift inside the inode structure happens. > > And then you give up. Been there, done that. > > /M > _______________________________________________ > Toasters mailing list > Toasters(a)teaparty.net > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.teaparty.net_mailma… > > _______________________________________________ > Toasters mailing list > Toasters(a)teaparty.net > http://www.teaparty.net/mailman/listinfo/toasters _______________________________________________ Toasters mailing list Toasters(a)teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

1 0

SnapMirror Bi-directional possible?
by Eric Peng 24 Jul '18

24 Jul '18

Just wanted to do a quick sanity check, is there any way to have SnapMirror replication go both directions between sites? We are running cDOT 9.x at both locations. I came across some older posts (several years ago) that seem to indicate that bi-directional snapmirror would not be possible. Just wondering if this might have changed with cDOT 9.x capabilities? Thanks, Eric

5 4

Mac OS X, SMB NTFS, and chmod/chown
by Ehrenwald, Ian 12 Jul '18

12 Jul '18

Hello I have users on Mac OS X that are unable to uncompress zip files in SMB shares which sit on top of NTFS security style volumes. I hadn't heard of this problem until I started migrating my users to new shares, previously they were all on UNIX security style volumes. If I am understanding my users correctly, they can reliably reproduce this problem this way: - In Finder, right click on a folder on a share, select Compress - Zip file is created successfully on share - Double click on Zip file that was created - Mac OS X archiver utility pops up an error box, "Unable to expand (zip name) into (target dir). (Error 13 - Permission denied). Examining the Zip file from the command line shows that it is a fully formed file, no corruption, etc. When I attempt to uncompress the Zip file using the command line on the same machine that created it, I can see that unzip is attempting to chmod the files and directories and getting a Permission Denied error back from the svm. Output: UserMachine:z TheUser$ unzip 3\:14\:18.zip Archive: 3:14:18.zip creating: 3:14:18/ chmod (directory attributes) error: Permission denied inflating: 3:14:18/FileNameXYZ.FC3.idml fchmod (file attributes) error: Permission denied creating: __MACOSX/ chmod (directory attributes) error: Permission denied creating: __MACOSX/3:14:18/ chmod (directory attributes) error: Permission denied inflating: __MACOSX/3:14:18/._FileNameXYZ.FC3.idml fchmod (file attributes) error: Permission denied inflating: 3:14:18/FileNameABC.ai fchmod (file attributes) error: Permission denied inflating: __MACOSX/3:14:18/._FileNameABC.ai fchmod (file attributes) error: Permission denied inflating: 3:14:18/FileNameLMN.png fchmod (file attributes) error: Permission denied inflating: __MACOSX/3:14:18/._FileNameLMN.png fchmod (file attributes) error: Permission denied inflating: 3:14:18/FileNameXYZ.FC3.indd fchmod (file attributes) error: Permission denied inflating: __MACOSX/3:14:18/._FileNameXYZ.FC3.indd fchmod (file attributes) error: Permission denied warning: cannot set permissions for __MACOSX/3:14:18/ Permission denied warning: set times/attribs failed for __MACOSX/3:14:18/ warning: cannot set permissions for __MACOSX/ Permission denied warning: set times/attribs failed for __MACOSX/ warning: cannot set permissions for 3:14:18/ Permission denied warning: set times/attribs failed for 3:14:18/ UserMachine:z TheUser$ This does make sense, since SMB/CIFS doesn't really understand UNIX ownership and attributes. Is the solution setting the security style of the volume to Mixed? Reading in detail about it at https://whyistheinternetbroken.wordpress.com/2017/01/24/mixed-perceptions-m… makes it seem like that's the right path to follow, but before I start down that trail I wanted to see what other people have done in this situation. Thanks for your help. Ian Ehrenwald Senior Infrastructure Engineer Hachette Book Group, Inc. 1.617.263.1948 / ian.ehrenwald(a)hbgusa.com This may contain confidential material. If you are not an intended recipient, please notify the sender, delete immediately, and understand that no disclosure or reliance on the information herein is permitted. Hachette Book Group may monitor email to and from our network.

7 14

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters