toasters July 2015

toasters@lists.teaparty.net

47 participants
40 discussions

Hot Shelf Removal - check for mailbox disks first before remove ownership
by Gelb, Scott 24 Jul '15

24 Jul '15

The hot shelf removal feature is great, but the documentation at https://library.netapp.com/ecmdocs/ECMP1367947/html/GUID-2B80FBD2-007D-4D19… is missing a check. An edge case if you had mroot on those disks prior, and then moved mroot to other disks/shelves for the hot removal. If the disks on the shelf or shelves have mailbox disks, right after you remove ownership from the disks (even though all zeroed spares), the node will crash. The ha-partner will not be able to takeover. Then the ha partner will also crash with an inconsistent mailbox. We had both nodes drop after removing ownership from a shelf that was being removed. It is an easy fix to get to maintenance mode and mailbox destroy local, but add that to your run book for hot shelf removal. We didn't have any data on the ha-pair for the maintenance (the great thing about cDOT which averted any outage even during this maintenance). We recovered after reassigning the disks and deleting the mailbox (had to do that twice) then both nodes came back up after coredump and mailbox was recreated and storage failover was able to enable again. ::> set adv ::*> stor failover mailbox-disk show One other note on the move mroot procedure https://kb.netapp.com/support/index?page=content&id=3013873&locale=en_US The article now lists to turn on nvfail (submitted that update a while ago), but some systems also had create_ucode enabled on vol0, so I check "vol options vol0" prior to move, then match the settings after the move in addition to nvfail. Also, if you have cdpd.enable on and ip.fastpath.enable off, both of those need to be reset since mroot move sets them back to cdpd off and fastpath on defaults (8.3P1).

3 2

Cluster evolution
by Milazzo Giacomo 22 Jul '15

22 Jul '15

Hi everybody, suppose that for technical (i.e. a 3240HA controller pair that does not fit with a 8060HA one in the same cluster) reason I've to start with two different 2 nodes cluster, each of the two managing its respective aggregates and node. So I can have now: CLUSTER1-01 and CLUSTER1-02 (3240HA) and CLUSTER2-01 and CLUSTER2-02 (8060HA) In the future customer could have budget to do a tech refresh on the first cluster changing the heads to an 8000 pairs. Well, what's about the merge of the CLUSTER1 nodes to the CLUSTER2 preserving all data in CLUSTER1 aggregates? So I mean that I would have in the end a 4 nodes cluster and not two separate 2 nodes ones. Is there some procedure? Regards.

6 15

FAS-2554 cDOT sanity check requested
by Roy McMorran 22 Jul '15

22 Jul '15

Hi all, We recently purchased a NetApp FAS-2554 and I'm getting what I feel is less than stellar performance. I have a case open with support. But I'm trying to 'calibrate my expectations' a bit, so I thought I'd send this out to the list. Any feedback would be appreciated. Here's a some info about the configuration cDOT, 8.3.1RC1 2-node switchless cluster 22 SATA drives (3TB) in two RAID-DP aggregates + 2 spares 22 10K SAS drives (900GB) in two RAID-DP aggregates + 2 spares No flash (yet?) 10Gb cluster interconnect 10Gb to server(s) with jumbo frames 2 vFilers - the one under test has a volume on the sas_data_1 aggregate which is serving NFS v3 (TCP) I'm currently testing with a Cisco UCS C220 M4S running CentOS 6.6 as the NFS client. We noticed that performance (on a database load) seemed poor, so I started some benchmark testing with "iozone". The test I'm looking at in particular is sequential writes with an 8K record size. The write throughput was 2MB/S! There are no other workloads on the filer or the server. NetApp suggested a workaround for the bug described here: http://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=876563 and that actually improved things - my sequential write throughput test can now do 15MB/S. Definitely better, but that still seems slow to me. So with all that background here is my question - does anyone have a similar configuration out there, and what sort of write speeds are you seeing? I'm just looking for a sanity check. Is 15MB/S reasonable? Thanks! Roy McMorran

9 18

FEAR report
by Brad Thompson 22 Jul '15

22 Jul '15

Has anyone ever heard of a NetApp FEAR report? I've had several people ask me about running it, but I can't figure out what they are talking about. -Brad

4 8

Latency Counters - Perfmon versus Resource Monitor
by Philbert Rupkins 21 Jul '15

21 Jul '15

Hello Toasters, I have a situation in which Resource Monitor shows 100 - 400 ms disk Response Time (latency) for various processes in the Disk-Disk Activity tab. Resource Monitor doesnt break Response Time down into read/write/other so I hopped over to perfmon. I enabled the following perfmon metrics for the block device showing 100 - 400ms response time in Resource Monitor: Physical Disk: Avg. Disk Bytes/Read Avg. Disk Bytes/Write Avg. Disk Bytes/Transfer Physical Disk: Avg. Disk sec/Read Avg. Disk sec/Write Avg. Disk Sec/Transfer Those permon counters were chosen to have a look at block size and latency. Perfmon reported very large values for Avg. Disk Bytes/Read (up to 500,000) which may explain some of the higher Response Time numbers in Resource Monitor. HOWEVER, perfmon showed very low numbers for Avg. Disk sec/Read whereas Resource Monitor showed 100 - 400 ms. Does anybody know what might account for the difference in Response Times between perfmon and Resource Monitor? Specifically, the difference between Avg. Disk sec/Transfer (perfmon) and the Response Time column in Resource Monitor? Windows Server 2008, Exchange 2010, NetApp v3250 running 8.1.4P3 7-Mode. All LUNS presented via iSCSI. -Phil

1 0

Nondisruptively change language of a volume containing LUNs?
by Basil 20 Jul '15

20 Jul '15

Hi folks, One of my colleagues made a typo on the language for volumes that are used exclusively for LUNs. I know that you can't change the language of a volume, but is there a way to move the LUNs to a new volume with the correct language set? Thanks!

4 6

RE: SVM DR in 8.3.1 - AGAIN
by Borzenkov, Andrei 20 Jul '15

20 Jul '15

Original question was - is it possible to start DR SVM with identity-preserve without stopping SnapMirror. It looks like it is possible. Regarding “future snap mirror updates” - I mentioned that if I switch back from identity-preserve=false to identity-preserve=true changes made in the past (during identiry-preserve=false relationship) are replicated. They are even applied to a running DR SVM, and e.g. LIF gets configured while DR SVM is running. I do not promote or endorse it - as I said “I am not sure if it works as intended”. But I can see use cases for it. From: Moore, Doug [mailto:Doug.Moore@netapp.com] Sent: Sunday, July 19, 2015 7:44 AM To: Borzenkov, Andrei Cc: NGC-g.milazzo-sinergy.it; Parisi, Justin; Toasters Subject: Re: SVM DR in 8.3.1 - AGAIN What are you trying to accomplish? The setup you have below is clearly not a use case that we advise. Anything that is sent in the initial ID preserve that is not then sent in the id discard will not be replicated in future snap mirror updates. So why would you do it that way? Doug Moore Technical Marketing Engineer Clustered Data ONTAP/Multi-Tenancy NetApp 303.349.4851<tel:303.349.4851> Direct / Mobile doug.moore(a)netapp.com<mailto:doug.moore@netapp.com> www.netapp.com<http://www.netapp.com/> On Jul 18, 2015, at 13:36, Borzenkov, Andrei <andrei.borzenkov(a)ts.fujitsu.com<mailto:andrei.borzenkov@ts.fujitsu.com>> wrote: Not sure if it works as intended, but - first create snapmirror with identity-preserve true; then delete (without breaking) and create again with identity-preserve false. Now you have SVM that contains original configuration but can be started without breaking SnapMirror. To continue just resync. OTOH it is logical. In case of identity-preserve is false, network configuration of DR SVM is not changed at all and where it came from originally does not matter. If you delete snapmirror and recreate with identity preserve again, changes made in between on source are propagated to DR. Of course, it does not have ability to select arbitrary snapshot as starting point (and as most part of SVM configuration is not kept in volume, it would not help anyway). I do not know if RDB even has anything similar to snapshots that could be employed here. From: toasters-bounces(a)teaparty.net<mailto:toasters-bounces@teaparty.net> [mailto:toasters-bounces@teaparty.net] On Behalf Of Milazzo Giacomo Sent: Wednesday, July 08, 2015 9:06 PM To: Parisi, Justin; Toasters Cc: Moore, Doug Subject: R: SVM DR in 8.3.1 - AGAIN We’re still experiencing a lot of issues with customer where first few days ago we’ve attempted to set up this SVM DR. As reported it seems that it does the work that someone could expect as…basic way. But at this customer the world is a little bit more complicated (does it is the only one using that DR way with 7Mode? I don’t think so!) We need to understand how to proceed to set up this DR. Or we (better NetApp!) must officially communicate to customers that they can wast their –perfectly working for test and troubles- DR approach for it’s not possible in cDOT! Let me resume the issue. This customer, since years, doesn’t use the vFiler DR direct approach but, at the snapmirrored destination flexclone volumes are created and on this run vfiler with –d option that start from the root volume of the vfiler. I.e. (the green stands for running, the red for stopped/offlined/mirrored) ✓ ✓ X ✓ ✓ Production ----sm---> DR ---flexclone---> DR_clone –vfilercreate --> vfiler in dr This permit to have a point in time instance with all the initial configurations (share/exports/MSAD domain join, users, security ecc). This also let have the choice to choose the snapshot used as starting point and, overall, this let to maintain the source-destination replicas active and updated. All of those is automated using WFA, so it’s enough a click and everything go! We’ve seen that SVM DR more ore less work as vfiler dr worked. ✓ ✓ X Production ----sm---> DR < break of mirror relationship – a MUST > X X ✓ Produzione ----sm---> DR We’ve also seen with some trick and maintaining the production and test network separated in different IP spaces (and ports, switches, vlan…) that we can have both running the source and the destination, but we must also break the sm relationship! ✓ X ✓ Production ----sm---> DR These releationships unfortunately are essential prerequisites for this customer. Let me say vols (in SVMs) must continue to replicate for the DR test SVM could be up also for 10 and more days! And customer cannot permit the risk to loose its replicated/updated copies… The questions are: is there something like vfiler create –d? (documentation does not report anything else than “normal” usage) Something that let us create an SVM with all configurations inside but starting from cloned vols at destination snapmirrored site? We’re really in trouble for customer is starting to suspect that this cDot could be a boomerang and, except for VSAN and VVOL support for VMware 6, the things they’ve loosed are more than the one they’ve gained! Any help will be appreciated! Da: toasters-bounces(a)teaparty.net<mailto:toasters-bounces@teaparty.net> [mailto:toasters-bounces@teaparty.net] Per conto di Milazzo Giacomo Inviato: lunedì 6 luglio 2015 17:33 A: Parisi, Justin Cc: Toasters; Moore, Doug Oggetto: R: SVM DR in 8.3.1 We need identity true. a lot of cifs configuration are not brought if false. Think to vfiler Dr of 7mode... Bye - Sent by mobile - -------- Messaggio originale -------- Da: "Parisi, Justin" <Justin.Parisi(a)netapp.com<mailto:Justin.Parisi@netapp.com>> Data: 06/07/2015 17:24 (GMT+01:00) A: Milazzo Giacomo <G.Milazzo(a)sinergy.it<mailto:G.Milazzo@sinergy.it>> Cc: "Borzenkov, Andrei" <andrei.borzenkov(a)ts.fujitsu.com<mailto:andrei.borzenkov@ts.fujitsu.com>>, Toasters <toasters(a)teaparty.net<mailto:toasters@teaparty.net>>, "Moore, Doug" <Doug.Moore(a)netapp.com<mailto:Doug.Moore@netapp.com>> Oggetto: RE: SVM DR in 8.3.1 You can always run the destination in the “Identity discard” mode to avoid IP clashes. That should allow you to test. I’ve added the SVM DR TME to this thread so he can address your questions/be aware of the questions we have out there. From: Milazzo Giacomo [mailto:G.Milazzo@sinergy.it] Sent: Monday, July 06, 2015 11:10 AM To: Parisi, Justin Cc: Borzenkov, Andrei; Toasters Subject: R: SVM DR in 8.3.1 Again, interesting for it summarize the two official docs. We’ve just created a test and it seems to work: unfortunately up to now we don’t have yet the physical network correctly cabled to the right switch. We’re concerned about the statement “source SVM must be stopped” to run in DR…this because we’ve an isolated network environment and when we run DR test the source still continue to work…fortunately we’ve test the “force” option to the start SVM command… A doubt. We’ve noted that when the SVM DR has been created automatically has been created its LIF for data communication (our test contained a CIFS server with a shared vol). This LIF has been created using the VLAN that was assigned to the ipspace we’ve created to the test network, and the same IP of source has been assigned. That’s what we expect, and it’s good. But, what happen if I’ve a lot of source SVM, each with its subnet and broadcast domain that must converge in the same test ipspace? How the lifs and their IP addresses are choosen? Regards, Da: Parisi, Justin [mailto:Justin.Parisi@netapp.com] Inviato: lunedì 6 luglio 2015 16:08 A: Borzenkov, Andrei; Milazzo Giacomo; Toasters Oggetto: RE: SVM DR in 8.3.1 I wrote a blog on SVM DR, if you are interested: https://whyistheinternetbroken.wordpress.com/2015/06/25/techstorage-virtual… From: toasters-bounces(a)teaparty.net<mailto:toasters-bounces@teaparty.net> [mailto:toasters-bounces@teaparty.net] On Behalf Of Borzenkov, Andrei Sent: Wednesday, July 01, 2015 10:05 AM To: NGC-g.milazzo-sinergy.it<http://NGC-g.milazzo-sinergy.it>; Toasters Subject: RE: SVM DR in 8.3.1 You are right, you need to create destination SVM in advance with right attributes. According to documentation “For remote data access or data distribution, you can provide read-only access to the NFS clients and SAN hosts from the destination SVM, if you chose to set the identity-preserve option to false.”. Also “SVM Disaster Recovery Preparation Express Guide” gives examples that indicate that you can (actually, must) configure network on destination SVM and access it. From: toasters-bounces(a)teaparty.net<mailto:toasters-bounces@teaparty.net> [mailto:toasters-bounces@teaparty.net] On Behalf Of Milazzo Giacomo Sent: Wednesday, July 01, 2015 4:33 PM To: Toasters Subject: SVM DR in 8.3.1 Hi all, I know is still RC1 but prior to upgrade a running 8.3 I would try to have more info on a new 8.3.1 feature: the SVM DR. I’ve a customer that on 7Mode ran the DR of CIFS vfiler (joined to a domain) using snapmirror of the vfiler vols, including the root one, and at the DR test/run the source vfiler was stopped and a vfiler create –d command, with the mirrored root specified, let the destination vfiler start with all domain infos, shares on the vole and ip addresses on the DR site with minimal effort. I’ve read the docs that come with 8.3.1 but is not clear. I don’t think that there’s something in SVM DR that permit to recreate the SVM on the destinanation: this must exist in advance, in stopped mode, right? What’s about the maintenance of the IPs? Should I have two identical subnets in both clusters? What if I cannot have two different subnets? Do I have a way to let the replicated SVM being in standby with the data lif with preregistered IPs? Regards

2 2

R: SVM DR in 8.3.1 - AGAIN
by Milazzo Giacomo 18 Jul '15

18 Jul '15

We’re still experiencing a lot of issues with customer where first few days ago we’ve attempted to set up this SVM DR. As reported it seems that it does the work that someone could expect as…basic way. But at this customer the world is a little bit more complicated (does it is the only one using that DR way with 7Mode? I don’t think so!) We need to understand how to proceed to set up this DR. Or we (better NetApp!) must officially communicate to customers that they can wast their –perfectly working for test and troubles- DR approach for it’s not possible in cDOT! Let me resume the issue. This customer, since years, doesn’t use the vFiler DR direct approach but, at the snapmirrored destination flexclone volumes are created and on this run vfiler with –d option that start from the root volume of the vfiler. I.e. (the green stands for running, the red for stopped/offlined/mirrored) ✓ ✓ X ✓ ✓ Production ----sm---> DR ---flexclone---> DR_clone –vfilercreate --> vfiler in dr This permit to have a point in time instance with all the initial configurations (share/exports/MSAD domain join, users, security ecc). This also let have the choice to choose the snapshot used as starting point and, overall, this let to maintain the source-destination replicas active and updated. All of those is automated using WFA, so it’s enough a click and everything go! We’ve seen that SVM DR more ore less work as vfiler dr worked. ✓ ✓ X Production ----sm---> DR < break of mirror relationship – a MUST > X X ✓ Produzione ----sm---> DR We’ve also seen with some trick and maintaining the production and test network separated in different IP spaces (and ports, switches, vlan…) that we can have both running the source and the destination, but we must also break the sm relationship! ✓ X ✓ Production ----sm---> DR These releationships unfortunately are essential prerequisites for this customer. Let me say vols (in SVMs) must continue to replicate for the DR test SVM could be up also for 10 and more days! And customer cannot permit the risk to loose its replicated/updated copies… The questions are: is there something like vfiler create –d? (documentation does not report anything else than “normal” usage) Something that let us create an SVM with all configurations inside but starting from cloned vols at destination snapmirrored site? We’re really in trouble for customer is starting to suspect that this cDot could be a boomerang and, except for VSAN and VVOL support for VMware 6, the things they’ve loosed are more than the one they’ve gained! Any help will be appreciated! Da: toasters-bounces(a)teaparty.net [mailto:toasters-bounces@teaparty.net] Per conto di Milazzo Giacomo Inviato: lunedì 6 luglio 2015 17:33 A: Parisi, Justin Cc: Toasters; Moore, Doug Oggetto: R: SVM DR in 8.3.1 We need identity true. a lot of cifs configuration are not brought if false. Think to vfiler Dr of 7mode... Bye - Sent by mobile - -------- Messaggio originale -------- Da: "Parisi, Justin" <Justin.Parisi(a)netapp.com<mailto:Justin.Parisi@netapp.com>> Data: 06/07/2015 17:24 (GMT+01:00) A: Milazzo Giacomo <G.Milazzo(a)sinergy.it<mailto:G.Milazzo@sinergy.it>> Cc: "Borzenkov, Andrei" <andrei.borzenkov(a)ts.fujitsu.com<mailto:andrei.borzenkov@ts.fujitsu.com>>, Toasters <toasters(a)teaparty.net<mailto:toasters@teaparty.net>>, "Moore, Doug" <Doug.Moore(a)netapp.com<mailto:Doug.Moore@netapp.com>> Oggetto: RE: SVM DR in 8.3.1 You can always run the destination in the “Identity discard” mode to avoid IP clashes. That should allow you to test. I’ve added the SVM DR TME to this thread so he can address your questions/be aware of the questions we have out there. From: Milazzo Giacomo [mailto:G.Milazzo@sinergy.it] Sent: Monday, July 06, 2015 11:10 AM To: Parisi, Justin Cc: Borzenkov, Andrei; Toasters Subject: R: SVM DR in 8.3.1 Again, interesting for it summarize the two official docs. We’ve just created a test and it seems to work: unfortunately up to now we don’t have yet the physical network correctly cabled to the right switch. We’re concerned about the statement “source SVM must be stopped” to run in DR…this because we’ve an isolated network environment and when we run DR test the source still continue to work…fortunately we’ve test the “force” option to the start SVM command… A doubt. We’ve noted that when the SVM DR has been created automatically has been created its LIF for data communication (our test contained a CIFS server with a shared vol). This LIF has been created using the VLAN that was assigned to the ipspace we’ve created to the test network, and the same IP of source has been assigned. That’s what we expect, and it’s good. But, what happen if I’ve a lot of source SVM, each with its subnet and broadcast domain that must converge in the same test ipspace? How the lifs and their IP addresses are choosen? Regards, Da: Parisi, Justin [mailto:Justin.Parisi@netapp.com] Inviato: lunedì 6 luglio 2015 16:08 A: Borzenkov, Andrei; Milazzo Giacomo; Toasters Oggetto: RE: SVM DR in 8.3.1 I wrote a blog on SVM DR, if you are interested: https://whyistheinternetbroken.wordpress.com/2015/06/25/techstorage-virtual… From: toasters-bounces(a)teaparty.net<mailto:toasters-bounces@teaparty.net> [mailto:toasters-bounces@teaparty.net] On Behalf Of Borzenkov, Andrei Sent: Wednesday, July 01, 2015 10:05 AM To: NGC-g.milazzo-sinergy.it; Toasters Subject: RE: SVM DR in 8.3.1 You are right, you need to create destination SVM in advance with right attributes. According to documentation “For remote data access or data distribution, you can provide read-only access to the NFS clients and SAN hosts from the destination SVM, if you chose to set the identity-preserve option to false.”. Also “SVM Disaster Recovery Preparation Express Guide” gives examples that indicate that you can (actually, must) configure network on destination SVM and access it. From: toasters-bounces(a)teaparty.net<mailto:toasters-bounces@teaparty.net> [mailto:toasters-bounces@teaparty.net] On Behalf Of Milazzo Giacomo Sent: Wednesday, July 01, 2015 4:33 PM To: Toasters Subject: SVM DR in 8.3.1 Hi all, I know is still RC1 but prior to upgrade a running 8.3 I would try to have more info on a new 8.3.1 feature: the SVM DR. I’ve a customer that on 7Mode ran the DR of CIFS vfiler (joined to a domain) using snapmirror of the vfiler vols, including the root one, and at the DR test/run the source vfiler was stopped and a vfiler create –d command, with the mirrored root specified, let the destination vfiler start with all domain infos, shares on the vole and ip addresses on the DR site with minimal effort. I’ve read the docs that come with 8.3.1 but is not clear. I don’t think that there’s something in SVM DR that permit to recreate the SVM on the destinanation: this must exist in advance, in stopped mode, right? What’s about the maintenance of the IPs? Should I have two identical subnets in both clusters? What if I cannot have two different subnets? Do I have a way to let the replicated SVM being in standby with the data lif with preregistered IPs? Regards

2 1

cDOT and NFS Volumes for VMDK datastores
by John Stoffel 17 Jul '15

17 Jul '15

Guys, I'm starting to re-read TR-3749 again, but since I'm running cDOT 8.2p# on my main two node cluster (soon to be four nodes) I wanted your opinions on how to provision NFS datastores. Right now, when we add new DataStores, we have to goto all 16 ESX hosts in the cluster and manually mount them. Which is doable, but maybe not scalable over time as we'd like to isolate groups and apps into their own NFS volumes if possible. So, knowing that cDOT allows me to setup a volume and then use junction mounts to add more volumes to that name space, does it make sense to do: 1. Create base volume, let's call it '/datastore1' 2. Create sub-volumes of various sizes and performance levels and mount them to: /datastore1/bob /datastore1/jim /datastore1/sam .... 3. When we spin up VMs and asign the data stores, you only need to drill down into the correct area, bob, jim or sam and put the data there. 4. I want to add new FlashPool volume, so I create it and junction mount it to /datastore1/fp1 5. I don't have to add any mounts to the ESX hosts, they just see more growth in the /datastore1/ mount point and keep working. So is this a dumb idea? Or not a supported idea? I know it limits my through put to just a single IP address for traffic, unless I spread out the load by having multiple /datastore#/ volumes spread across the nodes of the cluster, and various volumes junction mounted to each of these master /datastore#/ volumes. Right now we just create new volumes and mount them, but I looking for a more scalable, manageable method. THanks, John

6 8

"Created Date" on CIFS shares
by Edward Rolison 14 Jul '15

14 Jul '15

I've got a scripting task that needs me to work off the creation time of a file. I've been caught out before by 'ctime' being change time, rather than 'create time'. So I thought I was just stuck - however, I've noticed that the Windows 'file creation time' attribute .... actually looks correct. Which has me confused, because I can't see this value from 'Unix side' seemingly at all. I can't tell how Windows knows what the right value is. Is anyone able to enlighten me? Where's creation time stored, and is there a way to access it from Unix?

3 2

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters July 2015