toasters

toasters@lists.teaparty.net

13531 discussions

ONTAP9 SystemManager issues with quotas
by Sami Kapanen 11 Jan '17

11 Jan '17

Hi, We are having issues with ONTAP9 SystemManager, quota report doesn't show everything, and target filtering is not working. Also, it doesn't allow to sort the rows by username or space used columns. Others seen this? We have case open and escalated, sk

1 1

Atomicity of rename on NFS
by Edward Rolison 03 Jan '17

03 Jan '17

Hello fellow NetApp Admins. I have a bit of an odd one that I'm trying to troubleshoot - and whilst I'm not sure it's specifically filer related, it's NFS related (and is happening on a filer mount). What happens is this - there's a process that updates a file, and relies on 'rename()' being atomic- a journal is updated, and then reference pointer (file) is newly created, and renamed over an old one. The expectation is that this file will always be there - because "rename()" is defined as an atomic operation. But that's not quite what I'm getting - I have one nfs client doing it's (atomic) rename. And another client (different NFS host) reading it, and - occasionally - reporting 'no such file or directory'. This is causing an operation to fail, which in turn means that someone has to intervene in the process. This operation (and multiple extremely similar ones) happen at 5m intervals, and every few days (once a week maybe?) it fails for this reason, and our developers think that should be impossible. But as such - it looks like a pretty narrow race condition. So what I'm trying to figure out is first off: - Could this be a NetApp bug? We've moved from 7 mode to CDOT, and it didn't happen before. On the flip side though - I have no guarantee that it 'never happened before' because we weren't catching a race condition. (moving to new tin and improving performance does increase race condition likelihood after all) - Could this be a kernel bug? We're all on kernel 2.6.32-504.12.2.el6.x86_64 - and whilst we're deploying Centos 7, all the hosts involved aren't yet. (But that's potentially also just coincidence, as there's quite a few hosts, and they're all the same kernel versions). - Is it actually impossible for a file A renamed over file B to generate ENOENT on a different client? Specifically, in RFC3530 We have: " The RENAME operation must be atomic to the client.". So the client doing the rename sees an atomic operation - but the expectation is that a separate client will also perceive an 'atomic' change - once the cache is refreshed, the 'new' directory has the new files, and at no point was there 'no such file or directory' because it was either the old one, or the newly renamed one. Is this actually a valid thing to think? This is a bit of a complicated one, and has me clutching at straws a bit - I can't reliably reproduce it - a basic fast spinning loop script on multiple client to read-write-rename didn't hit it. I've got pcaps running hoping to catch it 'in flight' - but haven't yet managed to catch it happening. But any suggestions would be gratefully received.

5 12

SOLVED: snapmirror source and target aggregate usage don't match?
by Rue, Randy 23 Dec '16

23 Dec '16

Hello All, This provided an opportunity to find and implement some improvements in our dedupe/snapmirror configuration but that wasn't the real problem. We disabled dedupe on all but the two SVMs that were showing space savings of >10% or a couple of hundred GB, and changed the dedupe schedule to run hourly and with ample time to finish before the hourly snapmirror schedule. The real problem turned out to be that while we configure all our data volumes on the source cluster as thin provisioned, when DOT creates the target volume on the secondary "thick." And as the Storage Efficiency settings aren't exposed anywhere in the GUI on the secondary cluster for those volumes, it took some digging and five people staring at the CLI output before we figured this out. We changed all the target volumes to thin and will check/change new ones as they're created Life is good. Randy From: toasters-bounces(a)teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Rue, Randy Sent: Saturday, December 17, 2016 11:14 AM To: toasters(a)teaparty.net Subject: RE: snapmirror source and target aggregate usage don't match? Thank you for a NA docs pointer from a list member and the opportunity for a little RTFM before I reply to the list! Looks like my colleague's recollection of earlier versions still applies to 8.3. Essentially, we've been snapmirroring un-deduped data. My next question is, is it realistic to run dedupes hourly on 30-40 volumes totalling ~100TB? Because that's a much easier proposition than amending our SLAs to lengthen our snapshot cycles. And to get both on the same cycle, is it possible to make snapshots dependent on dedupe finishing, or do we just assume dedupe will complete, and if so, if it doesn't, what are the consequences? For example, if a dedupe that usually finishes at 5 minutes after the hour isn't done, and snapmirror runs then, will snapmirror then be syncing a full hour of full-sized changes? Last question for now: assuming both are on the same schedule, how do I get the current lost space back? Will it be reclaimed when the schedules are synced and the snapshots have rolled off? Or do I need to destroy and recreate the target volumes? Hope to hear from you, especially from any other shop running both snapmirror and dedupe. Randy (and if a solution requires DOT9 we do have an upgrade on our roadmap) Replying to just toy for now. Hopefully this will help and you can report back Look at this... https://library.netapp.com/ecm/ecm_download_file/ECMLP2348026 Page 142. I think you may need to work out some scheduling and that may help. Going to look a bit more...i have another idea, but that may only be ONTAP 9 related. Get Outlook for iOS<https://aka.ms/o0ukef> _____________________________ From: Rue, Randy <rrue(a)fredhutch.org<mailto:rrue@fredhutch.org>> Sent: Saturday, December 17, 2016 11:14 AM Subject: snapmirror source and target aggregate usage don't match? To: <toasters(a)teaparty.net<mailto:toasters@teaparty.net>> Hello All, We run two 8.3 filers with a list of vservers and their associated volumes, with each volume snapmirrored (volume level) from the active primary cluster to matching vserver/volumes on the passive secondary. Both clusters have a similar set of aggregates of just about equal size. Both clusters' aggregates contain the same list of volumes of the same size, with the same space total/used/available on both sets. But on the target cluster the same aggregates are reporting 30% more used space. This is about on par with the dedupe savings we're getting on the primary so when I first noticed this my thought was to check that dedupe was OK on the target. But if you look in the webUI, it reports that no "storage efficiency" is available on a replication target, and ended up thinking this meant that the secondary data would have to be full-sized. I even recall asking someone and having this confirmed, but can't recall if that came from the vendor SE or our VAR SE or a support tech or. Now we're approaching the space limit of the secondary cluster and I'm looking deeper. At this point, as it appears that for each volume the total/used/free space matches after dedupe on the source, I'm thinking that dedupe properties aren't exposed on the target but the data is still a true copy of the deduped original. This is supported by being able to view dedupe stats on the target via the CLI that show the same savings as on the source. Note that we're also snapshotting these volumes, and while we're deduping daily, we're snapshotting hourly. A colleague mentioned remembering that this could mean mirrored data that's not deduped yet is being replicated full-size. But if so, wouldn't this be reflected in the dedupe stats on the target? OK, just found that "storage aggregate show -fields usedsize,physical-used" on the primary/source cluster shows that used and physical-used are about identical for all aggrs. On the secondary/target, used is consistently larger than physical-used and the total difference makes up the 30% I'm "missing." Is this a problem with my reporting? Are we actually OK and I need to look at physical-used instead of used? Or if we're not OK, where is the space being used and can I get it back? Thanks in advance for your guidance... Randy

2 1

Flash Cache vs Flash Pool
by Ehrenwald, Ian 21 Dec '16

21 Dec '16

Hello Toasters I'm thinking about if/how to implement Flash Pool in our new cDOT clusters, and was wondering if anyone could provide a bit of real world guidance for me. We have two SAS aggregates, aggr_sas_600g_c1n1 with 8x24 600g and aggr_sas_1200g_c1n1 with 4x24 1.2t. I've been doing a bunch of reading about Flash Pool vs Flash Cache and am trying to better understand their strengths and weaknesses. Flash Pool accelerates writes as well as reads (Flash Cache is reads only), however with Flash Pool there seems to be the potential for slower cache access/throughput vs Flash Cache since the data needs to travel the SAS path vs Flash Cache which is probably DMA through PCIe. Maybe that's not a concern at all, I don't know. Additionally, it appears that using Flash Pool disables the Flash Cache functionality for the aggregates which are in hybrid mode (makes sense), but then we have expensive add-in cards doing nothing. Our theoretical Flash Pool would be 2x24 200g, giving us about 5.5t of usable caching space to sprinkle into these aggregates. I've been running AWA on the cluster against those two SAS aggregates for ~24 hours and have come up with these stats: ### FP AWA Stats ### Host mrk_c1n1 Memory 61054 MB ONTAP Version NetApp Release 8.3.2P5: Tue Aug 23 01:27:00 PDT 2016 Basic Information Aggregate aggr_sas_600g_c1n1 Current-time Tue Nov 22 17:06:53 EST 2016 Start-time Mon Nov 21 12:48:17 EST 2016 Total runtime (sec) 101918 Interval length (sec) 600 Total intervals 157 In-core Intervals 1024 Summary of the past 157 intervals max ------------ Read Throughput (MB/s): 339.039 Write Throughput (MB/s): 123.536 Cacheable Read (%): 56 Cacheable Write (%): 66 Max Projected Cache Size (GiB): 787.463 Summary Cache Hit Rate vs. Cache Size Referenced Cache Size (GiB): 714.650 Referenced Interval: ID 132 starting at Tue Nov 22 12:33:05 EST 2016 Size 20% 40% 60% 80% 100% Read Hit (%) 25 30 30 30 33 Write Hit (%) 1 2 2 2 2 The entire results and output of Automated Workload Analyzer (AWA) are estimates. The format, syntax, CLI, results and output of AWA may change in future Data ONTAP releases. AWA reports the projected cache size in capacity. It does not make recommendations regarding the number of data SSDs required. Please follow the guidelines for configuring and deploying Flash Pool; that are provided in tools and collateral documents. These include verifying the platform cache size maximums and minimum number and maximum number of data SSDs. Basic Information Aggregate aggr_sas_1200g_c1n1 Current-time Tue Nov 22 17:06:53 EST 2016 Start-time Mon Nov 21 12:40:57 EST 2016 Total runtime (sec) 102357 Interval length (sec) 600 Total intervals 158 In-core Intervals 1024 Summary of the past 158 intervals max ------------ Read Throughput (MB/s): 914.247 Write Throughput (MB/s): 257.318 Cacheable Read (%): 41 Cacheable Write (%): 26 Max Projected Cache Size (GiB): 2412.178 Summary Cache Hit Rate vs. Cache Size Referenced Cache Size (GiB): 2142.380 Referenced Interval: ID 113 starting at Tue Nov 22 09:04:41 EST 2016 Size 20% 40% 60% 80% 100% Read Hit (%) 34 38 38 38 41 Write Hit (%) 7 7 7 7 9 The entire results and output of Automated Workload Analyzer (AWA) are estimates. The format, syntax, CLI, results and output of AWA may change in future Data ONTAP releases. AWA reports the projected cache size in capacity. It does not make recommendations regarding the number of data SSDs required. Please follow the guidelines for configuring and deploying Flash Pool; that are provided in tools and collateral documents. These include verifying the platform cache size maximums and minimum number and maximum number of data SSDs. ### FP AWA Stats End ### Aggregate aggr_sas_600g_c1n1 has a lot of random overwrites (66%) that could have been cached. The volumes in that aggregate are pretty much exclusively Oracle databases. The other aggregate, aggr_sas_1200g_c1n1, doesn't seem hit as hard. Given those statistics, what would you do if your options were ~5.5t of Flash Pool vs buying another 2t Flash Cache card per node in this HA pair? I seem to be missing the 'Projected Read Offload' and 'Projected Write Offload' statistics which would have been very useful, mentioned at the Flash Pool documentation in https://library.netapp.com/ecmdocs/ECMP1368404/html/GUID-2C3EC0DF-FEFE-4871 -A161-4A3BAC87DB69.html Thanks for any insight you all can provide. -- Ian Ehrenwald Senior Infrastructure Engineer Hachette Book Group, Inc. 1.617.263.1948 / ian.ehrenwald(a)hbgusa.com This may contain confidential material. If you are not an intended recipient, please notify the sender, delete immediately, and understand that no disclosure or reliance on the information herein is permitted. Hachette Book Group may monitor email to and from our network.

4 8

Re: snapmirror source and target aggregate usage don't match?
by jordan slingerland 18 Dec '16

18 Dec '16

Maybe also review the dedup savings you are acheveing and consider disabling it on volumes getting 10% savings or so or less since the metadata required for dedup is something like 7%. On Dec 17, 2016 9:23 PM, "tmac" <tmacmd(a)gmail.com> wrote: Trial by fire? You will likely need to get a rough baseline of how long a single dedupe run takes on a source volume. Then look at doing multiple volumes and see how much the process slows down. With Spinnng media, the dedupe process can take a while on larger volumes. You may need to get a little creative and group some of the volumes together and created schedules for the deduping to happen at specific times. I really doubt you are going to be able to dedupe 30-40 volumes totalling ~100TB every hour. You should probably stagger and run throughout the day with different schedules you can create custom vol eff. policies that only run for a certain duration ("-" is default for no duration or up to 999 hours (whole numbers in hours)) and the qos-policy (for vol eff operations!) can be set to background (run, but do not impede) or best-effort (may slightly impact operations) as far as the timing, you may want to look at scripting with powershell or the NetApp SDK. Wait for a vol eff operation to finish, then do snapshots and/or mirroring. We could spend lots of time on this. I hope this helps at least a little. --tmac *Tim McCarthy, **Principal Consultant* *Proud Member of the #NetAppATeam <https://twitter.com/NetAppATeam>* *I Blog at TMACsRack <https://tmacsrack.wordpress.com/>* On Sat, Dec 17, 2016 at 2:13 PM, Rue, Randy <rrue(a)fredhutch.org> wrote: > Thank you for a NA docs pointer from a list member and the opportunity for > a little RTFM before I reply to the list! > > Looks like my colleague’s recollection of earlier versions still applies > to 8.3. Essentially, we've been snapmirroring un-deduped data. > > My next question is, is it realistic to run dedupes hourly on 30-40 > volumes totalling ~100TB? Because that's a much easier proposition than > amending our SLAs to lengthen our snapshot cycles. > > And to get both on the same cycle, is it possible to make snapshots > dependent on dedupe finishing, or do we just assume dedupe will complete, > and if so, if it doesn't, what are the consequences? For example, if a > dedupe that usually finishes at 5 minutes after the hour isn't done, and > snapmirror runs then, will snapmirror then be syncing a full hour of > full-sized changes? > > Last question for now: assuming both are on the same schedule, how do I > get the current lost space back? Will it be reclaimed when the schedules > are synced and the snapshots have rolled off? Or do I need to destroy and > recreate the target volumes? > > Hope to hear from you, especially from any other shop running both > snapmirror and dedupe. > > Randy > > (and if a solution requires DOT9 we do have an upgrade on our roadmap) > > > > Replying to just toy for now. Hopefully this will help and you can report > back > Look at this... > https://library.netapp.com/ecm/ecm_download_file/ECMLP2348026 > > Page 142. > > I think you may need to work out some scheduling and that may help. Going > to look a bit more...i have another idea, but that may only be ONTAP 9 > related. > > > Get Outlook for iOS <https://aka.ms/o0ukef> > > _____________________________ > From: Rue, Randy <rrue(a)fredhutch.org> > Sent: Saturday, December 17, 2016 11:14 AM > Subject: snapmirror source and target aggregate usage don't match? > To: <toasters(a)teaparty.net> > > > Hello All, > > > > We run two 8.3 filers with a list of vservers and their associated > volumes, with each volume snapmirrored (volume level) from the active > primary cluster to matching vserver/volumes on the passive secondary. > > > > Both clusters have a similar set of aggregates of just about equal size. > Both clusters’ aggregates contain the same list of volumes of the same > size, with the same space total/used/available on both sets. > > > > But on the target cluster the same aggregates are reporting 30% more used > space. > > > > This is about on par with the dedupe savings we’re getting on the primary > so when I first noticed this my thought was to check that dedupe was OK on > the target. But if you look in the webUI, it reports that no “storage > efficiency” is available on a replication target, and ended up thinking > this meant that the secondary data would have to be full-sized. I even > recall asking someone and having this confirmed, but can’t recall if that > came from the vendor SE or our VAR SE or a support tech or. > > > > Now we’re approaching the space limit of the secondary cluster and I’m > looking deeper. At this point, as it appears that for each volume the > total/used/free space matches after dedupe on the source, I’m thinking that > dedupe properties aren’t exposed on the target but the data is still a true > copy of the deduped original. This is supported by being able to view > dedupe stats on the target via the CLI that show the same savings as on the > source. > > > > Note that we’re also snapshotting these volumes, and while we’re deduping > daily, we’re snapshotting hourly. A colleague mentioned remembering that > this could mean mirrored data that’s not deduped yet is being replicated > full-size. But if so, wouldn’t this be reflected in the dedupe stats on the > target? > > > > OK, just found that “storage aggregate show -fields > usedsize,physical-used” on the primary/source cluster shows that used and > physical-used are about identical for all aggrs. On the secondary/target, > used is consistently larger than physical-used and the total difference > makes up the 30% I’m “missing.” > > > > Is this a problem with my reporting? Are we actually OK and I need to look > at physical-used instead of used? Or if we’re not OK, where is the space > being used and can I get it back? > > > > Thanks in advance for your guidance… > > > > Randy > > > > > > > > _______________________________________________ > Toasters mailing list > Toasters(a)teaparty.net > http://www.teaparty.net/mailman/listinfo/toasters > > _______________________________________________ Toasters mailing list Toasters(a)teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

1 0

RE: snapmirror source and target aggregate usage don't match?
by Rue, Randy 18 Dec '16

18 Dec '16

Thank you for a NA docs pointer from a list member and the opportunity for a little RTFM before I reply to the list! Looks like my colleague’s recollection of earlier versions still applies to 8.3. Essentially, we've been snapmirroring un-deduped data. My next question is, is it realistic to run dedupes hourly on 30-40 volumes totalling ~100TB? Because that's a much easier proposition than amending our SLAs to lengthen our snapshot cycles. And to get both on the same cycle, is it possible to make snapshots dependent on dedupe finishing, or do we just assume dedupe will complete, and if so, if it doesn't, what are the consequences? For example, if a dedupe that usually finishes at 5 minutes after the hour isn't done, and snapmirror runs then, will snapmirror then be syncing a full hour of full-sized changes? Last question for now: assuming both are on the same schedule, how do I get the current lost space back? Will it be reclaimed when the schedules are synced and the snapshots have rolled off? Or do I need to destroy and recreate the target volumes? Hope to hear from you, especially from any other shop running both snapmirror and dedupe. Randy (and if a solution requires DOT9 we do have an upgrade on our roadmap) Replying to just toy for now. Hopefully this will help and you can report back Look at this... https://library.netapp.com/ecm/ecm_download_file/ECMLP2348026 Page 142. I think you may need to work out some scheduling and that may help. Going to look a bit more...i have another idea, but that may only be ONTAP 9 related. Get Outlook for iOS<https://aka.ms/o0ukef> _____________________________ From: Rue, Randy <rrue(a)fredhutch.org<mailto:rrue@fredhutch.org>> Sent: Saturday, December 17, 2016 11:14 AM Subject: snapmirror source and target aggregate usage don't match? To: <toasters(a)teaparty.net<mailto:toasters@teaparty.net>> Hello All, We run two 8.3 filers with a list of vservers and their associated volumes, with each volume snapmirrored (volume level) from the active primary cluster to matching vserver/volumes on the passive secondary. Both clusters have a similar set of aggregates of just about equal size. Both clusters’ aggregates contain the same list of volumes of the same size, with the same space total/used/available on both sets. But on the target cluster the same aggregates are reporting 30% more used space. This is about on par with the dedupe savings we’re getting on the primary so when I first noticed this my thought was to check that dedupe was OK on the target. But if you look in the webUI, it reports that no “storage efficiency” is available on a replication target, and ended up thinking this meant that the secondary data would have to be full-sized. I even recall asking someone and having this confirmed, but can’t recall if that came from the vendor SE or our VAR SE or a support tech or. Now we’re approaching the space limit of the secondary cluster and I’m looking deeper. At this point, as it appears that for each volume the total/used/free space matches after dedupe on the source, I’m thinking that dedupe properties aren’t exposed on the target but the data is still a true copy of the deduped original. This is supported by being able to view dedupe stats on the target via the CLI that show the same savings as on the source. Note that we’re also snapshotting these volumes, and while we’re deduping daily, we’re snapshotting hourly. A colleague mentioned remembering that this could mean mirrored data that’s not deduped yet is being replicated full-size. But if so, wouldn’t this be reflected in the dedupe stats on the target? OK, just found that “storage aggregate show -fields usedsize,physical-used” on the primary/source cluster shows that used and physical-used are about identical for all aggrs. On the secondary/target, used is consistently larger than physical-used and the total difference makes up the 30% I’m “missing.” Is this a problem with my reporting? Are we actually OK and I need to look at physical-used instead of used? Or if we’re not OK, where is the space being used and can I get it back? Thanks in advance for your guidance… Randy

2 1

snapmirror source and target aggregate usage don't match?
by Rue, Randy 17 Dec '16

17 Dec '16

Hello All, We run two 8.3 filers with a list of vservers and their associated volumes, with each volume snapmirrored (volume level) from the active primary cluster to matching vserver/volumes on the passive secondary. Both clusters have a similar set of aggregates of just about equal size. Both clusters' aggregates contain the same list of volumes of the same size, with the same space total/used/available on both sets. But on the target cluster the same aggregates are reporting 30% more used space. This is about on par with the dedupe savings we're getting on the primary so when I first noticed this my thought was to check that dedupe was OK on the target. But if you look in the webUI, it reports that no "storage efficiency" is available on a replication target, and ended up thinking this meant that the secondary data would have to be full-sized. I even recall asking someone and having this confirmed, but can't recall if that came from the vendor SE or our VAR SE or a support tech or. Now we're approaching the space limit of the secondary cluster and I'm looking deeper. At this point, as it appears that for each volume the total/used/free space matches after dedupe on the source, I'm thinking that dedupe properties aren't exposed on the target but the data is still a true copy of the deduped original. This is supported by being able to view dedupe stats on the target via the CLI that show the same savings as on the source. Note that we're also snapshotting these volumes, and while we're deduping daily, we're snapshotting hourly. A colleague mentioned remembering that this could mean mirrored data that's not deduped yet is being replicated full-size. But if so, wouldn't this be reflected in the dedupe stats on the target? OK, just found that "storage aggregate show -fields usedsize,physical-used" on the primary/source cluster shows that used and physical-used are about identical for all aggrs. On the secondary/target, used is consistently larger than physical-used and the total difference makes up the 30% I'm "missing." Is this a problem with my reporting? Are we actually OK and I need to look at physical-used instead of used? Or if we're not OK, where is the space being used and can I get it back? Thanks in advance for your guidance... Randy

2 1

AW: System attention LED
by Alexander Griesser 16 Dec '16

16 Dec '16

Hi, one is running 9.1RC2 and the other one is running 8.1.2P4 (FAS2240) Alexander Griesser Head of Systems Operations ANEXIA Internetdienstleistungs GmbH E-Mail: AGriesser(a)anexia-it.com<mailto:AGriesser@anexia-it.com> Web: http://www.anexia-it.com<http://www.anexia-it.com/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 Von: CPO [mailto:cavinoquin@gmail.com] Gesendet: Donnerstag, 15. Dezember 2016 14:23 An: Alexander Griesser <AGriesser(a)anexia-it.com> Betreff: Re: System attention LED What version on data on tap? On Dec 14, 2016 3:22 PM, "Alexander Griesser" <AGriesser(a)anexia-it.com<mailto:AGriesser@anexia-it.com>> wrote: Hi there, I have two filers (1x FAS2240 and 1x FAS2650) and both have the system attention LED lit and I have no idea why. `System health status show` as well as config advisor are both A-OK and there are no visible indicators why this LED is still lit. Any idea how to find out the root cause of this LED on the front bezel or how I can check or maybe reset the status of the LED through CLI? Thanks, Alexander Griesser Head of Systems Operations ANEXIA Internetdienstleistungs GmbH E-Mail: AGriesser(a)anexia-it.com<mailto:AGriesser@anexia-it.com> Web: http://www.anexia-it.com<http://www.anexia-it.com/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 _______________________________________________ Toasters mailing list Toasters(a)teaparty.net<mailto:Toasters@teaparty.net> http://www.teaparty.net/mailman/listinfo/toasters

1 0

AW: System attention LED
by Alexander Griesser 15 Dec '16

15 Dec '16

Alright, thanks – it’s strange that one can only guess what the problem is with those LEDs :-/ Alexander Griesser Head of Systems Operations ANEXIA Internetdienstleistungs GmbH E-Mail: AGriesser(a)anexia-it.com<mailto:AGriesser@anexia-it.com> Web: http://www.anexia-it.com<http://www.anexia-it.com/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 Von: Vervloesem Wouter [mailto:wouter.vervloesem@neoria.be] Gesendet: Donnerstag, 15. Dezember 2016 13:52 An: Alexander Griesser <AGriesser(a)anexia-it.com> Cc: jordan slingerland <jordan.slingerland(a)gmail.com>; toasters(a)teaparty.net Betreff: Re: System attention LED I had a similar problem in the past. It turned out that one of the PSU's had lost its power in the past, and the LED wasn't cleared correctly. The solution (for me) was to reseat the PSU in the system, few seconds after that everything was fine. Met vriendelijke groeten, Wouter Vervloesem<https://be.linkedin.com/pub/wouter-vervloesem/5/a63/a41> Storage Consultant Neoria NV<http://www.neoria.be/> Prins Boudewijnlaan 41 - 2650 Edegem T +32 3 451 23 82 | M +32 496 52 93 61 Op 15 dec. 2016, om 13:30 heeft Alexander Griesser <AGriesser(a)anexia-it.com<mailto:AGriesser@anexia-it.com>> het volgende geschreven: Hi, the LED is still on and I guess it should have been fully charged by now if this would have been the issue. So, any other ideas besides opening a case? Regarding the FAS2240 I mentioned in my initial e-mail: SP netapp1*> system fru led show 0 FRU LED ID 0 is off SP netapp1*> system fru led show 1 FRU LED ID 1 is on I did google a bit for this symptom and therefore updated the SP firmware to the latest compatible one for my ontap version (2.1.3P2), but that did not fix the issue. Any fix for this symptom except for halting both nodes? Best, Alexander Griesser Head of Systems Operations ANEXIA Internetdienstleistungs GmbH E-Mail: AGriesser(a)anexia-it.com<mailto:AGriesser@anexia-it.com> Web: http://www.anexia-it.com<http://www.anexia-it.com/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 Von: toasters-bounces(a)teaparty.net<mailto:toasters-bounces@teaparty.net> [mailto:toasters-bounces@teaparty.net] Im Auftrag von Alexander Griesser Gesendet: Mittwoch, 14. Dezember 2016 23:29 An: jordan slingerland <jordan.slingerland(a)gmail.com<mailto:jordan.slingerland@gmail.com>> Cc: toasters(a)teaparty.net<mailto:toasters@teaparty.net> Betreff: AW: AW: System attention LED Maybe that has changed with OnTap 9.1? I’m running 9.1RC2 here on this filer and not because I’m brave, but because it got shipped with it ☺ Alexander Griesser Head of Systems Operations ANEXIA Internetdienstleistungs GmbH E-Mail: AGriesser(a)anexia-it.com<mailto:AGriesser@anexia-it.com> Web: http://www.anexia-it.com<http://www.anexia-it.com/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 Von: jordan slingerland [mailto:jordan.slingerland@gmail.com] Gesendet: Mittwoch, 14. Dezember 2016 23:27 An: Alexander Griesser <AGriesser(a)anexia-it.com<mailto:AGriesser@anexia-it.com>> Cc: toasters(a)teaparty.net<mailto:toasters@teaparty.net>; Brad Thompson <brad.thompson877(a)gmail.com<mailto:brad.thompson877@gmail.com>> Betreff: Re: AW: System attention LED Hmm, expected it to look like this listing charge voltage. Current and hours it can hold https://kb.netapp.com/support/s/article/how-to-check-the-nvram-battery-stat… On Dec 14, 2016 5:16 PM, "Alexander Griesser" <AGriesser(a)anexia-it.com<mailto:AGriesser@anexia-it.com>> wrote: Looks good to me: netapp01> environment status chassis list-sensors Sensor Name State Current Critical Warning Warning Critical Reading Low Low High High ------------------------------------------------------------------------------------------------- PSU2 FRU GOOD PSU1 FRU GOOD SP Status IPMI_HB_OK mSATA Status OK mSATA Pres PRESENT Best, Alexander Griesser Head of Systems Operations ANEXIA Internetdienstleistungs GmbH E-Mail: AGriesser(a)anexia-it.com<mailto:AGriesser@anexia-it.com> Web: http://www.anexia-it.com<http://www.anexia-it.com/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 Von: Brad Thompson [mailto:brad.thompson877@gmail.com<mailto:brad.thompson877@gmail.com>] Gesendet: Mittwoch, 14. Dezember 2016 23:09 An: Alexander Griesser <AGriesser(a)anexia-it.com<mailto:AGriesser@anexia-it.com>> Cc: toasters(a)teaparty.net<mailto:toasters@teaparty.net> Betreff: Re: System attention LED The battery might be charging. check: environment status chassis list-sensors On Wed, Dec 14, 2016 at 4:49 PM, Alexander Griesser <AGriesser(a)anexia-it.com<mailto:AGriesser@anexia-it.com>> wrote: Hi Brad, netapp01> storage show fault No faults found in storage subsystem netapp02> storage show fault No faults found in storage subsystem No luck with `system fru led show` on the SP of the 2650: SP netapp01*> system fru led show system fru show - display fru id inventory data system fru list - list fru id system fru log show - display fru log system fru log clear - clear fru log But this one seems to work (I’ve highlighted the ones where the LED seems to be in the „on“ state): SP netapp01*> system led show Invalid command usage Usage: system led show <FRU-LED-ID> Display status of the specified <FRU-LED-ID>. <FRU-LED-ID>: 1 - SP Controller Fault LED 2 - SP Chassis Locate LED 3 - SP System Fault LED 4 - SP SAS Port A Fault LED 5 - SP SAS Port B Fault LED 6 - SP CNA 0 Port 1 Fault LED 7 - SP CNA 0 Port 2 Fault LED 8 - SP CNA 1 Port 1 Fault LED 9 - SP CNA 1 Port 2 Fault LED 10 - SP SFP Port 0 Fault LED 11 - SP SFP Port 1 Fault LED 12 - SP CPU DIMM 1 Fault LED 13 - SP CPU DIMM 2 Fault LED 14 - SP NVME 1 Fault LED 15 - SP MSATA Port Fault LED 16 - SP Battery Fault LED 17 - SP RTC Battery Fault LED (Coin Cell Battery) all others are not valid SP netapp01*> SP netapp01*> SP netapp01*> system led show 1 FRU LED ID 1 is on SP netapp01*> system led show 2 FRU LED ID 2 is off SP netapp01*> system led show 3 FRU LED ID 3 is on. Set by SP SP netapp01*> system led show 4 FRU LED ID 4 is off SP netapp01*> system led show 5 FRU LED ID 5 is off SP netapp01*> system led show 6 FRU LED ID 6 is off SP netapp01*> system led show 7 FRU LED ID 7 is off SP netapp01*> system led show 8 FRU LED ID 8 is off SP netapp01*> system led show 9 FRU LED ID 9 is off SP netapp01*> system led show 10 FRU LED ID 10 is off SP netapp01*> system led show 11 FRU LED ID 11 is off SP netapp01*> system led show 12 FRU LED ID 12 is off SP netapp01*> system led show 13 FRU LED ID 13 is off SP netapp01*> system led show 14 FRU LED ID 14 is off SP netapp01*> system led show 15 FRU LED ID 15 is off SP netapp01*> system led show 16 FRU LED ID 16 is on SP netapp01*> system led show 17 FRU LED ID 17 is off On the second controller, all LEDs (except for ID 3, which has been set by partner SP and is therefore related) are off. So thanks a bunch, I do at least finally know where this LED comes from ☺ Since this filer was unboxed today and installed, I’m not aware of any previous faults – could it be that the battery is almost empty and needs to fully charge? Any way to check the battery charging state? Best, Alexander Griesser Head of Systems Operations ANEXIA Internetdienstleistungs GmbH E-Mail: AGriesser(a)anexia-it.com<mailto:AGriesser@anexia-it.com> Web: http://www.anexia-it.com<http://www.anexia-it.com/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 Von: Brad Thompson [mailto:brad.thompson877@gmail.com<mailto:brad.thompson877@gmail.com>] Gesendet: Mittwoch, 14. Dezember 2016 22:39 An: Alexander Griesser <AGriesser(a)anexia-it.com<mailto:AGriesser@anexia-it.com>> Cc: toasters(a)teaparty.net<mailto:toasters@teaparty.net> Betreff: Re: System attention LED Is clustering enabled (cf status)? That will turn on the fault led if it's not enabled. What does "storage show fault" say? Other than that, the fault light could be on due to a previous fault in the system that has been cleared. Are you aware of any previous faults? If you log in to the Service Processor you can run "priv set diag", followed by "system fru led show". That command will list out the available fru's that you can check the status on. For example: SP nodeA*> system fru led show Invalid command usage Usage: system fru led show <FRU-LED-ID> Display status of the specified <FRU-LED-ID>. <FRU-LED-ID>: 0 = SP Controller Fault LED 1 = SP Heartbeat LED 2 = SP System Fault LED 3 = SP Fan 1 Fault LED 4 = SP Fan 2 Fault LED 5 = SP Fan 3 Fault LED 6 = SP IOXM Fault LED all others are not valid SP nodeA*> system fru led show 2 FRU LED ID 2 is off You can check the status of each fru id to see which one is on. From there double check there aren't any actual faults with that component. Still no faults? You may just be able to turn off the led using the "system fru led set" command. On Wed, Dec 14, 2016 at 4:16 PM, Alexander Griesser <AGriesser(a)anexia-it.com<mailto:AGriesser@anexia-it.com>> wrote: Hi there, I have two filers (1x FAS2240 and 1x FAS2650) and both have the system attention LED lit and I have no idea why. `System health status show` as well as config advisor are both A-OK and there are no visible indicators why this LED is still lit. Any idea how to find out the root cause of this LED on the front bezel or how I can check or maybe reset the status of the LED through CLI? Thanks, Alexander Griesser Head of Systems Operations ANEXIA Internetdienstleistungs GmbH E-Mail: AGriesser(a)anexia-it.com<mailto:AGriesser@anexia-it.com> Web: http://www.anexia-it.com<http://www.anexia-it.com/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 _______________________________________________ Toasters mailing list Toasters(a)teaparty.net<mailto:Toasters@teaparty.net> http://www.teaparty.net/mailman/listinfo/toasters _______________________________________________ Toasters mailing list Toasters(a)teaparty.net<mailto:Toasters@teaparty.net> http://www.teaparty.net/mailman/listinfo/toasters _______________________________________________ Toasters mailing list Toasters(a)teaparty.net<mailto:Toasters@teaparty.net> http://www.teaparty.net/mailman/listinfo/toasters

2 1

qtree snapmirror inexplicably slow once a week
by BERNTSEN Basil 14 Dec '16

14 Dec '16

Hi folks, I'm at a loss. I have an unmaintained (fas2020) 7-mode HA pair in a site that's slated to be decommed, but for now, I need to keep the lights on. There is a qtree snapmirror that, aside from Mondays, runs in about 3-4 hours. Mondays, it takes 2-4 days. The change rate on the source doesn't appear to be any different. Also, this was working happily for years and only started acting up lately. There are only 5 disks on the system, and while they're at 100% now, they're also at 100% during a sample taken while this context wasn't updating, and while it was updating in a way that finished in 4 hours. So far, I've checked statit during the event and while idle, and I couldn't see anything that stuck out as a massive difference, except maybe name cache hits? I have 68% name cache hits during this transfer, and 100% while idle. I don't see anything suspicious in the wafl scan status, and reallocate status -v shows: Reallocation scans are on No reallocation status. CPU is idle, network has no errors. Does anyone have any suggestions? What kinds of things should I be looking at? Thanks, Basil ************************************************************************* This message and any attachments (the "message") are confidential, intended solely for the addressee(s), and may contain legally privileged information. Any unauthorized use or dissemination is prohibited. E-mails are susceptible to alteration. Neither SOCIETE GENERALE nor any of its subsidiaries or affiliates shall be liable for the message if altered, changed or falsified. Please visit http://sgasdisclosure.com for important information regarding SG Americas Securities, LLC (“SGAS”). Please visit http://swapdisclosure.sgcib.com for important information regarding swap transactions with SOCIETE GENERALE. *************************************************************************

3 3

"statistics aggregate show"
by Edward Rolison 13 Dec '16

13 Dec '16

I have been running (on CDOT 8.3): statistics aggregate show statistics volume show over ssh, via cron, and graphing them. Since updating to 8.3.2P6, I've now got a different error: Error: command failed: More than 50 samples are not allowed. Please delete existing sample(s) to create a new sample. Now, that's not a deprecation warning, but I'm wondering if the functionality has been disabled anyway. Or if there's a workaround that any of you fine folk have seen. Or - for that matter - what's the neatest way to grab some summary statistics suitable for graphing at 5m intervals. (I can probably hack something together using the API, but haven't really needed to until now) Thanks and regards, Ed.

2 1

FDMI reporting with qla2xxx driver on Linux
by Momonth 09 Dec '16

09 Dec '16

Hi, I recently noticed that qla2xxx (Qlogic HBAs) driver on Linux is capable of reporting FDMI. By default it sends "Linux-default", and I'm failing to see how to change it to actual host name as it becomes way more handy then =) Any ideas? Cheers, Vladimir

2 3

SP Image Downloads?
by Alexander Griesser 05 Dec '16

05 Dec '16

Hi there, I'm looking for SP firmware 3.1.2P1 for an AFF8040, but am only able to download 3.2.1 on the support site which is not compatible to my currently running version of OnTap. Does anyone have a deep link for this image or knows of a way to find download links for specific versions like the method used for downloading patch releases etc. for all other software shipments from NetApp? Thanks, Alexander Griesser Head of Systems Operations ANEXIA Internetdienstleistungs GmbH E-Mail: AGriesser(a)anexia-it.com<mailto:AGriesser@anexia-it.com> Web: http://www.anexia-it.com<http://www.anexia-it.com/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

4 12

ONTAP9 NFSv4 + CentOS 6.8 does not work
by Momonth 29 Nov '16

29 Nov '16

Hi All, We upgraded our 6 nodes cluster to ONTAP9 yesterday. Today I discovered some freshly installed CentOS 6.8 hosts not being able to mount file shares via NFS v4 protocol. The mount command just kind of hangs. I verified it's not a connectivity issue, ie connections to the filer's port 2049 are possible. Also NFS v3 works just fine. Did anyone experience anything like that? Cheers, Vladimir

7 11

CDOT snapmirror -forground
by Rhodes, Richard L. 28 Nov '16

28 Nov '16

Hello I am starting to look into a possible rollover of our 7-mode IBM N-Series systems to new NetApp CDOT systems. I run backups with snapmirror and snapvault that are controlled by a AIX ksh script. The ksh script runs the following cmds to perform the data transfers: snapmirror update -S ${filerpri}:${vol} -w ${filerdr}:${vol} snapvault snap create -w $vol $vol Note the '-w' parm. I run these cmds in forground, so all backups are single threaded and return a response back to the script. I'm looking at CDOT cmds manual for an equivalent '-w' parm. I find the following for the "snapmirror update" cmd: [-foreground | -w [true]] - Foreground Process This specifies whether the operation runs as a foreground process. If this parameter is specified, the default setting is true (the operation runs in the foreground). When set to true, the command will not return until the process completes. This parameter is only applicable to relationships with "Relationship Capability" of "Pre 8.2". Q) What does "applicable to relationships with "Relationship Capability" of "Pre 8.2" mean? Does this mean that a CDOT>8.2 system doesn't support this parm? Thanks! Rick ----------------------------------------- The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately, and delete the original message.

2 1

Flash Cache older 3020c??
by Chris hansen 26 Nov '16

26 Nov '16

I saw a few threads talking about flash cache and it made me think of my old 3020c that continues to chug along in my lab running 7.3.7. (568 days) Do any of these PAM II flash modules (111-00525+D0 for example) work in this FAS model or was there no flash option? I did some googling but there wasn’t much data on some of these older modules. -Chris

2 1

NFS V4 and nobody
by John Adams 16 Nov '16

16 Nov '16

I am mounting an export via NFS V4. At first, I saw that mounted clients showed user/group as nobody:nobody. But then I configured the nfs.v4.id.domain option on the filer to be the client machines DNS domain name. Then the username:groups were correct.. mostly... We use NIS on the client machines and the filer. For files that are owned by users or groups that are not in our NIS, the clients now shows nobody. Under NFS V3... this shows the UID/GID numbers (if they are not in NIS, and have no mappings). Anybody know how to make NFS V4 just show the UID/GID of entries that have no mapping instead of converting them to nobody? (UID's are no longer in our NIS after the account is deleted...) RHEL6/7. OnTap 8.2.4-7 Mode. Thanks.

3 3

Problems with SMB and macOS Sierra?
by James Andrewartha 09 Nov '16

09 Nov '16

Hi, We're testing macOS Sierra at the moment, and are seeing some weird problems connecting to SMB shares on our 8.2.4 7-mode filer. Listing directories is super-slow, and then after a while all the files get the no permissions icon and you can't open anything. Has anyone seen anything similar? Thanks, -- James Andrewartha Network & Projects Engineer Christ Church Grammar School Claremont, Western Australia Ph. (08) 9442 1757 Mob. 0424 160 877

4 7

Troubleshooting 'unreachable' on a CDOT port
by Edward Rolison 08 Nov '16

08 Nov '16

Hello fellow NetApp admins. Over the weekend, I hit a really quite "interesting" sort of a problem. One of those weekends that ... no one really wants to have. It involved a firmware upgrade on a switch going catastrophically wrong. causing chaos for several hours. And off the back of that - one of our two CDOT nodes, it's primary 'data' interface LACP group has .. seemingly died. I say "seemingly" because: - Snooping the interfaces sees packets going in and out. (Mostly "arp"). - But the switch side "snoop" doesn't see the ARP replies. And thus never 'learns' the mac, and doesn't route traffic to it. This happens on both ports of a LACP group, and even moving it to another switch entirely hasn't helped. Manually offlining the ports doesn't help (Except if I do both, it migrates the lif automatically). But switching the lifs over onto the other head - has fixed it, for now. (although obviously, we're failed over, and have reduced resilience). Has anyone run into anything similar? Or can give me some insight as to what could explain this perplexing behaviour? The 'source' of the problem was - probably - some serious network strangeness. Loops, vlans going up and down, all sorts of chaos. I haven't (yet) rebooted the failed node, as the vservers are running quite merrily. I'm wondering if there's some sort of DOS/arp flood protection that might be tripping us up. Thanks, Ed.

4 4

Re: sas.adapter.debug messages
by Mike Thompson 25 Oct '16

25 Oct '16

yeah it's supported per hardware universe gear isn't on support because money i could upgrade it, i'm just not going to, without support - had too many upgrades go sideways in the past, requiring intervention from support On Tue, Oct 25, 2016 at 10:55 AM, gene <genec220(a)verizon.net> wrote: > Then it is very possible that the gear isn’t supported by your older ONTAP > version. It might not even know about your gear and/or versions. Why > isn’t the gear on support? If you have a valid support account, can’t > you download a more current ONTAP version and use it regardless of the > non-support on the controllers? > > > > g > > > > *From:* toasters-bounces(a)teaparty.net [mailto:toasters-bounces@ > teaparty.net] *On Behalf Of *Mike Thompson > *Sent:* Tuesday, October 25, 2016 1:51 PM > *To:* NGC-tmacmd-gmail.com > *Cc:* toasters(a)teaparty.net Lists > *Subject:* Re: sas.adapter.debug messages > > > > gear isn't on support, so i'm not updating Ontap. > > all the SAS shelves brand new though, so should have latest firmware, but > i'll double check. > > >

2 1

sas.adapter.debug messages
by Mike Thompson 25 Oct '16

25 Oct '16

Hey all, Recently hooked up a ton of DS2246 w 900g drives to some 6080s and 6280s running (ancient) cDOT 8.0.5, and am seeing a lot of sas.adapter.debug messages when the aggregates in question are under heavy write load. Seeing this on multiple filers, multiple adapters, so I'm guessing it's a bug of some sort, which was likely fixed years ago? These don't appear to be impacting performance as far as I can tell, but I'm not certain (gear was just onlined, and using this as staging area for shuffling lots of volumes around). Are these particularly evil? Can't find much online about them, but they don't look very appealing: Oct 24 18:17:47 filer1a pmcsas_intrd_1: sas.adapter.debug: adapterName="5a", debug_string="OPEN timeout (0x13): dev 5b.13.13 (5:000039:7385204:66), cdb 0x2a:236f16d8:0200 (0/29311), NDU 0x0" Oct 24 18:17:51 filer1a pmcsas_intrd_1: sas.adapter.debug: adapterName="5a", debug_string="Bad device state completion: status 0x39, dev 5b.13.13, cdb 0x2a:236f16d8:0200" Oct 24 18:17:51 filer1a pmcsas_intrd_1: sas.adapter.debug: adapterName="5a", debug_string="Bad device state completion: status 0x39, dev 5b.13.13, cdb 0x2a:236f18d8:0200" Oct 24 18:17:51 filer1a pmcsas_intrd_1: sas.adapter.debug: adapterName="5a", debug_string="Bad device state completion: status 0x39, dev 5b.13.13, cdb 0x28:236f3620:0010" Oct 24 18:17:51 filer1a pmcsas_intrd_1: sas.adapter.debug: adapterName="5a", debug_string="Bad device state completion: status 0x39, dev 5b.13.13, cdb 0x28:1555e6f8:0080" Oct 24 18:17:51 filer1a pmcsas_intrd_1: sas.adapter.debug: adapterName="5a", debug_string="Bad device state completion: status 0x39, dev 5b.13.13, cdb 0x2a:236f1ad8:0200" Oct 24 18:17:51 filer1a pmcsas_intrd_1: sas.adapter.debug: adapterName="5a", debug_string="Bad device state completion: status 0x39, dev 5b.13.13, cdb 0x2f:041a4800:0400" Oct 24 19:21:46 filer1a pmcsas_intrd_2: sas.adapter.debug: adapterName="1a", debug_string="OPEN timeout (0x13): dev 1a.15.22 (5:000039:73850ad:19), cdb 0x2a:2368ed60:0200 (0/18656), NDU 0x0" Oct 24 19:21:49 filer1a pmcsas_intrd_2: sas.adapter.debug: adapterName="1a", debug_string="Bad device state completion: status 0x39, dev 1a.15.22, cdb 0x2a:2368ed60:0200" Oct 24 19:21:49 filer1a pmcsas_intrd_2: sas.adapter.debug: adapterName="1a", debug_string="Bad device state completion: status 0x39, dev 1a.15.22, cdb 0x2f:04dd0800:0400" Oct 24 19:24:01 filer1a pmcsas_intrd_2: sas.adapter.debug: adapterName="1a", debug_string="OPEN timeout (0x13): dev 1a.15.0 (5:000039:7481142:e5), cdb 0x2a:2382c048:0200 (0/33634), NDU 0x0" Oct 24 19:24:01 filer1a pmcsas_intrd_2: sas.adapter.debug: adapterName="1a", debug_string="Bad device state completion: status 0x39, dev 1a.15.0, cdb 0x2a:2382bc48:0200" Oct 24 19:24:01 filer1a pmcsas_intrd_2: sas.adapter.debug: adapterName="1a", debug_string="Bad device state completion: status 0x39, dev 1a.15.0, cdb 0x2a:2382c248:0200" Oct 24 19:30:02 filer1a pmcsas_intrd_2: sas.adapter.debug: adapterName="1a", debug_string="OPEN timeout (0x13): dev 1a.16.19 (5:000039:7480858:65), cdb 0x2a:23ca4d38:0200 (0/74568), NDU 0x0" Oct 24 19:30:02 filer1a pmcsas_intrd_2: sas.adapter.debug: adapterName="1a", debug_string="Bad device state completion: status 0x39, dev 1a.16.19, cdb 0x2a:23ca4f38:0200" Oct 24 19:31:21 filer1a pmcsas_intrd_2: sas.adapter.debug: adapterName="1a", debug_string="OPEN timeout (0x13): dev 1a.15.0 (5:000039:7481142:e5), cdb 0x2a:23f9dd80:0200 (0/439), NDU 0x0" Oct 24 19:31:21 filer1a pmcsas_intrd_2: sas.adapter.debug: adapterName="1a", debug_string="Bad device state completion: status 0x39, dev 1a.15.0, cdb 0x2a:23f9df80:0200" Oct 24 19:31:34 filer1a pmcsas_intrd_2: sas.adapter.debug: adapterName="1a", debug_string="OPEN timeout (0x13): dev 1a.15.2 (5:000039:7480859:bd), cdb 0x28:0871dd38:0008 (0/74660), NDU 0x0" Oct 24 19:31:34 filer1a pmcsas_intrd_2: sas.adapter.debug: adapterName="1a", debug_string="Bad device state completion: status 0x39, dev 1a.15.2, cdb 0x2a:23fca318:00e8" Oct 24 19:33:40 filer1a pmcsas_intrd_2: sas.adapter.debug: adapterName="1a", debug_string="OPEN timeout (0x13): dev 1a.16.6 (5:000039:7481143:01), cdb 0x2a:24176530:0198 (0/74786), NDU 0x0" Oct 24 19:33:40 filer1a pmcsas_intrd_2: sas.adapter.debug: adapterName="1a", debug_string="Bad device state completion: status 0x39, dev 1a.16.6, cdb 0x2a:241766c8:0200"

2 2

Culprit...
by Carl Howell 21 Oct '16

21 Oct '16

I've got an aggregate on a FAS2554c, running ONTAP 9, that runs at over 4500 read IOPS constantly. All of the volumes in the aggregate don't generate near that many IOPS and SnapMirror doesn't look like the cause either. I've used OCUM, statistics show...., etc, and haven't been able to find out what's causing this. Any suggestions? Thanks, --Carl

4 3

SnapMirror policy clarification
by Ehrenwald, Ian 17 Oct '16

17 Oct '16

Hello I'm trying to understand the differences between the snapshot policies included in cDOT 8.3, namely DPDefault and MirrorAllSnapshots. The documentation really isn't clear and apparently I can't see it myself. The behavior I'm seeing with a DP relationship using the DPDefault policy looks like it mirrors all snapshots and the active filesystem, which the MirrorAllSnapshots policy sounds like it should do. If I delete a snapshot on the source side and do an SM update, the snapshot is gone on the destination side too. I created a test to try and understand the differences between the policies, but maybe I'm looking at the wrong things because the results of the tests were identical: - Created testvolA on the source and SM_testvolA on the destination - Created testvolB on the source and SM_testvolB on the destination - Created SnapMirror relationship between testvolA and SM_testvolA with a SM policy of MirrorAllSnapshots - Created SnapMirror relationship between testvolB and SM_testvolB with a SM policy of DPDefault - Created a snapshot on each source volume - Initialized SnapMirror relationships - Examined output of snap list SM_testvol*, identical - Created another snapshot on each source volume - Updated SnapMirror relationships - Examined output of snap list SM_testvol*, identical - Deleted first snapshot on each source volume - Updated SnapMirror relationships - Examined output of snap list SM_testvol*, identical So, I'm having a difficult time understanding what the point of these policies are. Can someone enlighten me? Ian Ehrenwald Hachette Book Group, Inc. This may contain confidential material. If you are not an intended recipient, please notify the sender, delete immediately, and understand that no disclosure or reliance on the information herein is permitted. Hachette Book Group may monitor email to and from our network.

3 3

IOM3 and IOM6 on same loop
by John Adams 10 Oct '16

10 Oct '16

Hello, I currently have 3 DS4243's with IOM3 connected to my 8020 HA pair. I also have DS2246 with IOM6 as the last shelf on the loop. I want to add a another DS2246 with IOM6 to the loop. I know that IOM3 and IOM6 can exist on the same loop as long as there was only one IOM3 to IOM6 transition. Will there be any issue adding the 2nd shelf with IOM6 to this loop? i.e.: current: IOM3 -> IOM3 -> IOM3 -> IOM6 new: IOM3 -> IOM3 -> IOM3 -> IOM6 -> IOM6 Thanks.

5 5

RE: Frame discards on Brocade with 7-Mode FMC using 8G HBA and DS14
by andrei.borzenkov＠ts.fujitsu.com 04 Oct '16

04 Oct '16

Hi, Yes, UTA2 do not support direct connection of DS14 because they do not support FC-AL. They are supported in FMC though for connection to backend switches and that is what we have here. It is clear that setting HBA to 8G won’t make DS14 faster. Still customer has also ATTO 6500N and any future storage extension will go over ATTO as well so it is still beneficial to use full available speed. --- With best regards Andrei Borzenkov Senior system engineer FTS WEMEAI RUC RU SC TMS FOS [cid:image001.gif@01CBF835.B3FEDA90] FUJITSU Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation Tel.: +7 495 730 62 20 ( reception) Mob.: +7 916 678 7208 Fax: +7 495 730 62 14 E-mail: Andrei.Borzenkov(a)ts.fujitsu.com<mailto:Andrei.Borzenkov@ts.fujitsu.com> Web: ru.fujitsu.com<http://ts.fujitsu.com/> Company details: ts.fujitsu.com/imprint<http://ts.fujitsu.com/imprint.html> This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation. Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission. From: gene [mailto:genec220@verizon.net] Sent: Monday, October 03, 2016 8:51 PM To: 'Tim McCarthy'; Borzenkov, Andrei Subject: RE: Frame discards on Brocade with 7-Mode FMC using 8G HBA and DS14 Just making sure you got this since my post bounced from toasters… <assuming you were talking about DS14 connections and not FC SAN targets> The DS14 shelves are supported on the UTA2 ports. You would need FC HBAs such as the X1132, or X2054. “The FAS80x0 onboard UTA2 ports and X1143A UTA2 ports are not supported for connectivity to DS14 shelves. To connect to DS14 shelves, use the X1132A-R6 or X2054A-R6 Fibre Channel adapters.” Also, unless your complete path is 8Gb, you won’t get the 8Gb. The 8Gb links would auto-negotiate down to 4Gb. Shouldn’t error though. You won’t get any better perf out of the DS14’s by setting the path links to 8Gb either since the fastest the DS14MK4 can run is 4Gb. {good seeing you, although a mere glance, at Insight, Tim. I continue to learn from your posts. Hope you are well.} Good luck, gene From: gene [mailto:genec220@verizon.net] Sent: Monday, October 03, 2016 11:34 AM To: 'Tim McCarthy'; NGC-andrei.borzenkov-ts.fujitsu.com; NGC-andrei.borzenkov-ts.fujitsu.com Cc: toasters(a)teaparty.net<mailto:toasters@teaparty.net> Subject: RE: Frame discards on Brocade with 7-Mode FMC using 8G HBA and DS14 Hi Andrei, I do not want to assume even though you say “backend switches” so are you trying to use the UTA2 ports to connect to the DS14’s or as FC SAN target ports? gene From: toasters-bounces(a)teaparty.net<mailto:toasters-bounces@teaparty.net> [mailto:toasters-bounces@teaparty.net] On Behalf Of Tim McCarthy Sent: Monday, October 03, 2016 7:47 AM To: NGC-andrei.borzenkov-ts.fujitsu.com; NGC-andrei.borzenkov-ts.fujitsu.com Cc: toasters(a)teaparty.net<mailto:toasters@teaparty.net> Subject: Re: Frame discards on Brocade with 7-Mode FMC using 8G HBA and DS14 You would not believe the silly stuff I have seen customers do. Get Outlook for iOS<https://aka.ms/o0ukef> On Mon, Oct 3, 2016 at 7:40 AM -0400, "andrei.borzenkov(a)ts.fujitsu.com<mailto:andrei.borzenkov@ts.fujitsu.com>" <andrei.borzenkov(a)ts.fujitsu.com<mailto:andrei.borzenkov@ts.fujitsu.com>> wrote: What FC ports are you using on the FAS8040 (onboard/PCIe Card)? [Borzenkov, Andrei] onboard how about the sysconfig -a for the FC ports involved (not the whole thing!)? [Borzenkov, Andrei] slot 0: FC Host Adapter 0e (QLogic 8324 rev. 2, N-port, <UP>) Firmware rev: 7.5.0 Host Port Id: 0x10100 FC Node Name: 5:00a:098003:8826d0 FC Port Name: 5:00a:098003:8826d0 SFP Vendor: AVAGO SFP Part Number: AFBR-57F5MZ-NA1 SFP Serial Number: AC1609J00D5 SFP Capabilities: 4, 8 or 16 Gbit Others are the same Do you have the correct SFP+s in the FAS8040? [Borzenkov, Andrei] Whatever NetApp ships with FAS8040. Where did they come from? [Borzenkov, Andrei] NetApp Did you take them from the FAS3140 (X6539 -> 4Gb)? [Borzenkov, Andrei] Of course, not. You seriously expect link to come up at 8G with 4G SFPs? (if so, those are likely 4Gb) The supported ones on the FAS8040 are X6596->16Gb [Borzenkov, Andrei] Yes, I know. Of course, if you are using a PCIe crad, that would be different too. --tmac Tim McCarthy, Principal Consultant Proud Member of the #NetAppATeam<https://twitter.com/NetAppATeam> I Blog at TMACsRack<https://tmacsrack.wordpress.com/> [NetApp - In partnership with Alpine Testing Solutions][NetApp Certified Data Administrator, ONTAP]<https://www.certmetrics.com/netapp/public/badge.aspx?t=c&d=2012-11-05&i=35&…>[NetApp Certified Implementation Engineer - SAN Specialist, ONTAP]<https://www.certmetrics.com/netapp/public/badge.aspx?t=c&d=2012-11-08&i=36&…>[NetApp Certified Storage Installation Engineer, ONTAP]<https://www.certmetrics.com/netapp/public/badge.aspx?t=c&d=2015-10-13&i=38&…>[NetApp Certified Implementation Engineer - Data Protection Specialist]<https://www.certmetrics.com/netapp/public/badge.aspx?t=c&d=2015-10-15&i=11&…>[https://dl.dropboxusercontent.com/u/6874230/FlexPod2.jpg][http://dl.dropbox…] NetApp Candidate ID: NETAPP00041276 FlexPod Design: Oct 2015 - Jan 2018, S0N62WE1BMVEYF3M FlexPod Implementation: Oct 2015 - Jan 2018, JH3QJT4KLEQ41HPH RHCE6 110-107-141<https://www.redhat.com/wapps/training/certification/verify.html?certNumber=…> On Mon, Oct 3, 2016 at 5:32 AM, andrei.borzenkov(a)ts.fujitsu.com<mailto:andrei.borzenkov@ts.fujitsu.com> <andrei.borzenkov(a)ts.fujitsu.com<mailto:andrei.borzenkov@ts.fujitsu.com>> wrote: This weekend I did head swap from FAS3140 to FAS8040 in 7-Mode FMC. Storage is DS14mk4 and one DS2246 that was added later. Backend switches are Brocade 300 so we tried to change HBA ports from 4G to 8G hoping to boost overall performance. But almost immediately I observed intermittent Sat Oct 1 23:00:08 YEKT [netapp2:cf.fsm.takeoverOfPartnerDisabled:error]: Failover monitor: takeover of netapp1 disabled (status of backup mailbox is uncertain). Sat Oct 1 23:00:14 YEKT [netapp2:cf.fsm.takeoverOfPartnerEnabled:notice]: Failover monitor: takeover of netapp1 enabled On both heads. There was no load, all applications were still stopped at this point. There were no other errors visible in logs. Looking on switches, the only errors reported for ports were c3 discards. They were most frequent on DS14 ports, but also happened for HBA and even ATTO ports (1 and 13 in this example). Statistic is collected several minutes after resetting counters. netapp2_fcsw1:admin> porterrshow frames enc crc crc too too bad enc disc link loss loss frjt fbsy c3timeout tx rx in err g_eof shrt long eof out c3 fail sync sig tx rx ======================================================================================================================= 0: 124.5k 90.9k 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1: 327.5k 290.8k 0 0 0 0 0 0 0 3 0 0 0 0 0 0 3 2: 278.4k 244.0k 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3: 27.0k 29.1k 0 0 0 0 0 0 0 102 0 0 0 0 0 0 102 4: 426.5k 474.3k 0 0 0 0 0 0 0 324 0 0 0 0 0 324 0 5: 41.9k 48.9k 0 0 0 0 0 0 0 120 0 0 0 0 0 0 120 6: 98.8k 109.8k 0 0 0 0 0 0 0 6 0 0 0 0 0 0 6 7: 111.4k 133.1k 0 0 0 0 0 0 0 2 0 0 0 0 0 0 2 8: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 9: 56.2k 64.0k 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 10: 13.4k 14.9k 0 0 0 0 0 0 0 52 0 0 0 0 0 0 52 11: 46.4k 50.4k 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 12: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 13: 22.0k 23.8k 0 0 0 0 0 0 0 38 0 0 0 0 0 0 38 14: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 15: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 17: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 18: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 19: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 20: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 21: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 22: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 23: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 I could not find any best practice or recommendation regarding port speeds in such configuration (actually, documentation recommends setting port speed to maximum supported by connected device, which in this case means 8G). After forcing HBA back to 4G the above messages and c3 discards went away. So I'm looking for practical experience in this area. May be there are some settings we missed. Switches are running 7.0.2b (minimal version matching IMT). Shelf firmware is the latest. Data ONTAP is 8.2.4P4. --- With best regards Andrei Borzenkov Senior system engineer FTS WEMEAI RUC RU SC TMS FOS FUJITSU Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation Tel.: +7 495 730 62 20 ( reception) Mob.: +7 916 678 7208 Fax: +7 495 730 62 14 E-mail: Andrei.Borzenkov(a)ts.fujitsu.com<mailto:Andrei.Borzenkov@ts.fujitsu.com> Web: ru.fujitsu.com<http://ru.fujitsu.com> Company details: ts.fujitsu.com/imprint<http://ts.fujitsu.com/imprint> This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation. Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission. _______________________________________________ Toasters mailing list Toasters(a)teaparty.net<mailto:Toasters@teaparty.net> http://www.teaparty.net/mailman/listinfo/toasters

1 0

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters