toasters

toasters@lists.teaparty.net

2 participants
13520 discussions

cluster mode volume stats since boot
by Mike Thompson 18 Apr '17

18 Apr '17

any way to get volume stats (reads/writes/ops/etc) since system boot or volume move, in cdot 8.2 and above? I used to track all that stuff in a database in the early 8.x days, via 'statistics show -object volume', but that disappeared in 8.2 (or maybe it was 8.1, not sure) looks like you can only display real time volume stats in more recent ontap releases via cli. I used to collect this info hourly, stuff it in a db, and then diff it over time to display activity per-volume, which would tell us if a volume was busy or not, and what type of traffic it was getting, over time. that info is sorely missed from some of our applications, and hence I'm looking for per-volume accumulators. looks like i can get nfs/cifs ops per qtree still, but what i really want is reads and writes (and anything else) per volume. same numbers for aggregates would be useful as well. any ideas?

3 3

Re: Switching to ACP inband connectivity
by tmac 14 Apr '17

14 Apr '17

Yep. internal burt, but a burt none the less. save them from digging it up. Let us know how you make out. I was under a crunch putting a system into production and could not fool around with it any more. Thanks --tmac *Tim McCarthy, **Principal Consultant* *Proud Member of the #NetAppATeam <https://twitter.com/NetAppATeam>* *I Blog at TMACsRack <https://tmacsrack.wordpress.com/>* On Sat, Apr 8, 2017 at 12:29 PM, Alexander Griesser <AGriesser(a)anexia-it.com > wrote: > Alright, thanks – I’m opening a case now, cannot see the details of the > BURT, though, but will refer support to it. > > > > Best, > > > > *Alexander Griesser* > > Head of Systems Operations > > > > ANEXIA Internetdienstleistungs GmbH > > > > E-Mail: AGriesser(a)anexia-it.com > > Web: http://www.anexia-it.com > > > > Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt > > Geschäftsführer: Alexander Windbichler > > Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT > U63216601 > > > > *Von:* tmac [mailto:tmacmd@gmail.com] > *Gesendet:* Samstag, 8. April 2017 14:05 > *An:* Alexander Griesser <AGriesser(a)anexia-it.com> > *Cc:* toasters(a)teaparty.net > *Betreff:* Re: Switching to ACP inband connectivity > > > > I have seen this! Support was unable to reproduce for me. I will dig up a > case number later if you like. > > > > I will give you two options: > > 1. Open a support case with netapp and refer to the BURT 1050364 / Case > 2006580487 > > 2. The ultimate fix that I did was this: > > -> Downgrade *all* IOM firmware to 2.09. Wait for it to stabilize. > > -> Convert to in-band. Wait for it to stabilize and confirm IN-Band > > -> Convert to out-of-band and upgrade to 2.10 firmware. > > -> Wait for it to stabilize and then convert to out-of-band > > > > I got to believe there may be some race condition where the 2.08 -> 2.10 > firmware is not completing 100%. > > If you use the out-of-band, all looks ok, but switch to in-band and not so > much. > > > > The thought is something special happens on 2.09 that fixes everything > allowing the upgrade to 2.10 > > > > If possible, please open a case and refer to the data above. I think it is > in everyones best interest NetApp gets to the bottom of this to fix. > > > > Thanks > > > > > --tmac > > > > *Tim McCarthy, **Principal Consultant* > > *Proud Member of the #NetAppATeam <https://twitter.com/NetAppATeam>* > > *I Blog at **TMACsRack <https://tmacsrack.wordpress.com/>* > > > > > > On Sat, Apr 8, 2017 at 5:55 AM, Alexander Griesser < > AGriesser(a)anexia-it.com> wrote: > > Hey Toasters, > > > > I’ve just upgraded one of my HA-pairs to 9.1P2 and wanted to switch to ACP > inband connectivity following this KB: > > https://kb.netapp.com/support/s/article/ka31A00000012NsQAI/ > in-band-acp-setup-and-support > > > > According to this KB, I should be eligible for the switch: > > > > ::*> storage shelf acp module show -fields module-name,protocol-version, > firmware-version,shelf-serial-number,iom-type,state > > node module-name protocol-version firmware-version shelf-serial-number > iom-type state > > ------- ----------- ---------------- ---------------- ------------------- > -------- ------ > > nodeA 1.22.B 1.2.02.10 02.10 SHJMS1424000060 > iom6 active > > nodeA 1.22.A 1.2.02.10 02.10 SHJMS1424000060 > iom6 active > > nodeA 2.11.A 1.2.02.10 02.10 SHFMS1423000507 > iom6 active > > nodeA 2.11.B 1.2.02.10 02.10 SHFMS1423000507 > iom6 active > > nodeA 2.14.B 1.2.02.10 02.10 SHFMS1423000491 > iom6 active > > nodeA 2.14.A 1.2.02.10 02.10 SHFMS1423000491 > iom6 active > > nodeA 2.12.A 1.2.02.10 02.10 SHFMS1423000493 > iom6 active > > nodeA 2.12.B 1.2.02.10 02.10 SHFMS1423000493 > iom6 active > > nodeA 2.13.A 1.2.02.10 02.10 SHFMS1423000504 > iom6 active > > nodeA 2.13.B 1.2.02.10 02.10 SHFMS1423000504 > iom6 active > > nodeA 1.21.B 1.2.02.10 02.10 SHJMS1424000434 > iom6 active > > nodeA 1.21.A 1.2.02.10 02.10 SHJMS1424000434 > iom6 active > > nodeB > > 1.22.B 1.2.02.10 02.10 SHJMS1424000060 > iom6 active > > nodeB > > 1.22.A 1.2.02.10 02.10 SHJMS1424000060 > iom6 active > > nodeB > > 2.11.A 1.2.02.10 02.10 SHFMS1423000507 > iom6 active > > nodeB > > 2.11.B 1.2.02.10 02.10 SHFMS1423000507 > iom6 active > > nodeB > > 2.14.B 1.2.02.10 02.10 SHFMS1423000491 > iom6 active > > nodeB > > 2.14.A 1.2.02.10 02.10 SHFMS1423000491 > iom6 active > > nodeB > > 2.12.A 1.2.02.10 02.10 SHFMS1423000493 > iom6 active > > nodeB > > 2.12.B 1.2.02.10 02.10 SHFMS1423000493 > iom6 active > > nodeB > > 2.13.A 1.2.02.10 02.10 SHFMS1423000504 > iom6 active > > nodeB > > 2.13.B 1.2.02.10 02.10 SHFMS1423000504 > iom6 active > > nodeB > > 1.21.B 1.2.02.10 02.10 SHJMS1424000434 > iom6 active > > nodeB > > 1.21.A 1.2.02.10 02.10 SHJMS1424000434 > iom6 active > > 24 entries were displayed. > > > > > > The requirements state that the IOM6 modules must be upgraded to 2.09 (I > have 2.10) and ACP firmware must be at least 1.2.2.9 (I have 1.2.2.10). > > So I went ahead and ran this command: > > > > storage shelf acp configure -is-enabled true -channel *in-band* > > > > After a few minutes, I saw this: > > > > ::*> storage shelf acp show > > Node Channel Connectivity > > ------------------ --------------- ---------------------- > > nodeA in-band active > > nodeB in-band active > > 2 entries were displayed. > > > > But when I ran the initial check command again, I saw this output: > > > > ::*> storage shelf acp module show -fields module-name,protocol-version, > firmware-version,shelf-serial-number,iom-type,state > > node module-name protocol-version firmware-version shelf-serial-number > iom-type state > > ------- ----------- ---------------- ---------------- ------------------- > -------- ------------------------ > > nodeA 1.22.B 1.2.02.10 02.10 SHJMS1424000060 > iom6 firmware-update-required > > nodeA 1.22.A 1.2.02.10 02.10 SHJMS1424000060 > iom6 firmware-update-required > > nodeA 2.11.A 1.2.02.10 02.10 SHFMS1423000507 > iom6 firmware-update-required > > nodeA 2.11.B 1.2.02.10 02.10 SHFMS1423000507 > iom6 firmware-update-required > > nodeA 2.14.B 1.2.02.10 02.10 SHFMS1423000491 > iom6 firmware-update-required > > nodeA 2.14.A 1.2.02.10 02.10 SHFMS1423000491 > iom6 firmware-update-required > > nodeA 2.12.A 1.2.02.10 02.10 SHFMS1423000493 > iom6 firmware-update-required > > nodeA 2.12.B 1.2.02.10 02.10 SHFMS1423000493 > iom6 firmware-update-required > > nodeA 2.13.A 1.2.02.10 02.10 SHFMS1423000504 > iom6 firmware-update-required > > nodeA 2.13.B 1.2.02.10 02.10 SHFMS1423000504 > iom6 firmware-update-required > > nodeA 1.21.B 1.2.02.10 02.10 SHJMS1424000434 > iom6 firmware-update-required > > nodeA 1.21.A 1.2.02.10 02.10 SHJMS1424000434 > iom6 firmware-update-required > > nodeB > > 1.22.B 1.2.02.10 02.10 SHJMS1424000060 > iom6 firmware-update-required > > nodeB > > 1.22.A 1.2.02.10 02.10 SHJMS1424000060 > iom6 firmware-update-required > > nodeB > > 2.11.A 1.2.02.10 02.10 SHFMS1423000507 > iom6 firmware-update-required > > nodeB > > 2.11.B 1.2.02.10 02.10 SHFMS1423000507 > iom6 firmware-update-required > > nodeB > > 2.14.B 1.2.02.10 02.10 SHFMS1423000491 > iom6 firmware-update-required > > nodeB > > 2.14.A 1.2.02.10 02.10 SHFMS1423000491 > iom6 firmware-update-required > > nodeB > > 2.12.A 1.2.02.10 02.10 SHFMS1423000493 > iom6 firmware-update-required > > nodeB > > 2.12.B 1.2.02.10 02.10 SHFMS1423000493 > iom6 firmware-update-required > > nodeB > > 2.13.A 1.2.02.10 02.10 SHFMS1423000504 > iom6 firmware-update-required > > nodeB > > 2.13.B 1.2.02.10 02.10 SHFMS1423000504 > iom6 firmware-update-required > > nodeB > > 1.21.B 1.2.02.10 02.10 SHJMS1424000434 > iom6 firmware-update-required > > nodeB > > 1.21.A 1.2.02.10 02.10 SHJMS1424000434 > iom6 firmware-update-required > > 24 entries were displayed. > > > > Out of curiosity, I tried to run the ACP update again: > > > > nodeB> storage download acp > > > > Downloading ACP firmware can take several minutes > > and will not disrupt client access during that time. > > However, normal ACP recovery capabilities will not be > > available while the firmware upgrade is in progress. > > > > Are you sure you want to continue with ACP processor firmware update? y > > [storage download acp] Firmware will not be downloaded to module 2b.14.B > as ACPP state is not active, Current ACPP state is 0x9. > > [storage download acp] Firmware will not be downloaded to module 0a.14.A > as ACPP state is not active, Current ACPP state is 0x9. > > [storage download acp] Firmware will not be downloaded to module 0b.21.B > as ACPP state is not active, Current ACPP state is 0x9. > > [storage download acp] Firmware will not be downloaded to module 0a.13.A > as ACPP state is not active, Current ACPP state is 0x9. > > [storage download acp] Firmware will not be downloaded to module 0a.12.A > as ACPP state is not active, Current ACPP state is 0x9. > > [storage download acp] Firmware will not be downloaded to module 2a.21.A > as ACPP state is not active, Current ACPP state is 0x9. > > [storage download acp] Firmware will not be downloaded to module 2b.12.B > as ACPP state is not active, Current ACPP state is 0x9. > > [storage download acp] Firmware will not be downloaded to module 2b.13.B > as ACPP state is not active, Current ACPP state is 0x9. > > [storage download acp] Firmware will not be downloaded to module 0b.22.B > as ACPP state is not active, Current ACPP state is 0x9. > > [storage download acp] Firmware will not be downloaded to module 2a.22.A > as ACPP state is not active, Current ACPP state is 0x9. > > [storage download acp] Firmware will not be downloaded to module 0a.11.A > as ACPP state is not active, Current ACPP state is 0x9. > > [storage download acp] Firmware will not be downloaded to module 2b.11.B > as ACPP state is not active, Current ACPP state is 0x9. > > nodeB> exit > > > > Eventlog also said, that a FW Upgrade is required: > > > > ::*> event log show -severity infORMATIONAL > > Time Node Severity Event > > ------------------- ---------------- ------------- > --------------------------- > > 4/8/2017 11:41:15 nodeB INFORMATIONAL sfu.firmwareUpToDate: > Firmware is up-to-date on all disk shelves. > > 4/8/2017 11:39:37 nodeA INFORMATIONAL acp.need.firmware.update: > ACP module (NA) needs a firmware upgrade, (disk shelf serial number: > SHFMS1423000507). > > 4/8/2017 11:39:37 nodeA INFORMATIONAL acp.need.firmware.update: > ACP module (NA) needs a firmware upgrade, (disk shelf serial number: > SHFMS1423000504). > > 4/8/2017 11:39:37 nodeA INFORMATIONAL acp.need.firmware.update: > ACP module (NA) needs a firmware upgrade, (disk shelf serial number: > SHFMS1423000491). > > 4/8/2017 11:39:37 nodeA INFORMATIONAL acp.need.firmware.update: > ACP module (NA) needs a firmware upgrade, (disk shelf serial number: > SHFMS1423000493). > > 4/8/2017 11:39:37 nodeA INFORMATIONAL acp.need.firmware.update: > ACP module (NA) needs a firmware upgrade, (disk shelf serial number: > SHJMS1424000060). > > 4/8/2017 11:39:37 nodeA INFORMATIONAL acp.need.firmware.update: > ACP module (NA) needs a firmware upgrade, (disk shelf serial number: > SHJMS1424000434). > > 4/8/2017 11:39:37 nodeB INFORMATIONAL acp.need.firmware.update: > ACP module (NA) needs a firmware upgrade, (disk shelf serial number: > SHFMS1423000493). > > 4/8/2017 11:39:37 nodeB INFORMATIONAL acp.need.firmware.update: > ACP module (NA) needs a firmware upgrade, (disk shelf serial number: > SHFMS1423000504). > > 4/8/2017 11:39:37 nodeB INFORMATIONAL acp.need.firmware.update: > ACP module (NA) needs a firmware upgrade, (disk shelf serial number: > SHFMS1423000507). > > 4/8/2017 11:39:37 nodeB INFORMATIONAL acp.need.firmware.update: > ACP module (NA) needs a firmware upgrade, (disk shelf serial number: > SHFMS1423000491). > > 4/8/2017 11:39:37 nodeB INFORMATIONAL acp.need.firmware.update: > ACP module (NA) needs a firmware upgrade, (disk shelf serial number: > SHJMS1424000434). > > 4/8/2017 11:39:37 nodeB INFORMATIONAL acp.need.firmware.update: > ACP module (NA) needs a firmware upgrade, (disk shelf serial number: > SHJMS1424000060). > > > > I then switched back to OOB connectivity, and all is good again: > > > > ::*> storage shelf acp show > > Node Channel Connectivity > > ------------------ --------------- ---------------------- > > nodeA out-of-band full-connectivity > > nodeB out-of-band full-connectivity > > 2 entries were displayed. > > > > Tried to run the ACP upgrade again, getting this: > > > > nodeB> storage download acp > > > > Downloading ACP firmware can take several minutes > > and will not disrupt client access during that time. > > However, normal ACP recovery capabilities will not be > > available while the firmware upgrade is in progress. > > > > Are you sure you want to continue with ACP processor firmware update? y > > [storage download acp] ACP will check 12 modules for firmware download. > > [storage download acp] ACP module 192.168.0.124 is not eligible for update. > > [storage download acp] ACP module 192.168.0.126 is not eligible for update. > > [storage download acp] ACP module 192.168.0.180 is not eligible for update. > > [storage download acp] ACP module 192.168.0.202 is not eligible for update. > > [storage download acp] ACP module 192.168.0.208 is not eligible for update. > > [storage download acp] ACP module 192.168.0.248 is not eligible for update. > > [storage download acp] ACP module 192.168.1.112 is not eligible for update. > > [storage download acp] ACP module 192.168.1.170 is not eligible for update. > > [storage download acp] ACP module 192.168.1.238 is not eligible for update. > > [storage download acp] ACP module 192.168.2.66 is not eligible for update. > > [storage download acp] ACP module 192.168.3.130 is not eligible for update. > > [storage download acp] ACP module 192.168.3.138 is not eligible for update. > > nodeB> exit > > > > And now I’m back at the starting point. > > Any ideas what’s going on here? Why does it tell me > „firmware-update-required“ when I’m already on the latest versions? > > > > Thanks, > > > > *Alexander Griesser* > > Head of Systems Operations > > > > ANEXIA Internetdienstleistungs GmbH > > > > E-Mail: AGriesser(a)anexia-it.com > > Web: http://www.anexia-it.com > > > > Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt > > Geschäftsführer: Alexander Windbichler > > Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT > U63216601 > > > > > _______________________________________________ > Toasters mailing list > Toasters(a)teaparty.net > http://www.teaparty.net/mailman/listinfo/toasters > > >

2 6

Subselect-ish behavior?
by Ehrenwald, Ian 13 Apr '17

13 Apr '17

Conceptually this is what I’d like to do: volume show-footprint -fields total-footprint { volume show -clone-volume true -fields volume } Eg, show the total on disk footprint of all FlexClones. I could have sworn I saw a very brief example of this on Justin Parisi’s blog, taking the output of one command and feeding it into another. At the time, I thought “oh that’s cool” and moved on, but now can’t find reference to it. Either I don’t remember the syntax, it isn’t supported in 8.3.2, or it never happened and I’m slowly going crazy? -- Ian Ehrenwald Senior Infrastructure Engineer Hachette Book Group, Inc. 1.617.263.1948 / ian.ehrenwald(a)hbgusa.com This may contain confidential material. If you are not an intended recipient, please notify the sender, delete immediately, and understand that no disclosure or reliance on the information herein is permitted. Hachette Book Group may monitor email to and from our network.

4 7

RE: Windows NFS Client + cDOT
by Parisi, Justin 11 Apr '17

11 Apr '17

Well, access denied is definitely a different behavior than I was seeing. However, I was renaming files. When I rename a directory, I see the access issue: Z:\>ren "New Text Document.txt" test2.txt Z:\>ren "New folder" testdir Access is denied. Even when I change the access to 777, it still fails. # chmod 777 New\ folder [root@centos6 flexvol]# ls -la total 24 drwxrwxrwx. 5 root root 4096 Apr 11 09:41 . dr-xr-xr-x. 36 root root 4096 Apr 7 10:30 .. drwxrwxrwx. 2 root root 4096 Apr 11 09:39 New folder drwxr-xr-x. 2 root root 4096 Apr 11 09:41 New folder (2) drwxr-xr-x. 2 root root 4096 Apr 11 09:41 New folder (3) drwxrwxrwx. 10 root root 4096 Apr 11 09:05 .snapshot -rwxr-xr-x. 1 root root 0 Apr 10 15:45 test2.txt -rwxr-xr-x. 1 root root 0 Apr 10 15:46 test3.txt Z:\>ren "New folder" testdir Access is denied. So I dug around in our internal bug pages and found that if you don’t have showmount enabled, this can fail. So I enabled showmount on the NFS server and cleared the cache and remounted. Voila! ontap9-tme-8040::*> export-policy cache flush -vserver DEMO -cache all Warning: You are about to flush the "all (but showmount)" cache for Vserver "DEMO" on node "ontap9-tme-8040-02", which will result in increased traffic to the name servers. Do you want to proceed with flushing the cache? {y|n}: y ontap9-tme-8040::*> export-policy cache flush -vserver DEMO -cache showmount Warning: You are about to flush the "showmount" cache for Vserver "DEMO" on node "ontap9-tme-8040-02", which will result in increased traffic to the name servers. Do you want to proceed with flushing the cache? {y|n}: y C:\Users\Administrator>mount \\demo\flexvol Z: Z: is now successfully connected to \\demo\flexvol The command completed successfully. C:\Users\Administrator>Z: Z:\>ren "New folder (2)" testdir2 Z:\>dir Volume in drive Z has no label. Volume Serial Number is 80F0-372F Directory of Z:\ 04/11/2017 11:39 AM <DIR> . 04/11/2017 11:39 AM <DIR> .. 04/11/2017 09:41 AM <DIR> testdir2 04/10/2017 03:45 PM 0 test2.txt 04/10/2017 03:46 PM 0 test3.txt 04/11/2017 09:41 AM <DIR> New folder (3) 04/11/2017 09:39 AM <DIR> testdir 04/11/2017 11:05 AM <DIR> .snapshot 2 File(s) 24,576 bytes 6 Dir(s) 1,044,531,904,512 bytes free [cid:image003.png@01D2B2B9.E9D4FFB0] For completeness sake, I disabled showmount, cleared cache and remounted and saw it fail again: ontap9-tme-8040::*> nfs server modify -vserver DEMO -showmount disabled ontap9-tme-8040::*> export-policy cache flush -vserver DEMO -cache showmount Warning: You are about to flush the "showmount" cache for Vserver "DEMO" on node "ontap9-tme-8040-02", which will result in increased traffic to the name servers. Do you want to proceed with flushing the cache? {y|n}: y C:\Users\Administrator>umount X: Disconnecting X: \\10.193.67.237\flexvol The command completed successfully. C:\Users\Administrator>mount \\10.193.67.237\flexvol X: X: is now successfully connected to \\10.193.67.237\flexvol The command completed successfully. C:\Users\Administrator>X: X:\>dir Volume in drive X has no label. Volume Serial Number is 80F0-372F Directory of X:\ 04/11/2017 11:47 AM <DIR> . 04/11/2017 11:47 AM <DIR> .. 04/11/2017 09:39 AM <DIR> testdir2 04/10/2017 03:45 PM 0 test2.txt 04/10/2017 03:46 PM 0 test3.txt 04/11/2017 09:41 AM <DIR> New folder (3) 04/11/2017 09:41 AM <DIR> testdir 04/11/2017 11:05 AM <DIR> .snapshot 2 File(s) 24,576 bytes 6 Dir(s) 1,044,531,773,440 bytes free X:\>ren testdir testdirnew Access is denied. Looks like I’ll be adding that to the TR. ☺ From: Alexander Griesser [mailto:AGriesser@anexia-it.com] Sent: Tuesday, April 11, 2017 3:16 AM To: Parisi, Justin <Justin.Parisi(a)netapp.com>; NGC-tmacmd-gmail.com <tmacmd(a)gmail.com> Cc: toasters(a)teaparty.net Subject: AW: Windows NFS Client + cDOT Be glad that this hotfix did not install on your system – I’ve installed it on one test machine with 2k12 R2 and now it’s giving me BSODs after establishing either a remote session or logging in on the console in nfsrdr.sys, so this hotfix does not work at all I guess. Regarding the command line renaming: I’ve tried that initially and have sent the results earlier, you must have missed that. I’m unable to rename the files on the command line either, getting „Access denied“, here’s the quote again: On the command line, a different error (Access denied) is given: Z:\>move test test1 Access is denied. 0 dir(s) moved. Z:\>ren test test1 Access is denied. Any idea what I’m missing here now? Alexander Griesser Head of Systems Operations ANEXIA Internetdienstleistungs GmbH E-Mail: AGriesser(a)anexia-it.com<mailto:AGriesser@anexia-it.com> Web: http://www.anexia-it.com<http://www.anexia-it.com/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 Von: Parisi, Justin [mailto:Justin.Parisi@netapp.com] Gesendet: Montag, 10. April 2017 22:29 An: Alexander Griesser <AGriesser(a)anexia-it.com<mailto:AGriesser@anexia-it.com>>; NGC-tmacmd-gmail.com <tmacmd(a)gmail.com<mailto:tmacmd@gmail.com>> Cc: toasters(a)teaparty.net<mailto:toasters@teaparty.net> Betreff: RE: Windows NFS Client + cDOT Ok, started playing around with this today. This is my mount: Z:\>mount Local Remote Properties ------------------------------------------------------------------------------- Z: \\demo\flexvol<file://demo/flexvol> UID=0, GID=0 rsize=65536, wsize=65536 mount=hard, timeout=0.8 retry=1, locking=yes fileaccess=755, lang=ANSI casesensitive=no sec=sys When I try to rename via the GUI, I get this: [cid:image001.png@01D2B2A3.B085FC70] Via CLI, I get this: Z:\>ren "New Text Document (2).txt" test.txt Z:\>dir Volume in drive Z has no label. Volume Serial Number is 80F0-372F Directory of Z:\ 04/10/2017 03:49 PM <DIR> . 04/10/2017 03:49 PM <DIR> .. 04/10/2017 03:45 PM 0 New Text Document.txt 04/10/2017 03:46 PM 0 test.txt 04/10/2017 03:45 PM <DIR> .snapshot 2 File(s) 12,288 bytes 3 Dir(s) 1,044,535,574,528 bytes free A search for “invalid device” gets me this: https://support.microsoft.com/en-us/help/3025097/-invalid-device-error-when… I tried to apply it to my server, but it claims it’s not valid for Windows 2012R2, even though it’s specifically for Win 2012R2. ¯\_(ツ)_/¯ Packet traces and sktraces on the cluster suggest the issue isn’t on the cluster side; the rename request never happens from the client: [cid:image002.png@01D2B2A3.B085FC70] Does it fail for you the same way? Does rename work from CLI? On my end, at least, this seems to be a client issue. From: Alexander Griesser [mailto:AGriesser@anexia-it.com] Sent: Monday, April 10, 2017 4:26 AM To: NGC-tmacmd-gmail.com <tmacmd(a)gmail.com<mailto:tmacmd@gmail.com>> Cc: Parisi, Justin <Justin.Parisi(a)netapp.com<mailto:Justin.Parisi@netapp.com>>; toasters(a)teaparty.net<mailto:toasters@teaparty.net> Subject: AW: Windows NFS Client + cDOT Hey everyone, we’ve just set up a Windows 2k12 system and the renaming of files and folders doesn’t work there too, so it’s not a Windows 2016 problem. Any further ideas on how to debug this issue? Best, Alexander Griesser Head of Systems Operations ANEXIA Internetdienstleistungs GmbH E-Mail: AGriesser(a)anexia-it.com<mailto:AGriesser@anexia-it.com> Web: http://www.anexia-it.com<http://www.anexia-it.com/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 Von: Alexander Griesser Gesendet: Freitag, 7. April 2017 20:30 An: 'tmac' <tmacmd(a)gmail.com<mailto:tmacmd@gmail.com>> Cc: Parisi, Justin <Justin.Parisi(a)netapp.com<mailto:Justin.Parisi@netapp.com>>; toasters(a)teaparty.net<mailto:toasters@teaparty.net> Betreff: AW: Windows NFS Client + cDOT Yes, I’ve set anon in the export policy for the volume it self to 0 and I’ve set the registry keys on windows for AnonymousUID and AnonymousGID to 0 – without the registry keys, it will be „-2“ on the mount options in windows, whatever that negative value is used for… If the permissions were wrong, I could not create files at all in the first place, I guess, right? But I can create them and they show as UID 0 on the filer (also tested on a linux system where I mounted this volume) – and I can delete the files as well. The only thing which is not working, is renaming and I’m not sure why it’s refusing to do so. Maybe this is a Win 2k16 thinggie? I can try to spin up a Win2k12 system to see if this problem also persists there, that would at least rule out a misconfiguration on the filer I guess. I did not create separate policies for the SVM root, the SVM root only gets applied the default policy here and the default policy iss et to „ro all“, „rw never“ – as you can see below. Best, Alexander Griesser Head of Systems Operations ANEXIA Internetdienstleistungs GmbH E-Mail: AGriesser(a)anexia-it.com<mailto:AGriesser@anexia-it.com> Web: http://www.anexia-it.com<http://www.anexia-it.com/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 Von: tmac [mailto:tmacmd@gmail.com] Gesendet: Freitag, 7. April 2017 20:24 An: Alexander Griesser <AGriesser(a)anexia-it.com<mailto:AGriesser@anexia-it.com>> Cc: Parisi, Justin <Justin.Parisi(a)netapp.com<mailto:Justin.Parisi@netapp.com>>; toasters(a)teaparty.net<mailto:toasters@teaparty.net> Betreff: Re: Windows NFS Client + cDOT Not sure if this is it or not, but you have said that you set the anon ID's to 0. In this policy, it is set to 65535 Do you create separate policies for the SVM root and the data volumes? If you do, Root could/should be allow RO to all, rw to none. Then set the restrictions on the data volume policy. --tmac Tim McCarthy, Principal Consultant Proud Member of the #NetAppATeam<https://twitter.com/NetAppATeam> I Blog at TMACsRack<https://tmacsrack.wordpress.com/> On Fri, Apr 7, 2017 at 2:19 PM, Alexander Griesser <AGriesser(a)anexia-it.com<mailto:AGriesser@anexia-it.com>> wrote: Well, there are like 70 export policies on this SVM for 70 different volumes, I guess the policy for this volume as well as the default policy for the SVM will suffice here? If so, the export policy for this volume has already been sent earlier and here’s the default policy for this SVM: ::> export-policy rule show -vserver XXXXXXX -policyname default -instance Vserver: XXXXXXX Policy Name: default Rule Index: 1 Access Protocol: nfs List of Client Match Hostnames, IP Addresses, Netgroups, or Domains: 0/0 RO Access Rule: any RW Access Rule: never User ID To Which Anonymous Users Are Mapped: 65535 Superuser Security Types: none Honor SetUID Bits in SETATTR: true Allow Creation of Devices: true Best, Alexander Griesser Head of Systems Operations ANEXIA Internetdienstleistungs GmbH E-Mail: AGriesser(a)anexia-it.com<mailto:AGriesser@anexia-it.com> Web: http://www.anexia-it.com<http://www.anexia-it.com/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 Von: tmac [mailto:tmacmd@gmail.com<mailto:tmacmd@gmail.com>] Gesendet: Freitag, 7. April 2017 20:15 An: Alexander Griesser <AGriesser(a)anexia-it.com<mailto:AGriesser@anexia-it.com>> Cc: Parisi, Justin <Justin.Parisi(a)netapp.com<mailto:Justin.Parisi@netapp.com>>; toasters(a)teaparty.net<mailto:toasters@teaparty.net> Betreff: Re: Windows NFS Client + cDOT yes, yes.. export policy rule show -instance (please) --tmac Tim McCarthy, Principal Consultant Proud Member of the #NetAppATeam<https://twitter.com/NetAppATeam> I Blog at TMACsRack<https://tmacsrack.wordpress.com/>

1 0

upgrade help
by Scott Classen 10 Apr '17

10 Apr '17

Hello Toaster, I'm running into a problem downloading the 9.1 ONTAP image file to my filer. I've issued the following command netapp1::> cluster image package get -url http://my.webserver.gov/netapp/91_q_image.tgz Software get http://my.webserver.gov/netapp/91_q_image.tgz started on node netapp1-01 Downloading package. This may take up to 10 minutes. There is no update/install in progress Status of most recent operation: Run Status: Working Exit Status: Success Phase: Download Exit Message: Processing Package. Process package failed. Error: command failed: Error in processing the package, make sure the package version and URL are correct. netapp1::> I've submitted a help request ticket on mysupport.netapp.com <http://mysupport.netapp.com/> website, but not much useful help has been forthcoming yet. I've also tried using the OnCommand Web UI and receive a similar error. "Failed to add the software image to the cluster. Verify if the image and the specified URL are valid and ensure that there is sufficient space in the system management storage area." The url is valid. The tgz file has the correct md5sum. I'm sure this is something simple, but I'm not having any luck. Thanks, Scott ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Scott Classen, Ph.D. SIBYLS Beamline 12.3.1 sibyls.als.lbl.gov Advanced Light Source Lawrence Berkeley National Laboratory 1 Cyclotron Rd MS6R2100 Berkeley, CA 94720 mobile 510.206.4418 desk 510.495.2697 beamline 510.495.2134 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

2 3

Windows NFS Client + cDOT
by Alexander Griesser 10 Apr '17

10 Apr '17

Hey there, is it still true that Windows' integrated NFS client is unable to mount NFS shares from cDOT systems? Just asking because I was trying to do so again today out of curiosity and failed miserably again. If this is still the case, can anyone recommend alternative NFS clients for windows? Best, Alexander Griesser Head of Systems Operations ANEXIA Internetdienstleistungs GmbH E-Mail: AGriesser(a)anexia-it.com<mailto:AGriesser@anexia-it.com> Web: http://www.anexia-it.com<http://www.anexia-it.com/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

3 12

Switching to ACP inband connectivity
by Alexander Griesser 08 Apr '17

08 Apr '17

Hey Toasters, I've just upgraded one of my HA-pairs to 9.1P2 and wanted to switch to ACP inband connectivity following this KB: https://kb.netapp.com/support/s/article/ka31A00000012NsQAI/in-band-acp-setu… According to this KB, I should be eligible for the switch: ::*> storage shelf acp module show -fields module-name,protocol-version,firmware-version,shelf-serial-number,iom-type,state node module-name protocol-version firmware-version shelf-serial-number iom-type state ------- ----------- ---------------- ---------------- ------------------- -------- ------ nodeA 1.22.B 1.2.02.10 02.10 SHJMS1424000060 iom6 active nodeA 1.22.A 1.2.02.10 02.10 SHJMS1424000060 iom6 active nodeA 2.11.A 1.2.02.10 02.10 SHFMS1423000507 iom6 active nodeA 2.11.B 1.2.02.10 02.10 SHFMS1423000507 iom6 active nodeA 2.14.B 1.2.02.10 02.10 SHFMS1423000491 iom6 active nodeA 2.14.A 1.2.02.10 02.10 SHFMS1423000491 iom6 active nodeA 2.12.A 1.2.02.10 02.10 SHFMS1423000493 iom6 active nodeA 2.12.B 1.2.02.10 02.10 SHFMS1423000493 iom6 active nodeA 2.13.A 1.2.02.10 02.10 SHFMS1423000504 iom6 active nodeA 2.13.B 1.2.02.10 02.10 SHFMS1423000504 iom6 active nodeA 1.21.B 1.2.02.10 02.10 SHJMS1424000434 iom6 active nodeA 1.21.A 1.2.02.10 02.10 SHJMS1424000434 iom6 active nodeB 1.22.B 1.2.02.10 02.10 SHJMS1424000060 iom6 active nodeB 1.22.A 1.2.02.10 02.10 SHJMS1424000060 iom6 active nodeB 2.11.A 1.2.02.10 02.10 SHFMS1423000507 iom6 active nodeB 2.11.B 1.2.02.10 02.10 SHFMS1423000507 iom6 active nodeB 2.14.B 1.2.02.10 02.10 SHFMS1423000491 iom6 active nodeB 2.14.A 1.2.02.10 02.10 SHFMS1423000491 iom6 active nodeB 2.12.A 1.2.02.10 02.10 SHFMS1423000493 iom6 active nodeB 2.12.B 1.2.02.10 02.10 SHFMS1423000493 iom6 active nodeB 2.13.A 1.2.02.10 02.10 SHFMS1423000504 iom6 active nodeB 2.13.B 1.2.02.10 02.10 SHFMS1423000504 iom6 active nodeB 1.21.B 1.2.02.10 02.10 SHJMS1424000434 iom6 active nodeB 1.21.A 1.2.02.10 02.10 SHJMS1424000434 iom6 active 24 entries were displayed. The requirements state that the IOM6 modules must be upgraded to 2.09 (I have 2.10) and ACP firmware must be at least 1.2.2.9 (I have 1.2.2.10). So I went ahead and ran this command: storage shelf acp configure -is-enabled true -channel in-band After a few minutes, I saw this: ::*> storage shelf acp show Node Channel Connectivity ------------------ --------------- ---------------------- nodeA in-band active nodeB in-band active 2 entries were displayed. But when I ran the initial check command again, I saw this output: ::*> storage shelf acp module show -fields module-name,protocol-version,firmware-version,shelf-serial-number,iom-type,state node module-name protocol-version firmware-version shelf-serial-number iom-type state ------- ----------- ---------------- ---------------- ------------------- -------- ------------------------ nodeA 1.22.B 1.2.02.10 02.10 SHJMS1424000060 iom6 firmware-update-required nodeA 1.22.A 1.2.02.10 02.10 SHJMS1424000060 iom6 firmware-update-required nodeA 2.11.A 1.2.02.10 02.10 SHFMS1423000507 iom6 firmware-update-required nodeA 2.11.B 1.2.02.10 02.10 SHFMS1423000507 iom6 firmware-update-required nodeA 2.14.B 1.2.02.10 02.10 SHFMS1423000491 iom6 firmware-update-required nodeA 2.14.A 1.2.02.10 02.10 SHFMS1423000491 iom6 firmware-update-required nodeA 2.12.A 1.2.02.10 02.10 SHFMS1423000493 iom6 firmware-update-required nodeA 2.12.B 1.2.02.10 02.10 SHFMS1423000493 iom6 firmware-update-required nodeA 2.13.A 1.2.02.10 02.10 SHFMS1423000504 iom6 firmware-update-required nodeA 2.13.B 1.2.02.10 02.10 SHFMS1423000504 iom6 firmware-update-required nodeA 1.21.B 1.2.02.10 02.10 SHJMS1424000434 iom6 firmware-update-required nodeA 1.21.A 1.2.02.10 02.10 SHJMS1424000434 iom6 firmware-update-required nodeB 1.22.B 1.2.02.10 02.10 SHJMS1424000060 iom6 firmware-update-required nodeB 1.22.A 1.2.02.10 02.10 SHJMS1424000060 iom6 firmware-update-required nodeB 2.11.A 1.2.02.10 02.10 SHFMS1423000507 iom6 firmware-update-required nodeB 2.11.B 1.2.02.10 02.10 SHFMS1423000507 iom6 firmware-update-required nodeB 2.14.B 1.2.02.10 02.10 SHFMS1423000491 iom6 firmware-update-required nodeB 2.14.A 1.2.02.10 02.10 SHFMS1423000491 iom6 firmware-update-required nodeB 2.12.A 1.2.02.10 02.10 SHFMS1423000493 iom6 firmware-update-required nodeB 2.12.B 1.2.02.10 02.10 SHFMS1423000493 iom6 firmware-update-required nodeB 2.13.A 1.2.02.10 02.10 SHFMS1423000504 iom6 firmware-update-required nodeB 2.13.B 1.2.02.10 02.10 SHFMS1423000504 iom6 firmware-update-required nodeB 1.21.B 1.2.02.10 02.10 SHJMS1424000434 iom6 firmware-update-required nodeB 1.21.A 1.2.02.10 02.10 SHJMS1424000434 iom6 firmware-update-required 24 entries were displayed. Out of curiosity, I tried to run the ACP update again: nodeB> storage download acp Downloading ACP firmware can take several minutes and will not disrupt client access during that time. However, normal ACP recovery capabilities will not be available while the firmware upgrade is in progress. Are you sure you want to continue with ACP processor firmware update? y [storage download acp] Firmware will not be downloaded to module 2b.14.B as ACPP state is not active, Current ACPP state is 0x9. [storage download acp] Firmware will not be downloaded to module 0a.14.A as ACPP state is not active, Current ACPP state is 0x9. [storage download acp] Firmware will not be downloaded to module 0b.21.B as ACPP state is not active, Current ACPP state is 0x9. [storage download acp] Firmware will not be downloaded to module 0a.13.A as ACPP state is not active, Current ACPP state is 0x9. [storage download acp] Firmware will not be downloaded to module 0a.12.A as ACPP state is not active, Current ACPP state is 0x9. [storage download acp] Firmware will not be downloaded to module 2a.21.A as ACPP state is not active, Current ACPP state is 0x9. [storage download acp] Firmware will not be downloaded to module 2b.12.B as ACPP state is not active, Current ACPP state is 0x9. [storage download acp] Firmware will not be downloaded to module 2b.13.B as ACPP state is not active, Current ACPP state is 0x9. [storage download acp] Firmware will not be downloaded to module 0b.22.B as ACPP state is not active, Current ACPP state is 0x9. [storage download acp] Firmware will not be downloaded to module 2a.22.A as ACPP state is not active, Current ACPP state is 0x9. [storage download acp] Firmware will not be downloaded to module 0a.11.A as ACPP state is not active, Current ACPP state is 0x9. [storage download acp] Firmware will not be downloaded to module 2b.11.B as ACPP state is not active, Current ACPP state is 0x9. nodeB> exit Eventlog also said, that a FW Upgrade is required: ::*> event log show -severity infORMATIONAL Time Node Severity Event ------------------- ---------------- ------------- --------------------------- 4/8/2017 11:41:15 nodeB INFORMATIONAL sfu.firmwareUpToDate: Firmware is up-to-date on all disk shelves. 4/8/2017 11:39:37 nodeA INFORMATIONAL acp.need.firmware.update: ACP module (NA) needs a firmware upgrade, (disk shelf serial number: SHFMS1423000507). 4/8/2017 11:39:37 nodeA INFORMATIONAL acp.need.firmware.update: ACP module (NA) needs a firmware upgrade, (disk shelf serial number: SHFMS1423000504). 4/8/2017 11:39:37 nodeA INFORMATIONAL acp.need.firmware.update: ACP module (NA) needs a firmware upgrade, (disk shelf serial number: SHFMS1423000491). 4/8/2017 11:39:37 nodeA INFORMATIONAL acp.need.firmware.update: ACP module (NA) needs a firmware upgrade, (disk shelf serial number: SHFMS1423000493). 4/8/2017 11:39:37 nodeA INFORMATIONAL acp.need.firmware.update: ACP module (NA) needs a firmware upgrade, (disk shelf serial number: SHJMS1424000060). 4/8/2017 11:39:37 nodeA INFORMATIONAL acp.need.firmware.update: ACP module (NA) needs a firmware upgrade, (disk shelf serial number: SHJMS1424000434). 4/8/2017 11:39:37 nodeB INFORMATIONAL acp.need.firmware.update: ACP module (NA) needs a firmware upgrade, (disk shelf serial number: SHFMS1423000493). 4/8/2017 11:39:37 nodeB INFORMATIONAL acp.need.firmware.update: ACP module (NA) needs a firmware upgrade, (disk shelf serial number: SHFMS1423000504). 4/8/2017 11:39:37 nodeB INFORMATIONAL acp.need.firmware.update: ACP module (NA) needs a firmware upgrade, (disk shelf serial number: SHFMS1423000507). 4/8/2017 11:39:37 nodeB INFORMATIONAL acp.need.firmware.update: ACP module (NA) needs a firmware upgrade, (disk shelf serial number: SHFMS1423000491). 4/8/2017 11:39:37 nodeB INFORMATIONAL acp.need.firmware.update: ACP module (NA) needs a firmware upgrade, (disk shelf serial number: SHJMS1424000434). 4/8/2017 11:39:37 nodeB INFORMATIONAL acp.need.firmware.update: ACP module (NA) needs a firmware upgrade, (disk shelf serial number: SHJMS1424000060). I then switched back to OOB connectivity, and all is good again: ::*> storage shelf acp show Node Channel Connectivity ------------------ --------------- ---------------------- nodeA out-of-band full-connectivity nodeB out-of-band full-connectivity 2 entries were displayed. Tried to run the ACP upgrade again, getting this: nodeB> storage download acp Downloading ACP firmware can take several minutes and will not disrupt client access during that time. However, normal ACP recovery capabilities will not be available while the firmware upgrade is in progress. Are you sure you want to continue with ACP processor firmware update? y [storage download acp] ACP will check 12 modules for firmware download. [storage download acp] ACP module 192.168.0.124 is not eligible for update. [storage download acp] ACP module 192.168.0.126 is not eligible for update. [storage download acp] ACP module 192.168.0.180 is not eligible for update. [storage download acp] ACP module 192.168.0.202 is not eligible for update. [storage download acp] ACP module 192.168.0.208 is not eligible for update. [storage download acp] ACP module 192.168.0.248 is not eligible for update. [storage download acp] ACP module 192.168.1.112 is not eligible for update. [storage download acp] ACP module 192.168.1.170 is not eligible for update. [storage download acp] ACP module 192.168.1.238 is not eligible for update. [storage download acp] ACP module 192.168.2.66 is not eligible for update. [storage download acp] ACP module 192.168.3.130 is not eligible for update. [storage download acp] ACP module 192.168.3.138 is not eligible for update. nodeB> exit And now I'm back at the starting point. Any ideas what's going on here? Why does it tell me "firmware-update-required" when I'm already on the latest versions? Thanks, Alexander Griesser Head of Systems Operations ANEXIA Internetdienstleistungs GmbH E-Mail: AGriesser(a)anexia-it.com<mailto:AGriesser@anexia-it.com> Web: http://www.anexia-it.com<http://www.anexia-it.com/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

2 2

8.2 -> 8.3 cDOT upgrade
by John Stoffel 06 Apr '17

06 Apr '17

Guys, I'm in the planning and setup phase of a four node cDOT 8.2 to 8.3.x upgrade. One thing I need to do is setup all the admin LIFs for my VServers. I've got a bunch of them already, but some have data protocols enabled, and I don't think I can nuke them without alot of hassle. Is there a way under 8.2.3pX to see if a particular LIF is seeing traffic over a particular data protocol? For example, my 'foo' VServer have some LIFs, which have varying data protocols: vserver lif data-protocol -------- ----- ------------- foo admin none foo lif1 nfs foo lif2 nfs,cifs,fcache foo lif3 nfs,cifs,fcache foo lif4 nfs,cifs,fcache Can I see if lifs2-4 have CIFS traffic or not? I've been poking around, but it's not any thing obvious. In the grand scheme of things, I could just rename the probablematic LIFs to some other name and setup new 'admin' LIFs as required. And I do realize they're not required to be call 'admin', it's just a help. Thanks, John

2 3

7-Mode End of Support dates ?
by John Adams 06 Apr '17

06 Apr '17

Hello, I was recently told by NetApp that 8.2 is End of Support on 7/31/2017. Isn't 8.2.4 the last 7-mode version of On-Tap? Does this mean that 7-Mode is completely unsupported as of 7/31/2017? Is there currently a mad rush to convert from 7-Mode to clustered that I'm not aware of ? Thanks.

5 4

ONTAP 9 -- Full Disk Encryption (FDE)
by Eric Peng 05 Apr '17

05 Apr '17

Last year, we acquired an all-flash FAS array (AFF8040) for POC. While this array supports the FDE feature (via SafeNet), the POC requirements did not require us to enable this. As it sometimes goes, POC is potentially turning into "production" and we are currently taking a look at options to see if it makes sense to convert the POC into an actual purchase. Am hoping some of you may have a bit of experience with enabling FDE encryption on an array *after* data already exists on an array. Basically, need to know if the existing data can safely remain on the disks if FDE feature is to be introduced now? I've only come across a piece of NetApp marketing literature that indicates this should be non-disruptive, while the technical whitepapers focus on the nuts and bolts of "how to enable FDE". Can anyone share definitively whether we must first enable FDE on an array that has SED disks, or whether it is actually safe to introduce FDE on the array after it has been put into use? Would like to avoid having to migrate off the data and re-configure the array/cluster if I can. Thanks, Eric Peng | Enterprise Storage Engineer Esri | 380 New York St. | Redlands, CA 92373 | USA T 909 793 2853 x3567 | M 909 367 1691 epeng(a)esri.com<mailto:epeng@esri.com> | esri.com<http://esri.com/>

3 7

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters