toasters

toasters@lists.teaparty.net

13531 discussions

Filer controller down and auto failover issue
by Sayla, Mustafa 28 Jun '16

28 Jun '16

Last night we had an issue with our production filer where controller A went down but controller B did not know that A is down and did not took over. Only after I manually power off the controller A then controller B took over. I then brought the Controller A back up and did the give back and the filer worked fine for about 3 hours and same thing happened again. This time I shut down controller A and let it shut down and right now we are operating off of controller B only. Per netapp there is nothing in the logs that I sent them or in autosupport to suggest the root cause. They are asking us to bring controller A again and wait for it to happen again and if it does collect a core dump. This option is not acceptable to us as this affects production. Has anyone seen this before or have any ideas? Filer failover settings are all correct. We are running CDOT 8.3.1 on FAS8040. Mustafa Sayla Visit us on the Web at mesirowfinancial.com This communication may contain privileged and/or confidential information. It is intended solely for the use of the addressee. If you are not the intended recipient, you are strictly prohibited from disclosing, copying, distributing or using any of this information. If you received this communication in error, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy. Confidential, proprietary or time-sensitive communications should not be transmitted via the Internet, as there can be no assurance of actual or timely delivery, receipt and/or confidentiality. This is not an offer, or solicitation of any offer to buy or sell any security, investment or other product.

2 2

Interested in learning more about ONTAP 9?
by Parisi, Justin 28 Jun '16

28 Jun '16

In case you aren't aware, NetApp has an official podcast. This month, we've covered all things ONTAP 9 related. If you're interested, check it out! http://techontappodcast.com/

3 3

nfs access to volumes with ntfs security mode
by Peter D. Gray 27 Jun '16

27 Jun '16

Hi toasters We are migrating to cluster mode from 7 mode and having troubles with NFS access to volumes also doing CIFS. If I create a volume with NTFS security mode, and export aforesaid volume via NFS to a client, the client cannot read the volume. The mount suceeds, that is a filesystem handle is returned, but calls to access the directort fail, with message "file temporarily unavailable on the server, retrying..." The actual NFS error over the wire is is NFS3ERR_JUKEBOX. This is as root on the client. Non-root just gets permission denied. We have already set -ignore-nt-acl-for-root enabled on the vserver. The actuall settings on the vserver are: -access true \ -v3 enabled \ -udp enabled \ -tcp enabled \ -v4-id-domain its.uow.edu.au \ -v4.0 enabled \ -v4.1 enabled \ -v4.0-acl enabled \ -v4.1-acl enabled \ -v4.0-read-delegation enabled \ -v4.1-read-delegation enabled \ -v4.0-write-delegation enabled \ -v4.1-write-delegation enabled \ -v4.1-pnfs enabled \ -v4-numeric-ids enabled \ -mount-rootonly enabled \ -nfs-rootonly enabled \ -showmount enabled \ -vstorage disabled \ -ignore-nt-acl-for-root enabled So, what are we doing wrong or is this a bug? Regards, pdg Peter Gray Ph (direct): +61 2 4221 3770 Information Management & Technology Services Ph (switch): +61 2 4221 3555 University of Wollongong Fax: +61 2 4229 1958 Wollongong NSW 2522 Email: pdg(a)uow.edu.au Australia URL: http://pdg.uow.edu.au

2 1

Unable to terminate re-startable dumps
by John Adams 22 Jun '16

22 Jun '16

Hi All, We backup from a 3rd party product via NDMP. The past couple of days, SOME the backups have been failing on this filer with: There are no backup contexts available in the system. Delete some of the unwanted restartable backup contexts and retry the operation. DUMP: Maximum number of allowed dumps/restores (16) in progress. NDMP_OP_ERRORCODE is 'EMAXBACKUPS' DUMP: Dump failed to initialize. And looking at the filer: hostname*> backup status ID State Type Device Start Date Level Path -- ----------- ---- ------ ------------ ----- --------------- 2 RESTARTABLE dump 0 3 RESTARTABLE dump 0 4 RESTARTABLE dump 0 5 RESTARTABLE dump 0 6 RESTARTABLE dump 0 7 RESTARTABLE dump 0 8 RESTARTABLE dump 0 9 RESTARTABLE dump 0 10 RESTARTABLE dump 0 11 RESTARTABLE dump 0 12 RESTARTABLE dump 0 13 RESTARTABLE dump 0 14 RESTARTABLE dump 0 15 RESTARTABLE dump 0 16 RESTARTABLE dump 0 So I thought I had an accumulation of old backup sessions over time. hostname*> backup terminate 2 Unable to read from registry. There are no snapshots on the system that were created by backups right now. hostname*> version NetApp Release 8.2.3P3 7-Mode: Tue Apr 28 14:48:22 PDT 2015 Any ideas of what I could try here? Its strange that it shows dump as the device instead of NDMP. And its strange that the Start Date and Path's are blank. Thanks.

1 0

Raid disk layout - the ability to lose a shelf.
by jordan slingerland 22 Jun '16

22 Jun '16

Hello, I am deploying a new 8040 and it was requested that the aggregates / raid groups are laid out in such a way that no more than 2 disks in any raid group are within the same shelf. At first this sounds like it reduces single points of failure and could protect availability from the failure of a full disk shelf. I argue against this strategy and was wondering if anyone in this list had any feedback. My thought is that this configuration is marginally increasing availability at the sacrifice of additional risk to data integrity. With this strategy, each time a disk failed we would endure not only the initial rebuilt to spare, but a second rebuild when a disk replace is executed to put the original shelf/slot/disk back into the the active raid group. Additional, if a shelf failure were encountered, I question whether it would even be possible to limp along. In an example configuration, we would be down 24 disks, 4 or 5 would rebuild to the remaining spares available. Those rebuilds along should require significant cpu to occur concurrently and I expect would impact data services significantly. Additionally, at least 10 other raid groups would be either single or double degraded. I expect the performance degradation at this point would be so great that the most practical course of action would be to shutdown the system until the failed shelf could be replaced. Thanks for any input. I would like to know if anyone has any experience thinking through this type of scenario. Is considering this configuration interesting or perhaps silly? Are any best practice recommendations being violated? Thanks in advance. --Jordan

13 14

Netappdjenga
by Bert Kiers 21 Jun '16

21 Jun '16

Hi, A collegue fell off his chair when I proposed this game so I share the idea. Take a working Netapp metro cluster (preferably non-production). Players take turns to disconnect one connector until the cluster stops functioning. The player that disconnected the last one, loses. Grtnx, -- Bert

2 1

adding new SAS loop
by Peter D. Gray 17 Jun '16

17 Jun '16

Hi people, anybody know if adding hot adding new SAS loop onto an 8000 series with ontap 8.3 cluster mode is supported? If yes, where is this fact documented? (as opposed to added a new shelf into an existing loop which I know is supported) Regards, pdg

3 2

7-mode snap list with year
by Basil 16 Jun '16

16 Jun '16

Hi, I'm trying to write a module into a home-grown management tool that will alert us when snapshots are kept longer than their retention target. It works fine for CDOT, but I'm having trouble with the time stamps in 7-mode. They omit the year. I need to use cli to get this, I can't use powershell or the api. Does anyone have an idea?

2 2

High CPU
by Sayla, Mustafa 10 Jun '16

10 Jun '16

Is there a way to find out why in a CDOT cluster controller A is always between 50 and 55% CPU usage and controller B is less than 25%. Storage wise data is evenly balance but seem like some volumes are used more heavily than others and if I can identify them I would balance them to that CPU usage is even between the controllers. Thank you Mustafa Sayla Visit us on the Web at mesirowfinancial.com This communication may contain privileged and/or confidential information. It is intended solely for the use of the addressee. If you are not the intended recipient, you are strictly prohibited from disclosing, copying, distributing or using any of this information. If you received this communication in error, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy. Confidential, proprietary or time-sensitive communications should not be transmitted via the Internet, as there can be no assurance of actual or timely delivery, receipt and/or confidentiality. This is not an offer, or solicitation of any offer to buy or sell any security, investment or other product.

4 3

Risk of assigning disks in a loop to two controllers
by BERNTSEN Basil 10 Jun '16

10 Jun '16

Hi folks, I have a CDOT 8.2 system (in prod) where new shelves were installed on a SAS stack on the wrong node. I want to create an aggregate on node1, but the disks are on node2. Based on the guidelines<https://library.netapp.com/ecmdocs/ECMP1196912/html/GUID-99C1AA7C-79D9-4E4F…>, I'm thinking of delaying the aggregate creation and changing the shelves. My question is whether it's worth the delay. What are the risks of just assigning the disks to the partner of the node they're currently assigned to? The documentation says to "Always assign all disks on the same loop or stack to the same system." Why? Thanks! Basil ************************************************************************* This message and any attachments (the "message") are confidential, intended solely for the addressee(s), and may contain legally privileged information. Any unauthorized use or dissemination is prohibited. E-mails are susceptible to alteration. Neither SOCIETE GENERALE nor any of its subsidiaries or affiliates shall be liable for the message if altered, changed or falsified. Please visit http://sgasdisclosure.com for important information regarding SG Americas Securities, LLC (“SGAS”). Please visit http://swapdisclosure.sgcib.com for important information regarding swap transactions with SOCIETE GENERALE. *************************************************************************

6 5

Closing cifs sessions by id in 8.2
by Basil 31 May '16

31 May '16

Hi, I'm running 8.2.3 and need to close a specific cifs session by ID for troubleshooting and taking a trace. The command cifs session close appears to have been introduced in 8.3... Is there another way, maybe from node or system shell, to kill a specific session?

1 1

Support Bulletin 7010246
by Philbert Rupkins 27 May '16

27 May '16

Toasters, Does anybody have more information on support bulletin 7010246? I just received an email urging us to upgrade to ONTAP 8.1.4 P9 as earlier versions of 8.1.4 are "more susceptible to potential system disruptions and/or data loss due to drive failures exceeding the two drive failure protection offered by RAID-DP." We've been planning to upgrade to P9 so this isn't that big of a deal. However, I'm just curious if this is a new development or has this always been a problem with pre-P9 releases of 8.1.4? The support bulletin is light on background information. If this hasn't always been an issue, the only logical conclusion I can come to is NetApp is shipping new replacement drives that don't play well with pre-P9 releases of 8.1.4. Thank you all very much. Phil

1 0

(no subject)
by Yuvaraja Shivaram 20 May '16

20 May '16

hi, Anyone has any idea of resolving the below things for snapvault issue.. would like to fix this issue, stopping and restarting might work, but is that going start once again from beginning, is there any way to start with update command rather than starting from beginning.. Transfer aborted: destination qtree is not coalesced regards raj

1 0

Showmount -e?
by Basil 19 May '16

19 May '16

Does anyone know why showmount -e doesn't work in CDOT? Will it ever work?

3 3

incorrect language for volume
by Darry Cooley 16 May '16

16 May '16

We are in the process of migrating data from our 7mode filers (8.1.2P4 7-Mode) to our new CDOT filers (8.3.1P1), and we noticed that a few of our 7mode volumes (NFS volumes) do not have the correct language set. 7mode documentation says I can change the language of a volume, however: Note: If you are changing the NFS character set, you are asked to confirm > your choice, and also whether you want to halt the system so that > WAFL_check can be run to check for any files that will no longer be > accessible using NFS. The default answer for this question is yes. If you > do not want to halt the system, you must enter n. > I'd really prefer to not halt the filer. What are the ramifications of not halting the filer? Can I safely change the language and snapmirror over to the new filers without issue? Thanks. ...Darry -- Darry Cooley Factory Systems Engineer Broadcom Limited - WSD FS/FAST 4380 Ziegler Road, Fort Collins, CO 80525 darry.cooley(a)broadcom.com

3 3

Listing NTFS style ACLs from unix client via NFS (Take 2...)
by John Adams 12 May '16

12 May '16

Hi all.. I asked this a few weeks ago: If a qtree is using NTFS style permissions, and that same qtree is exported via NFS to a unix client...Is there a way to see the NTFS acl's from that unix client? The usual "ls -l" just shows what looks like mode 777. I got some good responses, but I'm not seeing what I want to see. Qtree on filer is security style NTFS. Qtree is exported to linux box via export file on filer: /vol/secgroup_group -sec=sys,rw=mfanfs,root=x.x.x.x,nosuid Qtree is mounted on linux box: > mount -o vers=4,acl secgroup:/vol/secgroup_group /secgroup > grep secgroup /proc/mounts secgroup:/vol/secgroup_group/ /secgroup nfs4 rw,relatime,vers=4,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=x.x.x.x,minorversion=0,local_lock=none,addr=x.x.x.x 0 0 NFS option on filer: > options nfs.ntacl nfs.ntacl_display_permissive_perms on >From a windows box, using the CIFS share to that qtree, I can right click on a file, select security, and then see/set the usual NTFS style ACLs. When use getfacl from the unix box, I still only see the unix style (User,Group,Other) permissions. What I'm hoping to see is the Windows style ACLs that are on the files. I can see them from the filer: secgroup@testfs1> fsecurity show /vol/secgroup_group/ccc/pit [/vol/secgroup_group/ccc/pit - File (inum 28476)] Security style: NTFS Effective style: NTFS DOS attributes: 0x0020 (---A----) Unix security: uid: xxxx(username) gid: 101 (groupname) mode: 0777 (rwxrwxrwx) NTFS security descriptor: Owner: DOMAIN\username Group: DOMAIN\Domain Users DACL: Allow - DOMAIN\budget - 0x001f01ff (Full Control) Allow - Everyone - 0x001200a9 (Read and Execute) - (Inherited) Allow - DOMAIN\username - 0x001f01ff (Full Control) - (Inherited) Allow - DOMAIN\group - 0x001f01ff (Full Control) - (Inherited) I am hoping to be able to see the above output, from the linux client. I'm looking for a way for users on linux clients to see the windows ACLs that are on this NTFS qtree. Any suggestions? Thanks.

3 2

automatic SP firmware upgrade after system node image get
by Iluhes 11 May '16

11 May '16

Folks,I have situation that SP firmware was upgraded after just "Get" of softwaresystem node image get -node * -package http:// and nothing else. It caused the SP to upgrade and watchdog reset some nodes as well.Is there a reason for that?Anyone ever come across this issue.

2 8

nfsstat in cmode 8.3
by Douglas Siggins 05 May '16

05 May '16

Greetings, Dipping my toes into cluster mode. It seems the ability to monitor specific nfs clients in 8.3 is no longer there? ::> statistics settings modify -display-rates true -client-stats enabled Warning: System performance may be significantly impacted. Are you sure? Do you want to continue? {y|n}: y ::> statistics show -object client I dont believe I am seeing the hostname field, has anyone used this prior? I know about nperf and pcpconfig in the sytemshell, but I'd prefer not to use "unsupported" tools.

3 5

FAS 3170 aggr issue
by Mark Flint 05 May '16

05 May '16

Hi all. I have 2 x FAS3170’s, both now running 8.0.5P1. On one unit, I can build an aggregate of over 70TB, yet on the other, it won’t allow this to happen. Both aggr’s were built using the CLI :- aggr create aggrgate1 –B 64 –r 16 –t raid_dp 112 ( 96 ) the 96 is the aggr built on the unit that is restricting aggr size. Anybody seen this before, or have any ideas? Thanks, `Mark

2 2

RE: 8.3.2 Upgrade Question
by Aaron Lewis 05 May '16

05 May '16

Thanks. So, I ran the command and got the following error: Error: command failed: Upgrade attempt failed. When I run the “system node upgrade-revert show” command, the node that had problems doesn’t even display. At this point, I’m thinking I may need to go through the process of installing the wonky node again. Any suggestions? From: Aaron Dunlap [mailto:Aaron.Dunlap@pivotts.com] Sent: Wednesday, May 4, 2016 1:58 PM To: Parisi, Justin <Justin.Parisi(a)netapp.com>; Aaron Lewis <aaron(a)ivoxy.com>; <toasters(a)teaparty.net> Lists <toasters(a)teaparty.net> Subject: RE: 8.3.2 Upgrade Question Me too… From: toasters-bounces(a)teaparty.net<mailto:toasters-bounces@teaparty.net> [mailto:toasters-bounces@teaparty.net] On Behalf Of Parisi, Justin Sent: Wednesday, May 4, 2016 3:51 PM To: NGC-aaron-ivoxy.com <aaron(a)ivoxy.com<mailto:aaron@ivoxy.com>>; <toasters(a)teaparty.net<mailto:toasters@teaparty.net>> Lists <toasters(a)teaparty.net<mailto:toasters@teaparty.net>> Subject: RE: 8.3.2 Upgrade Question Non disruptive every time I’ve run it… From: toasters-bounces(a)teaparty.net<mailto:toasters-bounces@teaparty.net> [mailto:toasters-bounces@teaparty.net] On Behalf Of Aaron Lewis Sent: Wednesday, May 04, 2016 4:48 PM To: <toasters(a)teaparty.net<mailto:toasters@teaparty.net>> Lists Subject: 8.3.2 Upgrade Question Hi all, Last night, I assisted a customer in upgrading a 4-node cluster from 8.2.1 to 8.3.2. We had an issue where the last node didn’t reboot correctly, due to the default image not changing, and had to boot into the backup image. From there, I was able to set the 8.3.2 image as default. Now everything is running 8.3.2 but “version” displays 8.2.1. A KB (#2021570) references running “system node upgrade-revert -node <nodename>” to fix this. Can anyone confirm that the command is non-disruptive? I feel like it should be, as the node is already running 8.3.2 code, but I wanted to make sure. Thanks! Regards, Aaron The information contained in this message and any attachments (collectively, the "Transmission") contains proprietary and/or confidential information of the sender’s employer which is protected from disclosure and intended solely for the named recipient(s). If you are not a named recipient, you are prohibited from copying, distributing or using this Transmission. Please contact the sender immediately by returning the e-mail and deleting the original Transmission.

2 1

disk firmware for IBM N series
by Mark Saunders 04 May '16

04 May '16

Hi all We have a mix of Netapp FAS and IBM N series filers. I used to be able to download new firmware for the N series through the IBM support site but this is now saying I don't have a valid support contract. Do IBM do anything to the firmware updates or could we get them from Netapp and apply them to the N series. I have looked at a couple and they appear to be identical but I know with the base OT there is something in the firmware to stop a Netapp version being installed to an N series. Regards Mark Data Centre Sysadmin Team Managed Services Phone:- 02476 694455 Ext 2567 The Sysadmin Team promoting PCMS Values ~Integrity~Respect~Commitment~ ~Continuous Improvement~ The information contained in this e-mail is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient of this e-mail, the use of this information or any disclosure, copying or distribution is prohibited and may be unlawful. If you received this in error, please contact the sender and delete the material from any computer. The views expressed in this e-mail may not necessarily be the views of the PCMS Group plc and should not be taken as authority to carry out any instruction contained. The PCMS Group reserves the right to monitor and examine the content of all e-mails. The PCMS Group plc is a company registered in England and Wales with company number 1459419 whose registered office is at PCMS House, Torwood Close, Westwood Business Park, Coventry CV4 8HX, United Kingdom. VAT No: GB 705338743

1 0

8.3.2 Upgrade Question
by Aaron Lewis 04 May '16

04 May '16

Hi all, Last night, I assisted a customer in upgrading a 4-node cluster from 8.2.1 to 8.3.2. We had an issue where the last node didn't reboot correctly, due to the default image not changing, and had to boot into the backup image. From there, I was able to set the 8.3.2 image as default. Now everything is running 8.3.2 but "version" displays 8.2.1. A KB (#2021570) references running "system node upgrade-revert -node <nodename>" to fix this. Can anyone confirm that the command is non-disruptive? I feel like it should be, as the node is already running 8.3.2 code, but I wanted to make sure. Thanks! Regards, Aaron

2 1

NetApp CN1610 Cluster Switches??
by Chris hansen 04 May '16

04 May '16

I may have access to some of these switches, are they just plain 10gb fiber NETWORK switches? The person that is getting rid of them says they have a box of 4gb SPFs to go with them and it doesn’t appear this is a converged switch but just a 10gb switch geared toward Netapp. I’ve read some documentation on configuration but doesn’t give a clear use case. Thanks -Chris

3 2

accessing service processor
by Rhodes, Richard L. 27 Apr '16

27 Apr '16

Hello, This is probably really dumb, but I'm stumped. I've been working on a support case with IBM (nSeries). They asked me to get some info from the service processor. So I ssh to the SP, get a prompt, try and login as naroot (using root's pwd), which fails. The last time I tried to ssh and log onto a SP was when these systems were setup +2 years ago. Back then it worked. Since then we've regularly rolled the root password every X months. It's my understanding that naroot at the service processor uses the OnTap root password. I've tried to access the SP on several nSeries system and they all fail the same way. IBM nSeries N3220 (NetApp FAS2240) OnTap 8.1.2P1 7-mode Any thoughts are appreciated. Rick What I've done: - I know the root password - I used it to log onto the filer on the main interface Using username "root". root@filerxxxx's password: filerxxxx> - It's status is online filerxxxx> sp status Service Processor Status: Online Firmware Version: 2.1.1 Mgmt MAC Address: xx:xx:xx:xx:xx:xx Ethernet Link: up, 100Mb, full duplex, auto-neg complete Using DHCP: no IPv4 configuration: IP Address: xxx.xx.xx.xxx Netmask: 255.255.255.0 Gateway: xxx.xx.xx.x IPv6 configuration: Disabled - all hosts are allowed to access the sp filerxxxx> options sp sp.autologout.enable on sp.autologout.timeout 60 sp.setup on sp.ssh.access * - it pings - it gives a login prompt, but it doesn't take the root password ssh naroot@filerxxxxs naroot@filerxxxxs's password: Permission denied, please try again. naroot@filerxxxxs's password: - I've tried ssh to the SP via the DNS name (filerxxxxs) and the IP (xxx.xx.xx.xxx) - I am aware of the 15min timeout period on 6 unsuccessful login attempts. - I've tried ssh via putty from a Win laptop, and, ssh from an AIX system. Both give the same result - it naroot/root password doesn't work. - I rebooted the SP filerxxxx> sp reboot NOTE : If your console connection is through the SP,it will be disconnected Do you want to reboot the SP now? (y/n) :? y Console connection through the SP will now be dropped - Reboot didn't help - still can't login as naroot with root's password ssh naroot@filerxxxxs naroot@filerxxxxs's password: Permission denied, please try again. ----------------------------------------- The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately, and delete the original message.

5 6

quota control query in CDOT
by Brian Parent 23 Apr '16

23 Apr '16

I'm writing a perl script to call the NetApp APIs for modifying quotas. I'd like to allow a vserver admin the ability to change the association of quota policy for their own vserver, but that doesn't appear to be allowed. As far as I can tell, only cluster admins can do that. This is a request to see if I'm misunderstanding things, as I'm fairly new to CDOT. Vserver admins can create, delete, modify, copy, and rename quota policies for their vserver, but not change the active policy. So, if they were to make a backup copy of their active policy, then make changes to the active policy, but make a mistake and have the need to revert to their backup copy, they would *not* be able to without the intervention of a cluster admin. CDOT (8.3.1) prevents deletion of the active quota policy, and prevents copying to a policy name that already exists (even as a cluster admin, even if quotas are turned off). Excerpted from "Clustered Data ONTAP 8.3 Logical Storage": • When you need to create a quota rule or change quota rules for a quota policy, you can choose either of the following approaches: ◦ If you are working in a quota policy that is assigned to an SVM, then you need not assign the quota policy to the SVM. ◦ If you are working in an unassigned quota policy and then assigning the quota policy to the SVM, then you must have a backup of the quota policy that you can revert to if required. For example, you can make a copy of the assigned quota policy, change the copy, assign the copy to the SVM, and rename the original quota policy. I guess what is unsaid in the doc is that such an approach is available only to cluster, and not vserver admins. Have I got this right? I thought the idea was to allow vserver admins the privileges to work on their own vserver without requiring cluster admin privileges. -- Brian Parent Information Technology Services Department IT Infrastructure Operations Group Workplace, Internal, Research, and Educational Platforms (WIRE) team UC San Diego (858) 534-6090

1 0

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters