toasters

toasters@lists.teaparty.net

2 participants
13529 discussions

VERITAS ORACLE NETAPPS AND SOLARIS
by Noll, Kevin 10 Dec '02

10 Dec '02

Does anyone use the following combination of services Oracle Group on Veritas running on Solaris mounted to Network Appliance? Have you ever had the problem were by the following happens? TAG_D 2002/12/10 06:12:35 (snape) VCS:13067:Agent is calling clean for resource(ora_dat1_weborig) because the resource became OFFLINE unexpectedly, on its own. This problem happens at least once a week. We are running multiple versions of Solaris, Oracle and Data onTap. Any help would be much appreciated Kevin Noll tel +44 7900 244 255 ******************************************************************************************** " This message contains information that may be privileged or confidential and is the property of the Cap Gemini Ernst & Young Group. It is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain, copy, disseminate, distribute, or use this message or any part thereof. If you receive this message in error, please notify the sender immediately and delete all copies of this message ". ********************************************************************************************

1 0

Re: Dump issue
by Ambrose_Earle＠shamrockfoods.com 10 Dec '02

10 Dec '02

We are seeing very similar performance issues from our F840Cs. We have opened a case with NetApps, but have received very little information back from them. In our case, a 3 drive volume backing up gzip'ed files maxes out our SDLT drives at nearly 11 MB/sec. We have a 6 drive volume of highly compressible information that will stream to tape at over 20 MB/sec. However, a 7 drive volume that only contains a few dozen very large Oracle datafiles, now at times struggles to write 4 MB/sec! The only correlation that I can make is that when disk utilization goes above 40%, the backup speed plummets below 10 MB/sec. Have you tried running sysstat or statit during the backup? My biggest concern is that this may be the first symptom of some major problem. If 7 drives can't read faster than 4MB/sec, then wouldn't you be a little worried?

1 0

RE: Suse & NetApp break oracle: flock problem
by Alex Harkema 10 Dec '02

10 Dec '02

Michael, attached you'll find the polyserv patch. Besides that you'll find a little program 'flock.c' which is able to execute old BSD-style flock or, at request, the POSIX fcntl lock. First method is not usable for NFS running the 'new' kernel (2.4.18-224), this in contrast to the 2nd method. Oracle uses first method (BSD-style) -alex -----Original Message----- From: Kiernan, Michael [mailto:mkiernan@netapp.com] Sent: Tuesday, December 10, 2002 9:13 AM To: 'Alex Harkema' Subject: RE: Suse & NetApp break oracle: flock problem Hi Alex, can you post details on the polyserve-flock patch? (a couple of minutes on Google failed to find it...) Thanks, Mike -----Original Message----- From: Alex Harkema [mailto:HarkemaA@vertis.nl] Sent: Monday, December 09, 2002 8:04 PM To: Kiernan, Michael Subject: RE: Suse & NetApp break oracle: flock problem Indeed it was what I was looking for. Thanks! Meanwhile, my cry for help activated some suse-people; I'm now awaiting their responses on the locking problem. Alex -----Original Message----- From: Kiernan, Michael [mailto:mkiernan@netapp.com] Sent: Monday, December 09, 2002 5:44 PM To: 'Alex Harkema' Subject: RE: Suse & NetApp break oracle: flock problem Hi Alex, look under the partners section on the website: http://www.netapp.com/partners/oracle/9irac.html <http://www.netapp.com/partners/oracle/9irac.html> whitepaper: http://www.netapp.com/tech_library/3164.html <http://www.netapp.com/tech_library/3164.html> hope that's what you're looking for... Mike -----Original Message----- From: Alex Harkema [mailto:HarkemaA@vertis.nl] Sent: Monday, December 09, 2002 5:33 PM To: Kiernan, Michael Cc: 'toasters(a)mathworks.com' Subject: RE: Suse & NetApp break oracle: flock problem Mike, thank you for your reply. I really suspect this is a linux-(kernel)problem. However I was curious if anyone had encountered the same issue. Besides that, we had not rebooted the machine when the error occurred. At the moment installations are running on the linux box (now still running the 2.4.18.-134 kernel, this one *not* containing the polyserver-flock patch) and I don't have the opportunity to reboot that one. However, I'll gladly send you the dumps afterward if we're still experiencing same problems. Can you tell me were I can find certifications for the oracle-on-linux/netapp products? Like, is 9iRac certified on Netapp? tia, Alex -----Original Message----- From: Kiernan, Michael [mailto:mkiernan@netapp.com] Sent: Monday, December 09, 2002 2:09 PM To: 'Alex Harkema' Cc: Rolf Fokkens Subject: RE: Suse & NetApp break oracle: flock problem Hi Alex, What did 'priv set advanced; lock_dump -h' show on the filer at the time you get the error ? Was Oracle shutdown cleanly prior to reboot ? One problem we're aware of with Linux rpc.statd is that it sometimes can fail to remove the locks on the filer on reboot, due to it locking with an unqualified nodename, and sending lock recovery packets with a fully-qualified domain name. If Oracle shutdown cleanly, however, the locks should have been released by that action. A pktt trace from the filer to the linux box (pktt start all -i <ip of oracle linux box>) while you reboot the upgraded box and restart oracle would be useful, as would the lock_dump output. I'll be happy to look at the data. Mike -----Original Message----- From: Alex Harkema [mailto:HarkemaA@vertis.nl] Sent: Monday, December 09, 2002 1:24 PM To: 'toasters(a)mathworks.com' Cc: Rolf Fokkens Subject: Suse & NetApp break oracle: flock problem Hi all! After upgrading the Linux kernel on our SuSE Linux Enterprise Server (SLES) to k_smp-2.4.18-224 we were unable to start our Oracle Database which was located on a NetApp Filer. All the feedback we got was "no locks available" Further investigation showed that k_smp-2.4.18-224 introduced a polyserve-flock patch which probably causes the problems. Has anyone ran into this? We'll do tests with k_smp-2.4.18-224 and/or 2.4.18-237 whithout the polyserve-flock patch and share this with those who express their interest. I address this report to both SuSE and NetApp for it's this combination that breaks Oracle. Because it's both in SuSE's and NetApp's interest I hope this report is appreciated though it may not have been sent through the proper channels. Alex Harkema Vertis

1 0

RE: Dump issue
by Kumar, Rahul 10 Dec '02

10 Dec '02

Hi As you can see its only 40gb data off from 119 gb vol0 Rahul -----Original Message----- From: Kumar, Sreejith [mailto:Sreejith_Kumar@mentorg.com] Sent: Tuesday, December 10, 2002 3:51 PM To: Kumar, Rahul Subject: Re: Dump issue Is your netapp over 70-80% capacity ? We had seen similar problems with dump when the filer reaches 70-80% capacity. Sreejith ----- Original Message ----- From: "Kumar, Rahul" <rahul.kumar(a)eds.com> To: "Chris Thompson" <cet1(a)cus.cam.ac.uk> Cc: <toasters(a)mathworks.com> Sent: Tuesday, December 10, 2002 3:08 PM Subject: RE: Dump issue > Hi All, > > This is the speed which I regularly used to get that too oon the same > tape.. We alternate the tapes.... Any help or issues... > Also the tape is attached to a remote host which is a dedicated solaris > box > > Regards > Rahul > > DUMP: creating "/vol/vol0/../snapshot_for_backup.11" snapshot. > DUMP: Using Full Volume Dump > DUMP: Date of this level 0 dump: Mon Dec 2 23:02:13 2002. > DUMP: Date of last level 0 dump: the epoch. > DUMP: Dumping /vol/vol0/ to root > DUMP: mapping (Pass I)[regular files] > DUMP: mapping (Pass II)[directories] > DUMP: estimated 42147106 KB. > DUMP: dumping (Pass III) [directories] > DUMP: dumping (Pass IV) [regular files] > DUMP: Mon Dec 2 23:08:12 2002 : We have written 971826 KB. > DUMP: Mon Dec 2 23:13:12 2002 : We have written 2639150 KB. > DUMP: Mon Dec 2 23:18:12 2002 : We have written 4115292 KB. > DUMP: Mon Dec 2 23:23:14 2002 : We have written 5679897 KB. > DUMP: Mon Dec 2 23:28:14 2002 : We have written 7306871 KB. > DUMP: Mon Dec 2 23:33:14 2002 : We have written 8830654 KB. > DUMP: Mon Dec 2 23:38:14 2002 : We have written 10476997 KB. > DUMP: Mon Dec 2 23:43:14 2002 : We have written 11551556 KB. > DUMP: Mon Dec 2 23:48:14 2002 : We have written 13048942 KB. > DUMP: Mon Dec 2 23:53:14 2002 : We have written 14688391 KB. > DUMP: Mon Dec 2 23:58:14 2002 : We have written 16221813 KB. > DUMP: Tue Dec 3 00:03:14 2002 : We have written 17364633 KB. > DUMP: Tue Dec 3 00:08:14 2002 : We have written 18843934 KB. > DUMP: Tue Dec 3 00:13:14 2002 : We have written 20516017 KB. > DUMP: Tue Dec 3 00:18:14 2002 : We have written 21848469 KB. > DUMP: Tue Dec 3 00:23:14 2002 : We have written 23232642 KB. > DUMP: Tue Dec 3 00:28:14 2002 : We have written 24980200 KB. > DUMP: Tue Dec 3 00:33:14 2002 : We have written 26645288 KB. > DUMP: Tue Dec 3 00:38:14 2002 : We have written 28013082 KB. > DUMP: Tue Dec 3 00:43:14 2002 : We have written 29653350 KB. > DUMP: Tue Dec 3 00:48:14 2002 : We have written 31112304 KB. > DUMP: Tue Dec 3 00:53:14 2002 : We have written 32806123 KB. > DUMP: Tue Dec 3 00:58:14 2002 : We have written 34071851 KB. > DUMP: Tue Dec 3 01:03:14 2002 : We have written 35518838 KB. > DUMP: Tue Dec 3 01:08:14 2002 : We have written 36825522 KB. > DUMP: Tue Dec 3 01:13:14 2002 : We have written 38524506 KB. > DUMP: Tue Dec 3 01:18:14 2002 : We have written 39836545 KB. > DUMP: Tue Dec 3 01:23:14 2002 : We have written 41444950 KB. > DUMP: dumping (Pass V) [ACLs] > DUMP: 42121234 KB > DUMP: DUMP IS DONE > > -----Original Message----- > From: Chris Thompson [mailto:cet1@cus.cam.ac.uk] > Sent: Tuesday, December 10, 2002 2:22 PM > To: Kumar, Rahul > Cc: toasters(a)mathworks.com > Subject: Re: Dump issue > > > rahul.kumar(a)eds.com (Rahul Kumar) writes: > > > > Any Ideas why the dump is running so slow. We are running the filer > > to > > > dump onto a SDLT tape drive > ^^^^^^^^^^^^^^^^^ > > Oh, yeah? In which case why does it say > > > DUMP: Dumping tape file 1 on /tmp/filer1tape > ^^^^^^^^^^^^^^^ > > ? > > If this is dumping to a tape on a remote host, I suppose that _could_ > be a symlink to a tape drive! But in that case, it may be the network > that is the source of your speed problem. > > Chris Thompson University of Cambridge Computing Service, > Email: cet1(a)ucs.cam.ac.uk New Museums Site, Cambridge CB2 3QH, > Phone: +44 1223 334715 United Kingdom.

1 0

RE: Dump issue
by Kumar, Rahul 10 Dec '02

10 Dec '02

Hi All, This is the speed which I regularly used to get that too oon the same tape.. We alternate the tapes.... Any help or issues... Also the tape is attached to a remote host which is a dedicated solaris box Regards Rahul DUMP: creating "/vol/vol0/../snapshot_for_backup.11" snapshot. DUMP: Using Full Volume Dump DUMP: Date of this level 0 dump: Mon Dec 2 23:02:13 2002. DUMP: Date of last level 0 dump: the epoch. DUMP: Dumping /vol/vol0/ to root DUMP: mapping (Pass I)[regular files] DUMP: mapping (Pass II)[directories] DUMP: estimated 42147106 KB. DUMP: dumping (Pass III) [directories] DUMP: dumping (Pass IV) [regular files] DUMP: Mon Dec 2 23:08:12 2002 : We have written 971826 KB. DUMP: Mon Dec 2 23:13:12 2002 : We have written 2639150 KB. DUMP: Mon Dec 2 23:18:12 2002 : We have written 4115292 KB. DUMP: Mon Dec 2 23:23:14 2002 : We have written 5679897 KB. DUMP: Mon Dec 2 23:28:14 2002 : We have written 7306871 KB. DUMP: Mon Dec 2 23:33:14 2002 : We have written 8830654 KB. DUMP: Mon Dec 2 23:38:14 2002 : We have written 10476997 KB. DUMP: Mon Dec 2 23:43:14 2002 : We have written 11551556 KB. DUMP: Mon Dec 2 23:48:14 2002 : We have written 13048942 KB. DUMP: Mon Dec 2 23:53:14 2002 : We have written 14688391 KB. DUMP: Mon Dec 2 23:58:14 2002 : We have written 16221813 KB. DUMP: Tue Dec 3 00:03:14 2002 : We have written 17364633 KB. DUMP: Tue Dec 3 00:08:14 2002 : We have written 18843934 KB. DUMP: Tue Dec 3 00:13:14 2002 : We have written 20516017 KB. DUMP: Tue Dec 3 00:18:14 2002 : We have written 21848469 KB. DUMP: Tue Dec 3 00:23:14 2002 : We have written 23232642 KB. DUMP: Tue Dec 3 00:28:14 2002 : We have written 24980200 KB. DUMP: Tue Dec 3 00:33:14 2002 : We have written 26645288 KB. DUMP: Tue Dec 3 00:38:14 2002 : We have written 28013082 KB. DUMP: Tue Dec 3 00:43:14 2002 : We have written 29653350 KB. DUMP: Tue Dec 3 00:48:14 2002 : We have written 31112304 KB. DUMP: Tue Dec 3 00:53:14 2002 : We have written 32806123 KB. DUMP: Tue Dec 3 00:58:14 2002 : We have written 34071851 KB. DUMP: Tue Dec 3 01:03:14 2002 : We have written 35518838 KB. DUMP: Tue Dec 3 01:08:14 2002 : We have written 36825522 KB. DUMP: Tue Dec 3 01:13:14 2002 : We have written 38524506 KB. DUMP: Tue Dec 3 01:18:14 2002 : We have written 39836545 KB. DUMP: Tue Dec 3 01:23:14 2002 : We have written 41444950 KB. DUMP: dumping (Pass V) [ACLs] DUMP: 42121234 KB DUMP: DUMP IS DONE -----Original Message----- From: Chris Thompson [mailto:cet1@cus.cam.ac.uk] Sent: Tuesday, December 10, 2002 2:22 PM To: Kumar, Rahul Cc: toasters(a)mathworks.com Subject: Re: Dump issue rahul.kumar(a)eds.com (Rahul Kumar) writes: > > Any Ideas why the dump is running so slow. We are running the filer to > dump onto a SDLT tape drive ^^^^^^^^^^^^^^^^^ Oh, yeah? In which case why does it say > DUMP: Dumping tape file 1 on /tmp/filer1tape ^^^^^^^^^^^^^^^ ? If this is dumping to a tape on a remote host, I suppose that _could_ be a symlink to a tape drive! But in that case, it may be the network that is the source of your speed problem. Chris Thompson University of Cambridge Computing Service, Email: cet1(a)ucs.cam.ac.uk New Museums Site, Cambridge CB2 3QH, Phone: +44 1223 334715 United Kingdom.

1 0

Dump issue
by Kumar, Rahul 10 Dec '02

10 Dec '02

Hi Any Ideas why the dump is running so slow. We are running the filer to dump onto a SDLT tape drive DUMP: creating "/vol/vol0/../snapshot_for_backup.0" snapshot. DUMP: Using Full Volume Dump DUMP: Dumping tape file 1 on /tmp/filer1tape DUMP: Date of this level 0 dump: Mon Dec 9 23:02:06 2002. DUMP: Date of last level 0 dump: the epoch. DUMP: Dumping /vol/vol0/ to root DUMP: mapping (Pass I)[regular files] DUMP: mapping (Pass II)[directories] DUMP: estimated 44482138 KB. DUMP: dumping (Pass III) [directories] DUMP: dumping (Pass IV) [regular files] DUMP: Mon Dec 9 23:08:37 2002 : We have written 328738 KB. DUMP: Mon Dec 9 23:13:37 2002 : We have written 813336 KB. DUMP: Mon Dec 9 23:18:37 2002 : We have written 1248349 KB. DUMP: Mon Dec 9 23:23:37 2002 : We have written 1780575 KB. DUMP: Mon Dec 9 23:28:37 2002 : We have written 2307507 KB. DUMP: Mon Dec 9 23:33:37 2002 : We have written 2782586 KB. DUMP: Mon Dec 9 23:38:37 2002 : We have written 3220125 KB. DUMP: Mon Dec 9 23:43:37 2002 : We have written 3751844 KB. DUMP: Mon Dec 9 23:48:37 2002 : We have written 4190113 KB. DUMP: Mon Dec 9 23:53:37 2002 : We have written 4641781 KB. DUMP: Mon Dec 9 23:58:37 2002 : We have written 5080513 KB. DUMP: Tue Dec 10 00:03:37 2002 : We have written 5606338 KB. DUMP: Tue Dec 10 00:08:37 2002 : We have written 6088137 KB. DUMP: Tue Dec 10 00:13:37 2002 : We have written 6572608 KB. DUMP: Tue Dec 10 00:18:37 2002 : We have written 7016947 KB. DUMP: Tue Dec 10 00:23:37 2002 : We have written 7542679 KB. DUMP: Tue Dec 10 00:28:37 2002 : We have written 7984060 KB. DUMP: Tue Dec 10 00:33:37 2002 : We have written 8503801 KB. DUMP: Tue Dec 10 00:38:37 2002 : We have written 8958353 KB. DUMP: Tue Dec 10 00:43:37 2002 : We have written 9390217 KB. DUMP: Tue Dec 10 00:48:37 2002 : We have written 9828824 KB. DUMP: Tue Dec 10 00:53:37 2002 : We have written 10260501 KB. Rahul

3 2

Raid Group question
by Art Hebert 10 Dec '02

10 Dec '02

I have a current volume (vol0) that has a raidgroup size of 8. I had 3 spares that weren't being used and I wanted to add them to the volume, but like an idiot didn't check the raidgroup size. Thus when I added the 3 disks to the volume (vol0) 1 went to raidgroup 0 and the other 2 went to raidgroup 1 as a parity drive and a data drive. This will create a performance problem from what I can tell not mention another parity drive being used. I'd like to get these two disks back under raid group 0 if I can. Suggestions on the best way to do this would be appreciated. I'm thinking the safest way is to back it up and recreate the volumes and restore. Thanks art hebert

1 0

RE: Placing Oracle onto a cluster
by Art Hebert 09 Dec '02

09 Dec '02

We have the following: Head One: redo logs dbf files dbs files ctl files system temp Head Two: redo logs archive logs ctl files Sun Server: oracle binaries ctl files Since its dedicated to our production database I am thinking of moving some of the data files over to Head Two to improve performance. We have seperate mount points for each database component so we can move as needed without too much effort. art hebert -----Original Message----- From: George Kahler [mailto:george@YorkU.CA] Sent: Saturday, December 07, 2002 7:30 AM To: toasters(a)mathworks.com Subject: Placing Oracle onto a cluster I would appreciate if someone could share how they break the various Oracle parts (pls excuse, I'm not a DBA), ie. data, logs, archives, & etc. amongst the various volumes in a NetApp cluster. I have an Oracle instance running on a singe F840 that very shortly is going to be clustered. The Oracle binaries are in a separate qtree and the rest the DBAs put in another qtree that I created for them. The qtrees and mount are by the SID. (HR) On the toaster /vol/db/<sid>/archs archive logs /ctl control files /data dbf files /rbs dbs files /redo redo logs /system /temp this gets mounted on a Sun DB server as filer:/vol/db/HR /rmounts/HR /oradata/HR --> /rmounts/DB/HR The DBAs refer to the database by a slink /oradata/HR in this case. A new mandate is to utilize the clustered F840s and to separate the database (.dbf), the redo and archive logs. In the cluster env there are now 4 volumes, 2 on each head. One for database, the other for logs (one has snapshots the other does not). Production DB is on one head, QA on the other. filer: A filer: B ------------------------ vol DB vol DB vol log vol log Do you folks that spread your logs end up with multiple nfs mounts per database? Do I suggest to my DBAs that I will have something like: A:/vol/db/HR/u01/data /rmounts/HR/u01 /rbs /system /temp A:/vol/log/HR/u02/redo1 /rmounts/HR/u02 /arch1 /ctl1 B:/vol/log/HR/u03/redo2 /rmounts/HR/u03 /arch2 /ctl2 /oradata/HR --> /rmounts/HR So I end up with 3 nfs mounts for each instance of the database? How do you people do this? There are about 10 other DBs waiting. What about a DSS; a reporting database? At the present time with all of the DB components on one volume all the DBAs have to snap the one volume and they can start up a DSS instance. In this new scenario, I guess they will still have to snap the DB volume and also snap one of the LOG volumes. What do you do? Sorry for being so winded but I was looking for specifics, unfortunately I couldn't find anything on either the NOW site or the archive list. Just the White Papers that describe the concept. Thanks, George ---------------------------------------------------------------------------- --- George Kahler e-mail: george(a)yorku.ca Sr. Systems Administrator humans: (416) 736-2100 x.22699 Computing and Network Services machines: (416) 736-5830 Ontario, Canada, M3J-1P3

1 0

RE: Suse & NetApp break oracle: flock problem
by Alex Harkema 09 Dec '02

09 Dec '02

Mike, thank you for your reply. I really suspect this is a linux-(kernel)problem. However I was curious if anyone had encountered the same issue. Besides that, we had not rebooted the machine when the error occurred. At the moment installations are running on the linux box (now still running the 2.4.18.-134 kernel, this one *not* containing the polyserver-flock patch) and I don't have the opportunity to reboot that one. However, I'll gladly send you the dumps afterward if we're still experiencing same problems. Can you tell me were I can find certifications for the oracle-on-linux/netapp products? Like, is 9iRac certified on Netapp? tia, Alex -----Original Message----- From: Kiernan, Michael [mailto:mkiernan@netapp.com] Sent: Monday, December 09, 2002 2:09 PM To: 'Alex Harkema' Cc: Rolf Fokkens Subject: RE: Suse & NetApp break oracle: flock problem Hi Alex, What did 'priv set advanced; lock_dump -h' show on the filer at the time you get the error ? Was Oracle shutdown cleanly prior to reboot ? One problem we're aware of with Linux rpc.statd is that it sometimes can fail to remove the locks on the filer on reboot, due to it locking with an unqualified nodename, and sending lock recovery packets with a fully-qualified domain name. If Oracle shutdown cleanly, however, the locks should have been released by that action. A pktt trace from the filer to the linux box (pktt start all -i <ip of oracle linux box>) while you reboot the upgraded box and restart oracle would be useful, as would the lock_dump output. I'll be happy to look at the data. Mike -----Original Message----- From: Alex Harkema [mailto:HarkemaA@vertis.nl] Sent: Monday, December 09, 2002 1:24 PM To: 'toasters(a)mathworks.com' Cc: Rolf Fokkens Subject: Suse & NetApp break oracle: flock problem Hi all! After upgrading the Linux kernel on our SuSE Linux Enterprise Server (SLES) to k_smp-2.4.18-224 we were unable to start our Oracle Database which was located on a NetApp Filer. All the feedback we got was "no locks available" Further investigation showed that k_smp-2.4.18-224 introduced a polyserve-flock patch which probably causes the problems. Has anyone ran into this? We'll do tests with k_smp-2.4.18-224 and/or 2.4.18-237 whithout the polyserve-flock patch and share this with those who express their interest. I address this report to both SuSE and NetApp for it's this combination that breaks Oracle. Because it's both in SuSE's and NetApp's interest I hope this report is appreciated though it may not have been sent through the proper channels. Alex Harkema Vertis

1 0

Suse & NetApp break oracle: flock problem
by Alex Harkema 09 Dec '02

09 Dec '02

Hi all! After upgrading the Linux kernel on our SuSE Linux Enterprise Server (SLES) to k_smp-2.4.18-224 we were unable to start our Oracle Database which was located on a NetApp Filer. All the feedback we got was "no locks available" Further investigation showed that k_smp-2.4.18-224 introduced a polyserve-flock patch which probably causes the problems. Has anyone ran into this? We'll do tests with k_smp-2.4.18-224 and/or 2.4.18-237 whithout the polyserve-flock patch and share this with those who express their interest. I address this report to both SuSE and NetApp for it's this combination that breaks Oracle. Because it's both in SuSE's and NetApp's interest I hope this report is appreciated though it may not have been sent through the proper channels. Alex Harkema Vertis

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters