toasters

toasters@lists.teaparty.net

2 participants
13529 discussions

RE: Real vs Raw disk space
by Steve Roche 13 Dec '01

13 Dec '01

According to my configuration tool you would realize 606GB of usable space assuming a 10% snapshot reserve. -----Original Message----- From: Geoff Hardin [mailto:geoff.hardin@dalsemi.com] Sent: Thursday, December 13, 2001 12:36 PM To: toasters Subject: Real vs Raw disk space Toasters, We are looking at purchasing some new DS14 shelves with 72 GB disks. In order to properly forecast our disk space growth, I was wondering if anyone out there had taken a look at the difference between real and usable disk space realized with these new shelves and drives. Basically, our environment is as follows: DS14 Shelf with 14 72 GB disks (1 RAID group of 8 disks, 1 RAID group of 6 disks), attached to a filer running DOT 6.1R1P1. We will have 1 spare disk and no snap reserve. Geoff Hardin geoff.hardin(a)dalsemi.com

2 1

Add Local Groups from the Command Line
by Nail, Larry 13 Dec '01

13 Dec '01

CIFS Question... Anyone know how to add "local" groups to a filer from the command line? I can't seem to find a method. ______________________________________ Lawrence D. Nail, MGTS Microsoft Certified Systems Engineer on Windows(r) 2000 Strategic Development Team Design Systems Texas Instruments Incorporated

2 1

CONCLUSION: UCD SNMP and NetApp Filers
by Ben Rockwood 13 Dec '01

13 Dec '01

Fellow Toasters: So after plowing around I found that it was probly time for me to actually learn SNMP alittle better than I previously had. I'll summerize my conclusions for postarity. So the first step is getting that MIB from NetApp. Then putting it in your /usr/local/share/snmp/mibs/ directory (yes, there is a way around this, but...). Then you can get cracking. By supplying a "-m netapp-mib-file" helped get around any problem I originally had. My biggest problem was getting the MIB file correctly loading. True you can walk the tree without it, but as most of the people who replied to me suggested, walking from .1 really sucks. (No offense, it's just alot of data) I found that all OID's can start at enterprises, rather than .iso. That helped. Then I found that restricting walking to the main arteries of the Filer MIB's groups was best, namely thoughs found in this document: http://now.netapp.com/NOW/knowledge/docs/ontap/rel60/html/sag/snmp3.htm# top Several commands that I liked were: # snmpwalk -m NETWORK-APPLIANCE-MIB -v 1 myfilerhost1 public .iso.org.dod.internet.private.enterprises.netapp.netapp1.raid # snmpget -m NETWORK-APPLIANCE-MIB -v 1 -Ov ralph public enterprises.netapp.netapp1.sysStat.environment.envFailedPowerSupplyMessa ge.0 The earlier command searching the RaidGroup information, which is IMHO the most intersting stuff from the Filers. The second grabbing a specific OID value. The "-Ov" arguement to snmpget says to suppress its natural urge to display the OID with the value, giving us JUST the value itself. Anyway, I mentioned that I was interested in all this stuff in order to find an easy and automated way to check for failed disks. I built a script to do just that. I'm attach it below. It's _really_ basic, but should help anyone interested in building such a script get a head start. I wanted this to be super straightforward, so don't get too down on my PERL. Thanx for all replies from the Guru Toasters out there. Ben Rockwood UNIX Systems Admin brockwood(a)homestead-inc.com ------------------------------------------------------------------------ ---- #!/usr/bin/perl #Usage: nacheckfilers toaster1 toaster2 toaster3 ...... # nacheckfilers v0.5 # Dependancies: PERL 5, UCD SNMP, NetApp MIB # Copyright (C) 2001 Homestead Inc. # Written by Ben Rockwood <brockwood(a)homestead-inc.com> # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. ## NOTES: This is an UNPOLISHED script for reference and modification. ## All SNMP gets are intended to be as simple to follow as possible. ## This script was inspired loosely (namely 2 lines, but...) on an earlier ## script by Daniel Quinlan posted to the Toasters group. ## In its current for this script does NO host checking and can therefore ## create pages of rubbish by supplying a non-existant or non-Filer host. ###################################### ## User Changable Config Parameters ## ###################################### #Set the path for UCD-SNMP $ENV{'PATH'}="/bin:/usr/bin:/usr/sbin:/usr/bin:/usr/local/bin"; #Make sure UCD-SNMP knows which MIB we wanna use #Note, the _actual_ file name is NETWORK-APPLIANCE-MIB.txt, the .txt should not be supplied below. #Note, the NetApp supplied MIB file is normally netapp_1_5.mib, link or rename it to NETWORK-APPLIANCE-MIB.txt $ENV{'MIBS'}="+NETWORK-APPLIANCE-MIB"; #Set the community string $COMMUNITY = "public"; #################### ## Main Run Loop ## #################### $ARGS = @ARGV; unless ($ARGS > 0) { die "Usage: nacheckfilers toaster1 toaster2 toaster3 ...\n"; } for ($i = 0; $i < $ARGS; $i++) { $TARGETHOST = $ARGV[$i]; __gen_filer_report(); } ############################# # Now do the heavy lifting ## ############################# sub __gen_filer_report { ## Setup some common stuff for convience $SNMPGET = "snmpget -v 1 -Ov $TARGETHOST $COMMUNITY "; $DATE = `date`; ## Heres where we get all the goods. chomp($failedfans = `$SNMPGET enterprises.netapp.netapp1.sysStat.environment.envFailedFanCount.0`); chomp($failedfansmsg = `$SNMPGET enterprises.netapp.netapp1.sysStat.environment.envFailedFanMessage.0`); chomp($failedps = `$SNMPGET enterprises.netapp.netapp1.sysStat.environment.envFailedPowerSupplyCount .0`); chomp($failedpsmsg = `$SNMPGET enterprises.netapp.netapp1.sysStat.environment.envFailedPowerSupplyMessa ge.0`); chomp($batterystat = `$SNMPGET enterprises.netapp.netapp1.sysStat.nvram.nvramBatteryStatus.0`); chomp($globalmsg = `$SNMPGET enterprises.netapp.netapp1.sysStat.misc.miscGlobalStatusMessage.0`); chomp($diskcount = `$SNMPGET enterprises.netapp.netapp1.raid.diskSummary.diskTotalCount.0`); chomp($activecount = `$SNMPGET enterprises.netapp.netapp1.raid.diskSummary.diskActiveCount.0`); chomp($rebuildingcount = `$SNMPGET enterprises.netapp.netapp1.raid.diskSummary.diskReconstructingCount.0`); chomp($reparitycount = `$SNMPGET enterprises.netapp.netapp1.raid.diskSummary.diskReconstructingParityCoun t.0`); chomp($verifyparitycount = `$SNMPGET enterprises.netapp.netapp1.raid.diskSummary.diskVerifyingParityCount.0`) ; chomp($failedcount = `$SNMPGET enterprises.netapp.netapp1.raid.diskSummary.diskFailedCount.0`); chomp($sparecount = `$SNMPGET enterprises.netapp.netapp1.raid.diskSummary.diskSpareCount.0`); chomp($failedmsg = `$SNMPGET enterprises.netapp.netapp1.raid.diskSummary.diskFailedMessage.0`); ## Here's where we output all this crap. print("\n\nStatus Report for $TARGETHOST: $DATE\n"); print(" The global message is: $globalmsg \n\n"); print(" There are $failedfans failed fans.\n"); if($failedfans != 0) { print("\n***FAILED FAN IN $TARGETHOST: $failedfanmsg ***\n\n"); } print(" There are $failedps failed power supplied.\n"); if($failedps != 0) { print("\n***FAILED POWER SUPPLY IN $TARGETHOST: $failedpsmsg ***\n\n"); } print(" Battery Status is: $batterystat \n"); print(" Total disk count is: $diskcount \n"); print(" The Active Disk Count is: $activecount \n"); print(" The Spare Disk Count is: $sparecount \n"); if($activecount + $sparecount != $diskcount) { print("\n***PROBLEM: ActiveDisks plus SpareDisks dont match TotalDisks. Investigate***\n\n"); } print(" The Failed Disk Count is: $failedcount \n"); if($failedcount != 0) { print("\n***FAILED DISK IN $TARGETHOST: $failedmsg ***\n\n"); } print("\n ------H--O--M--E--S--T--E--A--D--.--C--O--M-----\n"); } #------------------------- FIN -----------brockwood(a)homestead-inc.com---

1 0

RE: dump to overland jukebox dlt7000 w lvd scsi card problem
by Toal, Dave 13 Dec '01

13 Dec '01

Stephen, > disabled ndmpd Ok, so 'ndmpd status' returns "OFF"; and no sessions are active? I've had to 'ndmpd kill <session number>' occasionally, to free up the tape device after a (Veritas Net)backup problem. Rebooting the filer will also clear hung ndmpd sessions. Also, this sounds trivial, but bad scsi termination will cause write errors. mt will return good status (and even fsf & rewoffl) but dump won't be able to write to the tape device. > switched the jukebox with another unit If #2 was working under budtool, maybe its termination is good. Might try a different scsi cable as well? Another trivial point: the set-up @ork is an F740 with a dlt 7000 in a 9710 tape library. I've noticed that power-cycling the dlt or changing the connection to the filer seems to require rebooting the 740, before the dlt is useable. Sun boxes, on the other hand, can 'hot-swap' a scsi dlt without problems. -- Dave Toal unix resource thomson & thomson -----Original Message----- From: Darragh, Stephen J To: 'toasters(a)mathworks.com' Sent: 12/13/01 9:01 AM Subject: dump to overland jukebox dlt7000 w lvd scsi card problem Hello, I have a Overland mini library xpress 15 slot single dlt7000 that I attached to a f740 filer sysconfig -m and sysconfig -t report what appears to be the correct information and the lvd card is the correct rev and in the right slot. I even switched the jukebox with another unit that I use for BudTool NDMP backups of that filer. mt -f rst3a status shows write enabled and dlt7000 correctly. When I go to use the dump command, I get an error that it can't open the device for write, and asks for retry (which also fails) I disabled ndmpd before I tried to dump to rst3a I have an open case with NetApp, but am hoping someone else has already run into this Thanks Stephen Darragh UNIX Sys Admin BAE Systems

1 0

dump to overland jukebox dlt7000 w lvd scsi card problem
by Darragh, Stephen J 13 Dec '01

13 Dec '01

Hello, I have a Overland mini library xpress 15 slot single dlt7000 that I attached to a f740 filer sysconfig -m and sysconfig -t report what appears to be the correct information and the lvd card is the correct rev and in the right slot. I even switched the jukebox with another unit that I use for BudTool NDMP backups of that filer. mt -f rst3a status shows write enabled and dlt7000 correctly. When I go to use the dump command, I get an error that it can't open the device for write, and asks for retry (which also fails) I disabled ndmpd before I tried to dump to rst3a I have an open case with NetApp, but am hoping someone else has already run into this Thanks Stephen Darragh UNIX Sys Admin BAE Systems

1 0

Question about FC
by Ulf Zimmermann 12 Dec '01

12 Dec '01

Hello, I am unfortunatly new to FC and got a question about the FC-AL terminator used by NetApps, like the F720. Is this really a terminator, as in it has resistors, or is it a loopback adapter ? I found some cables for the NetApp I aquired but all they had was a loopback adapter instead of the terminator. -- Regards, Ulf. --------------------------------------------------------------------- Ulf Zimmermann, 1525 Pacific Ave., Alameda, CA-94501, #: 510-865-0204

1 0

RE: SecureAdmin
by Neto, Antonio Jose Rodrigues 12 Dec '01

12 Dec '01

Please update for 2.1.1D2. It's OK. Hope this help. neto -----Original Message----- From: Hanno Wagner [mailto:rince@kpnqwest.net] Sent: Wednesday, December 12, 2001 1:54 PM To: toasters(a)mathworks.com Subject: Re: SecureAdmin Hi, On Wed, Dec 12, 2001 at 09:23:21AM -0500, Stout, Noelette wrote: > That's really interesting... I use OpenSSH 2.3.0p1 on Solaris and I have no > problem connecting to the filer (runnin 6.1.1R2) via SecureAdmin 2.1.1. I have OpenSSH 2.9.1p2 on Solaris - and it closes the Connection with 6.1.1R1. Ciao, Hanno -- [A] KPNQwest Germany * Emmy-Noether-Strasse 9 * D-76131 Karlsruhe [T] +49 721 9652 0 [F] +49 721 9652 209 Teamleader Webhosting [E] hanno.wagner(a)kpnqwest.net [I] www.kpnqwest.de

1 0

RE: SecureAdmin
by Neto, Antonio Jose Rodrigues 12 Dec '01

12 Dec '01

Hi All, Please use SecureAdmin 2.1.1D2. It's OK. Hope this jelp. neto -----Original Message----- From: Stout, Noelette [mailto:NStout@IKON.com] Sent: Wednesday, December 12, 2001 12:23 PM To: 'Hannes Herret'; Lance A. Brown Cc: Kendall Libby; toasters(a)mathworks.com Subject: RE: SecureAdmin That's really interesting... I use OpenSSH 2.3.0p1 on Solaris and I have no problem connecting to the filer (runnin 6.1.1R2) via SecureAdmin 2.1.1. Noelette Stout Unix System Administrator IKON Office Solutions, Inc Digital Express tel. (770)326-4140 fax (770)326-4139 email: nstout(a)ikon.com -----Original Message----- From: Hannes Herret [mailto:hh@bacher.at] Sent: Wednesday, December 12, 2001 5:28 AM To: Lance A. Brown Cc: Kendall Libby; toasters(a)mathworks.com Subject: Re: SecureAdmin hi, i now had the same behaviour w/ 6.1.1R2P1 and secureadmin2.1.1 . see the log from openssh-host -> connection closed without reason !!! is there a debug of sshd available?? cu hannes =-> ssh -v -l root filer OpenSSH_2.9p1, SSH protocols 1.5/2.0, OpenSSL 0x0090601f debug1: Reading configuration data /usr/local/ssh/etc/ssh_config debug1: Seeded RNG with 27 bytes from programs debug1: Seeded RNG with 3 bytes from system calls debug1: Rhosts Authentication disabled, originating port will not be trusted. debug1: restore_uid debug1: ssh_connect: getuid 0 geteuid 0 anon 1 debug1: Connecting to filer [10.1.1.11] port 22. debug1: temporarily_use_uid: 0/1 (e=0) debug1: restore_uid debug1: temporarily_use_uid: 0/1 (e=0) debug1: restore_uid debug1: Connection established. debug1: read PEM private key done: type DSA debug1: read PEM private key done: type RSA debug1: identity file /root/.ssh/identity type 0 debug1: identity file /root/.ssh/id_rsa type 1 debug1: identity file /root/.ssh/id_dsa type -1 debug1: Remote protocol version 1.5, remote software version 2.1.1 debug1: no match: 2.1.1 debug1: Local version string SSH-1.5-OpenSSH_2.9p1 debug1: Waiting for server public key. debug1: Received server public key (512 bits) and host key (768 bits). debug1: Host 'filer' is known and matches the RSA1 host key. debug1: Found key in /root/.ssh/known_hosts:4 debug1: Encryption type: 3des debug1: Sent encrypted session key. debug1: Installing crc compensation attack detector. debug1: Received encrypted confirmation. debug1: Doing password authentication. root@filer's password: debug1: Requesting pty. debug1: Requesting shell. debug1: Entering interactive session. Connection to filer closed. debug1: Transferred: stdin 0, stdout 1, stderr 29 bytes in 0.5 seconds debug1: Bytes per second: stdin 0.0, stdout 2.0, stderr 58.0 debug1: Exit status 0 debug1: writing PRNG seed to file /root/.ssh/prng_seed root@dumbo : /root :> =-> "Lance A. Brown" wrote: > Greetings, > > This is bug 42404, according to my interactions with NetApp support. > It is fixed in 6.1.1R1 > > --[Lance] > > > > > SecureAdmin 2.1.1 > > ONTAP 6.1R1 > > > > New system, just slapped SecureAdmin on it... secure filer view works > > happily, ssh has ... issues. > > > > I can connect fine from a host running ssh v1.2.27, but connecting > > from OpenSSH hosts seems to fail. Said hosts are FreeBSD and Solaris > > running 2.3.0 and 2.5.1_p1 respectively. Initially, I thought it was > > a protocol 2 issue, but that's not it. Here's what happens: > > > > SSH Version OpenSSH_2.3.0 green(a)FreeBSD.org 20010321, protocol > > versions 1.5/2.0. > > Compiled with SSL (0x0090600f). > > debug: Reading configuration data /etc/ssh/ssh_config > > debug: Applying options for * > > debug: ssh_connect: getuid 0 geteuid 0 anon 0 > > debug: Connecting to (null) [xxx.xxx.xxx.xxx] port 22. > > debug: Allocated local port 1022. > > debug: Connection established. > > debug: Remote protocol version 1.5, remote software version 2.1.1 > > debug: match: 2.1.1 pat ^2\. > > > > debug: Local version string SSH-1.5-OpenSSH_2.3.0 green(a)FreeBSD.org > > 20010321 > > debug: Waiting for server public key. > > debug: Received server public key (512 bits) and host key (768 bits). > > debug: Host 'xxx.xxx.xxx.xxx' is known and matches the RSA host key. > > debug: Encryption type: 3des > > debug: Sent encrypted session key. > > debug: Installing crc compensation attack detector. > > debug: Received encrypted confirmation. > > debug: Doing password authentication. > > root(a)xxx.xxx.xxx.xxx's password: > > debug: Requesting pty. > > debug: Requesting X11 forwarding with authentication spoofing. > > Warning: Remote host denied X11 forwarding. > > debug: Requesting authentication agent forwarding. > > Warning: Remote host denied authentication agent forwarding. > > debug: Requesting shell. > > debug: Entering interactive session. > > Connection to xxx.xxx.xxx.xxx closed. > > debug: Transferred: stdin 0, stdout 1, stderr 36 bytes in 0.5 seconds > > debug: Bytes per second: stdin 0.0, stdout 2.0, stderr 73.4 > > debug: Exit status 0 > > > > I can't imagine that everyone out there using ssh on the filer is also > > still using the old ssh versions, so it's gotta be me. Right? > > > > K. > > -- ------------------------------------------------------------------- Hannes Herret IT-Service / Storage phone : +43 (1) 60 126-34 Bacher Systems EDV GmbH fax : +43 (1) 60 126-555 Wienerbergstr. 11B mailto:hh@bacher.at A-1101 Wien, Austria www : http://www.bacher.at/ Europe

2 1

RE: SecureAdmin
by Stout, Noelette 12 Dec '01

12 Dec '01

That's really interesting... I use OpenSSH 2.3.0p1 on Solaris and I have no problem connecting to the filer (runnin 6.1.1R2) via SecureAdmin 2.1.1. Noelette Stout Unix System Administrator IKON Office Solutions, Inc Digital Express tel. (770)326-4140 fax (770)326-4139 email: nstout(a)ikon.com -----Original Message----- From: Hannes Herret [mailto:hh@bacher.at] Sent: Wednesday, December 12, 2001 5:28 AM To: Lance A. Brown Cc: Kendall Libby; toasters(a)mathworks.com Subject: Re: SecureAdmin hi, i now had the same behaviour w/ 6.1.1R2P1 and secureadmin2.1.1 . see the log from openssh-host -> connection closed without reason !!! is there a debug of sshd available?? cu hannes =-> ssh -v -l root filer OpenSSH_2.9p1, SSH protocols 1.5/2.0, OpenSSL 0x0090601f debug1: Reading configuration data /usr/local/ssh/etc/ssh_config debug1: Seeded RNG with 27 bytes from programs debug1: Seeded RNG with 3 bytes from system calls debug1: Rhosts Authentication disabled, originating port will not be trusted. debug1: restore_uid debug1: ssh_connect: getuid 0 geteuid 0 anon 1 debug1: Connecting to filer [10.1.1.11] port 22. debug1: temporarily_use_uid: 0/1 (e=0) debug1: restore_uid debug1: temporarily_use_uid: 0/1 (e=0) debug1: restore_uid debug1: Connection established. debug1: read PEM private key done: type DSA debug1: read PEM private key done: type RSA debug1: identity file /root/.ssh/identity type 0 debug1: identity file /root/.ssh/id_rsa type 1 debug1: identity file /root/.ssh/id_dsa type -1 debug1: Remote protocol version 1.5, remote software version 2.1.1 debug1: no match: 2.1.1 debug1: Local version string SSH-1.5-OpenSSH_2.9p1 debug1: Waiting for server public key. debug1: Received server public key (512 bits) and host key (768 bits). debug1: Host 'filer' is known and matches the RSA1 host key. debug1: Found key in /root/.ssh/known_hosts:4 debug1: Encryption type: 3des debug1: Sent encrypted session key. debug1: Installing crc compensation attack detector. debug1: Received encrypted confirmation. debug1: Doing password authentication. root@filer's password: debug1: Requesting pty. debug1: Requesting shell. debug1: Entering interactive session. Connection to filer closed. debug1: Transferred: stdin 0, stdout 1, stderr 29 bytes in 0.5 seconds debug1: Bytes per second: stdin 0.0, stdout 2.0, stderr 58.0 debug1: Exit status 0 debug1: writing PRNG seed to file /root/.ssh/prng_seed root@dumbo : /root :> =-> "Lance A. Brown" wrote: > Greetings, > > This is bug 42404, according to my interactions with NetApp support. > It is fixed in 6.1.1R1 > > --[Lance] > > > > > SecureAdmin 2.1.1 > > ONTAP 6.1R1 > > > > New system, just slapped SecureAdmin on it... secure filer view works > > happily, ssh has ... issues. > > > > I can connect fine from a host running ssh v1.2.27, but connecting > > from OpenSSH hosts seems to fail. Said hosts are FreeBSD and Solaris > > running 2.3.0 and 2.5.1_p1 respectively. Initially, I thought it was > > a protocol 2 issue, but that's not it. Here's what happens: > > > > SSH Version OpenSSH_2.3.0 green(a)FreeBSD.org 20010321, protocol > > versions 1.5/2.0. > > Compiled with SSL (0x0090600f). > > debug: Reading configuration data /etc/ssh/ssh_config > > debug: Applying options for * > > debug: ssh_connect: getuid 0 geteuid 0 anon 0 > > debug: Connecting to (null) [xxx.xxx.xxx.xxx] port 22. > > debug: Allocated local port 1022. > > debug: Connection established. > > debug: Remote protocol version 1.5, remote software version 2.1.1 > > debug: match: 2.1.1 pat ^2\. > > > > debug: Local version string SSH-1.5-OpenSSH_2.3.0 green(a)FreeBSD.org > > 20010321 > > debug: Waiting for server public key. > > debug: Received server public key (512 bits) and host key (768 bits). > > debug: Host 'xxx.xxx.xxx.xxx' is known and matches the RSA host key. > > debug: Encryption type: 3des > > debug: Sent encrypted session key. > > debug: Installing crc compensation attack detector. > > debug: Received encrypted confirmation. > > debug: Doing password authentication. > > root(a)xxx.xxx.xxx.xxx's password: > > debug: Requesting pty. > > debug: Requesting X11 forwarding with authentication spoofing. > > Warning: Remote host denied X11 forwarding. > > debug: Requesting authentication agent forwarding. > > Warning: Remote host denied authentication agent forwarding. > > debug: Requesting shell. > > debug: Entering interactive session. > > Connection to xxx.xxx.xxx.xxx closed. > > debug: Transferred: stdin 0, stdout 1, stderr 36 bytes in 0.5 seconds > > debug: Bytes per second: stdin 0.0, stdout 2.0, stderr 73.4 > > debug: Exit status 0 > > > > I can't imagine that everyone out there using ssh on the filer is also > > still using the old ssh versions, so it's gotta be me. Right? > > > > K. > > -- ------------------------------------------------------------------- Hannes Herret IT-Service / Storage phone : +43 (1) 60 126-34 Bacher Systems EDV GmbH fax : +43 (1) 60 126-555 Wienerbergstr. 11B mailto:hh@bacher.at A-1101 Wien, Austria www : http://www.bacher.at/ Europe

4 3

Samba PDC & NetApp
by X Gynt 12 Dec '01

12 Dec '01

Did anyone have any success using Samba (2.2.2?) as the PDC for NetApp's /Gynt -- _______________________________________________ Sign-up for your own FREE Personalized E-mail at Mail.com http://www.mail.com/?sr=signup 1 cent a minute calls anywhere in the U.S.! http://www.getpennytalk.com/cgi-bin/adforward.cgi?p_key=RG9853KJ&url=http:/…

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters