toasters October 2012

toasters@lists.teaparty.net

35 participants
21 discussions

LUNs lost initiator setting after a power outage
by Andrew Laurence 16 Nov '12

16 Nov '12

So, this was fun. After a power outage on Saturday, our Exchange 2010 mailbox servers (all hosted on VMWare ESX) came up with only some of their LUNs. These LUNs are hosted on our HA-paired FAS2040 (OnTAP 7.3.4); each LUN hosts an Exchange database. Try as we might, we couldn't get these LUNs to mount. After opening a support case with each of NetApp and Microsoft, and conferencing in both engineers, and several hours, the Microsoft engineer finally suggested we check the target LUNs on the filer. Turns out that each of the missing LUNs no longer had the setting for the mailbox servers' initiators. After checking that box the LUNs mounted without issue. Anyone seen this happen before? The NetApp engineer didn't have any immediate ideas, and I haven't had luck finding a match in the list archives or in NOW. Thanks, Andrew -- Andrew Laurence Office of Information Technology atlauren(a)uci.edu University of California, Irvine

8 8

vFiler with vlan tagging?
by Paul Letta 25 Oct '12

25 Oct '12

Hello, I'm trying to deploy multiple vFilers on a single filer. The vFilers need to be on different vlans. I want to use the same VIF (made up of 2 interfaces) for this. Am I wrong in my thinking that I can use vlan tagging with these multiple vfilers while using the same vif? I think I should be able to use the default IPspace on the vif. I've had our network people trunk 2 vlans to the ports that make up the VIF. I then create 2 vFilers, each specifying its own IP address/subnet mask. (The IPs and masks are different for each vfiler). But what I'm missing is the connection between vfiler and vlan. With ESX, we trunk dozens of vlans to the nics that the esx server is using for the vswitch. We then configure the vm's to be on which ever vlan we need it to be. Of course that's ESX's virtual switch doing the work for me. How does this picture map into vfiler / physical filer space? Thanks, Paul

4 3

Re: vFiler with vlan tagging?
by Scott 25 Oct '12

25 Oct '12

First I created the vfilers and issued them front end IP addresses, aliased to the front end VIF on the controller. Then, I created a back end VIF for hosts to mount their storage (e0b & e1b). On that VIF I created 4 vlans. Once the vlans were created, I went back to the vfiler and added a back end IP, then aliased that IP to the particular vlan. Of course, the networking guy did his magic on the switch side creating the vlans. E: flags=0xa2f48863<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM,VLAN> mtu 1500 ether 02:a0:98:0f:01:79 (Enabled interface groups) E-34: flags=0x2b48863<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500 inet 10.54.34.6 netmask 0xffffff00 broadcast 10.54.34.255 ether 02:a0:98:0f:01:79 (Enabled interface groups) E-36: flags=0x2b48863<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500 inet 10.55.10.6 netmask 0xffffff00 broadcast 10.55.10.255 inet 10.55.10.220 netmask 0xffffff00 broadcast 10.55.10.255 ether 02:a0:98:0f:01:79 (Enabled interface groups) E-38: flags=0x2b48863<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500 inet 10.53.10.6 netmask 0xffffff00 broadcast 10.53.10.255 ether 02:a0:98:0f:01:79 (Enabled interface groups) E-40: flags=0x2b48863<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500 inet 10.56.10.6 netmask 0xffffff00 broadcast 10.56.10.255 ether 02:a0:98:0f:01:79 (Enabled interface groups) On Oct 25, 2012, at 4:33 PM, Coartney, Paul wrote: > How did you create the vfilers? Can you send me the output of > > Ifconfig -a > >> From vfiler0. > > On 10/25/12 1:24 PM, "Christopher S Eno" <s.eno(a)me.com> wrote: > >> In our case, I created a VIF of two physical interfaces, then assigned 4 >> different vlans to that VIF. Then, the vfiler's IP addresses were >> aliased to whichever vlan they needed. Using default IP space. >> >> >> On Oct 25, 2012, at 4:13 PM, Paul Letta wrote: >> >>> Hello, >>> >>> I'm trying to deploy multiple vFilers on a single filer. The vFilers >>> need to be on different vlans. >>> I want to use the same VIF (made up of 2 interfaces) for this. Am I >>> wrong in my thinking that I can >>> use vlan tagging with these multiple vfilers while using the same vif? >>> >>> I think I should be able to use the default IPspace on the vif. >>> >>> I've had our network people trunk 2 vlans to the ports that make up the >>> VIF. >>> I then create 2 vFilers, each specifying its own IP address/subnet mask. >>> (The IPs and masks are different for each vfiler). >>> >>> But what I'm missing is the connection between vfiler and vlan. >>> >>> With ESX, we trunk dozens of vlans to the nics that the esx server is >>> using for the vswitch. >>> We then configure the vm's to be on which ever vlan we need it to be. >>> Of course that's ESX's >>> virtual switch doing the work for me. >>> >>> How does this picture map into vfiler / physical filer space? >>> >>> >>> Thanks, >>> Paul >>> >>> >>> >>> >>> _______________________________________________ >>> Toasters mailing list >>> Toasters(a)teaparty.net >>> http://www.teaparty.net/mailman/listinfo/toasters >> >> _______________________________________________ >> Toasters mailing list >> Toasters(a)teaparty.net >> http://www.teaparty.net/mailman/listinfo/toasters >

1 0

test an upgrade without fully failing back an HA node?
by Randy Rue 24 Oct '12

24 Oct '12

Hello All, We just had a rough night trying to upgrade a 3020 from 7.2.4 to 7.3.7 where the upgraded node came up with no working network connections. Turns out the same setup / RC file that works great in 7.2.4 crumps in 7.3.7. After some flailing, we used revert_to to roll back. In the mean time, our VMware farm had several hundred VMs suffer a ~1 hour loss of their disks and ~40 VMs needed help ranging from just a reboot to a full rollback of the disk image. This Sunday we're scheduled to upgrade a v3170 pair that supports about three times as many VMs and uses the same network design. We have some reasons to hope that the same issue won't bite us, and also have better options to test before we have to risk the virtual farm, and some other ideas on how to mitigate risk and impact. But it was also suggested that an even better way to avoid a problem would be if we could upgrade a node and then test the RC file before it was actually "given back." If I do a takeover from Node B, then upgrade Node A, is there a way to bring up Node A far enough to test that its RC file loaded successfully but before it tries to go back into production? Hope to hear from you, Randy

7 7

Max CIFS Multiplex
by Ian Dotson 23 Oct '12

23 Oct '12

The online "Health Check" was telling us that: sam.misconfig.badValue.cifsMaxMpx: Max Multiplex should not be near or equal to the value of the option cifs.max_mpx. The value of cifs.max_mpx should be set higher. Please discuss what values are acceptable for this parameter with a NetApp technical support representative. Then, use "options set cifs.max_mpx ..." to set this option to a new value. We're seeing unusually large high-water mark for cifs max multiplex -- up to 511 (from 'cifs stats -c') We do have some client IIS servers (Server 2003, IIS 6.0) that we suspect are the culprits. "cifs stat -h <ip>" doesn't display the max number of multiplexed cifs sessions requested for a host, only current stats. Is there a way to determine which client had asked for this huge number of multiplexed cifs sessions? Thanks, Ian Dotson

3 2

2220 or 2240? Performance issues?
by Milazzo Giacomo 23 Oct '12

23 Oct '12

Hi all, I've a startup project that will expect a very simple Hyper-V 2012 iSCSI cluster. Competitor is an EqualLogic appliance 24x300 10k SAS HDs. There's a very interesting promo in these days offering a 2220 12x900 SAS at very competitive price vs a 2240-2 24x450 SAS so I was evaluating the 2220. But I'm concerned about performances. The two server nodes will be "only" 32 GB RAM/2 CPU Xeon 8C so I can presume that the number of VM servers will be no more that 7-8 I've lab data from NetApp that state, at lab conditions, that for short random ops with iSCSI the 2220 can do roughy 10k IOPS vs the 30k of the 2240-2, for short write they are quite similar (40k IOPS). For large ops the 2240-2's performances are very higher that 2220. In addition to this I've also considered that the 2220 will have ONLY 6 mechanics for each controller while the 2240 will have 12 each... What do you think about this little cluster? Can I trust in 2220? Regards

3 5

Re: smtape vs. dump
by Peter Kain 22 Oct '12

22 Oct '12

On Fri, Oct 19, 2012 at 2:49 PM, Peter Kain <pkain42(a)gmail.com> wrote: > Thanks for the quick replies: > > The volume is the destination for a SnapMirror from the production filer. > > The volume contains a single LUN. > > FAS3210> df -h /vol/papayasg1_mirror > Filesystem total > used avail capacity Mounted on > /vol/papayasg1_mirror/ 260GB 212GB > 47GB 82% /vol/papayasg1_mirror/ > /vol/papayasg1_mirror/.snapshot 0GB 11GB 0GB > ---% /vol/papayasg1_mirror/.snapshot > > The LUN is 200GB > > FAS3210> lun show /vol/papayasg1_mirror/papayasg1db > /vol/papayasg1_mirror/papayasg1db 200.0g (214778511360) (r/o, online) > > NetApp OnCommand System Manager reports that the LUN is 60% used with 79GB > free(not sure how to do this at the command line). So that would be 120GB > used and that matches what I see when i run a dump. When I run smtape > backup it reports 139GB backed up. Not sure why but I think it has to do > with SnapShots. > > Here's today's test run to a tape drive: > > -- dump takes 19 minutes 34 seconds to back up 120GB@105MB/sec , which I > consider to be rather good for an LTO4 tape drive > > -- smtape backed up 139.481 GB bytes in 0:44:56 so 52MB/sec > > > FAS3210> sysstat -u 3 > CPU Total Net kB/s Disk kB/s Tape kB/s Cache Cache > CP CP Disk > ops/s in out read write read write age hit > time ty util > 2% 0 0 0 0 0 0 0 >60 100% > 0% - 0% > 1% 0 0 0 0 0 0 0 >60 100% > 0% - 0% > 1% 0 0 0 9 297 0 0 >60 92% > 4% T 3% > 5% 0 0 0 0 0 0 0 >60 100% > 0% - 0% > > > FAS3210> mt -f nrst0h status > Tape drive: IBM LTO 4 ULT3580 > Status: ready, write enabled > Format: LTO 4 800GB > fileno = 0 blockno = 0 resid = 0 > > > > FAS3210> date; dump 0f nrst0h /vol/papayasg1_mirror; date > Fri Oct 19 12:58:34 PDT 2012 > > DUMP: Using Full Volume Dump > DUMP: Dumping tape file 1 on nrst0h > DUMP: Date of this level 0 dump: Fri Oct 19 02:50:36 2012. > > DUMP: Date of last level 0 dump: the epoch. > DUMP: Dumping /vol/papayasg1_mirror to nrst0h > > DUMP: mapping (Pass I)[regular files] > DUMP: mapping (Pass II)[directories] > DUMP: estimated 126938643 KB. > > DUMP: dumping (Pass III) [directories] > DUMP: dumping (Pass IV) [regular files] > DUMP: Fri Oct 19 13:03:35 2012 : We have written 33948408 KB. > 32GB@110MBsec > DUMP: Fri Oct 19 13:08:35 2012 : We have written 68791721 KB. > 33GB@113MBsec > DUMP: Fri Oct 19 13:13:35 2012 : We have written 99138845 KB. > 28GB@98MB/sec > DUMP: dumping (Pass V) [ACLs] > DUMP: 126442444 KB > DUMP: This dump has written to 1 tapefile(s). > DUMP: DUMP IS DONE > Fri Oct 19 13:18:08 PDT 2012 > > > FAS3210> smtape backup /vol/papayasg1_mirror nrst0h > > > FAS3210> rdfile /etc/log/backup > > (null) Fri Oct 19 13:20:18 PDT /vol/papayasg1_mirror CLI-Backup (nrst0h) > 25 Fri Oct 19 13:20:18 PDT /vol/papayasg1_mirror BKP-Start (level 0 backup > of Backup Set ID b93ab842-1bbc-11e2-8766-00a09813e8e0) > 25 Fri Oct 19 13:20:18 PDT /vol/papayasg1_mirror BKP-Params > (originator=CLI mode=dw-data tape_record_size=240KB tape=nrst0h > all_snapshots tape_seeding) > 25 Fri Oct 19 13:20:19 PDT /vol/papayasg1_mirror BKP-DW-Start > 25 Fri Oct 19 14:05:12 PDT /vol/papayasg1_mirror BKP-DW-End (phase > completed in 0:44:53; 34870380 blocks moved) > 25 Fri Oct 19 14:05:14 PDT /vol/papayasg1_mirror BKP-Tape-Stats > (wait_time=488s wait_count=264987 total_count=582596 > avail_count=345482/79992/79015/78107) > 25 Fri Oct 19 14:05:14 PDT /vol/papayasg1_mirror BKP-End (backed up > 139.481 GB bytes in 0:44:56; performance=186.251 GB/hour) > FAS3210> > > FAS3210> snap list papayasg1_mirror > Volume papayasg1_mirror > working... > > %/used %/total date name > ---------- ---------- ------------ -------- > 0% ( 0%) 0% ( 0%) Oct 19 02:50 > FAS3210(1573774893)_papayasg1_mirror.405 (snapmirror) > 0% ( 0%) 0% ( 0%) Oct 19 02:50 @snapmir@{D1FE5C41-8C5A-4BAF-BE56-C7E32FCF8C0E} > > 0% ( 0%) 0% ( 0%) Oct 18 19:03 > FAS3210(1573774893)_papayasg1_mirror.404 > 0% ( 0%) 0% ( 0%) Oct 18 19:03 @snapmir@{9937F7F6-FCC7-4D74-8A79-847C620B5825} > > 0% ( 0%) 0% ( 0%) Oct 18 19:01 exchsnap__papaya__recent > 1% ( 1%) 0% ( 0%) Oct 17 19:01 > exchsnap__papaya_10-17-2012_19.00.24__daily > 1% ( 1%) 1% ( 0%) Oct 16 19:01 > exchsnap__papaya_10-16-2012_19.00.27__daily > 2% ( 0%) 1% ( 0%) Oct 16 03:17 > FAS3210(1573774893)_papayasg1_mirror.399 (snapmirror) > 2% ( 0%) 1% ( 0%) Oct 15 19:01 > exchsnap__papaya_10-15-2012_19.00.25__daily > 2% ( 0%) 1% ( 0%) Oct 15 13:07 > exchsnap__papaya_10-15-2012_13.06.38__daily > 3% ( 0%) 1% ( 0%) Oct 14 19:01 > exchsnap__papaya_10-14-2012_19.00.26__daily > 3% ( 0%) 1% ( 0%) Oct 13 19:01 > exchsnap__papaya_10-13-2012_19.00.38__daily > 3% ( 0%) 2% ( 0%) Oct 12 19:01 > exchsnap__papaya_10-12-2012_19.00.31__daily > 4% ( 0%) 2% ( 0%) Oct 11 19:01 > exchsnap__papaya_10-11-2012_19.00.26__daily > 4% ( 1%) 2% ( 0%) Oct 10 19:01 > exchsnap__papaya_10-10-2012_19.00.32__daily > 5% ( 1%) 2% ( 0%) Oct 09 19:01 > exchsnap__papaya_10-09-2012_19.00.48__daily > 6% ( 1%) 3% ( 0%) Oct 08 19:01 > exchsnap__papaya_10-08-2012_19.00.30__daily > 6% ( 1%) 3% ( 0%) Oct 07 19:01 > exchsnap__papaya_10-07-2012_19.00.22__daily > 6% ( 0%) 3% ( 0%) Oct 06 19:01 > exchsnap__papaya_10-06-2012_19.00.34__daily > 7% ( 0%) 3% ( 0%) Oct 05 19:01 > exchsnap__papaya_10-05-2012_19.00.29__daily > 7% ( 1%) 4% ( 0%) Oct 04 19:01 > exchsnap__papaya_10-04-2012_19.00.28__daily > 9% ( 2%) 5% ( 1%) Sep 26 21:51 > FAS3210(1573774893)_papayasg1_mirror.358 (snapmirror) > > > Pete > > > > On Fri, Oct 19, 2012 at 12:01 PM, Kennedy, Jeffrey <jkennedy(a)qualcomm.com>wrote: > >> Do you have a large volume with a small amount of data? Dump will only >> backup the data but smtape backs up all the blocks regardless if there’s >> data or not.**** >> >> ** ** >> >> Jeff Kennedy**** >> >> Qualcomm, Incorporated**** >> >> QCT Engineering Compute**** >> >> 858-651-6592**** >> >> ** ** >> >> *From:* toasters-bounces(a)teaparty.net [mailto: >> toasters-bounces(a)teaparty.net] *On Behalf Of *Peter Kain >> *Sent:* Friday, October 19, 2012 10:24 AM >> *To:* toasters(a)teaparty.net >> *Subject:* smtape vs. dump**** >> >> ** ** >> >> I'm looking at backing up our filer to tape for DR purposes. I've tested >> both smtape and dump and was surprised that dump was 3 times faster than >> smtape. The test was preformed on a idle filer. The filer is the >> destination for SnapMirrors at night and idle during the day. >> >> Given that smtape is a block backup I'd expect it to be as fast or faster >> than dump. The test Volume contains a single LUN. >> >> FAS3210> version >> NetApp Release 8.0.2P6 7-Mode: Fri Jan 27 14:48:08 PST 2012 >> >> >> - filer is idle >> >> FAS3210> sysstat -u 1 >> CPU Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk >> ops/s in out read write read write age hit time ty util >> 2% 0 1 0 0 0 0 0 2s 100% 0% - 0% >> 1% 0 2 2 0 0 0 0 2s 100% 0% - 0% >> >> - dump to null takes about 13 minutes >> >> FAS3210> date; dump f null /vol/papayasg1_mirror; date >> Tue Oct 16 12:48:06 PDT 2012 >> DUMP: Using Full Volume Dump >> >> DUMP: Dumping tape file 1 on null >> DUMP: Date of this level 0 dump: Tue Oct 16 03:17:08 2012. >> DUMP: Date of last level 0 dump: the epoch. >> DUMP: Dumping /vol/papayasg1_mirror to null >> DUMP: mapping (Pass I)[regular files] >> DUMP: mapping (Pass II)[directories] >> DUMP: estimated 126361315 KB. >> DUMP: dumping (Pass III) [directories] >> DUMP: dumping (Pass IV) [regular files] >> DUMP: Tue Oct 16 12:53:07 2012 : We have written 52672652 >> KB. 50GB@171MB/sec >> DUMP: Tue Oct 16 12:58:07 2012 : We have written 99181873 KB. >> 44GB@151MB/sec >> DUMP: dumping (Pass V) [ACLs] >> DUMP: 125867380 KB 120GB@155MB/sec >> DUMP: DUMP IS DONE >> Tue Oct 16 13:01:18 PDT 2012 >> >> - smtape backup to null takes 44 minutes >> >> FAS3210> rdfile /etc/log/backup >> (null) Tue Oct 16 13:05:32 PDT /vol/papayasg1_mirror CLI-Backup (null) >> 19 Tue Oct 16 13:05:32 PDT / >> vol/papayasg1_mirror BKP-Start (level 0 backup of Backup Set ID >> 2bc95ed2-195f-11e2-8766-00a09813e8e0) >> 19 Tue Oct 16 13:05:32 PDT /vol/papayasg1_mirror BKP-Params >> (originator=CLI mode=dw-data tape_record_size=240KB tape=null all_snapshots >> tape_seeding) >> 19 Tue Oct 16 13:05:33 PDT /vol/papayasg1_mirror BKP-DW-Start >> 19 Tue Oct 16 13:50:26 PDT /vol/papayasg1_mirror BKP-DW-End (phase >> completed in 0:44:53; 34394279 blocks moved) >> 19 Tue Oct 16 13:50:26 PDT /vol/papayasg1_mirror BKP-Tape-Stats >> (wait_time=0s wait_count=0 total_count=0 avail_count=0/0/0/0) >> 19 Tue Oct 16 13:50:26 PDT /vol/papayasg1_mirror BKP-End (backed up >> 137.577 GB bytes in 0:44:54; performance=183.844 GB/hour) >> >> >> -- >> Pete Kain >> 310-506-4475**** >> > > > > -- > Pete Kain > 310-506-4475 > -- Pete Kain 310-506-4475

1 0

Raidgroupsize and I/O Performance
by Steffen Knauf 22 Oct '12

22 Oct '12

Hello, i'll try to improve our I/0 Perfomance. We have a raidgroup with 16 disks (SAS), 1 aggregate and 1 volume (dedup enabled). The volume is the storage for 100 VM's on the VMware Cluser (access via NFS). Does it make sense to increase the raidgroup? 90% Percent of the Disk I/O are Read Ops, so i'll buy a PAM Card for our FAS3240,too. And what's your Experience with the Raidgroupsize of a Raidgroup with SATA Disks (now: 11+1)? Thanks and greets ! Steffen

7 16

NSF client listing
by Weasner, Matthew K 19 Oct '12

19 Oct '12

This is probably a silly question, but what is the most reliable way to get a list of NFS clients that have an export mounted? OnTap 7.3.3, and the share is exported R/W to the world. Thanks, Matt Sent from my iPad The information contained in this message may be privileged, confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify your representative immediately and delete this message from your computer. Thank you.

7 10

smtape vs. dump
by Peter Kain 19 Oct '12

19 Oct '12

I'm looking at backing up our filer to tape for DR purposes. I've tested both smtape and dump and was surprised that dump was 3 times faster than smtape. The test was preformed on a idle filer. The filer is the destination for SnapMirrors at night and idle during the day. Given that smtape is a block backup I'd expect it to be as fast or faster than dump. The test Volume contains a single LUN. FAS3210> version NetApp Release 8.0.2P6 7-Mode: Fri Jan 27 14:48:08 PST 2012 - filer is idle FAS3210> sysstat -u 1 CPU Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk ops/s in out read write read write age hit time ty util 2% 0 1 0 0 0 0 0 2s 100% 0% - 0% 1% 0 2 2 0 0 0 0 2s 100% 0% - 0% - dump to null takes about 13 minutes FAS3210> date; dump f null /vol/papayasg1_mirror; date Tue Oct 16 12:48:06 PDT 2012 DUMP: Using Full Volume Dump DUMP: Dumping tape file 1 on null DUMP: Date of this level 0 dump: Tue Oct 16 03:17:08 2012. DUMP: Date of last level 0 dump: the epoch. DUMP: Dumping /vol/papayasg1_mirror to null DUMP: mapping (Pass I)[regular files] DUMP: mapping (Pass II)[directories] DUMP: estimated 126361315 KB. DUMP: dumping (Pass III) [directories] DUMP: dumping (Pass IV) [regular files] DUMP: Tue Oct 16 12:53:07 2012 : We have written 52672652 KB. 50GB@171MB/sec DUMP: Tue Oct 16 12:58:07 2012 : We have written 99181873 KB. 44GB@151MB/sec DUMP: dumping (Pass V) [ACLs] DUMP: 125867380 KB 120GB@155MB/sec DUMP: DUMP IS DONE Tue Oct 16 13:01:18 PDT 2012 - smtape backup to null takes 44 minutes FAS3210> rdfile /etc/log/backup (null) Tue Oct 16 13:05:32 PDT /vol/papayasg1_mirror CLI-Backup (null) 19 Tue Oct 16 13:05:32 PDT / vol/papayasg1_mirror BKP-Start (level 0 backup of Backup Set ID 2bc95ed2-195f-11e2-8766-00a09813e8e0) 19 Tue Oct 16 13:05:32 PDT /vol/papayasg1_mirror BKP-Params (originator=CLI mode=dw-data tape_record_size=240KB tape=null all_snapshots tape_seeding) 19 Tue Oct 16 13:05:33 PDT /vol/papayasg1_mirror BKP-DW-Start 19 Tue Oct 16 13:50:26 PDT /vol/papayasg1_mirror BKP-DW-End (phase completed in 0:44:53; 34394279 blocks moved) 19 Tue Oct 16 13:50:26 PDT /vol/papayasg1_mirror BKP-Tape-Stats (wait_time=0s wait_count=0 total_count=0 avail_count=0/0/0/0) 19 Tue Oct 16 13:50:26 PDT /vol/papayasg1_mirror BKP-End (backed up 137.577 GB bytes in 0:44:54; performance=183.844 GB/hour) -- Pete Kain 310-506-4475

2 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters October 2012