toasters November 2005

toasters@lists.teaparty.net

59 participants
55 discussions

RE: Block-Level Backup of Filer Volumes
by Holland, William L 15 Nov '05

15 Nov '05

We use Backup Express and it works fine. We SnapVault from our main filers to an R200 then NDMP backup to tape with Backup Express. -----Original Message----- From: owner-toasters(a)mathworks.com [mailto:Jason.Palmer@drkw.com] Sent: Tuesday, November 15, 2005 8:29 AM To: 'Ben Rockwood'; toasters(a)mathworks.com Subject: RE: Block-Level Backup of Filer Volumes Netvault (Backbone) - Tested SMT sucessfully Backup Express (Syncsort) - Not tested by me yet but awaiting to eval These allow the backup of the volume using Snapmirror to tape (SMT) Back at the time I tested NetVault it was pretty basic, but I got the volume + block level snapshots sucessfully off to a Virtual Backup Library (Another nice feature) and then back from the virtual tape library onto another Filers volume intact. Not sure is either of these products have enhanced their block level capabilities since I looked at them last year. I would also be interested to see what others say and come back with Jason -----Original Message----- From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Ben Rockwood Sent: 15 November 2005 08:14 To: toasters(a)mathworks.com Subject: Block-Level Backup of Filer Volumes I'm looking into various methods for tape backup of our filers. Due to the large number of files a file level backup would be significantly slower than a block-level backup method, however I'm not sure if one exists. I was hoping someone could clarify. NDMP utilizes dump, which is file-based so that doesn't solve the performance issues, and from what I can tell SnapVault is only for disk-to-disk backup, not tape. Even if I direct connect a tape robot to the filer I'd still be using dump and the throughput isn't being limited by the network capacity, so that doesn't help either. I'm hoping there is a block-level method I'm unaware of. NDMP in my tests has been dog slow and standard backups via NFS mount don't sound appealing. Suggestions are welcome. benr. ---------------------------------------------------------------------------- ---- The information contained herein is confidential and is intended solely for the addressee. Access by any other party is unauthorised without the express written permission of the sender. If you are not the intended recipient, please contact the sender either via the company switchboard on +44 (0)20 7623 8000, or via e-mail return. If you have received this e-mail in error or wish to read our e-mail disclaimer statement and monitoring policy, please refer to http://www.drkw.com/disc/email/ or contact the sender. 3167 ---------------------------------------------------------------------------- ----

1 0

RE: Block-Level Backup of Filer Volumes
by Palmer, Jason 15 Nov '05

15 Nov '05

Netvault (Backbone) - Tested SMT sucessfully Backup Express (Syncsort) - Not tested by me yet but awaiting to eval These allow the backup of the volume using Snapmirror to tape (SMT) Back at the time I tested NetVault it was pretty basic, but I got the volume + block level snapshots sucessfully off to a Virtual Backup Library (Another nice feature) and then back from the virtual tape library onto another Filers volume intact. Not sure is either of these products have enhanced their block level capabilities since I looked at them last year. I would also be interested to see what others say and come back with Jason -----Original Message----- From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Ben Rockwood Sent: 15 November 2005 08:14 To: toasters(a)mathworks.com Subject: Block-Level Backup of Filer Volumes I'm looking into various methods for tape backup of our filers. Due to the large number of files a file level backup would be significantly slower than a block-level backup method, however I'm not sure if one exists. I was hoping someone could clarify. NDMP utilizes dump, which is file-based so that doesn't solve the performance issues, and from what I can tell SnapVault is only for disk-to-disk backup, not tape. Even if I direct connect a tape robot to the filer I'd still be using dump and the throughput isn't being limited by the network capacity, so that doesn't help either. I'm hoping there is a block-level method I'm unaware of. NDMP in my tests has been dog slow and standard backups via NFS mount don't sound appealing. Suggestions are welcome. benr. -------------------------------------------------------------------------------- The information contained herein is confidential and is intended solely for the addressee. Access by any other party is unauthorised without the express written permission of the sender. If you are not the intended recipient, please contact the sender either via the company switchboard on +44 (0)20 7623 8000, or via e-mail return. If you have received this e-mail in error or wish to read our e-mail disclaimer statement and monitoring policy, please refer to http://www.drkw.com/disc/email/ or contact the sender. 3167 --------------------------------------------------------------------------------

1 0

Cat5/6/Coax/Optical test gear
by Jeff Mohler 11 Nov '05

11 Nov '05

What do Y'all here have as recommendations for a handheld test set to certify the listed types of signal paths? Theres so many..thought Id ask the list. Thanks!

1 0

Re: howto document for initial volume architecting?
by Chris Thompson 11 Nov '05

11 Nov '05

Jeff Mery writes: > > Here's how to do the tracking-only trees (from our active quota file). > Enabling it causes no problems. We actually turn on certain tracking > quotas in the middle of the day from time to time. > > #Qtree Quotas > #--------------------------------------------------------------------------------------------- > #Target Type Disk Files Thold Sdisk > Sfile Comment > #--------------------------------------------------------------------------------------------- > > <------------snip > /vol/perforce/ActiveXContainer tree - > /vol/perforce/ASIC tree - > /vol/perforce/ATC tree - > /vol/perforce/CodeMart tree - > /vol/perforce/ComponentWorks tree - > /vol/perforce/CVI tree - > /vol/perforce/DAQ tree - > /vol/perforce/DAQConfig tree - > /vol/perforce/DevSuite tree - > <------------snip Alternatively, #Target Type Disk Files * tree@/vol/perforce - will make a default tracking-but-unlimited quota for every qtree on the volume not given its own quota. Apart from not having to list the qtrees explictly, this has the advantage of also applying to any qtrees created after the "quota on". -- Chris Thompson Email: cet1(a)cam.ac.uk

2 1

RE: howto document for initial volume architecting?
by Lai, Derek 10 Nov '05

10 Nov '05

Lori, Sounds like a very interesting day. Why is Samba even needed now? Derek -----Original Message----- From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Lori Barfield Sent: Thursday, November 10, 2005 8:29 AM To: John Stoffel; Greg.Libert(a)plexus.com; toasters(a)mathworks.com Subject: Re: howto document for initial volume architecting? i'm so grateful for the help yesterday! just for fun, i will share the day i had back with you all. i hope i get the chance to return the assistance some day. thanks to the two best netapp salesmen on earth ;) i spent the morning yesterday printing pdfs and planning data migrations to ontap 7 and flexvols. the flexvol sizing calculation i am especially grateful for- it isn't obvious and i would certainly have guessed wrong. armed with a clear direction about where my day was going, i marched into the office...and right into a user-impacting outage due to a full volume. up until yesterday i had never logged into any of the filers and knew nothing about them. so in the midst of the firefight, i had several revelations about the configuration. as for the full volume, i found that snapshots were being kept for only one week and used 17gb. but there was 180gb of snap reserve. so i cut the snap reserve in half, quieting the tumult, and explained to the boss that ten weeks of snap reserve is awfully nice to have, and if you're going to set that much space aside, you might want to keep a little more backup data around. :) about the configurations, dozens of qtrees exist, many duplicated from filer to filer, but some very tiny, as if perhaps there was going to be mirroring at one time, but instead over the years the second qtree got used just a little on the second filer. none of the qtrees is running quotas and they all have the default unix permissions, so the implementation was never exploited. some of the directories at /vol/vol1/ aren't in qtrees, so consistency suffered over the years as well. however, the computer gods truely love me...the boss already upgraded the filers to ontap 7! each filer has two flexvols, one for system and one for data. so, to plan the rearchitecting and data migration project, i spent the rest of the day making a spreadsheet describing the existing data directories and filling in nfs ops statistics for all the qtrees, the one real blessing of that feature in this setup. because of the single volume for everything i don't have snap statistics for each tree, but i created columns for that in my spreadsheet and we'll iterate on the configuration as stats come in later. stupidly, i counted the space each data directory is taking with du -sk, so i got the snaps as well and all the numbers are off. i guess today i'm going to be adding up directory sizes with a massive find -prune loop. (or is there a better way?) at the end of the day, we met to discuss the spreadsheet, and decided the intuitive thing to do was create a new data architecture that mirrored the corporate org chart. this will facilitate setting up samba file sharing that makes sense. right now filer space is all mounted at / but in the future each top-level organization will have its own top-level directory on the sun servers for the mount points for the flexvol/qtrees it needs. with samba, right now folders for multiperson efforts are kind of shared all over the place whereever users happen to be working. going forward, for consistency each team will have a group-shareable folder for all the team members (whether it's needed now or not). and potentially, each major project will have a shareable folder that will span teams, not individuals. but project folders will all belong in a hierarchy owned by a project lead or someone, so the data can be maintained and removed later when it's no longer needed. i plan to add the flexvol space calculation to the spreadsheet today so it can be used as a tool to redistribute space on the fly during architecture discussions. thank you all again. :) ...lori On 11/9/05, Lori Barfield <itdirector(a)gmail.com> wrote: > thank you, greg and john, for the recommendations and the pdf links. > i was not aware of FlexVols and it looks like this is suddenly going > to be a much more challenging exercise. :)

1 0

RE: snapmirroring over WAN
by Henry.Chan＠Emulex.Com 10 Nov '05

10 Nov '05

Unfortunately just SnapMirror. SnapVault has no option to create tape baseline. -----Original Message----- From: John Stoffel [mailto:john.stoffel@taec.toshiba.com] Sent: Thu 11/10/2005 11:16 AM To: Chan, Henry Cc: dleeds(a)edmunds.com; bparent(a)calvin.ucsd.edu; toasters(a)mathworks.com Subject: RE: snapmirroring over WAN Henry> We also had to back up our volumes to a remote site via Henry> snapmirror accross a T1 WAN. One of the vols was over 1TB and Henry> it would have taken days to create the initial baseline over Henry> the T1 so we snapped to SDLT tape, sent it down to CA and Henry> created the baseline in a "snap"! Now our nightly VSMs take Henry> about 2 hours for 5 gigs (our average delta size). Can you do this with SnapVaults, or just snapmirrors? This would be a great way to get an initial snapvault setup for my remote sites, since they are on fast links, but far away and the TCP bandwidth delay is killing my performance. John

2 1

Re: howto document for initial volume architecting?
by Lori Barfield 10 Nov '05

10 Nov '05

i'm so grateful for the help yesterday! just for fun, i will share the day i had back with you all. i hope i get the chance to return the assistance some day. thanks to the two best netapp salesmen on earth ;) i spent the morning yesterday printing pdfs and planning data migrations to ontap 7 and flexvols. the flexvol sizing calculation i am especially grateful for- it isn't obvious and i would certainly have guessed wrong. armed with a clear direction about where my day was going, i marched into the office...and right into a user-impacting outage due to a full volume. up until yesterday i had never logged into any of the filers and knew nothing about them. so in the midst of the firefight, i had several revelations about the configuration. as for the full volume, i found that snapshots were being kept for only one week and used 17gb. but there was 180gb of snap reserve. so i cut the snap reserve in half, quieting the tumult, and explained to the boss that ten weeks of snap reserve is awfully nice to have, and if you're going to set that much space aside, you might want to keep a little more backup data around. :) about the configurations, dozens of qtrees exist, many duplicated from filer to filer, but some very tiny, as if perhaps there was going to be mirroring at one time, but instead over the years the second qtree got used just a little on the second filer. none of the qtrees is running quotas and they all have the default unix permissions, so the implementation was never exploited. some of the directories at /vol/vol1/ aren't in qtrees, so consistency suffered over the years as well. however, the computer gods truely love me...the boss already upgraded the filers to ontap 7! each filer has two flexvols, one for system and one for data. so, to plan the rearchitecting and data migration project, i spent the rest of the day making a spreadsheet describing the existing data directories and filling in nfs ops statistics for all the qtrees, the one real blessing of that feature in this setup. because of the single volume for everything i don't have snap statistics for each tree, but i created columns for that in my spreadsheet and we'll iterate on the configuration as stats come in later. stupidly, i counted the space each data directory is taking with du -sk, so i got the snaps as well and all the numbers are off. i guess today i'm going to be adding up directory sizes with a massive find -prune loop. (or is there a better way?) at the end of the day, we met to discuss the spreadsheet, and decided the intuitive thing to do was create a new data architecture that mirrored the corporate org chart. this will facilitate setting up samba file sharing that makes sense. right now filer space is all mounted at / but in the future each top-level organization will have its own top-level directory on the sun servers for the mount points for the flexvol/qtrees it needs. with samba, right now folders for multiperson efforts are kind of shared all over the place whereever users happen to be working. going forward, for consistency each team will have a group-shareable folder for all the team members (whether it's needed now or not). and potentially, each major project will have a shareable folder that will span teams, not individuals. but project folders will all belong in a hierarchy owned by a project lead or someone, so the data can be maintained and removed later when it's no longer needed. i plan to add the flexvol space calculation to the spreadsheet today so it can be used as a tool to redistribute space on the fly during architecture discussions. thank you all again. :) ...lori On 11/9/05, Lori Barfield <itdirector(a)gmail.com> wrote: > thank you, greg and john, for the recommendations and the pdf links. > i was not aware of FlexVols and it looks like this is suddenly going > to be a much more challenging exercise. :)

6 7

RE: howto document for initial volume architecting?
by Brian Pauley 10 Nov '05

10 Nov '05

Alternatively, you can use a "du"-like script which avoids traversing the ".snapshot" directory: #!/bin/perl # # Usage mydu [dir] # opendir (DIR, $ARGV[0]) || die "Can't open $ARGV[0] $!"; @dirs = readdir(DIR); closedir (DIR); foreach $d (@dirs) { next if ($d eq "." || $d eq ".." || $d eq ".snapshot"); system ("du -s $ARGV[0]/$d"); } Brian Pauley Systems Engineer Dataline, Inc 2551 Eltham Avenue, Suite O Norfolk, VA 23513 Phone (757)858-0613 ext. 264 Fax (757)858-0606 email: brian.pauley(a)dataline.com -----Original Message----- From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Andrew Siegel Sent: Thursday, November 10, 2005 2:13 PM To: Lori Barfield Cc: toasters(a)mathworks.com Subject: Re: howto document for initial volume architecting? Lori Barfield wrote: > > stupidly, i > counted the space each data directory is taking with du -sk, so i got > the snaps as well and all the numbers are off. i guess today i'm > going to be adding up directory sizes with a massive find -prune loop. > (or is there a better way?) One trick for doing du's without encountering any .snapshot directories is to do the du within a snapshot.

1 0

RE: snapmirroring over WAN
by Henry.Chan＠Emulex.Com 10 Nov '05

10 Nov '05

We also had to back up our volumes to a remote site via snapmirror accross a T1 WAN. One of the vols was over 1TB and it would have taken days to create the initial baseline over the T1 so we snapped to SDLT tape, sent it down to CA and created the baseline in a "snap"! Now our nightly VSMs take about 2 hours for 5 gigs (our average delta size). Henry -----Original Message----- From: owner-toasters(a)mathworks.com on behalf of Leeds, Daniel Sent: Thu 11/10/2005 9:21 AM To: Brian Parent; toasters(a)mathworks.com Cc: Subject: RE: snapmirroring over WAN yeah but that initial transfer can be fun if you are mirroring a couple terabyte volumes or if the application is generating alot of changes. a DS3 wan link would be no problem but they may need to rate limit snapmirror on much smaller wan links. -----Original Message----- From: owner-toasters(a)mathworks.com on behalf of Brian Parent Sent: Thu 11/10/2005 9:08 AM To: toasters(a)mathworks.com Subject: Re: snapmirroring over WAN If I recall correctly, snapmirror uses very little bandwidth, and shouldn't be a problem on any modern network. It only copies the blocks which were changed, rather than entire files. Re: > Date: Thu, 10 Nov 2005 08:49:38 -0800 > From: Lori Barfield <itdirector(a)gmail.com> > To: toasters(a)mathworks.com > Subject: snapmirroring over WAN > > for offsite backups, we hope to snapmirror our new fas 3020 on a > regular schedule via vpn over the internet to an older filer at our colo. > obviously, intermittent latency is a concern since we don't have a > dedicated link. and so is the potential for overpowering the modest > firewall/network hardware currently in place. does anyone have a > suggestion for how we might throttle the snapmirrors to improve > reliability in the transmission? > > and how are you folks handling encryption for remote mirroring? > we have a small pix, sonicwall, or cisco to choose from...i'm > guessing we may have to try all three but i hope to pick the one > most likely to succeed the first time. > > > ...lori

2 1

snapmirroring over WAN
by Lori Barfield 10 Nov '05

10 Nov '05

for offsite backups, we hope to snapmirror our new fas 3020 on a regular schedule via vpn over the internet to an older filer at our colo. obviously, intermittent latency is a concern since we don't have a dedicated link. and so is the potential for overpowering the modest firewall/network hardware currently in place. does anyone have a suggestion for how we might throttle the snapmirrors to improve reliability in the transmission? and how are you folks handling encryption for remote mirroring? we have a small pix, sonicwall, or cisco to choose from...i'm guessing we may have to try all three but i hope to pick the one most likely to succeed the first time. ...lori

5 5

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters November 2005