toasters March 2003

toasters@lists.teaparty.net

92 participants
102 discussions

RE: Snapmirror throughput question
by Hill, Aaron 07 Mar '03

07 Mar '03

At this stage I do not think this is a snapmirror issue. There are a lot of people out there successfully doing transfers over higher latency links than the 20Mbit I have. I was unintentionally misleading with the "ftp between filers" statement. The transfer was actually between 2 unix hosts that had directories mounted on the filers. So there are differences in the way the tcp traffic will be handled. It does establish that the network setup we currently have CAN be saturated. However, it does not establish that we have it setup optimally for the snapmirror we are trying to do. After talking with some very good NetApp engineers, we decided to do a more specific test limited to filer-filer communication by doing; 1) A snapmirror on the same network segment to an R100. This is underway now using exactly the same volumes as I have mentioned. 2) An NDMPcopy directly to the DR/target filer. This will need to wait unitl a "quiet" time on the WAN so I can hog it and not affect the already slow snapmirror. The results of these should be able to help us further define whether snapmirror is the issue or not. A tool suggested by another toasters member is pchar. I am very keen to put it on my network between sites and see what it spits out. Just don't feel like working all weekend!! Thanks, Aaron -----Original Message----- From: Todd C. Merrill [mailto:tmerrill@mathworks.com] Sent: Saturday, March 08, 2003 7:18 AM To: Hill, Aaron Subject: Re: Snapmirror throughput question On Thu, 6 Mar 2003, Hill, Aaron wrote: > 6) ftp transfers have been tested between filers and these saturate > the 20Mbit immediately How do you do this, ftp directly between filers? The fact that *this* filer-to-filer communication saturates the link seems to be quite damning to snapmirror... Until next time... The MathWorks, Inc. 508-647-7000 x7792 3 Apple Hill Drive, Natick, MA 01760-2098 508-647-7001 FAX tmerrill(a)mathworks.com http://www.mathworks.com ---

1 0

RE: Snapmirror throughput question
by Hill, Aaron 07 Mar '03

07 Mar '03

Hey, thanks for the info. Here is the network path; Source Filer GigE ----- GigE on 4006 Cisco switch 100/FastE ---- 100/FastE 3660 Router 100/FastE ==== 20M VLAN ==== 100/FastE 3600 Router 100/FastE ---- 100/FastE 2924 Cisco Switch 100/FastE ---- 100/FastE Target Filer The VLAN carrier, UECOMM tells us no traffic shaping on the VLAN given to us. Our network team has no traffic shaping on the 3660 nor the 3600. See anything else in there causing an issue? Aaron -----Original Message----- From: Michael van Elst [mailto:mlelstv@serpens.de] Sent: Friday, March 07, 2003 10:44 AM To: Hill, Aaron Subject: Re: Snapmirror throughput question On Thu, Mar 06, 2003 at 04:40:11PM -0600, Hill, Aaron wrote: > Hi people, > What are the maximum data throughputs that Snapmirroring people are seeing > across a WAN? Is anyone seeing 15-20+ Mbit bandwidth usage from a snapmirror > data stream? I had a setup with three F840 clusters mirroring to a single F840 cluster using a 100Mbps tunnel. During the initial full copy this saturated the tunnel, the subsequent incremental updates were on a much lower level (4-8Mbps), because there weren't that many changes to the data. When I paused the mirror for a while and then resumed it, the then much bigger update again saturated the tunnel. The filers all had GigE and for the tunnel a dedicated GigE each, so that snapmirror traffic didn't compete with regular accesses. The filers were connected to Catalyst6509s that had a 100TX link to the tunnel routers. I am a bit confused what the "dedicated 20Mbit Ethernet link" of yours is exactly. If that's some kind of allocated bandwidth or subject to traffic shaping, it could easily explain the bad utilization. Greetings, -- Michael van Elst Internet: mlelstv(a)serpens.de "A potential Snark may lurk in every tree."

3 2

RE: Snapmirror throughput question
by Borders, Rich 07 Mar '03

07 Mar '03

Hi guys, I am replying in-line ... > -----Original Message----- > From: Michael van Elst [mailto:mlelstv@serpens.de] > Sent: Thursday, March 06, 2003 8:48 PM > To: Hill, Aaron > Cc: 'toasters(a)mathworks.com' > > On Thu, Mar 06, 2003 at 06:31:18PM -0600, Hill, Aaron wrote: > > Hey, thanks for the info. > > > > Here is the network path; > > > > Source Filer GigE ----- GigE on 4006 Cisco switch 100/FastE ---- > > 100/FastE 3660 Router 100/FastE ==== 20M VLAN ==== 100/FastE 3600 > > Router 100/FastE > > ---- 100/FastE 2924 Cisco Switch 100/FastE ---- 100/FastE > Target Filer I see a pair of 3660's in the middle. Are they one of the first ones you had? They are fast and they are 'divisional' class routers. They can operate HSSI interfaces up to DS3 speeds effectively, but take a look at the processing of them. What is the cpu utilization. You are most likely using the FA0/0 and FA 0/1, but just asking. Are you doing any routing protocols, ect? I am digging so please bear with me. I am pointing out potential issues. Also make sure you have the full version IOS on these routers. When I got my first pair, I was doing DS3's to a web site, and I got a pre-release of the cisco IOS on there. Just something to check. We did upgrade, because of a bug. ( harware related ) Look at the couters on the int's for ANY errors. If you packets are seeing any issues, then you will see it there. Also, at 20 megabit, you will see about 1.6 megabytes MAX theoretical. Are you ever seeing that anywhere? FTP, makefiles ect.? Try to establish the baseline. I saw you say you got it from ftp, but that speed? Just curious. > > The VLAN carrier, UECOMM tells us no traffic shaping on the > VLAN given > > to us. I have seen AT&T do this here in the US. You can get good speeds, but it is burst. > > Our network team has no traffic shaping on the 3660 nor the 3600. > > > > See anything else in there causing an issue? Not off hand. Anything else happening on the WAN when you are doing this? Are you getting dedicate CIR rates from the carrier. Do you have any TSU's or DSU like terminaing devices. Check framing. I think that it is possible that the filer can POUND that link into a submissive state. That is not to say we shouldn't get max speeds, but snapmirror CAN be unrelenting. You need to try and harden the link, and packet shaping can help. The original request to turn on flow control is good if you have the option. This will aid the source and destination in any fragmentation or retrans type issues. Are you seeing any errors in the ifstat information. CRC, checksum errors, anything? > I would start looking at flow-control settings for the GigE > on the 4006 and also verify the MTU sizes on each interface > to rule out packet fragmentation. Sounds good. > But I still suspect the 20Mbps bandwidth limit to have a > significant effect. To make this more visible you could > enable traffic shaping on your routers with a slightly smaller limit. This will keep you from crippling the link. Snapmirror was designed with campus speeds in mind. The throttle came when wan and DR was introduced. If you can throttle to 1.7 megabit, the you will see a possible performance increase. Sort of like stopping a potential 'downward spiral'. > -- > Michael van Elst > Internet: mlelstv(a)serpens.de > "A potential Snark may lurk > in every tree." Rich B

1 0

RE: SnapMirror delays.
by Borders, Rich 07 Mar '03

07 Mar '03

Have you tried the command line version < snapmirror status -l > for extended output. It may be a little more helpful. It seems you are checking status, and I am looking a little more for the 'actual' data moving. Good SM Link: http://now.netapp.com/AskNOW/highlight_html.jsp?url=http%3A%2F%2Fnow.net app.com%2FNOW%2Fknowledge%2Fdocs%2Fontap%2Frel631r1%2Fhtml%2Fontap%2Fdpg %2Fmirror19.htm&sentenceId=60257568&titleIds=60257567%2C60257566%2C60257 562#1548676 The actual data moving is block level, so there really is not a true concept of 'files' If you have a DB in the snap destination that is moving only changed blocks IN the DB then you are not really moving a file. Just block updates IN the file. You may be aware of this, so maybe you could be more specific in your question. Also, just for more info, is this QSM, or volume level? Rich > -----Original Message----- > From: Ngoh, Clarence [mailto:clarence.ngoh@cba.com.au] > Sent: Thursday, March 06, 2003 5:04 PM > To: toasters(a)mathworks.com > > Hi folks > > I am having some difficulties tracking down a particular user > or qtree that is causing delays on our SnapMirror process. > The most relevant information of our setup are this: > > Filer head : FAS940 > OS: DOT 6.3.1 > Filername : Let's call it "FilerA" > OS to troubleshoot from : Windows, but open sourced solutions > welcomed. > > Filer A contains several different qtrees with different > business units. > Every so often, the snapshots are SnapMirrored to our DR > site. Of late, we noticed that FilerA is lagging behind the > DR site by a fair amount of time > -- 16 hours. The amount of data transferred for this period > is around 15 GB. > > Please do not offer solutions regarding network latency, as > the amount of data change is the cause of the lag. > > I would like to trace the particular user or qtree that is > making this changes. We are using DFM 2.1 for W2K server. > DFM did not register data deletions or additions for any > business unit -- I suspect then that data changes are the > culprits, but where is this coming from? > > > Cheers > Clarence. > > > ************** IMPORTANT MESSAGE ************** > This e-mail message is intended only for the addressee(s) and > contains information which may be confidential. If you are > not the intended recipient please advise the sender by return > email, do not use or disclose the contents, and delete the > message and any attachments from your system. Unless > specifically indicated, this email does not constitute formal > advice or commitment by the sender or the Commonwealth Bank > of Australia (ABN 48 123 123 124) or its subsidiaries. > ************************************************** > >

1 0

RE: Snapmirror throughput question
by Hill, Aaron 07 Mar '03

07 Mar '03

The VLAN is over a UECOMM Fibre Metropolitan Area Network. I really am not sure exactly what carrier equipment is on that loop. I am guessing fibre switches. As far as I can tell, all they do is give us a VLAN with bandwidth throttled to whatever we pay for. For example, for our initial transfers we have had them ramp it up to 100Mbit for 24 hours. Thanks, Aaron -----Original Message----- From: Mohler, Jeff [mailto:jeff.mohler@netapp.com] Sent: Friday, March 07, 2003 12:28 PM To: Hill, Aaron; 'Michael van Elst' Cc: 'toasters(a)mathworks.com' Subject: RE: Snapmirror throughput question This Vlan, is it delivered over ATM? -----Original Message----- From: Hill, Aaron [mailto:aaron.hill@navitaire.com] Sent: Thursday, March 06, 2003 4:31 PM To: 'Michael van Elst' Cc: 'toasters(a)mathworks.com' Subject: RE: Snapmirror throughput question Hey, thanks for the info. Here is the network path; Source Filer GigE ----- GigE on 4006 Cisco switch 100/FastE ---- 100/FastE 3660 Router 100/FastE ==== 20M VLAN ==== 100/FastE 3600 Router 100/FastE ---- 100/FastE 2924 Cisco Switch 100/FastE ---- 100/FastE Target Filer The VLAN carrier, UECOMM tells us no traffic shaping on the VLAN given to us. Our network team has no traffic shaping on the 3660 nor the 3600. See anything else in there causing an issue? Aaron -----Original Message----- From: Michael van Elst [mailto:mlelstv@serpens.de] Sent: Friday, March 07, 2003 10:44 AM To: Hill, Aaron Subject: Re: Snapmirror throughput question On Thu, Mar 06, 2003 at 04:40:11PM -0600, Hill, Aaron wrote: > Hi people, > What are the maximum data throughputs that Snapmirroring people are seeing > across a WAN? Is anyone seeing 15-20+ Mbit bandwidth usage from a snapmirror > data stream? I had a setup with three F840 clusters mirroring to a single F840 cluster using a 100Mbps tunnel. During the initial full copy this saturated the tunnel, the subsequent incremental updates were on a much lower level (4-8Mbps), because there weren't that many changes to the data. When I paused the mirror for a while and then resumed it, the then much bigger update again saturated the tunnel. The filers all had GigE and for the tunnel a dedicated GigE each, so that snapmirror traffic didn't compete with regular accesses. The filers were connected to Catalyst6509s that had a 100TX link to the tunnel routers. I am a bit confused what the "dedicated 20Mbit Ethernet link" of yours is exactly. If that's some kind of allocated bandwidth or subject to traffic shaping, it could easily explain the bad utilization. Greetings, -- Michael van Elst Internet: mlelstv(a)serpens.de "A potential Snark may lurk in every tree."

1 0

RE: Snapmirror throughput question
by Mohler, Jeff 07 Mar '03

07 Mar '03

This Vlan, is it delivered over ATM? -----Original Message----- From: Hill, Aaron [mailto:aaron.hill@navitaire.com <mailto:aaron.hill@navitaire.com> ] Sent: Thursday, March 06, 2003 4:31 PM To: 'Michael van Elst' Cc: 'toasters(a)mathworks.com' Subject: RE: Snapmirror throughput question Hey, thanks for the info. Here is the network path; Source Filer GigE ----- GigE on 4006 Cisco switch 100/FastE ---- 100/FastE 3660 Router 100/FastE ==== 20M VLAN ==== 100/FastE 3600 Router 100/FastE ---- 100/FastE 2924 Cisco Switch 100/FastE ---- 100/FastE Target Filer The VLAN carrier, UECOMM tells us no traffic shaping on the VLAN given to us. Our network team has no traffic shaping on the 3660 nor the 3600. See anything else in there causing an issue? Aaron -----Original Message----- From: Michael van Elst [mailto:mlelstv@serpens.de <mailto:mlelstv@serpens.de> ] Sent: Friday, March 07, 2003 10:44 AM To: Hill, Aaron Subject: Re: Snapmirror throughput question On Thu, Mar 06, 2003 at 04:40:11PM -0600, Hill, Aaron wrote: > Hi people, > What are the maximum data throughputs that Snapmirroring people are seeing > across a WAN? Is anyone seeing 15-20+ Mbit bandwidth usage from a snapmirror > data stream? I had a setup with three F840 clusters mirroring to a single F840 cluster using a 100Mbps tunnel. During the initial full copy this saturated the tunnel, the subsequent incremental updates were on a much lower level (4-8Mbps), because there weren't that many changes to the data. When I paused the mirror for a while and then resumed it, the then much bigger update again saturated the tunnel. The filers all had GigE and for the tunnel a dedicated GigE each, so that snapmirror traffic didn't compete with regular accesses. The filers were connected to Catalyst6509s that had a 100TX link to the tunnel routers. I am a bit confused what the "dedicated 20Mbit Ethernet link" of yours is exactly. If that's some kind of allocated bandwidth or subject to traffic shaping, it could easily explain the bad utilization. Greetings, -- Michael van Elst Internet: mlelstv(a)serpens.de "A potential Snark may lurk in every tree."

1 0

RE: A humble plea from a novice.
by Ngoh, Clarence 06 Mar '03

06 Mar '03

John Do you have an account with now.netapp.com? It provides a wealth of resources online and this includes manuals that come with the toasters :) You may find the following manuals useful: 1. The system administrator's guide 2. Data Protection Guide 3. Command Reference. This should get you started with the basic commands and setups. Regards, Clarence Ngoh Institutional Banking Services, Technology Office Level 5, 48 Martin Place Sydney, 2000 (o): +612 9378 5967 | (f): 02 9378 5202 | (e):clarence.ngoh@cba.com.au > -----Original Message----- > From: John Enger [mailto:john@johnenger.com] > Sent: Friday, 7 March 2003 09:38 AM > To: toasters(a)mathworks.com > Subject: A humble plea from a novice. > > > I have just gotten hold of an old 2nd hand NetApp F330. > I got it because I'm curious and l love to learn new things > hands on. Most of my skills have been picked up by troubling > and searching for info and so forth, but when it comes to the > NetApp, things are very different. > > What I need is a good source of info on how it works, a prom > monitor command map or similar. Where/how the OS is stored, > very common info. I have been searching the net for this, but > found nothing so far. > > I found you guys, but your discussions are way too advanced > for me to participate in or to understand much of. At least > until I get my own up and running... :o) > > So does anyone of you have a tip on where such info might be > available? If it's at all available? > > > Thanks! > > John > > ************** IMPORTANT MESSAGE ************** This e-mail message is intended only for the addressee(s) and contains information which may be confidential. If you are not the intended recipient please advise the sender by return email, do not use or disclose the contents, and delete the message and any attachments from your system. Unless specifically indicated, this email does not constitute formal advice or commitment by the sender or the Commonwealth Bank of Australia (ABN 48 123 123 124) or its subsidiaries. **************************************************

1 0

Snapmirror throughput question
by Hill, Aaron 06 Mar '03

06 Mar '03

Hi people, What are the maximum data throughputs that Snapmirroring people are seeing across a WAN? Is anyone seeing 15-20+ Mbit bandwidth usage from a snapmirror data stream? The reason I ask is because I have a dedicated 20Mbit Ethernet link between 2 filers and am only seeing a maximum of 6-8Mbit of that available 20Mbit being used by snapmirror. Details; 1) No CPU or disk bottlenecking on source or target 2) Source is F880 target F820 3) 3 snapmirror streams at volume level 4) Approximately 30-60 Gb of data per volume is snapmirrored per day. 5) 20Mbit WAN is about 95% dedicated to Snapmirror. The rest is small ftp, http traffic etc. 6) ftp transfers have been tested between filers and these saturate the 20Mbit immediately 7) snapmirror.conf is set to transfer at 6am each day. Current kbs=10000. i.e. 1 transfer per volume per day. This is for Disaster recovery synchronization requirements. 8) No packets are being dropped on the switch interface. 9) Single 1000Mbit Interface on source filer, 100Mbit interface on target. 10) Neither interface shows saturation. 11) Unix clients only at both source and target filers. 98%+ would be NFS traffic. Now, NetApp support informs me that even if I had a dedicated VLAN between filer interfaces across the 20Mbit, I cannot expect to see full utilization. I do not believe this. Someone out there must be transferring large amounts of data across a WAN and have seen higher throughput. If you are seeing higher throughput rates, what type of setup do you have? Things I am currently considering; - multiple qtrees snapmirrored. i.e. more snapmirror streams results in more throughput - multiple snapmirror transfers per day - dedicated VLAN between source and target Any expert advice here would be appreciated. Thanks, Aaron www.navitaire.com

1 0

A humble plea from a novice.
by John Enger 06 Mar '03

06 Mar '03

I have just gotten hold of an old 2nd hand NetApp F330. I got it because I'm curious and l love to learn new things hands on. Most of my skills have been picked up by troubling and searching for info and so forth, but when it comes to the NetApp, things are very different. What I need is a good source of info on how it works, a prom monitor command map or similar. Where/how the OS is stored, very common info. I have been searching the net for this, but found nothing so far. I found you guys, but your discussions are way too advanced for me to participate in or to understand much of. At least until I get my own up and running... :o) So does anyone of you have a tip on where such info might be available? If it's at all available? Thanks! John

1 0

SnapMirror delays.
by Ngoh, Clarence 06 Mar '03

06 Mar '03

Hi folks I am having some difficulties tracking down a particular user or qtree that is causing delays on our SnapMirror process. The most relevant information of our setup are this: Filer head : FAS940 OS: DOT 6.3.1 Filername : Let's call it "FilerA" OS to troubleshoot from : Windows, but open sourced solutions welcomed. Filer A contains several different qtrees with different business units. Every so often, the snapshots are SnapMirrored to our DR site. Of late, we noticed that FilerA is lagging behind the DR site by a fair amount of time -- 16 hours. The amount of data transferred for this period is around 15 GB. Please do not offer solutions regarding network latency, as the amount of data change is the cause of the lag. I would like to trace the particular user or qtree that is making this changes. We are using DFM 2.1 for W2K server. DFM did not register data deletions or additions for any business unit -- I suspect then that data changes are the culprits, but where is this coming from? Cheers Clarence. ************** IMPORTANT MESSAGE ************** This e-mail message is intended only for the addressee(s) and contains information which may be confidential. If you are not the intended recipient please advise the sender by return email, do not use or disclose the contents, and delete the message and any attachments from your system. Unless specifically indicated, this email does not constitute formal advice or commitment by the sender or the Commonwealth Bank of Australia (ABN 48 123 123 124) or its subsidiaries. **************************************************

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters March 2003