Hill, Aaron wrote:

Hey, thanks for the info.

Here is the network path;

Source Filer GigE ----- GigE on 4006 Cisco switch 100/FastE ---- 100/FastE 3660 Router 100/FastE ==== 20M VLAN ==== 100/FastE 3600 Router 100/FastE ---- 100/FastE 2924 Cisco Switch 100/FastE ---- 100/FastE Target Filer

The VLAN carrier, UECOMM tells us no traffic shaping on the VLAN given to us.

Our network team has no traffic shaping on the 3660 nor the 3600.

See anything else in there causing an issue?

Aaron

-----Original Message----- From: Michael van Elst [mailto:mlelstv@serpens.de] Sent: Friday, March 07, 2003 10:44 AM To: Hill, Aaron Subject: Re: Snapmirror throughput question

On Thu, Mar 06, 2003 at 04:40:11PM -0600, Hill, Aaron wrote:

...

Hi people, What are the maximum data throughputs that Snapmirroring people are

seeing

...

across a WAN? Is anyone seeing 15-20+ Mbit bandwidth usage from a

snapmirror

...

data stream?

I had a setup with three F840 clusters mirroring to a single F840 cluster using a 100Mbps tunnel. During the initial full copy this saturated the tunnel, the subsequent incremental updates were on a much lower level (4-8Mbps), because there weren't that many changes to the data. When I paused the mirror for a while and then resumed it, the then much bigger update again saturated the tunnel.

The filers all had GigE and for the tunnel a dedicated GigE each, so that snapmirror traffic didn't compete with regular accesses. The filers were connected to Catalyst6509s that had a 100TX link to the tunnel routers.

I am a bit confused what the "dedicated 20Mbit Ethernet link" of yours is exactly. If that's some kind of allocated bandwidth or subject to traffic shaping, it could easily explain the bad utilization.

Greetings,

if you want to test your network path deep inside, i would sugest you * pchar * u could find it on the net (google) for free : http://www.employees.org/~bmah/Software/pchar/ http://www.employees.org/~bmah/Talks/pchar-NGI-99-Slides.pdf this tool is handy but it takes a while to complete (as you pass throught a wan, start it in the morning and check it in the afternoon; also modify default options like -R (Repetitions per hop))

for info : a sample output (no more than 15 hops can be treated) << # pchar -v filer85 pchar to filer85 (192.168.10.234) using UDP/IPv4 Using raw socket input Packet size increments from 32 to 1500 by 32 46 test(s) per repetition 32 repetition(s) per hop 0: 172.31.240.253 (bsun18) Partial loss: 0 / 1472 (0%) Partial char: rtt = 1.388522 ms, (b = 0.000133 ms/B), r2 = 0.866755 stddev rtt = 0.006350, stddev b = 0.000008 Partial queueing: avg = 0.000134 ms (1010 bytes) Hop char: rtt = 1.388522 ms, bw = 60166.975881 Kbps Hop queueing: avg = 0.000134 ms (1010 bytes) 1: 172.31.250.253 (172.31.250.253) Partial loss: 0 / 1472 (0%) Partial char: rtt = 0.217381 ms, (b = 0.000067 ms/B), r2 = 0.902658 stddev rtt = 0.002676, stddev b = 0.000003 Partial queueing: avg = 0.000060 ms (1010 bytes) Hop char: rtt = --.--- ms, bw = --.--- Kbps Hop queueing: avg = -0.000074 ms (0 bytes) 2: 192.168.10.234 (filer85) Path length: 2 hops Path char: rtt = 0.217381 ms r2 = 0.902658 Path bottleneck: 60166.975881 Kbps Path pipe: 1634 bytes Path queueing: average = 0.000060 ms (1010 bytes) Start time: Fri Nov 22 12:04:33 2002 End time: Fri Nov 22 12:17:20 2002 #

...

u would check Path bottleneck at the summary output as you can see Start time and End time : pchar needs 13 minutes to compute for 2 hops

and the usage : << genghis:~# pchar Usage: pchar [-a analysis] [-b burst] [-c] [-d debuglevel] [-g gap] [-G gaptype] [-h] [-H hops] [-I increment] [-m mtu] [-n] [-p protocol] [-P port] [-q] [-R reps] [-s hop] [-S] [-t timeout] [-T tos] [-v] [-V] [-w file] -r file | host] -a analysis Set analysis type (default is lsq) lsq Least sum of squares linear fit kendall Linear fit using Kendall's test statistic lms Least median of squares linear fit lmsint Least median of squares linear fit (integer computations) -b Burst size (default = 1) -c Ignore route changes -d debuglevel Set debugging output level -g gap Inter-test gap in seconds (default = 0.25) -G gaptype Inter-test gap type (default is fixed) fixed Fixed gap exp Exponentially distributed random -H hops Maximum number of hops (default = 30) -h Print this help information -I increment Packet size increment (default = 32) -l host Set origin address of probes (defaults to hostname) -m mtu Maximum packet size to check (default = 1500) -M mode Operational mode (defaults to pchar) pchar Path characterization trout Tiny traceroute -n Don't resolve addresses to hostnames -p protocol Network protocol (default is ipv4udp) ipv4udp UDP over IPv4 ipv4raw UDP over IPv4 (raw sockets) ipv4icmp ICMP over IPv4 (raw sockets) ipv6icmp ICMPv6 over IPv6 (raw sockets) ipv6tcp TCP over IPv6 ipv6udp UDP over IPv6 -P port Starting port number (default = 32768) -q Quiet output -r file Read data from a file (- for stdin) -R reps Repetitions per hop (default = 32) -s hop Starting hop number (default = 1) -S Do SNMP queries per-hop -t timeout ICMP timeout in seconds (default = 3) -T tos Set IP type-of-service field (default = 0) -v Verbose output -V Print version information -w file Write data to a file (- for stdout)

...

and last : don't worry about some rtts or bws being displayed as * --.--- * : <<

If memory serves me right, Support FPS wrote:

Hi Stephane--

Sorry for the delay...I was out of town and unable to read my email for a few days.

...

Second : bellow is a pchar output, i would like to know : a - is it normal that we don't have some information (rtt displayed as --,---) in the 2nd hop b - is it normal that in the summary, Path char rtt is 0.222667 ( the last rtt ) but not 1.330503 + 0.222667 ( the sum of rtt ) ? another way to say it : isn't strange to see overall rtt being recorded as the smallest rtt gotten ?

Comments in-line:

...

here is the command and output : << # ./pchar -v filer85 pchar to filer85 (192.168.10.234) using UDP/IPv4 Using raw socket input Packet size increments from 32 to 1500 by 32 46 test(s) per repetition 32 repetition(s) per hop 0: 172.31.240.253 (bsun18) Partial loss: 0 / 1472 (0%) Partial char: rtt = 1.330503 ms, (b = 0.000154 ms/B), r2 = 0.909658 stddev rtt = 0.005918, stddev b = 0.000007 Partial queueing: avg = 0.000119 ms (773 bytes) Hop char: rtt = 1.330503 ms, bw = 51893.838057 Kbps Hop queueing: avg = 0.000119 ms (773 bytes) 1: 172.31.250.253 (172.31.250.253) Partial loss: 0 / 1472 (0%) Partial char: rtt = 0.222667 ms, (b = 0.000067 ms/B), r2 = 0.904762 stddev rtt = 0.002636, stddev b = 0.000003 Partial queueing: avg = 0.000102 ms (773 bytes) Hop char: rtt = --.--- ms, bw = --.--- Kbps Hop queueing: avg = -0.000017 ms (0 bytes) 2: 192.168.10.234 (filer85)

A funny thing happened here in that the RTT along two hops was computed to be larger than the RTT for just the first hop by itself. (In other words, 0.222667 > 1.330503.) A common cause for this is that the router at hop #1 (172.31.250.253) takes longer to generate an ICMP time-exceeded message than it does to forward a packet.

In the normal case, we compute the RTT for the second hop by taking the RTT for the first two hops and subtracting the RTT for the first hop alone. If we tried that here, we'd get a negative number (0.222667 - 1.330503 = -1.107836). Clearly this makes no sense. At one point, pchar used to print the negative RTT, but I got too many emails asking what a negative RTT meant. So I changed it to just print the "--.---" string instead.

...

Path length:       2 hops
Path char:         rtt = 0.222667 ms r2 = 0.904762
Path bottleneck:   51893.838057 Kbps
Path pipe:         1444 bytes
Path queueing:     average = 0.000102 ms (773 bytes)
Start time:        Tue Nov 19 16:53:51 2002
End time:          Tue Nov 19 17:06:36 2002

...

...

The RTT for the entire path is equal to the partial RTT for two hops, because this is a two-hop path. The fact that this is smaller than the RTT for the first hop doesn't really matter (again, remember that many routers can forward packets faster than they can generate ICMP messages).

In fact, the path RTT is always going to be equal to the partial path RTT from the source to the last hop (except for some pathological cases where we cannot compute this value).

Hope this helps!

Bruce.

...