Bill Holland pointed me to this link which might be of use to you
http://now.netapp.com/NOW/knowledge/docs/ontap/rel724/html/ontap/onlinebk/4m...
In my case I've staggered the mirror several hours apart so they shouldn't kick off simultaneously - I was actually reasonably suprised (I guess I shouldn't have been) that there was a limit at all.
The other thread mentioned running a wafl_iron type command to check the source - is there anything else on the source that could affect establishing a new mirror ? Old snaps ? Old mirrors ? Snap schedules etc ?
Don't suppose anyone has a definitive way of re-establishing a mirror over a suspect connection (surely if I throttle the bandwidth it should just take its time to establish a baseline) ?
Cheers, Raj.
On Mon, May 5, 2008 at 7:11 AM, Mike Partyka mpartyka@acmn.com wrote:
Yeah, I was thinking the same thing, a packet trace but I am waiting for support to come to the same conclusion. After the upgrade yesterday morning I decided I was stumped and opened a ticket this morning. They are currently looking into the problem. Hopefully I'll hear back today sometime and I will share what the list what the eventual resolution is.
Regards
Mike
From: Kenneth Heal [mailto:kheal@hotmail.com] Sent: Sunday, May 04, 2008 2:07 PM
To: Mike Partyka; tmacmd@gmail.com; owner-toasters@mathworks.com; Raj Patel; NetApp Toasters List Subject: RE: Oddball SnapMirror issue - Status: Pending with restart checkpoint
Hi Mike,
Thx for the quick reply. That does indeed shoot my theory/hope out the water. And I am inclined to agree that going lower on the window size is not likely to help, especially as both your boxes are in the same datacentre without any nasty firewalls or WAN links in between them. This is also the window size recommended in the kb for such problems.
At this I would be inclined to take a packet trace, fire off ASUPs, open a support case and upload a gzipped copy of the pktt trace. Have to give myself beat on this one... though I would be keen to know what the eventual resolution is.
cheers, Kenneth
https://now.netapp.com/Knowledgebase/solutionarea.asp?id=kb17202 ________________________________
Subject: RE: Oddball SnapMirror issue - Status: Pending with restart checkpoint Date: Sun, 4 May 2008 13:56:45 -0500 From: mpartyka@acmn.com To: kheal@hotmail.com; tmacmd@gmail.com; owner-toasters@mathworks.com; phigmov@gmail.com; toasters@mathworks.com
After failing to get the initialization going on the 270 and 3050 (running 7.0.5 and 7.0.6 respectively) yesterday morning we upgraded both the filers (src and dst) to 7.2.4. I immediately after tried the mirror again but no dice the error occurs around the same place/time in the initialization.
I did miss the following error in the /etc/messages file:
Sat May 3 11:51:23 CDT [worker_thread_98:notice]: snapmirror: Message from Read Socket : Connection
Sat May 3 11:51:23 CDT [snapmirror.dst.err:error]: SnapMirror destination transfer from 10.0.10.238data : snapmirror transfer failed to complete.
Sat May 3 11:51:24 CDT [snapmirror.dst.err:error]: SnapMirror destination transfer from 10.0.10.238data : snapmirror transfer failed to complete.
I understand this might mean the snapmirror.window_size is too large but it's set 32768 which is pretty small already. Usually you increase this value to increase performance but I don't think I want to go much smaller than this.
From: Kenneth Heal [mailto:kheal@hotmail.com] Sent: Sunday, May 04, 2008 1:48 PM To: Mike Partyka; tmacmd@gmail.com; owner-toasters@mathworks.com; Raj Patel; NetApp Toasters List Subject: RE: Oddball SnapMirror issue - Status: Pending with restart checkpoint
Hi all
I don't see a bug which is a precise match to this, but I do see that both scenarios were using 7.0.x releases, and I see a fair few SnapMirror bugs have been fixed in 7.2.4; so I am wondering if in either of the scenarios it is possible to move both filers to 7.2.4 (I semi-fear it isn't especially for the source filers concerned) and/or if anyone has seen this on a 7.2.x release.
cheers Kenneth
http://now.netapp.com/NOW/cgi-bin/relcmp.on?&rrel=7.0.6&rrel=7.2.4&a... ________________________________
Subject: RE: Oddball SnapMirror issue Date: Sun, 4 May 2008 13:24:05 -0500 From: mpartyka@acmn.com To: tmacmd@gmail.com; owner-toasters@mathworks.com; phigmov@gmail.com;
toasters@mathworks.com
Is there any reason to prefer wafliron over WAFL_check? Sounds like they do the same thing but you have the option to only check not automatically fix with WAFL_check.
-Mike
-----Original Message----- From: tmacmd@gmail.com [mailto:tmacmd@gmail.com] Sent: Sunday, May 04, 2008 12:59 PM To: Mike Partyka; owner-toasters@mathworks.com; Raj Patel; NetApp Toasters List Subject: Re: Oddball SnapMirror issue
I would try a wafl iron on the source volume/aggr
Just because you do not see any filesystem problems, does not mean ther are not any.
--tmac
Sent from my Verizon Wireless BlackBerry
-----Original Message----- From: "Mike Partyka" mpartyka@acmn.com
Date: Sun, 4 May 2008 09:28:18 To:"Raj Patel" phigmov@gmail.com, toasters@mathworks.com Subject: RE: Oddball SnapMirror issue
I'm having a similar experience trying to setup a Snapmirror between a pair of filers in the same datacenter (Not separated by a firewall). The source is a 3050 running DOT 7.0.5 and the destination is a 270 running 7.0.6. The volume is a 420G volume serving unstructured CIFS data. When I start the initialize everything works fine until it gets to about 82 or 83G, then the initialize aborts. The log contains some very non-specific messages, here is the current snapmirror log:
sys Sat May 3 09:12:55 CDT SnapMirror_off (shutdown) log Sat May 3 09:15:31 CDT FILER_REBOOTED sys Sat May 3 09:15:34 CDT SnapMirror_on (registry) dst Sat May 3 10:09:36 CDT 10.0.10.238:data hci2:rcv_data Request (Initialize) dst Sat May 3 10:09:42 CDT 10.0.10.238:data hci2:rcv_data Start dst Sat May 3 11:51:24 CDT 10.0.10.238:data hci2:rcv_data Abort (snapmirror transfer failed to complete)
Just as the Raj says when it fails to initialize the destination volume is in limbo, you can't online it due to the failed initialize. Here is the error:
vol online: Volume 'rcv_data' was left in an inconsistent state by an aborted vol copy or an aborted snapmirror initial (level 0) transfer. In order to bring it online, you must either destroy and re-create the volume, or complete an initial snapmirror transfer or vol copy.
I have considered running WAFL_check but WAFL isn't reporting an inconsistent state so i'm not sure that would be very effective. Yesterday I upgraded both filers to DOT 7.2.4 and updated all firmware then retried with the exact same results.
The only thing I can think of doing now is running a packet capture on the filer while it runs and see what that tells me.
-Mike
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Raj Patel Sent: Sunday, May 04, 2008 1:29 AM To: George T Chen Cc: toasters@mathworks.com Subject: Re: Oddball SnapMirror issue
Hi George,
The working transfers do just update 10 to 20Mb - very small turnover.
Unfortunately the two I need to mirror are from scratch - no baseline snapshot. The checkpoint restart occurring during the initialisation phase. Once the initialisation phase stalls further updates fail as the volume is not online (obviusly because the init failed).
I tried setting a once-a-day schedule at a particular time so it wouldn't trip over itself or other snapmirror operations to no avail.
As other volumes are updating with small update it made me wonder if it wasn't the router ipsec tunnel or firewall prematurely closing a connection for a large baseline transfer.
I'll attach the log & config when I get back into work.
Cheers, Raj.
On Sun, May 4, 2008 at 4:36 PM, George T Chen gtchen@yahoo-inc.com wrote:
Since you have one volume already transferring, then there's no
network
or firewall issue--any problem at that level would affect all
volumes,
not just a few.
A "Pending with restart checkpoint" appears you abort an ongoing transfer. Checkpoint occur every ?? megabytes and gives Ontap a
place
to restart instead of from scratch. It's hard to debug without more info, but I would start by:
- doing a snapmirror break on the volume (not just an abort)
- verify that there is a common baseline snapshot on both source and
destination 3) restart with a snapmirror resync command
Depending on step 2, you may be required to go to a snapmirror initialize.
What do the /etc/log/snapmirror and /etc/messages file say?
-gtchen
-----Original Message----- From: owner-toasters@mathworks.com
[mailto:owner-toasters@mathworks.com]
On Behalf Of Raj Patel Sent: Saturday, May 03, 2008 2:00 AM To: toasters@mathworks.com Subject: Oddball SnapMirror issue
We've got two FAS 270's in different cities. They're connected by a 10mb pipe with routers (running ipsec) & firewalls (checkpoint
splat)
seperating each datacenter.
The primary san is fine and runs all our prod volumes (7.0.5) which are mirrored to our secondary san (7.0.6).
Recently I had to recreate the mirror relationship for some volumes
as
they'd fallen far out of sync due to some firewall work.
What I am seeing is one volume is syncing fine, one has a small lag and two are stuck with a status of 'Pending with restart
checkpoint'
after I re-initialised the transfer.
snapmirror status -l shows this for one of the two that just don't
get
properly initialised
Source: 10.1.45.7:sqlprod01 Destination: adcsan1:sqlprod01_mirror Status: Pending with restart checkpoint Progress: 38376 KB State: Unknown Lag: - Mirror Timestamp: - Base Snapshot: - Current Transfer Type: Retry Current Transfer Error: volume is not online; cannot execute
operation
Contents: - Last Transfer Type: - Last Transfer Size: - Last Transfer Duration: - Last Transfer From: -
Our firewalls rules have been relaxed to allow free-flow between
these
devices (instead of just the SnapMirror ports) and the routers and circuit haven't changed at all between it working fine and not
working
now. The volume that is mirroring OK seems fine and still syncs
fine -
granted the updates are small whereas the three non-working volumes have to sync quite a lot of data.
I've tried deleting the mirrored volumes, recreating them, setting
up
the mirror relationship again (with a variety of scheduling and bandwidth throttling options) and doing a destination SAN reboot.
What are the best options to troubleshoot this or insuring a successful mirror ? Has anyone had issues with dropped or stalled SnapMirror baseline transfers via an IPSec tunnel or Firewall ?
Thanks in advance, Raj.
PS As an addendum it looks like it starts a transfer, stalls and
from
then on subsequent mirrors fail because its not online (ie the initialisation fails ?)
What I don't understand is why it just can't carry on with the initialisation regardless of the interruption by resuming the
mirror
operation ?
Express yourself instantly with MSN Messenger! MSN Messenger
Express yourself instantly with MSN Messenger! MSN Messenger