Greetings,
I upgraded heads on our filers. The new 6290s had their own disks and I have 8.1.2P4 on them. The old 6080s were 7.3.5.1P4. For the update I kept the 6290 root/vol0 drives and changed ownership of all the other drives. The aggregates were brought online and all the volumes came up OK. I'm now seeing none of my backups are running because the snapvault snapshot is missing. I still have the acs snapshot. I've tried the snapvault start -S command and it says it is already a snapvault relationship. When I try to tell it to do a snapvault update, it says the base snapshot no longer exists on the source.
What happened to all my snapvault snapshots? Why did the snapvault snapshots get released? I thought those were in the volume and should have carried over in the hardware migration. I still have access to the old root volumes and backups of the old root volumes if there is anything that can be salvaged. I would rather not have to re-baseline 100+TB of source data.
Thanks,
Jeff
Possibly because the system is confused because of the new system id/serial which is part of the snapshot name. You did use exactly the same system names ? They are case sensitive as far as snapshot naming is concerned. Or Protection Manager may have deleted them because it is confused about something. Can you restart them with "snapvault start -S -r" ?
On Fri, Jul 5, 2013 at 3:08 AM, Jeff Cleverley jeff.cleverley@avagotech.com wrote:
Greetings,
I upgraded heads on our filers. The new 6290s had their own disks and I have 8.1.2P4 on them. The old 6080s were 7.3.5.1P4. For the update I kept the 6290 root/vol0 drives and changed ownership of all the other drives. The aggregates were brought online and all the volumes came up OK. I'm now seeing none of my backups are running because the snapvault snapshot is missing. I still have the acs snapshot. I've tried the snapvault start -S command and it says it is already a snapvault relationship. When I try to tell it to do a snapvault update, it says the base snapshot no longer exists on the source.
What happened to all my snapvault snapshots? Why did the snapvault snapshots get released? I thought those were in the volume and should have carried over in the hardware migration. I still have access to the old root volumes and backups of the old root volumes if there is anything that can be salvaged. I would rather not have to re-baseline 100+TB of source data.
Thanks,
Jeff
-- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Filip,
We don't use any level of DFM to manage the snapshots, so the baseline name doesn't contain any of that. That doesn't mean it isn't embedded in the meta data somewhere.
The hosts were renamed to the same hosts they were before so that part is OK. I tried both the snapvault start -S and snapvault start -r and neither works. I also tried specifying the the acs snapshot using the "-s" option on a snapvault update. Everything fails saying the base snapshot is no longer there.
I've moved aggregates from one partner to another and was able to modify the snapvault relationship to reflect the new source location and that worked. That should have had the original filer ID/serial number in it if that was the case.
For right now I'm not getting very far with NetApp since they are running a skeleton crew this week. We have another business group that is purchasing updated heads. I've had to tell them not to do the upgrade until we figure out what the problem is.
Apparently the only solution right now is to re-baseline 400+ TB. I'm not sure I even have that much available so we'll end up having to delete a bunch of snapshots and possibly some destination volumes.
Thanks,
Jeff
On Fri, Jul 5, 2013 at 12:09 AM, Filip Sneppe filip.sneppe@gmail.comwrote:
Possibly because the system is confused because of the new system id/serial which is part of the snapshot name. You did use exactly the same system names ? They are case sensitive as far as snapshot naming is concerned. Or Protection Manager may have deleted them because it is confused about something. Can you restart them with "snapvault start -S -r" ?
On Fri, Jul 5, 2013 at 3:08 AM, Jeff Cleverley jeff.cleverley@avagotech.com wrote:
Greetings,
I upgraded heads on our filers. The new 6290s had their own disks and I have 8.1.2P4 on them. The old 6080s were 7.3.5.1P4. For the update I
kept
the 6290 root/vol0 drives and changed ownership of all the other drives. The aggregates were brought online and all the volumes came up OK. I'm
now
seeing none of my backups are running because the snapvault snapshot is missing. I still have the acs snapshot. I've tried the snapvault start
-S
command and it says it is already a snapvault relationship. When I try
to
tell it to do a snapvault update, it says the base snapshot no longer
exists
on the source.
What happened to all my snapvault snapshots? Why did the snapvault snapshots get released? I thought those were in the volume and should
have
carried over in the hardware migration. I still have access to the old
root
volumes and backups of the old root volumes if there is anything that
can be
salvaged. I would rather not have to re-baseline 100+TB of source data.
Thanks,
Jeff
-- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Hi,
I think the following happened.
Snapshots locked by snapvault are registered in the root volume and in the volumes itself. Therefore since the new controllers used new root volumes the systems were unaware of any snapshot used as a base for snapvault relationships. This would also have been apparent by the lack of snapshots with "snapvault" behind it in the "snap list" output.
After swapping root volumes there is still not a problem. If you execute "snapvault updates" for all the relationships, either scheduled from the secondary or manual, ONTAP would have discovered the base snapshots again (no resync required) and registered them as a snapvault base snapshot in the new root volumes.
I now assume that the base snapshots got deleted by the snapvault schedules set on the primary or another process before the snapvault updates were executed.
The following KB describes something similar: https://kb.netapp.com/support/index?page=content&id=2012214
-- Pascal Dukers ASML IT Datacenter/Network General Phone +31(0)402684341
-----Original Message----- From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Jeff Cleverley Sent: Friday, July 05, 2013 11:23 PM To: Filip Sneppe Cc: Toasters@teaparty.net Subject: Re: Snapvault snapshots lost after upgrade.
Filip,
We don't use any level of DFM to manage the snapshots, so the baseline name doesn't contain any of that. That doesn't mean it isn't embedded in the meta data somewhere.
The hosts were renamed to the same hosts they were before so that part is OK. I tried both the snapvault start -S and snapvault start -r and neither works. I also tried specifying the the acs snapshot using the "-s" option on a snapvault update. Everything fails saying the base snapshot is no longer there.
I've moved aggregates from one partner to another and was able to modify the snapvault relationship to reflect the new source location and that worked. That should have had the original filer ID/serial number in it if that was the case.
For right now I'm not getting very far with NetApp since they are running a skeleton crew this week. We have another business group that is purchasing updated heads. I've had to tell them not to do the upgrade until we figure out what the problem is.
Apparently the only solution right now is to re-baseline 400+ TB. I'm not sure I even have that much available so we'll end up having to delete a bunch of snapshots and possibly some destination volumes.
Thanks,
Jeff
On Fri, Jul 5, 2013 at 12:09 AM, Filip Sneppe filip.sneppe@gmail.com wrote:
Possibly because the system is confused because of the new system id/serial which is part of the snapshot name. You did use exactly the same system names ? They are case sensitive as far as snapshot naming is concerned. Or Protection Manager may have deleted them because it is confused about something. Can you restart them with "snapvault start -S -r" ?
On Fri, Jul 5, 2013 at 3:08 AM, Jeff Cleverley jeff.cleverley@avagotech.com wrote:
Greetings,
I upgraded heads on our filers. The new 6290s had their own disks and
I
have 8.1.2P4 on them. The old 6080s were 7.3.5.1P4. For the update I
kept
the 6290 root/vol0 drives and changed ownership of all the other
drives.
The aggregates were brought online and all the volumes came up OK.
I'm now
seeing none of my backups are running because the snapvault
snapshot is
missing. I still have the acs snapshot. I've tried the snapvault start -S command and it says it is already a snapvault relationship. When I try
to
tell it to do a snapvault update, it says the base snapshot no longer
exists
on the source.
What happened to all my snapvault snapshots? Why did the snapvault snapshots get released? I thought those were in the volume and
should have
carried over in the hardware migration. I still have access to the old
root
volumes and backups of the old root volumes if there is anything that
can be
salvaged. I would rather not have to re-baseline 100+TB of source
data.
Thanks,
Jeff
-- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
-- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611
Pascal,
I had looked at this KB but it didn't really seem to apply since it was talking about moving the root volume to the new environment. It would indicate that the snapvault information being used is not being read from the disks, but from the root volume (registry files?).
If this really is the case, it sounds like you can never migrate primary filers without taking the root volume with you.
I know this does not apply to the secondary filers. I recently took all the drives from a 3140 and connected them to a 3270. Once I changed the system ID and updated the snapvault relationship to point to the new filer, everything was happy. We've done that on several occasions with no issues.
I'm still waiting to hear back with something definitive from NetApp.
Thanks,
Jeff
On Sun, Jul 7, 2013 at 11:40 PM, Pascal Dukers pascal.dukers@asml.comwrote:
Hi,
I think the following happened.
Snapshots locked by snapvault are registered in the root volume and in the volumes itself. Therefore since the new controllers used new root volumes the systems were unaware of any snapshot used as a base for snapvault relationships. This would also have been apparent by the lack of snapshots with "snapvault" behind it in the "snap list" output.
After swapping root volumes there is still not a problem. If you execute "snapvault updates" for all the relationships, either scheduled from the secondary or manual, ONTAP would have discovered the base snapshots again (no resync required) and registered them as a snapvault base snapshot in the new root volumes.
I now assume that the base snapshots got deleted by the snapvault schedules set on the primary or another process before the snapvault updates were executed.
The following KB describes something similar: https://kb.netapp.com/support/index?page=content&id=2012214
-- Pascal Dukers ASML IT Datacenter/Network General Phone +31(0)402684341
-----Original Message----- From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net
]
On Behalf Of Jeff Cleverley Sent: Friday, July 05, 2013 11:23 PM To: Filip Sneppe Cc: Toasters@teaparty.net Subject: Re: Snapvault snapshots lost after upgrade.
Filip,
We don't use any level of DFM to manage the snapshots, so the baseline
name
doesn't contain any of that. That doesn't mean it isn't embedded in the
meta
data somewhere.
The hosts were renamed to the same hosts they were before so that part is
OK.
I tried both the snapvault start -S and snapvault start -r and neither
works. I also
tried specifying the the acs snapshot using the "-s" option on a
snapvault update.
Everything fails saying the base snapshot is no longer there.
I've moved aggregates from one partner to another and was able to modify
the
snapvault relationship to reflect the new source location and that
worked. That
should have had the original filer ID/serial number in it if that was the
case.
For right now I'm not getting very far with NetApp since they are running
a
skeleton crew this week. We have another business group that is
purchasing
updated heads. I've had to tell them not to do the upgrade until we
figure out
what the problem is.
Apparently the only solution right now is to re-baseline 400+ TB. I'm
not sure I
even have that much available so we'll end up having to delete a bunch of snapshots and possibly some destination volumes.
Thanks,
Jeff
On Fri, Jul 5, 2013 at 12:09 AM, Filip Sneppe filip.sneppe@gmail.com
wrote:
Possibly because the system is confused because of the new system id/serial which is part of the snapshot name. You did use exactlythe
same system names ? They are case sensitive as far as snapshotnaming
is concerned. Or Protection Manager may have deleted them becauseit
is confused about something. Can you restart them with "snapvault start -S -r" ? On Fri, Jul 5, 2013 at 3:08 AM, Jeff Cleverley <jeff.cleverley@avagotech.com> wrote: > Greetings, > > I upgraded heads on our filers. The new 6290s had their owndisks and
I > have 8.1.2P4 on them. The old 6080s were 7.3.5.1P4. For the
update I
kept > the 6290 root/vol0 drives and changed ownership of all the other drives. > The aggregates were brought online and all the volumes came up
OK.
I'm now > seeing none of my backups are running because the snapvault snapshot is > missing. I still have the acs snapshot. I've tried the
snapvault start -S
> command and it says it is already a snapvault relationship.When I try
to > tell it to do a snapvault update, it says the base snapshot no
longer
exists > on the source. > > What happened to all my snapvault snapshots? Why did the
snapvault
> snapshots get released? I thought those were in the volume andshould have > carried over in the hardware migration. I still have access to
the old
root > volumes and backups of the old root volumes if there is anything
that
can be > salvaged. I would rather not have to re-baseline 100+TB of
source
data. > > Thanks, > > Jeff > > -- > Jeff Cleverley > Unix Systems Administrator > 4380 Ziegler Road > Fort Collins, Colorado 80525 > 970-288-4611 >
> _______________________________________________ > Toasters mailing list > Toasters@teaparty.net > http://www.teaparty.net/mailman/listinfo/toasters >-- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611
-- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt.
Jeff,
I had looked at this KB but it didn't really seem to apply since it was talking about moving the root volume to the new environment. It would indicate that the snapvault information being used is not being read from the disks, but from the root volume (registry files?).
For the snapvault source the snapvault relationship and the snapshot that will be softlocked because it is the base snapshot are stored in the registry. So when you start with a brand new root volume, the knowledge of snapvault relationships and softlocked snapshots is lost. However this does not imply that your base snapshot is lost after a headswap with new root volume.
As "proof" of the registry information an example for a softlock from the registry file of an actual filer:
"state.sv_softlock.cce124.001d69e5.066.internal.1=locked"
This shows that volume cce124 has a softlock on snapshot id "066".
Snapshot id "066" in volume cce124 has the name "sv_daily.0":
Volume cce124 snapid status date ownblks release fsRev name ------ ------ ------------ ------- ------- ----- -------- 66 complete Jul 08 21:00 738206 8.1 22331 sv_daily.0
And indeed it is softlocked for snapvault:
%/used %/total date name ---------- ---------- ------------ -------- 0% ( 0%) 0% ( 0%) Jul 08 21:00 sv_daily.0 (snapvault,acs)
If this really is the case, it sounds like you can never migrate primary filers without taking the root volume with you.
As I already described, just execute snapvault updates from the snapvault destination to register the snapvault relationship and softlocks again in the new root volume. As long as the base snapshots have not been deleted this will work.
I know this does not apply to the secondary filers. I recently took all the drives from a 3140 and connected them to a 3270. Once I changed the system ID and updated the snapvault relationship to point to the new filer, everything was happy. We've done that on several occasions with no issues.
With moving all disks, do you mean you also reused the original root volume (with all the registry information)? To be honest I do not have experience with new root volumes on secondary filers. But for primary filers I have seen it gone wrong for the reasons I described.
Pascal.
Pascal,
Your explanation is the best I've heard and makes sense based on what we've seen. Since the information is in the registry, it makes me ask the next question - Is there a way to modify the registry?
I know it is not a normal file. It would be interesting to try and modify the registry entry for which snapshot ID is the baseline, or find a way to change the snapshot ID of a snapshot. In a case like this I would be willing to potentially lose some data created between snapshot for backups vs baselining 400TB :-) I'm guessing there isn't a way to make that work and most certainly would not be supported if there was.
If this really is the case, it sounds like you can never migrate primary filers
without taking the root volume with you.
As I already described, just execute snapvault updates from the snapvault destination to register the snapvault relationship and softlocks again in the new root volume. As long as the base snapshots have not been deleted this will work.
I'll keep that in mind next time :-)
I know this does not apply to the secondary filers. I recently took all
the drives
from a 3140 and connected them to a 3270. Once I changed the system ID
and
updated the snapvault relationship to point to the new filer, everything
was
happy. We've done that on several occasions with no issues.
With moving all disks, do you mean you also reused the original root volume (with all the registry information)? To be honest I do not have experience with new root volumes on secondary filers. But for primary filers I have seen it gone wrong for the reasons I described.
I wasn't clear with that. The 3270 already was running as a NearStore with its own backups and root volume. The 3140 root disks were not used. I did do the snapvault start command with the new location to update everything which probably made everything happy.
Thanks again for all the help and explanations on this.
Jeff
Pascal.
-- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt.
-
Filip Sneppe -
Jeff Cleverley -
Pascal Dukers