Hey Skottie!

Good to hear from you - been a while!

Unfortunately, though those commands are available in this ancient 8.0.2 release we are running on this cluster, they cannot be run in the state that this particular node is in.

It's in a state kind of like prior to joining or creating a cluster. There are no volume commands available to be run (even if i explicitly type them out) It's got the cluster join, cluster create, ping-cluster commands available - it's like it's orphaned itself from the rest of the cluster. Though if I ping-cluster it sees and can connect to all the other nodes fine.

I'm tempted to try and rejoin it to the cluster via 'cluster join' again, but don't want to possibly screw up the rest of the cluster.

Thanks for the help!

On Tue, Oct 15, 2013 at 2:09 PM, Scott Miller <Scott.Miller@dreamworks.com> wrote:

try:

set -priv diag
volume add-other-volumes

which claims it's used to import 7-mode volumes, but I've
used it to re-sync the cluster volume database when
a root volume went walk-about.

*and*

also in priv mode:

volume lost-found show to see what the cluster volume DB
things might be missing.

These commands are in cDOT 8.2P3, not sure in earlier versions.

-skottie

On 10/15/2013 01:47 PM, Mike Thompson wrote:

Hey all,

I've got a 8.0.2 c-mode cluster that recently had a single node joined
to it, and a few empty aggregates created on it. We had a extended
power outtage that required a lot of gear in the data center to get shut
down, and since this node in the cluster didn't have any live data or
VIFs on it, it got shut down.

A few days later, and we are now powering it up, but get this upon login:

"The contents of the root volume may have changed and the local management
databases may be out of sync with the replicated databases due to
corruption of
NVLOG data during takeover. This node is not fully operational. Contact
support
personnel for the root volume recovery procedures."

The node comes up fine, can see all it's aggregates, and the other nodes
in the cluster can see it via the cluster network, but the node is
indeed not fully functional and part of the cluster again. It's
aggregates and other info are not visible from the other nodes in the
cluster.

Did a wafl_check of the root aggr and vol0 and that came back clean. I
seem to recall having been through this before in the past, but can't
find anything in my notes.

This particular cluster is not under support, due to some genius
decisions by management, so I'm on my own with this.

There are a few empty aggregates on this node, no volumes other than the
root vol. Maybe I can force unjoin it from the cluster and rebuild
it? Would rather not try to do that. If there is a way to sync up the
dbs on the rootvol so it will come back into the cluster, that would be
ideal.

Any ideas?

_______________________________________________
Toasters mailing list
Toasters@teaparty.net
http://www.teaparty.net/mailman/listinfo/toasters