Hey all,
I've got a 8.0.2 c-mode cluster that recently had a single node joined to it, and a few empty aggregates created on it. We had a extended power outtage that required a lot of gear in the data center to get shut down, and since this node in the cluster didn't have any live data or VIFs on it, it got shut down.
A few days later, and we are now powering it up, but get this upon login:
"The contents of the root volume may have changed and the local management databases may be out of sync with the replicated databases due to corruption of NVLOG data during takeover. This node is not fully operational. Contact support personnel for the root volume recovery procedures."
The node comes up fine, can see all it's aggregates, and the other nodes in the cluster can see it via the cluster network, but the node is indeed not fully functional and part of the cluster again. It's aggregates and other info are not visible from the other nodes in the cluster.
Did a wafl_check of the root aggr and vol0 and that came back clean. I seem to recall having been through this before in the past, but can't find anything in my notes.
This particular cluster is not under support, due to some genius decisions by management, so I'm on my own with this.
There are a few empty aggregates on this node, no volumes other than the root vol. Maybe I can force unjoin it from the cluster and rebuild it? Would rather not try to do that. If there is a way to sync up the dbs on the rootvol so it will come back into the cluster, that would be ideal.
Any ideas?