Hello all,

It just died all of a sudden. Automatic Recovery was off. Pls advise.

Cannot access the cluster level , node level is fine.

Sim::> event log show -severity emerGENCY

Time Node Severity Event

------------------- ---------------- ------------- ---------------------------

7/26/2015 09:57:50 Sim-01 EMERGENCY callhome.root.vol.recovery.reqd: Call home for ROOT VOLUME NOT WORKING PROPERLY: RECOVERY REQUIRED.

7/26/2015 09:56:36 Sim-01 EMERGENCY rdb.recovery.failed: Error: Unable to find a master.. Unable to recover the local database of Data Replication Module: Management.

2 entries were displayed.

Sim::> event log show -severity erroR

Time Node Severity Event

------------------- ---------------- ------------- ---------------------------

7/26/2015 10:00:51 Sim-01 ERROR mgmtgwd.ntp.job.failure: The ntp_resolve_job could not be started.

7/26/2015 09:56:39 Sim-01 ERROR spm.mgwd.process.exit: Management Gateway (mgwd) subsystem with ID 12774 exited as a result of signal signal 6. The subsystem will attempt to restart.

7/26/2015 09:56:36 Sim-01 ERROR ucore.panicString: 'mgwd: assertion (false) at corrupt_rdb.cc:587 failed, raising SIGABRT(6) at RIP 0x82e1ed7bc (pid 12774, uid 0, timestamp 1437918996)'

7/26/2015 09:53:50 Sim-01 ERROR spm.mgwd.process.exit: Management Gateway (mgwd) subsystem with ID 12403 exited as a result of signal normal exit (1). The subsystem will attempt to restart.

7/26/2015 09:52:15 Sim-01 ERROR spm.vldb.process.exit: Volume Location Database(VLDB) subsystem with ID 4568 exited as a result of signal normal exit (1). The subsystem will attempt to restart.

7/26/2015 09:52:15 Sim-01 ERROR rdb.env.IOFailure: Error encountered in root volume I/O: Could not force changes to /mroot/etc/cluster_config/rdb/VLDB/_unit.config.bak.

7/25/2015 19:29:45 Sim-01 ERROR mgmtgwd.mroot.nearlyfull: The root volume of node 'Sim-01' is nearly full. When it is full the node cannot operate properly.

7/25/2015 19:29:45 Sim-01 ERROR mgmtgwd.configbr.noSNCBackup: This cluster is effectively a single node and there is no configuration backup configured.

7/25/2015 19:27:45 Sim-01 ERROR mgmtgwd.mroot.nearlyfull: The root volume of node 'Sim-01' is nearly full. When it is full the node cannot operate properly.

7/22/2015 18:34:29 Sim-01 ERROR mgmtgwd.configbr.noSNCBackup: This cluster is effectively a single node and there is no configuration backup configured.

7/22/2015 18:32:11 Sim-01 ERROR mgmtgwd.mroot.nearlyfull: The root volume of node 'Sim-01' is nearly full. When it is full the node cannot operate properly.

7/21/2015 10:20:13 Sim-01 ERROR mgmtgwd.configbr.noSNCBackup: This cluster is effectively a single node and there is no configuration backup configured.

7/21/2015 09:16:17 Sim-01 ERROR mgmtgwd.configbr.noSNCBackup: This cluster is effectively a single node and there is no configuration backup configured.

7/21/2015 09:14:53 Sim-01 ERROR mgmtgwd.configbr.backupFailed: The node configuration backup Sim-01.daily.2015-07-21.09_03_46.7z cannot be created. Error: 'Failed to create tarball for backup Sim-01.daily.2015-07-21.09_03_46.7z. Error: Failed to create package with name /mroot/etc/backups/config/Sim-01.daily.2015-07-21.09_03_46.7z'.

7/21/2015 09:14:51 Sim-01 ERROR mgmtgwd.configbr.backupFailed: The node configuration backup Sim-01.8hour.2015-07-21.09_03_46.7z cannot be created. Error: 'Failed to create tarball for backup Sim-01.8hour.2015-07-21.09_03_46.7z. Error: Failed to create package with name /mroot/etc/backups/config/Sim-01.8hour.2015-07-21.09_03_46.7z'.

7/21/2015 09:14:50 Sim-01 ERROR mgmtgwd.mroot.nearlyfull: The root volume of node 'Sim-01' is nearly full. When it is full the node cannot operate properly.

7/21/2015 09:14:37 Sim-01 ERROR mgmtgwd.mroot.full: The root volume of node 'Sim-01' is full. This prevents the node from operating properly.

7/21/2015 09:07:23 Sim-01 ERROR mgmtgwd.mroot.nearlyfull: The root volume of node 'Sim-01' is nearly full. When it is full the node cannot operate properly.

21 entries were displayed.

Sim::>