Hello all,
It just died all of a sudden. Automatic Recovery was off. Pls advise.
Cannot access the cluster level , node level is fine.
Sim::> event log show -severity emerGENCY
Time Node Severity Event
------------------- ---------------- ------------- ---------------------------
7/26/2015 09:57:50 Sim-01 EMERGENCY callhome.root.vol.recovery.reqd: Call home for ROOT VOLUME NOT WORKING PROPERLY: RECOVERY REQUIRED.
7/26/2015 09:56:36 Sim-01 EMERGENCY rdb.recovery.failed: Error: Unable to find a master.. Unable to recover the local database of Data Replication Module: Management.
2 entries were displayed.
Sim::> event log show -severity erroR
Time Node Severity Event
------------------- ---------------- ------------- ---------------------------
7/26/2015 10:00:51 Sim-01 ERROR mgmtgwd.ntp.job.failure: The ntp_resolve_job could not be started.
7/26/2015 09:56:39 Sim-01 ERROR spm.mgwd.process.exit: Management Gateway (mgwd) subsystem with ID 12774 exited as a result of signal signal 6. The subsystem will attempt to restart.
7/26/2015 09:56:36 Sim-01 ERROR ucore.panicString: 'mgwd: assertion (false) at corrupt_rdb.cc:587 failed, raising SIGABRT(6) at RIP 0x82e1ed7bc (pid 12774, uid 0, timestamp 1437918996)'
7/26/2015 09:53:50 Sim-01 ERROR spm.mgwd.process.exit: Management Gateway (mgwd) subsystem with ID 12403 exited as a result of signal normal exit (1). The subsystem will attempt to restart.
7/26/2015 09:52:15 Sim-01 ERROR spm.vldb.process.exit: Volume Location Database(VLDB) subsystem with ID 4568 exited as a result of signal normal exit (1). The subsystem will attempt to restart.
7/26/2015 09:52:15 Sim-01 ERROR rdb.env.IOFailure: Error encountered in root volume I/O: Could not force changes to /mroot/etc/cluster_config/rdb/VLDB/_unit.config.bak.
.
7/25/2015 19:29:45 Sim-01 ERROR mgmtgwd.mroot.nearlyfull: The root volume of node 'Sim-01' is nearly full. When it is full the node cannot operate properly.
7/25/2015 19:29:45 Sim-01 ERROR mgmtgwd.configbr.noSNCBackup: This cluster is effectively a single node and there is no configuration backup configured.
7/25/2015 19:29:45 Sim-01 ERROR mgmtgwd.configbr.noSNCBackup: This cluster is effectively a single node and there is no configuration backup configured.
7/25/2015 19:27:45 Sim-01 ERROR mgmtgwd.mroot.nearlyfull: The root volume of node 'Sim-01' is nearly full. When it is full the node cannot operate properly.
7/22/2015 18:34:29 Sim-01 ERROR mgmtgwd.configbr.noSNCBackup: This cluster is effectively a single node and there is no configuration backup configured.
7/22/2015 18:34:29 Sim-01 ERROR mgmtgwd.configbr.noSNCBackup: This cluster is effectively a single node and there is no configuration backup configured.
7/22/2015 18:32:11 Sim-01 ERROR mgmtgwd.mroot.nearlyfull: The root volume of node 'Sim-01' is nearly full. When it is full the node cannot operate properly.
7/21/2015 10:20:13 Sim-01 ERROR mgmtgwd.configbr.noSNCBackup: This cluster is effectively a single node and there is no configuration backup configured.
7/21/2015 09:16:17 Sim-01 ERROR mgmtgwd.configbr.noSNCBackup: This cluster is effectively a single node and there is no configuration backup configured.
7/21/2015 09:16:17 Sim-01 ERROR mgmtgwd.configbr.noSNCBackup: This cluster is effectively a single node and there is no configuration backup configured.
7/21/2015 09:14:53 Sim-01 ERROR mgmtgwd.configbr.backupFailed: The node configuration backup Sim-01.daily.2015-07-21.09_03_46.7z cannot be created. Error: 'Failed to create tarball for backup Sim-01.daily.2015-07-21.09_03_46.7z. Error: Failed to create package with name /mroot/etc/backups/config/Sim-01.daily.2015-07-21.09_03_46.7z'.
7/21/2015 09:14:51 Sim-01 ERROR mgmtgwd.configbr.backupFailed: The node configuration backup Sim-01.8hour.2015-07-21.09_03_46.7z cannot be created. Error: 'Failed to create tarball for backup Sim-01.8hour.2015-07-21.09_03_46.7z. Error: Failed to create package with name /mroot/etc/backups/config/Sim-01.8hour.2015-07-21.09_03_46.7z'.
7/21/2015 09:14:50 Sim-01 ERROR mgmtgwd.mroot.nearlyfull: The root volume of node 'Sim-01' is nearly full. When it is full the node cannot operate properly.
7/21/2015 09:14:37 Sim-01 ERROR mgmtgwd.mroot.full: The root volume of node 'Sim-01' is full. This prevents the node from operating properly.
7/21/2015 09:07:23 Sim-01 ERROR mgmtgwd.mroot.nearlyfull: The root volume of node 'Sim-01' is nearly full. When it is full the node cannot operate properly.
21 entries were displayed.
Sim::>
Looks like you inadvertently filled up your root volume.
Doing something there you should be....
May need to log in as diag user and clean up some space on /mroot
--tmac
*Tim McCarthy* *Principal Consultant*
On Sun, Jul 26, 2015 at 10:23 AM, Uddhav Regmi uregmi111@gmail.com wrote:
I am checking on this
Sim-01> df -gm
Filesystem total used avail capacity Mounted on
/vol/vol0/ 766MB 672MB 94MB 88% /vol/vol0/
/vol/vol0/.snapshot 40MB 0MB 40MB 0% /vol/vol0/.snapshot
/vol/vol101/ 19MB 0MB 18MB 1% /vol/vol101/
/vol/vol101/.snapshot 1MB 0MB 0MB 61% /vol/vol101/.snapshot
/vol/svm101_root/ 19MB 0MB 18MB 1% /vol/svm101_root/
/vol/svm101_root/.snapshot 1MB 0MB 0MB 58% /vol/svm101_root/.snapshot
Last time when I had this issue, it was different and I turned off all snapshot. Looks like 88%, there is something , somewhere this is getting filled up
Hmmmm…..nothing inside volume of vserver ??? Interesting…
Sim::volume*> ?
Sim::volume*> help
Press "?" or Tab for help.
Sim::volume*> cd ..
Sim::*> vserver
Sim::vserver*> ?
audit> Manage auditing of protocol requests that the Vserver services
check> The check directory
cifs> Manage the CIFS configuration of a Vserver
nfs> Manage the NFS configuration of a Vserver
services> The services directory
vscan> Manage Vscan
From: tmac [mailto:tmacmd@gmail.com] Sent: Sunday, July 26, 2015 10:37 AM To: Uddhav Regmi uregmi111@gmail.com Cc: Toasters Toasters@teaparty.net Subject: Re: unable to find master : Cdot : 8.3RC1
Looks like you inadvertently filled up your root volume.
Doing something there you should be....
May need to log in as diag user and clean up some space on /mroot
--tmac
Tim McCarthy
Principal Consultant
On Sun, Jul 26, 2015 at 10:23 AM, Uddhav Regmi <uregmi111@gmail.com mailto:uregmi111@gmail.com > wrote:
Hello all,
It just died all of a sudden. Automatic Recovery was off. Pls advise.
Cannot access the cluster level , node level is fine.
Sim::> event log show -severity emerGENCY
Time Node Severity Event
------------------- ---------------- ------------- ---------------------------
7/26/2015 09:57:50 Sim-01 EMERGENCY callhome.root.vol.recovery.reqd: Call home for ROOT VOLUME NOT WORKING PROPERLY: RECOVERY REQUIRED.
7/26/2015 09:56:36 Sim-01 EMERGENCY rdb.recovery.failed: Error: Unable to find a master.. Unable to recover the local database of Data Replication Module: Management.
2 entries were displayed.
Sim::> event log show -severity erroR
Time Node Severity Event
------------------- ---------------- ------------- ---------------------------
7/26/2015 10:00:51 Sim-01 ERROR mgmtgwd.ntp.job.failure: The ntp_resolve_job could not be started.
7/26/2015 09:56:39 Sim-01 ERROR spm.mgwd.process.exit: Management Gateway (mgwd) subsystem with ID 12774 exited as a result of signal signal 6. The subsystem will attempt to restart.
7/26/2015 09:56:36 Sim-01 ERROR ucore.panicString: 'mgwd: assertion (false) at corrupt_rdb.cc:587 failed, raising SIGABRT(6) at RIP 0x82e1ed7bc (pid 12774, uid 0, timestamp 1437918996)'
7/26/2015 09:53:50 Sim-01 ERROR spm.mgwd.process.exit: Management Gateway (mgwd) subsystem with ID 12403 exited as a result of signal normal exit (1). The subsystem will attempt to restart.
7/26/2015 09:52:15 Sim-01 ERROR spm.vldb.process.exit: Volume Location Database(VLDB) subsystem with ID 4568 exited as a result of signal normal exit (1). The subsystem will attempt to restart.
7/26/2015 09:52:15 Sim-01 ERROR rdb.env.IOFailure: Error encountered in root volume I/O: Could not force changes to /mroot/etc/cluster_config/rdb/VLDB/_unit.config.bak.
.
7/25/2015 19:29:45 Sim-01 ERROR mgmtgwd.mroot.nearlyfull: The root volume of node 'Sim-01' is nearly full. When it is full the node cannot operate properly.
7/25/2015 19:29:45 Sim-01 ERROR mgmtgwd.configbr.noSNCBackup: This cluster is effectively a single node and there is no configuration backup configured.
7/25/2015 19:29:45 Sim-01 ERROR mgmtgwd.configbr.noSNCBackup: This cluster is effectively a single node and there is no configuration backup configured.
7/25/2015 19:27:45 Sim-01 ERROR mgmtgwd.mroot.nearlyfull: The root volume of node 'Sim-01' is nearly full. When it is full the node cannot operate properly.
7/22/2015 18:34:29 Sim-01 ERROR mgmtgwd.configbr.noSNCBackup: This cluster is effectively a single node and there is no configuration backup configured.
7/22/2015 18:34:29 Sim-01 ERROR mgmtgwd.configbr.noSNCBackup: This cluster is effectively a single node and there is no configuration backup configured.
7/22/2015 18:32:11 Sim-01 ERROR mgmtgwd.mroot.nearlyfull: The root volume of node 'Sim-01' is nearly full. When it is full the node cannot operate properly.
7/21/2015 10:20:13 Sim-01 ERROR mgmtgwd.configbr.noSNCBackup: This cluster is effectively a single node and there is no configuration backup configured.
7/21/2015 09:16:17 Sim-01 ERROR mgmtgwd.configbr.noSNCBackup: This cluster is effectively a single node and there is no configuration backup configured.
7/21/2015 09:16:17 Sim-01 ERROR mgmtgwd.configbr.noSNCBackup: This cluster is effectively a single node and there is no configuration backup configured.
7/21/2015 09:14:53 Sim-01 ERROR mgmtgwd.configbr.backupFailed: The node configuration backup Sim-01.daily.2015-07-21.09_03_46.7z cannot be created. Error: 'Failed to create tarball for backup Sim-01.daily.2015-07-21.09_03_46.7z. Error: Failed to create package with name /mroot/etc/backups/config/Sim-01.daily.2015-07-21.09_03_46.7z'.
7/21/2015 09:14:51 Sim-01 ERROR mgmtgwd.configbr.backupFailed: The node configuration backup Sim-01.8hour.2015-07-21.09_03_46.7z cannot be created. Error: 'Failed to create tarball for backup Sim-01.8hour.2015-07-21.09_03_46.7z. Error: Failed to create package with name /mroot/etc/backups/config/Sim-01.8hour.2015-07-21.09_03_46.7z'.
7/21/2015 09:14:50 Sim-01 ERROR mgmtgwd.mroot.nearlyfull: The root volume of node 'Sim-01' is nearly full. When it is full the node cannot operate properly.
7/21/2015 09:14:37 Sim-01 ERROR mgmtgwd.mroot.full: The root volume of node 'Sim-01' is full. This prevents the node from operating properly.
7/21/2015 09:07:23 Sim-01 ERROR mgmtgwd.mroot.nearlyfull: The root volume of node 'Sim-01' is nearly full. When it is full the node cannot operate properly.
21 entries were displayed.
Sim::>
_______________________________________________ Toasters mailing list Toasters@teaparty.net mailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
You're reaching out to toasters after wedging yourself with a cdot simulator?
.
On Jul 26, 2015, at 7:49 AM, Uddhav Regmi <uregmi111@gmail.commailto:uregmi111@gmail.com> wrote:
I am checking on this
Sim-01> df -gm Filesystem total used avail capacity Mounted on /vol/vol0/ 766MB 672MB 94MB 88% /vol/vol0/ /vol/vol0/.snapshot 40MB 0MB 40MB 0% /vol/vol0/.snapshot /vol/vol101/ 19MB 0MB 18MB 1% /vol/vol101/ /vol/vol101/.snapshot 1MB 0MB 0MB 61% /vol/vol101/.snapshot /vol/svm101_root/ 19MB 0MB 18MB 1% /vol/svm101_root/ /vol/svm101_root/.snapshot 1MB 0MB 0MB 58% /vol/svm101_root/.snapshot
Last time when I had this issue, it was different and I turned off all snapshot. Looks like 88%, there is something , somewhere this is getting filled up
Hmmmm…..nothing inside volume of vserver ??? Interesting…
Sim::volume*> ?
Sim::volume*> help Press "?" or Tab for help.
Sim::volume*> cd ..
Sim::*> vserver
Sim::vserver*> ? audit> Manage auditing of protocol requests that the Vserver services check> The check directory cifs> Manage the CIFS configuration of a Vserver nfs> Manage the NFS configuration of a Vserver services> The services directory vscan> Manage Vscan
From: tmac [mailto:tmacmd@gmail.com] Sent: Sunday, July 26, 2015 10:37 AM To: Uddhav Regmi <uregmi111@gmail.commailto:uregmi111@gmail.com> Cc: Toasters <Toasters@teaparty.netmailto:Toasters@teaparty.net> Subject: Re: unable to find master : Cdot : 8.3RC1
Looks like you inadvertently filled up your root volume.
Doing something there you should be....
May need to log in as diag user and clean up some space on /mroot
--tmac
Tim McCarthy Principal Consultant
On Sun, Jul 26, 2015 at 10:23 AM, Uddhav Regmi <uregmi111@gmail.commailto:uregmi111@gmail.com> wrote: Hello all,
It just died all of a sudden. Automatic Recovery was off. Pls advise.
Cannot access the cluster level , node level is fine.
Sim::> event log show -severity emerGENCY Time Node Severity Event ------------------- ---------------- ------------- --------------------------- 7/26/2015 09:57:50 Sim-01 EMERGENCY callhome.root.vol.recovery.reqd: Call home for ROOT VOLUME NOT WORKING PROPERLY: RECOVERY REQUIRED. 7/26/2015 09:56:36 Sim-01 EMERGENCY rdb.recovery.failed: Error: Unable to find a master.. Unable to recover the local database of Data Replication Module: Management. 2 entries were displayed.
Sim::> event log show -severity erroR Time Node Severity Event ------------------- ---------------- ------------- --------------------------- 7/26/2015 10:00:51 Sim-01 ERROR mgmtgwd.ntp.job.failure: The ntp_resolve_job could not be started. 7/26/2015 09:56:39 Sim-01 ERROR spm.mgwd.process.exit: Management Gateway (mgwd) subsystem with ID 12774 exited as a result of signal signal 6. The subsystem will attempt to restart. 7/26/2015 09:56:36 Sim-01 ERROR ucore.panicString: 'mgwd: assertion (false) at corrupt_rdb.cc:587 failed, raising SIGABRT(6) at RIP 0x82e1ed7bc (pid 12774, uid 0, timestamp 1437918996)' 7/26/2015 09:53:50 Sim-01 ERROR spm.mgwd.process.exit: Management Gateway (mgwd) subsystem with ID 12403 exited as a result of signal normal exit (1). The subsystem will attempt to restart. 7/26/2015 09:52:15 Sim-01 ERROR spm.vldb.process.exit: Volume Location Database(VLDB) subsystem with ID 4568 exited as a result of signal normal exit (1). The subsystem will attempt to restart. 7/26/2015 09:52:15 Sim-01 ERROR rdb.env.IOFailure: Error encountered in root volume I/O: Could not force changes to /mroot/etc/cluster_config/rdb/VLDB/_unit.config.bak. . 7/25/2015 19:29:45 Sim-01 ERROR mgmtgwd.mroot.nearlyfull: The root volume of node 'Sim-01' is nearly full. When it is full the node cannot operate properly. 7/25/2015 19:29:45 Sim-01 ERROR mgmtgwd.configbr.noSNCBackup: This cluster is effectively a single node and there is no configuration backup configured. 7/25/2015 19:29:45 Sim-01 ERROR mgmtgwd.configbr.noSNCBackup: This cluster is effectively a single node and there is no configuration backup configured. 7/25/2015 19:27:45 Sim-01 ERROR mgmtgwd.mroot.nearlyfull: The root volume of node 'Sim-01' is nearly full. When it is full the node cannot operate properly. 7/22/2015 18:34:29 Sim-01 ERROR mgmtgwd.configbr.noSNCBackup: This cluster is effectively a single node and there is no configuration backup configured. 7/22/2015 18:34:29 Sim-01 ERROR mgmtgwd.configbr.noSNCBackup: This cluster is effectively a single node and there is no configuration backup configured. 7/22/2015 18:32:11 Sim-01 ERROR mgmtgwd.mroot.nearlyfull: The root volume of node 'Sim-01' is nearly full. When it is full the node cannot operate properly. 7/21/2015 10:20:13 Sim-01 ERROR mgmtgwd.configbr.noSNCBackup: This cluster is effectively a single node and there is no configuration backup configured. 7/21/2015 09:16:17 Sim-01 ERROR mgmtgwd.configbr.noSNCBackup: This cluster is effectively a single node and there is no configuration backup configured. 7/21/2015 09:16:17 Sim-01 ERROR mgmtgwd.configbr.noSNCBackup: This cluster is effectively a single node and there is no configuration backup configured. 7/21/2015 09:14:53 Sim-01 ERROR mgmtgwd.configbr.backupFailed: The node configuration backup Sim-01.daily.2015-07-21.09_03_46.7z cannot be created. Error: 'Failed to create tarball for backup Sim-01.daily.2015-07-21.09_03_46.7z. Error: Failed to create package with name /mroot/etc/backups/config/Sim-01.daily.2015-07-21.09_03_46.7z'. 7/21/2015 09:14:51 Sim-01 ERROR mgmtgwd.configbr.backupFailed: The node configuration backup Sim-01.8hour.2015-07-21.09_03_46.7z cannot be created. Error: 'Failed to create tarball for backup Sim-01.8hour.2015-07-21.09_03_46.7z. Error: Failed to create package with name /mroot/etc/backups/config/Sim-01.8hour.2015-07-21.09_03_46.7z'. 7/21/2015 09:14:50 Sim-01 ERROR mgmtgwd.mroot.nearlyfull: The root volume of node 'Sim-01' is nearly full. When it is full the node cannot operate properly. 7/21/2015 09:14:37 Sim-01 ERROR mgmtgwd.mroot.full: The root volume of node 'Sim-01' is full. This prevents the node from operating properly. 7/21/2015 09:07:23 Sim-01 ERROR mgmtgwd.mroot.nearlyfull: The root volume of node 'Sim-01' is nearly full. When it is full the node cannot operate properly. 21 entries were displayed.
Sim::>
_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
No, it is full, based on your log messages:
7/21/2015 09:14:53 Sim-01 ERROR mgmtgwd.configbr.backupFailed: The node configuration backup Sim-01.daily.2015-07-21.09_03_46.7z cannot be created. Error: 'Failed to create tarball for backup Sim-01.daily.2015-07-21.09_03_46.7z. Error: Failed to create package with name /mroot/etc/backups/config/Sim-01.daily.2015-07-21.09_03_46.7z'.
7/21/2015 09:14:51 Sim-01 ERROR mgmtgwd.configbr.backupFailed: The node configuration backup Sim-01.8hour.2015-07-21.09_03_46.7z cannot be created. Error: 'Failed to create tarball for backup Sim-01.8hour.2015-07-21.09_03_46.7z. Error: Failed to create package with name /mroot/etc/backups/config/Sim-01.8hour.2015-07-21.09_03_46.7z'.
7/21/2015 09:14:50 Sim-01 ERROR mgmtgwd.mroot.nearlyfull: The root volume of node 'Sim-01' is nearly full. When it is full the node cannot operate properly.
7/21/2015 09:14:37 Sim-01 ERROR mgmtgwd.mroot.full: The root volume of node 'Sim-01' is full. This prevents the node from operating properly.
7/21/2015 09:07:23 Sim-01 ERROR mgmtgwd.mroot.nearlyfull: The root volume of node 'Sim-01' is nearly full. When it is full the node cannot operate properly.
The system cannot update any of the databases and it is puking on you.
Not sure what or how you are using you simulator, but I have never had the mroot fill on me unless I did something dumb like a packet trace or core files that consumed space.
(even then, I never let the mroot fill up that full)
--tmac
*Tim McCarthy* *Principal Consultant*
On Sun, Jul 26, 2015 at 10:48 AM, Uddhav Regmi uregmi111@gmail.com wrote:
-
Francis Kim -
tmac -
Uddhav Regmi