Nice – I didn’t know about debug log show – that will come in handy You can also pull messages.log via powershell: Read-NcFile -Path /vol/vol0/etc/log/mlog/messages.log -VserverContext <node>
mark
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Parisi, Justin Sent: Tuesday, March 10, 2015 9:31 AM To: Borzenkov, Andrei; Iluhes Cc: Toasters Subject: RE: CMOD /etc/messages of SVM
A few things here…
- EMS logs can be viewed via “event log show” in cluster shell. This is as close to /etc/messages as you need in most cases.
- You can filter EMS with “event log show” by message name, severity, timestamp, etc.
::*> event log show ? [ -detail | -detailtime | -instance | -fields <fieldname>, ... ] [[-node] <nodename>] Node [[-seqnum] <Sequence Number>] Sequence# [ -time <"MM/DD/YYYY HH:MM:SS"> ] Time [ -severity {EMERGENCY|ALERT|CRITICAL|ERROR|WARNING|NOTICE|INFORMATIONAL|DEBUG} ] Severity (default: <DEBUG) [ -ems-severity {NODE_FAULT|SVC_FAULT|NODE_ERROR|SVC_ERROR|WARNING|NOTICE|INFO|DEBUG|VAR} ] *EMS Severity [ -source <text> ] Source [ -messagename <Message Name> ] Message Name [ -event <text> ] Event [ -kernelgen <integer> ] *Kernel Generation Number [ -kernelseqnum <integer> ] *Kernel Sequence Number [ -action <text> ] Corrective Action [ -description <text> ] Description
Example:
::*> event log show -severity ERR Time Node Severity Event ------------------- ---------------- ------------- --------------------------- 3/10/2015 08:33:16 node2 ERROR secd.ldap.connectFailure: vserver (TRUST) could not make a connection over the network to LDAP server (italy) at address (10.228.225.125) and received error (Invalid credentials) 3/10/2015 04:36:11 node2 ERROR secd.ldap.connectFailure: vserver (TRUST) could not make a connection over the network to LDAP server (italy) at address (10.228.225.125) and received error (Invalid credentials) 3/10/2015 00:28:06 node2 ERROR secd.ldap.connectFailure: vserver (TRUST) could not make a connection over the network to LDAP server (italy) at address (10.228.225.125) and received error (Invalid credentials)
With timestamps, you can use specific dates, ranges or just the last few minutes.
::*> event log show -time >30m Time Node Severity Event ------------------- ---------------- ------------- --------------------------- 3/10/2015 09:33:55 node2 NOTICE rdb.ha.verified: Verified that cluster high availability (HA) is configured correctly, and that on-disk mailboxes are intact. 3/10/2015 09:33:54 node1 NOTICE rdb.ha.verified: Verified that cluster high availability (HA) is configured correctly, and that on-disk mailboxes are intact. 3/10/2015 09:30:17 node1 WARNING cmd.quota.throughput: The Counter Manager's Archiver data creation rate is high for the amount of root space available. The system can only keep 0 days of data at this rate, and would need 224956 MB of extra space. 3 entries were displayed.
EMS is mostly what you’d use unless you need to get more granular with troubleshooting and need to look at management host processes like secd, mgwd, etc.
Then you could use systemshell and commands like more, tail, less, etc. Those logs are located in /mroot/etc/mlog.
There is also a way to show those logs from the cluster shell via a diag level command:
::*> set diag
Warning: These diagnostic commands are for use by NetApp personnel only. Do you want to continue? {y|n}: y
::*> debug log files modify -incl-files secd
Then I can use debug log show to view secd logs:
::*> debug log show ? [ -advanced | -instance | -fields <fieldname>, ... ] [[-timestamp] <Date>] *Timestamp [[-node] <nodename>] *Node(s) [[-seq] <integer>] *Sequence [ -log-line <text> ] *Log Line
::*> debug log show -timestamp >24h Time Node Log ------------------------ ----------- ----------------------- Mon Mar 09 11:37:03 2015 node2 [kern_secd:info:4304] Mon Mar 09 11:37:03 2015 node2 [kern_secd:info:4304] .------------------------------------------------------------------------------. Mon Mar 09 11:37:03 2015 node2 [kern_secd:info:4304] | RPC FAILURE: | Mon Mar 09 11:37:03 2015 node2 [kern_secd:info:4304] | secd_rpc_ldap_get_group_info has failed | Mon Mar 09 11:37:03 2015 node2 [kern_secd:info:4304] | Result = 0, RPC Result = 6910 | Mon Mar 09 11:37:03 2015 node2 [kern_secd:info:4304] | RPC received at Mon Mar 9 11:37:03 2015 | Mon Mar 09 11:37:03 2015 node2 [kern_secd:info:4304] |------------------------------------------------------------------------------' Mon Mar 09 11:37:03 2015 node2 [kern_secd:info:4304] | [000.000.028] debug: Worker Thread 34546426048 processing RPC 214:secd_rpc_ldap_get_group_info with request ID:54969 which sat in the queue for 0 seconds. { in run() at secd/server/secd_rpc_server.cpp:2118 } Mon Mar 09 11:37:03 2015 node2 [kern_secd:info:4304] | [000.000.075] debug: Setting thread context. VServerId = 5 (name='NAS'), Protocol = NONE, lifId = 0 { in setThreadContext() at secd/utils/secd_thread_data_manager.cpp:387 }
There is also a little-known command called debug log view, which allows vi-like capability with the cluster shell:
::*> debug log view
View (H for help): Summary of View Mode Commands
h H Display this help. q Q Quit or Exit view mode. ^C Terminate input. Three-in-a-row terminate view mode. ^L r ^R Re-display or repaint the screen.
----------------------------------------------------------------------
MOVING
^N RETURN e ^E j ^J Forward one line. ^P y ^Y k ^K Backward one line. f SPACE ^V f ^F Forward one page of text. b ^B ESC-v Backward one page of text. F Forward forever (follow end of log)
----------------------------------------------------------------------
JUMPING
< ESC-< Jump to the start of the log(s).
ESC-> Jump to the end of the log(s).
----------------------------------------------------------------------
SEARCHING /pat Search forward using the specified pattern ?pat Search backward using the specified pattern ESC-u Toggle highlighting text that matches the pattern
----------------------------------------------------------------------
MISC
vlog Add the specified log to the current view. v+log Add the specified log to the current view. v-log Remove the specified log from the current view. v+* Add all the logs to the current view. v-* Remove all the logs to the current view NOTE: If none left, messages is displayed
List of current logs: apache_access, apache_error, bcomd, cm-daemon, command-history, crs, debug, fpolicy, hashd, messages,
Dseq Toggle display of sequence number and log file name
TR-4303 covers some more info on logging:
http://www.netapp.com/us/system/pdf-reader.aspx?pdfuri=tcm:10-130859-16&...
Also, I wrote a blog post on some CLI ninja tricks if you want to check it out:
https://whyistheinternetbroken.wordpress.com/2015/02/16/techbecome-a-cluster...
From: toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Borzenkov, Andrei Sent: Tuesday, March 10, 2015 1:37 AM To: Iluhes Cc: Toasters Subject: RE: CMOD /etc/messages of SVM
The only way I am aware of is diag account and systemshell. Unfortunately there are many log files and not much information what they contain.
From: Iluhes [mailto:iluhes@yahoo.com] Sent: Tuesday, March 10, 2015 8:30 AM To: Borzenkov, Andrei Cc: Toasters Subject: Re: CMOD /etc/messages of SVM
I have seen that. Thanks Andrei! what I was looking is for as to tail -f to see live messages. Is there a way?
On Mar 10, 2015, at 12:14 AM, "Borzenkov, Andrei" <andrei.borzenkov@ts.fujitsu.commailto:andrei.borzenkov@ts.fujitsu.com> wrote: There is no /etc/messages in cDOT. You can access log directory via HTTPS; it is enabled by default (and documented) since 8.2.1, see KB 1013814 how to enable it on earlier versions.
From: toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Iluhes Sent: Tuesday, March 10, 2015 12:36 AM To: Toasters Subject: CMOD /etc/messages of SVM
Can I still mount what used to be /vol/vol0, aka root volume from NFS SVM and read /etc/messages?