Use centralized logging to diagnose problems
We can use centralized logging to enable mustGather traces, perform a per-request trace analysis, and follow the flow of a request through both the ODR and application server tiers.
- Centralized logging is a tool to help collect the data necessary for troubleshooting and diagnosing a problem. However, for performance reasons, the features of centralized logging are not enabled all the time. Therefore, enable the centralized logging features when you anticipate capturing a particular problem.
- When you run the mustgather.py script, trace specifications are overwritten. To restore the trace specifications, save the trace strings before running the script.
- If we use the remote log collection feature of the mustgather.py script, ensure that the target file system has the appropriate amount of space necessary to contain the gathered log files.
Use centralized logging, we can enable tracing based on the type of problem that you experience, for example, 503 HTTP response codes. With centralized logging, specific types of mustGather documents can be identified by predefined strings. Some examples of predefined mustGather documents include application editioning, 404 error and 503 response codes, health monitoring, and more.
We can also enable rule-based tracing which allows us to trace requests based on specific rules.
- Run mustgather.py to enable tracing, collect mustGather documents, and disable tracing.
All the necessary logs within the cell are compiled on the system where the script runs. The supported commands are included in the following list:
- enable [mustgatherType]
- Enables tracing
- collect [mustgatherType] [destination]
- Collects a specific type of mustGather document
- disable [mustgatherType]
- Disables tracing
- Determine the type of mustGather document that we need. The supported types of mustGather documents are included in the following list:
404 404 HTTP response code 503 503 HTTP response code 504 504 HTTP response code agent node agent appedition Application edition manager arfm Autonomic request flow manager dc Dynamic cluster hadmgr High availability deployment manager hmm Health monitoring odr On demand router operations Visualization issues with the Extended Deployment and Operations tabs reports Visualization issues with the Reports tab reportsPerf Visualization issues with performance data that is displayed on the Reports tab repository Extended Repository Service sip SIP request routing
- Run the enable command with the appropriate mustGather type specified to set the appropriate tracing on all relevant servers in the cell.
wsadmin -lang jython -f c:\WebSphere\AppServer\bin\mustgather.py enable 404
- Recreate the desired scenario.
- Run the collect command with the appropriate mustGather type and a destination file specified to collect the local and remote mustGather documents.
wsadmin -lang jython -f c:\WebSphere\AppServer\bin\mustgather.py collect 404 "c:\\mydocs\\collection.zip"
- Run the disable command with the appropriate mustGather type specified. Running this command sets the trace to *=info on all relevant servers in the cell.
wsadmin -lang jython -f c:\WebSphere\AppServer\bin\mustgather.py disable 404
- Run the setupReqBasedTracing.py script to enable or disable request-based tracing rules. The supported commands are included in the following list:
- enableReqBasedTracing
- Sets up a request-based tracing rule. A rule consists of an expression and identifier called a rule ID. Optionally, a rule can also contain ODR trace and application server trace strings. Start and end markers are placed in the log files for requests that match one or more rules.
The ODR logs a start marker when a request that matches one or more rules arrives and logs an end marker before the request is dispatched to the backend application server. The ODR also logs a start marker when a response is received from the application server and an end marker after response is sent back to the user. The application server logs a start marker when a matched request arrives from the ODR and an end marker before a response is sent to the ODR. These markers make it possible to find a particular request, or set of requests, and to correlate them with processing of that request on the application server.
- listRuleIDs
- Lists all the rules. This command will output all the rules that are set on all the ODRs. After an ODR is restarted, the rules must be recreated.
- disableReqBasedTracing
- Disables a request-based tracing rule.
For more information on rule expressions, read about HTTP operands.
- Enable request-based tracing.
./wsadmin.sh -lang jython -f setupReqBasedTracing.py enableReqBasedTracing -ruleExpression:<expression> -odrTraceSpec:<trace strings> -appServerTraceSpec:<trace string> -ruleID:<rule ID>
where
- -ruleExpression:<expression>
- Specifies an expression used to match the requests. (Required)
- -odrTraceSpec:<trace string>
- Specifies an ODR trace string set at run time for the requests that match the specified expression. If the parameter is not specified, the trace specification is not dynamically set. (Optional)
- -appServerTraceSpec:<trace string>
- Specifies an application server trace string set at run time for the requests that match the specified expression. If the parameter is not specified, the trace specification is not dynamically set. (Optional)
- -ruleID:<rule ID>
- ID for the request-based tracing rule. If the parameter is not specified, a rule ID is generated by the script in the form of ruleID-<time stamp>. (Optional)
- List all the rules.
./wsadmin.sh -lang jython -f setupReqBasedTracing.py listRuleIDs
- Disable request-based tracing.
./wsadmin.sh -lang jython -f setupReqBasedTracing.py disableReqBasedTracing -ruleIDs:<rule ID1>,<rule ID2>...,<ruleIDn>
where
- -ruleIDs:<rule ID1>,<rule ID2>...,<ruleIDn>
- List of rule IDs to disable. (Required)
Intelligent Management: HTTP operands