WLM Delay Monitoring

(ZOS) WLM Delay Monitoring

WebSphere Application Server for z/OS uses Workload Manager (WLM) services to report transaction begin-to-end response times and execution delay times.

The WLM data collected by Resource Measurement Facility (RMF™) is captured in two phases of the RMF report:

BTE - the begin-to-end phase applies to requests handled by the controller
EXE - the execution phase applies to requests handled by the servant
Use this status information to determine where possible performance bottlenecks are occurring. This feature is available on z/OS V1R2 and with WLM APAR OW51848 and RMF APAR OW52227.
For details on workload management, see z/OS MVS™ Planning: Workload Management. For details on WLM delay monitoring, refer to z/OS MVS Programming: Workload Management Services. Both are available on the z/OS Internet Library website.

When a new transaction enters the system, the WAS for z/OS application control region (ACR) starts the classify service. Delays associated with the WAS for z/OS ACR service class are counted separately for the BTE phase and the EXE phase. This support allows WLM to associate a performance block (PB) with an enclave to record delays that occur in the flow of a transaction. The state samples are collected on an ongoing basis and reported as a percentage of average transaction response time. The following table shows the states, their codes, the section of the RMF report where each is reported, the meaning, and suggested response.

State Code Report Meaning Response
ACTIVE ACTIVE SUB Both BTE and EXE WebSphere is actively processing request
ACTIVE_APPLIC ACTIVE APPL Both BTE and EXE Application is running Use application monitoring tool to determine the cause of the delay.
WAITING TYPE1 TYP1
Not used
WAITING TYPE2 TYP2
Not used

WAITING TYPE3 TYP3
Not used
WAITING TYPE 4 TYP4 BTE OTS call to RRS. Occurs only in controller when controller is trying to commit a distributed transaction.

Investigate the delay on the other server.
Look for any network problems.
Consider combining application into one server to avoid delay.

WAITING REGIST TO WORKTABLE WORK
Not used

WAITING OTHER_PRODUCT OTHER
Not used

WAITING DISTRIB DIST BTE Controller as a client went outbound waiting for a response.

Investigate the delay on the other server.
Look for any network problems.
Consider combining application into one server to avoid delay.

WAITING SESS_NETWORK REMT
Not used

WAITING SESS_SYSPLEX SYSP
Not used

WAITING REGULAR_THREAD REGT BTE Waiting for a thread in the controller. Work is bottlenecked in the controller because it is receiving more requests than it can process. Create a new Server in the same cluster. By doing so, you now have two control regions to share the work and are able to spread out processing to forestall any processing contention.
WAITING SSL_THREAD SSLT BTE Waiting for an SSL thread in the controller. Work is bottlenecked in the controller because it is receiving more requests for SSL handshakes than it can process. Create a new Server in the same cluster. By doing so, you now have two control regions to share the work and are able to spread out processing to forestall any processing contention.

Increase SSL threads.
Look at SSL configuration.
Create a new Server in the same cluster.

WAITING SESS_LOCALMVS LOCL
Not used
WAITING TYPE6 TYP6 EXE Indicates a delay in ZIOP processing. Contact IBM Support for assistance.
WAITING TYPE7 TYP7 BTE Indicates a configuration problem in DNS or TCP/IP

Make sure all of the DNS servers are running.
Look at OPING or ONSLOOKUP.

WAITING TYPE8 TYP8 EXE J2C Resource manager delay

Call a J2C connector to resource managers such as DB2, IMS, and CICS.
Check other resource managers using their monitoring tools.

WAITING TYPE9 TYP9
Not used
WAITING TYPE10 TY10 EXE Servant called to the CR to perform one of the following actions:

LocateFunction
ControlFunction
FlushService
FRCAService
PushService
GIOPOutboundResponseFragmentService
GIOPOutboundRequestFragmentService
SIBusMessageListenerDispatch

Check for a delay in the controller.
WAITING TYPE11 TY11 EXE Servant issued a call to the controller, and the controller then issued a call to a different distributed object server using RMI/IIOP.

Investigate the delay on the other server. The delay might point to session caches.
Look for network problems.
Avoid outbound calls.

WAITING TYPE12 TY12
Not used
WAITING TYPE13 TY13
Not used
WAITING TYPE14 TY14
Not used
WAITING TYPE15 TY15
Not used

Monitor overall system health
RMF report examples

State	Code	Report	Meaning	Response
ACTIVE	ACTIVE SUB	Both BTE and EXE	WebSphere is actively processing request
ACTIVE_APPLIC	ACTIVE APPL	Both BTE and EXE	Application is running	Use application monitoring tool to determine the cause of the delay.
WAITING TYPE1	TYP1		Not used
WAITING TYPE2	TYP2		Not used
WAITING TYPE3	TYP3		Not used
WAITING TYPE 4	TYP4	BTE	OTS call to RRS. Occurs only in controller when controller is trying to commit a distributed transaction.	Investigate the delay on the other server. Look for any network problems. Consider combining application into one server to avoid delay.
WAITING REGIST TO WORKTABLE	WORK		Not used
WAITING OTHER_PRODUCT	OTHER		Not used
WAITING DISTRIB	DIST	BTE	Controller as a client went outbound waiting for a response.	Investigate the delay on the other server. Look for any network problems. Consider combining application into one server to avoid delay.
WAITING SESS_NETWORK	REMT		Not used
WAITING SESS_SYSPLEX	SYSP		Not used
WAITING REGULAR_THREAD	REGT	BTE	Waiting for a thread in the controller. Work is bottlenecked in the controller because it is receiving more requests than it can process.	Create a new Server in the same cluster. By doing so, you now have two control regions to share the work and are able to spread out processing to forestall any processing contention.
WAITING SSL_THREAD	SSLT	BTE	Waiting for an SSL thread in the controller. Work is bottlenecked in the controller because it is receiving more requests for SSL handshakes than it can process.	Create a new Server in the same cluster. By doing so, you now have two control regions to share the work and are able to spread out processing to forestall any processing contention. Increase SSL threads. Look at SSL configuration. Create a new Server in the same cluster.
WAITING SESS_LOCALMVS	LOCL		Not used
WAITING TYPE6	TYP6	EXE	Indicates a delay in ZIOP processing.	Contact IBM Support for assistance.
WAITING TYPE7	TYP7	BTE	Indicates a configuration problem in DNS or TCP/IP	Make sure all of the DNS servers are running. Look at OPING or ONSLOOKUP.
WAITING TYPE8	TYP8	EXE	J2C Resource manager delay	Call a J2C connector to resource managers such as DB2, IMS, and CICS. Check other resource managers using their monitoring tools.
WAITING TYPE9	TYP9		Not used
WAITING TYPE10	TY10	EXE	Servant called to the CR to perform one of the following actions: LocateFunction ControlFunction FlushService FRCAService PushService GIOPOutboundResponseFragmentService GIOPOutboundRequestFragmentService SIBusMessageListenerDispatch	Check for a delay in the controller.
WAITING TYPE11	TY11	EXE	Servant issued a call to the controller, and the controller then issued a call to a different distributed object server using RMI/IIOP.	Investigate the delay on the other server. The delay might point to session caches. Look for network problems. Avoid outbound calls.
WAITING TYPE12	TY12		Not used
WAITING TYPE13	TY13		Not used
WAITING TYPE14	TY14		Not used
WAITING TYPE15	TY15		Not used