(ZOS) Use RRS panels to resolve InDoubt units of recovery

Use this task to better understand messages received when using peer restart and recovery.

Peer Restart and Recovery (PRR) functionality is deprecated. We should use the integrated high availability support for the transaction service subcomponent, instead of Peer Restart and Recovery for transaction recovery. See the topic Transaction support in WAS for more information about the integrated high availability support for the transaction service subcomponent.

There are RRS version requirements that we must heed when using peer restart and recovery. For more information on these requirements, see z/OS MVS™ Programming: Resource Recovery.

If we receive the console message:

BBOT0015D OTS UNABLE TO RESOLVE ALL INCOMPLETE TRANSACTIONS FOR SERVER
string.  REPLY CONTINUE OR TERMINATE.

Tasks

1. Note the server name specified for string. in the message.
2. Go to the SYSPRINT, that is the status queue for that server, and search for messages BBOT0019 - BBOT0022 that refer to that server.

3. Read the resulting messages.

4. Stop the server.

   RRS does not allow an operator to resolve an InDoubt UR if the DSRM for that UR is active at the time. Therefore, we must stop the server. To do this, reply TERMINATE to the CONTINUE/TERMINATE WTOR.

What to do next

Use the following non-console messages to trigger restart and recovery automation. These messages provide information about daemon activities. Pay particular attention to the URID, XID FormatId, XID Gtrid, and XIDBqual attributes. We need to use these pieces of information when we manually resolve the relevant units of work via the RRS panels.

• BBOO003E WEBSPHERE FOR z/OS CONTROL REGION string ENDED ABNORMALLY, REASON= hstring.
• BBOO009E WEBSPHERE FOR z/OS DAEMON string ENDED ABNORMALLY, REASON= hstring.
• BBOO0171I WEBSPHERE FOR z/OS CONTROL REGION string NOT STARTING ON CONFIGURED SYSTEM string
• The following messages, which are written only in recovery and restart mode, provide details about transactions that cannot be resolved during restart and recovery:
  • BBOT0008I TRANSACTION SERVICE RESTART INITIATED ON SERVER string
  • BBOT0009I TRANSACTION SERVICE RESTART UR STATUS COUNTS FOR SERVER string: IN-BACKOUT= dstring, IN-DOUBT= dstring, IN-COMMIT= dstring
  • BBOT0010I TRANSACTION SERVICE RESTART AND RECOVERY ON SERVER string IS COMPLETE
  • BBOT0011I SERVER string IS COLD STARTING WITH RRS
  • BBOT0012I SERVER string IS WARM STARTING WITH RRS
  • BBOT0013I TRANSACTION SERVICE RESTART AND RECOVERY ON SERVER string IS COMPLETE. THE SERVER IS STOPPING.
  • BBOT0014I TRANSACTION SERVICE RECOVERY PROCESSING FOR RRS URID ' string ' IN SERVER string IS COMPLETE.
  • BBOT0016I TRANSACTION SERVICE RESTART AND RECOVERY FOR SERVER string IS NOT COMPLETE. THE SERVER IS STOPPING DUE TO OPERATOR REPLY.
  • BBOT0017I TRANSACTION SERVICE RESTART AND RECOVERY FOR SERVER string IS CONTINUING DUE TO OPERATOR REPLY.
  • BBOT0018I TRANSACTION SERVICE RESTART AND RECOVERY FOR SERVER string IS STILL PROCESSING dstring INCOMPLETE UNIT(S) OF RECOVERY.
  • BBOT0019W UNABLE TO RESOLVE THE OUTCOME OF THE TRANSACTION BRANCH DESCRIBED BY URID: ' string ' XID FORMATID: ' string ' XID GTRID: ' string ' XID BQUAL: ' string ' BECAUSE THE OTS RECOVERY COORDINATOR FOR SERVER string ON HOST string: dstring COULD NOT BE REACHED.
  • BBOT0020W UNABLE TO PROVIDE THE SUBORDINATE OTS RESOURCE IN SERVER string ON HOST string: dstring WITH THE OUTCOME OF THE TRANSACTION DESCRIBED ON THIS SERVER BY URID: ' string ' XID FORMATID: ' string' XID GTRID: ' string ' XID BQUAL: ' string ' BECAUSE THIS SERVER HAS BEEN UNABLE TO RESOLVE THE OUTCOME WITH A SUPERIOR NODE.
  • BBOT0021W UNABLE TO string THE SUBORDINATE OTS RESOURCE IN SERVER string ON HOST string: dstring FOR THE TRANSACTION DESCRIBED ON THIS SERVER BY URID: ' string ' XID FORMATID: ' string ' XID GTRID: ' string ' XID BQUAL: ' string ' OR ANOTHER RESOURCE INVOLVED IN THIS UNIT OF RECOVERY BECAUSE ONE OR MORE RESOURCES COULD NOT BE REACHED OR HAVE NOT YET REPLIED.
  • BBOT0022W UNABLE TO FORGET THE TRANSACTION WITH HEURISTIC OUTCOME DESCRIBED ON THIS SERVER BY URID: ' string ' XID FORMATID: 'string ' XID GTRID: ' string ' XID BQUAL: ' string ' BECAUSE THE SUPERIOR COORDINATOR FOR SERVER string ON HOST string: dstring HAS NOT INVOKED FORGET ON THE REGISTERED RESOURCE.

Tips for RRS Operations

See z/OS MVS Programming: Resource Recovery, for RRS operations guidelines.

Tips for RRS operations:

• If we have configured your log streams to the coupling facility, then monitor the log streams to ensure offload is not occurring. RRS will perform better if its recovery logs do not offload.

  Proper sizing of the RRS logs is important. Too small and we get reduced throughput since logger is off-loading the logs too frequently. Too large and we could overflow your coupling facility..

• Keep the main and delayed (only contains active or live data) logs in your coupling facility. Make sure the CF definitions don't overflow.

  A commit cannot occur until the log record is written..

• Until you stabilize your workloads, it is a good idea to use the archive log. If we have an archive log configured, RRS will unconditionally use it. However, there is a performance penalty for using it.

Subtopics

• Resolving InDoubt units if we receive a BBOT00xxW message
  If we receive a BBOT00xxW message, we must use RRS panels to view the outcome of other branches in the transaction and set the outcomes of InDoubt units to match the outcome of those other branches.

• Resource Recovery Services Operations
  This topic provides tips for using the z/OS Resource Recovery Services with this product.

Related:

Transactional high availability

Resource Recovery Services (RRS)