z/OS: Problem Determination

  1. z/OS: Problem Determination

  2. Figures

  3. Tables

  4. About this book

  5. Who should read this book

  6. What we need to know to understand this book

  7. How to use this book

  8. Summary of changes

  9. Changes for this edition (GC34-6600-00)

  10. Establishing the cause of the problem

  11. Introduction to problem determination

  12. How this book can help you

  13. How this book is organized

  14. Preliminary checks

  15. Has WebSphere MQ for z/OS run successfully before?

  16. Are there any error messages or return codes that explain the problem?

  17. Can you reproduce the problem?

  18. Have any changes been made since the last successful run?

  19. Has the application run successfully before?

  20. Does the problem affect specific parts of the network?

  21. Does the problem occur at specific times of the day?

  22. Does the problem affect all users of the application?

  23. Does the problem occur with all z/OS, CICS, or IMS systems?

  24. Is the problem intermittent?

  25. Have you applied any APARs or PTFs?

  26. Do you have a program error?

  27. What to do next

  28. Examining the problem in greater depth

  29. Have you obtained some incorrect output?

  30. Have you received an unexpected error message?

  31. Has there been an abend?

  32. Common causes of abends

  33. Address space dumps and transaction dumps

  34. Abnormal program termination

  35. Have you failed to receive a response from an MQSC command?

  36. Is there a problem with the WebSphere MQ queues?

  37. Are some of your queues working?

  38. Are the correct queues defined?

  39. Does the problem affect only remote or cluster queues?

  40. Does the problem affect only shared queues?

  41. Is your application or WebSphere MQ for z/OS running slowly?

  42. Has your application or WebSphere MQ for z/OS stopped processing work?

  43. Check for error messages

  44. No error messages issued

  45. Has the queue manager or channel initiator terminated abnormally?

  46. WebSphere MQ for z/OS might still be running

  47. Where to look next

  48. Dealing with the problem

  49. Dealing with program abends

  50. Batch abends

  51. CICS transaction abends

  52. IMS transaction abends

  53. Dealing with waits and loops

  54. Distinguishing between waits and loops

  55. Waits

  56. Loops

  57. Symptoms of waits and loops

  58. Dealing with waits

  59. Is a batch or TSO program waiting?

  60. Is a CICS transaction waiting?

  61. Is DB2 waiting?

  62. Is RRS active?

  63. Is WebSphere MQ for z/OS waiting?

  64. Dealing with loops

  65. Is a batch application looping?

  66. Is a batch job producing a large amount of output?

  67. Does a CICS region show a lot of CPU activity?

  68. Does an IMS region show a lot of CPU activity?

  69. Is the queue manager showing a lot of CPU activity?

  70. Is a queue, page set, or Coupling Facility structure filling up unexpectedly?

  71. Are a task, and WebSphere MQ for z/OS, showing a lot of CPU activity?

  72. Dealing with performance problems

  73. z/OS system considerations

  74. WebSphere MQ for z/OS considerations

  75. Log buffer pools

  76. Buffer pool size

  77. Distribution of data sets on available DASD

  78. Distribution of queues on page sets

  79. Distribution of queues on Coupling Facility structures

  80. Limitation of concurrent threads

  81. Using the WebSphere MQ trace for administration

  82. CICS constraints

  83. Application design considerations

  84. Effect of message length

  85. Effect of message persistence

  86. Searching for a particular message

  87. Queues that contain messages of different lengths

  88. Frequency of syncpoints

  89. Advantages of the MQPUT1 call

  90. Dealing with incorrect output

  91. Messages do not appear when expected

  92. Problems with missing messages when using distributed queuing

  93. Problems with getting messages when using message grouping

  94. Finding messages sent to a cluster queue

  95. Finding messages sent to the WebSphere MQ-IMS bridge

  96. Messages contain unexpected or corrupted information

  97. Diagnostic aids and techniques

  98. Diagnostic aids

  99. WebSphere MQ for z/OS recovery actions

  100. Program errors

  101. WebSphere MQ for z/OS abends

  102. WebSphere MQ for z/OS diagnostic information

  103. z/OS abends

  104. CICS abends

  105. IMS abends

  106. Diagnostic information produced

  107. Error messages

  108. Dumps

  109. Console logs and job output

  110. Symptom strings

  111. Queue information

  112. Other sources of information

  113. Your own documentation

  114. Manuals for the products you are using

  115. Source listings and link-edit maps

  116. Change log

  117. System configuration charts

  118. Information from the DISPLAY CONN command

  119. Diagnostic aids for CICS

  120. Diagnostic aids for IMS

  121. Diagnostic aids for DB2

  122. WebSphere MQ dumps

  123. How to use dumps for problem determination

  124. Getting a dump

  125. Using the z/OS DUMP command

  126. Processing a dump

  127. Using the WebSphere MQ for z/OS dump display panels

  128. Using line mode IPCS

  129. Using IPCS in batch

  130. Analyzing the dump

  131. Dump title variation with PSW and ASID

  132. SYSUDUMP information

  133. Snap dumps

  134. SYS1.LOGREC information

  135. Finding the applicable SYS1.LOGREC information

  136. When SVC dumps are not produced

  137. Suppressing WebSphere MQ for z/OS dumps using z/OS DAE

  138. Using trace for problem determination

  139. The user parameter and IBM internal traces

  140. Starting the trace

  141. Formatting the information

  142. If trace data is not produced

  143. Interpreting the information

  144. The Log and Trace Analyzer tool

  145. Examples of trace output

  146. Other types of trace

  147. The channel initiator trace

  148. The line trace

  149. The CICS adapter trace

  150. System SSL trace

  151. z/OS traces

  152. Finding solutions to similar problems

  153. Searching the IBM database

  154. Search argument process

  155. Techniques for varying the search

  156. The keyword format

  157. Free format

  158. Structured database (SDB) format

  159. Building a keyword string

  160. Component-identifier keyword

  161. Release-level keyword

  162. Type-of-failure keyword

  163. Abend keyword

  164. IEA911E message

  165. CSQV086E message

  166. CSECT keyword

  167. Load module modifier keyword

  168. Recovery routine modifier keyword

  169. Wait and loop keywords

  170. Message keyword

  171. Procedure for WebSphere MQ for z/OS messages

  172. Performance keyword

  173. Documentation keyword

  174. Incorrect output keyword

  175. Working with IBM to solve your problem

  176. Reporting a problem to the IBM software support group

  177. When to contact the support center

  178. Dealing with the support center

  179. What the support center needs to know

  180. What happens next

  181. Collecting documentation for the problem

  182. General documentation needed for all problems with WebSphere MQ for z/OS

  183. Sending the documentation to the change team

  184. Resolving a problem

  185. The APAR process

  186. Applying the fix

  187. The APAR becomes a PTF

  188. Appendixes

  189. Appendix A. SDB format symptom-to-keyword cross reference

  190. Appendix B. WebSphere MQ component and resource manager identifiers

  191. Appendix C. CICS adapter trace entries

  192. Appendix D. Examples of CEDF output

  193. MQOPEN

  194. MQCLOSE

  195. MQPUT

  196. MQPUT1

  197. MQGET

  198. MQINQ

  199. MQSET

  200. Appendix E. Notices

  201. Programming interface information

  202. Trademarks

  203. Index