Skip-record processing

Use skip-record processing to skip read and write record errors in transactional batch jobs. Specify skip-record policies in the xJCL.

Skip-record processing

Each batch data stream has its own skip-record policy configuration. You enable skip-record processing by specifying a non-zero value for the com.ibm.batch.bds.skip.count batch data stream property in the xJCL.

We can refine skip record processing using the com.ibm.batch.bds.skip.include.exception.class.<n> property to specify what record errors to skip and the com.ibm.batch.bds.skip.exclude.exception.class.<n> property to specify what record errors not to skip. The two properties are mutually exclusive.

The batch framework tracks skip record processing on a per step basis in the local job status database. This tracking is done only for batch data streams from the batch framework. At the end of step processing, a message is written to the job log. The message indicates the number of records that were skipped per batch data stream and the number of records per second per batch data stream. The number of records per second might not match the number of records that were processed by the batch data stream. If the actual number of records processed took less than a second, the value of the records per second is extrapolated from the amount of time it took to process the actual number of records.

The following list contains each of the skip-record properties followed by a description.

com.ibm.batch.bds.skip.count

Number of records that a batch data stream can skip due to an error in reading or writing a record. After the limit is reached, the batch data stream does not skip any more read or write errors.

When an input record is skipped, the batch data stream moves to the next record and fetches it. Control does not return to the caller until a record is read successfully, an error that does not involve skipping records occurs, or the skip limit is reached.

When an output record is skipped, the batch data stream returns to the caller normally.

If the batch data stream suffers a read or write error after the skip limit is reached, then the read or write exception is returned to the caller. The record is not skipped.

If we register a skip listener with the batch data stream, the skip listener receives control on every skipped record. For skipped reads, the SkipListener.onSkippedRead(Throwable t) method is invoked. The skip listener receives control before the next record is fetched. For skipped writes, the SkipListener.onSkippedWrite(Object record, Throwable t) method is invoked. The skipped record is passed in the first argument. The skip listener receives control before the batch data stream returns to the caller.

The running skip count for a batch data stream persists at every checkpoint. When a job step starts again, the skip count is restored from the last committed checkpoint.

Skip-record processing is disabled by default.

Any batch data stream implementation that extends the com.ibm.websphere.batch.devframework.datastreams.bdsadapter.AbstractBatchDataInputStreamRecordMetrics class or the com.ibm.websphere.batch.devframework.datastreams.bdsadapter.AbstractBatchDataOutputStreamRecordMetrics class automatically inherits skip-record support. All batch data streams defined under the com.ibm.websphere.batch.devframework.datastreams package contain skip-record support.

com.ibm.batch.bds.skip.include.exception.class.<n>

List of exceptions for the batch data stream to skip when reading or writing records. The batch data stream skips only the exceptions on the list.

The <n> variable is an integer. Start the variable at 1 and increment it by one for each exception.

If we do not specify any exceptions, then the default is that all exceptions are included in the list of read/writer errors to skip.

The following example uses the property:

<batch-data-streams>
   <bds>
     <logical-name>inputBDS</logical-name>
     <props>
       <prop name="PATTERN_IMPL_CLASS" value="com.ibm.ws.batch.sample.bds.WCGSampleBDS"/>
       <prop name="file.encoding" value="8859_1"/>
       <prop name="FILENAME" value="/tmp/input.txt" />
       <prop name="com.ibm.batch.bds.skip.count" value="5" />
       <prop name="com.ibm.batch.bds.skip.include.exception.class.1"
             value="java.io.IOException" />
       <prop name="com.ibm.batch.bds.skip.include.exception.class.2"
             value="com.xyz.bds.error.BadDataException" />
     </props>
     <impl-class>com.ibm.websphere.batch.devframework.datastreams.patterns.TextFileReader</impl-class>
   </bds>
</batch-data-streams>

The batch data stream skips records for input/output exceptions and for bad data exceptions.

com.ibm.batch.bds.skip.exclude.exception.class.<n>

List of exceptions that cannot be skipped when reading or writing records.

The <n> variable is an integer. Start the variable at 1 and increment it by one for each exception.

If we do not specify any exceptions, then the default is that no records are excluded from the list of read/write record errors to skip.

The following example uses the property:

<batch-data-streams>
   <bds>
     <logical-name>inputBDS</logical-name>
     <props>
       <prop name="PATTERN_IMPL_CLASS" value="com.ibm.ws.batch.sample.bds.WCGSampleBDS"/>
       <prop name="file.encoding" value="8859_1"/>
       <prop name="FILENAME" value="/tmp/input.txt" />
       <prop name="com.ibm.batch.bds.skip.count" value="3" />
       <prop name="com.ibm.batch.bds.skip.exclude.exception.class.1"
             value="java.io.FileNotFoundException" />
     </props>
     <impl-class>com.ibm.websphere.batch.devframework.datastreams.patterns.TextFileReader</impl-class>
   </bds>
</batch-data-streams>

The batch data stream does not skip records for file not found exceptions.

Skip listeners

We can register a skip listener with a batch data stream to listen for skipped records. The skip listener receives control whenever a record is skipped.

The following example code registers the skip listener.

AbstractBatchDataInputStream _inputBDS =
   (AbstractBatchDataInputStream)BatchDataStreamMgr.getBatchDataStream("inputBDS", getJobStepID());
((AbstractBatchDataStreamRecordMetrics)_inputBDS).addSkipListener(new MySkipListener());

Developing a simple transactional batch application

Retry-step processing