Data Load utility framework process and components

The data load framework loads your input data into your target database.

Note: We can load data into a workspace. When you load data into a workspace, the Data Load utility respects the locking policy set in the workspace. However, the table object mediator does not support workspace locking. The data load framework consists of four main components:

1. DataReader: The DataReader reads the input data from a data source and returns an object that is passed to the BusinessObjectBuilder.

2. BusinessObjectBuilder: The BusinessObjectBuilder populates a data object based on the object that is passed from the DataReader. The data object is then passed to the BusinessObjectMediator.

3. BusinessObjectMediator: The BusinessObjectMediator transforms the data object into a list of physical objects that is then passed to the DataWriter.

4. DataWriter: The DataWriter saves the physical objects to the database using JDBC or a list file in the database native loadable format.

The DataReader, BusinessObjectBuilder, BusinessObjectMediator, and DataWriter are also names for the interfaces. The implementation of these interfaces is defined in the data load business object configuration file. The following code snippet is a sample data business object configuration file:

<_config:DataloadBusinessObjectConfiguration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.ibm.com/xmlns/prod/commerce/foundation/config ../../../xml/config/xsd/wc-dataload-businessobject.xsd"
  xmlns:_config="http://www.ibm.com/xmlns/prod/commerce/foundation/config">
  <_config:DataLoader className="com.ibm.commerce.foundation.dataload.BusinessObjectLoader">
    <_config:DataReader className="com.ibm.commerce.foundation.dataload.datareader.CSVReader" firstLineIsHeader="true" useHeaderAsColumnName="true"/>
      <_config:BusinessObjectBuilder className="com.ibm.commerce.foundation.dataload.businessobjectbuilder.BaseBusinessObjectBuilder"
        packageName="com.ibm.commerce.catalog.facade.datatypes.CatalogPackage" dataObjectType="CatalogGroupType" >
        <_config:DataMapping>
          <_config:mapping xpath="CatalogGroupIdentifier/ExternalIdentifier/GroupIdentifier" value="GroupIdentifier" />
          <_config:mapping xpath="displaySequence" value="Sequence" />
          <_config:mapping xpath="Attributes/field1" value="Field1"/>
          <_config:mapping xpath="Attributes/field2" value="Field2"/>
          <_config:mapping xpath="" value="Delete"  deleteValue="1"/>
        </_config:DataMapping>
        <_config:BusinessObjectMediator className="com.ibm.commerce.catalog.dataload.mediator.CatalogGroupMediator" componentId="com.ibm.commerce.catalog" >
        <_config:DataWriter className="com.ibm.commerce.foundation.dataload.datawriter.JDBCDataWriter" />
      </_config:BusinessObjectMediator>
    </_config:BusinessObjectBuilder>
  </_config:DataLoader>
</_config:DataloadBusinessObjectConfiguration>

When you define the data load business object configuration file, ensure that you specify the right implementation class for the DataReader, BusinessObjectBuilder, BusinessObjectMediator, and DataWriter. From the interface, the object flow from the DataReader to the BusinessObjectBuilder is a generic Java object. Similarly, the object flow from the BusinessObjectBuilder to the BusinessObjectMediator, and from the BusinessObjectMediator to the DataWriter are all generic Java objects. The specific implementation class expects that a specific type of object is passed around. For example, the CSVReader reads a line of data from a CSV file and returns a Map. Therefore, BaseBusinessObjectBuilder is expecting to have a map that is passed in. So the CSVReader and the BaseBusinessObjectBuilder can be used together.

DataReader

The DataReader is an interface for a physical data reader. The following classes implement the DataReader interface:

• CSVReader

  This class reads the contents of a CSV file, one line at a time, and builds a Map object. The key in the Map is either specified in the configuration or the first line of the CSV file.

• XMLReader

  This class reads the contents of an XML file, one element at a time, and builds a Map object. The key in the Map is either specified in the configuration file or in the root element of the XML file.

See the com.ibm.commerce.foundation.dataload.datareader.DataReader API for more information.

• CSV

  <_config:DataReader className="com.ibm.commerce.foundation.dataload.datareader.CSVReader" firstLineIsHeader="true" useHeaderAsColumnName="true"/>

• XML

  <_config:DataReader className="com.ibm.commerce.foundation.dataload.datareader.XMLReader" />

BusinessObjectBuilder

The BusinessObjectBuilder is an interface for building a business object. The following classes implement the BusinessObjectBuilder interface:

• BaseBusinessObjectBuilderThis class populates a business object that is based on the input object. It expects the input object to be a Map object. It builds the specific business object that is based on the attributes specified in the configuration: packageName and dataObjectType. The business object is passed to the instance of the BusinessObjectMediator specified in the configuration.

  Note: Use this class if the implementation class of the DataReader is the CSVReader, and the BusinessObjectMediator class is expecting a business object as an input.

• TableObjectBuilderThis class populates a list of ExtendedTableDataObject based on the table/column definition specified in the configuration. The list of ExtendedTableDataObject can be passed into the TableObjectMediator.

  Note: Use this class if the implementation class of the BusinessObjectMediator is the TableObjectMediator.

See the com.ibm.commerce.foundation.dataload.businessobjectbuilder.BusinessObjectBuilder API for more information.

<_config:BusinessObjectBuilder className="com.ibm.commerce.foundation.dataload.businessobjectbuilder.BaseBusinessObjectBuilder" 
	packageName="com.ibm.commerce.catalog.facade.datatypes.CatalogPackage" dataObjectType="CatalogGroupType" >

BusinessObjectMediator

The BusinessObjectMediator is an interface for transforming a business object into a list of physical objects. The following classes implement the BusinessObjectMediator interface:

• WebSphere Commerce logic noun-based mediatorThere are several implementation classes available for the following components:

  • catalog

  • inventory

  • price

  • member

• Table-based mediator

  The implemented class for this mediator is the TableObjectMediator. It can be used with the TableObjectBuilder.

See the com.ibm.commerce.foundation.dataload.businessobjectmediator.BusinessObjectMediator API for more information.

<_config:BusinessObjectMediator className="com.ibm.commerce.catalog.dataload.mediator.CatalogGroupMediator" componentId="com.ibm.commerce.catalog" >

DataWriter

The DataWriter is an interface for a physical data writer. The following classes implement the DataWriter interface:

• JDBCDataWriter

  This class writes the physical objects that are created by the BusinessObjectMediator directly into the database. The JDBC data writer persists the physical object into the database directly with the JDBC batch APIs. Initial loads can be configured to use either the JDBC data writer or the native file data writer. Delta loads should be configured to use the JDBC data writer.

• NativeDBDataWriterThis class generates only database native loadable files. The native file data writer persists the physical object into a file in a native database loadable format. This file can then be loaded into the database with the database native load utility. Initial loads that require large amounts of data can be configured to generate and load data with this database native load file format for optimum performance.

  Note: The NativeDBDataWriter supports only DB2 and Oracle.

See the com.ibm.commerce.foundation.dataload.datawriter.DataWriter API for more information.

<_config:DataWriter className="com.ibm.commerce.foundation.dataload.datawriter.JDBCDataWriter" />