Develop > Persistence layer > Data assets > Work with the Data load utility > Data load architectural overview

Data load framework process and components

The data load framework loads the input data into the target database.

You can load data into a workspace. When loading data into a workspace, the data load utility respects the locking policy set in the workspace. However, the table object mediator does not support workspace locking.

The data load framework consists of four main components:

1. DataReader: The DataReader reads the input data from a data source and returns an object which is passed to the BusinessObjectBuilder.

2. BusinessObjectBuilder: The BusinessObjectBuilder populates a data object based on the object passed from the DataReader. The data object is then passed to the BusinessObjectMediator.

3. BusinessObjectMediator: The BusinessObjectMediator transforms the data object into a list of physical objects which is then passed to the DataWriter.

4. DataWriter: The DataWriter saves the physical objects to the database using JDBC or a list file in the database native loadable format.

The DataReader, BusinessObjectBuilder, BusinessObjectMediator, and DataWriter are also names for the interfaces. The implementation of these interfaces is defined in the data load business object configuration file. The following code snippet is a sample data business object configuration file:

<_config:DataloadBusinessObjectConfiguration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.ibm.com/xmlns/prod/commerce/foundation/config ../../../xml/config/xsd/wc-dataload-businessobject.xsd"
    xmlns:_config="http://www.ibm.com/xmlns/prod/commerce/foundation/config">     <_config:DataLoader className="com.ibm.commerce.foundation.dataload.BusinessObjectLoader">     <_config:DataReader className="com.ibm.commerce.foundation.dataload.datareader.CSVReader" firstLineIsHeader="true" useHeaderAsColumnName="true"/>  
<_config:BusinessObjectBuilder className="com.ibm.commerce.foundation.dataload.businessobjectbuilder.BaseBusinessObjectBuilder" 
    packageName="com.ibm.commerce.catalog.facade.datatypes.CatalogPackage" dataObjectType="CatalogGroupType" >         <_config:DataMapping>               <_config:mapping xpath="CatalogGroupIdentifier/ExternalIdentifier/GroupIdentifier" value="GroupIdentifier" />             <_config:mapping xpath="displaySequence" value="Sequence" />             <_config:mapping xpath="Attributes/field1" value="Field1"/>             <_config:mapping xpath="Attributes/field2" value="Field2"/>             <_config:mapping xpath="" value="Delete"  deleteValue="1"/>         </_config:DataMapping>           <_config:BusinessObjectMediator className="com.ibm.commerce.catalog.dataload.mediator.CatalogGroupMediator" componentId="com.ibm.commerce.catalog" >               <_config:DataWriter className="com.ibm.commerce.foundation.dataload.datawriter.JDBCDataWriter" />           </_config:BusinessObjectMediator>       </_config:BusinessObjectBuilder>  
</_config:DataLoader>
</_config:DataloadBusinessObjectConfiguration>

When defining the data load business object configuration file, ensure that you specify the right implementation class for the DataReader, BusinessObjectBuilder, BusinessObjectMediator, and DataWriter. From the interface, the object flow from the DataReader to the BusinessObjectBuilder is a generic Java object. Similarly, the object flow from the BusinessObjectBuilder to the BusinessObjectMediator, and from the BusinessObjectMediator to the DataWriter are all generic Java objects. The specific implementation class expects that a specific type of object is passed around. For example, the CSVReader reads a line of data from a CSV file and returns a Map. Therefore, BaseBusinessObjectBuilder is expecting to have a Map passed in. So the CSVReader and the BaseBusinessObjectBuilder can be used together.

DataReader

The DataReader is an interface for a physical data reader. The following classes implement the DataReader interface:

• CSVReader

  This class reads the contents of a CSV file, one line at a time, and builds a Map object. The key in the Map is either specified in the configuration or the first line of the CSV file.

See the com.ibm.commerce.foundation.dataload.datareader.DataReader API for more information.

<_config:DataReader className="com.ibm.commerce.foundation.dataload.datareader.CSVReader" firstLineIsHeader="true" useHeaderAsColumnName="true"/>

BusinessObjectBuilder

The BusinessObjectBuilder is an interface for building a business object. The following classes implement the BusinessObjectBuilder interface:

• BaseBusinessObjectBuilder

  This class populates a business object based on the input object. It expects the input object to be a Map object. It builds the specific business object based on the attributes specified in the configuration: packageName and dataObjectType. The business object is passed to the instance of the BusinessObjectMediator specified in the configuration.

  Use this class if the implementation class of the DataReader is the CSVReader, and the BusinessObjectMediator class is expecting a business object as an input.

• TableObjectBuilder

  This class populates a list of ExtendedTableDataObject based on the table/column definition specified in the configuration. The list of ExtendedTableDataObject can be passed into the TableObjectMediator.

  Use this class if the implementation class of the BusinessObjectMediator is the TableObjectMediator.

See the com.ibm.commerce.foundation.dataload.businessobjectbuilder.BusinessObjectBuilder API for more information.

<_config:BusinessObjectBuilder className="com.ibm.commerce.foundation.dataload.businessobjectbuilder.BaseBusinessObjectBuilder" 
    packageName="com.ibm.commerce.catalog.facade.datatypes.CatalogPackage" dataObjectType="CatalogGroupType" >

BusinessObjectMediator

The BusinessObjectMediator is an interface for transforming a business object into a list of physical objects. The following classes implement the BusinessObjectMediator interface:

• WebSphere Commerce logic noun-based mediator

  There are several implementation classes available for the catalog, inventory, and price components.

• Table-based mediator

  The implemented class for this mediator is the TableObjectMediator. It can be used with the TableObjectBuilder.

See the com.ibm.commerce.foundation.dataload.businessobjectmediator.BusinessObjectMediator API for more information.

<_config:BusinessObjectMediator className="com.ibm.commerce.catalog.dataload.mediator.CatalogGroupMediator" componentId="com.ibm.commerce.catalog" >

DataWriter

The DataWriter is an interface for a physical data writer. The following classes implement the DataWriter interface:

• JDBCDataWriter

  This class writes the physical objects created by the BusinessObjectMediator directly into the database. The JDBC data writer persists the physical object into the database directly using the JDBC batch APIs. Initial loads can be configured to use either the JDBC data writer or the native file data writer. Delta loads should be configured to use the JDBC data writer.

• NativeDBDataWriter

  This class only generates database native loadable files. The native file data writer persists the physical object into a file in a native database loadable format. This file can then be loaded into the database using the database native load utility. Initial loads requiring large amounts of data can be configured to generate and load data using this database native load file format for optimum performance.

  The NativeDBDataWriter only supports DB2 and Oracle.

See the com.ibm.commerce.foundation.dataload.datawriter.DataWriter API for more information.

<_config:DataWriter className="com.ibm.commerce.foundation.dataload.datawriter.JDBCDataWriter" />

Related concepts

Data load architectural overview

Related tasks

Customize the data load utility

Configure the CSV data reader

Related reference

Data load business object mediators

Data load utility samples

+

Search Tips   |   Advanced Search