Overview of the Data Load utility

Overview of the Data Load utility

The Data Load utility is an enhanced business object based loading utility. This utility provides an efficient solution for loading information into the WebSphere Commerce database. We can also customize the Data Load utility to load other types of data. The Data Load utility is the recommended loading utility.
The Data Load utility supports loading data into a workspace. When you load data into a workspace we can make and preview changes to managed assets, without affecting what is running on the site. The following general user roles interact with the Data Load utility:
Business user

Responsible for managing the business data.

Developer

Responsible for defining the data source template, business object mappings, and customizing the Data Load utility.

Site administrator

Responsible for the day-to-day operation of the Data Load utility.

Data Load utility - user interaction diagram

The following diagram describes how the user roles interact with the data load utility:

The business user provides the developer with the business data.
The developer creates a data source template, which defines how source data must be formatted before the data is loaded.
The developer also creates the business object configuration file. The business object configuration file defines how the Data Load utility maps the input data to the business object and how to transform the business object to physical data.
The site administrator uses the business object configuration file to define and create the load order configuration file.
The site administrator sets the store and database settings in the environment configuration file.
The business data is formatted according to the rules of the data source template before the data is loaded to the database.
The formatted source data is provided to the site administrator.
The site administrator runs the Data Load utility along with the three configuration files (environment, load order, and business object configuration files) to load the formatted source data into the WebSphere Commerce database. After the utility runs, the site administrator also verifies the results of the load.
The business data is available in WebSphere Commerce to be managed by the business user.

For more information about how the Data Load process is structured, see Data Load utility architectural overview. For more information about how the Data Load utility works and what components are included in the process, see Data Load utility framework process and components.

Running the Data Load utility

To use the Data Load utility to load data into the WebSphere Commerce database, we must first configure or create the required files. To run the Data Load utility a business object configuration file, load order configuration file, environment configuration file, and data source file are all required. See
Configure and running the Data Load utility.

Best practices
When we are using the Data Load utility to load data, follow these best practices:

Data Load utility best practices
If the data we are loading is catalog, price, inventory, member, or Commerce Composer:

General data load best practices
Data Load best practices for Catalog
Data Load utility best practices for Inventory
Data Load best practices for Price
Data Load best practices for Member
Data Load utility best practices for Commerce Composer

Data Load utility limitations

Newly created business objects might not show up immediately in the storefront due to caching. To resolve this problem, invalidate the dynamic cache manually. For more information about cache invalidation, see Removing cache entries through the Cache Monitor.
When you update business objects, the changes might not show up in the Management Center. This issue can occur due to caching. To resolve this problem, invalidate the data cache.
We can load data in only CSV and XML formatted files. By default, most sample configuration files provided with WebSphere Commerce are configured to load CSV files. To load XML files, configure the utility to use the XML data reader. Your XML input files must also use a CSV-like structure. See File format for Data Load input files.
To load data in other formats, or in an XML structure that is not supported by the default XML data reader, we must create our own custom data reader.
Configure the Data Load utility to use a business object mediator to map your input file data with the appropriate WebSphere Commerce business objects. By default, we are provided with business object mediators for loading data for business objects in the following components:

Catalog
Inventory
Price and Catalog filter
Member
Location
Commerce Composer
Promotions
Marketing
To load other data, we can use the TableObjectMediator, or we can create our own custom business object mediators.
The TableObjectMediator does not support workspace locking.
If the site uses WebSphere Commerce search, the delta search index might not rebuild correctly when you delete some catalog objects with the Data Load utility in delete mode. When you delete a child object of a catalog entry or category with the utility in delete mode, both the child and parent object are removed from the delta search index rebuild. This removal can cause the parent catalog entry or category to no longer be indexed or display correctly in the storefront.
Use the utility in replace mode to delete catalog objects when the site uses WebSphere Commerce search. To delete objects with the utility in replace mode, include the value 1 for the Delete column of an object in your input file. If you do decide to delete catalog objects with the utility in delete mode, run a full index rebuild after the load operation completes.
The member component business object mediators do not support the following actions with the Data Load utility:

Deleting users or organizations
Loading user passwords
Modifying the parent organization of a member
Modifying the distinguished name (DN) of an organization
Modifying the entity type of an organization

When we are loading userData into custom extension database tables, we cannot load data into columns that have a data type that is not supported by the utility.

Procedures and samples
The following topics are available to help you learn more about the Data Load utility.

Role Topic Summary
Site Administrator File format for Data Load input files A data load input file contains the actual information that the Data Load utility populates into the database. Learn about how to construct such files to ensure that the loading process is successful.
Creating data in XML format Learn how to create a data load input file in the supported XML format.
Creating data in CSV format Learn how to create a data load input file in the supported CSV format
Configure the Data Load utility to run a file difference preprocess If you routinely load the same generated Data Load input file from an external system or source, we can choose to run a file difference preprocess. We can run this preprocess as part of the Data Load process to ensure that we are loading only new changes when loading your newest input file.
Configure the CSV data reader The CVS data reader is already provided with the Data Load utility. Learn how to configure the provided CSV data reader to change the way data is read from your CSV input files.
Configure the XML data reader An XML data reader is provided by default with the Data Load utility. Learn how to configure this provided data reader to change the way data is read from your XML formatted input files.
Configure the component business object builder The BaseBusinessObjectBuilder and TableObjectBuilder business object builders by defining sub elements and attribute values in the business object configuration file.
Configure the business object configuration file Learn how to configure the business object configuration file that defines how to load the data into the database. In this file, you specify the implementation classes for our Data Reader, Business Object Builder, and Business Object Mediator components.
Configure the data load order The data load order configuration file controls the load order of the Data Load utility. Learn how to configure your data load order file.
Configure a column exclusion list We can configure a column exclusion list that causes the Data Load utility to avoid loading data into the specified columns of a table.
Configure the data load environment settings Learn how to configure the environment variables used by the Data Load utility in the environment settings file.To configure the Data Load utility to load data into a workspace, add the workspace attribute to the business context in the environment settings file.
Note: When you load data into a workspace, the Data Load utility respects the locking policy set in the workspace. For more information about workspace locking policies, see Workspaces locking policies.
Data Load utility command syntax Learn how to run the utility command that runs the Data Load process
Verifying the results of the data load Learn how to verify that a load operation with the Data Load utility is successfully completed.
Loading data into workspaces using the Data Load utility Load data into workspaces with the data load utility
Loading values for single and multiple value attributes Load data for attribute values for attributes with a single value and attributes with multiple values
Reuse attribute assigned values with the Data Load utility Configure the utility to reuse assigned values for attributes when the same value is needed for multiple catalog entries.
Loading promotions with the Data Load utility Load data to create or change promotions for a store.
Loading customer segment members by email address with the Data Load utility Load a list of email addresses to create a customer segment and populate the segment with members associated with the email address.
Loading member group members by email address with the Data Load utility Load members into a member group based on the member email addresses.
IBM Omni-Channel Merchandising price list integration Integrate the utility with IBM Omni-Channel Merchandising, formerly known as IBM DemandTec so that a scheduled job can use an SFTP transport to retrieve price list CSV files from IBM Omni-Channel Merchandising. The Data Load utility can then load the price data into WebSphere Commerce.
Configure a Data Load utility scheduler job Use the WebSphere Commerce Administration Console to configure the Data Load utility to run as a scheduled job.
Loading marketing objects with the Data Load utility Load data to create marketing activities, campaigns, customer segments, content, attachments, and e-Marketing Spots.
Samples Scenario: Initial load The initial load is the scenario when you finish creating and configuring a new WebSphere Commerce instance; you then load your initial data into the WebSphere Commerce database.
Scenario: Delta load The delta load scenario is when the WebSphere Commerce server is up and running, then we want to insert, update, or delete the catalog, inventory, or price data.
Scenario: Catalog entry update load The utility can run in an update mode to update catalog entry data. The update mode replaces or adds data for only the columns specified in the input file. All other columns remain unchanged.
Scenarios: Workspace locking for the Data Load utility These scenarios detail the workspace locking policies that are in effect when the Data Load utility loads data into a workspace. These locking policies affect the loading of data into the workspace database.
Sample: Setting up the Data Load utility This sample demonstrates how to set up the Data Load utility for its first use.
Catalog samples These samples use CSV or XML files to demonstrate how to run the Data Load utility to load catalog data
Inventory samples These samples use CSV files to demonstrate how to run the Data Load utility to load inventory data.
Member samples These samples use CSV or XML files to demonstrate how to run the Data Load utility to load member data.
These samples include a sample for loading customer segment member data.
Price samples These samples use CSV files to demonstrate how to run the Data Load utility to load price data.
Location samples These samples use CSV files to demonstrate how to run the Data Load utility to load location data.
Store configuration samples This sample uses a CSV or XML file to demonstrate how to load configuration name-value pair properties for a store.
Commerce Composer samples These samples use CSV file to demonstrate how to load Commerce Composer assets with the Data Load utility.
Promotion samples These samples use input files to demonstrate how to load promotions, promotion folders, and load promotions into the promotion folders.
Sample: Loading marketing data These samples use input files to demonstrate how to load marketing activities, campaigns, customer segments, content, attachments, and e-Marketing Spots.
Examples Examples: Mapping catalog data These examples use a CSV file to demonstrate how to insert, replace, or delete catalog data.
Examples: Mapping inventory data These examples use a CSV file to demonstrate how to insert, replace, or delete inventory data.
Examples: Mapping pricing data These examples use a CSV file to demonstrate how to insert, replace, or delete price data.
Examples: Mapping catalog filter data These examples use a CSV file to demonstrate how to insert, replace, or delete catalog filter data.
Examples: Mapping member data These examples use a CSV file to demonstrate how to insert, replace, or delete member data.
Developer Data Load utility architectural overview An understanding of how the Data Load utility works, and the components that make up the Data Load utility.
Customizing the Data Load utility Learn how to customize elements of the Data Load utility to create custom data readers, column handlers, business object mediators, and load extension tables. By customizing the Data Load utility we can complete the following tasks:

Load data from sources that are not in a CSV or specific XML format.
Resolve data from database columns that are based on input values that cannot be mapped to the column value through the default WebSphere Commerce configuration.
Create a custom mediator to load data for extended or custom business objects.
Load UserData into custom extension tables.

Configure an SFTP transport to retrieve external files for the Data Load utility Customize an SFTP transport for a Data Load utility scheduled job to use to retrieve input files from an external source to load into WebSphere Commerce.

Role	Topic	Summary
Site Administrator	File format for Data Load input files	A data load input file contains the actual information that the Data Load utility populates into the database. Learn about how to construct such files to ensure that the loading process is successful.
	Creating data in XML format	Learn how to create a data load input file in the supported XML format.
	Creating data in CSV format	Learn how to create a data load input file in the supported CSV format
	Configure the Data Load utility to run a file difference preprocess	If you routinely load the same generated Data Load input file from an external system or source, we can choose to run a file difference preprocess. We can run this preprocess as part of the Data Load process to ensure that we are loading only new changes when loading your newest input file.
	Configure the CSV data reader	The CVS data reader is already provided with the Data Load utility. Learn how to configure the provided CSV data reader to change the way data is read from your CSV input files.
	Configure the XML data reader	An XML data reader is provided by default with the Data Load utility. Learn how to configure this provided data reader to change the way data is read from your XML formatted input files.
	Configure the component business object builder	The BaseBusinessObjectBuilder and TableObjectBuilder business object builders by defining sub elements and attribute values in the business object configuration file.
	Configure the business object configuration file	Learn how to configure the business object configuration file that defines how to load the data into the database. In this file, you specify the implementation classes for our Data Reader, Business Object Builder, and Business Object Mediator components.
	Configure the data load order	The data load order configuration file controls the load order of the Data Load utility. Learn how to configure your data load order file.
	Configure a column exclusion list	We can configure a column exclusion list that causes the Data Load utility to avoid loading data into the specified columns of a table.
	Configure the data load environment settings	Learn how to configure the environment variables used by the Data Load utility in the environment settings file.To configure the Data Load utility to load data into a workspace, add the workspace attribute to the business context in the environment settings file. Note: When you load data into a workspace, the Data Load utility respects the locking policy set in the workspace. For more information about workspace locking policies, see Workspaces locking policies.
	Data Load utility command syntax	Learn how to run the utility command that runs the Data Load process
	Verifying the results of the data load	Learn how to verify that a load operation with the Data Load utility is successfully completed.
	Loading data into workspaces using the Data Load utility	Load data into workspaces with the data load utility
	Loading values for single and multiple value attributes	Load data for attribute values for attributes with a single value and attributes with multiple values
	Reuse attribute assigned values with the Data Load utility	Configure the utility to reuse assigned values for attributes when the same value is needed for multiple catalog entries.
	Loading promotions with the Data Load utility	Load data to create or change promotions for a store.
	Loading customer segment members by email address with the Data Load utility	Load a list of email addresses to create a customer segment and populate the segment with members associated with the email address.
	Loading member group members by email address with the Data Load utility	Load members into a member group based on the member email addresses.
	IBM Omni-Channel Merchandising price list integration	Integrate the utility with IBM Omni-Channel Merchandising, formerly known as IBM DemandTec so that a scheduled job can use an SFTP transport to retrieve price list CSV files from IBM Omni-Channel Merchandising. The Data Load utility can then load the price data into WebSphere Commerce.
	Configure a Data Load utility scheduler job	Use the WebSphere Commerce Administration Console to configure the Data Load utility to run as a scheduled job.
	Loading marketing objects with the Data Load utility	Load data to create marketing activities, campaigns, customer segments, content, attachments, and e-Marketing Spots.
Samples	Scenario: Initial load	The initial load is the scenario when you finish creating and configuring a new WebSphere Commerce instance; you then load your initial data into the WebSphere Commerce database.
	Scenario: Delta load	The delta load scenario is when the WebSphere Commerce server is up and running, then we want to insert, update, or delete the catalog, inventory, or price data.
	Scenario: Catalog entry update load	The utility can run in an update mode to update catalog entry data. The update mode replaces or adds data for only the columns specified in the input file. All other columns remain unchanged.
	Scenarios: Workspace locking for the Data Load utility	These scenarios detail the workspace locking policies that are in effect when the Data Load utility loads data into a workspace. These locking policies affect the loading of data into the workspace database.
	Sample: Setting up the Data Load utility	This sample demonstrates how to set up the Data Load utility for its first use.
	Catalog samples	These samples use CSV or XML files to demonstrate how to run the Data Load utility to load catalog data
	Inventory samples	These samples use CSV files to demonstrate how to run the Data Load utility to load inventory data.
	Member samples	These samples use CSV or XML files to demonstrate how to run the Data Load utility to load member data. These samples include a sample for loading customer segment member data.
	Price samples	These samples use CSV files to demonstrate how to run the Data Load utility to load price data.
	Location samples	These samples use CSV files to demonstrate how to run the Data Load utility to load location data.
	Store configuration samples	This sample uses a CSV or XML file to demonstrate how to load configuration name-value pair properties for a store.
	Commerce Composer samples	These samples use CSV file to demonstrate how to load Commerce Composer assets with the Data Load utility.
	Promotion samples	These samples use input files to demonstrate how to load promotions, promotion folders, and load promotions into the promotion folders.
	Sample: Loading marketing data	These samples use input files to demonstrate how to load marketing activities, campaigns, customer segments, content, attachments, and e-Marketing Spots.
Examples	Examples: Mapping catalog data	These examples use a CSV file to demonstrate how to insert, replace, or delete catalog data.
	Examples: Mapping inventory data	These examples use a CSV file to demonstrate how to insert, replace, or delete inventory data.
	Examples: Mapping pricing data	These examples use a CSV file to demonstrate how to insert, replace, or delete price data.
	Examples: Mapping catalog filter data	These examples use a CSV file to demonstrate how to insert, replace, or delete catalog filter data.
	Examples: Mapping member data	These examples use a CSV file to demonstrate how to insert, replace, or delete member data.
Developer	Data Load utility architectural overview	An understanding of how the Data Load utility works, and the components that make up the Data Load utility.
	Customizing the Data Load utility	Learn how to customize elements of the Data Load utility to create custom data readers, column handlers, business object mediators, and load extension tables. By customizing the Data Load utility we can complete the following tasks: Load data from sources that are not in a CSV or specific XML format. Resolve data from database columns that are based on input values that cannot be mapped to the column value through the default WebSphere Commerce configuration. Create a custom mediator to load data for extended or custom business objects. Load UserData into custom extension tables.
	Configure an SFTP transport to retrieve external files for the Data Load utility	Customize an SFTP transport for a Data Load utility scheduled job to use to retrieve input files from an external source to load into WebSphere Commerce.