Data Extract utility

The Data Extract utility, which uses the Data Load utility framework, extracts data from the WebSphere Commerce database into an output file.

We can run this utility in the staging and production environments. However, we are recommended to run the utility in an environment that has all of the information for that you need to extract for an object. For example, the staging environment might not have inventory or pricing information for a catalog entry. In this case, run the utility on the production environment.

For general information about the Data Extract utility, see Overview of the Data Extract utility.

Note: If you integrate WebSphere Commerce with IBM Product Recommendations, use this utility to generate Enterprise Product Report (EPR) data for use with IBM Product Recommendations. Before you extract this data, configure the utility to generate ECDF and EPCMF files in the correct format to load into IBM Product Recommendations. See Data extraction utility for dynamic recommendations in IBM Product Recommendations.

Command syntax

Parameter values

full_path

Full path location of the order configuration file for the Data Extract utility.

-DXmlValidation

(Boolean) Used to turn on or off XML validation. For accuracy, the defined configurations inside the XML files are validated against the Data Extract utility and Data Load utility XSDs. By default, XML validation is turned on (true). To turn off validation, enter -DXmlValidation=false.

-DLoadOrder

Specifies the order that data is extracted from the database. This parameter overrides the LoadOrder element in the order configuration file. We can specify any number of object items to extract that we want, with each item separated by a comma. Ensure that parameter values do not contain white space. If there is white space, enclose the parameters in double quotation marks. For example,

"-DLoadOrder=loadItemName1, loadItemName2"

-DConsoleHandler.level

Used to override the console logging level that is defined in the utilities_root\dataextract\logging.properties file. For example, enter -DConsoleHandler.level=SEVERE to change the console logging level to SEVERE.

-DFileHandler.level

Used to override the file logging level that is defined in the utilities_root\dataextract\logging.properties file. For example, enter -DFileHandler.level=WARNING to change the file logging level to WARNING.

-Dpackage_name.level

Where package_name is the name of any package. This parameter is used to override the log level for the specified package. If the package_name is blank (-D.level=level name), then the log level is changed for all packages. The level name can be: OFF, SEVERE, WARNING, INFO, CONFIG, FINE, FINER, FINEST, or ALL. To help troubleshoot errors that can occur when you run the Data Extract utility, we can override logging levels. For example, override the following Data Load utility packages that the have logging available that the Data Extract utility can use:

-Dcom.ibm.commerce.foundation.dataload.businessobjectbuilder.level
-Dcom.ibm.commerce.foundation.dataload.businessobjectmediator.level
-Dcom.ibm.commerce.foundation.dataload.config.level
-Dcom.ibm.commerce.foundation.dataload.datareader.level
-Dcom.ibm.commerce.foundation.dataload.datawriter.level
-Dcom.ibm.commerce.foundation.dataload.idresolve.level
-Dcom.ibm.commerce.foundation.dataload.database.level
-Dcom.ibm.commerce.foundation.dataload.level 

For more information about changing the logging level for a package, see Configure logging for runtime environments. For more information about logging, see Logging services.

Running the Data Extract utility

1. (Developer) On a command line, go to the WCDE_installdir\bin directory.

2. (Linux) Open a command line in the Utility server Docker container. Change the directory to utilities_root/bin directory.
   For information about entering and leaving containers, see Running utilities from the Utility server Docker container.

3. Run the following command to use Data Extract utility to run the specified order configuration file. The file identifies the business object configuration files that define how to extract the data.

   • (Linux) ./dataextract.sh ../path_to_order_configuration_file/wc-dataextract.xml

   • (Developer) dataextract ..\path_to_order_configuration_file\wc-dataextract.xml

   Where

   path_to_order_configuration_file

   The relative path to the order configuration file that identifies the configuration files that we are using with the Data Extract utility.

   wc-dataextract.xml

   The order configuration file. Sample files are provided with WebSphere Commerce in the following directories:

   • (Linux) utilities_root/samples/DataExtract

   • (Developer) WCDE_installdir\samples\DataExtract

   By default the sample order configuration file is typically named wc-dataextract.xml or wc-dataextract-object.xml, where object is the type of object that is being extracted. These files are structured similar to the order configuration files used to configure the Data Load utility, but they include some extensions.

   As an example, the following command runs the Data Extract utility to extract promotion objects

   • (Linux) ./dataextract.sh ../samples/DataExtract/Promotion/wc-dataextract.xml

   • (Developer) dataextract ..\samples\DataExtract\Promotion\wc-dataextract.xml

Note: Since the Data Extract utility uses the Data Load utility framework, we can also configure the Data Load utility to extract data. We can configure the Data Load utility command to use some of the Data Extract utility configuration files in the subfolders of the following directory:

• (Linux) utilities_root/samples/DataExtract

• (Developer) WCDE_installdir\samples\DataExtract

We cannot configure the Data Load utility to extract catalog data for generating Enterprise Product Report (EPR) data for use with IBM Product Recommendations. We must use the Data Extract utility to extract this data.

Troubleshooting

We can encounter errors when you run the utility for many reasons. Errors can be caused by incorrect command parameters, configurations, and more. To help troubleshoot errors that are encountered, review the wc-dataextract.log file as the first step in determining the source of your error. This file can be located within the following directory

• (Linux) utilities_root\logs

• WCDE_installdir\logs

By default, this log file provides only a summary report. We can however specify the level of package logging when you run the utility to generate more information about the extract process within this log file. For example, to run the utility with a finer log level for catalog information, we can specify the following command that includes the -Dpackage_name.level parameter:

dataextract.sh wc-dataextract.xml -Dcom.ibm.commerce.catalog.dataload.level=FINER

Related concepts
Overview of the Data Extract utility