Create and index shards

We can set up and index a specified number of sharding cores by defining them in an input properties file and running the parallel-process utility in the Utility server Docker container.

Before beginning

1. Decide on the sharding configuration we want to use. For example, we can select a language-specific core that needs to be sharded.

2. Decide on the number of shards to create.

Task info

During this task, you perform the following high-level steps:

1. Create and populate the input properties file used to process indexing shards.

2. Run the di-parallel-process utility to index the shards.

Procedure

Prepare the input properties file

1. Create or update the input properties file that will be used for indexing, based on the environment and requirements. The sharding input properties file is used by the di-parallel-process utility to process indexing shards. It contains the following sections of properties:

   • System properties such as the password property file path.

   • Database properties, which are used to establish database connections with the database server.

   • Global preprocessing and indexing properties, which are used for preprocessing and indexing by all shards.

   • Master search server properties, which specify the master index cores where all shard data is merged.

   • Horizontal Shard properties, which specify the horizontal shard properties.

   The Utility Docker image contains two sample properties files. The example di-parallel-process.properties file uses the en_US locale and contains commented-out definitions for shards A, B, C and V (V is for vertical sharding). We can find the following files in the directory /opt/WebSphere/CommerceServer90/properties/parallelprocess/.

   di-parallel-process.properties

   The sample sharding input properties file for a Linux operating system that uses DB2. Refer to the latest di-parallel-process.properties configuration file in the WebSphere Commerce Utility docker container if we are using the Oracle database.

   password.properties

   The sample password properties file, referenced by the sample sharding input properties file. It contains passwords encrypted by the wcs_encrypt utility.

   For more information about the properties file and expected values, see Sharding input properties file.

Run the shard indexing utility

2. Go to the following directory:

   • (Developer) WCDE_installdir\bin

   • /opt/WebSphere/CommerceServer90/bin in the Utility server Docker container.

3. Run the utility:

   • (Developer) di-parallel-process.bat input_properties_file

   • di-parallel-process.sh input_properties_file [-dbuser dbuser] [-dbuserpwd dbuserpwd] [-searchuser searchuser] [-searchuserpwd searchuserpwd]

   Where:

   input_properties_file

   The path for the utility's input properties file can be either relative or absolute. For example, ../../di-parallel-process-linux-oracle.properties. The default path in the Utility server Docker container is /opt/WebSphere/CommerceServer90/properties/parallelprocess/di-parallel-process.properties.

   dbuser

   Optional: The name of the user who is connecting to the database.

   dbuserpwd

   Optional: The password for the user who is connecting to the database.

   searchuser

   Optional: The search application administrative user name.

   searchuserpwd

   Optional: The search application administrative user password.

4. Ensure that the utility runs successfully. Either check the exit code, or the wc-dataimport-parallel-processor.log file for more information. The path for wc-dataimport-parallel-processor.log is:

   • (Developer) WCDE_installdir\logs.

   • /opt/WebSphere/CommerceServer90/logs/.

   If we want the log to provide greater detail, set the log level to FINEST in the /opt/WebSphere/AppServer/profiles/default/installedApps/localhost/ts.ear/xml/config/dataimport/parallel-processor-logging.properties properties file.

What to do next

Replicate the merged index into the repeater, then to the other nodes.