Sharding input properties file
The sharding input properties file is a file created that is passed in to the di-parallel-process utility when indexing shards.The sharding input properties file contains the following sections and properties:
System properties
System properties are shared by other utilities. For example, passwords that are common among utilities.
- PasswordFileDir
- The location of a password.properties file created that contains passwords encrypted by the wcs_encrypt utility.
Database properties
Database properties are used to establish database connections with the database server. All database properties start with the Database keyword, followed by the property name, which is separated by a period. The following list shows the database properties:
- Database.dbType
- The database type. The accepted values are db2 or oracle.
- Database.dbName
- The database name.
- Database.dbHost
- The remote database host name.
- Database.dbServerPort
- The remote database port number.
- Database.dbUser
- The database user name.
- Database.dbSchemaName
- The database schema name. This value is typically the uppercase version of the database user name.
- Database.jdbcURL
- The database JDBC URL.
- Database.jdbcDriverLocation
- The database JDBC driver location.
- Database.jdbcDriverType
- The database JDBC driver type.
Global preprocessing and indexing properties
Global properties are used for preprocessing and indexing by all shards, if not otherwise overwritten by the shard itself. All global properties start with the Global keyword, followed by the property name, which is separated by a period.
Global properties can be grouped into the following categories: server properties, preprocessing properties, and indexing properties. The following list shows the global server properties:
- Global.index-server-name
- The search server host name.
- Global.index-server-port
- The search server port number.
- Global.index-server-user
- The search server user name.
The following list shows the global preprocessing properties:
- Global.preprocessing-locale
- A single preprocessing locale value to be preprocessed. Default is All, where all locales are preprocessed.
A comma-separated list of locales is allowed.
- Global.auto-preprocess
- Automatically preprocess all shards.
- Global.preprocessing-drop-temp-table
- Indicates whether to drop tables when preprocessing the search index.
Passing in a value of false uses a TRUNCATE statement on the tables.
Default is true, which uses a DROP statement on the tables.
Note: This parameter supports only DB2 9.7 or later, or Oracle databases.
- Global.preprocessing-truncate-table-sql
- Overwrites the default TRUNCATE table statement.
- Global.preprocessing-dir
- The path to the preprocessing XML files in the WebSphere Commerce Utility Docker container.
- Global.preprocessing-initial-global-table-dir
- The preprocessing XML under this path is processed just once before sharding. This path is the preferred location for preprocessing configurations for initial temporary tables that do not need to be sharded or created multiple times.
- To reduce the customer's configuration efforts, auto-sharding is supported in WebSphere Commerce Version 9.0.0.5+ and enabled by default in Version 9.0.0.7+. When this feature is enabled, the parallel process utility can automatically divide the entry table CATENTRY into equal pieces by row number, and create temporary tables based on those pieces. The shard names are generated automatically according to the number-of-shards value, starting from A. For example, if number-of-shards equals 3, the shards would be named A, B and C.CAUTION:We cannot use auto-sharding and manually specified shards at the same time. If you do, Global.locale will be unable to overwrite each shard's index-core-name and index-core-dir for the autosharding process, resulting in unpredictable behavior.
The following list shows auto-sharding properties.
Note: Auto-sharding is compatible with vertical sharding and the combination is recommended.
- Global.auto-shard
- Flag to automatically do shard. In WebSphere Commerce Version 9.0.07+ this property is enabled by default.
- Global.number-of-shards
- Number of shards into which to divide the index. The permitted value range is from 1 to 26. In Version 9.0.0.7+ the default is 3.
- Global.preprocessing-shard-entry-table
- Shard entry table or view. If this value is not set, the CATENTRY table is processed as the default entry table.
- Global.locale.unstructured-index-core-name
- Name to be used to override the unstructured-index-core-name.
- Global.locale.unstructured-index-core-dir
- Directory to be used to override the unstructured-index-core-dir.
- Global.locale.structured-index-core-name
- Name to be used to override the structured-index-core-name.
- Global.locale.structured-index-core-dir
- If Global.auto-index is set to true, the Global.locale... configurations overwrite each shard's index-core-name and index-core-dir configurations in Horizontal Shard properties. locale is a supported locale, for example en_US or fr_FR. The above index-core-name and index-core-dir can automatically be appended with the shard name.
- For example, if customer set number-of-shards to 3 with the following configurations:
Global.en_US.unstructured-index-core-name=MC_10001_CatalogEntry_Unstructured_en_US Global.en_US.unstructured-index-core-dir=/search/index/solr/MC_10001/en_US/Unstructured Global.en_US.structured-index-core-name=MC_10001_CatalogEntry_en_US Global.en_US.structured-index-core-dir=/search/index/solr/MC_10001/en_US/CatalogEntryThen shards A, B, and C are created automatically. For the auto-index, the values are as follows.
- Shard A's unstructured-index-core-name is MC_10001_CatalogEntry_Unstructured_en_US_A. Its unstructured-index-core-dir is /search/index/solr/MC_10001/en_US/Unstructured_A/. Its structured-index-core-name is MC_10001_CatalogEntry_en_US_A, and its structured-index-core-dir is /search/index/solr/MC_10001/en_US/CatalogEntry_A/.
- Shard B's unstructured-index-core-name is MC_10001_CatalogEntry_Unstructured_en_US_B. Its unstructured-index-core-dir is /search/index/solr/MC_10001/en_US/Unstructured_B/. Its structured-index-core-name is MC_10001_CatalogEntry_en_US_B, and its structured-index-core-dir is /search/index/solr/MC_10001/en_US/CatalogEntry_B/.
- Shard C's unstructured-index-core-name is MC_10001_CatalogEntry_Unstructured_en_US_C. Its unstructured-index-core-dir is /search/index/solr/MC_10001/en_US/Unstructured_C/. Its structured-index-core-name is MC_10001_CatalogEntry_en_US_C, and its structured-index-core-dir is /search/index/solr/MC_10001/en_US/CatalogEntry_C/.
The following list shows the global indexing properties:
- Global.auto-index
- Automatically start indexing shards after they are preprocessed.
- Global.create-core
- Create the shard core in the solr.xml file, if it does not exist.
- Global.unload-core
- Unload the shard core from the solr.xml file, if it does not exist.
- Gloabl.index-status-Interval
- The time interval to monitor the indexing process. Default is 10000.
- Global.auto-merge-index
- Automatically merge the index into the master server after all shard indexing is complete.
Master search server properties
There is typically only one set of master properties. They start with the keyword Master, followed by either the common keyword or the locale name (for example, en_US), followed by the property name, which is separated by a period.
The master properties must include at least the master-specific properties. Only the global server properties can be overwritten in the master properties. Otherwise, if not redefined, the global properties are used.
Note: The following properties use the en_US locale with shards A and C as a sample. Configure your sharding input properties file according to the locale and shard types that we are using.
- Master.en_US.structured-index-core-name
- Master.en_US.unstructured-index-core-name
- The master index cores where all shard data is merged.
Horizontal Shard properties
There is typically a set of horizontal shard properties for each horizontal shard. The horizontal shard properties start with the Shard keyword, followed by the shard name, followed by either the common keyword or the locale name, followed by the property name, all separated by periods.
The horizontal shard properties must include at least the horizontal shard's specific properties. All global properties can be overwritten in the horizontal shard, except for auto-merge-index. If no properties are redefined, the global properties are used. The following list shows the horizontal shard-specific properties, using shard A as a sample:
- Shard.A.common.preprocessing-dir
- The path to the preprocessing XML files in the WebSphere Commerce Utility Docker container. This property is common and cannot be overwritten.
- Shard.A.common.preprocessing-start-range-value
- Shard.A.common.preprocessing-end-range-value
- The horizontal shard range start and end values.
- Shard.A.common.configuration-type
- The shard type to configure. For example, horizontal or vertical. When there is a combination of horizontal and vertical shard types, this property must be set so. In this case, horizontal.
- Shard.A.en_US.unstructured-index-core-name
- Shard.A.en_US.unstructured-index-core-dir
- Shard.A.en_US.structured-index-core-name
- Shard.A.en_US.structured-index-core-dir
- The shard unstructured and structured core names and directories.
Vertical Shard properties
There is typically a set of vertical shard properties for each vertical shard. The vertical shard properties start with the Shard keyword, followed the shard name, either the common keyword or the locale name, followed by the property name, all separated by periods.
The vertical shard properties must include at least the vertical shard-specific properties. Only the global preprocessing properties can be overwritten in the vertical shard. Otherwise, if not redefined, the global properties are used.
The following list shows the vertical shard-specific properties, using shard V as a sample:
- Shard.V.common.preprocessing-dir
- The path to the preprocessing XML files in the WebSphere Commerce Utility Docker container. This property is common and cannot be overwritten.
- Shard.V.common.configuration-type
- The shard type to configure. For example, horizontal or vertical. When there is a combination of horizontal and vertical shard types, this property must be set so. In this case, vertical.
Overriding properties
To override a shard or master global property, use the following convention to overwrite it at the common level:Master.common.index-server-name=localhost Shard.A.common.index-server-name=localhostTo override a shard or master global property, use the following convention to overwrite it at the locale level:
Master.en_US.index-server-name=localhost Shard.A.en_US.index-server-name=localhost
Sample properties files
The Utility Docker image contains two sample properties files. The example di-parallel-process.properties file uses the en_US locale with shards A, B, and C. These shard definitions are commented-out in favor of auto-sharding, but have been kept as examples because manual sharding is still supported. We can find the sample files in the directory /opt/WebSphere/CommerceServer90/properties/parallelprocess/.
- di-parallel-process.properties
- The sample sharding input properties file for a Linux operating system that uses DB2. Refer to the latest di-parallel-process.properties configuration file in the WebSphere Commerce Utility docker container if we are using the Oracle database.
- password.properties
- The sample password properties file, referenced by the sample sharding input properties file. It contains passwords encrypted by the wcs_encrypt utility.