Index synchronization and delta updates in WebSphere Commerce Search
The WebSphere Commerce Search index can typically fall out of synchronization with the latest production data over time. To maintain accuracy of the search result data, reindexing must be done at a certain time during normal business operation. There are three forms of search reindexing, and depending on your business requirements, costs and benefits are associated with each of the following options: Full reindexing, delta reindexing, and emergency updates. Consider the following scenarios when we select a reindexing option:
- Full reindexing
- Performing a full reindexing rebuilds the entire search index. After the indexing completes, the search engine automatically uses the updated index. Full reindexing is effective when many updates are applied over time and the index starts to degrade in performance due to fragmentation. However, a full reindexing can take considerable time and therefore is best done on an extended schedule, for example on a weekly basis.
- Delta reindexing
- A delta reindexing makes only incremental updates to the existing operational search index. Delta reindexing is effective when the amount of changed data is manageable and small enough to be applied as a batch. Typically, delta reindexing is done at regular intervals in a staging environment. After the changes are ready, it is published from the staging environment over to production through replication, thus minimizing the business impact. However, you may encounter business scenarios where a delta reindex might be too complicated, and therefore a full reindexing is preferred.
- To run a delta reindexing, set the fullbuild flag to false when the system builds the search index. Otherwise, the default value is true, which does a full preprocess and index build.
Building the index
Use the build index REST call to do both data preprocessing and index building.
For full information about the index build REST call, see Building the WebSphere Commerce Search index.
The build process extracts and flattens WebSphere Commerce data and then outputs the data into a set of temporary tables inside the WebSphere Commerce database. The data in the temporary tables is then used by the index building utility to populate the data into search indexes by using the Data Import Handler (DIH). When are using multiple indexes, for example, each language with its own separate index, the index is built multiple times.
See Building the search index.
Events that trigger search reindexes
The search index is triggered for reindexing when certain actions take place within WebSphere Commerce. Reindexing is necessary to avoid working with stale data, ensuring that the content we are working with or previewing is up-to-date. In the Management Center, with workspaces enabled:
- Task group commit
- Store preview
- Show facets (catalog entry index)
- UpdateSearchIndex scheduler job
Note: This scheduler task triggers reindexing only on the base index. It skips workspace indexes to prevent unnecessary reindexing that is caused by uncommitted changes.
In the Management Center, with workspaces disabled:
- Store preview
- Show facets (catalog entry index)
- UpdateSearchIndex scheduler job
In the Aurora starter store:
- UpdateSearchIndex scheduler job.
Note: This scheduler job detects changes that are made from the authoring environment using the TI_DELTA_CATENTRY and TI_DELTA_CATGROUP temporary database tables. That is, after the quick publish task is approved, the runtime automatically publishes the search change history to the temporary tables in both the staging and production databases. It then starts a reindexing for emergency fixes from the production environment.
Using utilities:
- The buildindex RESTful call. Refer to Building the WebSphere Commerce Search index.
When to perform full search index builds
The WebSphere Commerce Search index is automatically built when certain business tasks are performed, as outlined in Common business tasks and their impact to the WebSphere Commerce Search index. In several cases, common business tasks result in delta index builds that do not pose a significant risk to production system performance. However, doing several delta index builds without occasional full index builds might result in the search index gradually degrading over time due to fragmentation. To avoid this issue, doing full search index builds when possible ensures that the search index performs well over time.
When Lucene receives a delete request, it does not delete entries from the index, but instead marks them for deletion and adds updated records to the end of the index. This marking results in the catalog unevenly spreading out across different segment data files in the search index, and might result in increased search response times. If you have a dedicated indexing server, consider scheduling a periodic full search index build. Make this build a background task that runs once per month, so that the deleted entries are flushed out, and to optimize the data.
Scheduler jobs
WebSphere Commerce Search contains scheduler jobs to trigger for index synchronization.
See Scheduling WebSphere Commerce Search jobs.
Search index approaches
Although catalog entries are mentioned in index synchronization and delta updates in WebSphere Commerce search, the same approach applies for the other search indexes or BODs such as ChangeCatalogGroup.
Search index optimization
There are several search optimization hints and tips to consider when administering WebSphere Commerce search. See Search index optimization.
- Common business tasks and their impact to the WebSphere Commerce Search index
There are impacts to the search index when certain business tasks are performed. Ensure that we are aware of your daily tasks when they might require search index rebuilds.- Catalog Upload operations in WebSphere Commerce Search
Certain actions require WebSphere Commerce Search index updates when we are uploading catalog data in the Management Center. Search index updates apply in cases where we are using the Catalog Upload or Data Load utility.
Related concepts
Workspaces in WebSphere Commerce Search