Tune Index Load

We can tune Index Load for optimal performance by configuring the tunable values and evaluating the results.

About this task

Index Load starts with a single input source, uses multithreaded processing, and ends with a single batch service writing to a single index.

The following diagram shows the available tunable areas of Index Load:

Where the following main tunable areas exist:

Parallel threads

The number of threads to be used for parallel indexing.

Prefetch size

The number of rows to return for each database (SQL) call.

Commit count (hard commit)

The number of index documents to keep in memory before writing to the Solr index.

Batch count (soft commit)

The number of index documents to keep in the Index Load runtime buffer before pushing them into the Solr memory stack.

The higher the batch count value, the higher the indexing throughput but with more garbage generated.

The following gauges of measurement exist to determine performance, can be viewed from the Index Load status page:

Rate

The average number of documents indexed per second to the Solr stack.

Read time

The average amount of time spent running SQL calls.

Flush time

The average amount of time spent on Solr soft commits.

Commit time

The average amount of time spent on Solr hard commits.

Indexing time

The overall end-to-end time spent on indexing.

We can use these statistics to tune the main tunable areas of Index Load. The following diagram shows how Index Load works with chunks of data, and how we can tune the prefetch, threads, and batch count for performance:

Where:

1. The Key Manager uses Prefetch to get small chunks of data from the database and evenly distributes it across all worker threads. This technique avoids overloading the database when processing a very large of result set size. Very large result set sizes might not even fit into the database transaction log. By using smaller chunks of data, the query time is improved, and the Index Load thread workload is more evenly distributed.

2. The prefetch size (ParallelPrefetchSize) defines the lookahead block size, whereas the next range SQL (ParallelNextRangeSQL) is used to address large empty ID range gaps. The next range SQL is only used when the lookahead contains no data. That is, when a gap is detected. This SQL is used to return the next available ID, and therefore avoids unnecessary crawling.

3. The prefetch size, thread count, ranges, and batch count are all factors to consider when tuning Index Load.

Procedure

1. Use the following overall tuning technique to achieve optimal performance of Index Load:

   1. Start with a fixed time window or small data set to tune before scaling up. This helps calibrate the tasks with your hardware.

   2. Determine the optimal prefetch size, when the read time is slightly higher than 0, and ideally when the total elapsed read time is less than 1 minute. The prefetch size is used to control read time.

   3. Reach the best throughput, when flush and commit times are close to 0. That is, when the overall Solr index overhead is 0. The batch count is used to control the flush time (soft commit). The commit count is used to control the commit time.

   4. When the optimal settings are found, reduce the number of threads until throughput rate drops. This technique helps find the maximum power to set to each pipeline.

2. Use the following general settings to help maximize the indexing rate:

   1. Use a 64 bit JVM and assign as much heap to it as possible to reduce overall garbage collection.

   2. Use a hardcoded key range, so that Index Load does not waste time scanning empty ranges.

   3. Use a ThreadLaunchTimeDelay to avoid processing all indexing threads in parallel at startup and overloading database resources.

3. Tune the following values to determine how Index Load works with chunks of data:

   1. Tune the prefetch size to control how many rows are read from the data source at a time. This must be tuned to balance database load against data availability for worker threads.

   2. Tune the thread count to control how many parallel threads are processing these rows. This must be tuned to balance overall CPU load with ability to process data in parallel.

   3. Tune the ParallelNextRangeSQL to avoid gaps in input ID ranges. Each range is fetched and distributed across the worker threads.

   4. Tune the batch count to control the sizes of the batches that are sent to Solr.

4. Tune the configurable performance attributes in the wc-indexload-profileName.xml and wc-indexload-businessobject.xml Index Load configuration files. See Index Load configuration files for indexing from database.

5. Select one of the following tuning methods that affect the overall indexing rate, and adjust the tuning parameters accordingly.

   Note: Combining multiple tuning methods might result in unpredictable results and negatively impact the overall indexing rate.

   Option	Description
   Recommended: Memory allocation-based configuration	Set the following values in the solrconfig.xml file: Lucene ramBufferSizeMB batchSize disable commitCount
   Document count-based Index Load configuration	Set the following values when configuring Index Load: batchSize commitCount
   Document count-based Solr configuration	Set the following values in the solrconfig.xml file: maxDocs of Solr autoCommit batchSize disable commitCount
   Document count-based Lucene configuration	Set the following values in the solrconfig.xml file: Lucene maxBufferedDocs batchSize disable commitCount

   Where:

   Lucene ramBufferSizeMB

   Defines the amount of memory space, in MB, to be used for buffering indexed documents. Once the accumulated document updates exceed the allocated memory space, a disk flush occurs, which can also create new segments, or trigger an index segment merge.

   Default is 100 MB.

   Lucene maxBufferedDocs

   Defines the number of document updates to buffer in memory before they are flushed as a new segment. Once the accumulated document updates exceed this value, a disk flush occurs, which can also create new segments or trigger an index segment merge.

   Default is disabled.

   maxDocs of Solr autoCommit

   A Solr-level parameter that defines the maximum number of indexed documents to be buffered in memory before a disk flush occurs. Compared to the Lucene level maxBufferedDocs, this setting does not guarantee a low-level disk flush. When the Lucene maxBufferedDocs value is greater than this autoCommit size, this setting is irrelevant.

   Default is disabled.

   Index Load commitCount

   Defines the maximum number of uncommitted documents to be buffered in memory before a disk flush occurs. This is an application-level control and flushes to physical storage using the Solr hard commit API. Set the value to 0 to disable it.

   Default is 1.

   Index Load batchSize

   Defines the number of documents to be kept in memory before soft committing to Solr. This action does not guarantee a disk flush, as the final decision also depends on the values of maxDocs, and the Lucene ramBufferSizeMB and maxBufferedDocs values, if any of them are configured. Set the value to 0 to disable it.

   Default is 1.

   Note: The commit count value supersedes the batch size value when the batch size is larger than the commit count.

6. Monitor Index Load and use the metrics while indexing to help refine tuning parameters and improve performance throughput. See Monitoring Index Load.