Configure the number of crawling threads

Overview

Edit settings in search-config.xml to specify the maximum number of threads used when crawling. The maximum number of threads that you should specify is the number of applications you have installed in the deployment.

By default, the maximum number of threads allowed when crawling is 2, however we can change this value by modifying search-config.xml. When you change the maximum number of crawling threads, you might also need to adjust the thread settings for the SearchCrawlingWorkManager on each node. The Search application will use whichever setting is lower.

Update maximum of crawling threads

1. Start wsadmin...

2. Initialize the Search environment, and start the Search script interpreter:
   cd app_server_root/profiles/Dmgr01/bin
   ./wsadmin.sh -lang jython
   execfile("searchAdmin.py")

   If successful ..

   Search Administration initialized

3. Check out search-config.xml:

   SearchCellConfig.checkOutConfig("working_dir", "cellName")

   To get Cell: print AdminControl.getCell()

4. Run:

   SearchCellConfig.setMaxCrawlerThreads(String maxThreadNumber)

   Maximum number of seedlist threads that can be used when crawling. By default, the value is set to 2.

   Specify the number of threads allowed.

   For example:

   SearchCellConfig.setMaxCrawlerThreads("3")

5. Check in search-config.xml:

   SearchCellConfig.checkInConfig()
   AdminConfig.save()
   exit

6. Stop the server or servers hosting the Search application, and then restart the Search servers.

   The next time the scheduled task runs, persisted seedlists are retained after indexing finishes.

Parent topic:
Manage the Search index

Related:
Reload the Search application
Configure the number of indexing threads
Performing a background crawl
Update Search work manager settings
Create work managers for Search