Migrate WebSphere Commerce Version 8 Search

Migrate WebSphere Commerce Version 8 Search
Migrate the WebSphere Commerce Search index and configurations from WebSphere Commerce Version 8 to WebSphere Commerce Version 9.
Starting with Version 9.0.0.5, WebSphere Commerce Version 9 uses Solr 7.3.1. Your index data from an earlier version of Solr is unsupported by Solr 7.3.1 and must be updated. Review the following information to understand what WebSphere Commerce Search architecture and functionality has changed in WebSphere Commerce Version 9.

The WebSphere Commerce Search server has its own container within your production environment. You deploy the WebSphere Commerce Search server as part of our CI/CD pipeline
Deep sequence is disabled by default in WebSphere Commerce Version 9.
Result grouping is disabled by default in WebSphere Commerce Version 9.
The programming model for WebSphere Commerce Search is changed to coincide with the new build and deployment process in WebSphere Commerce Version 9. The basis of the new programming model is to separate custom WebSphere Commerce Search assets and configuration settings from the product code, which reduces the resource cost of maintenance and operation. The following WebSphere Commerce Version 8 customizations must be updated for the new programming model:

Solr runtime is upgraded to 5.5.4, so any customizations to Solr must be updated to follow new programming model.
WebSphere Commerce Search utilities are replaced by the container utility service, which includes di-preprocess, di-buildindex, di-calculateprice, and indexprop. The SetupSearchIndex utility is discontinued. The index core directory is now automatically synchronized with the SRCHCONF table and the SRCHCONFEXT table when the WebSphere Commerce Search server is started. To create a new master core, extension core, or language we must maintain the SRCHCONF and SRCHCONFEXT tables. The workspace core is created automatically if the WebSphere Commerce Search server detects that the workspace schema is on an authoring environment.
In WebSphere Commerce Version 9, the table view is used for preprocessing and index building, so any customizations to preprocessing and index building must be reconfigured according to the new programming guide.
In WebSphere Commerce Version 9, the common foundation-based scheduler is enabled on the WebSphere Commerce Search server. Authoring environments use the scheduler to replicate indexes from authoring environments to the WebSphere Commerce Search repeater.
WebSphere Commerce Version 9 moved to JAX-RS 2.0(JSR-339). Also, the documentation API is Swagger 2.0.
WebSphere Commerce Version 8 used direct JDBC calls, which went through DSL(data service layer) to the database. In WebSphere Commerce Version 9, JPA 2.1 (EclipseLink) native query is used. Custom queries from previous versions are ported over to the new query service. No additional configure is required.
In WebSphere Commerce Version 9, when Price or Inventory are worked as extended core, SolrJoin preserves the document relationship between the main CatalogEntry core and the Price/Inventory subcore. MultipleQueryComponent and MultipleFacetComponent, which were used to join or filter the subcores in WebSphere Commerce Version 8, are now discontinued. To handle facet and result fields from extension indexes, a new SearchCatalogEntryExtensionIndexPostprocessor makes a subquery against each of the extension indexes, then joins back with the main index. A new join parameter was also introduced in wc-search.xml, so any WebSphere Commerce Version 8 customization to an extension index needs to be implemented to use SolrJoin.

Before starting

Drop all your temporary tables, including our custom temporary tables, but exclude the following temporary tables:

TI_DELTA_CATENTRY
TI_DELTA_CATGROUP
TI_DELTA_INVENTORY

Your temporary tables use a TI_ prefix. Whereas, our custom temporary tables use a XI_ prefix.
Changes were made to the temporary tables between the previous versions of WebSphere Commerce and WebSphere Commerce Version 9. Failure to drop the temporary tables might result in preprocess errors, for example, SQLSTAE=56098. For more information about temporary WebSphere Commerce Search tables, see Temporary table schema definition.

Procedure
Migrate your index cores. In WebSphere Commerce Version 9.0.0.1+, the setupSearchIndex utility is discontinued. Now, we must maintain your SRCHCONF table and SRCHCONFEXT table to set up your index distribution.
Migrate the main CatalogEntry / CatalogGroup core by updating your SRCHCONF table by running the following SQLs.
(DB2)
Update srchconf set config='IndexScopeTag=0,createAll=true,PreProcessConfigDirectory=/v3/db2,Version=3' where indextype='CatalogEntry';
(Oracle)
update srchconf set config='IndexScopeTag=0,createAll=true,PreProcessConfigDirectory=/v3/oracle,Version=3' where indextype='CatalogEntry';
(DB2)
update srchconf set config='IndexScopeTag=1,createAll=true,PreProcessConfigDirectory=/v3/db2/CatalogGroup,Version=3' where indextype='CatalogGroup’;
(Oracle)
update srchconf set config='IndexScopeTag=1,createAll=true,PreProcessConfigDirectory=/v3/oracle/CatalogGroup,Version=3' where indextype='CatalogGroup’;
Notes:

IndexScopeTag is an internal tag used to mark the temporary table, which is generated during preprocess. If we are migrating multiple master catalogs, we must incrementally increase this value for each entry you add.
createAll is used to determine whether to create a workspace counterpart core for this specific index. Setting this value to true creates the workspace counterpart.

Rerun the previous SQLs for every catalog entry and catalog group to migrate to WebSphere Commerce Version 9.0.0.1+.
If you need to migrate price or inventory subcores, we must register the subcores into the SRCHCONFEXT table. Use the following SQL as an example on how to register a core to your SRCHCONFEXT table.
insert into srchconfext(srchconfext_id,indextype,indexscope,indexsubtype,config) values(-1,'CatalogEntry','3074457345616676668','Price','');
Note: After you register your subcores, we must run separate build index to build your subcores. Complete this build index after the master core is built. For more information about building subcores, see Building the WebSphere Commerce Search index.
Migrate your languages. In previous WebSphere Commerce versions, different languages used separate schema definitions. Even though different languages still use different index data in WebSphere Commerce Version 9.0.0.1+, languages now share a schema definition. Registers your languages in your SRCHCONF table using the following SQL example.
If you used en_US, zh_CN, and fr_FR languages, you run the following SQL.
Update srchconf set languages=’-1,-2,-7’ where indextype=’CatalogEntry’ and indexscope=’10001’;
Restart the WebSphere Commerce Search server. As a result, the WebSphere Commerce Search server creates your cores and subcores during start-up.
Migrate your index schema customizations. In WebSphere Commerce Version 9.0.0.1+, the catalog entry fieldType definitions use two templates:

A non-customizable template: search-config/src/main/resources/managed-solr/config/v3/common/schema-field-types.xml.
Note: When the first index is created, this XML file is copied to the resources/search/index/managed-solr/config/v3/common directory. After the index creation, other indexes share this fieldType definition.
A customizable template: search-config-ext/src/main/resources/index/managed-solr/config/v3/common/x-schema-field-types.xml.
Note: When the first index is created, this XML file is copied to resources/search/index/managed-solr/config/v3-index-ext/common directory. After the index creation, other indexes share this customizable fieldType definition.
Migrate our custom schema fieldType using the following example.
If we created a new field Type x_textSpell for the fr_FR locale in a previous WebSphere Commerce version, you updated the schema.xml file, which was located under core configuration directory specific to the fr_FR locale. In WebSphere Commerce Version 9.0.0.1+, add the field type x_textSpell_fr to the file x-schema-field-types.xml as follows.

<fieldType omitNorms="true" positionIncrementGap="100" class="solr.TextField" name="x_textSpell_fr">
<analyzer type="index">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" words="${stopwords_fr:../../../v3/common/stopwords.txt}" ignoreCase="true"/>
<filter class="solr.WordDelimiterFilterFactory" preserveOriginal="0" splitOnNumerics="1" splitOnCaseChange="1" 
    catenateAll="0" catenateNumbers="0" catenateWords="0" generateNumberParts="1" generateWordParts="1"/>
<filter class="solr.ShingleFilterFactory" fillerToken="" tokenSeparator=" " maxShingleSize="3" minShingleSize="2" 
    outputUnigrams="true"/>
<filter class="solr.PatternReplaceFilterFactory" replace="all" replacement=" " pattern="\s{2,}"/>
<filter class="solr.TrimFilterFactory"/>
<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
</analyzer>

<analyzer type="query">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" words="${stopwords_fr:../../../v3/common/stopwords.txt}" ignoreCase="true"/>
</analyzer>
</fieldType>
After the new x_textSpell_fr field type is defined in the x-schema-field-types.xml file, define an entry into your x-schema.xml file using the following example.
<field name="spellCheck" type="x_textSpell_fr" indexed="true" stored="false" multiValued="true" />
Notes:
To all support languages, we must create a field type for every language in the x-schema-field-types.xml, then add following field definition into x-schema.xml file.
<field name="spellCheck" type="x_textSpell_${lang:en}" indexed="true" stored="false" multiValued="true" />
Migrate your Solr supported file assets.
Migrate our customized stop words. In WebSphere Commerce Version 8, you could change the stopwords.txt file for each core configuration directory. For example, if you customized stopwords.txt for the English language, you need only to change the stopwords.txt file under the en_US core. When you migrate to WebSphere Commerce Version 9.0.0.1+, the index schemas are shared among multiple languages. In order to customize stopwords.txt for a specific language, WebSphere Commerce Version 9.0.0.1+ requires that you point to the file in the SRCHCONFEXT database table. Run the following SQL command to put a stopwords.txt into the config field:
Update srchconfext set config='stopwords_en=../../../../managed-solr/config/v3-index-ext/common/stopwords.txt' 
where indextype=’CatalogEntry’ and indexscope=’$MASTERCATALOGID’;
When the WebSphere Commerce Search server is restarted, the following contents are added into x-core.properties under the en_US index data directory.
stopwords_en=../../../../managed-solr/config/v3-index-ext/common/stopwords.txt
If you need to change the stopwords.txt that is under workspace_dir\search-config-ext\src\index\managed-solr\config\v3\common directory, the new stopwords.txt will be copied to the Liberty_installdir\resources\search\index\managed-solr\config\v3-index-ext\common directory as part of setting up your WCB and CI/CD pipeline. For example, in WebSphere Commerce Version 8, you added a new English stop word: can. The following steps show you how to migrate this customization.
Add can to the bottom of your workspace_dir\search-config-ext\src\index\managed-solr\config\v3\common\stopwords.txt file on WebSphere Commerce Version 9.0.0.1+.
Add the customized stopwords.txt path in CONFIG column of SRCHCONFEXT table, by running the following SQL statement:
Update srchconfext set config='stopwords_en=../../../../managed-solr/config/v3-index-ext/common/stopwords.txt' 
where indextype='CatalogEntry' and indexscope='10001';
Migrate our customized pre-process files.
In WebSphere Commerce Version 8, when creating a customized preprocess.xml file to index external data, the customized file is under the solrhome/pre-processConfig/MC_$MasterCatalog/Dbtype directory. In WebSphere Commerce Version 9.0.0.1+, such pre-process XML files reside under the WCDE_installdir/xml/search/dataImport/v3/Dbtype/ directory.
Copy the wc-dataimport-preprocess-custom.xml file from your Feature Pack 8 environment to the WCDE_installdir/xml/search/dataImport/v3/Dbtype/ directory on WebSphere Commerce Version 9.0.0.1+. Also copy any files that are referenced by the wc-dataimport-preprocess-custom.xml file to WebSphere Commerce Version 9.0.0.1+, and update the wc-dataimport-preprocess-custom.xml file to ensure that it points to correct path file if you change the file location on WebSphere Commerce Version 9.0.0.1+. For example, if you specify an input file in the wc-dataimport-preprocess-custom.xml,

<_config:property name="inputFile" value="W:\WCDE_INT70\bin\Ratings.xml"/>
Then we must copy the Ratings.xml file from WebSphere Commerce Version 8 to WebSphere Commerce Version 9.0.0.1+, and put it under location WCDE_installdir\bin, and update wc-dataimport-preprocess-custom.xml to point to the new location.

<_config:property name="inputFile" value="W:\WCDE90\bin\Ratings.xml"/>
Migrate our customized data import handler files.
To index more data in WebSphere Commerce Version 8, you change the wc-data-config.xml file in the specific core directory. For example, to index more data from the X_RATING table in the English language, you edit wc-data-config.xml under the en_US core directory to allow query and deltaImportQuery to join the X_RATING table. In WebSphere Commerce Version 9.0.0.1+, the Solr DataImportHandler (DIH) gets data from the view VI_CE_#INDEX_SCOPE_TAG#_#lang_tag# and X_VI_CE_#INDEX_SCOPE_TAG#_#lang_tag#. Add this new customizable table into the view X_VI_CE_#INDEX_SCOPE_TAG#_#lang_tag# in wc-dataimport-preprocss-x-finalbuild.xml. See Configure the Data Import Handler mapping.
After the customized tables are added into X_VI_CE table, during DIH, customized data can be returned and accessible to Solr. Define the field-column mapping so that Solr can transfer the new customized column into Solr fields. Update the search-config-ext/src/main/resources/index/managed-solr/config/v3/CatalogEntry/x-data-config.xml file to add the field mapping.
The following is an example of the new mapping. With this mapping, the RATING column from the X_VI_CE view maps to the customerRanking field in the catalog entry index.
<field column="RATING" name="customerRanking"/>
Migrate the SRCHATTRPROP table to enable price range queries. In WebSphere Commerce version 8.0, the default price range facet takes the following format.
"price_USD:{* 100} 100;{100 200} 200;{200 300} 300;{300 400} 400;{400 500} 500;{500 *}"
In Solr version 7.3.1, this format causes a syntax error in the query parser. If we are using WebSphere Commerce version 9.0.0.5+, change the query string into the following format.
 "price_USD:{* TO 100];{100 TO 200];{200 TO 300];{300 TO 400];{400 TO 500];{500 TO *}" 
Note: Convert previous range query formats that take the form "({lower upper} upper)" into "({lower TO upper])". Migrate any other customization that involves the old query format to the new one.
Migrate the default schema field types. Starting with Solr version 7.0.0, the Trie*Field fields are deprecated. Replace them with *PointField. The default setting retains the old data type fields (for example, int, tint, sint) and creates new fields (for example, pint, pints). Although the old fields still function, for future compatibility upgrade the old data type to the new one. Some deprecated fields are still used, for example, protected field types, for compatible considerations.

Migrate any customized solrconfig.xml parameters. For Solr version 7.3.1, move the configuration parameter solr.mergeFactor in the solrconfig.xml file into the SRCHCONFEXT.CONFIG column. It is replaced by two parameters: solr.mergePolicy.maxMergeAtOnce and solr.mergePolicy.segmentsPerTier. If you previously set the value to something like <mergeFactor>5</mergeFactor>, replace it with solr.mergePolicy.maxMergeAtOnce=5,solr.mergePolicy.segmentsPerTier=5.
Migrate the WebSphere Commerce Search runtime configuration.
Migrate the UserData property name mapping. In WebSphere Commerce Version 8, when we want to enable some specific feature, such as result grouping or searchBasedSuggestion, you edit the Search.ear/xml/config/com.ibm.commerce.catalog/wc-component.xml file. When you add more data into the index at run time, you also add new field mappings into wc-component.xml. Here is an example of a field1 mapping in the wc-component.xml file:
<_config:valuemappingservice>  
    <_config:valuemapping externalName="CatalogEntryUserDataFieldNameMapping" 
       internalName="CatalogEntryUserDataFieldNameMapping">
      <_config:valuemap externalValue="x_customerRanking" internalValue="customerRanking_display"/>
    </_config:valuemapping>
  </_config:valuemappingservice>
In WebSphere Commerce Version 9.0.0.1+, you add such configuration information into the Workspace_dir/search-config-ext/runtime.config/com.ibm.commerce.search/wc-component.xml file. See Changing properties in the component configuration file (wc-component.xml).
Note: Except for the result group, all other features need only to be updated in the WebSphere Commerce Search server component configuration file.
Migrate the WebSphere Commerce Search configuration.
In order to create or change a WebSphere Commerce Search profile in WebSphere Commerce Version 8, you configure the Search.ear/xml/config/com.ibm.commerce.catalog-ext/wc-search.xml file. In WebSphere Commerce Version 9.0.0.1+, you update the Toolkit_Workspace/search-config-ext/runtime.config/com.ibm.commerce.search/wc-search.xml file. For example, WebSphere Commerce Version 8 has the following customized WebSphere Commerce search profiles.
<_config:profile name="IBM_findProductsBySearchTerm" indexName="CatalogEntry">
		<_config:query>
			<_config:postprocessor
				classname="com.ibm.commerce.foundation.server.services.rest.search.postprocessor.solr.SolrRESTSearchCatalogEntryViewUserDataQueryPostprocessor" />
		</_config:query>
		<_config:result>
			<_config:field name="customerRanking_display" />
		</_config:result>            
	</_config:profile>
	
	<_config:profile name="IBM_findProductsByCategory" indexName="CatalogEntry">   	
    	<_config:query> 
    	<_config:provider classname="com.mycompany.search.internal.expression.provider.CustomizedExpressionProvider"/> 
    	<_config:postprocessor classname="com.mycompany.search.internal.expression.postprocessor.CustomizedGiftPostprocessor" />  
        </_config:query>        
    </_config:profile>
    
    <_config:profile name="IBM_findProductByIds_Details" indexName="CatalogEntry"> 	
    	<_config:query> 
    	<_config:postprocessor classname="com.mycompany.search.internal.expression.postprocessor.CustomizedGiftPostprocessor" />  
        </_config:query>          
    </_config:profile>
In WebSphere Commerce Version 9.0.0.1+, you add this configuration information into the Workspace_dir/search-config-ext/runtime.config/com.ibm.commerce.search/wc-component.xml file. For instructions on configuring this file, see Changing properties in the WebSphere Commerce Search configuration file (wc-search.xml).
After running the WCB process and pipeline deployment, the runtime configuration files are deployed intoLiberty_installdir/resources/search/runtime/config/com.ibm.commerce.search-ext directory.
In WebSphere Commerce Version 8, we might create an SQL query in the Search_eardir/xml/config/com.ibm.commerce.catalog/wc-query-utilities.tpl file and rely on the JDBCQueryService to retrieve data from the WebSphere Commerce Search server. In WebSphere Commerce Version 9.0.0.1+, we must update the workspace_dir/search-config-ext/runtime.config/com.ibm.commerce.search/wc-query.tpl file. Later, when we deploy our customizations, the .tpl file is deployed to the Liberty_installdir/resources/search/runtime/config/com.ibm.commerce.search-ext directory.
In previous versions, we might create custom expression providers, preprocessors, or postprocessors under the WebSphereCommerceServerExtensionsLogic project. In WebSphere Commerce Version 9.0.0.1+, we must create these customizations in the workspace_dir/search-logic-ext directory.
What to do next
The next step in the migration process is to build and deploy our custom containers. After those containers are deployed, we can build your index. For more information about building your index, see Building the WebSphere Commerce Search index.