WebSphere Commerce Search index schema

The WebSphere Commerce Search index process involves a search index schema, and is built from temporary tables.

Indexing WebSphere Commerce content with WebSphere Commerce Search requires a well-structured index schema design, so that populating and querying the index is efficient when searches are submitted.

Solr server instance

A self-contained run time environment, which includes Solr server process, Solr home and Solr Cores.

Solr Core

Enables one Solr instance to contain multiple configurations and indexes, thus being more efficient than dedicated Solr instances for each configuration and index. Due to the hot core reloading and swapping features, Solr Core is recommended.

Solr index

Solr maintains one or more indexes, which are searchable collections of items called documents. When we use Solr to support catalog search, the documents in the index represent catalog entries.

Adding a document to an index is often referred to as indexing the document.

Each document is composed of a set of attributes called fields. For example, a catalog entry document can have fields such as Partnumber, Name, and Description.

When a document is added to an index, each field in the document can be indexed or stored, or indexed and stored. An indexed field is one that can be used for searching, sorting, and faceting. If the field is indexed, document IDs can be determined from field values. A stored field is one where its value can be retrieved by a search. Alternatively, if the field is stored, field values can be determined from document IDs.

Solr schema

Defined in the Solr configuration file, schema.xml. It defines the schema fields and data type of the fields.

Solr data import handler (DIH)

Provides a configuration-driven method of importing data from relational databases or XML to the Solr index.

Solr home

The root directory of the Solr configuration files and index data files. Each Solr instance can map to one Solr home.

The WebSphere Commerce Search schema-related information is stored in the schema.xml file, with other configuration information stored in the solrconfig.xml file. We can customize these files directly. The unique key of the index is the catentry_id field. That is, the index document is only for CATENTRY. If other WebSphere Commerce objects must be indexed, for example, CATGROUP, separate index document must be created. The default query operator is set to OR.

The following default field types are available, providing advanced index functions:

wc_text

Used for searchable fields. The field is tokenized and lower-cased to support not case-sensitive searches. The dictionary functions are enabled, such as synonyms, stop words, and stemming.

wc_keywordText

Used for generic sorting and faceting. The field is non-tokenized. No dictionary functions are enabled. In general, WebSphere Commerce Search requires a sorting or faceting field to be indexed, but not multivalued or tokenized.

wc_keywordTextLowerCase

Used for not case-sensitive sorting and faceting. The field is non-tokenized, and lower-cased. No dictionary functions are enabled. In general, WebSphere Commerce Search requires a sorting or faceting field to be indexed, but not multivalued or tokenized.

Search dictionary files

Synonyms, stop words, and stemming are controlled in the following ways:

Synonyms

Synonym expansion is implemented by using WebSphere Commerce web services.

Stop words

Basic United States English stop words are contained in the stopwords.txt file. This stop word list contains words that are merged from the default Solr stop words and ODE stop words. We can manually maintain stop words in the stopwords.txt file if necessary.

Stemming

The EnglishPorterFilterFactory is used for the English language, as this factory supports user-defined protected words, and performs relatively well. The SnowballPorterFilterFactory is used for other languages. Stemming is only enabled in the wc_text field type. We can manually maintain protected words in the protwords.txt file if necessary.

Highlighting search keywords

Highlighting keywords is a default query component. The query controls how the highlighting works. The field must be stored in the index for highlighting to be enabled. The field option stored must be set to true in the schema.xml file.

Spelling correction

The indexed content is used to create the term dictionary, so that the generated dictionary data is relevant to the indexed data. Spell checking is enabled in the search request handler in the wc_spellcheck component in the solrconfig.xml file. The wc_textSpell field type and spellCheck field is created in the schema.xml file. The spellCheck field contains the field's name, shortDescription, and keyWord by default.

The following sample query searches for coffee with spellcheck enabled:

http://localhost/solr/CatalogEntry/select?indent=on&version=2.2&q=shortDescription%3Acofffee
&fq=&start=0&rows=10&fl=*%2CsCore&qt=standard&wt=standard&explainOther=&hl.fl=&
spellcheck=true&spellcheck.collate=true

Spell checker component

A more efficient spell checker, DirectSolrSpellChecker, is used instead of the spell check index. This spell checker component uses data directly from the CatalogEntry index, instead of relying on a separate stand-alone index. Therefore, no additional index builds are required to synchronize changes between the base index and the spell checker index. See SpellCheckComponent.

Automatic keyword suggestions

The TermsComponent is implemented in WebSphere Commerce for auto-suggest functions. The component provides fast field faceting over the entire index. That is, it is not restricted by the base query or any filters. The document frequencies returned are the number of documents that match the term, including any documents marked for deletion but not yet removed from the index.

Retrieving terms from the index order is relatively fast since the implementation directly uses the Lucene TermEnum functions to iterate over the term dictionary.

The wc_termsComponent search component and wc_terms request handler are created in the solrconfig.xml file. The auto-suggest query points to the wc_terms request handler to get the auto-suggest terms. For example:

http://localhost/solr/MC_10001_CatalogEntry_en_US/terms?terms.fl=shortDescription&terms.sort=index&erms.limit=5&terms.prefix=ligh

Default search scope

The following columns are searched on by default, that is, instead of searching on the entire search index, if no index field is given:

• Product description (name)
• Short description (shortDescription)
• Part number (partNumber_ntk)
• Keyword
• Attribute Dictionary attribute values

However, the WebSphere Commerce Search runtime always passes in a search scope and is defined as part of the search profile. See WebSphere Commerce Search configuration file (wc-search.xml).

Schema changes for related structured and unstructured content

When structured content contains a relationship with unstructured content, it must contain a new field in the structured schema.xml file to represent the unstructured information. This new field can query the structured objects by their unstructured content. For example, when you search for products by the attachments' content information, the following new field definition resembles the following form:

<field name="unstructure" type="wc_text" indexed="true" stored="false" multiValued="true" />

Where the stored="false" snippet enables unstructured content to not be retrieved by queries.

See

1. WebSphere Commerce Search index schema definition
   

2. Temporary table schema definition
   

3. Logical representation of indexed columns
   

4. WebSphere Commerce Search extension indexes
   

Related concepts
Workspaces in WebSphere Commerce Search