Management Center: Search rule experiment statistics

Search rule experiment statistics

By understanding how the server collects and calculates statistics for the test elements in our search rule experiments, we can effectively evaluate the results of the experiment.
Here is an example of the statistical data that is provided for a search rule experiment:

The following table describes the statistical data:

Statistic Description
Total number of unique customers The number of individual customers who participated in the experiment.
Unique Customers The number of individual customers who were assigned the path and reached the test element.
Clicks The number of times any link on the first page ¹ of search results was clicked, which indicates that the search results had relevance.
This does not necessarily mean that the customer clicked a link for a product that is directly related to the search rule action. For example, if our search rule action is to boost Brand A products in the search results, the customer does not need to click a Brand A product to increase the Clicks count.
If the customer clicks a featured product displayed on the search results page, this click is not counted.
Click Orders The number of orders that are placed by customers who were influenced by the search results.
For the order to count in the statistics, the customer must add any item that is listed in the first page¹ of search results to the shopping cart from the product display page that displays right after the customer clicks a link on the search results page.
Click Revenue For orders that counted as Click orders, the total value of all those orders.
For example, if a path has a count of 10 Click Orders, and each order totaled $10, the Click Revenue value is $100. This value is for the entire order, not just for the specific items in the order that were listed on the first page¹ of search results.

1. By default, only the first page of search results is considered in click, order, and revenue statistics. However, a site administrator can change the default value of the SearchStatisticsResultPagesTrackingThreshold parameter to consider a different number of search result pages, for example, the first three pages.

We can view these statistics while the experiment is running and after it ends. See Viewing the statistics of a search rule experiment.

Enable the collection of revenue statistics
To collect revenue statistics, a site administrator must enable the following statistical counters before your experiment starts:

ExperimentEvaluationEventListener
OrderSubmissionEvent

Frequency of statistical updates

The following table identifies how often statistics are updated in the Statistics tab while an experiment is running.

In the Clicks column The server gathers and displays these numbers using the SaveMarketingStatistics scheduled job. By default, this job runs every 15 minutes.
A site administrator can change the frequency of this job; see Scheduling the SaveMarketingStatistics job.
In the Click Orders and Click Revenue columns The server gathers and displays these numbers using the RaiseECEvent scheduled job. By default, this job runs every 5 minutes.
A site administrator can change the frequency of this job.

Experiment statistics in a staging environment
Search rule statistics are captured on the production environment. To view experiment statistics in a staging environment for analysis, a system administrator can transfer the statistical data from the production environment to the staging environment by running the ExportStats and ImportStats commands. When you run the commands, specify the DMEXPSTATS and DMELESTATS tables.

Related tasks
Viewing the statistics of a search rule experiment
Creating search rule experiments
Selecting the winning path for a search rule experiment
Suspending or resuming search rule experiments

Statistic	Description
Total number of unique customers	The number of individual customers who participated in the experiment.
Unique Customers	The number of individual customers who were assigned the path and reached the test element.
Clicks	The number of times any link on the first page ¹ of search results was clicked, which indicates that the search results had relevance. This does not necessarily mean that the customer clicked a link for a product that is directly related to the search rule action. For example, if our search rule action is to boost Brand A products in the search results, the customer does not need to click a Brand A product to increase the Clicks count. If the customer clicks a featured product displayed on the search results page, this click is not counted.
Click Orders	The number of orders that are placed by customers who were influenced by the search results. For the order to count in the statistics, the customer must add any item that is listed in the first page¹ of search results to the shopping cart from the product display page that displays right after the customer clicks a link on the search results page.
Click Revenue	For orders that counted as Click orders, the total value of all those orders. For example, if a path has a count of 10 Click Orders, and each order totaled $10, the Click Revenue value is $100. This value is for the entire order, not just for the specific items in the order that were listed on the first page¹ of search results.
1. By default, only the first page of search results is considered in click, order, and revenue statistics. However, a site administrator can change the default value of the SearchStatisticsResultPagesTrackingThreshold parameter to consider a different number of search result pages, for example, the first three pages.

In the Clicks column	The server gathers and displays these numbers using the SaveMarketingStatistics scheduled job. By default, this job runs every 15 minutes. A site administrator can change the frequency of this job; see Scheduling the SaveMarketingStatistics job.
In the Click Orders and Click Revenue columns	The server gathers and displays these numbers using the RaiseECEvent scheduled job. By default, this job runs every 5 minutes. A site administrator can change the frequency of this job.