Client identification for search of the portal by external search engines

 

+
Search Tips   |   Advanced Search

 

For the portal to recognize external search engines, portal provides a client that covers several popular search engines. This client is implemented according to the Composite Capability/Preference Profiles (CC/PP) standard. It has the capability HTML_SEARCH set. To add more search engines, we can configure the client as required.

The client has been implemented with the following settings:

User agent:

(.*(B|b)ot.*)|(.*BOT.*)|(.*(S|s)pider.*)|(.*(S|s)earch.*)|(.*(C|c)rawl(er)?.*)| (.*(G|g)rabber.*)|(.*(Y|y)ahoo.*)|(.*(S|s)lurp.*)|(.*Lycos.*)|(.*Wget.*)

This user agent covers most available large search engines, such as Google, Yahoo!, Lycos, or MSN. This pattern list also accommodates all other search engines that include segments of bot, spider, search, or crawler.

Capability:

For each search engine that you want to be able to crawl the portal, we need to set the capability HTML_SEARCH. Search engines usually visit a Web site twice, the first time to crawl the site, and the second time to validate the content. When a search engine visits a site for the second time, it usually does so by using a normal browser. Therefore enter additional capabilities for supporting the different browser settings. Examples: (HTML_4_0, HTML_IFRAME, HTML_FRAME, HTML_NESTED_TABLE, HTML_2_0, HTML_JAVASCRIPT, HTML_3_2, HTML_3_0, HTML_CSS, HTML_TABLE).

Manufacturer:

Search

Markup:

HTML

To include search engines that are not covered by the default set, we can do so by either of the following ways:

• Adding search engines by using the administration portlet Manage Clients

• Adding search engines by using the XML configuration interface

Notes:

1. The search mechanism works correctly for the portal only if the search engine robots are identified to the portal in advance.

2. Search on the portal by external search engines requires additional configuration beyond client identification. For more details about this refer to Search by external search services.

 

Add search engines by using the administration portlet Manage Clients

To add search engines by using the administration portlet Manage Clients...

1. Navigate to the Manage Clients portlet by clicking Main Menu > Administration > Portal Settings > Supported Clients. Portal opens the Manage Clients portlet.

2. Depending on whether you want to add more search clients to the default user agent or add a complete new client, perform one of the following steps:
   • Select the client that starts with (.*(B|b)ot.*)|(.*BOT.*)|(.*(S|s)pider.*) ... from the list of clients and edit it. Use this option if you simply want to add one or more search engines.

   • Add a new client. For example, we can use this option, to give the newly added search engine priority by setting it to the First position in the client list.

   For details about how to do this refer to the Manage Clients portlet help.

3. Update or fill the fields and select the options as required:

   User agent:

   Type the new search engine user agent.

   Markup:

   html

   Manufacturer

   Search Engine Manufacturer. This field is optional.

   Capability:

   HTML_SEARCH, HTML_4_0, HTML_IFRAME, HTML_FRAME, HTML_NESTED_TABLE, HTML_2_0, HTML_JAVASCRIPT, HTML_3_2, HTML_3_0, HTML_CSS, HTML_TABLE

   Position:

   First. Set the specified search engine to the first position, so that it is correctly recognized. The reason for this is that the pattern matching for the comparison of the user agents to the supported clients is done from concrete and specific to general.

   For a more detailed description of the fields and options refer to the Manage Clients portlet help.

4. Click OK to save changes.

 

Add search engines by using the XML configuration interface

To add search engines by using the XML configuration interface import them by an XML script file. For more information refer to The XML configuration interface. To verify the search mechanism works correctly, we need to add the capability HTML_SEARCH. An example XML script is:

<?xml version="1.0" encoding="UTF-8"?>
<request    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
            xsi:noNamespaceSchemaLocation="PortalConfig_1.4.xsd"
            type="update" create-oids="true">
    <portal action="locate">

        <client action="update" uniquename="wps.client.search.Our Search Engine Name" 
                manufacturer="Our Search Engine Manufacturer" markup="html">

            <useragent-pattern>Our User-Agent Pattern</useragent-pattern>

            <client-capability update="set">HTML_SEARCH</client-capability>
            <client-capability update="set">HTML_4_0</client-capability>
            <client-capability update="set">HTML_IFRAME</client-capability>
            <client-capability update="set">HTML_FRAME</client-capability>
            <client-capability update="set">HTML_NESTED_TABLE</client-capability>
            <client-capability update="set">HTML_2_0</client-capability>
            <client-capability update="set">HTML_JAVASCRIPT</client-capability>
            <client-capability update="set">HTML_3_2</client-capability>
            <client-capability update="set">HTML_3_0</client-capability>
            <client-capability update="set">HTML_CSS</client-capability>
            <client-capability update="set">HTML_TABLE</client-capability>

        </client>
    </portal>
</request>

 

Parent Topic

Search by external search services

 

Related concepts

The XML configuration interface

 

---