Content negotiation for HTTP Server (powered by Apache)

Content negotiation for HTTP Server (powered by Apache)

In the IBM HTTP Server for i5/OS, content negotiation, type-map files, MultiViews, negotiation methods, dimensions of negotiation, negotiation algorithm, media types, and wildcards is supported.
Information for this topic supports the latest PTF levels for HTTP Server for iSeries . IBM recommends that you install the latest PTFs to upgrade to the latest level of the HTTP Server for i5/OS. Some of the topics documented here are not available prior to this update. See IBM Service for more information.
A resource may be available in several different representations. For example, it might be available in different languages or different media types, or a combination. One way of selecting the most appropriate choice is to give the user an index page, and let them select; however it is often possible for the server to choose automatically. This works because browsers can send as part of each request information about what representations it prefers. For example, a browser could indicate that it would like to see information in French, if possible, else English will do. Browsers indicate their preferences by headers in the request. To request only French representations, the browser would send:
Accept-Language: fr
Note that this preference will only be applied when there is a choice of representations and they vary by language.
As an example of a more complex request, this browser has been configured to accept French and English, but prefers French, and to accept various media types, preferring HTML over plain text or other text types, and preferring GIF or JPEG over other media types, but also allowing any other media type as a last resort:
Accept-Language: fr; q=1.0, en; q=0.5
Accept: text/html; q=1.0, text/*; q=0.8, image/gif; q=0.6,
   image/jpeg; q=0.6, image/*; q=0.5, */*; q=0.1
The HTTP Server (powered by Apache) supports 'server driven' content negotiation, as defined in the HTTP/1.1 specification. It fully supports the Accept, Accept-Language, Accept-Charset and Accept-Encoding request headers. The HTTP Server (powered by Apache) also supports 'transparent' content negotiation, which is an experimental negotiation protocol defined in RFC 2295 and RFC 2296. It does not offer support for 'feature negotiation' as defined in these RFCs.
A resource is a conceptual entity identified by a URI (RFC 2396). The HTTP Server (powered by Apache) provides access to representations of the resource(s) within its namespace, with each representation in the form of a sequence of bytes with a defined media type, character set, encoding, or other. Each resource may be associated with zero, one, or more than one representation at any given time. If multiple representations are available, the resource is referred to as negotiable and each of its representations is termed a variant. The ways in which the variants for a negotiable resource vary are called the dimensions of negotiation.

Parent topic:
Concepts of functions of HTTP Server

Content negotiation

In order to negotiate a resource, the server needs to be given information about each of the variants. This is done in one of two ways:

Using a type -map (for example, a *.var file) which names the files containing the variants explicitly.

Using a 'MultiViews' search, where the server does an implicit filename pattern match and chooses from among the results.

Using a type-map file

A type map is a document which is associated with the handler named type-map (or, for backwards-compatibility with older HTTP Server (powered by Apache) configurations, the mime type application/x-type-map). Note that to use this feature, have a handler set in the configuration that defines a file suffix as type-map; this is best done with an AddHandler in the server configuration file, as shown below.
AddHandler type-map var
Type map files have an entry for each available variant; these entries consist of contiguous HTTP-format header lines. Entries for different variants are separated by blank lines. Blank lines are illegal within an entry. It is conventional to begin a map file with an entry for the combined entity as a whole (although this is not required, and if present will be ignored). An example map file is:
URI: jkl 
URI: jkl.en.html Content-type: text/html Content-language: en 
URI: jkl.fr.de.html Content-type: text/html;charset=iso-8859-2
Content-language: fr, de
If the variants have different source qualities, that may be indicated by the "qs" parameter to the media type, as in this picture (available as jpeg, gif, or ASCII-art):
URI: jkl 
URI: jkl.jpeg Content-type: image/jpeg; Qs=0.8

URI: jkl.gif Content-type: image/gif; Qs=0.5

URI: jkl.txt Content-type: text/plain; Qs=0.01
The "Qs" value can vary in the range 0.000 to 1.000. Note that any variant with a "Qs" value of 0.000 will never be chosen. Variants with no "Qs" parameter value are given a "Qs" factor of 1.0. The "Qs" parameter indicates the relative 'quality' of this variant compared to the other available variants, independent of the client's capabilities. For example, a jpeg file is usually of higher source quality than an ASCII file if its attempting to represent a photograph; however, if the resource being represented is an original ASCII art, then an ASCII representation would have a higher source quality than a jpeg representation. A "Qs" value is therefore specific to a given variant depending on the nature of the resource it represents.
The full list of headers recognized are:

URI

The uri of the file containing the variant (of the given media type, encoded with the given content encoding). These are interpreted as URLs relative to the map file; they must be on the same server, and they must refer to files to which the client would be granted access if they were to be requested directly.

Content-Type

The media type --- charset, level and "Qs" parameters may be given. These are often referred to as MIME types; typical media types are image/gif, text/plain, or text/html; level=3.

Content-Language

The languages of the variant, specified as an Internet standard language tag from RFC 1766 (for example, en for English, or kr for Korean).

Content-Encoding

If the file is compressed, or otherwise encoded, rather than containing the actual raw data, this states how it was done. The HTTP Server (powered by Apache) only recognizes encodings that are defined by an AddEncoding directive. This normally includes the encodings x-compress for compressed files, and x-gzip for gzip'd files. The x- prefix is ignored for encoding comparisons.

Content-Length

The size of the file. Specifying content lengths in the type-map allows the server to compare file sizes without checking the actual files.

Description

A human-readable textual description of the variant. If the HTTP Server (powered by Apache) cannot find any appropriate variant to return, it will return an error response which lists all available variants instead. Such a variant list will include the human-readable variant descriptions.

MultiViews

MultiViews is a per-directory option, meaning it can be set with an Options directive within a <Directory>, <Location> or <Files> container in the configuration file, or (if AllowOverride is properly set) in .htaccess files. Note that Options All does not set MultiViews; you have to ask for it by name.
The effect of MultiViews is as follows: if the server receives a request for /some/dir/jkl, if /some/dir has MultiViews enabled, and /some/dir/jkl does not exist, then the server reads the directory looking for files named jkl.*, and effectively fakes up a type map which names all those files, assigning them the same media types and content-encodings it would have if the client had asked for one of them by name. It then chooses the best match to the client's requirements.
MultiViews may also apply to searches for the file named by the DirectoryIndex directive, if the server is trying to index a directory. If the configuration files specify:
DirectoryIndex index
The server will arbitrate between index.html and index.html3 if both are present.
If one of the files found when reading the directive is a CGI script, it is not obvious what should happen. The code gives that case special treatment --- if the request was a POST, or a GET with QUERY_ARGS or PATH_INFO, the script is given an extremely high quality rating, and generally invoked; otherwise it is given an extremely low quality rating, which generally causes one of the other views (if any) to be retrieved.

The negotiation methods

After the HTTP Server (powered by Apache) has obtained a list of the variants for a given resource, either from a type-map file or from the filenames in the directory, it invokes one of two methods to decide on the 'best' variant to return, if any. It is not necessary to know any of the details of how negotiation actually takes place in order to use the HTTP Server (powered by Apache) content negotiation features. However the rest of this document explains the methods used for those interested.
There are two negotiation methods:

Server driven negotiation with the HTTP Server (powered by Apache) algorithm is used in the normal case. The HTTP Server (powered by Apache) algorithm is explained in more detail below. When this algorithm is used, the HTTP Server (powered by Apache) can sometimes 'fiddle' the quality factor of a particular dimension to achieve a better result. The ways the HTTP Server (powered by Apache) can fiddle quality factors is explained in more detail below.

Transparent content negotiation is used when the browser specifically requests this through the mechanism defined in RFC 2295. This negotiation method gives the browser full control over deciding on the 'best' variant, the result is therefore dependent on the specific algorithms used by the browser. As part of the transparent negotiation process, the browser can ask the HTTP Server (powered by Apache) to run the 'remote variant selection algorithm' defined in RFC 2296.

Dimensions of negotiation

Media Type

Browser indicates preferences with the Accept header field. Each item can have an associated quality factor. Variant description can also have a quality factor (the "Qs" parameter).

Language

Browser indicates preferences with the Accept-Language header field. Each item can have a quality factor. Variants can be associated with none, one or more than one language.

Encoding

Browser indicates preference with the Accept-Encoding header field. Each item can have a quality factor.

Charset

Browser indicates preference with the Accept-Charset header field. Each item can have a quality factor. Variants can indicate a charset as a parameter of the media type.

Client (Browser)

The User-Agent HTTP header is used to determine browser type.

The negotiation algorithm

The HTTP Server (powered by Apache) can use the following algorithm to select the 'best' variant (if any) to return to the browser. This algorithm is not further configurable. It operates as follows:

First, for each dimension of the negotiation, check the appropriate Accept* header field and assign a quality to each variant. If the Accept* header for any dimension implies that this variant is not acceptable, eliminate it. If no variants remain, go to step 4.

Select the 'best' variant by a process of elimination. Each of the following tests is applied in order. Any variants not selected at each test are eliminated. After each test, if only one variant remains, select it as the best match and proceed to step 3. If more than one variant remains, move on to the next test.

Multiply the quality factor from the Accept header with the quality-of-source factor for this variant's media type, and select the variants with the highest value.

Select the variants with the highest language quality factor.

Select the variants with the best language match, using either the order of languages in the Accept-Language header (if present), or else the order of languages in the LanguagePriority directive (if present).

Select the variants with the highest 'level' media parameter (used to give the version of text/html media types).

Select variants with the best charset media parameters, as given on the Accept-Charset header line. Charset ISO-8859-1 is acceptable unless explicitly excluded. Variants with a text/* media type but not explicitly associated with a particular charset are assumed to be in ISO-8859-1.

Select those variants which have associated charset media parameters that are not ISO-8859-1. If there are no such variants, select all variants instead.

Select the variants with the best encoding. If there are variants with an encoding that is acceptable to the user-agent, select only these variants. Otherwise if there is a mix of encoded and non-encoded variants, select only the non-encoded variants. If either all variants are encoded or all variants are not encoded, select all variants.

Select the variants that correspond to the User-Agent header received on the HTTP Request.

Select the variants with the smallest content length.

Select the first variant of those remaining. This will be either the first listed in the type-map file, or when variants are read from the directory, the one whose file name comes first when sorted using ASCII code order.

The algorithm has now selected one 'best' variant, so return it as the response. The HTTP response header Vary is set to indicate the dimensions of negotiation (browsers and caches can use this information when caching the resource).

To get here means no variant was selected (because none are acceptable to the browser). Return a 406 status (meaning "No acceptable representation") with a response body consisting of an HTML document listing the available variants. Also set the HTTP Vary header to indicate the dimensions of variance.

Editing quality values

The HTTP Server (powered by Apache) sometimes changes the quality values from what would be expected by a strict interpretation of the HTTP Server (powered by Apache) negotiation algorithm above. This is to get a better result from the algorithm for browsers which do not send full or accurate information. Some of the most popular browsers send Accept header information which would otherwise result in the selection of the wrong variant in many cases. If a browser sends full and correct information these fiddles will not be applied.

Media types and wildcards

The Accept: request header indicates preferences for media types. It can also include 'wildcard' media types, such as "image/*" or "*/*" where the * matches any string. So a request including Accept: image/*, */* would indicate that any type starting "image/" is acceptable, as is any other type (so the first "image/*" is redundant). Some browsers routinely send wildcards in addition to explicit types they can handle. For example, Accept: text/html, text/plain, image/gif, image/jpeg, */*.
The intention of this is to indicate that the explicitly listed types are preferred, but if a different representation is available, that is OK too. However under the basic algorithm, as given above, the */* wildcard has exactly equal preference to all the other types, so they are not being preferred. The browser should really have sent a request with a lower quality (preference) value for *.*, such as: Accept: text/html, text/plain, image/gif, image/jpeg, */*; q=0.01.
The explicit types have no quality factor, so they default to a preference of 1.0 (the highest). The wildcard */* is given a low preference of 0.01, so other types will only be returned if no variant matches an explicitly listed type.
If the Accept: header contains no "q" factors at all, the HTTP Server (powered by Apache) sets the "q" value of "*/*", if present, to 0.01 to emulate the desired behavior. It also sets the "q" value of wildcards of the format "type/*" to 0.02 (so these are preferred over matches against "*/*"). If any media type on the Accept: header contains a "q" factor, these special values are not applied, so requests from browsers which send the correct information to start with work as expected.

Variants with no language

If some of the variants for a particular resource have a language attribute, and some do not, those variants with no language are given a very low language quality factor of 0.001.
The reason for setting this language quality factor for variant with no language to a very low value is to allow for a default variant which can be supplied if none of the other variants match the browser's language preferences. For example, consider the situation with three variants:

jkl.en.html, language en

jkl.fr.html, language fr

jkl.html, no language

The meaning of a variant with no language is that it is always acceptable to the browser. If the request Accept-Language header includes either en or fr (or both) one of jkl.en.html or jkl.fr.html will be returned. If the browser does not list either en or fr as acceptable, jkl.html will be returned instead.

Extensions to transparent content negotiation

The HTTP Server (powered by Apache) extends the transparent content negotiation protocol (RFC 2295) as follows. A new {encoding ..} element is used in variant lists to label variants which are available with a specific content-encoding only. The implementation of the RVSA/1.0 algorithm (RFC 2296) is extended to recognize encoded variants in the list, and to use them as candidate variants whenever their encodings are acceptable according to the Accept-Encoding request header. The RVSA/1.0 implementation does not round computed quality factors to 5 decimal places before choosing the best variant.

Notes on hyperlinks and naming conventions

If you are using language negotiation you can choose between different naming conventions, because files can have more than one extension, and the order of the extensions is normally irrelevant (see mod_mime for details).
A typical file has a MIME-type extension (for example, html), maybe an encoding extension (for example, gz), and of course a language extension (for example, en) when we have different language variants of this file.
Examples:

jkl.en.html

jkl.html.en

jkl.en.html.gz

Examples of filenames together with valid and invalid hyperlinks:

Filename Valid hyperlink Invalid hyperlink

jkl.html.en
jkl
jkl.html
-
jkl.en.html jkl jkl.html
jkl.html.en.gz
jkl
jkl.html

jkl.gz
jkl.html.gz
jkl.en.html.gz jkl
jkl.html
jkl.html.gz
jkl.gz
jkl.gz.html.en
jkl
jkl.gz
jkl.gz.html
jkl.html
jkl.html.gz.en
jkl
jkl.html
jkl.html.gz
jkl.gz

Looking at the table above you will notice that it is always possible to use the name without any extensions in an hyperlink (for example, jkl). The advantage is that you can hide the actual type of a document rsp. file and can change it later, for example, from html to shtml or cgi without changing any hyperlink references.
If you want to continue to use a MIME-type in your hyperlinks (for example jkl.html) the language extension (including an encoding extension if there is one) must be on the right hand side of the MIME-type extension (for example, jkl.html.en).

Notes on caching

When a cache stores a representation, it associates it with the request URL. The next time that URL is requested, the cache can use the stored representation. But, if the resource is negotiable at the server, this might result in only the first requested variant being cached and subsequent cache hits might return the wrong response. To prevent this, the HTTP Server (powered by Apache) normally marks all responses that are returned after content negotiation as non-cacheable by HTTP/1.0 clients. The HTTP Server (powered by Apache) also supports the HTTP/1.1 protocol features to allow caching of negotiated responses.
For requests which come from an HTTP/1.0 compliant client (either a browser or a cache), the directive CacheNegotiatedDocs can be used to allow caching of responses which were subject to negotiation. This directive can be given in the server config or virtual host, and takes no arguments. It has no effect on requests from HTTP/1.1 clients.

Filename	Valid hyperlink	Invalid hyperlink
jkl.html.en	jkl jkl.html	-
jkl.en.html	jkl	jkl.html
jkl.html.en.gz	jkl jkl.html	jkl.gz jkl.html.gz
jkl.en.html.gz	jkl	jkl.html jkl.html.gz jkl.gz
jkl.gz.html.en	jkl jkl.gz jkl.gz.html	jkl.html
jkl.html.gz.en	jkl jkl.html jkl.html.gz	jkl.gz