The Authentication Header (AH) protocol provides data origin authentication, data integrity, and replay protection. However, AH does not provide data confidentiality, which means that all of your data is sent in the clear.
AH ensures data integrity with the checksum that a message authentication code, like MD5, generates. To ensure data origin authentication, AH includes a secret shared key in the algorithm that it uses for authentication. To ensure replay protection, AH uses a sequence number field within the AH header. It is worth noting here, that these three distinct functions are often lumped together and referred to as authentication. In the simplest terms, AH ensures that your data has not been tampered with en route to its final destination.
Although AH authenticates as much of the IP datagram as possible, the values of certain fields in the IP header cannot be predicted by the receiver. AH does not protect these fields, known as mutable fields. However, AH always protects the payload of the IP packet.
The Internet Engineering Task Force (IETF) formally defines AH in Request for Comment (RFC) 2402, IP Authentication Header. You can view this RFC on the Internet at the following Web site: http://www.rfc-editor.org.
You can apply AH in two ways: transport mode or tunnel mode. In transport mode, the IP header of the datagram is the outermost IP header, followed by the AH header and then the payload of the datagram. AH authenticates the entire datagram, except the mutable fields. However, the information contained in the datagram is transported in the clear and is, therefore, subject to eavesdropping. Transport mode requires less processing overhead than tunnel mode, but does not provide as much security.
Tunnel mode creates a new IP header and uses it as the outermost IP header of the datagram. The AH header follows the new IP header. The original datagram (both the IP header and the original payload) comes last. AH authenticates the entire datagram, which means that the responding system can detect whether the datagram changed while in transit.
When either end of a security association is a gateway, use tunnel mode. In tunnel mode the source and destination addresses in the outermost IP header do not need to be the same as those in the original IP header. For example, two security gateways may operate an AH tunnel to authenticate all traffic between the networks they connect together. In fact, this is a very typical configuration.
The main advantage to using tunnel mode, is that tunnel mode totally protects the encapsulated IP datagram. In addition, tunnel mode makes it possible to use private addresses.
In many cases, your data only requires authentication. While the Encapsulating Security Payload (ESP) protocol can perform authentication, AH does not affect your system performance as does ESP. Another advantage of using AH, is that AH authenticates the entire datagram. ESP, however, does not authenticate the leading IP header or any other information that comes before the ESP header.
In addition, ESP requires strong cryptographic algorithms in order to be put into effect. Strong cryptography is restricted in some regions, while AH is not regulated and can be used freely around the world.
If you use the AH protocol then you might want to enable Extended Sequence Number (ESN). ESN allows you to transmit large volumes of data at a high speed with out re-keying. The VPN connection uses a 64-bit sequence numbers instead of 32-bit numbers over IPSec. Using 64-bit sequence numbers allows more time before re-keying, which prevents sequence number exhaustion and minimizes the use of system resources.
AH uses algorithms known as hashed message authentication codes (HMAC). Specifically, VPN uses either HMAC-MD5 or HMAC-SHA. Both MD5 and SHA take variable-length input data and a secret key to produce fixed-length output data (called a hash value). If the hashes of two messages match, then it is very likely that the messages are the same. Both MD5 and SHA encode the message length in their output, but SHA is regarded as more secure because it produces larger hashes.
The Internet Engineering Task Force (IETF) formally defines HMAC-MD5 in Request for Comments (RFC) 2085, HMAC-MD5 IP Authentication with Replay Prevention. The Internet Engineering Task Force (IETF) formally defines HMAC-SHA in Request for Comments (RFC) 2404, The Use of HMAC-SHA-1-96 within ESP and AH. You can view these RFCs on the Internet at the following Web site: http://www.rfc-editor.org.
Related concepts
Encapsulating Security Payload