The Encapsulating Security Payload (ESP) protocol provides data confidentiality, and also optionally provides data origin authentication, data integrity checking, and replay protection.
The difference between ESP and the Authentication Header (AH) protocol is that ESP provides encryption, while both protocols provide authentication, integrity checking, and replay protection. With ESP, both communicating systems use a shared key for encrypting and decrypting the data they exchange.
If you decide to use both encryption and authentication, then the responding system first authenticates the packet and then, if the first step succeeds, the system proceeds with decryption. This type of configuration reduces processing overhead, as well as reduces your vulnerability to denial-of-service attacks.
You can apply ESP in two ways: transport mode or tunnel mode. In transport mode, the ESP header follows the IP header of the original IP datagram. If the datagram already has an IPSec header, then the ESP header goes before it. The ESP trailer and the optional authentication data follow the payload.
Transport mode does not authenticate or encrypt the IP header, which might expose your addressing information to potential attackers while the datagram is in transit. Transport mode requires less processing overhead than tunnel mode, but does not provide as much security. In most cases, hosts use ESP in transport mode.
Tunnel mode creates a new IP header and uses it as the outermost IP header of the datagram, followed by the ESP header and then the original datagram (both the IP header and the original payload). The ESP trailer and the optional authentication data are appended to the payload. When you use both encryption and authentication, ESP completely protects the original datagram because it is now the payload data for the new ESP packet. ESP, however, does not protect the new IP header. Gateways must use ESP in tunnel mode.
ESP uses a symmetric key that both communicating parties use to encrypt and decrypt the data they exchange. The sender and the receiver must agree on the key before secure communication takes place between them. VPN uses Data Encryption Standard (DES), triple-DES (3DES), RC5, RC4, or Advanced Encryption Standard (AES) for encryption.
If you choose the AES algorithm for encryption then you might want to enable Extended Sequence Number (ESN). ESN allows you to transmit large volumes of data at a high speed. The VPN connection uses a 64-bit sequence numbers instead of 32-bit numbers over IPSec. Using 64-bit sequence numbers allows more time before re-keying, which prevents sequence number exhaustion and minimizes the use of system resources.
The Internet Engineering Task Force (IETF) formally defines DES in Request for Comment (RFC) 1829, The ESP DES-CBC Transform. The Internet Engineering Task Force (IETF) formally defines 3DES in RFC 1851, The ESP Triple DES Transform. You can view these and other RFCs on the Internet at the following Web address: http://www.rfc-editor.org.
ESP uses HMAC-MD5 and HMAC-SHA algorithms to provide authentication functions. Both MD5 and SHA take variable-length input data and a secret key to produce fixed-length output data (called a hash value). If the hashes of two messages match, then it is very likely that the messages are the same. Both MD5 and SHA encode the message length in their output, but SHA is regarded as more secure because it produces larger hashes.
The Internet Engineering Task Force (IETF) formally defines HMAC-MD5 in Request for Comments (RFC) 2085, HMAC-MD5 IP Authentication with Replay Prevention. The Internet Engineering Task Force (IETF) formally defines HMAC-SHA in Request for Comments (RFC) 2404, The Use of HMAC-SHA-1-96 within ESP and AH. You can view these and other RFCs on the Internet at the following Web address: http://www.rfc-editor.org.
Related concepts
Authentication Header