We have already discussed the concept of IPSec, in this article we will consider IPSec in more detail.

So, the name IPSec comes from IP Security.
IPSec is a set of protocols and algorithms that are used to protect IP packets at the Layer3 level.

IPSec allows you to guarantee:
- Confidentiality - using encryption
- Data integrity - via Hashing and HMAC\
- Authentication - through the use of Digital Signatures or Pre-shared key (PSK).

We list the main IPsec protocols:
■ ESP and AH: The two main protocols used in IPsec.
Encapsulating Security Payload (ESP), can do everything that is required for IPsec, and
Authentication Header (AH), can do everything except encryption, encryption of the data, - therefore, ESP is most often used.
■ Encryption algorithms for confidentiality: DES, 3DES, AES.
■Hashing algorithms for integrity: MD5, SHA.
■ Authentication algorithms: Pre-shared keys (PSK), RSA digital signatures.
■ key management: An example would be Diffie-Hellman (DH), which can be used to
dynamically generate symmetrical keys to be used by symmetrical algorithms; PKI,
which supports the function of digital certificates issued by trusted CAs; and Internet
Key Exchange (IKE), which does a lot of the negotiating and management for us for
IPsec to operate.

Why IPSec is needed

Consider the following simple topology for connecting two offices.

We need to connect the two offices and fulfill the following goals:

• Confidentiality- provided through data encryption.
• data integrity- provided through hashing, or through Hashed Message Authentication Code (HMAC), - methods to ensure that the data has not been changed.
• Authentication- provided using pre-shared keys (PSK), or digital signatures. And when using HMAC, authentication happens all the time.
• anti-replay protection- all VPN packets are numbered, which is a protection against their repetition.

IPSec protocols and ports

IKEv1 Phase 1	UDP port 500	IKEv1 Phase 1 uses UDP:500 for its negotiation.
NAT-T (NAT traversal)	UDP port 4500	NAT Traversal is used by devices to traverse NAT. If both devices connect to each other through NAT: they want to put a fake UDP port 4500 header on each IPsec packet (before the ESP header) to survive a NAT device that otherwise may have a problem tracking an ESP session (Layer 4 protocol 50)
ESP	Layer 4 Protocol 50	All IPSec packets are Layer 4 protocol of ESP (IP Protocol #50), all data is encapsulated in it. Usually it is ESP (and not AH) that is used. In the case of using NAT-T, the ESP header is closed by the second UDP header.
AH	Layer 4 protocol 51	AH packets are Layer 4 protocol of AH (IP Protocol #51). AH does not support payload encryption and is therefore rarely used.

IPSec operation

To raise a secure VPN connection, IPSec uses the protocol Internet Key Exchange (IKE).
IKE is a framework provided Internet Security Association, as well as Key Management Protocol (ISAKMP)

So in our configuration, both routers will act as VPN gateway or IPsec peers.

Suppose a user on the 10.0.0.0 network sends a packet to the 172.16.0.0 network.
Since the tunnel has not yet been created, R1 will initiate negotiations with the second router, R2.

Step 1: Negotiate the IKEv1 Phase 1 Tunnel

The first step between routers rises Internet Key Exchange (IKE) Phase 1 tunnel.
Such a tunnel is not intended to transfer user data, but is used for official purposes, to protect management traffic.

Elevating the IKE Phase 1 tunnel can be done in two modes:
-main mode
- aggressive mode
Main mode requires the exchange of a large number of packets, but is also considered more secure.

To raise an IKE Phase 1 tunnel, the following elements must be negotiated:

• Hash algorithm: It could be message digest 5 algorithm (MD5) or Secure Hash
  Algorithm (SHA).
• Encryption algorithm: Digital Encryption Standard (DES)(weak, not recommended), Triple DES (3DES)(slightly better) or Advanced Encryption Standard (AES)(recommended) AES can use keys of different lengths: the longer the more secure.
• Diffie-Hellman (DH) group to use: The DH “group” refers to the modulus size (length of
  the key) to use for the DH key exchange. Group 1 uses 768 bits, group 2 uses 1024, and
  group 5 uses 1536. More secure DH groups are part of the next-generation encryption
  (NGE):
  - Group 14 or 24: Provides 2048-bit DH
  - Groups 15 and 16: Support 3072-bit and 4096-bit DH
  - Group 19 or 20: Supports the 256-bit and 384-bit ECDH groups, respectively

  The task of DH is to generate keying material (symmetric keys). These keys will be used to transfer data.
  DH itself is asymmetrical , but it generates symmetrical keys.

• Authentication method: can be in the form pre-shared key (PSK) or RSA signatures
• life time: IKE Phase 1 tunnel lifetime. The only parameter that may not match. The shorter the Lifetime, the more often the keys will be changed, and the safer it is.

Step 2: Run the DH Key Exchange

Once the routers have agreed on an IKE Phase 1 policy, they can start the DH key exchange process. DH allows two devices that do not yet have a secure connection between them to securely exchange symmetric keys to be used by symmetric algorithms such as AES.

Step 3: Authenticate the Peer

The last thing that will be done in IKE Phase 1 is mutual host authentication, which can be done in two ways (PSK or RSA digital signatures)
If the authentication is successful, the IKE Phase 1 tunnel is considered up. The tunnel is bidirectional.

Step 4: IKE Phase 2

After the IKE Phase 1 tunnel has risen, routers begin to raise the IKE Phase 1 tunnel.
As already mentioned, the IKE Phase 1 tunnel is purely a service, management tunnel and all negotiation traffic passes through it to raise the IKE Phase 2 tunnel.
IKE Phase 2 tunnel also uses hashing and encryption algorithms.
Elevating the IKE Phase 2 tunnel can be done in one of the following modes:
- quick mode

The IKE Phase 2 tunnel actually consists of two unidirectional tunnels, i.e. we can say that they are created:
One IKE Phase 1 tunnel, which is bidirectional, used for service functions.
And two IKE Phase 2 tunnels, which are unidirectional, and which are used to encrypt useful traffic.
All these tunnels are also called as security agreements between the two VPN peers or security associations (SA).
Each SA has its own unique number.

Now, after the IKE Phase 2 tunnel has been raised, all packets coming out of the external interfaces will be encrypted.

Setting example

Consider an example of configuring IPsec using this scheme as an example.

1. Configure Interesting Traffic
   First, we must define the traffic that we will encrypt.
   Router R1

   ip access-list extended VPN-ACL permit ip 192.168.1.0 0.0.0.255 192.168.2.0 0.0.0.255

   Router R2

   ip access-list extended VPN-ACL permit ip 192.168.2.0 0.0.0.255 192.168.1.0 0.0.0.255

2. Configure Phase 1 (ISAKMP)
   Phase 1 brings up a tunnel used for service purposes: exchange of shared secret keys, authenticate, negotiate IKE security policies, etc.
   Multiple isakmp policies can be created with different priorities.

   Router R1

   crypto isakmp key secretkey address 200.200.200.1

   Router R2

   crypto isakmp policy 1 encryption 3des hash md5 authentication pre-share group 2

   crypto isakmp key secretkey address 100.100.100.1

   Here key is the PSK(Preshared Key) used by routers for IKE Phase 1 authentication.

3. Configure Phase 2 (IPSEC)
   The purpose of the IKE Phase 2 Tunnel is to transfer useful traffic between the hosts of two offices.
   The parameters of the Phase 2 Tunnel are grouped into sets called transform sets.
   Router R1

   crypto ipsec transform-set TRSET esp-3des esp-md5-hmac ! crypto map VPNMAP 10 ipsec-isakmp set peer 200.200.200.1 set transform-set TRSET match address VPN-ACL ! interface FastEthernet0/0 crypto map VPNMAP

   Router R2

   crypto ipsec transform-set TRSET esp-3des esp-md5-hmac ! crypto map VPNMAP 10 ipsec-isakmp set peer 100.100.100.1 set transform-set TRSET match address VPN-ACL ! interface FastEthernet0/0 crypto map VPNMAP

   Both hosts used crypto ipsec transform-set TRSET esp-3des esp-md5-hmac.
   This means that 3des will be used for encryption and md5-hmac for authentication.

   crypto map is applied to the interface. The crypto map tracks traffic that meets the given conditions. Our crypto map will work with a router with the address 100.100.100.1, set by ACL internal traffic and will apply transform-set TRSET to this traffic.

IPSec Check

In general, the list of useful commands is as follows:
show crypto isakmp policy
show crypto map
show crypto isakmp sa detail
show crypto ipsec sa
show crypto engine connections active

In practice, the following is most useful:

IPSec relies on a number of technological solutions and encryption methods, but the operation of IPSec can be summarized in the following main steps:

Step 1. Starting the IPSec Process. Traffic that needs to be encrypted according to the IPSec security policy negotiated by the IPSec parties starts the IKE process.

Step 2 First phase of IKE. The IKE process authenticates the IPSec parties and negotiates the IKE security association parameters, which creates a secure channel for negotiating the IPSec security association parameters during the second phase of IKE.

Step 3 Second phase of IKE. The IKE process negotiates the IPSec security association parameters and establishes the appropriate IPSec security associations for the communicating party devices.

Step 4 Data transfer. Communication takes place between IPSec communicating parties, which is based on IPSec parameters and keys stored in the security association database.

Step 5 Terminating an IPSec Tunnel. IPSec security associations are terminated either as a result of their deletion or because they have exceeded their lifetime limit.

IPsec operating modes

There are two modes of operation for IPSec: transport and tunnel.

In transport mode, only the informative part of the IP packet is encrypted. Routing is not affected because the header of the IP packet is not changed. Transport mode is typically used to establish a connection between hosts.

In tunnel mode, the entire IP packet is encrypted. In order for it to be transmitted over the network, it is placed in another IP packet. Thus, a secure IP tunnel is obtained. Tunnel mode can be used to connect remote computers to a virtual private network or to securely transfer data over open channels links (Internet) between gateways to connect different parts of the virtual private network.

IPSec transform negotiation

During the operation of the IKE protocol, IPSec transformations (IPSec security algorithms) are negotiated. The IPSec transforms and their associated encryption algorithms are as follows:

AH protocol (Authentication Header - authentication header). A security protocol that provides authentication and (optionally) a replay detection service. The AH protocol acts as a digital signature and ensures that the data in the IP packet is not tampered with. The AH protocol does not provide a data encryption and decryption service. This protocol can be used either alone or in conjunction with the ESP protocol.

ESP (Encapsulating Security Payload) protocol. A security protocol that provides privacy and data protection, and optionally an authentication and replay detection service. Cisco IPSec-enabled products use ESP to encrypt the payload of IP packets. The ESP protocol can be used alone or in conjunction with AH.

DES standard (Data Encription Standard - data encryption standard). Packet data encryption and decryption algorithm. The DES algorithm is used in both IPSec and IKE. The DES algorithm uses a 56-bit key, which means not only higher consumption of computing resources, but also stronger encryption. The DES algorithm is a symmetric encryption algorithm that requires identical secret encryption keys in the devices of each of the IPSec communicating parties. The Diffie-Hellman algorithm is used to generate symmetric keys. IKE and IPSec use the DES algorithm to encrypt messages.

"Triple" DES (3DES). A variant of DES based on the use of three iterations of standard DES with three different keys, effectively tripling the strength of DES. The 3DES algorithm is used within IPSec to encrypt and decrypt a data stream. This algorithm uses a 168-bit key, which guarantees high encryption strength. IKE and IPSec use the 3DES algorithm to encrypt messages.

AES(advanced encryption standard)). The AES protocol uses the Rine Dale4 encryption algorithm, which provides much stronger encryption. Many cryptographers believe that AES cannot be hacked at all. AES is now the federal information processing standard. It is defined as an encryption algorithm for use by US government organizations to protect sensitive but non-classified information. The problem with AES is that it requires a lot of processing power compared to similar protocols.

IPSec conversion also uses two standard hashing algorithms to provide data authentication.

Algorithm MD5 (Message Digest 5). The hashing algorithm used to authenticate data packets. Cisco products use an MD5-calculated HMAC (Hashed Message Authentication Code) variant of the message authentication code that is provided additional protection using hashing. Hashing is a one-way (i.e., irreversible) encryption process that produces a fixed-length output for an input message of arbitrary length. IKE, AH and ESP use MD5 for data authentication.

Algorithm SHA-1 (Secure Hash Algorithm-1 -- secure hashing algorithm 1). The hashing algorithm used to authenticate data packets. Cisco products use a variant of the HMAC code that is computed using SHA-1. IKE, AH and ESP use SHA-1 for data authentication.

Within the IKE protocol, symmetric keys are created using the Diffie-Hellman algorithm, which uses DES, 3DES, MD5, and SHA. The Diffie-Hellman protocol is a cryptographic protocol based on the use of public keys. It allows two parties to agree on a shared secret key without having a sufficiently reliable communication channel. Shared secrets are required for the DES and HMAC algorithms. The Diffie-Hellman algorithm is used within IKE to generate session keys. Diffie-Hellman (DH) groups - define the "strength" of the encryption key that is used in the key exchange procedure. The higher the group number, the "stronger" and more secure the key. However, one should take into account the fact that with an increase in the DH group number, the "strength" and the security level of the key increase, but at the same time the load on the central processor increases, since more time and resources are needed to generate a "stronger" key.

WatchGuard devices support DH groups 1, 2 and 5:

DH group 1: 768-bit key

DH group 2: 1024-bit key

DH group 5: 1536-bit key

Both devices that communicate via VPN must use the same DH group. The DH group to be used by the devices is selected during the IPSec Phase 1 procedure.

IPSec protocols Organization of a secure channel https://www.site/lan/protokoly-ipsec https://www.site/@@site-logo/logo.png

IPSec protocols

Organization of a secure channel

IPSec protocols

Organization of a secure channel using AH, ESP and IKE.

Internet Protocol Security (IPSec) is referred to in the Internet standards as a system. Indeed, IPSec is a coherent set of open standards that today has a well-defined core, and at the same time, it can be quite simply supplemented with new protocols, algorithms and functions.

The main purpose of IPSec protocols is to provide secure data transmission over IP networks. The use of IPSec guarantees:

• integrity, i.e. that the data was not corrupted, lost or duplicated during transmission;
• authenticity, i.e. that the data was transmitted by the sender who proved that he is who he claims to be;
• confidentiality, i.e. that the data is transmitted in a form that prevents unauthorized viewing.

(Note that, in accordance with the classical definition, the concept of data security includes one more requirement - the availability of data, which in the considered context can be interpreted as a guarantee of their delivery. IPSec protocols do not solve this problem, leaving it to the TCP transport layer protocol.)

SECURE CHANNELS AT DIFFERENT LEVELS

IPSec is just one of many, albeit the most popular today, technology for secure data transmission over a public (unsecured) network. For technologies of this purpose, a generalized name is used - a secure channel (secure channel). The term "channel" emphasizes the fact that data protection is provided between two network nodes (hosts or gateways) along some virtual path laid in a packet-switched network.

A secure channel can be built using system tools implemented at different levels of the OSI model (see Figure 1). If a protocol of one of the upper layers (application, presentation or session) is used to protect data, then this method of protection does not depend on which networks (IP or IPX, Ethernet or ATM) are used to transport data, which can be considered an undoubted advantage. On the other hand, the application then becomes dependent on a specific security protocol, i.e. for applications such a protocol is not transparent.

A secure channel at the highest, application level, has another disadvantage - a limited scope. The protocol protects only a well-defined network service - file, hypertext or mail. For example, the S/MIME protocol only protects messages Email. Therefore, for each service, it is necessary to develop an appropriate secure version of the protocol.

The most well-known secure channel protocol operating at the next, presentational level is the Secure Socket Layer (SSL) protocol and its new open implementation Transport Layer Security (TLS). Reducing the protocol level makes it a much more versatile protection tool. Now any applications and any application layer protocols can use a single security protocol. However, applications still need to be rewritten - explicit calls to secure channel protocol functions must be built into them.

The lower in the stack the secure channel facilities are implemented, the easier it is to make them transparent to applications and application protocols. At the network and data link layers, application dependence on security protocols disappears altogether. However, here we are faced with another problem - the dependence of the security protocol on a particular network technology. Indeed, in different parts of a large composite network, generally speaking, different channel protocols are used, so it is impossible to lay a secure channel through this heterogeneous environment using a single link layer protocol.

Consider, for example, the Point-to-Point Tunneling Protocol (PPTP) secure channel running on link layer. It is based on the PPP protocol, which is widely used for point-to-point connections such as leased lines. The PPTP protocol not only provides protection transparency for applications and services of the application layer, but also does not depend on the applied network layer protocol: in particular, the PPTP protocol can carry packets both in IP networks and in networks based on IPX, DECnet protocols or NetBEUI. However, since the PPP protocol is not used in all networks (in most local networks, the Ethernet protocol works at the data link layer, and in the global networks - ATM, frame relay protocols), then PPTP cannot be considered a universal tool.

IPSec, which operates at the network layer, is a compromise. On the one hand, it is transparent to applications, and on the other hand, it can work in almost all networks, as it is based on the widely used IP protocol: currently only 1% of computers in the world do not support IP at all, the remaining 99% use it either as a single protocol, or as one of several protocols.

DISTRIBUTION OF FUNCTIONS BETWEEN IPSEC PROTOCOLS

There are three protocols at the core of IPSec: the Authenti-cation Header (AH), the Encapsulation Security Payload (ESP), and the Internet Key Exchange (IKE). Functions for maintaining a secure channel are distributed between these protocols as follows:

• the AH protocol guarantees data integrity and authenticity;
• the ESP protocol encrypts the transmitted data, guaranteeing confidentiality, but it can also support authentication and data integrity;
• the IKE protocol solves the ancillary task of automatically providing the endpoints of the channel with the secret keys necessary for the operation of the authentication and data encryption protocols.

As can be seen from the brief description of the functions, the capabilities of the AH and ESP protocols partially overlap. The AH protocol is only responsible for ensuring data integrity and authentication, while the ESP protocol is more powerful, as it can encrypt data, and in addition, perform the functions of the AH protocol (although, as we will see later, authentication and integrity are provided by it in a somewhat truncated form ). The ESP protocol can support encryption and authentication/integrity functions in any combination, i.e. either both groups of functions, or only authentication/integrity, or only encryption.

Any symmetric encryption algorithm that uses secret keys can be used to encrypt data in IPSec. Data integrity and authentication are also based on one of the encryption techniques - encryption using a one-way function, also called a hash function or digest function.

This function, applied to encrypted data, results in a digest value consisting of a fixed small number of bytes. The digest is sent in an IP packet along with the original message. The recipient, knowing which one-way encryption function was used to compose the digest, recomputes it using the original message. If the values ​​of the received and computed digests are the same, this means that the contents of the packet were not subjected to any changes during transmission. Knowing the digest does not make it possible to recover the original message and therefore cannot be used for protection, but it does allow you to check the integrity of the data.

The digest is a kind of checksum for the original message. However, there is also a significant difference. The use of a checksum is a means of verifying the integrity of messages transmitted over untrusted links and is not intended to combat malicious activity. Indeed, the presence of a checksum in the transmitted packet will not prevent an attacker from replacing the original message by adding a new checksum value to it. Unlike the checksum, a secret key is used to calculate the digest. If a one-way function was used to get the digest, with a parameter (which is the secret key) known only to the sender and receiver, any modification to the original message would be immediately detected.

The separation of security functions between the two protocols AH and ESP is caused by the practice in many countries to restrict the export and / or import of means that ensure the confidentiality of data through encryption. Each of these two protocols can be used both independently and simultaneously with the other, so that in cases where encryption is due to current restrictions cannot be used, the system can only be supplied with AH protocol. Naturally, protecting data only with the help of the AH protocol will in many cases be insufficient, since in this case the receiving party will only be sure that the data was sent by the node from which they are expected and arrived in the form in which they were received. sent. The AH protocol cannot protect against unauthorized viewing along the way of data, since it does not encrypt them. To encrypt data, it is necessary to use the ESP protocol, which can also check their integrity and authenticity.

SAFE ASSOCIATION

In order for the AH and ESP protocols to do their job of protecting transmitted data, the IKE protocol establishes a logical connection between the two endpoints, which in the IPSec standards is called the "Security Association" (Security Association, SA). The establishment of SA begins with mutual authentication of the parties, because all security measures lose their meaning if data is transmitted or received by the wrong person or from the wrong person. The SA parameters that you select next determine which of the two protocols, AH or ESP, is used to protect data, what functions the security protocol performs: for example, only authentication and integrity checks, or, in addition, also protection against false reproduction. Highly important parameter secure association is the so-called cryptographic material, i.e. the secret keys used in the operation of the AH and ESP protocols.

The IPSec system also allows a manual method of establishing a secure association, in which the administrator configures each end node so that they maintain the negotiated association parameters, including secret keys.

The AH or ESP protocol already functions within the established SA logical connection, with its help, the required protection of transmitted data is carried out using the selected parameters.

The secure association parameters must suit both endpoints of the secure channel. Therefore, when using the automatic SA establishment procedure, IKE protocols operating on opposite sides of the channel choose parameters during the negotiation process, just as two modems determine the maximum exchange rate acceptable to both sides. For each task solved by the AH and ESP protocols, several authentication and encryption schemes are offered - this makes IPSec a very flexible tool. (Note that the choice of the digest function for solving the authentication problem does not affect the choice of algorithm for data encryption.)

To ensure compatibility, the standard version of IPsec defines a certain mandatory "tool" set: in particular, one of the MD5 or SHA-1 one-way encryption functions can always be used to authenticate data, and DES is certainly among the encryption algorithms. At the same time, manufacturers of products that include IPSec are free to expand the protocol with other authentication and encryption algorithms, which they do with success. For example, many IPSec implementations support the popular Triple DES encryption algorithm, as well as relatively new algorithms - Blowfish, Cast, CDMF, Idea, RC5.

IPSec standards allow gateways to use either one SA to transmit traffic from all hosts communicating over the Internet, or to create an arbitrary number of SAs for this purpose, for example, one for each TCP connection. An SA is a unidirectional (simplex) logical connection in IPSec, so two SAs must be established for two-way communications.

TRANSPORT AND TUNNEL MODES

The AH and ESP protocols can protect data in two modes: transport and tunnel. In transport mode, the transmission of an IP packet through the network is performed using the original header of this packet, while in tunnel mode, the original packet is placed in a new IP packet and data transmission over the network is performed based on the header of the new IP packet. The use of one or another mode depends on the requirements for data protection, as well as on the role played in the network by the node that terminates the secure channel. Thus, a node can be a host (end node) or a gateway (intermediate node). Accordingly, there are three IPSec implementation patterns: host-to-host, gateway-to-gateway, and host-to-gateway.

In the first scheme, a secure channel, or, in this context, the same thing, a secure association, is established between two network end nodes (see Figure 2). The IPSec protocol in this case runs on the end node and protects the data coming to it. For the host-to-host scheme, transport mode is most commonly used, although tunnel mode is also allowed.

In accordance with the second scheme, a secure channel is established between two intermediate nodes, the so-called security gateways (Security Gateway, SG), each of which runs the IPSec protocol. Secure communication can occur between any two end nodes connected to networks that are located behind Security Gateways. End nodes are not required to support IPSec and pass their traffic unsecured over trusted enterprise intranets. Traffic destined for the public network passes through the Security Gateway, which provides protection using IPSec, acting on its own behalf. Gateways can only use tunnel mode.

The "host-gateway" scheme is often used for remote access. Here, a secure channel is established between a remote host running IPSec and a gateway that protects traffic for all hosts that are part of the enterprise Intranet. The remote host can use both transport and tunnel mode when sending packets to the gateway, while the gateway sends a packet to the host only in tunnel mode. This scheme can be complicated by creating in parallel one more secure channel - between the remote host and any host belonging to the internal network protected by the gateway. This combined use of two SAs allows you to reliably protect traffic in the internal network as well.

Natalya Olifer

Document Operations

Viewed: 8033

0 Let's take a look at the details of the technologies that make up IPSec. The standards used within IPSec are quite complex to understand, so in this section we will look at each of the components of IPSec in detail. To understand what IPSEC is, refer to the "IPSEC as a Network Traffic Security Protocol" document published earlier on this site. This article is a continuation of the above document.

IPSec uses the following technologies:

• AN protocol;
• ESP protocol;
• DES encryption standard;
• 3DES encryption standard;
• IKE protocol;
• Diffie-Hellman key agreement method;
• hashed message authentication codes (HMACs);
• RSA protection;
• certification authorities.

AN protocol

This protocol provides authentication and data integrity for IP packets sent between two systems. AN protocol is not
provides confidentiality (i.e. encryption) of packets. Authentication is accomplished by applying a one-way, key-dependent hash function to the packet, generating a "profile" of the message. A change in any part of the packet along the transmission path will be detected by the recipient by applying a similar one-way hash function to the received data and comparing the calculated message profile value with the one specified by the sender. The authenticity of the received information is guaranteed by the fact that both systems use the same secret key for one-way hashing. The scheme of operation of the AN protocol is shown below. This performs the following steps.

1. The IP header and payload of the packet is hashed.
2. The resulting hash code is used to construct a new AH header that is attached to the original packet between the header and the payload block.
3. The new packet is sent to the second IPSec side.
4. The receiving side computes a hash code value for the IP header and payload, extracts the transmitted hash code value from the AH header, and compares the two values. The corresponding hash code values ​​must match exactly. If even one bit of the packet changes in the path, the hash code of the packet computed by the receiver will not match the value specified in the AH header.

The AH protocol provides authentication for as many IP header fields as possible, as well as for the data fields of higher layer protocols. However, some IP header fields may change along the way. The values ​​of variable fields (for example, the TTL field indicating the packet's lifetime) are changed by intermediate network devices through which the packet passes, and the sender cannot predict such changes. Variable field values ​​should not be protected by the AH protocol. Thus, the protection that AH provides to the IP header is somewhat limited. The AH protocol can also optionally provide replay protection by specifying the sequence number of the packet in the IP header. A complete description of the AH protocol is contained in RFC 2402.

ESP Protocol

ESP is a security protocol that provides confidentiality (i.e., encryption), source authentication, and data integrity, as well as (optionally) a replay protection service and limited traffic privacy by countering attempts to parse the data stream.

The ESP protocol provides privacy through encryption at the IP packet level. At the same time, many algorithms of the symmetric encryption scheme are supported. The default algorithm for IPSec is DES with a 56-bit key. This cipher must be present to ensure interoperability between all IPSec-enabled products. Cisco products also support the 3DES algorithm, which provides stronger encryption. Privacy can be selected independently from other services.

Data origin authentication and connectionless integrity support are used together and are optional (i.e., not required). These capabilities can also be combined with a privacy service.
The playback protection service can only be selected if data source authentication is selected and the selection of this service is the sole responsibility of the recipient. Although by default the sender is required to automatically increment the sequence number used for replay protection, this service is only effective if the recipient verifies the sequence number. Traffic privacy requires the choice of tunnel mode. This is most effective in a security gateway, where source-destination masquerading can be performed on all traffic at once. It should be noted here that although both privacy and authentication are options, at least one of these services must be selected.
The set of services provided by the ESP protocol depends on the parameters that are specified in the IPSec configuration and selected when creating the IPSec security association. However, choosing confidentiality without integrity/authentication (either within ESP or separately via AH) leaves an adversary open to certain types of attacks, which can limit the usefulness of the privacy service so employed.
The ESP header is inserted into the packet after the IP header, before the upper layer protocol header (in transport mode) or before the encapsulated IP header (in tunnel mode). A complete description of the ESP protocol is contained in RFC 2406.

ESP encryption using HMAC

Within the ESP protocol, packets can also be authenticated using an optional authentication field. In Cisco IOS Software and PIX Firewalls, this service is called HMAC ESP. Authentication values ​​are calculated after encryption is done. The IPSec standard in use today describes the SHA1 and MD5 algorithms as mandatory for HMAS.
The main difference between ESP authentication and AH authentication is their scope. ESP does not protect any IP header fields unless ESP (tunnel mode) encapsulation is intended. Figure shows which fields are protected when using HMAC ESP.

Note that the encryption only covers the payload data, while the ESP with ESP HMAC hashing covers the ESP header and payload data. The IP header is not protected. The HMAC ESP service cannot be used on its own, but must be combined with the ESP encryption protocol.

IPSec tunnel and transport modes

IPSec operates in either tunnel mode or transport mode. The figure shows a diagram of the implementation of the tunnel mode. In this mode, the entire original IP datagram is encrypted and becomes payload in a new IP packet with a new IP header and an additional IPSec header (abbreviated as HDR in the figure). Tunnel mode allows a network device (such as the PIX Firewall) to act as an IPSec gateway or proxy that performs encryption for hosts behind the firewall. The source router encrypts the packet and sends it over the IPSec tunnel. The destination PIX Firewall decrypts the received IPSec packet, extracts the original IP datagram, and passes it on to the destination system. The main advantage of tunnel mode is that end systems do not need to be modified to enable them to use IPSec. Tunnel mode also prevents an adversary from analyzing the data stream. In a tunnel mode exchange, an adversary has the ability to determine only the endpoints of the tunnel, but not the true source and destination of packets passing through the tunnel, even if the tunnel endpoints are on the source and destination systems.

The diagram in Figure below illustrates the transport mode. Here, only the IP payload is encrypted, and the original IP header is left intact.
An IPSec header is added. The advantage of this mode is that only a few bytes are added to each packet. In addition, devices open network can see the true sender and receiver addresses of the packet.

This allows the use of special features of intermediate networks (for example, guaranteed quality of service) based on information in the IP header. However, the layer 4 header is encrypted, which limits the ability to parse the packet. Unfortunately, the transmission of the IP header in the clear in the transport mode allows the attacker to perform some analysis of the data flow. For example, an attacker can find out how many packets were transmitted by IPSec parties operating in transport mode. But the intruder can only learn that the IP packets were being forwarded. It will not be able to determine if they were an email message or some other application if the ESP protocol was used.

Using tunnel and transport modes

Let's look at a few examples illustrating the rules for choosing a tunnel or transport mode. The figure below shows situations in which tunnel mode is used. This mode is most commonly used to encrypt traffic between IPSec security gateways, such as between a Cisco router and a PIX Firewall. IPSec gateways perform IPSec functions for devices behind such gateways (in the figure shown, this Personal Computer Alice and HR servers). In this example, Alice gains secure access to the HR servers through an IPSec tunnel established between the gateways.

Tunnel mode is also used for communication of end stations in which software IPSec, such as for linking the CiscoSecure VPN client and IPSec gateway.
In this example, tunnel mode is used to create an IPSec tunnel between the Cisco router and the server running the IPSec software. Note that in Cisco IOS Software and the PIX Firewall, tunnel mode for IPSec communications is the default mode.
Transport mode is used between IPSec-enabled end stations, or between an end station and a gateway if the gateway is interpreted as a host. On fig. Example D is shown below, illustrating the use of transport mode to create an encrypted IPSec tunnel from Alice's computer, which is running the client software. Microsoft Windows 2000 to a Cisco VPN 3000 concentrator, allowing Alice to use an L2TP tunnel over IPSec.

Using AH and ESP

In certain situations, the problem of choosing between AH and ESP may seem difficult to resolve, but it can be simplified by following a few rules. If you need to know that data from an authenticated source is transmitted without compromising integrity and does not need to be kept confidential, use the AH protocol, which protects upper layer protocols and IP header fields that are not changed in transit. Security means that the corresponding values ​​cannot be changed because this will be detected by the IPSec second party and any modified IP datagram will be discarded. The AH protocol does not provide protection against channel eavesdropping and header and data viewing by an intruder. But since the header and data cannot be silently changed, the changed packets are rejected.

If you need to keep data private (ensure confidentiality), use ESP. This protocol encrypts the upper layer protocols in transport mode and the entire original IP datagram in tunnel mode, so it is not possible to extract information about the packets by sniffing the transmission channel. The ESP protocol can also provide an authentication service for packets. However, when using ESP in transport mode, the outer original IP header is not protected, and in tunnel mode, the new IP header is not protected. When using IPSec, users are more likely to use tunnel mode than transport mode.

IPsec is not a single protocol, but a system of protocols designed to protect data at the network layer of IP networks. This article will describe the theory of using IPsec to create a VPN tunnel.

Introduction

VPN based on IPsec technology can be divided into two parts:

• Internet Key Exchange Protocol (IKE)
• IPsec protocols (AH/ESP/both)

The first part (IKE) is the negotiation phase, during which the two VPN points decide which methods will be used to secure the IP traffic sent between them. In addition, IKE is also used to manage connections, for this the concept of Security Associations (SA) is introduced for each connection. SAs only point in one direction, so a typical IPsec connection uses two SAs.

The second part is the IP data that needs to be encrypted and authenticated before transmission using the methods agreed in the first part (IKE). There are different IPsec protocols that can be used: AH, ESP, or both.

The sequence for establishing a VPN over IPsec can be briefly described as:

• IKE negotiates IKE layer security
• IKE negotiates IPsec layer security
• protected data is transmitted via VPN IPsec

IKE, Internet Key Exchange

To encrypt and authenticate data, you need to select the encryption / authentication method (algorithm) and the keys used in them. The task of the Internet Key Exchange protocol, IKE, in this case is to distribute these “session keys” and agree on algorithms that will protect data between VPN points.

The main tasks of IKE:

• Authenticate each other's VPN points
• Organization of new IPsec connections (through creation of SA pairs)
• Managing current connections

IKE keeps track of connections by assigning each of them a certain Security Associations, SA. SA describes the parameters of a particular connection, including the IPsec protocol (AH/ESP or both), session keys used to encrypt/decrypt and/or authenticate data. SA is unidirectional, so there are multiple SAs per connection. In most cases where only ESP or AH is used, only two SAs are created for each connection, one for inbound traffic and one for outbound traffic. When ESP and AH are used together, SA requires four.

The IKE negotiation process goes through several stages (phases). These phases include:

1. IKE first phase (IKE Phase-1):
   - The protection of the IKE itself (ISAKMP tunnel) is negotiated
2. IKE second phase (IKE Phase-2):
   - Negotiate IPsec security
   — Receiving data from the first phase to generate session keys

IKE and IPsec connections are limited in duration (in seconds) and in the amount of data transferred (in kilobytes). This is done to improve security.
The duration of an IPsec connection is typically shorter than IKE. Therefore, when an IPsec connection expires, a new IPsec connection is recreated through the second phase of the negotiation. The first phase of the negotiation is used only when recreating the IKE connection.

To negotiate IKE, the concept of an IKE proposal (IKE Proposal) is introduced - this is a proposal on how to protect data. The VPN point that initiates the IPsec connection sends a list (proposal) that specifies different connection security methods.
Negotiations can be conducted both about establishing a new IPsec connection, and about establishing a new IKE connection. In the case of IPsec, the protected data is the traffic sent through the VPN tunnel, while in the case of IKE, the protected data is the data of the IKE negotiations themselves.
The VPN point that has received the list (offer) selects the most suitable one from it and indicates it in the response. If none of the offers can be selected, the VPN gateway refuses.
The proposal contains all the necessary information for choosing an encryption and authentication algorithm, etc.

Phase 1 IKE - IKE Security Negotiation (ISAKMP Tunnel)
In the first phase of negotiation, VPN points authenticate each other based on a common key (Pre-Shared Key). Hash algorithm is used for authentication: MD5, SHA-1, SHA-2.
However, before authenticating each other, so as not to transmit information in clear text, VPN points exchange lists of proposals (Proposals), described earlier. Only after the offer that suits both VPN points is selected, does each other's VPN point authenticate.
Authentication can be done different ways: via pre-shared keys, certificates or . Shared keys are the most common authentication method.
IKE negotiation of the first phase can occur in one of two modes: main (main) and aggressive (aggressive). The main mode is longer, but also more secure. In its process, six messages are exchanged. Aggressive mode is faster, limited to three messages.
The main work of the first phase of IKE lies in the Diffie-Hellman key exchange. It is based on public key encryption, each side encrypts the authentication parameter (Pre-Shared Key) with the neighbor's public key, which received given message decrypts it with its private key. Another way of authenticating parties to each other is by using certificates.

Phase 2 IKE - IPsec Security Negotiation
In the second phase, the choice of how to protect the IPsec connection is made.
The second phase uses the keying material extracted from the Diffie-Hellman key exchange that occurred in the first phase. Based on this material, session keys are created, which are used to protect data in the VPN tunnel.

If the mechanism is used Perfect Forwarding Secrecy (PFS), then a new Diffie-Hellman key exchange will be used for each phase 2 handshake. Slightly reducing the speed of work, this procedure ensures that the session keys are independent of each other, which increases protection, since even if one of the keys is compromised, it cannot be used to select the others.

There is only one mode of operation of the second phase of the IKE negotiation, it is called quick mode - fast mode. In the process of negotiating the second phase, three messages are exchanged.

At the end of the second phase, a VPN connection is established.

IKE settings.
During connection establishment, several parameters are used, without negotiation of which it is impossible to establish a VPN connection.

• End node identification
  How nodes authenticate each other. The most commonly used is the shared key. Shared key authentication uses the Diffie-Hellman algorithm.
• Local and remote network/host
  Specifies the traffic that will be allowed through the VPN tunnel.
• Tunnel or transport mode.
  IPsec can operate in two modes: tunnel and transport. The choice of the mode depends on the protected objects.
  tunnel mode used for protection between remote objects, i.e. The IP packet is completely encapsulated in a new one, and only the connection between two VPN points will be visible to an observer from the outside. The real IP addresses of the source and destination will be visible only after the packet is decapsulated when it is received at the VPN receiving point. Therefore, tunnel mode is most commonly used for VPN connections.
  Transport mode protects the data of the IP packet (TCP, UDP and upper layer protocols), and the header of the original IP packet will be preserved. Thus, the original source and destination will be visible to the observer, but not the transmitted data. This mode is most often used when securing a connection in local network between hosts.
• Remote Gateway
  A secure connection destination VPN point that will decrypt/authenticate the data from the other side and send it to the final destination.
• IKE mode of operation
  IKE negotiation can operate in two modes: basic and aggressive.
  The difference between the two is that aggressive mode uses fewer packets to achieve faster connection establishment. On the other hand, the aggressive mode does not pass some negotiation parameters, such as Diffie-Hellman groups and PFS, which requires their preliminary identical configuration on the connection points.
• IPsec protocols
  There are two IPsec protocols, Authentication Header (AH) and Encapsulating Security Payload (ESP), which perform encryption and authentication functions.
  ESP allows you to encrypt, authenticate individually or simultaneously.
  AH only allows authentication. The difference with ESP authentication is that AH authenticates the outer IP header as well, allowing you to verify that the packet actually came from the source specified in it.
• IKE encryption
  Specifies the IKE encryption algorithm to use and its keys. Various symmetric encryption algorithms are supported, for example: DES, 3DES, AES.
• IKE authentication
  The authentication algorithm used in the IKE handshake. Can be: SHA, MD5.
• IKE Diffie-Hellman (DH) groups
  Group used by the DF for key exchange in IKE. The larger the group, the larger the size of the exchange keys.
• IKE Connection Lifespan
  It is specified both by time (seconds) and by the size of the transferred data (kilobytes). As soon as one of the counters reaches the threshold, a new first phase starts. If no data has been transmitted since the creation of the IKE connection, no new connections will be created until one of the parties wants to create a VPN connection.
• PFS
  With PFS disabled, key generation material will be retrieved in the first phase of the IKE negotiation at the time of the key exchange. In the second phase of the IKE negotiation, session keys will be created based on the received material. With PFS enabled, when creating new session keys, the material for them will be used each time a new one. Thus, if the key is compromising, it is not possible to create new keys based on it.
  PFS can be used in two modes: first PFS on keys, will start a new key exchange in the first phase of IKE every time a negotiation starts
  second phase. The second mode, PFS on identities, will delete the SA of the first phase every time the second phase negotiation passes, thereby ensuring that no second phase negotiation is encrypted with an identical previous key.
• IPsec DH groups
  These DF groups are similar to those used in IKE, only used for PFS.
• IPsec encryption
  Algorithm used to encrypt data. Used when using ESP in encryption mode. Example algorithms: DES, 3DES, AES.
• IPsec authentication
  Algorithm used to authenticate transmitted data. Used in case of AH or ESP in authentication mode. Example algorithms: SHA, MD5.
• IPsec lifetime
  The lifetime of a VPN connection is indicated both by time (seconds) and by the size of the transferred data (kilobytes). The first counter to reach the limit will start re-creating the session keys. If no data has been transmitted since the creation of the IKE connection, no new connections will be created until one of the parties wants to create a VPN connection.

IKE Authentication Methods

• Manual mode
  The simplest of the methods, in which IKE is not used, and the authentication and encryption keys, as well as some other parameters, are set manually on both points of the VPN connection.
• Via Pre-Shared Keys (PSK)
  A pre-shared key on both points of the VPN connection. The difference from the previous method is that IKE is used, which allows you to authenticate endpoints and use changing session keys, instead of fixed encryption keys.
• Certificates
  Each VPN point uses: its private key, its public key, its own certificate that includes its public key and is signed by a trusted certificate authority. Unlike the previous method, it allows you to avoid entering one common key at all VPN connection points, replacing it with personal certificates signed by a trusted authority.

IPsec protocols

IPsec protocols are used to protect transmitted data. The choice of protocol and its keys occurs during IKE negotiation.

AH (Authentication Header)

AH provides the ability to authenticate the transmitted data. To do this, a cryptographic hash function is used with respect to the data contained in the IP packet. The output of this function (hash) is transmitted along with the packet and allows the remote VPN point to confirm the integrity of the original IP packet, confirming that it has not been modified along the way. In addition to the IP packet data, the AH also authenticates part of its header.

In transport mode, the AH embeds its header after the original IP packet.
In tunnel mode, the AH embeds its header after the outer (new) IP header and before the inner (original) IP header.

ESP (Encapsulating Security Payload)

The ESP protocol is used for encryption, for authentication, or both, against an IP packet.

In transport mode, the ESP protocol inserts its header after the original IP header.
In ESP tunnel mode, the header is after the outer (new) IP header and before the inner (original).

Two main differences between ESP and AH:

• ESP, in addition to authentication, also provides the ability to encrypt (AH does not provide this)
• ESP in tunnel mode only authenticates the original IP header (AH authenticates the outer one as well).

Work behind NAT (NAT Traversal)
A separate specification has been implemented to support operation behind NAT. If the VPN point supports this specification, IPsec supports NAT operation, but there are certain requirements.
NAT support consists of two parts:

• At the IKE level, end devices communicate with each other about support, NAT Traversal, and the version of the supported specification.
• At the ESP level, the generated packet is encapsulated in UDP.

NAT Traversal is used only if both points support it.
Definition of NAT: Both VPNs send hashes of their IP addresses along with the source UDP port of the IKE negotiation. This information is used by the recipient to determine if the IP address and/or port of the source has changed. If these parameters have not been changed, then traffic does not pass through NAT and the NAT Traversal mechanism is not needed. If the address or port has been changed, then there is a NAT between the devices.

Once the endpoints determine that NAT Traversal is needed, the IKE negotiation is moved from UDP port 500 to port 4500. This is because some devices do not correctly handle an IKE session on port 500 when using NAT.
Another problem arises from the fact that the ESP protocol is a transport layer protocol and sits directly on top of IP. Because of this, the concepts of TCP / UDP port are not applicable to it, which makes it impossible for more than one client to connect through NAT to one gateway. To solve this problem, ESP is packaged into a UDP datagram and sent to port 4500, the same one used by IKE when NAT Traversal is enabled.
NAT Traversal is built into the protocols that support it and works without prior configuration.