IPSEC & its Protocols

By | May 6, 2013

IPSEC – Internet Protocol Security

  • IPSEC is a suite of protocols used to establish Secure IP communication by encrypting each packet in the Communication.
  • It works at IP Layer of TCP/IP Protocol Suite.
  • Also includes protocols to establish mutual authentication between hosts.

There are two main modes of IPSEC operation.

Transparent Mode

  • In this mode only Data/Payload is encrypted.
  • ESP/AH Header is inserted after IP Header
  • Used mostly in host to host communication.

Tunnel Mode

  • In this type of mode full IP packet is encrypted.
  • A new IP header is created.
  • Used mostly in network to network or host to network communications.
  • It is the default mode on Cisco Routers.

IPSEC Protocols


  •  Internet Security and key management protocol is is protocol and part of IPSEC protocol suite.
  • It is used for authentication a peer , management of Security Association. etc.
  • It traverses UDP port 500
  • ISAKMP is mainly used for key exchange in IKE (Internet Key Exchange Protocol)


  • Internet key exchange is a protocol which is responsible for key agreement using public key cryptography thus used to set up Security Associations in IPSEC protocol suite.
  • Uses X.509 certificates for authentication and DH groups (Diffie–Hellman key exchange ) to set up a secret session.
  • IKE has two modes: Phase 1 and Phase 2. Also there is a optional IKE phase 1.5 which is used to implement X.Auth for user authentication.
  • IKE Phase 1 is is used to set up an administrative tunnel through which IKE phase 2 can be negotiated.
  • For Phase 1 to establish successfully following parameters should match on both ends of IPSEC tunnel: authentication method, DH group, encryption algorithm, exchange mode, hash alorithm, NAT-T, DPD and lifetime (optional Parameter)
  • IKE Phase 2 is used to protect the acual IP traffic with IPSEC and only uses quick mode of IKE operation.
  • For Phase 1 to establish successfully following parameters should match on both ends of IPSEC tunnel:  IPSEC protocol, mode (tunnel or transport), authentication method, PFS (DH) group, lifetime

IKE has following three modes of operation:

Main Mode

  • It happens in three two way exchanges
  • Algo and hashes used to secure IKE communication are agreed in this mode.
  • Main Mode uses DH exchange to generate shared secret.
  • 6 messages are shared in three exchanges

Agressive Mode

  • Fever communications are done in this mode.
  • In first exchange all info is agreed upon .
  • 2nd exchange is to confirm the exchange from receiver.
  • The main disadvantage in this mode is that all info is shared before there is secure channel.
  • 3 messages are shared  in two exchanges.

Quick Mode

  • Same as aggressive, except the negotiation is secured with the IKE Security Association (SA).


  • Encapsulating Security Payload is a protocol used only for payload to provide data encryption, integrity and authentication of peer.
  • IP Protocol number for ESP is 50.


  • Authentication header is used for entire IP Packet to provide integrity and peer authentication but not the encryption for the packet.
  • IP Protocol number for ESP is 50.

IPSEC Encryption Algorithms:


  • DES Stands for Data Encryption Standard
  • Symmetric Algorithm with 56 bit key length.
  • Used to encrypt messages using private key.


  • Tripple DES or 3 Data Encryption Standard
  • Symmetric algorithm with 168 bit key length.
  • Two 56-bit keys are selected for encrytion. Data is encrypted through DES three times, the first time by the first key, the second time by the second key and the third time by the first key once more.


  • ASE stands for Advanced encryption standard
  • Symmetric algorithm with 128, 192, 256 bit key length
  • 128 block size, means 128 bit block of data is encrypted at a time.


  • RSA stands for Ron Rivest, Adi Shamir and Leonard Adleman
  • RSA is a public-key cryptosystem for both encryption and authentication
  • Asymmetric algorithm with 1024 bit key length.
  • Two large prime numbers are used and are multipied which derives two set of numbers that are used as public and private key.
  • The algorithm involves multiplying two large prime numbers and through additional operations deriving a set of two numbers that constitutes the public key and another set that is the private key

IPSEC Hashing Algorithms


  • Message Digest Algorithm.
  • Produces 128 Bit hash value.
  • Mostly used to check data integrity.


  • Secure Hash Algorithm.
  • Produces 160 bit hash value.


  • SHA-256/224 – 256/224 hash value or message digest with blocksize of 512.
  • SHA-512/384 512/384 hash value with block size of 1024

2 thoughts on “IPSEC & its Protocols

  1. Peter Arabomen

    Nice Write up, i hope to see the configuration in your next write up

    1. ciscohite

      Thanks Peter, We indeed plan to write a complete series of VPN configuration on various devices.

Comments are closed.