Configuring a Dial-up VPN Using Windows 7 Client with L2TP Over IPSec

The Windows 7 client, when configuring L2TP over IPsec, has a couple special requirements in order to establish a successful connection configure L2TP over IPsec for Windows XP. Most of the procedure applies the same for the Windows 7 client; however there are two components which are different for the Windows 7 client.

  •  generate certificates on the Windows 7 client
  • add additional custom Phase2 proposal on the firewall, when you get to the ‘Configuring Autokey IKE VPN (Phase 2)’ section.

 Generate certificates on Windows 7

follow the procedure and the screenshots below on how to generate the certificate for Windows 7.Use Microsoft Active Directory Certificate Services to manage certificates as outlined.

The web-based certificate enrolment method will *not* work with a Vista or Windows 7 client since the Xenroll ActiveX control has been deprecated. There is a procedure from Microsoft for installing a hot fix to the Certificate Web Enrolment pages.  However, we found that this did not work because it left us unable to generate certificates which would be stored in the Computer Account as required by the VPN client.

nstead, use the MMC snap-in for Certificates to generate the certificate from the client.

The steps are:

    1. Add the Certificates snap-in to MMC:

    1. Use the Request New Certificate option:

    1. Choose the “…Click here to configure settings” option:

    1. Choose the “Web Server” option and click Enroll:

    1. Add the appropriate values to the Subject name in line with the KB article:


Then continue on with creating the Dial-up Connection on page 15 of the Application Note.  follow the configuration changes below.

II. Add additional custom Phase2 proposal on the firewall

After adding the certificates, Phase1 of the VPN may establish fine with certificates, but the Phase 2 may not come up.  The following error could be reported in the 3rd message from the Windows 7 client to the firewall.

For example, in the ‘debug ike detail’ output collected during the VPN establishment, the firewall sends the second phase 2 message to the Windows 7 client, and the firewall receives a “HASH DELETE” from the client to clear the SA.  This mean there is an issue that the Windows 7 client is not liking a message sent by the firewall and sending a HASH DELETE to clear the SA.

## 2009-12-02 13:37:00 : IKE<1.1.1.1> Phase 2 Responder constructing 2nd message.
## 2009-12-02 13:37:00 : IKE<1.1.1.1> Construct ISAKMP header.
## 2009-12-02 13:37:00 : IKE<1.1.1.1> Msg header built (next payload #8)
## 2009-12-02 13:37:00 : IKE<1.1.1.1> Construct [HASH]
## 2009-12-02 13:37:00 : IKE<1.1.1.1> Construct [SA] for IPSEC
## 2009-12-02 13:37:00 : IKE<0.0.0.0        >   Set IPSEC SA attrs tunnel(2) SHA-1 grp0 lifetime(3600/250000)
## 2009-12-02 13:37:00 : IKE<0.0.0.0        >   Before NAT-T attr unmap: P2 prop tunnel = 2.
## 2009-12-02 13:37:00 : IKE<0.0.0.0        >   After NAT-T attr unmap: P2 prop tunnel = 2.
## 2009-12-02 13:37:00 : IKE<1.1.1.1> IP<1.1.1.1> mask<255.255.255.255> prot<17> port<1701>
## 2009-12-02 13:37:00 : IKE<1.1.1.1> Initiator P2 ID built:
## 2009-12-02 13:37:00 : IKE<1.1.1.1> Responder P2 ID built:
## 2009-12-02 13:37:00 : IKE<1.1.1.1> Construct [NONCE] for IPSec
## 2009-12-02 13:37:00 : IKE<1.1.1.1> Construct [ID] for Phase 2
## 2009-12-02 13:37:00 : id payload constructed. type(1),ip(403473b4),mask(ffffffff), prot(17), port(1701)
## 2009-12-02 13:37:00 : IKE<1.1.1.1> Construct [ID] for Phase 2
## 2009-12-02 13:37:00 : id payload constructed. type(1),ip(403473b2),mask(ffffffff), prot(17), port(1701)
## 2009-12-02 13:37:00 : IKE<1.1.1.1> send out RESPONDER_LIFETIME notification. prot=3,
## 2009-12-02 13:37:00 : IKE<1.1.1.1> life_kb=0
## 2009-12-02 13:37:00 : IKE<1.1.1.1> Construct [NOTIF] (RESPONDER-LIFETIME) for IPSEC
## 2009-12-02 13:37:00 : IKE<1.1.1.1> construct QM HASH
## 2009-12-02 13:37:00 : IKE<1.1.1.1  > Xmit*: [HASH] [SA] [NONCE] [ID] [ID] [NOTIF]
## 2009-12-02 13:37:00 : IKE<1.1.1.1> Encrypt P2 payload (len 192)
## 2009-12-02 13:37:00 : IKE<1.1.1.1> Responder sending IPv4 IP 1.1.1.1/port 500
## 2009-12-02 13:37:00 : IKE<1.1.1.1> Send Phase 2 packet (len=196)
## 2009-12-02 13:37:00 : IKE<1.1.1.1> oakley_process_quick_mode():exit
## 2009-12-02 13:37:00 : IKE<1.1.1.1> IKE msg done: PKI state<0> IKE state<3/1017113f>
## 2009-12-02 13:37:00 : IKE<1.1.1.1> ike packet, len 112, action 0
## 2009-12-02 13:37:00 : IKE<1.1.1.1> Catcher: received 84 bytes from socket.
## 2009-12-02 13:37:00 : IKE<1.1.1.1> ****** Recv packet if <ethernet2> of vsys <Root> ******

## 2009-12-02 13:37:00 : IKE<1.1.1.1> Catcher: get 84 bytes. src port 500
## 2009-12-02 13:37:00 : IKE<0.0.0.0        >   ISAKMP msg: len 84, nxp 8[HASH], exch 5[INFO], flag 01  E
## 2009-12-02 13:37:00 : IKE<1.1.1.1> Create conn entry…
## 2009-12-02 13:37:00 : IKE<1.1.1.1>   …done(new 6401cd21)
## 2009-12-02 13:37:00 : IKE<1.1.1.1> Decrypting payload (length 56)
## 2009-12-02 13:37:00 : IKE<1.1.1.1  > Recv*: [HASH] [DELETE]
## 2009-12-02 13:37:00 : IKE<1.1.1.1> Process [DELETE]:
## 2009-12-02 13:37:00 : IKE<1.1.1.1> DELETE payload received, deleting Phase-1 SA

## 2009-12-02 13:37:00 : IKE<1.1.1.1>   Delete conn entry…
## 2009-12-02 13:37:00 : IKE<1.1.1.1>  …found conn entry(6401cd21)
## 2009-12-02 13:37:00 : IKE<1.1.1.1> IKE msg done: PKI state<0> IKE state<3/1017113f>
## 2009-12-02 13:37:00 : IKE<0.0.0.0        >   from FLOAT port.
## 2009-12-02 13:37:01 : nhtb_list_update_status: vpn Claire_Work
## 2009-12-02 13:37:01 :   ** link ready return 8
## 2009-12-02 13:37:01 : sa_link_status_for_tunl_ifp: saidx 0, preliminary status 8
## 2009-12-02 13:37:01 : nhtb_list_update_status: vpn Alex_work
## 2009-12-02 13:37:01 : nhtb_list_update_status: vpn Claire2_work
## 2009-12-02 13:37:05 : IKE<0.0.0.0        >   IKE: phase-2 packet re-trans timer expired
## 2009-12-02 13:37:05 : IKE<1.1.1.1> phase-2 packet re-trans timer expired.
At this time, the logs on the Windows 7 client do not report the reason why the Windows 7 client is sending a “HASH DELETE” message. With permutation and combination, we can see that the main reason for this was the phase 2 IPSEC proposal mismatch between the firewall and the Windows 7.

On a Windows XP setup, configuring the firewall to use the “COMPATIBLE” phase 2 proposal (on page 32 of the Application Note) will work fine. However on the Windows 7 and Windows Vista client this will end up with an error and the phase 2 never completes.

To fix this, when you get to page 32 of the Application Note, configure a phase 2 custom proposal on the firewall and apply it to the VPN configuration:

    1. Create two new proposals as follows:

set ike p2-proposal "nopfs-esp-3des-sha-windows7" no-pfs esp 3des sha-1 second 3600 kbyte 250000
set ike p2-proposal "nopfs-esp-aes128-sha-windows7" no-pfs esp aes128 sha-1 second 3600 kbyte 250000

  1. Then change the IPsec config as follows to reference these two p2 proposals (p.33 of the Application Note):
set vpn "WindowsVPN-vpn" gateway "WindowsVPN-gateway" no-replay transport idletime 0 proposal "nopfs-esp-3des-sha-windows7"  "nopfs-esp-3des-md5"  "nopfs-esp-aes128-sha-windows7"  "nopfs-esp-des-md5"

Important note: The LIFE SIZE has to be set for a value 250000; if LIFE SIZE is not set to 250000, it will not work.   All built-in P2 proposals set on the Juniper firewall are set to a LIFE SIZE of 0.

After adding the above configuration, the L2TP over IPSEC connection connects, and the following debug info is reported after the new phase 2 proposal:

## 2009-12-02 19:51:26 : IKE<1.1.1.1> Phase 2 Responder constructing 2nd message.
## 2009-12-02 19:51:26 : IKE<1.1.1.1> Construct ISAKMP header.
## 2009-12-02 19:51:26 : IKE<1.1.1.1> Msg header built (next payload #8)
## 2009-12-02 19:51:26 : IKE<1.1.1.1> Construct [HASH]
## 2009-12-02 19:51:26 : IKE<1.1.1.1> Construct [SA] for IPSEC
## 2009-12-02 19:51:26 : IKE<0.0.0.0        >   Set IPSEC SA attrs tunnel(2) SHA-1 grp0 lifetime(3600/250000)
## 2009-12-02 19:51:26 : IKE<0.0.0.0        >   Before NAT-T attr unmap: P2 prop tunnel = 2.
## 2009-12-02 19:51:26 : IKE<0.0.0.0        >   After NAT-T attr unmap: P2 prop tunnel = 2.
## 2009-12-02 19:51:26 : IKE<1.1.1.1> IP<1.1.1. 1> mask<255.255.255.255> prot<17> port<1701>
## 2009-12-02 19:51:26 : IKE<1.1.1.1> Initiator P2 ID built:
## 2009-12-02 19:51:26 : IKE<1.1.1.1> Responder P2 ID built:
## 2009-12-02 19:51:26 : IKE<1.1.1.1> Construct [NONCE] for IPSec
## 2009-12-02 19:51:26 : IKE<1.1.1.1> Construct [ID] for Phase 2
## 2009-12-02 19:51:26 : id payload constructed. type(1),ip(403473b4),mask(ffffffff), prot(17), port(1701)
## 2009-12-02 19:51:26 : IKE<1.1.1.1> Construct [ID] for Phase 2
## 2009-12-02 19:51:26 : id payload constructed. type(1),ip(403473b2),mask(ffffffff), prot(17), port(1701)
## 2009-12-02 19:51:26 : IKE<1.1.1.1> construct QM HASH
## 2009-12-02 19:51:26 : IKE<1.1.1.1  > Xmit*: [HASH] [SA] [NONCE] [ID] [ID]
## 2009-12-02 19:51:26 : IKE<1.1.1.1> Encrypt P2 payload (len 168)
## 2009-12-02 19:51:26 : IKE<1.1.1.1> Responder sending IPv4 IP 1.1.1.1/port 500
## 2009-12-02 19:51:26 : IKE<1.1.1.1> Send Phase 2 packet (len=172)
## 2009-12-02 19:51:26 : IKE<1.1.1.1> oakley_process_quick_mode():exit
## 2009-12-02 19:51:26 : IKE<1.1.1.1> IKE msg done: PKI state<0> IKE state<3/1017113f>
## 2009-12-02 19:51:26 : IKE<1.1.1.1> ike packet, len 88, action 0
## 2009-12-02 19:51:26 : IKE<1.1.1.1> Catcher: received 60 bytes from socket.
## 2009-12-02 19:51:26 : IKE<1.1.1.1> ****** Recv packet if <ethernet2> of vsys <Root> ******
## 2009-12-02 19:51:26 : IKE<1.1.1.1> Catcher: get 60 bytes. src port 500
## 2009-12-02 19:51:26 : IKE<0.0.0.0        >   ISAKMP msg: len 60, nxp 8[HASH], exch 32[QM], flag 01  E
## 2009-12-02 19:51:26 : IKE<1.1.1.1> Decrypting payload (length 32)
## 2009-12-02 19:51:26 : IKE<1.1.1.1  > Recv*: [HASH]
## 2009-12-02 19:51:26 : IKE<0.0.0.0        >   extract payload (32):
## 2009-12-02 19:51:26 : IKE<1.1.1.1> QM in state OAK_QM_AUTH_AWAIT.
## 2009-12-02 19:51:26 : IKE<1.1.1.1> xauth_cleanup()
## 2009-12-02 19:51:26 : IKE<1.1.1.1> Done cleaning up IKE Phase 1 SA
## 2009-12-02 19:51:26 : IKE<1.1.1.1> Start by finding matching member SA (verify 82/82)
## 2009-12-02 19:51:26 : IKE<1.1.1.1> Verify sa: index 82
## 2009-12-02 19:51:26 : IKE<1.1.1.1> IKE: Matching policy: gw ip <1.1.1.1> peer entry id<4>
## 2009-12-02 19:51:26 : IKE<0.0.0.0        >   protocol mis-matched expect<0>, received lcl<17> rmt<17>.

## 2009-12-02 19:51:26 : IKE<1.1.1.1> L2TP-O-IPSec, sa ID 174
## 2009-12-02 19:51:26 : IKE<1.1.1.1> Proxy ID match: Located matching Phase 2 SA <174>.
## 2009-12-02 19:51:26 : IKE<1.1.1.1> sa ID for phase 2 sa is <174>. IP version is 4.
## 2009-12-02 19:51:26 : IKE<1.1.1.1> Single user entry.
## 2009-12-02 19:51:26 : IKE<0.0.0.0        >   life (sec or kb): lcl 3600, peer 3600, set 3600.
## 2009-12-02 19:51:26 : IKE<0.0.0.0        >   life (sec or kb): lcl 250000, peer 250000, set 250000.
## 2009-12-02 19:51:26 : IKE<1.1.1.1> gen_qm_key()
## 2009-12-02 19:51:26 : IKE<1.1.1.1> load_sa_keys(): enter.
## 2009-12-02 19:51:26 : IKE<1.1.1.1> gen_qm_key()
## 2009-12-02 19:51:26 : IKE<1.1.1.1> load_sa_keys(): enter.
## 2009-12-02 19:51:26 : IKE<1.1.1.1> ikmpd.c 3668. sa ID for phase 2 sa is <174>. IP version is 4.
## 2009-12-02 19:51:26 : IKE<0.0.0.0        >   spi hash node removed: type<2>,spi<c952285e>,ip<64.52.115.178>
## 2009-12-02 19:51:26 : IKE<0.0.0.0        >   spi hash node removed: type<2>,spi<48dda2f0>,ip<1.1.1.1>
## 2009-12-02 19:51:26 : IKE<1.1.1.1> clean_all_sa_state_node_from_list->
## 2009-12-02 19:51:26 : IKE<1.1.1.1> no relocate earlier SA-state, not active.
## 2009-12-02 19:51:26 : IKE<1.1.1.1> key_modify: sa index <82> bk_idx <82>.
## 2009-12-02 19:51:26 : IKE<0.0.0.0        >   insert_sa_state_to_spi_hash spi<c9522861>, sa_index<82>, Incoming
## 2009-12-02 19:51:26 : IKE<0.0.0.0        >   insert_sa_state_to_spi_hash spi<102efa97>, sa_index<82>, Outgoing
## 2009-12-02 19:51:26 : IKE<1.1.1.1>
crypto_ctx 82, 16, 16, 16, 0, 0, 24, 0, 12, 36
## 2009-12-02 19:51:26 : IKE<1.1.1.1> modify esp tunnel: src (peer) ipv4 <1.1.1.1>
## 2009-12-02 19:51:26 : IKE<1.1.1.1> modifying esp tunnel: self <ipv4 64.52.115.178>
## 2009-12-02 19:51:26 : IKE<1.1.1.1> update auto NHTB status for sa 82
## 2009-12-02 19:51:26 : IKE<1.1.1.1> after mod, out nsptunnel <02d7c3d8>.
## 2009-12-02 19:51:26 : IKE<1.1.1.1> Phase 2 msg-id <00000001>: Completed Quick Mode negotiation with SPI <c9522861>, tunnel ID <174>, and lifetime <3600> seconds/<250000> KB.

The above IPSEC proposal looks like it is a requirement for the new Windows 7 and Vista client to establish the L2TP over IPSEC connection.