Troubleshoot Cisco Site-to-Site VPNs

Cisco2

Troubleshoot Cisco Site-to-Site VPNs

This article is a how to on troubleshooting a Cisco router site to site VPN that isn’t connecting or isn’t passing traffic. Troubleshooting Cisco site to site VPNs may not seem as straight forward as some other routers or firewalls as we don’t always get a decent GUI with lots of visual feedback telling us our VPN works or doesn’t work. However with the right commands it can be straight forward to troubleshoot a Cisco site-to-site VPN and isolate where the failure is and why it fails.

Basic example Site-to-Site VPN scenario:

Troubleshoot Cisco Site-to-Site VPNs

Here you can see one router on each site separated by the internet, I haven’t shown the public IPs but have shown the internal networks at either end of the VPN as these are important to ensuring the right traffic is passed over the VPN. The external IP is also important but you only really need to know that whenever you are asked for a peer address that means the external IP of the router you want to establish a tunnel with.

Here is an example of one of the router’s configurations:

.
crypto isakmp policy 1
 encr 3des
 hash md5
 authentication pre-share
 group 2
 lifetime 43200
crypto isakmp key 0 yourpsk address XXX.XXX.XXX.XXX

crypto ipsec transform-set exampleset esp-3des esp-md5-hmac

crypto map vpn 10 ipsec-isakmp
 set peer XXX.XXX.XXX.XXX
 set security-association lifetime kilobytes 10240
 set security-association lifetime seconds 43200
 set transform-set exampleset
 set pfs group2
 match address 120
.
.
interface Dialer1
 ip address negotiated
 ip access-group 100 in
 ip nat outside
 ip inspect firewall out
 encapsulation ppp
 no ip route-cache
 no ip mroute-cache
 dialer pool 1
 dialer-group 1
 no cdp enable
 ppp authentication chap callin
 ppp chap hostname dslusername@username
 ppp chap password 0 password
 ppp ipcp dns request
 ppp ipcp mask request
 crypto map vpn
.
ip nat inside source route-map nonat interface Dialer1 overload
ip nat inside source static tcp 10.0.0.2 25 interface Dialer1 25
ip nat inside source static tcp 10.0.0.2 3389 interface Dialer1 3389
.
.
access-list 100 remark ----- Inbound ACL -----
access-list 100 permit tcp any any eq 22
access-list 100 permit tcp any any eq smtp
access-list 100 permit tcp any any eq 3389
access-list 100 permit udp any any eq snmp
access-list 100 permit udp any any eq isakmp
access-list 100 permit icmp any any unreachable
access-list 100 permit icmp any any time-exceeded
access-list 100 permit icmp any any echo
access-list 100 permit esp any any
access-list 100 deny   ip any any
access-list 110 remark ----- NATed Addresses -----
access-list 110 deny   ip 10.0.0.0 0.0.0.255 10.1.0.0 0.0.0.255
access-list 110 permit ip 10.0.0.0 0.0.0.255 any
access-list 120 remark ----- Match List for Crypto Sequence 10 -----
access-list 120 permit ip 10.0.0.0 0.0.0.255 10.1.0.0 0.0.0.255
access-list 120 deny   ip any any
.
.
route-map nonat permit 10
match ip address 110
.

Firstly lets look at some steps to check over your configuration. A site-to-site VPN has two phases so we need to isolate which phase is failing and why.

1. Check your peer addresses and key

Most fundamentally the peer IP addresses of your routers need to be correct at both ends so the routers can find one another, you will also need to make the crypto key is the same at both ends so the peers can authenticate with one another. The line you are interested in here is:

crypto isakmp key 0 yourpsk address XXX.XXX.XXX.XXX

You need to make sure the PSK matches on the routers at both ends of the tunnel. The IP address should be the public/external IP of the router you are wishing to establish a tunnel with.

The peer address is also set under the crypto map and must also be accurate and set to the external/public IP of the peer. Here is the example configuration line:

set peer XXX.XXX.XXX.XXX

2. Check the ‘interesting traffic’ is specified and correct

To even initiate phase 1 your router needs to know what traffic it should actually be trying to tunnel. This is done with an ACL. In the crypto map you specify the ACL that should be used to determine the interesting traffic in the line:

match address 120

Here access-list 120 has been named as the ACL that specifies the traffic that should go via the VPN. So now to take a look at the actual ACL:

access-list 120 remark ----- Match List for Crypto Sequence 10 -----
access-list 120 permit ip 10.0.0.0 0.0.0.255 10.1.0.0 0.0.0.255
access-list 120 deny   ip any any

The remark line is not really necessary and neither is the deny as deny ip any any is implied for any traffic that is not specified as permitted. Access lists are processed top down so if you do have a deny line your permit must be above it. The permit line should specify the internal network from where the traffic is originating (the source) and the internal network of where the traffic is going to (the destination). The example tells the router that any traffic from the network 10.0.0.0/24 trying to reach the 10.1.0.0/24 network should go via the VPN. This configuration at the other end of the VPN will look very similar but the source and destination networks will be the other way around.

With this in mind remember that to initiate the IPSEC tunnel in the first place some traffic has to actually be directed over it, don’t overlook that when checking if the tunnel is up!

3. NAT or more specifically what not to NAT

The traffic you want to go over the VPN (the same traffic that we talked about above) above needs to be excluded from NAT. Normally all outgoing traffic will be translated to a public IP address but we don’t want to translate the traffic as it is to remain within our private network. Firstly take a look at these example lines of NAT configuration:

ip nat inside source route-map nonat interface Dialer1 overload
ip nat inside source static tcp 10.0.0.2 25 interface Dialer1 25
ip nat inside source static tcp 10.0.0.2 3389 interface Dialer1 3389

Here we can see a couple of static rules to NAT SMTP and RDP to internal servers but the rule we are particularly interested in is the top rule, which specifies a route-map for traffic to NAT. Here is the route-map configuration:

route-map nonat permit 10
match ip address 110

Here the route-map named ‘nonat’ (the route-map specified in the NAT lines we have looked at) is set to match addresses with an ACL much like the crypto map we looked at in step 2. Here is the ACL:

access-list 110 remark ----- NATed Addresses -----
access-list 110 deny   ip 10.0.0.0 0.0.0.255 10.1.0.0 0.0.0.255
access-list 110 permit ip 10.0.0.0 0.0.0.255 any

Again the remark line is just for information and is not actually needed. The difference with this ACL is you can see that the ACL begins with a deny, to deny traffic from one internal network to the internal network at the other end of our VPN. So traffic from the network 10.0.0.0/24 going to 10.1.0.0/24 will not be translated. The rule that follows says that ALL other traffic coming from 10.0.0.0/24 will be translated. Remember that access-lists are processed top down so our VPN traffic is already excluded from NAT before it reaches the permit rule.

The configuration on the router at the other end will look almost identical except the network addresses will be switched around as per below:

access-list 110 remark ----- NATed Addresses -----
access-list 110 deny   ip 10.1.0.0 0.0.0.255 10.0.0.0 0.0.0.255
access-list 110 permit ip 10.1.0.0 0.0.0.255 any

4. Checking encryption methods and other values to ensure they match

Above we have covered some of the most common causes of our VPN tunnel failing so lets go over the remaining configuration and another potential reason for VPN tunnels not behaving.

Firstly here is the isakmp policy:

crypto isakmp policy 1
 encr 3des
 hash md5
 authentication pre-share
 group 2
 lifetime 43200

There are various options here that are potentially valid, the basic rule is to ensure is that the configuration is the same at both ends of the tunnel. Encryption and hash options are probably the most common mistakes at either end, here 3des and md5 have been chosen so they must also be set on the peer.

The same basic rule applies to the crypto map settings but before we look at that lets look at the transform set. You can create multiple transform sets to specify different esp encryption and hash options for later use in your crypto maps. Here is our example configuration:

crypto ipsec transform-set exampleset esp-3des esp-md5-hmac

In this example the name of the set is exampleset and 3des and md5 have been chosen again although this is not to be confused with the isakmp policy, this doesn’t have to be the same as the isakmp policy. The encryption and hash in the isakmp policy is for phase 1 and the transform set is for phase 2. However, as with the isakmp policy, the same transform options must be used on both peers (although the name can be different).

Next is the crypto map, some of these setting we have seen already but here is the complete configuration:

crypto map vpn 10 ipsec-isakmp
 set peer XXX.XXX.XXX.XXX
 set security-association lifetime kilobytes 10240
 set security-association lifetime seconds 43200
 set transform-set exampleset
 set pfs group2
 match address 120

Firstly we name the map, in this case it is named ‘vpn’. You can have multiple VPNs on your router (how many depends on model/licence), when creating additional tunnels use the same name but use a different number. Here you can see we have specified which transform set we want to use and regardless of what we call the set at either end the settings must match. With the exception of the values we have already discussed the remaining values have to match, e.g. the lifetime settings and pfs group at both ends should be the same.

5. Ensure the crypto map is applied to the outside interface and check MTU

As the title says your crypto map must be attached to your outside interface. The example here is a DSL router so the outside interface is Dialer1, see the last line in this section of example configuration:

interface Dialer1
 ip address negotiated
 ip access-group 100 in
 ip nat outside
 ip inspect firewall out
 encapsulation ppp
 no ip route-cache
 no ip mroute-cache
 dialer pool 1
 dialer-group 1
 no cdp enable
 ppp authentication chap callin
 ppp chap hostname dslusername@username
 ppp chap password 0 password
 ppp ipcp dns request
 ppp ipcp mask request
 crypto map vpn

While we are looking at the outside interface we should also check that the MTU value is consistent at both ends of the VPN, having a mismatching MTU will cause packet fragmentation and unreliable traffic over your VPN. The MTU value should also be no larger than what the internet connection can support or again fragmentation will occur, your ISP may be able to indicate a preferred value or you can determine it yourself but I am not going to cover that here. In the UK somewhere close to 1400 is usually sufficiently low. To see the MTU use the command ‘show interface Dialer1’ (replace dialer1 with the name of your outside interface).

6. The inbound ACLs

Despite your VPN traffic being tunnelled through your firewall the inbound ACL still needs to have a couple of ports open for our tunnel to be able to establish in the first place. We need to permit isakmp and ESP on our inbound access-list, here is the access-list in our example configuration:

access-list 100 remark ----- Inbound ACL -----
access-list 100 permit tcp any any eq 22
access-list 100 permit tcp any any eq smtp
access-list 100 permit tcp any any eq 3389
access-list 100 permit udp any any eq snmp
access-list 100 permit udp any any eq isakmp
access-list 100 permit icmp any any unreachable
access-list 100 permit icmp any any time-exceeded
access-list 100 permit icmp any any echo
access-list 100 permit esp any any
access-list 100 deny   ip any any

Again the deny is not really needed as it is implied and access-lists are applied top down so make sure your permit’s are above any deny that might include your traffic.

Generally what you see in this example is all you need however sometimes it is necessary to use a slightly different rule for isakmp. Isakmp usually uses UDP 500 but it is also able to use UDP 4500, it does this when the router is behind NAT-T. If this is the case you can add a permit rule to your access-list as per this example:

access-list 100 permit udp any any eq non500-isakmp

7. Checking the status of your VPN

There are a few very useful commands you should know that will show you the status of your site-to-site VPN’s and with them you can get some clues as to where any problems lie.

Firstly a command that will tell you straight away if your tunnel is up (IP addresses in these examples are not real):

TEST_ROUTER#sh crypto session
Crypto session current status

Interface: Dialer1
Session status: UP-IDLE
Peer: 195.104.126.125 port 500
  IKE SA: local 195.152.66.51/500 remote 195.104.126.125/500 Active
  IPSEC FLOW: deny ip 0.0.0.0/0.0.0.0 0.0.0.0/0.0.0.0
        Active SAs: 0, origin: crypto map

Interface: Dialer1
Session status: UP-ACTIVE
Peer: 195.104.126.125 port 4500
  IKE SA: local 195.152.66.51/4500 remote 195.104.126.125/4500 Active
  IPSEC FLOW: permit ip 10.1.0.0/255.255.255.0 10.0.0.0/255.255.255.0
        Active SAs: 2, origin: crypto map

Here we can see the tunnel is UP-ACTIVE to peer 195.104.126.125 on port 4500. This command will help identify if your tunnel is up but doesn’t by itself give you much useful troubleshooting information.

The following two commands will give you much more useful information for troubleshooting as well as also identify whether phase 1 and 2 of your VPN are established.

TEST_ROUTER#sh crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst             src             state          conn-id slot status
195.152.66.51    195.104.126.125 QM_IDLE           2012    0 ACTIVE
195.104.126.125 195.152.66.51    QM_IDLE           2011    0 ACTIVE

IPv6 Crypto ISAKMP SA

TEST_ROUTER#sh crypto ipsec sa

interface: Dialer1
    Crypto map tag: vpn, local addr 195.152.66.51

   protected vrf: (none)
   local  ident (addr/mask/prot/port): (10.1.0.0/255.255.255.0/0/0)
   remote ident (addr/mask/prot/port): (10.0.0.0/255.255.255.0/0/0)
   current_peer 195.104.126.125 port 4500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 18884, #pkts encrypt: 18884, #pkts digest: 18884
    #pkts decaps: 20236, #pkts decrypt: 20236, #pkts verify: 20236
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 1, #recv errors 0

     local crypto endpt.: 195.152.66.51, remote crypto endpt.: 195.104.126.125
     path mtu 1452, ip mtu 1452, ip mtu idb Dialer1
     current outbound spi: 0x4A1F9F2A(1243586346)

     inbound esp sas:
      spi: 0xC01C426B(3223077483)
        transform: esp-3des esp-md5-hmac ,
        in use settings ={Tunnel, }
        conn id: 301, flow_id: Motorola SEC 2.0:301, crypto map: vpn
        sa timing: remaining key lifetime (k/sec): (1591/38353)
        IV size: 8 bytes
        replay detection support: Y
        Status: ACTIVE

     inbound ah sas:

     inbound pcp sas:

     outbound esp sas:
      spi: 0x4A1F9F2A(1243586346)
        transform: esp-3des esp-md5-hmac ,
        in use settings ={Tunnel, }
        conn id: 302, flow_id: Motorola SEC 2.0:302, crypto map: vpn
        sa timing: remaining key lifetime (k/sec): (7640/38353)
        IV size: 8 bytes
        replay detection support: Y
        Status: ACTIVE

     outbound ah sas:

     outbound pcp sas:

The first command ‘show crypto isakmp sa’ shows you the state of isakmp negotiation. Any problems here may indicate issues with your isakmp policy such as mismatches in encryption or hash, a missing permit on your inbound ACL for isakmp or an incorrect PSK.

The second command ‘show crypto IPsec sa’ shows you the state of the IPsec tunnel. This command can show you again whether your tunnel is up, where you see inbound esp and outbound esp have a status of active then the tunnel is up. If the status of these is not active or you can’t see the packet counters going up you could have problems with not specifying the interesting traffic correctly or not having excluded the interesting traffic from NAT. Other causes could be the transform set not matching at both ends or esp not being permitted on your inbound ACLs.

8. Debugging

Lastly if you still don’t have a working VPN you can enable debugging with ‘debug crypto isakmp’  or if it is IPsec that is failing ‘debug crypto ipsec’ The output allows you to see in better detail exactly where the failure is occurring. You can see the real-time output of debugging with the command ‘terminal monitor’.

Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes:

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>