FGSP per-tunnel failover for IPsec

During FGSP per-tunnel failover for IPsec, the same IPsec dialup server configured on each FGSP member may establish tunnels with dialup clients as the primary gateway. The IPsec SAs are synchronized to all other FGSP peers that have FGSP synchronization for IPsec enabled. Other FGSP members may establish a tunnel with other clients on the same dialup server and synchronize their SAs to other peers.

Upon the failure of the FGSP member that is the primary gateway for a tunnel, the upstream router will fail over the tunnel traffic to another FGSP member. The other FGSP member will move from standby to the primary gateway for that tunnel and continue to forward traffic.

config vpn ipsec phase1-interface
    edit <name>
        set fgsp-sync {enable | disable}
    next
end

Example

In this example, the FGSP peers are connected on port4 over 172.31.1.1-4/24. Each peer has a loopback interface, lb1, with the same IP address. This loopback interface is used as the local gateway on each of the phase 1 connections to avoid each FGSP member having different IPs on port2. The DC Router uses ECMP to distribute traffic to each FGSP peer. It is assumed that the networking addresses are already configured properly.

Interface/setting	DC1_VM1	DC1_VM2	DC1_VM3	DC1_VM4
port2	192.168.125.254/24	192.168.126.254/24	192.168.127.254/24	192.168.128.254/24
port3	172.31.125.254/24	172.31.126.254/24	172.31.127.254/24	172.31.128.254/24
port4	172.31.1.1/24	172.31.1.2/24	172.31.1.3/24	172.31.1.4/24
lb1	192.168.202.31/32	192.168.202.31/32	192.168.202.31/32	192.168.202.31/32
fgsp-sync	Enabled	Enabled	Enabled	Disabled

Out of the four FGSP peers, DC1_VM1, DC1_VM2, and DC1_VM3 have fgsp-sync enabled in their IPsec phase 1 configurations. This allows the three FGSP members to synchronize IPsec SAs as clients establish dialup tunnels to them individually. DC1_VM4, which does not have fgsp-sync configured, will not participate in synchronizing IPsec SAs or establishing tunnels. The DC Router uses ECMP to route traffic to the destination 192.168.202.31 through each of the participating FGSP peers.

In a larger scale there may be many more IPsec dialup clients connecting, with each eligible FGSP peer being the primary gateway for a set of dialup tunnels, and is in standby for the rest of the tunnels. If an FGSP peer fails, traffic will fail over to other peers, and these peers will become primary gateways for the respective dialup tunnels.

To configure the FGSP peers (DC1_VM1):

The following steps are to configure DC1_VM1. The other peers have similar configurations based on the preceding table. In the config vpn ipsec phase1-interface settings, all peers should have the same local gateway external interface (192.168.202.31).

1. Configure the FGSP cluster and session synchronization:

   config system cluster-sync
       edit 1
           set peerip 172.31.1.2
       next
       edit 2
           set peerip 172.31.1.3
       next
       edit 3
           set peerip 172.31.1.4
       next
   end

2. Configure the additional FGSP attributes:

   config system standalone-cluster
       set standalone-group-id 1
       set group-member-id 1
   end

3. Configure the VPN tunnel phase 1 settings:

   config vpn ipsec phase1-interface
       edit "vpn1"
           set type dynamic
           set interface "port2"
           set ike-version 2
           set local-gw 192.168.202.31
           set keylife 90000
           set peertype one
           set net-device disable
           set proposal aes128-sha1
           set dpd on-idle
           set dhgrp 2
           set fgsp-sync enable
           set nattraversal disable
           set peerid "Nokia_Peer"
           set psksecret xxxxx
           set dpd-retryinterval 60
       next
   end

4. Configure the VPN tunnel phase 2 settings:

   config vpn ipsec phase2-interface
       edit "vpn1"
           set phase1name "vpn1"
           set proposal aes128-sha1
           set keylifeseconds 10800
       next
   end

To verify the configuration:

1. Once the FGSP members establish peering with each other, verify the standalone peers on DC1_VM1:

   DC1_VM1 # diagnose sys ha standalone-peers
   Group=1, ID=1
   Detected-peers=3
   Kernel standalone-peers: num=3.
   peer0: vfid=0, peerip:port = 172.31.1.2:708, standalone_id=2
           session-type: send=0, recv=0
            packet-type: send=0, recv=0
   peer1: vfid=0, peerip:port = 172.31.1.3:708, standalone_id=3
           session-type: send=0, recv=0
            packet-type: send=0, recv=0
   peer2: vfid=0, peerip:port = 172.31.1.4:708, standalone_id=4
           session-type: send=0, recv=0
            packet-type: send=0, recv=0
   Kernel standalone dev_base:
           standalone_id=0:
           standalone_id=1:
                   phyindex=0: mac=00:0c:29:22:00:6b, linkfail=1
                   phyindex=1: mac=00:0c:29:22:00:75, linkfail=1
                   phyindex=2: mac=00:0c:29:22:00:7f, linkfail=1
                   phyindex=3: mac=00:0c:29:22:00:89, linkfail=1
                   phyindex=4: mac=00:0c:29:22:00:93, linkfail=1
                   phyindex=5: mac=00:0c:29:22:00:9d, linkfail=1
                   phyindex=6: mac=00:0c:29:22:00:a7, linkfail=1
                   phyindex=7: mac=00:0c:29:22:00:b1, linkfail=1
                   phyindex=8: mac=00:0c:29:22:00:bb, linkfail=1
                   phyindex=9: mac=00:0c:29:22:00:c5, linkfail=1
           standalone_id=2:
                   phyindex=0: mac=00:0c:29:06:4e:d6, linkfail=1
                   phyindex=1: mac=00:0c:29:06:4e:e0, linkfail=1
                   phyindex=2: mac=00:0c:29:06:4e:ea, linkfail=1
                   phyindex=3: mac=00:0c:29:06:4e:f4, linkfail=1
                   phyindex=4: mac=00:0c:29:06:4e:fe, linkfail=1
                   phyindex=5: mac=00:0c:29:06:4e:08, linkfail=1
                   phyindex=6: mac=00:0c:29:06:4e:12, linkfail=1
                   phyindex=7: mac=00:0c:29:06:4e:1c, linkfail=1
                   phyindex=8: mac=00:0c:29:06:4e:26, linkfail=1
                   phyindex=9: mac=00:0c:29:06:4e:30, linkfail=1
           standalone_id=3:
                   phyindex=0: mac=00:0c:29:70:b9:6c, linkfail=1
                   phyindex=1: mac=00:0c:29:70:b9:76, linkfail=1
                   phyindex=2: mac=00:0c:29:70:b9:80, linkfail=1
                   phyindex=3: mac=00:0c:29:70:b9:8a, linkfail=1
                   phyindex=4: mac=00:0c:29:70:b9:94, linkfail=1
                   phyindex=5: mac=00:0c:29:70:b9:9e, linkfail=1
                   phyindex=6: mac=00:0c:29:70:b9:a8, linkfail=1
                   phyindex=7: mac=00:0c:29:70:b9:b2, linkfail=1
                   phyindex=8: mac=00:0c:29:70:b9:bc, linkfail=1
                   phyindex=9: mac=00:0c:29:70:b9:c6, linkfail=1
           standalone_id=4:
                   phyindex=0: mac=00:0c:29:5c:d3:23, linkfail=1
                   phyindex=1: mac=00:0c:29:5c:d3:2d, linkfail=1
                   phyindex=2: mac=00:0c:29:5c:d3:37, linkfail=1
                   phyindex=3: mac=00:0c:29:5c:d3:41, linkfail=1
                   phyindex=4: mac=00:0c:29:5c:d3:4b, linkfail=1
                   phyindex=5: mac=00:0c:29:5c:d3:55, linkfail=1
                   phyindex=6: mac=00:0c:29:5c:d3:5f, linkfail=1
                   phyindex=7: mac=00:0c:29:5c:d3:69, linkfail=1
                   phyindex=8: mac=00:0c:29:5c:d3:73, linkfail=1
                   phyindex=9: mac=00:0c:29:5c:d3:7d, linkfail=1
           standalone_id=5:
           ...
           standalone_id=15:

2. Initiate a dialup tunnel connection from the IPsec Client 2 FortiGate (192.168.1.2).

3. Verify the tunnel list for vpn1_1 on each peer. The output shows the bi-directional SAs for that particular tunnel are synchronized to all participating FGSP peers.

   1. DC1_VM1:

      DC1_VM1 # diagnose vpn tunnel list name vpn1_1
      list ipsec tunnel by names in vd 0
      ------------------------------------------------------
      name=vpn1_1 ver=2 serial=a4 192.168.202.31:0->192.168.1.2:0 tun_id=192.168.1.2 tun_id6=::10.0.0.15 dst_mtu=1500 dpd-link=on weight=1
      bound_if=6 lgwy=static/1 tun=intf/0 mode=dial_inst/3 encap=none/8840 options[2288]=npu rgwy-chg frag-rfc  run_state=0 role=sync-primary accept_traffic=1 overlay_id=0
      
      parent=vpn1 index=1
      proxyid_num=1 child_num=0 refcnt=6 ilast=6 olast=6 ad=/0
      stat: rxp=0 txp=0 rxb=0 txb=0
      dpd: mode=on-idle on=1 idle=60000ms retry=3 count=0 seqno=20
      natt: mode=none draft=0 interval=0 remote_port=0
      fec: egress=0 ingress=0
      proxyid=vpn1 proto=0 sa=1 ref=2 serial=3 add-route
        src: 0:0.0.0.0-255.255.255.255:0
        dst: 0:10.10.1.0-10.10.1.255:0
        SA:  ref=3 options=682 type=00 soft=0 mtu=1438 expire=10480/0B replaywin=2048
             seqno=1 esn=0 replaywin_lastseq=00000000 qat=0 rekey=0 hash_search_len=1
        life: type=01 bytes=0/0 timeout=10788/10800
        dec: spi=a575b631 esp=aes key=16 5de449f75c7d70258f4972506dd164e2
             ah=sha1 key=20 7e65d641be6bc52655619ff542c67c61713de523
        enc: spi=10aa45b0 esp=aes key=16 65ad3b4849386deb4f3028079a657257
             ah=sha1 key=20 b5f1e1c6786f69482b5d271347a69a0cbb83ed58
        dec:pkts/bytes=0/0, enc:pkts/bytes=0/0
        npu_flag=00 npu_rgwy=192.168.1.2 npu_lgwy=192.168.202.31 npu_selid=b2 dec_npuid=0 enc_npuid=0

   2. DC1_VM2:

      DC1_VM2 # diagnose vpn tunnel list name vpn1_1
      list ipsec tunnel by names in vd 0
      ------------------------------------------------------
      name=vpn1_1 ver=2 serial=a3 192.168.202.31:0->192.168.1.2:0 tun_id=192.168.1.2 tun_id6=::10.0.0.15 dst_mtu=0 dpd-link=on weight=1
      bound_if=6 lgwy=static/1 tun=intf/0 mode=dial_inst/3 encap=none/8712 options[2208]=npu frag-rfc  run_state=0 role=standby accept_traffic=1 overlay_id=0
      
      parent=vpn1 index=1
      proxyid_num=1 child_num=0 refcnt=6 ilast=43063501 olast=43063501 ad=/0
      stat: rxp=0 txp=0 rxb=0 txb=0
      dpd: mode=on-idle on=1 idle=60000ms retry=3 count=0 seqno=0
      natt: mode=none draft=0 interval=0 remote_port=0
      fec: egress=0 ingress=0
      proxyid=vpn1 proto=0 sa=1 ref=2 serial=3 add-route
        src: 0:0.0.0.0-255.255.255.255:0
        dst: 0:10.10.1.0-10.10.1.255:0
        SA:  ref=3 options=682 type=00 soft=0 mtu=1280 expire=10466/0B replaywin=2048
             seqno=10000001 esn=0 replaywin_lastseq=00000000 qat=0 rekey=0 hash_search_len=1
        life: type=01 bytes=0/0 timeout=10788/10800
        dec: spi=a575b631 esp=aes key=16 5de449f75c7d70258f4972506dd164e2
             ah=sha1 key=20 7e65d641be6bc52655619ff542c67c61713de523
        enc: spi=10aa45b0 esp=aes key=16 65ad3b4849386deb4f3028079a657257
             ah=sha1 key=20 b5f1e1c6786f69482b5d271347a69a0cbb83ed58
        dec:pkts/bytes=0/0, enc:pkts/bytes=0/0
        npu_flag=00 npu_rgwy=192.168.1.2 npu_lgwy=192.168.202.31 npu_selid=ab dec_npuid=0 enc_npuid=0

   3. DC1_VM3:

      DC1_VM3 # diagnose vpn tunnel list name vpn1_1
      list ipsec tunnel by names in vd 0
      ------------------------------------------------------
      name=vpn1_1 ver=2 serial=ac 192.168.202.31:0->192.168.1.2:0 tun_id=192.168.1.2 tun_id6=::10.0.0.15 dst_mtu=0 dpd-link=on weight=1
      bound_if=6 lgwy=static/1 tun=intf/0 mode=dial_inst/3 encap=none/8712 options[2208]=npu frag-rfc  run_state=0 role=standby accept_traffic=1 overlay_id=0
      
      parent=vpn1 index=1
      proxyid_num=1 child_num=0 refcnt=6 ilast=43063499 olast=43063499 ad=/0
      stat: rxp=0 txp=0 rxb=0 txb=0
      dpd: mode=on-idle on=1 idle=60000ms retry=3 count=0 seqno=0
      natt: mode=none draft=0 interval=0 remote_port=0
      fec: egress=0 ingress=0
      proxyid=vpn1 proto=0 sa=1 ref=2 serial=2 add-route
        src: 0:0.0.0.0-255.255.255.255:0
        dst: 0:10.10.1.0-10.10.1.255:0
        SA:  ref=3 options=682 type=00 soft=0 mtu=1280 expire=10462/0B replaywin=2048
             seqno=10000001 esn=0 replaywin_lastseq=00000000 qat=0 rekey=0 hash_search_len=1
        life: type=01 bytes=0/0 timeout=10788/10800
        dec: spi=a575b631 esp=aes key=16 5de449f75c7d70258f4972506dd164e2
             ah=sha1 key=20 7e65d641be6bc52655619ff542c67c61713de523
        enc: spi=10aa45b0 esp=aes key=16 65ad3b4849386deb4f3028079a657257
             ah=sha1 key=20 b5f1e1c6786f69482b5d271347a69a0cbb83ed58
        dec:pkts/bytes=0/0, enc:pkts/bytes=0/0
        npu_flag=00 npu_rgwy=192.168.1.2 npu_lgwy=192.168.202.31 npu_selid=b4 dec_npuid=0 enc_npuid=0

   4. DC1_VM4:

      DC1_VM4 # diagnose vpn tunnel list name vpn1_1
      list ipsec tunnel by names in vd 0

   The IPsec tunnel role=sync-primaryon DC1_VM1 indicates that the IPsec tunnel was established on the FortiGate and traffic is being forwarded. On DC1_VM2 and DC1_VM3, the IPsec tunnel role=standby indicates that they are synchronized from the FGSP peer and are in standby for traffic forwarding.

   The IPsec SAs do not synchronize to DC1_VM4 because fgsp-sync is disabled.

4. When a failure occurs on DC1_VM1, the tunnel traffic will fail over to either DC1_VM2 or DC1_VM3. Its tunnel role will become role=sync-primary.