VXLAN over IPsec using a VXLAN tunnel endpoint

This example describes how to implement VXLAN over IPsec VPN using a VXLAN tunnel endpoint (VTEP).

This example shows a specific configuration that uses a hub-and-spoke topology. However, the same logic can be applied to a static VPN with or without XAuth. In this hub-and-spoke topology, dialup VPN is convenient because it uses a single phase 1 dialup definition on the hub FortiGate. Additional spoke tunnels are added without any changes to the hub, other than adding a user account for each additional spoke. Spoke-to-spoke communication is established through the hub. This example assumes the authentication users and user groups have already been created.

IPsec tunnel interfaces are used to support VXLAN tunnel termination. An IP address is set for each tunnel interface. Ping access is allowed for troubleshooting purposes.

VTEPs are created on each of the hub and spokes in order to forward VXLAN traffic through the IPsec tunnels. VXLAN encapsulates OSI layer 2 Ethernet frames within layer 3 IP packets. You will need to either combine the internal port and VXLAN interface into a soft switch, or create a virtual wire pair so that devices behind port1 have direct layer 2 access to remote peers over the VXLAN tunnel. This example uses a switch interface on the hub and a virtual wire pair on the spokes to demonstrate the two different methods.

Finally, in order to apply an IPsec VPN interface on the VXLAN interface setting, net-device must be disabled in the IPsec VPN phase 1 settings. All VXLAN interfaces in this example share the same VXLAN network ID (vni).

To configure the hub FortiGate:

1. Configure the phase 1 and phase 2 interfaces:

   config vpn ipsec phase1-interface
      edit "SPOKES"
         set type dynamic
         set interface "port2"
         set mode aggressive
         set peertype one
         set net-device disable
         set proposal aes256-sha256
         set xauthtype auto
         set authusrgrp "SPOKES"
         set peerid "SPOKES"
         set psksecret <secret>
      next
   end
   config vpn ipsec phase2-interface
      edit "SPOKES"
         set phase1name "SPOKES"
         set proposal aes128-sha1 aes256-sha1 aes128-sha256 aes256-sha256 aes128gcm aes256gcm chacha20poly1305
      next
   end

2. Configure the IPsec VPN policy that allows VXLAN traffic between spokes:

   config firewall policy
      edit 1
         set name "VXLAN_SPOKE_to_SPOKE"
         set srcintf "SPOKES"
         set dstintf "SPOKES"
         set srcaddr "NET_192.168.255.0"
         set dstaddr "NET_192.168.255.0"
         set action accept
         set schedule "always"
         set service "UDP_4789"
         set logtraffic all
         set fsso disable
      next
   end

3. Configure the IPsec tunnel interfaces (the remote IP address is not used, but it is necessary for this configuration):

   config system interface
      edit "SPOKES"
         set vdom "root"
         set ip 192.168.255.1 255.255.255.255
         set allowaccess ping
         set type tunnel
         set remote-ip 192.168.255.254 255.255.255.0
         set snmp-index 12
         set interface "port2"
      next
   end

4. Configure the VXLAN interface (the remote IP is the tunnel interfaces IPs of the spokes):

   config system VXLAN
      edit "SPOKES_VXLAN"
         set interface "SPOKES"
         set vni 1
         set remote-ip "192.168.255.2" "192.168.255.3"
      next
   end

To configure the spoke FortiGates:

1. Configure the phase 1 and phase 2 interfaces:

   config vpn ipsec phase1-interface
      edit "HUB"
         set interface "port2"
         set mode aggressive
         set peertype any
         set net-device disable
         set proposal aes256-sha256
         set localid "SPOKES"
         set xauthtype client
         set authusr "SPOKE1"
         set authpasswd <secret>
         set remote-gw <hub public IP>
         set psksecret <secret>
      next
   end
   config vpn ipsec phase2-interface
      edit "HUB"
         set phase1name "HUB"
         set proposal aes128-sha1 aes256-sha1 aes128-sha256 aes256-sha256 aes128gcm aes256gcm chacha20poly1305
         set auto-negotiate enable
         set src-subnet 192.168.255.2 255.255.255.255
      next
   end

   The hub FortiGate inserts a reverse route pointing to newly established tunnel interfaces for any of the subnets that the spoke FortiGate's source quick mode selectors provides. This is why you should set the tunnel IP address here.

2. Configure the IPsec VPN policy:

   config firewall policy
      edit 1
         set name "VTEP_IPSEC_POLICY"
         set srcintf "HUB"
         set dstintf "HUB"
         set srcaddr "none"
         set dstaddr "none"
         set action accept
         set schedule "always"
         set service "PING"
         set logtraffic disable
         set fsso disable
      next
   end

3. Configure the IPsec tunnel interfaces:

   config system interface
      edit "HUB"
         set vdom "root"
         set ip 192.168.255.2 255.255.255.255
         set allowaccess ping
         set type tunnel
         set remote-ip 192.168.255.1 255.255.255.0
         set snmp-index 12
         set interface "port2"
      next
   end

4. Configure the VXLAN interface (the remote IP is the tunnel interface IP of the hub):

   config system VXLAN
      edit "HUB_VXLAN"
         set interface "HUB"
         set vni 1
         set remote-ip "192.168.255.1"
      next
   end

   To establish a VXLAN tunnel between spokes, you can add a spoke's tunnel IP address in remote-ip.

   To add more remote IP addresses to a VXLAN interface, the interface cannot be in use. You may want to provision future spokes' remote IP addresses at this point to avoid traffic disruption. Otherwise, you must delete the reference (the policy in this example) before adding remote IP addresses.

To bind the VXLAN interface to the internal interface:

1. Configure a switch interface on the hub:

   config system switch-interface
       edit "SW"
           set vdom "root"
           set member "port1" "SPOKES_VXLAN"
           set intra-switch-policy {implicit | explicit}
       next
   end

   Allowing intra-switch traffic is implicitly allowed by default. Use set intra-switch-policy explicit to require firewall policies to allow traffic between switch interfaces.

2. Configure a virtual wire pair on the spokes:

   config system virtual-wire-pair
       edit "VWP"
           set member "HUB_VXLAN" "port1"
       next
   end

   The virtual wire pair requires an explicit policy to allow traffic between interfaces.

To test the configuration:

1. Ping the hub FortiGate from the spoke FortiGate:

   user@pc-spoke1:~$ ping 192.168.1.1 -c 3PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
   64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=1.24 ms
   64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.672 ms
   64 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=0.855 ms
   --- 192.168.1.1 ping statistics ---
   3 packets transmitted, 3 received, 0% packet loss, time 2002ms
   rtt min/avg/max/mdev = 0.672/0.923/1.243/0.239 ms

2. Sniff traffic on the hub FortiGate:

   # diagnose sniffer packet any 'icmp or (udp and port 4789)' 4 0 ainterfaces=[any]
   filters=[icmp or (udp and port 4789)]
   15:00:01.438230 SPOKES in 192.168.255.2.4790 -> 192.168.255.1.4789: udp 106
   <<<<1
   15:00:01.438256 SPOKES_VXLAN in 192.168.1.2 -&gt; 192.168.1.1: icmp: echo request
   <<<<2
   15:00:01.438260 port1 out 192.168.1.2 -&gt; 192.168.1.1: icmp: echo request
   <<<<3
   15:00:01.438532 port1 in 192.168.1.1 -&gt; 192.168.1.2: icmp: echo reply
   15:00:01.438536 SPOKES_VXLAN out 192.168.1.1 -&gt; 192.168.1.2: icmp: echo reply
   15:00:01.438546 SPOKES out 192.168.255.1.4851 -&gt; 192.168.255.2.4789: udp 106