VXLAN troubleshooting

The following commands can be used to troubleshoot VXLAN connectivity:

• diagnose sys vxlan fdb list <VXLAN_interface>

• diagnose sys vxlan fdb stat <VXLAN_interface>

• diagnose netlink brctl name host <switch_interface>

• diagnose debug sniffer packet any 'udp and port 4789' 4 0 l

• diagnose debug enable

• diagnose debug flow filter port 4789

• diagnose debug flow trace start <repeat_#>

Topology

The following topology is used as an example configuration to demonstrate VXLAN troubleshooting steps.

In this example, two FortiGates are configured as VXLAN tunnel endpoints (VTEPs). A VXLAN is configured to allow L2 connectivity between the networks behind each FortiGate. The VXLAN interface and port6 are placed on the same L2 network using a software switch (sw100). An L2 network is formed between PC1 and PC2.

The VTEPs have the following MAC address tables:

Interface/endpoint	VTEP 1	VTEP 2
vxlan100	7e:f2:d1:84:75:0f	ca:fa:31:23:8d:c1
port6	00:0c:29:4e:5c:1c	00:0c:29:d0:3e:0d
sw100	00:0c:29:4e:5c:1c	00:0c:29:d0:3e:0d

The MAC address of PC1 is 00:0c:29:90:4f:bf. The MAC address of PC2 is 00:0c:29:f0:88:2c.

To configure the VTEP 1 FortiGate:

1. Configure the local interface:

   config system vxlan
       edit "vxlan100"
           set interface "port2"
           set vni 100
           set remote-ip "192.168.2.87"
       next
   end

2. Configure the interface settings:

   config system interface
       edit "port2"
           set vdom "root"
           set ip 192.168.2.86 255.255.255.0
           set allowaccess ping https ssh http fabric
       next
       edit "vxlan100"
           set vdom "root"
           set type vxlan
           set interface "port2"
       next
   end

3. Configure the software switch:

   config system switch-interface
       edit "sw100"
           set vdom "root"
           set member "port6" "vxlan100"
       next
   end

4. Configure the software switch interface settings:

   config system interface
       edit "sw100"
           set vdom "root"
           set ip 10.10.100.86 255.255.255.0
           set allowaccess ping
           set type switch
           set device-identification enable
           set lldp-transmission enable
           set role lan
       next
   end

To configure the VTEP 2 FortiGate:

1. Configure the local interface:

   config system vxlan
       edit "vxlan100"
           set interface "port2"
           set vni 100
           set remote-ip "192.168.2.86"
       next
   end

2. Configure the interface settings:

   config system interface
       edit "port2"
           set vdom "root"
           set ip 192.168.2.87 255.255.255.0
           set allowaccess ping https ssh snmp http
       next
       edit "vxlan100"
           set vdom "root"
           set type vxlan
           set interface "port2"
       next
   end

3. Configure the software switch:

   config system switch-interface
       edit "sw100"
           set vdom "root"
           set member "port6" "vxlan100"
       next
   end

4. Configure the software switch interface settings:

   config system interface
       edit "sw100"
           set vdom "root"
           set ip 10.10.100.87 255.255.255.0
           set allowaccess ping
           set type switch
           set device-identification enable
           set lldp-transmission enable
           set role lan
           set snmp-index 42
       next
   end

To run diagnostics and debugs:

1. Start a ping from PC1 10.10.100.10 to PC2 10.10.100.20:

   C:\Users\fortidocs>ping 10.10.100.20
   
   Pinging 10.10.100.20 with 32 bytes of data:
   Reply from 10.10.100.20: bytes=32 time=2ms TTL=128
   Reply from 10.10.100.20: bytes=32 time=1ms TTL=128
   Reply from 10.10.100.20: bytes=32 time=1ms TTL=128
   Reply from 10.10.100.20: bytes=32 time<1ms TTL=128
   
   Ping statistics for 10.10.100.20:
       Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
   Approximate round trip times in milli-seconds:
       Minimum = 0ms, Maximum = 2ms, Average = 1ms

2. Verify the ARP table:

   C:\Users\fortidocs>arp /a
   
   Interface: 10.10.100.10 --- 0x21
     Internet Address      Physical Address      Type
     10.10.100.20          00-0c-29-f0-88-2c     dynamic
     10.10.100.86          00-0c-29-4e-5c-1c     dynamic
     10.10.100.255         ff-ff-ff-ff-ff-ff     static
     224.0.0.22            01-00-5e-00-00-16     static
     224.0.0.252           01-00-5e-00-00-fc     static

3. Run diagnostics on the VTEP 1 FortiGate.

   1. Verify the forwarding database of VXLAN interface vxlan100:

      # diagnose sys vxlan fdb list vxlan100
      mac=00:00:00:00:00:00 state=0x0082 remote_ip=192.168.2.87 port=4789 vni=100 ifindex=6
      mac=00:0c:29:f0:88:2c state=0x0002 remote_ip=192.168.2.87 port=4789 vni=100 ifindex=6
      
      total fdb num: 2

      The MAC address 00:0c:29:f0:88:2c is learned from PC2 10.10.100.20.

   2. Verify the summary of statistics from the VXLAN’s forwarding database:

      # diagnose sys vxlan fdb stat vxlan100
      fdb_table_size=256 fdb_table_used=2 fdb_entry=2 fdb_max_depth=1 cleanup_idx=0 cleanup_timer=252

   3. Verify the software switch’s forwarding table:

      # diagnose netlink brctl name host sw100
      show bridge control interface sw100 host.
      fdb: hash size=32768, used=6, num=6, depth=1, gc_time=4, ageing_time=3, simple=switch
      Bridge sw100 host table
      port no device  devname mac addr                ttl     attributes
        1     7       port6   00:0c:29:4e:5c:1c       0       Local Static
        2     33      vxlan100        7e:f2:d1:84:75:0f       0       Local Static
        2     33      vxlan100        00:00:00:00:00:00       26       Hit(26)
        1     7       port6   00:0c:29:90:4f:bf       0        Hit(0)
        1     7       port6   00:0c:29:d0:3e:ef       7        Hit(7)
        2     33      vxlan100        00:0c:29:f0:88:2c       0        Hit(0)

      The MAC address of port6 is 00:0c:29:4e:5c:1c. The MAC address of vxlan100 is 7e:f2:d1:84:75:0f. The MAC address 00:0c:29:f0:88:2c of PC2 is learned from the remote network.

4. Run diagnostics on the VTEP 2 FortiGate.

   1. Verify the forwarding database of VXLAN interface vxlan100:

      # diagnose sys vxlan fdb list vxlan100
      mac=00:00:00:00:00:00 state=0x0082 remote_ip=192.168.2.86 port=4789 vni=100 ifindex=6
      mac=00:0c:29:90:4f:bf state=0x0002 remote_ip=192.168.2.86 port=4789 vni=100 ifindex=6
      
      total fdb num: 2

      The MAC address 00:0c:29:90:4f:bf is learned from PC1 10.10.100.10.

   2. Verify the summary of statistics from the VXLAN’s forwarding database:

      # diagnose sys vxlan fdb stat vxlan100
      fdb_table_size=256 fdb_table_used=2 fdb_entry=2 fdb_max_depth=1 cleanup_idx=0 cleanup_timer=304

   3. Verify the software switch’s forwarding table:

      # diagnose netlink brctl name host sw100
      show bridge control interface sw100 host.
      fdb: hash size=32768, used=5, num=5, depth=1, gc_time=4, ageing_time=3, simple=switch
      Bridge sw100 host table
      port no device  devname mac addr                ttl     attributes
        2     50      vxlan100        00:00:00:00:00:00       10       Hit(10)
        2     50      vxlan100        00:0c:29:90:4f:bf       2        Hit(2)
        1     7       port6   00:0c:29:d0:3e:0d       0       Local Static
        2     50      vxlan100        ca:fa:31:23:8d:c1       0       Local Static
        1     7       port6   00:0c:29:f0:88:2c       0        Hit(0)

      The MAC address of port6 is 00:0c:29:d0:3e:0d. The MAC address of vxlan100 is ca:fa:31:23:8d:c1. The MAC address 00:0c:29:90:4f:bf of PC1 is learned from the remote network.

5. Perform a sniffer trace on the VTEP 1 FortiGate to view the life of the packets as they pass through the FortiGate:

   # diagnose sniffer packet any 'host 10.10.100.20 or (udp and host 192.168.2.87)' 4 0 l
   Using Original Sniffing Mode
   interfaces=[any]
   filters=[host 10.10.100.20 or (udp and host 192.168.2.87)]
   2022-11-04 14:35:18.567602 port6 in arp who-has 10.10.100.20 tell 10.10.100.10
   2022-11-04 14:35:18.567629 vxlan100 out arp who-has 10.10.100.20 tell 10.10.100.10
   2022-11-04 14:35:18.567642 port2 out 192.168.2.86.4804 -> 192.168.2.87.4789: udp 68
   2022-11-04 14:35:18.567658 sw100 in arp who-has 10.10.100.20 tell 10.10.100.10
   2022-11-04 14:35:18.568239 port2 in 192.168.2.87.4789 -> 192.168.2.86.4789: udp 68
   2022-11-04 14:35:18.568263 vxlan100 in arp reply 10.10.100.20 is-at 00:0c:29:f0:88:2c
   2022-11-04 14:35:18.568272 port6 out arp reply 10.10.100.20 is-at 00:0c:29:f0:88:2c
   2022-11-04 14:35:18.568425 port6 in 10.10.100.10 -> 10.10.100.20: icmp: echo request
   2022-11-04 14:35:18.568435 vxlan100 out 10.10.100.10 -> 10.10.100.20: icmp: echo request
   2022-11-04 14:35:18.568443 port2 out 192.168.2.86.4805 -> 192.168.2.87.4789: udp 82
   2022-11-04 14:35:18.568912 port2 in 192.168.2.87.4789 -> 192.168.2.86.4789: udp 68
   2022-11-04 14:35:18.568925 vxlan100 in arp who-has 10.10.100.10 tell 10.10.100.20
   2022-11-04 14:35:18.568935 port6 out arp who-has 10.10.100.10 tell 10.10.100.20
   2022-11-04 14:35:18.568945 sw100 in arp who-has 10.10.100.10 tell 10.10.100.20
   2022-11-04 14:35:18.569070 port6 in arp reply 10.10.100.10 is-at 00:0c:29:90:4f:bf
   2022-11-04 14:35:18.569076 vxlan100 out arp reply 10.10.100.10 is-at 00:0c:29:90:4f:bf
   2022-11-04 14:35:18.569081 port2 out 192.168.2.86.4806 -> 192.168.2.87.4789: udp 68
   2022-11-04 14:35:18.569417 port2 in 192.168.2.87.4789 -> 192.168.2.86.4789: udp 82
   2022-11-04 14:35:18.569427 vxlan100 in 10.10.100.20 -> 10.10.100.10: icmp: echo reply
   2022-11-04 14:35:18.569431 port6 out 10.10.100.20 -> 10.10.100.10: icmp: echo reply

   In the output, the following packet sequence is seen on the FortiGate:

   1. The FortiGate receives an ARP request from PC1 10.10.100.10 on port6.

   2. The ARP request is forwarded to vxlan100 on the same software switch, where it gets encapsulated and sent out as a UDP port 4789 packet on port2.

   3. A reply is received on port2 from the remote VTEP with the ARP response encapsulated in UDP port 4789 again.

   4. The ARP reply is forwarded back out of port6 to PC1.

   5. PC1 sends the ICMP request using the same steps.

6. Perform the same sniffer trace filter with a level 6 verbose level. In this example, the packet capture is converted into a Wireshark file.

   

   The packet that leaves the physical port2 is encapsulated in UDP and has a VXLAN header with VNI 100 as the identifier. There is an additional 50 B overhead of the UDP encapsulated VXLAN packets as opposed to the unencapsulated packets (for example, packet 4 versus packets 1 and 2).