Fortinet black logo

Administration Guide

VXLAN with MP-BGP EVPN

VXLAN with MP-BGP EVPN

FortiOS supports VXLAN as implemented according to RFC 7348. Currently, VXLAN relies on determining the MAC address of the destination host by using address resolution protocol (ARP) broadcast frames encapsulated in multicast packets.

  • A multicast group is maintained with all the VXLAN tunnel endpoints (VTEPs) associated with the same VXLAN, namely, with the same VXLAN network identifier (VNI).

  • The multicast packets that encapsulate ARP broadcast frames are sent to this multicast group, and then the destination host replies to the source host using unicast IP packet encapsulated using VXLAN.

  • The source and destination FortiGates as VTEPs each maintain a mapping of MAC addresses to remote VTEPs.

As with non-VXLAN traffic, VXLAN relies on the preceding ARP process, commonly known as flood-and-learn that floods the network with broadcast frames encapsulated as multicast packets to learn MAC addresses. In the RFC 7348 implementation of VXLAN, the data plane is simultaneously used as a control plane.

The following topology demonstrates how flood-and-learn uses ARP broadcast traffic flooded throughout the VXLAN for PC A to learn PC D's MAC address when PC A tries to connect to PC D.

Multiprotocol Border Gateway Protocol Ethernet Virtual Private Network (MP-BGP EVPN) support for VXLAN allows for learning MAC addresses in a way that is more suitable for large deployments than flood-and-learn.

MP-BGP EVPN is a standards-based control plane that supports the distribution of attached host MAC and IP addresses using MP-BGP, namely, using the EVPN address family and MAC addresses treated as routing entries in BGP. As a control plane that is separate from the data plane, MP-BGP EVPN avoids flood-and-learn in the network, and the wide use of BGP as an external gateway protocol on the internet proves its ability to scale well with large deployments. The following topology demonstrates how MP-BGP EVPN distributes route type 2 MAC/IP advertisement routes among VTEPs in the VXLAN, and minimizes ARP broadcast traffic required for PC A to learn PC D's MAC address when PC A tries to connect to PC D.

MP-BGP EVPN supports the following features:

  • Route type 2 (MAC/IP advertisement route) and route type 3 (inclusive multicast Ethernet tag route)

  • Intra-subnet communication

  • Single-homing use cases

  • VLAN-based service, namely, there is only one broadcast domain per EVPN instance (EVI). This is due to the current VXLAN design that supports a single VNI for a VXLAN interface.

  • EVPN running on IPv4 unicast VXLAN

  • Egress replication for broadcast, unknown unicast, and multicast (BUM) traffic

  • VXLAN MAC learning from traffic

  • IP address local learning

  • ARP suppression

Note

For more information about MP-BGP EVPN, see RFC 7432. For more information about EVPN and VXLAN, see RFC 8365.

Basic MP-BGP EVPN configuration

The MP-BGP EVPN feature builds on the CLI commands used for configuring VXLAN using a VXLAN tunnel endpoint (VTEP). See General VXLAN configuration and topologies for more details.

After configuring VXLAN using a VTEP, the following CLI commands are configured to enable MP-BGP EVPN on each VTEP.

To configure MP-BGP EVPN on each VTEP:
  1. Configure the EVPN settings:

    config system evpn
        edit <id>
            set rd {AA | AA:NN | A.B.C.D:NN}
            set import-rt <AA:NN>
            set export-rt <AA:NN>
            set ip-local-learning {enable | disable}
            set arp-suppression {enable | disable}
        next
    end

    The ip-local-learning setting is used to enable/disable monitoring the local ARP table of the switch interface to learn the IP/MAC bindings, and advertise them to neighbors. This setting is disabled by default, but must be enabled when configuring MP-BGP EVPN.

    The arp-suppression setting is used to enable/disable using proxy ARP to perform suppression of ARP discovery using the flood-and-learn approach. This setting is disabled by default. When enabled, proxy ARP entries are added on the switch interface to suppress the ARP flooding of known IP/MAC bindings, which were learned by the MP-BGP EVPN control plane.

  2. Configure the EVPN settings within the VXLAN settings:

    config system vxlan
        edit <name>
            set interface <string>
            set vni <integer>
            set evpn-id <integer>
            set learn-from-traffic {enable | disable}
        next
    end

    The learn-from-traffic setting is used to enable/disable learning of remote VNIs from VXLAN traffic. This setting is disabled by default, and should only be enabled when local and all remote peers are using same VNI value, and some of the peers do not have MP-BGP EVPN capability.

  3. Configure the BGP settings:

    config router bgp
        set ibgp-multipath {enable | disable}
        set recursive-next-hop {enable | disable}
        set graceful-restart {enable | disable}
        config neighbor
            edit <WAN_IP_of_other_VTEP>
                set ebgp-enforce-multihop {enable | disable}
                set next-hop-self {enable | disable}
                set next-hop-self-vpnv4 {enable | disable}
                set soft-reconfiguration {enable | disable}
                set soft-reconfiguration-evpn {enable | disable}
                set remote-as <AS_number>
            next
        end
    end
  4. Configure the EVPN setting within the HA settings:

    config system ha
        set evpn-ttl <integer>
    end

Example

In this example, two FortiGates are configured as VXLAN tunnel endpoints (VTEPs). A VXLAN is configured to allow L2 connectivity between the networks behind each FortiGate. The VXLAN interface vxlan1 and port2 are placed on the same L2 network using a software switch (sw1). An L2 network is formed between PC1 and PC2. MP-BGP EVPN is used as the control plane to learn and distribute MAC address information within a single L2 domain identified using a specific VNI.

The VTEPs have the following MAC address tables:

Interface/endpoint

VTEP1

VTEP2

vxlan1

82:51:d1:44:bf:93

d2:21:00:c9:e6:98

port2

50:00:00:03:00:01

50:00:00:04:00:01

sw1

50:00:00:03:00:01

50:00:00:04:00:01

The MAC address of PC1 is 00:50:00:00:06:00. The MAC address of PC2 is 00:50:00:00:07:00.

This example assumes that the WAN interface and default route settings have already been configured on the VTEP 1 and VTEP 2 FortiGates. These configurations are omitted from the example. All peers are configured for MP-BGP EVPN.

To configure the VTEP1 FortiGate:
  1. Configure the loopback interface:

    config system interface
        edit "loopback1"
            set vdom "root"
            set ip 1.1.1.1 255.255.255.255
            set allowaccess ping https ssh http
            set type loopback
        next
    end
  2. Configure the EVPN settings:

    config system evpn
        edit 100
            set rd "100:100"
            set import-rt "1:1"
            set export-rt "1:1"
            set ip-local-learning enable
            set arp-suppression enable
        next
    end
  3. Configure the local interface and EVPN settings within the VXLAN settings:

    config system vxlan
        edit "vxlan1"
            set interface "loopback1"
            set vni 1000
            set evpn-id 100
        next
    end
  4. Configure the EVPN settings within the BGP settings:

    config router bgp
        set as 65001
        set router-id 1.1.1.1
        set ibgp-multipath enable
        set recursive-next-hop enable
        set graceful-restart enable
        config neighbor
            edit "172.25.160.101"
                set ebgp-enforce-multihop enable
                set next-hop-self enable
                set next-hop-self-vpnv4 enable
                set soft-reconfiguration enable
                set soft-reconfiguration-evpn enable 
                set remote-as 65001
            next
        end
        config network
            edit 1
                set prefix 1.1.1.1 255.255.255.255
            next
        end
    end

    172.27.16.237 is the WAN IP address of the VTEP2 FortiGate.

  5. Configure the software switch:

    config system switch-interface
        edit "sw1"
            set vdom "root"
            set member "port2" "vxlan1"
            set intra-switch-policy explicit
        next
    end
  6. Configure the software switch interface settings:

    config system interface
        edit "sw1"
            set vdom "root"
            set ip 172.18.1.253 255.255.255.0
            set allowaccess ping
            set type switch
        next
    end
  7. Configure the firewall policies between the member interfaces in the software switch:

    config firewall policy
         edit 1
             set srcintf "port2"
             set dstintf "vxlan1"
             set action accept
             set srcaddr "all"
             set dstaddr "all"
             set schedule "always"
             set service "ALL"
         next
         edit 2
             set srcintf "vxlan1"
             set dstintf "port2"
             set action accept
             set srcaddr "all"
             set dstaddr "all"
             set schedule "always"
             set service "ALL"
         next	 
    end
To configure the VTEP2 FortiGate:
  1. Configure the loopback interface:

    config system interface
        edit "loopback2"
            set vdom "root"
            set ip 2.2.2.2 255.255.255.255
            set allowaccess ping https ssh http
            set type loopback
        next
    end
  2. Configure the EVPN settings:

    config system evpn
        edit 100
            set rd "100:100"
            set import-rt "1:1"
            set export-rt "1:1"
            set ip-local-learning enable
            set arp-suppression enable
        next
    end
  3. Configure the local interface and EVPN settings within the VXLAN settings:

    config system vxlan
        edit "vxlan1"
            set interface "loopback2"
            set vni 1000
            set evpn-id 100
        next
    end
  4. Configure the EVPN settings within the BGP settings:

    config router bgp
        set as 65001
        set router-id 2.2.2.2
        set ibgp-multipath enable
        set recursive-next-hop enable
        set graceful-restart enable
        config neighbor
            edit "172.25.160.100" 
                set ebgp-enforce-multihop enable
                set next-hop-self enable
                set next-hop-self-vpnv4 enable
                set soft-reconfiguration enable
                set soft-reconfiguration-evpn enable 
                set remote-as 65001
            next
        end
        config network
            edit 1
                set prefix 2.2.2.2 255.255.255.255
            next
        end
    end

    172.27.16.236 is the WAN IP address of the VTEP1 FortiGate.

  5. Configure the software switch:

    config system switch-interface
        edit "sw1"
            set vdom "root"
            set member "port2" "vxlan1"
            set intra-switch-policy explicit
        next
    end
  6. Configure the software switch interface settings:

    config system interface
        edit "sw1"
            set vdom "root"
            set ip 172.18.1.254 255.255.255.0
            set allowaccess ping
            set type switch
        next
    end
  7. Configure the firewall policies between the member interfaces in the software switch:

    config firewall policy
         edit 1
             set srcintf "port2"
             set dstintf "vxlan1"
             set action accept
             set srcaddr "all"
             set dstaddr "all"
             set schedule "always"
             set service "ALL"
         next
         edit 2
             set srcintf "vxlan1"
             set dstintf "port2"
             set action accept
             set srcaddr "all"
             set dstaddr "all"
             set schedule "always"
             set service "ALL"
         next
    end
To verify the MP-BGP EVPN status on the VTEP1 FortiGate:
  1. From a host computer with IP address 172.18.1.11, perform the following.

    1. Check the ARP cache:

      # arp
      Address                  HWtype  HWaddress           Flags Mask            Iface
      172.18.1.253             ether   50:00:00:03:00:01   C                     ens3
    2. Ping the host computer with IP address 172.18.1.33:

      # ping 172.18.1.33 -c 4
      PING 172.18.1.33 (172.18.1.33) 56(84) bytes of data.
      64 bytes from 172.18.1.33: icmp_seq=1 ttl=64 time=1325 ms
      64 bytes from 172.18.1.33: icmp_seq=2 ttl=64 time=319 ms
      64 bytes from 172.18.1.33: icmp_seq=3 ttl=64 time=3.96 ms
      64 bytes from 172.18.1.33: icmp_seq=4 ttl=64 time=1.66 ms
      
      --- 172.18.1.33 ping statistics ---
      4 packets transmitted, 4 received, 0% packet loss, time 3007ms
      rtt min/avg/max/mdev = 1.660/412.614/1325.209/542.530 ms
    3. Check the ARP cache again:

      # arp
      Address                  HWtype  HWaddress           Flags Mask            Iface
      172.18.1.33              ether   00:50:00:00:07:00   C                     ens3
      172.18.1.253             ether   50:00:00:03:00:01   C                     ens3
  2. On the VTEP1 FortiGate, run the switch and VXLAN debug commands.

    1. Verify the forwarding database for vxlan1:

      # diagnose sys vxlan fdb list vxlan1
      mac=00:00:00:00:00:00 state=0x0082 remote_ip=2.2.2.2 port=4789 vni=1000 ifindex0
      mac=00:50:00:00:07:00 state=0x0082 remote_ip=2.2.2.2 port=4789 vni=1000 ifindex0
      
      total fdb num: 2
    2. Verify the forwarding database statistics for vxlan1:

      # diagnose sys vxlan fdb stat vxlan1
      fdb_table_size=256 fdb_table_used=2 fdb_entry=2 fdb_max_depth=1 cleanup_idx=0 c2
    3. Verify the bridging information for sw1:

      # diagnose netlink brctl name host sw1
      show bridge control interface sw1 host.
      fdb: hash size=32768, used=5, num=5, depth=1, gc_time=4, ageing_time=3, arp-sups
      Bridge sw1 host table
      port no device  devname mac addr                ttl     attributes
        2     15      vxlan1  00:00:00:00:00:00       28       Hit(28)
        2     15      vxlan1  00:50:00:00:07:00       18       Hit(18)
        2     15      vxlan1  82:51:d1:44:bf:93       0       Local Static
        1     4       port2   00:50:00:00:06:00       14       Hit(14)
        1     4       port2   50:00:00:03:00:01       0       Local Static 
      
  3. Run the BGP EVPN commands and observe the route type 2 (MAC/IP advertisement route) and route type 3 (inclusive multicast Ethernet tag route).

    1. Verify the BGP L2 VPN EVPN summary information:

      # get router info bgp evpn summary
      
      VRF 0 BGP router identifier 1.1.1.1, local AS number 65001
      BGP table version is 2
      1 BGP AS-PATH entries
      0 BGP community entries
      
      Neighbor       V         AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/Pd
      172.25.160.101 4      65001       9       9        1    0    0 00:04:02        3
      
      Total number of neighbors 1
    2. Verify the BGP L2 VPN EVPN network information:

      # get router info bgp evpn network
         Network          Next Hop            Metric     LocPrf Weight RouteTag Path
      Route Distinguisher: 100:100 (Default for VRF 0)
      *> [2][0][48][00:50:00:00:06:00][0]/72
                          1.1.1.1             0             100  32768        0 i <-/>
      *> [2][0][48][00:50:00:00:06:00][32][172.18.1.11]/104
                          1.1.1.1             0             100  32768        0 i <-/>
      *>i[2][0][48][00:50:00:00:07:00][0]/72
                          2.2.2.2             0             100      0        0 i <-/>
      *>i[2][0][48][00:50:00:00:07:00][32][172.18.1.33]/104
                          2.2.2.2             0             100      0        0 i <-/>
      *> [3][0][32][1.1.1.1]/80
                          1.1.1.1             0             100  32768        0 i <-/>
      *>i[3][0][32][2.2.2.2]/80
                          2.2.2.2             0             100      0        0 i <-/>
      
         Network          Next Hop            Metric     LocPrf Weight RouteTag Path
      Route Distinguisher: 100:100 (received from VRF 0)
      *>i[2][0][48][00:50:00:00:07:00][0]/72
                          2.2.2.2             0             100      0        0 i <-/>
      *>i[2][0][48][00:50:00:00:07:00][32][172.18.1.33]/104
                          2.2.2.2             0             100      0        0 i <-/>
      *>i[3][0][32][2.2.2.2]/80
                          2.2.2.2             0             100      0        0 i <-/> 
      
    3. Verify the BGP L2 VPN EVPN context:

      # get router info bgp evpn context
      L2VPN EVPN context for VRF 0
      ID 100 vlan-based, RD is [100:100]
       Import RT: RT:1:1
       Export RT: RT:1:1
        Bridge domain 0 VNI 1000
         Encapsulation 8(VXLAN)
         Source interface loopback1
         Source address 1.1.1.1
    4. Verify the BGP L2 VPN EVPN information for VRF 0:

      # get router info bgp evpn vrf 0
         Network          Next Hop            Metric     LocPrf Weight RouteTag Path
      Route Distinguisher: 100:100 (Default for VRF 0)
      *> [2][0][48][00:50:00:00:06:00][0]/72
                          1.1.1.1             0             100  32768        0 i <-/>
      *> [2][0][48][00:50:00:00:06:00][32][172.18.1.11]/104
                          1.1.1.1             0             100  32768        0 i <-/>
      *>i[2][0][48][00:50:00:00:07:00][0]/72
                          2.2.2.2             0             100      0        0 i <-/>
      *>i[2][0][48][00:50:00:00:07:00][32][172.18.1.33]/104
                          2.2.2.2             0             100      0        0 i <-/>
      *> [3][0][32][1.1.1.1]/80
                          1.1.1.1             0             100  32768        0 i <-/>
      *>i[3][0][32][2.2.2.2]/80
                          2.2.2.2             0             100      0        0 i <-/>
      
         Network          Next Hop            Metric     LocPrf Weight RouteTag Path
      Route Distinguisher: 100:100 (received from VRF 0)
      *>i[2][0][48][00:50:00:00:07:00][0]/72
                          2.2.2.2             0             100      0        0 i <-/>
      *>i[2][0][48][00:50:00:00:07:00][32][172.18.1.33]/104
                          2.2.2.2             0             100      0        0 i <-/>
      *>i[3][0][32][2.2.2.2]/80
                          2.2.2.2             0             100      0        0 i <-/> 
      
    5. Verify the BGP L2 VPN EVPN information for RD 100:100:

      # get router info bgp evpn rd 100:100
         Network          Next Hop            Metric     LocPrf Weight RouteTag Path
      Route Distinguisher: 100:100 (Default for VRF 0)
      *> [2][0][48][00:50:00:00:06:00][0]/72
                          1.1.1.1             0             100  32768        0 i <-/>
      *> [2][0][48][00:50:00:00:06:00][32][172.18.1.11]/104
                          1.1.1.1             0             100  32768        0 i <-/>
      *>i[2][0][48][00:50:00:00:07:00][0]/72
                          2.2.2.2             0             100      0        0 i <-/>
      *>i[2][0][48][00:50:00:00:07:00][32][172.18.1.33]/104
                          2.2.2.2             0             100      0        0 i <-/>
      *> [3][0][32][1.1.1.1]/80
                          1.1.1.1             0             100  32768        0 i <-/>
      *>i[3][0][32][2.2.2.2]/80
                          2.2.2.2             0             100      0        0 i <-/>
      
         Network          Next Hop            Metric     LocPrf Weight RouteTag Path
      Route Distinguisher: 100:100 (received from VRF 0)
      *>i[2][0][48][00:50:00:00:07:00][0]/72
                          2.2.2.2             0             100      0        0 i <-/>
      *>i[2][0][48][00:50:00:00:07:00][32][172.18.1.33]/104
                          2.2.2.2             0             100      0        0 i <-/>
      *>i[3][0][32][2.2.2.2]/80
                          2.2.2.2             0             100      0        0 i <-/> 
      
    6. Verify the neighbor EVPN advertised routes for 172.25.160.101:

      # get router info bgp neighbors 172.25.160.101 advertised-routes evpn
      Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
                    S Stale
      Origin codes: i - IGP, e - EGP, ? - incomplete
      
         Network          Next Hop            Metric LocPrf Weight RouteTag Path
      Route Distinguisher: 100:100 (Default for VRF 0) (Default for VRF 0)
      *>i[2][0][48][00:50:00:00:06:00][0]/72
                          1.1.1.1                           100  32768        0 i <-/>
      *>i[2][0][48][00:50:00:00:06:00][32][172.18.1.11]/104
                          1.1.1.1                           100  32768        0 i <-/>
      *>i[3][0][32][1.1.1.1]/80
                          1.1.1.1                           100  32768        0 i <-/>
      
      Total number of prefixes 3
    7. Verify the neighbor EVPN received routes for 172.25.160.101:

      # get router info bgp neighbors 172.25.160.101 received-routes evpn
      Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
                    S Stale
      Origin codes: i - IGP, e - EGP, ? - incomplete
      
         Network          Next Hop            Metric LocPrf Weight RouteTag Path
      Route Distinguisher: 100:100 (received from VRF 0) (received from VRF 0)
      *>i[2][0][48][00:50:00:00:07:00][0]/72
                          2.2.2.2                           100      0        0 i <-/>
      *>i[2][0][48][00:50:00:00:07:00][32][172.18.1.33]/104
                          2.2.2.2                           100      0        0 i <-/>
      *>i[3][0][32][2.2.2.2]/80
                          2.2.2.2                           100      0        0 i <-/>
      
      Total number of prefixes 3 
      
    8. Verify the neighbor EVPN routes:

      # get router info bgp neighbors 172.25.160.101 routes evpn
      Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
                    S Stale
      Origin codes: i - IGP, e - EGP, ? - incomplete
      
         Network          Next Hop            Metric LocPrf Weight RouteTag Path
      Route Distinguisher: 100:100 (Default for VRF 0) (Default for VRF 0)
      *>i[2][0][48][00:50:00:00:07:00][0]/72
                          2.2.2.2             0             100      0        0 i <-/>
      *>i[2][0][48][00:50:00:00:07:00][32][172.18.1.33]/104
                          2.2.2.2             0             100      0        0 i <-/>
      *>i[3][0][32][2.2.2.2]/80
                          2.2.2.2             0             100      0        0 i <-/>
      Route Distinguisher: 100:100 (received from VRF 0) (received from VRF 0)
      *>i[2][0][48][00:50:00:00:07:00][0]/72
                          2.2.2.2             0             100      0        0 i <-/>
      *>i[2][0][48][00:50:00:00:07:00][32][172.18.1.33]/104
                          2.2.2.2             0             100      0        0 i <-/>
      *>i[3][0][32][2.2.2.2]/80
                          2.2.2.2             0             100      0        0 i <-/>
      
      Total number of prefixes 6 
      
  4. Run the following EVPN get commands.

    1. Verify the EVPN instances:

      # get l2vpn evpn instance
      EVPN instance: 100
      IP local learning enabled
      ARP suppression enabled
      HA primary
         Number of bridge domain: 1
         Bridge domain: TAGID 0 VNI 1000 ADDR 1.1.1.1 VXLAN vxlan1 SWITCH sw1
    2. Verify the EVPN table:

      # get l2vpn evpn table
      EVPN instance 100
       Broadcast domain VNI 1000 TAGID 0
      
        EVPN instance 100
       Broadcast domain VNI 1000 TAGID 0
      
        EVPN MAC table:
        MAC               VNI      Remote Addr     Binded Address
        00:50:00:00:07:00 1000     2.2.2.2         172.18.1.33
                          1000     2.2.2.2         -
      
        EVPN IP table:
        Address         VNI      Remote Addr     MAC
        172.18.1.33     1000     2.2.2.2         00:50:00:00:07:00
      
        EVPN Local MAC table:
        "Inactive" means this MAC/IP pair will not be sent to peer.
        Flag code: S - Static F - FDB. Trailing * means HA
        MAC               Flag Status   Binded Address
        00:50:00:00:06:00      Active   172.18.1.11
                          F    Active   -
      
        EVPN Local IP table:
        Address         MAC
        172.18.1.11     00:50:00:00:06:00
      
        EVPN PEER table:
        VNI      Remote Addr     Binded Address
        1000     2.2.2.2         2.2.2.2 
      
  5. Run the proxy ARP diagnose command:

    # diagnose ip parp list
    Address           Hardware Addr     Interface
    172.18.1.33       00:50:00:00:07:00 sw1
    

VXLAN with MP-BGP EVPN

FortiOS supports VXLAN as implemented according to RFC 7348. Currently, VXLAN relies on determining the MAC address of the destination host by using address resolution protocol (ARP) broadcast frames encapsulated in multicast packets.

  • A multicast group is maintained with all the VXLAN tunnel endpoints (VTEPs) associated with the same VXLAN, namely, with the same VXLAN network identifier (VNI).

  • The multicast packets that encapsulate ARP broadcast frames are sent to this multicast group, and then the destination host replies to the source host using unicast IP packet encapsulated using VXLAN.

  • The source and destination FortiGates as VTEPs each maintain a mapping of MAC addresses to remote VTEPs.

As with non-VXLAN traffic, VXLAN relies on the preceding ARP process, commonly known as flood-and-learn that floods the network with broadcast frames encapsulated as multicast packets to learn MAC addresses. In the RFC 7348 implementation of VXLAN, the data plane is simultaneously used as a control plane.

The following topology demonstrates how flood-and-learn uses ARP broadcast traffic flooded throughout the VXLAN for PC A to learn PC D's MAC address when PC A tries to connect to PC D.

Multiprotocol Border Gateway Protocol Ethernet Virtual Private Network (MP-BGP EVPN) support for VXLAN allows for learning MAC addresses in a way that is more suitable for large deployments than flood-and-learn.

MP-BGP EVPN is a standards-based control plane that supports the distribution of attached host MAC and IP addresses using MP-BGP, namely, using the EVPN address family and MAC addresses treated as routing entries in BGP. As a control plane that is separate from the data plane, MP-BGP EVPN avoids flood-and-learn in the network, and the wide use of BGP as an external gateway protocol on the internet proves its ability to scale well with large deployments. The following topology demonstrates how MP-BGP EVPN distributes route type 2 MAC/IP advertisement routes among VTEPs in the VXLAN, and minimizes ARP broadcast traffic required for PC A to learn PC D's MAC address when PC A tries to connect to PC D.

MP-BGP EVPN supports the following features:

  • Route type 2 (MAC/IP advertisement route) and route type 3 (inclusive multicast Ethernet tag route)

  • Intra-subnet communication

  • Single-homing use cases

  • VLAN-based service, namely, there is only one broadcast domain per EVPN instance (EVI). This is due to the current VXLAN design that supports a single VNI for a VXLAN interface.

  • EVPN running on IPv4 unicast VXLAN

  • Egress replication for broadcast, unknown unicast, and multicast (BUM) traffic

  • VXLAN MAC learning from traffic

  • IP address local learning

  • ARP suppression

Note

For more information about MP-BGP EVPN, see RFC 7432. For more information about EVPN and VXLAN, see RFC 8365.

Basic MP-BGP EVPN configuration

The MP-BGP EVPN feature builds on the CLI commands used for configuring VXLAN using a VXLAN tunnel endpoint (VTEP). See General VXLAN configuration and topologies for more details.

After configuring VXLAN using a VTEP, the following CLI commands are configured to enable MP-BGP EVPN on each VTEP.

To configure MP-BGP EVPN on each VTEP:
  1. Configure the EVPN settings:

    config system evpn
        edit <id>
            set rd {AA | AA:NN | A.B.C.D:NN}
            set import-rt <AA:NN>
            set export-rt <AA:NN>
            set ip-local-learning {enable | disable}
            set arp-suppression {enable | disable}
        next
    end

    The ip-local-learning setting is used to enable/disable monitoring the local ARP table of the switch interface to learn the IP/MAC bindings, and advertise them to neighbors. This setting is disabled by default, but must be enabled when configuring MP-BGP EVPN.

    The arp-suppression setting is used to enable/disable using proxy ARP to perform suppression of ARP discovery using the flood-and-learn approach. This setting is disabled by default. When enabled, proxy ARP entries are added on the switch interface to suppress the ARP flooding of known IP/MAC bindings, which were learned by the MP-BGP EVPN control plane.

  2. Configure the EVPN settings within the VXLAN settings:

    config system vxlan
        edit <name>
            set interface <string>
            set vni <integer>
            set evpn-id <integer>
            set learn-from-traffic {enable | disable}
        next
    end

    The learn-from-traffic setting is used to enable/disable learning of remote VNIs from VXLAN traffic. This setting is disabled by default, and should only be enabled when local and all remote peers are using same VNI value, and some of the peers do not have MP-BGP EVPN capability.

  3. Configure the BGP settings:

    config router bgp
        set ibgp-multipath {enable | disable}
        set recursive-next-hop {enable | disable}
        set graceful-restart {enable | disable}
        config neighbor
            edit <WAN_IP_of_other_VTEP>
                set ebgp-enforce-multihop {enable | disable}
                set next-hop-self {enable | disable}
                set next-hop-self-vpnv4 {enable | disable}
                set soft-reconfiguration {enable | disable}
                set soft-reconfiguration-evpn {enable | disable}
                set remote-as <AS_number>
            next
        end
    end
  4. Configure the EVPN setting within the HA settings:

    config system ha
        set evpn-ttl <integer>
    end

Example

In this example, two FortiGates are configured as VXLAN tunnel endpoints (VTEPs). A VXLAN is configured to allow L2 connectivity between the networks behind each FortiGate. The VXLAN interface vxlan1 and port2 are placed on the same L2 network using a software switch (sw1). An L2 network is formed between PC1 and PC2. MP-BGP EVPN is used as the control plane to learn and distribute MAC address information within a single L2 domain identified using a specific VNI.

The VTEPs have the following MAC address tables:

Interface/endpoint

VTEP1

VTEP2

vxlan1

82:51:d1:44:bf:93

d2:21:00:c9:e6:98

port2

50:00:00:03:00:01

50:00:00:04:00:01

sw1

50:00:00:03:00:01

50:00:00:04:00:01

The MAC address of PC1 is 00:50:00:00:06:00. The MAC address of PC2 is 00:50:00:00:07:00.

This example assumes that the WAN interface and default route settings have already been configured on the VTEP 1 and VTEP 2 FortiGates. These configurations are omitted from the example. All peers are configured for MP-BGP EVPN.

To configure the VTEP1 FortiGate:
  1. Configure the loopback interface:

    config system interface
        edit "loopback1"
            set vdom "root"
            set ip 1.1.1.1 255.255.255.255
            set allowaccess ping https ssh http
            set type loopback
        next
    end
  2. Configure the EVPN settings:

    config system evpn
        edit 100
            set rd "100:100"
            set import-rt "1:1"
            set export-rt "1:1"
            set ip-local-learning enable
            set arp-suppression enable
        next
    end
  3. Configure the local interface and EVPN settings within the VXLAN settings:

    config system vxlan
        edit "vxlan1"
            set interface "loopback1"
            set vni 1000
            set evpn-id 100
        next
    end
  4. Configure the EVPN settings within the BGP settings:

    config router bgp
        set as 65001
        set router-id 1.1.1.1
        set ibgp-multipath enable
        set recursive-next-hop enable
        set graceful-restart enable
        config neighbor
            edit "172.25.160.101"
                set ebgp-enforce-multihop enable
                set next-hop-self enable
                set next-hop-self-vpnv4 enable
                set soft-reconfiguration enable
                set soft-reconfiguration-evpn enable 
                set remote-as 65001
            next
        end
        config network
            edit 1
                set prefix 1.1.1.1 255.255.255.255
            next
        end
    end

    172.27.16.237 is the WAN IP address of the VTEP2 FortiGate.

  5. Configure the software switch:

    config system switch-interface
        edit "sw1"
            set vdom "root"
            set member "port2" "vxlan1"
            set intra-switch-policy explicit
        next
    end
  6. Configure the software switch interface settings:

    config system interface
        edit "sw1"
            set vdom "root"
            set ip 172.18.1.253 255.255.255.0
            set allowaccess ping
            set type switch
        next
    end
  7. Configure the firewall policies between the member interfaces in the software switch:

    config firewall policy
         edit 1
             set srcintf "port2"
             set dstintf "vxlan1"
             set action accept
             set srcaddr "all"
             set dstaddr "all"
             set schedule "always"
             set service "ALL"
         next
         edit 2
             set srcintf "vxlan1"
             set dstintf "port2"
             set action accept
             set srcaddr "all"
             set dstaddr "all"
             set schedule "always"
             set service "ALL"
         next	 
    end
To configure the VTEP2 FortiGate:
  1. Configure the loopback interface:

    config system interface
        edit "loopback2"
            set vdom "root"
            set ip 2.2.2.2 255.255.255.255
            set allowaccess ping https ssh http
            set type loopback
        next
    end
  2. Configure the EVPN settings:

    config system evpn
        edit 100
            set rd "100:100"
            set import-rt "1:1"
            set export-rt "1:1"
            set ip-local-learning enable
            set arp-suppression enable
        next
    end
  3. Configure the local interface and EVPN settings within the VXLAN settings:

    config system vxlan
        edit "vxlan1"
            set interface "loopback2"
            set vni 1000
            set evpn-id 100
        next
    end
  4. Configure the EVPN settings within the BGP settings:

    config router bgp
        set as 65001
        set router-id 2.2.2.2
        set ibgp-multipath enable
        set recursive-next-hop enable
        set graceful-restart enable
        config neighbor
            edit "172.25.160.100" 
                set ebgp-enforce-multihop enable
                set next-hop-self enable
                set next-hop-self-vpnv4 enable
                set soft-reconfiguration enable
                set soft-reconfiguration-evpn enable 
                set remote-as 65001
            next
        end
        config network
            edit 1
                set prefix 2.2.2.2 255.255.255.255
            next
        end
    end

    172.27.16.236 is the WAN IP address of the VTEP1 FortiGate.

  5. Configure the software switch:

    config system switch-interface
        edit "sw1"
            set vdom "root"
            set member "port2" "vxlan1"
            set intra-switch-policy explicit
        next
    end
  6. Configure the software switch interface settings:

    config system interface
        edit "sw1"
            set vdom "root"
            set ip 172.18.1.254 255.255.255.0
            set allowaccess ping
            set type switch
        next
    end
  7. Configure the firewall policies between the member interfaces in the software switch:

    config firewall policy
         edit 1
             set srcintf "port2"
             set dstintf "vxlan1"
             set action accept
             set srcaddr "all"
             set dstaddr "all"
             set schedule "always"
             set service "ALL"
         next
         edit 2
             set srcintf "vxlan1"
             set dstintf "port2"
             set action accept
             set srcaddr "all"
             set dstaddr "all"
             set schedule "always"
             set service "ALL"
         next
    end
To verify the MP-BGP EVPN status on the VTEP1 FortiGate:
  1. From a host computer with IP address 172.18.1.11, perform the following.

    1. Check the ARP cache:

      # arp
      Address                  HWtype  HWaddress           Flags Mask            Iface
      172.18.1.253             ether   50:00:00:03:00:01   C                     ens3
    2. Ping the host computer with IP address 172.18.1.33:

      # ping 172.18.1.33 -c 4
      PING 172.18.1.33 (172.18.1.33) 56(84) bytes of data.
      64 bytes from 172.18.1.33: icmp_seq=1 ttl=64 time=1325 ms
      64 bytes from 172.18.1.33: icmp_seq=2 ttl=64 time=319 ms
      64 bytes from 172.18.1.33: icmp_seq=3 ttl=64 time=3.96 ms
      64 bytes from 172.18.1.33: icmp_seq=4 ttl=64 time=1.66 ms
      
      --- 172.18.1.33 ping statistics ---
      4 packets transmitted, 4 received, 0% packet loss, time 3007ms
      rtt min/avg/max/mdev = 1.660/412.614/1325.209/542.530 ms
    3. Check the ARP cache again:

      # arp
      Address                  HWtype  HWaddress           Flags Mask            Iface
      172.18.1.33              ether   00:50:00:00:07:00   C                     ens3
      172.18.1.253             ether   50:00:00:03:00:01   C                     ens3
  2. On the VTEP1 FortiGate, run the switch and VXLAN debug commands.

    1. Verify the forwarding database for vxlan1:

      # diagnose sys vxlan fdb list vxlan1
      mac=00:00:00:00:00:00 state=0x0082 remote_ip=2.2.2.2 port=4789 vni=1000 ifindex0
      mac=00:50:00:00:07:00 state=0x0082 remote_ip=2.2.2.2 port=4789 vni=1000 ifindex0
      
      total fdb num: 2
    2. Verify the forwarding database statistics for vxlan1:

      # diagnose sys vxlan fdb stat vxlan1
      fdb_table_size=256 fdb_table_used=2 fdb_entry=2 fdb_max_depth=1 cleanup_idx=0 c2
    3. Verify the bridging information for sw1:

      # diagnose netlink brctl name host sw1
      show bridge control interface sw1 host.
      fdb: hash size=32768, used=5, num=5, depth=1, gc_time=4, ageing_time=3, arp-sups
      Bridge sw1 host table
      port no device  devname mac addr                ttl     attributes
        2     15      vxlan1  00:00:00:00:00:00       28       Hit(28)
        2     15      vxlan1  00:50:00:00:07:00       18       Hit(18)
        2     15      vxlan1  82:51:d1:44:bf:93       0       Local Static
        1     4       port2   00:50:00:00:06:00       14       Hit(14)
        1     4       port2   50:00:00:03:00:01       0       Local Static 
      
  3. Run the BGP EVPN commands and observe the route type 2 (MAC/IP advertisement route) and route type 3 (inclusive multicast Ethernet tag route).

    1. Verify the BGP L2 VPN EVPN summary information:

      # get router info bgp evpn summary
      
      VRF 0 BGP router identifier 1.1.1.1, local AS number 65001
      BGP table version is 2
      1 BGP AS-PATH entries
      0 BGP community entries
      
      Neighbor       V         AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/Pd
      172.25.160.101 4      65001       9       9        1    0    0 00:04:02        3
      
      Total number of neighbors 1
    2. Verify the BGP L2 VPN EVPN network information:

      # get router info bgp evpn network
         Network          Next Hop            Metric     LocPrf Weight RouteTag Path
      Route Distinguisher: 100:100 (Default for VRF 0)
      *> [2][0][48][00:50:00:00:06:00][0]/72
                          1.1.1.1             0             100  32768        0 i <-/>
      *> [2][0][48][00:50:00:00:06:00][32][172.18.1.11]/104
                          1.1.1.1             0             100  32768        0 i <-/>
      *>i[2][0][48][00:50:00:00:07:00][0]/72
                          2.2.2.2             0             100      0        0 i <-/>
      *>i[2][0][48][00:50:00:00:07:00][32][172.18.1.33]/104
                          2.2.2.2             0             100      0        0 i <-/>
      *> [3][0][32][1.1.1.1]/80
                          1.1.1.1             0             100  32768        0 i <-/>
      *>i[3][0][32][2.2.2.2]/80
                          2.2.2.2             0             100      0        0 i <-/>
      
         Network          Next Hop            Metric     LocPrf Weight RouteTag Path
      Route Distinguisher: 100:100 (received from VRF 0)
      *>i[2][0][48][00:50:00:00:07:00][0]/72
                          2.2.2.2             0             100      0        0 i <-/>
      *>i[2][0][48][00:50:00:00:07:00][32][172.18.1.33]/104
                          2.2.2.2             0             100      0        0 i <-/>
      *>i[3][0][32][2.2.2.2]/80
                          2.2.2.2             0             100      0        0 i <-/> 
      
    3. Verify the BGP L2 VPN EVPN context:

      # get router info bgp evpn context
      L2VPN EVPN context for VRF 0
      ID 100 vlan-based, RD is [100:100]
       Import RT: RT:1:1
       Export RT: RT:1:1
        Bridge domain 0 VNI 1000
         Encapsulation 8(VXLAN)
         Source interface loopback1
         Source address 1.1.1.1
    4. Verify the BGP L2 VPN EVPN information for VRF 0:

      # get router info bgp evpn vrf 0
         Network          Next Hop            Metric     LocPrf Weight RouteTag Path
      Route Distinguisher: 100:100 (Default for VRF 0)
      *> [2][0][48][00:50:00:00:06:00][0]/72
                          1.1.1.1             0             100  32768        0 i <-/>
      *> [2][0][48][00:50:00:00:06:00][32][172.18.1.11]/104
                          1.1.1.1             0             100  32768        0 i <-/>
      *>i[2][0][48][00:50:00:00:07:00][0]/72
                          2.2.2.2             0             100      0        0 i <-/>
      *>i[2][0][48][00:50:00:00:07:00][32][172.18.1.33]/104
                          2.2.2.2             0             100      0        0 i <-/>
      *> [3][0][32][1.1.1.1]/80
                          1.1.1.1             0             100  32768        0 i <-/>
      *>i[3][0][32][2.2.2.2]/80
                          2.2.2.2             0             100      0        0 i <-/>
      
         Network          Next Hop            Metric     LocPrf Weight RouteTag Path
      Route Distinguisher: 100:100 (received from VRF 0)
      *>i[2][0][48][00:50:00:00:07:00][0]/72
                          2.2.2.2             0             100      0        0 i <-/>
      *>i[2][0][48][00:50:00:00:07:00][32][172.18.1.33]/104
                          2.2.2.2             0             100      0        0 i <-/>
      *>i[3][0][32][2.2.2.2]/80
                          2.2.2.2             0             100      0        0 i <-/> 
      
    5. Verify the BGP L2 VPN EVPN information for RD 100:100:

      # get router info bgp evpn rd 100:100
         Network          Next Hop            Metric     LocPrf Weight RouteTag Path
      Route Distinguisher: 100:100 (Default for VRF 0)
      *> [2][0][48][00:50:00:00:06:00][0]/72
                          1.1.1.1             0             100  32768        0 i <-/>
      *> [2][0][48][00:50:00:00:06:00][32][172.18.1.11]/104
                          1.1.1.1             0             100  32768        0 i <-/>
      *>i[2][0][48][00:50:00:00:07:00][0]/72
                          2.2.2.2             0             100      0        0 i <-/>
      *>i[2][0][48][00:50:00:00:07:00][32][172.18.1.33]/104
                          2.2.2.2             0             100      0        0 i <-/>
      *> [3][0][32][1.1.1.1]/80
                          1.1.1.1             0             100  32768        0 i <-/>
      *>i[3][0][32][2.2.2.2]/80
                          2.2.2.2             0             100      0        0 i <-/>
      
         Network          Next Hop            Metric     LocPrf Weight RouteTag Path
      Route Distinguisher: 100:100 (received from VRF 0)
      *>i[2][0][48][00:50:00:00:07:00][0]/72
                          2.2.2.2             0             100      0        0 i <-/>
      *>i[2][0][48][00:50:00:00:07:00][32][172.18.1.33]/104
                          2.2.2.2             0             100      0        0 i <-/>
      *>i[3][0][32][2.2.2.2]/80
                          2.2.2.2             0             100      0        0 i <-/> 
      
    6. Verify the neighbor EVPN advertised routes for 172.25.160.101:

      # get router info bgp neighbors 172.25.160.101 advertised-routes evpn
      Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
                    S Stale
      Origin codes: i - IGP, e - EGP, ? - incomplete
      
         Network          Next Hop            Metric LocPrf Weight RouteTag Path
      Route Distinguisher: 100:100 (Default for VRF 0) (Default for VRF 0)
      *>i[2][0][48][00:50:00:00:06:00][0]/72
                          1.1.1.1                           100  32768        0 i <-/>
      *>i[2][0][48][00:50:00:00:06:00][32][172.18.1.11]/104
                          1.1.1.1                           100  32768        0 i <-/>
      *>i[3][0][32][1.1.1.1]/80
                          1.1.1.1                           100  32768        0 i <-/>
      
      Total number of prefixes 3
    7. Verify the neighbor EVPN received routes for 172.25.160.101:

      # get router info bgp neighbors 172.25.160.101 received-routes evpn
      Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
                    S Stale
      Origin codes: i - IGP, e - EGP, ? - incomplete
      
         Network          Next Hop            Metric LocPrf Weight RouteTag Path
      Route Distinguisher: 100:100 (received from VRF 0) (received from VRF 0)
      *>i[2][0][48][00:50:00:00:07:00][0]/72
                          2.2.2.2                           100      0        0 i <-/>
      *>i[2][0][48][00:50:00:00:07:00][32][172.18.1.33]/104
                          2.2.2.2                           100      0        0 i <-/>
      *>i[3][0][32][2.2.2.2]/80
                          2.2.2.2                           100      0        0 i <-/>
      
      Total number of prefixes 3 
      
    8. Verify the neighbor EVPN routes:

      # get router info bgp neighbors 172.25.160.101 routes evpn
      Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
                    S Stale
      Origin codes: i - IGP, e - EGP, ? - incomplete
      
         Network          Next Hop            Metric LocPrf Weight RouteTag Path
      Route Distinguisher: 100:100 (Default for VRF 0) (Default for VRF 0)
      *>i[2][0][48][00:50:00:00:07:00][0]/72
                          2.2.2.2             0             100      0        0 i <-/>
      *>i[2][0][48][00:50:00:00:07:00][32][172.18.1.33]/104
                          2.2.2.2             0             100      0        0 i <-/>
      *>i[3][0][32][2.2.2.2]/80
                          2.2.2.2             0             100      0        0 i <-/>
      Route Distinguisher: 100:100 (received from VRF 0) (received from VRF 0)
      *>i[2][0][48][00:50:00:00:07:00][0]/72
                          2.2.2.2             0             100      0        0 i <-/>
      *>i[2][0][48][00:50:00:00:07:00][32][172.18.1.33]/104
                          2.2.2.2             0             100      0        0 i <-/>
      *>i[3][0][32][2.2.2.2]/80
                          2.2.2.2             0             100      0        0 i <-/>
      
      Total number of prefixes 6 
      
  4. Run the following EVPN get commands.

    1. Verify the EVPN instances:

      # get l2vpn evpn instance
      EVPN instance: 100
      IP local learning enabled
      ARP suppression enabled
      HA primary
         Number of bridge domain: 1
         Bridge domain: TAGID 0 VNI 1000 ADDR 1.1.1.1 VXLAN vxlan1 SWITCH sw1
    2. Verify the EVPN table:

      # get l2vpn evpn table
      EVPN instance 100
       Broadcast domain VNI 1000 TAGID 0
      
        EVPN instance 100
       Broadcast domain VNI 1000 TAGID 0
      
        EVPN MAC table:
        MAC               VNI      Remote Addr     Binded Address
        00:50:00:00:07:00 1000     2.2.2.2         172.18.1.33
                          1000     2.2.2.2         -
      
        EVPN IP table:
        Address         VNI      Remote Addr     MAC
        172.18.1.33     1000     2.2.2.2         00:50:00:00:07:00
      
        EVPN Local MAC table:
        "Inactive" means this MAC/IP pair will not be sent to peer.
        Flag code: S - Static F - FDB. Trailing * means HA
        MAC               Flag Status   Binded Address
        00:50:00:00:06:00      Active   172.18.1.11
                          F    Active   -
      
        EVPN Local IP table:
        Address         MAC
        172.18.1.11     00:50:00:00:06:00
      
        EVPN PEER table:
        VNI      Remote Addr     Binded Address
        1000     2.2.2.2         2.2.2.2 
      
  5. Run the proxy ARP diagnose command:

    # diagnose ip parp list
    Address           Hardware Addr     Interface
    172.18.1.33       00:50:00:00:07:00 sw1