Active dynamic BGP neighbor triggered by ADVPN shortcut

When a customer using SD-WAN with ADVPN has numerous IPv4 and IPv6 routes per spoke and there are many spokes in the topology, using ADVPN with a route reflector-based design poses the following challenges:

• The hub FortiGate will experience high CPU usage due to the amount of processing required to reflect the routes to the spoke FortiGates.
• Spoke FortiGates will learn many unnecessary routes.

For such cases, it is more suitable to deploy an IPv4- and IPv6-supported solution without a route-reflector that involves an active dynamic BGP neighbor triggered by an ADVPN shortcut. This solution allows a spoke FortiGate to form a BGP neighbor with another spoke FortiGate only after the shortcut tunnel between them has been established. As a result, the spoke only learns routes from its BGP neighbors.

How this solution differs from typical SD-WAN with ADVPN

In a topology where the Spoke 1 and Spoke 2 FortiGates are connected directly to the Hub FortiGate, route reflection will not be enabled. The Hub FortiGate is only configured with each spoke's summary route. An ADVPN shortcut tunnel is established between the Spoke 1 and Spoke 2 FortiGates. The valid routing between the Spoke 1 and Spoke 2 FortiGate is still through the Hub FortiGate at this point.

When a host behind Spoke 1 tries to connect to a host behind Spoke 2, Spoke 1 first reaches the Hub based on the valid routing table. The Hub determines that the destination is reachable, and the ADVPN shortcut tunnel between the spokes is established. Then, Spoke 1 and Spoke 2 will actively initiate a BGP connection to each other over the shortcut. Once established, they will exchange their routing information using BGP. On both spokes, BGP will resolve those routes on the shortcut and update the routing table accordingly.

For this solution, the following IPv4/IPv6 BGP configuration settings are required:

• The hub FortiGate should be configured with neighbor-group and neighbor-range/neighbor-range6.

• Each spoke FortiGate should be configured with neighbor-group and neighbor-range/neighbor-range6 like the hub. More importantly, each spoke should be configured with set passive disable to ensure spokes are able to initiate dynamic BGP connections between each other.

• The hub FortiGate should have route reflection disabled (by default) where each neighbor-group setting should have set route-reflector-client disable.

In the configuration, each of the spokes will form a BGP neighbor relationship with the hub. This is unchanged from the typical SD-WAN with ADVPN configuration.

Example

This example configuration contains the following structure:

• Use SD-WAN member 1 (via ISP1) and its dynamic shortcuts for Financial Department traffic.

• Use SD-WAN member 2 (via ISP2) and its dynamic shortcuts for Engineering Department traffic.

• Internal subnets of Spoke 1:

  • IPv4: 22.1.1.0/24

  • IPv6: 2001:22::/64

• Internal subnets of Spoke 2:

  • IPv4: 33.1.1.0/24

    • Financial Department: 33.1.1.1 to 33.1.1.100

    • Engineering Department: 33.1.1.101 to 33.1.1.200

  • IPv6: 2001:33::/64

    • Financial Department: 2001:33::1 to 2001:33::100

    • Engineering Department: 2001:33::101 to 2001:33::200

To configure the Hub FortiGate:

1. Configure the BGP settings (neighbor group and ranges):

   config router bgp
       set as 65100
       set router-id 10.200.1.1
       set ibgp-multipath enable
       config neighbor-group
           edit "EDGE"
               set activate6 disable
               set remote-as 65100
               set update-source "lo1"
               set route-reflector-client disable
           next
           edit "EDGEv6"
               set activate disable
               set remote-as 65100
               set update-source "lo1"
               set route-reflector-client disable
           next
       end
       config neighbor-range
           edit 2
               set prefix 10.200.1.0 255.255.255.0
               set neighbor-group "EDGE"
           next
       end
       config neighbor-range6
           edit 2
               set prefix6 2001::10:200:1:0/112
               set neighbor-group "EDGEv6"
           next
       end
       config network
           edit 2
               set prefix 10.200.1.0 255.255.255.0
           next
           edit 4
               set prefix 33.0.0.0 255.0.0.0
           next
           edit 5
               set prefix 22.0.0.0 255.0.0.0
           next
       end
       config network6
           edit 4
               set prefix6 2001:33::/32
           next
           edit 2
               set prefix6 2001:22::/32 
           next
       end
   end

2. Configure the static routes.

   1. For IPv4:

      config router static
          edit 33
              set dst 33.0.0.0 255.0.0.0
              set blackhole enable
              set vrf 0
          next
          edit 22
              set dst 22.0.0.0 255.0.0.0
              set blackhole enable
              set vrf 0
          next
      end

   2. For IPv6:

      config router static6
          edit 33
              set dst 2001:33::/32
              set blackhole enable
              set vrf 0
          next
          edit 22
              set dst 2001:22::/32
              set blackhole enable
              set vrf 0
          next
      end

The following IPv4 summary routes are advertised:

• 33.0.0.0/8

• 22.0.0.0/8

The following IPv6 summary routes are advertised:

• 2001:33::/32

• 2001:22::/32

Because route reflection has been disabled in this example, initially, Spoke 1 will not know the local subnet of Spoke 2, and Spoke 2 will not know the local subnet of Spoke 1. Therefore, for traffic routing, summary routes are configured on the hub as blackhole routes and then advertised to the spokes using BGP.

For example, for traffic from the local subnet of Spoke 2 destined for the local subnet of Spoke 1:

• For the IPv4 case, the summary route 22.0.0.0/8, which includes the local subnet of Spoke 1 (22.1.1.0/24), is advertised to Spoke 2. When Spoke 2 sends traffic destined for 22.1.1.0/24 to the Hub, the Hub forwards this traffic to Spoke 1 since they are BGP neighbors.
• For the IPv6 case, the summary route 2001:22::/32, which includes the local subnet of Spoke 1 (2001:22::/64), is advertised to Spoke 2. When Spoke 2 sends traffic destined for 2001:22::/64 to the Hub, the Hub forwards this traffic to Spoke 1 since they are BGP neighbors.

Although traffic from spoke-to-spoke goes through the hub first, it is expected that the spoke will eventually go through the shortcut tunnel.

To configure the Spoke 1 FortiGate:

1. Configure the SD-WAN settings:

   config system sdwan
       set status enable
       config zone
           edit "virtual-wan-link"
           next
       end
       config members
           edit 1
               set interface "spoke1-1"
               set cost 10
           next
           edit 2
               set interface "spoke-2"
               set cost 20
           next
       end
       config health-check
           edit "ping"
               set server "11.11.11.11"
               set source 10.200.1.100
               set members 1 2
               config sla
                   edit 1
                       set latency-threshold 200
                       set jitter-threshold 50
                   next
               end
           next
       end
       config service
           edit 1
               set dst "financial-department"
               set priority-members 1
           next
           edit 2
               set dst "engineering-department"
               set priority-members 2
           next
           edit 61
               set addr-mode ipv6
               set priority-members 1
               set dst6 "financial-department-IPv6"
           next
           edit 62
               set addr-mode ipv6
               set priority-members 2
               set dst6 "engineering-department-IPv6"
           next
       end
   end

2. Configure the BGP settings (neighbor group and ranges):

   config router bgp
       set as 65100
       set router-id 10.200.1.100
       set ibgp-multipath enable
       config neighbor
           edit "10.200.1.1"
               set activate6 disable
               set remote-as 65100
               set connect-timer 10
               set update-source "lo1"
           next
           edit "2001::10:200:1:1"
               set advertisement-interval 1
               set activate disable
               set remote-as 65100
               set update-source "lo1"
           next
       end
       config neighbor-group
           edit "spokes"
               set activate6 disable
               set passive disable 
               set remote-as 65100
               set update-source "lo1"
           next
           edit "spokesv6"
               set activate disable
               set passive disable 
               set remote-as 65100
               set update-source "lo1"
           next
       end
       config neighbor-range
           edit 1
               set prefix 10.200.1.0 255.255.255.0
               set neighbor-group "spokes"
           next
       end
       config neighbor-range6
           edit 1
               set prefix6 2001::10:200:1:0/112
               set neighbor-group "spokesv6"
           next
       end
       config network
           edit 3
               set prefix 22.1.1.0 255.255.255.0
           next
       end
       config network6
           edit 1
               set prefix6 2001:22::/64 
           next
       end 
   end

Verifying the configuration before a spoke-to-spoke shortcut VPN is established

IPv4 use case

To verify the status on Spoke 1:

1. Verify the BGP status:

   # get router info bgp summary
   VRF 0 BGP router identifier 10.200.1.100, local AS number 65100
   BGP table version is 5
   1 BGP AS-PATH entries
   0 BGP community entries
   Neighbor   V         AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
   10.200.1.1 4      65100     222     225        3    0    0 00:15:14        3
   Total number of neighbors 1

2. Verify the BGP routing table:

   # get router info routing-table bgp
   Routing table for VRF=0
   B       11.11.11.11/32 [200/0] via 10.200.1.1 (recursive via spoke1-1 tunnel 11.1.1.11), 00:15:19
                                                 (recursive via spoke1-2 tunnel 111.1.1.11), 00:15:19, [1/0]
   B       22.0.0.0/8 [200/0] via 10.200.1.1 (recursive via spoke1-1 tunnel 11.1.1.11), 00:15:19
                                             (recursive via spoke1-2 tunnel 111.1.1.11), 00:15:19, [1/0]
   B       33.0.0.0/8 [200/0] via 10.200.1.1 (recursive via spoke1-1 tunnel 11.1.1.11), 00:15:19
                                             (recursive via spoke1-2 tunnel 111.1.1.11), 00:15:19, [1/0]

IPv6 use case

To verify the status on Spoke 1:

1. Verify the BGP status:

   # get router info6 bgp summary
   VRF 0 BGP router identifier 10.200.1.100, local AS number 65100
   BGP table version is 6
   1 BGP AS-PATH entries
   0 BGP community entries
   Neighbor         V         AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
   2001::10:200:1:1 4      65100     223     224        4    0    0 00:15:21        3
   Total number of neighbors 1

2. Verify the BGP routing table:

   # get router info6 routing-table bgp
   Routing table for VRF=0
   B       2001::11:11:11:11/128 [200/0] via 2001::10:200:1:1 (recursive via spoke1-1 tunnel ::11.1.1.11), 00:15:29
                                                              (recursive via spoke1-2 tunnel ::111.1.1.11), 00:15:29, [1024/0]
   B       2001:22::/32 [200/0] via 2001::10:200:1:1 (recursive via spoke1-1 tunnel ::11.1.1.11), 00:15:29
                                                     (recursive via spoke1-2 tunnel ::111.1.1.11), 00:15:29, [1024/0]
   B       2001:33::/32 [200/0] via 2001::10:200:1:1 (recursive via spoke1-1 tunnel ::11.1.1.11), 00:15:29
                                                     (recursive via spoke1-2 tunnel ::111.1.1.11), 00:15:29, [1024/0]

Verifying the configuration after a single spoke-to-spoke shortcut VPN is established

IPv4 use case

To trigger a single spoke-to-spoke shortcut VPN, on host 22.1.1.22, ping the host 33.1.1.33 in the Financial Department. Because of the SD-WAN rule, use SD-WAN member 1 (via ISP1) and its dynamic shortcuts to reach hosts in the Financial Department.

To verify the status on Spoke 1:

1. Verify the BGP status:

   # get router info bgp summary
   VRF 0 BGP router identifier 10.200.1.100, local AS number 65100
   BGP table version is 6
   1 BGP AS-PATH entries
   0 BGP community entries
   Neighbor     V         AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
   10.200.1.1   4      65100     252     254        3    0    0 00:17:22        3 
   10.200.1.101 4      65100       6       6        5    0    0 00:00:14        1  
   Total number of neighbors 2

   Spoke 1 has as its BGP neighbors:

   • Hub FortiGate at 10.200.1.1
   • Spoke 2 FortiGate at 10.200.1.101

2. Verify the BGP routing table:

   # get router info routing-table bgp
   Routing table for VRF=0
   B       11.11.11.11/32 [200/0] via 10.200.1.1 (recursive via spoke1-1 tunnel 11.1.1.11), 00:17:26
                                                 (recursive via spoke1-2 tunnel 111.1.1.11), 00:17:26, [1/0]
   B       22.0.0.0/8 [200/0] via 10.200.1.1 (recursive via spoke1-1 tunnel 11.1.1.11), 00:17:26
                                             (recursive via spoke1-2 tunnel 111.1.1.11), 00:17:26, [1/0]
   B       33.0.0.0/8 [200/0] via 10.200.1.1 (recursive via spoke1-1 tunnel 11.1.1.11), 00:17:26
                                             (recursive via spoke1-2 tunnel 111.1.1.11), 00:17:26, [1/0]
   B       33.1.1.0/24 [200/0] via 10.200.1.101 (recursive via spoke1-1_0 tunnel 13.1.1.3), 00:00:18, [1/0]

   The remote route learned from Spoke 2 through the spoke1_1_0 tunnel and using BGP is 33.1.1.0/24.

IPv6 use case

To trigger a single spoke-to-spoke shortcut VPN over IPv6, on host 2001:22::22/64, ping the host 2001:33::33/64 in the Financial Department. Because of the SD-WAN rule, use SD-WAN member 1 (via ISP1) and its dynamic shortcuts to reach hosts in the Financial Department.

To verify the status on Spoke 1:

1. Verify the BGP status:

   # get router info6 bgp summary
   VRF 0 BGP router identifier 10.200.1.100, local AS number 65100
   BGP table version is 7
   1 BGP AS-PATH entries
   0 BGP community entries
   Neighbor           V         AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
   2001::10:200:1:1   4      65100     253     254        4    0    0 00:17:28        3 
   2001::10:200:1:101 4      65100       7       7        6    0    0 00:00:21        1
   Total number of neighbors 2

   Spoke 1 has as its BGP neighbors:

   • Hub FortiGate at 2001::10:200:1:1
   • Spoke 2 FortiGate at 2001::10:200:1:101

2. Verify the BGP routing table:

   # get router info6 routing-table bgp
   Routing table for VRF=0
   B       2001::11:11:11:11/128 [200/0] via 2001::10:200:1:1 (recursive via spoke1-1 tunnel ::11.1.1.11), 00:17:30
                                                              (recursive via spoke1-2 tunnel ::111.1.1.11), 00:17:30, [1024/0]
   B       2001:22::/32 [200/0] via 2001::10:200:1:1 (recursive via spoke1-1 tunnel ::11.1.1.11), 00:17:30
                                                     (recursive via spoke1-2 tunnel ::111.1.1.11), 00:17:30, [1024/0]
   B       2001:33::/32 [200/0] via 2001::10:200:1:1 (recursive via spoke1-1 tunnel ::11.1.1.11), 00:17:30
                                                     (recursive via spoke1-2 tunnel ::111.1.1.11), 00:17:30, [1024/0]
   B       2001:33::/64 [200/0] via 2001::10:200:1:101 (recursive via spoke1-1_0 tunnel ::13.1.1.3), 00:00:24, [1024/0]

   The remote route learned from Spoke 2 through the spoke1-1_0 tunnel and using BGP is 2001:33::/64.

Verifying the configuration after a second spoke-to-spoke shortcut VPN is established

IPv4 use case

To trigger a second spoke-to-spoke shortcut VPN, on host 22.1.1.22, ping the host 33.1.1.133 in the Engineering Department. Because of the SD-WAN rule, use SD-WAN member 2 (via ISP2) and its dynamic shortcuts to reach hosts in the Engineering Department.

To verify the status on Spoke 1:

1. Verify the BGP status:

   # get router info bgp summary
   VRF 0 BGP router identifier 10.200.1.100, local AS number 65100
   BGP table version is 6
   1 BGP AS-PATH entries
   0 BGP community entries
   Neighbor     V         AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
   10.200.1.1   4      65100     263     265        3    0    0 00:18:12        3   
   10.200.1.101 4      65100      17      17        5    0    0 00:01:04        1  
   Total number of neighbors 

   Spoke 1 continues to have its BGP neighbors:

   • Hub FortiGate at 10.200.1.1
   • Spoke 2 FortiGate at 10.200.1.101

2. Verify the BGP routing table:

   # get router info routing-table bgp
   Routing table for VRF=0
   B       11.11.11.11/32 [200/0] via 10.200.1.1 (recursive via spoke1-1 tunnel 11.1.1.11), 00:18:17
                                                 (recursive via spoke1-2 tunnel 111.1.1.11), 00:18:17, [1/0]
   B       22.0.0.0/8 [200/0] via 10.200.1.1 (recursive via spoke1-1 tunnel 11.1.1.11), 00:18:17
                                             (recursive via spoke1-2 tunnel 111.1.1.11), 00:18:17, [1/0]
   B       33.0.0.0/8 [200/0] via 10.200.1.1 (recursive via spoke1-1 tunnel 11.1.1.11), 00:18:17
                                             (recursive via spoke1-2 tunnel 111.1.1.11), 00:18:17, [1/0]
   B       33.1.1.0/24 [200/0] via 10.200.1.101 (recursive via spoke1-1_0 tunnel 13.1.1.3), 00:01:09   
                                                (recursive via spoke1-2_0 tunnel 113.1.1.3), 00:01:09, [1/0]

   The remote route learned from Spoke 2 through the spoke1-2_0 tunnel and using BGP is 33.1.1.0/24.

IPv6 use case

To trigger a second spoke-to-spoke shortcut VPN over IPv6, on host 2001:22::22/64, ping the host 2001:33::133/64 in the Engineering Department. Because of the SD-WAN rule, use SD-WAN member 2 (via ISP2) and its dynamic shortcuts to reach hosts in the Engineering Department.

To verify the status on Spoke 1:

1. Verify the BGP status:

   # get router info6 bgp summary
   VRF 0 BGP router identifier 10.200.1.100, local AS number 65100
   BGP table version is 7
   1 BGP AS-PATH entries
   0 BGP community entries
   Neighbor           V         AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
   2001::10:200:1:1   4      65100     264     265        4    0    0 00:18:18        3   
   2001::10:200:1:101 4      65100      19      19        6    0    0 00:01:11        1  
   Total number of neighbors 2

   Spoke 1 continues to have its BGP neighbors:

   • Hub FortiGate at 2001::10:200:1:1
   • Spoke 2 FortiGate at 2001::10:200:1:101

2. Verify the BGP routing table:

   # get router info6 routing-table bgp
   Routing table for VRF=0
   B       2001::11:11:11:11/128 [200/0] via 2001::10:200:1:1 (recursive via spoke1-1 tunnel ::11.1.1.11), 00:18:20
                                                              (recursive via spoke1-2 tunnel ::111.1.1.11), 00:18:20, [1024/0]
   B       2001:22::/32 [200/0] via 2001::10:200:1:1 (recursive via spoke1-1 tunnel ::11.1.1.11), 00:18:20
                                                     (recursive via spoke1-2 tunnel ::111.1.1.11), 00:18:20, [1024/0]
   B       2001:33::/32 [200/0] via 2001::10:200:1:1 (recursive via spoke1-1 tunnel ::11.1.1.11), 00:18:20
                                                     (recursive via spoke1-2 tunnel ::111.1.1.11), 00:18:20, [1024/0]
   B       2001:33::/64 [200/0] via 2001::10:200:1:101 (recursive via spoke1-1_0 tunnel ::13.1.1.3), 00:01:14   
                                                       (recursive via spoke1-2_0 tunnel ::113.1.1.3), 00:01:14, [1024/0] 

   The remote route learned from Spoke 2 through the spoke1-2_0 tunnel and using BGP is 2001:33::/64.