SD-WAN in large scale deployments

Phase 2 selectors can be used to inject IKE routes on the ADVPN shortcut tunnel. When configuration method (mode-cfg) is enabled in IPsec phase 1 configuration, enabling mode-cfg-allow-client-selector allows custom phase 2 selectors to be configured. By also enabling the addition of a route to the peer destination selector (add-route) in the phase 1 configuration, IKE routes based on the phase 2 selectors can be injected. This means that routes do not need to be reflected on the hub to propagate them between spokes, avoiding possible BGP daemon process load issues and improving network scalability in a large-scale ADVPN network.

Route map rules can apply priorities to BGP routes. On the hub, priorities can be set in a route map's rules, and the route map can be applied on BGP routes. This allows the hub to mark the preferred path learned from the spokes with a priority value (lower priority is preferred), instead of using multiple SD-WAN policy routes on the hub. When a preferred outbound route map (route-map-out-preferable) is also configured in an SD-WAN neighbor on the spoke, deploying SD-WAN rules on the hub to steer traffic from the hub to a spoke is unnecessary.

SD-WAN members' local cost can be exchanged on the ADVPN shortcut tunnel so that spokes can use the remote cost as tiebreak to select a preferred shortcut. If multiple shortcuts originate from the same member to different members on the same remote spoke, then the remote cost on the shortcuts is used as the tiebreak to decide which shortcut is preferred.

In this example, SD-WAN is configured on an ADVPN network with a BGP neighbor per overlay.

Instead of reflecting BGP routes with the route-reflector on the hub, when the shortcuts are triggered, IKE routes on the shortcuts are directly injected based on the configured phase 2 selectors to allow routes to be exchanged between spokes.

Routes between the hub and the spokes are exchanged by BGP, and the spokes use the default route to send spoke-to-spoke traffic to the hub and trigger the shortcuts.

Instead of configuring SD-WAN rules on the hub, different priorities are configured on the BGP routes by matching different BGP communities to steer traffic from the hub to the spokes.

To configure Spoke 1:

1. Configure phase 1:

   config vpn ipsec phase1-interface
       edit "spoke11-p1"
           ...
           set ike-version 2
           set net-device enable
           set add-route enable
           set mode-cfg enable
           set auto-discovery-receiver enable
           set mode-cfg-allow-client-selector enable
           set link-cost 11              
           ...
       next
       edit "spoke12-p1"
          ...
           set ike-version 2
           set net-device enable
           set add-route enable
           set mode-cfg enable
           set auto-discovery-receiver enable
           set mode-cfg-allow-client-selector enable
           set link-cost 21
       next
   end

2. Configure phase 2:

   config vpn ipsec phase2-interface
       edit "spoke11-p2"
           ...
           set src-name "LAN_Net"
           set dst-name "all"
       next
       edit "spoke12-p2"
           ...
           set src-name "LAN_Net"
           set dst-name "all"
       next
   end

3. Configure an address group:

   Spoke 1 uses LAN subnet 10.1-3.100.0/24.

   config firewall addrgrp
       edit "LAN_Net"
           set member "10.1.100.0" "10.2.100.0" "10.3.100.0"
       next
   end

4. Configure route maps:

   • If overlay 1 to the hub is in SLA, attach "65000:1" to the BGP routes advertised to the hub over overlay 1.
   • If overlay 2 to the hub is in SLA, attach "65000:2" to the BGP routes advertised to the hub over overlay 2.
   • If any overlay to the hub is out of SLA, attach "65000:9999" to the BGP routes advertised to the hub over any overlay.

   config router route-map
       edit "HUB_CARRIER1"
           config rule
               edit 1
                   set set-community "65000:1"
                   ...
               next
           end
           ...
       next
       edit "HUB_CARRIER2"
           config rule
               edit 1
                   set set-community "65000:2"
                   ...
               next
           end
           ...
       next
       edit "HUB_BAD"
           config rule
               edit 1
                   set set-community "65000:9999"
                   ...
               next
           end
           ...
       next
   end

5. Configure BGP and SD-WAN members and neighbors:

   config router bgp
       set as 65412
       config neighbor
           edit "10.10.15.253"
               set remote-as 65412
               set route-map-out "HUB_BAD"
               set route-map-out-preferable "HUB_CARRIER1"
               ...
           next
           edit "10.10.16.253"
               set remote-as 65412
               set route-map-out "HUB_BAD"
               set route-map-out-preferable "HUB_CARRIER2"
               ...
           next
       end
   end

   config system sdwan
       config members
           edit 1
               set interface "spoke11-p1"
               set cost 10
           next
           edit 2
               set interface "spoke12-p1"
               set cost 20
           next
       end
       config neighbor
           edit "10.10.15.253"
               set member 1
               set health-check "1"
               set sla-id 1
           next
           edit "10.10.16.253"
               set member 2
               set health-check "11"
               set sla-id 1
           next
       end
   end

To configure Spoke 2:

1. Configure phase 1:

   config vpn ipsec phase1-interface
       edit "spoke21-p1"
           ...
           set ike-version 2
           set net-device enable
           set add-route enable
           set mode-cfg enable
           set auto-discovery-receiver enable
           set mode-cfg-allow-client-selector enable
           set link-cost 101              
           ...
       next
       edit "spoke22-p1"
          ...
           set ike-version 2
           set net-device enable
           set add-route enable
           set mode-cfg enable
           set auto-discovery-receiver enable
           set mode-cfg-allow-client-selector enable
           set link-cost 201
       next
   end

2. Configure phase 2:

   config vpn ipsec phase2-interface
       edit "spoke21-p2"
           ...
           set src-name "LAN_Net"
           set dst-name "all"
       next
       edit "spoke22-p2"
           ...
           set src-name "LAN_Net"
           set dst-name "all"
       next
   end

3. Configure an address group:

   Spoke 2 uses LAN subnet 192.168.5-7.0/24.

   config firewall addrgrp
       edit "LAN_Net"
           set member "192.168.5.0" "192.168.6.0" "192.168.7.0"
       next
   end

4. Configure route maps:

   • If overlay 1 to the hub is in SLA, attach "65000:1" to the BGP routes advertised to the hub over overlay 1.
   • If overlay 2 to the hub is in SLA, attach "65000:2" to the BGP routes advertised to the hub over overlay 2.
   • If any overlay to the hub is out of SLA, attach "65000:9999" to the BGP routes advertised to the hub over any overlay.

   config router route-map
       edit "HUB_CARRIER1"
           config rule
               edit 1
                   set set-community "65000:1"
                   ...
               next
           end
           ...
       next
       edit "HUB_CARRIER2"
           config rule
               edit 1
                   set set-community "65000:2"
                   ...
               next
           end
           ...
       next
       edit "HUB_BAD"
           config rule
               edit 1
                   set set-community "65000:9999"
                   ...
               next
           end
           ...
       next
   end

5. Configure BGP and SD-WAN members and neighbors:

   config router bgp
       set as 65412
       config neighbor
           edit "10.10.15.253"
               set remote-as 65412
               set route-map-out "HUB_BAD"
               set route-map-out-preferable "HUB_CARRIER1"
               ...
           next
           edit "10.10.16.253"
               set remote-as 65412
               set route-map-out "HUB_BAD"
               set route-map-out-preferable "HUB_CARRIER2"
               ...
           next
       end
   end

   config system sdwan
       config members
           edit 1
               set interface "spoke21-p1"
               set cost 10
           next
           edit 2
               set interface "spoke22-p1"
               set cost 20
           next
       end
       config neighbor
           edit "10.10.15.253"
               set member 1
               set health-check "1"
               set sla-id 1
           next
           edit "10.10.16.253"
               set member 2
               set health-check "11"
               set sla-id 1
           next
       end
   end

To configure the hub:

1. Configure the route maps:

   • Set the priority to 100 for routes with community 65000:1, indicating that they are in SLA for overlay 1.
   • Set the priority to 200 for routes with community 65000:2, indicating that they are in SLA for overlay 2.
   • Set the priority to 9999 for routes with community 65000:9999, indicating that they are out of SLA for any overlay.

   config router route-map
        edit "Set_Pri"
           config rule
               edit 1
                   set match-community "comm_65000:1"
                   set set-priority 100
               next
               edit 2
                   set match-community "comm_65000:2"
                   set set-priority 200
               next
               edit 3
                   set match-community "comm_65000:9999"
                   set set-priority 9999
               next
           end
       next
   end

2. Configure BGP:

   config router bgp
       set as 65412
       config neighbor-group
           edit "advpn"
               set remote-as 65412
               set route-map-in "Set_Pri"
               ...
           next
           edit "advpn2"
               set remote-as 65412
               set route-map-in "Set_Pri"
               ...
           next
       end
       config neighbor-range
           edit 1
               set prefix 10.10.15.0 255.255.255.0
               set neighbor-group "advpn"
           next
           edit 2
               set prefix 10.10.16.0 255.255.255.0
               set neighbor-group "advpn2"
           next
       end
   end

To test the configuration:

1. Check the routing tables on the spokes:

   Spoke 1:

   spoke-1 (root) # get router info routing-table all
   B*      0.0.0.0/0 [200/0] via 10.10.15.253 (recursive is directly connected, spoke11-p1), 00:01:17, [1/0]         // default route to hub
                            [200/0] via 10.10.16.253 (recursive is directly connected, spoke12-p1), 00:01:17, [1/0]
   B       9.0.0.0/24 [200/0] via 10.10.15.253 (recursive is directly connected, spoke11-p1), 00:01:17, [1/0]        // route to the server behind hub
                             [200/0] via 10.10.16.253 (recursive is directly connected, spoke12-p1), 00:01:17, [1/0]
   C       10.1.100.0/24 is directly connected, port2             // route to PC 1
   C       10.10.15.0/24 is directly connected, spoke11-p1        // overlay 1
   C       10.10.15.1/32 is directly connected, spoke11-p1
   C       10.10.16.0/24 is directly connected, spoke12-p1        // overlay 2
   C       10.10.16.1/32 is directly connected, spoke12-p1

   Spoke 2:

   spoke-2 (root) # get router info routing-table all
   B*      0.0.0.0/0 [200/0] via 10.10.15.253 (recursive is directly connected, spoke21-p1), 00:46:14, [1/0]         // default route to hub
                            [200/0] via 10.10.16.253 (recursive is directly connected, spoke22-p1), 00:46:14, [1/0]
   B       9.0.0.0/24 [200/0] via 10.10.15.253 (recursive is directly connected, spoke21-p1), 00:46:18, [1/0]        // route to the server behind hub
                             [200/0] via 10.10.16.253 (recursive is directly connected, spoke22-p1), 00:46:18, [1/0]
   C       10.10.15.0/24 is directly connected, spoke21-p1        // overlay 1
   C       10.10.15.2/32 is directly connected, spoke21-p1
   C       10.10.16.0/24 is directly connected, spoke22-p1        // overlay 2
   C       10.10.16.2/32 is directly connected, spoke22-p1
   C       192.168.5.0/24 is directly connected, port2            // route to PC 2

2. Send traffic from PC 1 to PC 2 and trigger the shortcut:

   The IKE routes on the shortcut are directly injected based on the phase 2 selectors, and spoke-to-spoke traffic then goes directly through the shortcut instead of going through the hub.

   Spoke 1:

   spoke-1 (root) # get router info routing-table static
   S       192.168.5.0/24 [15/0] via spoke11-p1_0 tunnel 172.16.200.4 vrf 0, [1/0]
   S       192.168.6.0/24 [15/0] via spoke11-p1_0 tunnel 172.16.200.4 vrf 0, [1/0]
   S       192.168.7.0/24 [15/0] via spoke11-p1_0 tunnel 172.16.200.4 vrf 0, [1/0]

   spoke-1 (root) # diagnose sniffer packet any 'host 192.168.5.44' 4
   interfaces=[any]
   filters=[host 192.168.5.44]
   1.446306 port2 in 10.1.100.22 -> 192.168.5.44: icmp: echo request
   1.446327 spoke11-p1_0 out 10.1.100.22 -> 192.168.5.44: icmp: echo request
   1.446521 spoke11-p1_0 in 192.168.5.44 -> 10.1.100.22: icmp: echo reply
   1.446536 port2 out 192.168.5.44 -> 10.1.100.22: icmp: echo reply

   Spoke 2:

   spoke-2 (root) # get router info routing-table static
   S       10.1.100.0/24 [15/0] via spoke21-p1_0 tunnel 10.10.15.1 vrf 0, [1/0]
   S       10.2.100.0/24 [15/0] via spoke21-p1_0 tunnel 10.10.15.1 vrf 0, [1/0]
   S       10.3.100.0/24 [15/0] via spoke21-p1_0 tunnel 10.10.15.1 vrf 0, [1/0]

3. Confirm that the overlays are in SLA on the spokes:

   Spoke 1:

   spoke-1 (root) # diagnose sys sdwan  neighbor
   Neighbor(10.10.15.253): member(1)role(standalone)
           Health-check(1:1)  sla-pass selected alive
   Neighbor(10.10.16.253): member(2)role(standalone)
           Health-check(11:1)  sla-pass selected alive

   Spoke 2:

   spoke-2 (root) # diagnose sys sdwan neighbor
   Neighbor(10.10.15.253): member(1)role(standalone)             
           Health-check(1:1)  sla-pass selected alive
   Neighbor(10.10.16.253): member(2)role(standalone)
           Health-check(11:1)  sla-pass selected alive

4. On the hub, check that the routes received from the spokes have the expected priorities:

   hub (root) # diagnose ip route list | grep proto=11
   tab=254 vf=0 scope=0 type=1 proto=11 prio=100 0.0.0.0/0.0.0.0/0->10.1.100.0/24 pref=0.0.0.0 gwy=10.10.15.1 dev=101(hub-phase1)
   tab=254 vf=0 scope=0 type=1 proto=11 prio=200 0.0.0.0/0.0.0.0/0->10.1.100.0/24 pref=0.0.0.0 gwy=10.10.16.1 dev=102(hub2-phase1)
   tab=254 vf=0 scope=0 type=1 proto=11 prio=100 0.0.0.0/0.0.0.0/0->192.168.5.0/24 pref=0.0.0.0 gwy=10.10.15.2 dev=101(hub-phase1)
   tab=254 vf=0 scope=0 type=1 proto=11 prio=200 0.0.0.0/0.0.0.0/0->192.168.5.0/24 pref=0.0.0.0 gwy=10.10.16.2 dev=102(hub2-phase1)

   The priority set by the hub's route map is based on the community string received from the spoke. The route with a lower priority value is selected, so traffic to Spoke 1 goes out on the hub-phase1 tunnel:

   hub (root) # diagnose sniffer packet any 'host 9.0.0.2' 4
   interfaces=[any]
   filters=[host 9.0.0.2]
   2.735456 R190 in 9.0.0.2 -> 10.1.100.22: icmp: echo request
   2.735508 hub-phase1 out 9.0.0.2 -> 10.1.100.22: icmp: echo request
   2.735813 hub-phase1 in 10.1.100.22 -> 9.0.0.2: icmp: echo reply
   2.735854 R190 out 10.1.100.22 -> 9.0.0.2: icmp: echo reply

5. If overlay 1 goes out of SLA, the priorities of the routes on the hub are updated and traffic from the hub to Spoke 1 goes through overlay 2:

   Spoke 1:

   spoke-1 (root) # diagnose sys sdwan  neighbor
   Neighbor(10.10.15.253): member(1)role(standalone)
           Health-check(1:1)  sla-fail alive
   Neighbor(10.10.16.253): member(2)role(standalone)
           Health-check(11:1)  sla-pass selected alive
   

   Spoke 2:

   spoke-2 (root) # diagnose sys sdwan neighbor
   Neighbor(10.10.15.253): member(1)role(standalone)
           Health-check(1:1)  sla-fail alive
   Neighbor(10.10.16.253): member(2)role(standalone)
           Health-check(11:1)  sla-pass selected alive
   

   Hub:

   hub (root) # diagnose ip route list | grep proto=11
   tab=254 vf=0 scope=0 type=1 proto=11 prio=200 0.0.0.0/0.0.0.0/0->10.1.100.0/24 pref=0.0.0.0 gwy=10.10.16.1 dev=102(hub2-phase1)
   tab=254 vf=0 scope=0 type=1 proto=11 prio=9999 0.0.0.0/0.0.0.0/0->10.1.100.0/24 pref=0.0.0.0 gwy=10.10.15.1 dev=101(hub-phase1)
   tab=254 vf=0 scope=0 type=1 proto=11 prio=200 0.0.0.0/0.0.0.0/0->192.168.5.0/24 pref=0.0.0.0 gwy=10.10.16.2 dev=102(hub2-phase1)
   tab=254 vf=0 scope=0 type=1 proto=11 prio=9999 0.0.0.0/0.0.0.0/0->192.168.5.0/24 pref=0.0.0.0 gwy=10.10.15.2 dev=101(hub-phase1)
   

   hub (root) # diagnose sniffer packet any 'host 9.0.0.2' 4
   interfaces=[any]
   filters=[host 9.0.0.2]
   3.550181 R190 in 9.0.0.2 -> 10.1.100.22: icmp: echo request
   3.550234 hub2-phase1 out 9.0.0.2 -> 10.1.100.22: icmp: echo request
   3.550713 hub2-phase1 in 10.1.100.22 -> 9.0.0.2: icmp: echo reply
   3.550735 R190 out 10.1.100.22 -> 9.0.0.2: icmp: echo reply

6. Verify the service diagnostics on Spoke 1:

   # diagnose sys sdwan service4
   
   Service(1): Address Mode(IPV4) flags=0x200 use-shortcut-sla
    Tie break: cfg
     Gen(4), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(sla), sla-compare-order
     Members(2):
       1: Seq_num(1 spoke11-p1), alive, sla(0x1), gid(0), cfg_order(0), local cost(10), selected
       2: Seq_num(2 spoke12-p1), alive, sla(0x1), gid(0), cfg_order(1), local cost(20), selected
     Src address(1):
           10.1.100.0-10.1.100.255
   
     Dst address(1):
           0.0.0.0-255.255.255.255

7. Verify the service diagnostics on Spoke 2:

   # diagnose sys sdwan service4
   
   Service(1): Address Mode(IPV4) flags=0x200 use-shortcut-sla
    Tie break: cfg
     Gen(2), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(sla), sla-compare-order
     Members(2):
       1: Seq_num(1 spoke21-p1), alive, sla(0x1), gid(0), cfg_order(0), local cost(10), selected
       2: Seq_num(2 spoke22-p1), alive, sla(0x1), gid(0), cfg_order(1), local cost(20), selected
     Src address(1):
           192.168.5.0-192.168.5.255
   
     Dst address(1):
           0.0.0.0-255.255.255.255

8. Trigger shortcuts between Spoke 1 and Spoke 2:

   • Shortcuts spoke11-p1_1 and spoke11-p1_0 originate from spoke11-p1.

   • spoke11-p1_1 corresponds to spoke21-p1_0 on Spoke 2.

   • spoke11-p1_0 corresponds to spoke22-p1_0 on Spoke 2.

   Spoke 1:

   # diagnose sys sdwan service4
   
   Service(1): Address Mode(IPV4) flags=0x200 use-shortcut-sla
    Tie break: cfg
     Gen(11), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(sla), sla-compare-order
     Member sub interface(4):
       3: seq_num(1), interface(spoke11-p1):
          1: spoke11-p1_0(80)
          2: spoke11-p1_1(81)
     Members(4):
       1: Seq_num(1 spoke11-p1_1), alive, sla(0x1), gid(0), remote cost(101), cfg_order(0), local cost(10), selected
       2: Seq_num(1 spoke11-p1_0), alive, sla(0x1), gid(0), remote cost(201), cfg_order(0), local cost(10), selected
       3: Seq_num(1 spoke11-p1), alive, sla(0x1), gid(0), cfg_order(0), local cost(10), selected
       4: Seq_num(2 spoke12-p1), alive, sla(0x1), gid(0), cfg_order(1), local cost(20), selected
     Src address(1):
           10.1.100.0-10.1.100.255
   
     Dst address(1):
           0.0.0.0-255.255.255.255

   Spoke 2:

   # diagnose sys sdwan service4
   
   Service(1): Address Mode(IPV4) flags=0x200 use-shortcut-sla
    Tie break: cfg
     Gen(15), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(sla), sla-compare-order
     Member sub interface(4):
       2: seq_num(1), interface(spoke21-p1):
          1: spoke21-p1_0(75)
       4: seq_num(2), interface(spoke22-p1):
          1: spoke22-p1_0(74)
     Members(4):
       1: Seq_num(1 spoke21-p1_0), alive, sla(0x1), gid(0), remote cost(11), cfg_order(0), local cost(10), selected
       2: Seq_num(1 spoke21-p1), alive, sla(0x1), gid(0), cfg_order(0), local cost(10), selected
       3: Seq_num(2 spoke22-p1_0), alive, sla(0x1), gid(0), remote cost(11), cfg_order(1), local cost(20), selected
       4: Seq_num(2 spoke22-p1), alive, sla(0x1), gid(0), cfg_order(1), local cost(20), selected
     Src address(1):
           192.168.5.0-192.168.5.255
   
     Dst address(1):
           0.0.0.0-255.255.255.255

   The spoke11-p1_1 shortcut on Spoke 1 is preferred over spoke11-p1_0 due to the lower remote link cost of 101 when they have the same local SD-WAN member cost of 10.

9. Verify the policy route list on Spoke 1:

   # diagnose firewall proute list
   list route policy info(vf=root):
   
   id=2131755009(0x7f100001) vwl_service=1(1) vwl_mbr_seq=1 1 1 2 dscp_tag=0xfc 0xfc flags=0x0 tos=0x00 tos_mask=0x00 protocol=0 sport=0-65535 iif=0(any) dport=1-65535 path(4) oif=81(spoke11-p1_1) oif=80(spoke11-p1_0) oif=54(spoke11-p1) oif=55(spoke12-p1)
   source(1): 10.1.100.0-10.1.100.255
   destination(1): 0.0.0.0-255.255.255.255
   hit_count=176 last_used=2022-07-12 11:56:08