Fortinet black logo

Administration Guide

SD-WAN in large scale deployments

SD-WAN in large scale deployments

Phase 2 selectors can be used to inject IKE routes on the ADVPN shortcut tunnel. When configuration method (mode-cfg) is enabled in IPsec phase 1 configuration, enabling mode-cfg-allow-client-selector allows custom phase 2 selectors to be configured. By also enabling the addition of a route to the peer destination selector (add-route) in the phase 1 configuration, IKE routes based on the phase 2 selectors can be injected. This means that routes do not need to be reflected on the hub to propagate them between spokes, avoiding possible BGP daemon process load issues and improving network scalability in a large-scale ADVPN network.

Route map rules can apply priorities to BGP routes. On the hub, priorities can be set in a route map's rules, and the route map can be applied on BGP routes. This allows the hub to mark the preferred path learned from the spokes with a higher priority, instead of using multiple SD-WAN policy routes on the hub. When a preferred outbound route map (route-map-out-preferable) is also configured in an SD-WAN neighbor on the spoke, deploying SD-WAN rules on the hub to steer traffic from the hub to a spoke is unnecessary.

SD-WAN members' local cost can be exchanged on the ADVPN shortcut tunnel so that spokes can use the remote cost as tiebreak to select a preferred shortcut. If multiple shortcuts originate from the same member to different members on the same remote spoke, then the remote cost on the shortcuts is used as the tiebreak to decide which shortcut is preferred.

In this example, SD-WAN is configured on an ADVPN network with a BGP neighbor per overlay.

Instead of reflecting BGP routes with the route-reflector on the hub, when the shortcuts are triggered, IKE routes on the shortcuts are directly injected based on the configured phase 2 selectors to allow routes to be exchanged between spokes.

Routes between the hub and the spokes are exchanged by BGP, and the spokes use the default route to send spoke-to-spoke traffic to the hub and trigger the shortcuts.

Instead of configuring SD-WAN rules on the hub, different priorities are configured on the BGP routes by matching different BGP communities to steer traffic from the hub to the spokes.

To configure Spoke 1:
  1. Configure phase 1:

    config vpn ipsec phase1-interface
        edit "spoke11-p1"
            ...
            set ike-version 2
            set net-device enable
            set add-route enable
            set mode-cfg enable
            set auto-discovery-receiver enable
            set mode-cfg-allow-client-selector enable
            set link-cost 11              
            ...
        next
        edit "spoke12-p1"
           ...
            set ike-version 2
            set net-device enable
            set add-route enable
            set mode-cfg enable
            set auto-discovery-receiver enable
            set mode-cfg-allow-client-selector enable
            set link-cost 21
        next
    end
  2. Configure phase 2:

    config vpn ipsec phase2-interface
        edit "spoke11-p2"
            ...
            set src-name "LAN_Net"
            set dst-name "all"
        next
        edit "spoke12-p2"
            ...
            set src-name "LAN_Net"
            set dst-name "all"
        next
    end
  3. Configure an address group:

    Spoke 1 uses LAN subnet 10.1-3.100.0/24.

    config firewall addrgrp
        edit "LAN_Net"
            set member "10.1.100.0" "10.2.100.0" "10.3.100.0"
        next
    end
  4. Configure route maps:

    • If overlay 1 to the hub is in SLA, attach "65000:1" to the BGP routes advertised to the hub over overlay 1.
    • If overlay 2 to the hub is in SLA, attach "65000:2" to the BGP routes advertised to the hub over overlay 2.
    • If any overlay to the hub is out of SLA, attach "65000:9999" to the BGP routes advertised to the hub over any overlay.
    config router route-map
        edit "HUB_CARRIER1"
            config rule
                edit 1
                    set set-community "65000:1"
                    ...
                next
            end
            ...
        next
        edit "HUB_CARRIER2"
            config rule
                edit 1
                    set set-community "65000:2"
                    ...
                next
            end
            ...
        next
        edit "HUB_BAD"
            config rule
                edit 1
                    set set-community "65000:9999"
                    ...
                next
            end
            ...
        next
    end
  5. Configure BGP and SD-WAN members and neighbors:

    config router bgp
        set as 65412
        config neighbor
            edit "10.10.15.253"
                set remote-as 65412
                set route-map-out "HUB_BAD"
                set route-map-out-preferable "HUB_CARRIER1"
                ...
            next
            edit "10.10.16.253"
                set remote-as 65412
                set route-map-out "HUB_BAD"
                set route-map-out-preferable "HUB_CARRIER2"
                ...
            next
        end
    end
    config system sdwan
        config members
            edit 1
                set interface "spoke11-p1"
                set cost 10
            next
            edit 2
                set interface "spoke12-p1"
                set cost 20
            next
        end
        config neighbor
            edit "10.10.15.253"
                set member 1
                set health-check "1"
                set sla-id 1
            next
            edit "10.10.16.253"
                set member 2
                set health-check "11"
                set sla-id 1
            next
        end
    end
To configure Spoke 2:
  1. Configure phase 1:

    config vpn ipsec phase1-interface
        edit "spoke21-p1"
            ...
            set ike-version 2
            set net-device enable
            set add-route enable
            set mode-cfg enable
            set auto-discovery-receiver enable
            set mode-cfg-allow-client-selector enable
            set link-cost 101              
            ...
        next
        edit "spoke22-p1"
           ...
            set ike-version 2
            set net-device enable
            set add-route enable
            set mode-cfg enable
            set auto-discovery-receiver enable
            set mode-cfg-allow-client-selector enable
            set link-cost 201
        next
    end
  2. Configure phase 2:

    config vpn ipsec phase2-interface
        edit "spoke21-p2"
            ...
            set src-name "LAN_Net"
            set dst-name "all"
        next
        edit "spoke22-p2"
            ...
            set src-name "LAN_Net"
            set dst-name "all"
        next
    end
  3. Configure an address group:

    Spoke 2 uses LAN subnet 192.168.5-7.0/24.

    config firewall addrgrp
        edit "LAN_Net"
            set member "192.168.5.0" "192.168.6.0" "192.168.7.0"
        next
    end
  4. Configure route maps:

    • If overlay 1 to the hub is in SLA, attach "65000:1" to the BGP routes advertised to the hub over overlay 1.
    • If overlay 2 to the hub is in SLA, attach "65000:2" to the BGP routes advertised to the hub over overlay 2.
    • If any overlay to the hub is out of SLA, attach "65000:9999" to the BGP routes advertised to the hub over any overlay.
    config router route-map
        edit "HUB_CARRIER1"
            config rule
                edit 1
                    set set-community "65000:1"
                    ...
                next
            end
            ...
        next
        edit "HUB_CARRIER2"
            config rule
                edit 1
                    set set-community "65000:2"
                    ...
                next
            end
            ...
        next
        edit "HUB_BAD"
            config rule
                edit 1
                    set set-community "65000:9999"
                    ...
                next
            end
            ...
        next
    end
  5. Configure BGP and SD-WAN members and neighbors:

    config router bgp
        set as 65412
        config neighbor
            edit "10.10.15.253"
                set remote-as 65412
                set route-map-out "HUB_BAD"
                set route-map-out-preferable "HUB_CARRIER1"
                ...
            next
            edit "10.10.16.253"
                set remote-as 65412
                set route-map-out "HUB_BAD"
                set route-map-out-preferable "HUB_CARRIER2"
                ...
            next
        end
    end
    config system sdwan
        config members
            edit 1
                set interface "spoke21-p1"
                set cost 10
            next
            edit 2
                set interface "spoke22-p1"
                set cost 20
            next
        end
        config neighbor
            edit "10.10.15.253"
                set member 1
                set health-check "1"
                set sla-id 1
            next
            edit "10.10.16.253"
                set member 2
                set health-check "11"
                set sla-id 1
            next
        end
    end
To configure the hub:
  1. Configure the route maps:

    • Set the priority to 100 for routes with community 65000:1, indicating that they are in SLA for overlay 1.
    • Set the priority to 200 for routes with community 65000:2, indicating that they are in SLA for overlay 2.
    • Set the priority to 9999 for routes with community 65000:9999, indicating that they are out of SLA for any overlay.
    config router route-map
         edit "Set_Pri"
            config rule
                edit 1
                    set match-community "comm_65000:1"
                    set set-priority 100
                next
                edit 2
                    set match-community "comm_65000:2"
                    set set-priority 200
                next
                edit 3
                    set match-community "comm_65000:9999"
                    set set-priority 9999
                next
            end
        next
    end
  2. Configure BGP:

    config router bgp
        set as 65412
        config neighbor-group
            edit "advpn"
                set remote-as 65412
                set route-map-in "Set_Pri"
                ...
            next
            edit "advpn2"
                set remote-as 65412
                set route-map-in "Set_Pri"
                ...
            next
        end
        config neighbor-range
            edit 1
                set prefix 10.10.15.0 255.255.255.0
                set neighbor-group "advpn"
            next
            edit 2
                set prefix 10.10.16.0 255.255.255.0
                set neighbor-group "advpn2"
            next
        end
    end
To test the configuration:
  1. Check the routing tables on the spokes:

    Spoke 1:

    spoke-1 (root) # get router info routing-table all
    B*      0.0.0.0/0 [200/0] via 10.10.15.253 (recursive is directly connected, spoke11-p1), 00:01:17, [1/0]         // default route to hub
                             [200/0] via 10.10.16.253 (recursive is directly connected, spoke12-p1), 00:01:17, [1/0]
    B       9.0.0.0/24 [200/0] via 10.10.15.253 (recursive is directly connected, spoke11-p1), 00:01:17, [1/0]        // route to the server behind hub
                              [200/0] via 10.10.16.253 (recursive is directly connected, spoke12-p1), 00:01:17, [1/0]
    C       10.1.100.0/24 is directly connected, port2             // route to PC 1
    C       10.10.15.0/24 is directly connected, spoke11-p1        // overlay 1
    C       10.10.15.1/32 is directly connected, spoke11-p1
    C       10.10.16.0/24 is directly connected, spoke12-p1        // overlay 2
    C       10.10.16.1/32 is directly connected, spoke12-p1

    Spoke 2:

    spoke-2 (root) # get router info routing-table all
    B*      0.0.0.0/0 [200/0] via 10.10.15.253 (recursive is directly connected, spoke21-p1), 00:46:14, [1/0]         // default route to hub
                             [200/0] via 10.10.16.253 (recursive is directly connected, spoke22-p1), 00:46:14, [1/0]
    B       9.0.0.0/24 [200/0] via 10.10.15.253 (recursive is directly connected, spoke21-p1), 00:46:18, [1/0]        // route to the server behind hub
                              [200/0] via 10.10.16.253 (recursive is directly connected, spoke22-p1), 00:46:18, [1/0]
    C       10.10.15.0/24 is directly connected, spoke21-p1        // overlay 1
    C       10.10.15.2/32 is directly connected, spoke21-p1
    C       10.10.16.0/24 is directly connected, spoke22-p1        // overlay 2
    C       10.10.16.2/32 is directly connected, spoke22-p1
    C       192.168.5.0/24 is directly connected, port2            // route to PC 2
  2. Send traffic from PC 1 to PC 2 and trigger the shortcut:

    The IKE routes on the shortcut are directly injected based on the phase 2 selectors, and spoke-to-spoke traffic then goes directly through the shortcut instead of going through the hub.

    Spoke 1:

    spoke-1 (root) # get router info routing-table static
    S       192.168.5.0/24 [15/0] via spoke11-p1_0 tunnel 172.16.200.4 vrf 0, [1/0]
    S       192.168.6.0/24 [15/0] via spoke11-p1_0 tunnel 172.16.200.4 vrf 0, [1/0]
    S       192.168.7.0/24 [15/0] via spoke11-p1_0 tunnel 172.16.200.4 vrf 0, [1/0]
    spoke-1 (root) # diagnose sniffer packet any 'host 192.168.5.44' 4
    interfaces=[any]
    filters=[host 192.168.5.44]
    1.446306 port2 in 10.1.100.22 -> 192.168.5.44: icmp: echo request
    1.446327 spoke11-p1_0 out 10.1.100.22 -> 192.168.5.44: icmp: echo request
    1.446521 spoke11-p1_0 in 192.168.5.44 -> 10.1.100.22: icmp: echo reply
    1.446536 port2 out 192.168.5.44 -> 10.1.100.22: icmp: echo reply

    Spoke 2:

    spoke-2 (root) # get router info routing-table static
    S       10.1.100.0/24 [15/0] via spoke21-p1_0 tunnel 10.10.15.1 vrf 0, [1/0]
    S       10.2.100.0/24 [15/0] via spoke21-p1_0 tunnel 10.10.15.1 vrf 0, [1/0]
    S       10.3.100.0/24 [15/0] via spoke21-p1_0 tunnel 10.10.15.1 vrf 0, [1/0]
  3. Confirm that the overlays are in SLA on the spokes:

    Spoke 1:

    spoke-1 (root) # diagnose sys sdwan  neighbor
    Neighbor(10.10.15.253): member(1)role(standalone)
            Health-check(1:1)  sla-pass selected alive
    Neighbor(10.10.16.253): member(2)role(standalone)
            Health-check(11:1)  sla-pass selected alive

    Spoke 2:

    spoke-2 (root) # diagnose sys sdwan neighbor
    Neighbor(10.10.15.253): member(1)role(standalone)             
            Health-check(1:1)  sla-pass selected alive
    Neighbor(10.10.16.253): member(2)role(standalone)
            Health-check(11:1)  sla-pass selected alive
  4. On the hub, check that the routes received from the spokes have the expected priorities:

    hub (root) # diagnose ip route list | grep proto=11
    tab=254 vf=0 scope=0 type=1 proto=11 prio=100 0.0.0.0/0.0.0.0/0->10.1.100.0/24 pref=0.0.0.0 gwy=10.10.15.1 dev=101(hub-phase1)
    tab=254 vf=0 scope=0 type=1 proto=11 prio=200 0.0.0.0/0.0.0.0/0->10.1.100.0/24 pref=0.0.0.0 gwy=10.10.16.1 dev=102(hub2-phase1)
    tab=254 vf=0 scope=0 type=1 proto=11 prio=100 0.0.0.0/0.0.0.0/0->192.168.5.0/24 pref=0.0.0.0 gwy=10.10.15.2 dev=101(hub-phase1)
    tab=254 vf=0 scope=0 type=1 proto=11 prio=200 0.0.0.0/0.0.0.0/0->192.168.5.0/24 pref=0.0.0.0 gwy=10.10.16.2 dev=102(hub2-phase1)

    The priority set by the hub's route map is based on the community string received from the spoke. The route with a lower priority value is selected, so traffic to Spoke 1 goes out on the hub-phase1 tunnel:

    hub (root) # diagnose sniffer packet any 'host 9.0.0.2' 4
    interfaces=[any]
    filters=[host 9.0.0.2]
    2.735456 R190 in 9.0.0.2 -> 10.1.100.22: icmp: echo request
    2.735508 hub-phase1 out 9.0.0.2 -> 10.1.100.22: icmp: echo request
    2.735813 hub-phase1 in 10.1.100.22 -> 9.0.0.2: icmp: echo reply
    2.735854 R190 out 10.1.100.22 -> 9.0.0.2: icmp: echo reply
  5. If overlay 1 goes out of SLA, the priorities of the routes on the hub are updated and traffic from the hub to Spoke 1 goes through overlay 2:

    Spoke 1:

    spoke-1 (root) # diagnose sys sdwan  neighbor
    Neighbor(10.10.15.253): member(1)role(standalone)
            Health-check(1:1)  sla-fail alive
    Neighbor(10.10.16.253): member(2)role(standalone)
            Health-check(11:1)  sla-pass selected alive
    

    Spoke 2:

    spoke-2 (root) # diagnose sys sdwan neighbor
    Neighbor(10.10.15.253): member(1)role(standalone)
            Health-check(1:1)  sla-fail alive
    Neighbor(10.10.16.253): member(2)role(standalone)
            Health-check(11:1)  sla-pass selected alive
    

    Hub:

    hub (root) # diagnose ip route list | grep proto=11
    tab=254 vf=0 scope=0 type=1 proto=11 prio=200 0.0.0.0/0.0.0.0/0->10.1.100.0/24 pref=0.0.0.0 gwy=10.10.16.1 dev=102(hub2-phase1)
    tab=254 vf=0 scope=0 type=1 proto=11 prio=9999 0.0.0.0/0.0.0.0/0->10.1.100.0/24 pref=0.0.0.0 gwy=10.10.15.1 dev=101(hub-phase1)
    tab=254 vf=0 scope=0 type=1 proto=11 prio=200 0.0.0.0/0.0.0.0/0->192.168.5.0/24 pref=0.0.0.0 gwy=10.10.16.2 dev=102(hub2-phase1)
    tab=254 vf=0 scope=0 type=1 proto=11 prio=9999 0.0.0.0/0.0.0.0/0->192.168.5.0/24 pref=0.0.0.0 gwy=10.10.15.2 dev=101(hub-phase1)
    
    hub (root) # diagnose sniffer packet any 'host 9.0.0.2' 4
    interfaces=[any]
    filters=[host 9.0.0.2]
    3.550181 R190 in 9.0.0.2 -> 10.1.100.22: icmp: echo request
    3.550234 hub2-phase1 out 9.0.0.2 -> 10.1.100.22: icmp: echo request
    3.550713 hub2-phase1 in 10.1.100.22 -> 9.0.0.2: icmp: echo reply
    3.550735 R190 out 10.1.100.22 -> 9.0.0.2: icmp: echo reply
  6. Verify the service diagnostics on Spoke 1:
    # diagnose sys sdwan service
    
    Service(1): Address Mode(IPV4) flags=0x200 use-shortcut-sla
     Tie break: cfg
      Gen(4), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(sla), sla-compare-order
      Members(2):
        1: Seq_num(1 spoke11-p1), alive, sla(0x1), gid(0), cfg_order(0), local cost(10), selected
        2: Seq_num(2 spoke12-p1), alive, sla(0x1), gid(0), cfg_order(1), local cost(20), selected
      Src address(1):
            10.1.100.0-10.1.100.255
    
      Dst address(1):
            0.0.0.0-255.255.255.255
  7. Verify the service diagnostics on Spoke 2:
    # diagnose sys sdwan service
    
    Service(1): Address Mode(IPV4) flags=0x200 use-shortcut-sla
     Tie break: cfg
      Gen(2), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(sla), sla-compare-order
      Members(2):
        1: Seq_num(1 spoke21-p1), alive, sla(0x1), gid(0), cfg_order(0), local cost(10), selected
        2: Seq_num(2 spoke22-p1), alive, sla(0x1), gid(0), cfg_order(1), local cost(20), selected
      Src address(1):
            192.168.5.0-192.168.5.255
    
      Dst address(1):
            0.0.0.0-255.255.255.255
  8. Trigger shortcuts between Spoke 1 and Spoke 2:
    • Shortcuts spoke11-p1_1 and spoke11-p1_0 originate from spoke11-p1.

    • spoke11-p1_1 corresponds to spoke21-p1_0 on Spoke 2.

    • spoke11-p1_0 corresponds to spoke22-p1_0 on Spoke 2.

    Spoke 1:

    # diagnose sys sdwan service
    
    Service(1): Address Mode(IPV4) flags=0x200 use-shortcut-sla
     Tie break: cfg
      Gen(11), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(sla), sla-compare-order
      Member sub interface(4):
        3: seq_num(1), interface(spoke11-p1):
           1: spoke11-p1_0(80)
           2: spoke11-p1_1(81)
      Members(4):
        1: Seq_num(1 spoke11-p1_1), alive, sla(0x1), gid(0), remote cost(101), cfg_order(0), local cost(10), selected
        2: Seq_num(1 spoke11-p1_0), alive, sla(0x1), gid(0), remote cost(201), cfg_order(0), local cost(10), selected
        3: Seq_num(1 spoke11-p1), alive, sla(0x1), gid(0), cfg_order(0), local cost(10), selected
        4: Seq_num(2 spoke12-p1), alive, sla(0x1), gid(0), cfg_order(1), local cost(20), selected
      Src address(1):
            10.1.100.0-10.1.100.255
    
      Dst address(1):
            0.0.0.0-255.255.255.255

    Spoke 2:

    # diagnose sys sdwan service
    
    Service(1): Address Mode(IPV4) flags=0x200 use-shortcut-sla
     Tie break: cfg
      Gen(15), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(sla), sla-compare-order
      Member sub interface(4):
        2: seq_num(1), interface(spoke21-p1):
           1: spoke21-p1_0(75)
        4: seq_num(2), interface(spoke22-p1):
           1: spoke22-p1_0(74)
      Members(4):
        1: Seq_num(1 spoke21-p1_0), alive, sla(0x1), gid(0), remote cost(11), cfg_order(0), local cost(10), selected
        2: Seq_num(1 spoke21-p1), alive, sla(0x1), gid(0), cfg_order(0), local cost(10), selected
        3: Seq_num(2 spoke22-p1_0), alive, sla(0x1), gid(0), remote cost(11), cfg_order(1), local cost(20), selected
        4: Seq_num(2 spoke22-p1), alive, sla(0x1), gid(0), cfg_order(1), local cost(20), selected
      Src address(1):
            192.168.5.0-192.168.5.255
    
      Dst address(1):
            0.0.0.0-255.255.255.255

    The spoke11-p1_1 shortcut on Spoke 1 is preferred over spoke11-p1_0 due to the lower remote link cost of 101 when they have the same local SD-WAN member cost of 10.

  9. Verify the policy route list on Spoke 1:
    # diagnose firewall proute list
    list route policy info(vf=root):
    
    id=2131755009(0x7f100001) vwl_service=1(1) vwl_mbr_seq=1 1 1 2 dscp_tag=0xfc 0xfc flags=0x0 tos=0x00 tos_mask=0x00 protocol=0 sport=0-65535 iif=0(any) dport=1-65535 path(4) oif=81(spoke11-p1_1) oif=80(spoke11-p1_0) oif=54(spoke11-p1) oif=55(spoke12-p1)
    source(1): 10.1.100.0-10.1.100.255
    destination(1): 0.0.0.0-255.255.255.255
    hit_count=176 last_used=2022-07-12 11:56:08

SD-WAN in large scale deployments

Phase 2 selectors can be used to inject IKE routes on the ADVPN shortcut tunnel. When configuration method (mode-cfg) is enabled in IPsec phase 1 configuration, enabling mode-cfg-allow-client-selector allows custom phase 2 selectors to be configured. By also enabling the addition of a route to the peer destination selector (add-route) in the phase 1 configuration, IKE routes based on the phase 2 selectors can be injected. This means that routes do not need to be reflected on the hub to propagate them between spokes, avoiding possible BGP daemon process load issues and improving network scalability in a large-scale ADVPN network.

Route map rules can apply priorities to BGP routes. On the hub, priorities can be set in a route map's rules, and the route map can be applied on BGP routes. This allows the hub to mark the preferred path learned from the spokes with a higher priority, instead of using multiple SD-WAN policy routes on the hub. When a preferred outbound route map (route-map-out-preferable) is also configured in an SD-WAN neighbor on the spoke, deploying SD-WAN rules on the hub to steer traffic from the hub to a spoke is unnecessary.

SD-WAN members' local cost can be exchanged on the ADVPN shortcut tunnel so that spokes can use the remote cost as tiebreak to select a preferred shortcut. If multiple shortcuts originate from the same member to different members on the same remote spoke, then the remote cost on the shortcuts is used as the tiebreak to decide which shortcut is preferred.

In this example, SD-WAN is configured on an ADVPN network with a BGP neighbor per overlay.

Instead of reflecting BGP routes with the route-reflector on the hub, when the shortcuts are triggered, IKE routes on the shortcuts are directly injected based on the configured phase 2 selectors to allow routes to be exchanged between spokes.

Routes between the hub and the spokes are exchanged by BGP, and the spokes use the default route to send spoke-to-spoke traffic to the hub and trigger the shortcuts.

Instead of configuring SD-WAN rules on the hub, different priorities are configured on the BGP routes by matching different BGP communities to steer traffic from the hub to the spokes.

To configure Spoke 1:
  1. Configure phase 1:

    config vpn ipsec phase1-interface
        edit "spoke11-p1"
            ...
            set ike-version 2
            set net-device enable
            set add-route enable
            set mode-cfg enable
            set auto-discovery-receiver enable
            set mode-cfg-allow-client-selector enable
            set link-cost 11              
            ...
        next
        edit "spoke12-p1"
           ...
            set ike-version 2
            set net-device enable
            set add-route enable
            set mode-cfg enable
            set auto-discovery-receiver enable
            set mode-cfg-allow-client-selector enable
            set link-cost 21
        next
    end
  2. Configure phase 2:

    config vpn ipsec phase2-interface
        edit "spoke11-p2"
            ...
            set src-name "LAN_Net"
            set dst-name "all"
        next
        edit "spoke12-p2"
            ...
            set src-name "LAN_Net"
            set dst-name "all"
        next
    end
  3. Configure an address group:

    Spoke 1 uses LAN subnet 10.1-3.100.0/24.

    config firewall addrgrp
        edit "LAN_Net"
            set member "10.1.100.0" "10.2.100.0" "10.3.100.0"
        next
    end
  4. Configure route maps:

    • If overlay 1 to the hub is in SLA, attach "65000:1" to the BGP routes advertised to the hub over overlay 1.
    • If overlay 2 to the hub is in SLA, attach "65000:2" to the BGP routes advertised to the hub over overlay 2.
    • If any overlay to the hub is out of SLA, attach "65000:9999" to the BGP routes advertised to the hub over any overlay.
    config router route-map
        edit "HUB_CARRIER1"
            config rule
                edit 1
                    set set-community "65000:1"
                    ...
                next
            end
            ...
        next
        edit "HUB_CARRIER2"
            config rule
                edit 1
                    set set-community "65000:2"
                    ...
                next
            end
            ...
        next
        edit "HUB_BAD"
            config rule
                edit 1
                    set set-community "65000:9999"
                    ...
                next
            end
            ...
        next
    end
  5. Configure BGP and SD-WAN members and neighbors:

    config router bgp
        set as 65412
        config neighbor
            edit "10.10.15.253"
                set remote-as 65412
                set route-map-out "HUB_BAD"
                set route-map-out-preferable "HUB_CARRIER1"
                ...
            next
            edit "10.10.16.253"
                set remote-as 65412
                set route-map-out "HUB_BAD"
                set route-map-out-preferable "HUB_CARRIER2"
                ...
            next
        end
    end
    config system sdwan
        config members
            edit 1
                set interface "spoke11-p1"
                set cost 10
            next
            edit 2
                set interface "spoke12-p1"
                set cost 20
            next
        end
        config neighbor
            edit "10.10.15.253"
                set member 1
                set health-check "1"
                set sla-id 1
            next
            edit "10.10.16.253"
                set member 2
                set health-check "11"
                set sla-id 1
            next
        end
    end
To configure Spoke 2:
  1. Configure phase 1:

    config vpn ipsec phase1-interface
        edit "spoke21-p1"
            ...
            set ike-version 2
            set net-device enable
            set add-route enable
            set mode-cfg enable
            set auto-discovery-receiver enable
            set mode-cfg-allow-client-selector enable
            set link-cost 101              
            ...
        next
        edit "spoke22-p1"
           ...
            set ike-version 2
            set net-device enable
            set add-route enable
            set mode-cfg enable
            set auto-discovery-receiver enable
            set mode-cfg-allow-client-selector enable
            set link-cost 201
        next
    end
  2. Configure phase 2:

    config vpn ipsec phase2-interface
        edit "spoke21-p2"
            ...
            set src-name "LAN_Net"
            set dst-name "all"
        next
        edit "spoke22-p2"
            ...
            set src-name "LAN_Net"
            set dst-name "all"
        next
    end
  3. Configure an address group:

    Spoke 2 uses LAN subnet 192.168.5-7.0/24.

    config firewall addrgrp
        edit "LAN_Net"
            set member "192.168.5.0" "192.168.6.0" "192.168.7.0"
        next
    end
  4. Configure route maps:

    • If overlay 1 to the hub is in SLA, attach "65000:1" to the BGP routes advertised to the hub over overlay 1.
    • If overlay 2 to the hub is in SLA, attach "65000:2" to the BGP routes advertised to the hub over overlay 2.
    • If any overlay to the hub is out of SLA, attach "65000:9999" to the BGP routes advertised to the hub over any overlay.
    config router route-map
        edit "HUB_CARRIER1"
            config rule
                edit 1
                    set set-community "65000:1"
                    ...
                next
            end
            ...
        next
        edit "HUB_CARRIER2"
            config rule
                edit 1
                    set set-community "65000:2"
                    ...
                next
            end
            ...
        next
        edit "HUB_BAD"
            config rule
                edit 1
                    set set-community "65000:9999"
                    ...
                next
            end
            ...
        next
    end
  5. Configure BGP and SD-WAN members and neighbors:

    config router bgp
        set as 65412
        config neighbor
            edit "10.10.15.253"
                set remote-as 65412
                set route-map-out "HUB_BAD"
                set route-map-out-preferable "HUB_CARRIER1"
                ...
            next
            edit "10.10.16.253"
                set remote-as 65412
                set route-map-out "HUB_BAD"
                set route-map-out-preferable "HUB_CARRIER2"
                ...
            next
        end
    end
    config system sdwan
        config members
            edit 1
                set interface "spoke21-p1"
                set cost 10
            next
            edit 2
                set interface "spoke22-p1"
                set cost 20
            next
        end
        config neighbor
            edit "10.10.15.253"
                set member 1
                set health-check "1"
                set sla-id 1
            next
            edit "10.10.16.253"
                set member 2
                set health-check "11"
                set sla-id 1
            next
        end
    end
To configure the hub:
  1. Configure the route maps:

    • Set the priority to 100 for routes with community 65000:1, indicating that they are in SLA for overlay 1.
    • Set the priority to 200 for routes with community 65000:2, indicating that they are in SLA for overlay 2.
    • Set the priority to 9999 for routes with community 65000:9999, indicating that they are out of SLA for any overlay.
    config router route-map
         edit "Set_Pri"
            config rule
                edit 1
                    set match-community "comm_65000:1"
                    set set-priority 100
                next
                edit 2
                    set match-community "comm_65000:2"
                    set set-priority 200
                next
                edit 3
                    set match-community "comm_65000:9999"
                    set set-priority 9999
                next
            end
        next
    end
  2. Configure BGP:

    config router bgp
        set as 65412
        config neighbor-group
            edit "advpn"
                set remote-as 65412
                set route-map-in "Set_Pri"
                ...
            next
            edit "advpn2"
                set remote-as 65412
                set route-map-in "Set_Pri"
                ...
            next
        end
        config neighbor-range
            edit 1
                set prefix 10.10.15.0 255.255.255.0
                set neighbor-group "advpn"
            next
            edit 2
                set prefix 10.10.16.0 255.255.255.0
                set neighbor-group "advpn2"
            next
        end
    end
To test the configuration:
  1. Check the routing tables on the spokes:

    Spoke 1:

    spoke-1 (root) # get router info routing-table all
    B*      0.0.0.0/0 [200/0] via 10.10.15.253 (recursive is directly connected, spoke11-p1), 00:01:17, [1/0]         // default route to hub
                             [200/0] via 10.10.16.253 (recursive is directly connected, spoke12-p1), 00:01:17, [1/0]
    B       9.0.0.0/24 [200/0] via 10.10.15.253 (recursive is directly connected, spoke11-p1), 00:01:17, [1/0]        // route to the server behind hub
                              [200/0] via 10.10.16.253 (recursive is directly connected, spoke12-p1), 00:01:17, [1/0]
    C       10.1.100.0/24 is directly connected, port2             // route to PC 1
    C       10.10.15.0/24 is directly connected, spoke11-p1        // overlay 1
    C       10.10.15.1/32 is directly connected, spoke11-p1
    C       10.10.16.0/24 is directly connected, spoke12-p1        // overlay 2
    C       10.10.16.1/32 is directly connected, spoke12-p1

    Spoke 2:

    spoke-2 (root) # get router info routing-table all
    B*      0.0.0.0/0 [200/0] via 10.10.15.253 (recursive is directly connected, spoke21-p1), 00:46:14, [1/0]         // default route to hub
                             [200/0] via 10.10.16.253 (recursive is directly connected, spoke22-p1), 00:46:14, [1/0]
    B       9.0.0.0/24 [200/0] via 10.10.15.253 (recursive is directly connected, spoke21-p1), 00:46:18, [1/0]        // route to the server behind hub
                              [200/0] via 10.10.16.253 (recursive is directly connected, spoke22-p1), 00:46:18, [1/0]
    C       10.10.15.0/24 is directly connected, spoke21-p1        // overlay 1
    C       10.10.15.2/32 is directly connected, spoke21-p1
    C       10.10.16.0/24 is directly connected, spoke22-p1        // overlay 2
    C       10.10.16.2/32 is directly connected, spoke22-p1
    C       192.168.5.0/24 is directly connected, port2            // route to PC 2
  2. Send traffic from PC 1 to PC 2 and trigger the shortcut:

    The IKE routes on the shortcut are directly injected based on the phase 2 selectors, and spoke-to-spoke traffic then goes directly through the shortcut instead of going through the hub.

    Spoke 1:

    spoke-1 (root) # get router info routing-table static
    S       192.168.5.0/24 [15/0] via spoke11-p1_0 tunnel 172.16.200.4 vrf 0, [1/0]
    S       192.168.6.0/24 [15/0] via spoke11-p1_0 tunnel 172.16.200.4 vrf 0, [1/0]
    S       192.168.7.0/24 [15/0] via spoke11-p1_0 tunnel 172.16.200.4 vrf 0, [1/0]
    spoke-1 (root) # diagnose sniffer packet any 'host 192.168.5.44' 4
    interfaces=[any]
    filters=[host 192.168.5.44]
    1.446306 port2 in 10.1.100.22 -> 192.168.5.44: icmp: echo request
    1.446327 spoke11-p1_0 out 10.1.100.22 -> 192.168.5.44: icmp: echo request
    1.446521 spoke11-p1_0 in 192.168.5.44 -> 10.1.100.22: icmp: echo reply
    1.446536 port2 out 192.168.5.44 -> 10.1.100.22: icmp: echo reply

    Spoke 2:

    spoke-2 (root) # get router info routing-table static
    S       10.1.100.0/24 [15/0] via spoke21-p1_0 tunnel 10.10.15.1 vrf 0, [1/0]
    S       10.2.100.0/24 [15/0] via spoke21-p1_0 tunnel 10.10.15.1 vrf 0, [1/0]
    S       10.3.100.0/24 [15/0] via spoke21-p1_0 tunnel 10.10.15.1 vrf 0, [1/0]
  3. Confirm that the overlays are in SLA on the spokes:

    Spoke 1:

    spoke-1 (root) # diagnose sys sdwan  neighbor
    Neighbor(10.10.15.253): member(1)role(standalone)
            Health-check(1:1)  sla-pass selected alive
    Neighbor(10.10.16.253): member(2)role(standalone)
            Health-check(11:1)  sla-pass selected alive

    Spoke 2:

    spoke-2 (root) # diagnose sys sdwan neighbor
    Neighbor(10.10.15.253): member(1)role(standalone)             
            Health-check(1:1)  sla-pass selected alive
    Neighbor(10.10.16.253): member(2)role(standalone)
            Health-check(11:1)  sla-pass selected alive
  4. On the hub, check that the routes received from the spokes have the expected priorities:

    hub (root) # diagnose ip route list | grep proto=11
    tab=254 vf=0 scope=0 type=1 proto=11 prio=100 0.0.0.0/0.0.0.0/0->10.1.100.0/24 pref=0.0.0.0 gwy=10.10.15.1 dev=101(hub-phase1)
    tab=254 vf=0 scope=0 type=1 proto=11 prio=200 0.0.0.0/0.0.0.0/0->10.1.100.0/24 pref=0.0.0.0 gwy=10.10.16.1 dev=102(hub2-phase1)
    tab=254 vf=0 scope=0 type=1 proto=11 prio=100 0.0.0.0/0.0.0.0/0->192.168.5.0/24 pref=0.0.0.0 gwy=10.10.15.2 dev=101(hub-phase1)
    tab=254 vf=0 scope=0 type=1 proto=11 prio=200 0.0.0.0/0.0.0.0/0->192.168.5.0/24 pref=0.0.0.0 gwy=10.10.16.2 dev=102(hub2-phase1)

    The priority set by the hub's route map is based on the community string received from the spoke. The route with a lower priority value is selected, so traffic to Spoke 1 goes out on the hub-phase1 tunnel:

    hub (root) # diagnose sniffer packet any 'host 9.0.0.2' 4
    interfaces=[any]
    filters=[host 9.0.0.2]
    2.735456 R190 in 9.0.0.2 -> 10.1.100.22: icmp: echo request
    2.735508 hub-phase1 out 9.0.0.2 -> 10.1.100.22: icmp: echo request
    2.735813 hub-phase1 in 10.1.100.22 -> 9.0.0.2: icmp: echo reply
    2.735854 R190 out 10.1.100.22 -> 9.0.0.2: icmp: echo reply
  5. If overlay 1 goes out of SLA, the priorities of the routes on the hub are updated and traffic from the hub to Spoke 1 goes through overlay 2:

    Spoke 1:

    spoke-1 (root) # diagnose sys sdwan  neighbor
    Neighbor(10.10.15.253): member(1)role(standalone)
            Health-check(1:1)  sla-fail alive
    Neighbor(10.10.16.253): member(2)role(standalone)
            Health-check(11:1)  sla-pass selected alive
    

    Spoke 2:

    spoke-2 (root) # diagnose sys sdwan neighbor
    Neighbor(10.10.15.253): member(1)role(standalone)
            Health-check(1:1)  sla-fail alive
    Neighbor(10.10.16.253): member(2)role(standalone)
            Health-check(11:1)  sla-pass selected alive
    

    Hub:

    hub (root) # diagnose ip route list | grep proto=11
    tab=254 vf=0 scope=0 type=1 proto=11 prio=200 0.0.0.0/0.0.0.0/0->10.1.100.0/24 pref=0.0.0.0 gwy=10.10.16.1 dev=102(hub2-phase1)
    tab=254 vf=0 scope=0 type=1 proto=11 prio=9999 0.0.0.0/0.0.0.0/0->10.1.100.0/24 pref=0.0.0.0 gwy=10.10.15.1 dev=101(hub-phase1)
    tab=254 vf=0 scope=0 type=1 proto=11 prio=200 0.0.0.0/0.0.0.0/0->192.168.5.0/24 pref=0.0.0.0 gwy=10.10.16.2 dev=102(hub2-phase1)
    tab=254 vf=0 scope=0 type=1 proto=11 prio=9999 0.0.0.0/0.0.0.0/0->192.168.5.0/24 pref=0.0.0.0 gwy=10.10.15.2 dev=101(hub-phase1)
    
    hub (root) # diagnose sniffer packet any 'host 9.0.0.2' 4
    interfaces=[any]
    filters=[host 9.0.0.2]
    3.550181 R190 in 9.0.0.2 -> 10.1.100.22: icmp: echo request
    3.550234 hub2-phase1 out 9.0.0.2 -> 10.1.100.22: icmp: echo request
    3.550713 hub2-phase1 in 10.1.100.22 -> 9.0.0.2: icmp: echo reply
    3.550735 R190 out 10.1.100.22 -> 9.0.0.2: icmp: echo reply
  6. Verify the service diagnostics on Spoke 1:
    # diagnose sys sdwan service
    
    Service(1): Address Mode(IPV4) flags=0x200 use-shortcut-sla
     Tie break: cfg
      Gen(4), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(sla), sla-compare-order
      Members(2):
        1: Seq_num(1 spoke11-p1), alive, sla(0x1), gid(0), cfg_order(0), local cost(10), selected
        2: Seq_num(2 spoke12-p1), alive, sla(0x1), gid(0), cfg_order(1), local cost(20), selected
      Src address(1):
            10.1.100.0-10.1.100.255
    
      Dst address(1):
            0.0.0.0-255.255.255.255
  7. Verify the service diagnostics on Spoke 2:
    # diagnose sys sdwan service
    
    Service(1): Address Mode(IPV4) flags=0x200 use-shortcut-sla
     Tie break: cfg
      Gen(2), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(sla), sla-compare-order
      Members(2):
        1: Seq_num(1 spoke21-p1), alive, sla(0x1), gid(0), cfg_order(0), local cost(10), selected
        2: Seq_num(2 spoke22-p1), alive, sla(0x1), gid(0), cfg_order(1), local cost(20), selected
      Src address(1):
            192.168.5.0-192.168.5.255
    
      Dst address(1):
            0.0.0.0-255.255.255.255
  8. Trigger shortcuts between Spoke 1 and Spoke 2:
    • Shortcuts spoke11-p1_1 and spoke11-p1_0 originate from spoke11-p1.

    • spoke11-p1_1 corresponds to spoke21-p1_0 on Spoke 2.

    • spoke11-p1_0 corresponds to spoke22-p1_0 on Spoke 2.

    Spoke 1:

    # diagnose sys sdwan service
    
    Service(1): Address Mode(IPV4) flags=0x200 use-shortcut-sla
     Tie break: cfg
      Gen(11), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(sla), sla-compare-order
      Member sub interface(4):
        3: seq_num(1), interface(spoke11-p1):
           1: spoke11-p1_0(80)
           2: spoke11-p1_1(81)
      Members(4):
        1: Seq_num(1 spoke11-p1_1), alive, sla(0x1), gid(0), remote cost(101), cfg_order(0), local cost(10), selected
        2: Seq_num(1 spoke11-p1_0), alive, sla(0x1), gid(0), remote cost(201), cfg_order(0), local cost(10), selected
        3: Seq_num(1 spoke11-p1), alive, sla(0x1), gid(0), cfg_order(0), local cost(10), selected
        4: Seq_num(2 spoke12-p1), alive, sla(0x1), gid(0), cfg_order(1), local cost(20), selected
      Src address(1):
            10.1.100.0-10.1.100.255
    
      Dst address(1):
            0.0.0.0-255.255.255.255

    Spoke 2:

    # diagnose sys sdwan service
    
    Service(1): Address Mode(IPV4) flags=0x200 use-shortcut-sla
     Tie break: cfg
      Gen(15), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(sla), sla-compare-order
      Member sub interface(4):
        2: seq_num(1), interface(spoke21-p1):
           1: spoke21-p1_0(75)
        4: seq_num(2), interface(spoke22-p1):
           1: spoke22-p1_0(74)
      Members(4):
        1: Seq_num(1 spoke21-p1_0), alive, sla(0x1), gid(0), remote cost(11), cfg_order(0), local cost(10), selected
        2: Seq_num(1 spoke21-p1), alive, sla(0x1), gid(0), cfg_order(0), local cost(10), selected
        3: Seq_num(2 spoke22-p1_0), alive, sla(0x1), gid(0), remote cost(11), cfg_order(1), local cost(20), selected
        4: Seq_num(2 spoke22-p1), alive, sla(0x1), gid(0), cfg_order(1), local cost(20), selected
      Src address(1):
            192.168.5.0-192.168.5.255
    
      Dst address(1):
            0.0.0.0-255.255.255.255

    The spoke11-p1_1 shortcut on Spoke 1 is preferred over spoke11-p1_0 due to the lower remote link cost of 101 when they have the same local SD-WAN member cost of 10.

  9. Verify the policy route list on Spoke 1:
    # diagnose firewall proute list
    list route policy info(vf=root):
    
    id=2131755009(0x7f100001) vwl_service=1(1) vwl_mbr_seq=1 1 1 2 dscp_tag=0xfc 0xfc flags=0x0 tos=0x00 tos_mask=0x00 protocol=0 sport=0-65535 iif=0(any) dport=1-65535 path(4) oif=81(spoke11-p1_1) oif=80(spoke11-p1_0) oif=54(spoke11-p1) oif=55(spoke12-p1)
    source(1): 10.1.100.0-10.1.100.255
    destination(1): 0.0.0.0-255.255.255.255
    hit_count=176 last_used=2022-07-12 11:56:08