Embedded SD-WAN SLA priorities in ICMP probes

In SD-WAN hub-and-spoke topologies, each spoke can embed an SLA priority (priority-in-sla and priority-out-sla) in ICMP probes and send them to the hub. The hub can use the received SLA priorities from each spoke to manage route priority for hub-to-spoke traffic.

SLA status can also be embedded in ICMP probes. See Embedded SD-WAN SLA status in ICMP probes for more information.

Spoke-initiated speed tests can also use the embedded information. When a spoke continually embeds an out-of SLA priority into ICMP probes on the overlay, the hub can use the received out-of SLA priority information to manage route priorities and detour hub-to-spoke user traffic to other tunnels.

For configuration on a spoke, the config system sdwan command includes these options:

config system sdwan
    config members
        edit <entry>
          set priority-in-sla <integer>
          set priority-out-sla <integer>
        next
    end 
end  

set priority-in-sla <integer>	Preferred priority of routes to this member when this member is in SLA (0 - 65535, default = 0).
set priority-out-sla <integer>	Preferred priority of routes to this member when this member is out of SLA (0 - 65535, default = 0).

Example

The SD-WAN topology with ADVPN and BGP neighbor on loopback is used for the following two examples:

• Path selection based on embedded SLA priorities

• Speed test rerouting based on embedded SLA priorities

The examples use the following SD-WAN settings on the spoke:

SD-WAN settings on the spoke relevant to the examples:

config system sdwan
    set speedtest-bypass-routing enable
    config members
        edit <id>
            set priority-in-sla <value>
            set priority-out-sla <value>
        next
    end
    config health-check
        edit <name>
            set embed-measured-health enable 
            set sla-id-redistribute <id>
            config sla
                edit 1
                   <desired SLA thresholds>                                                   
                next
            end
        next
    end
end

SD-WAN settings on the hub relevant to the examples:

config system sdwan
    config health-check
        edit <name>
            set detect-mode remote
            set sla-id-redistribute <id>
                config sla
                    edit <id>
                        set link-cost-factor remote
                    next
                end
            next
        end
    next
end

Path selection based on embedded SLA priorities

In this example, spokes are configured to embed the SLA priority in ICMP probes, and the hub is configured to use the information. With this configuration:

• The spoke's overlay priority on each overlay is embedded in the ICMP probes and transported to the hub.

• The hub sets priorities on IKE routes over different overlays based on the received overlay priorities.

• On the hub, BGP routes, which are used to guide hub-to-spoke traffic, rely on IKE routes to be resolved to tunnels and inherit the priorities from the IKE routes.

• The hub can steer traffic to spokes by resolved BGP routes with different inherited priorities.

To embed SD-WAN SLA status and priorities in ICMP probes:

1. Configure SD-WAN and BGP on Spoke-1 (Branch1_A_FGT):

   1. Configure SD-WAN:

      config system sdwan
          set status enable
          set speedtest-bypass-routing enable
          config zone
              edit "overlay"
              next
          end
          config members
              edit 4
                  set interface "H1_T11"
                  set zone "overlay"
                  set source 172.31.0.65
                  set priority 10
                  set priority-in-sla 10
                  set priority-out-sla 20
              next
              edit 5
                  set interface "H1_T22"
                  set zone "overlay"
                  set source 172.31.0.65
                  set priority 10
                  set priority-in-sla 15
                  set priority-out-sla 25
              next
          end
          config health-check
              edit "HUB"
                  set server "172.31.100.100"
                  set embed-measured-health enable
                  set sla-id-redistribute 1                           
                  set members 4 5
                  config sla
                      edit 1
                          set link-cost-factor latency
                          set latency-threshold 50
                      next
                  end
              next
          end
      end

   2. Configure BGP:

      config router bgp
          set as 65001
          set router-id 172.31.0.65
          ......
          config neighbor
              edit "172.31.0.1"
                  ......
                  set remote-as 65001
                  set update-source "Loopback0"
              next
          end
          config network
              edit 1
                  set prefix 10.0.3.0 255.255.255.0
              next
          end
       ......
      end

   3. View the health-check settings:

      # diagnose sys sdwan health-check
      Health Check(HUB):
      Seq(4 H1_T11): state(alive), packet-loss(0.000%) latency(0.246), jitter(0.015), mos(4.404), bandwidth-up(999998), bandwidth-dw(999996), bandwidth-bi(1999994) sla_map=0x1
      Seq(5 H1_T22): state(alive), packet-loss(0.000%) latency(0.197), jitter(0.005), mos(4.404), bandwidth-up(999998), bandwidth-dw(999997), bandwidth-bi(1999995) sla_map=0x1

2. Configure SD-WAN and BGP on Spoke-2 (Branch2_FGT):

   1. Configure SD-WAN:

      config system sdwan
          set status enable
          set speedtest-bypass-routing enable
          config zone
              edit "overlay"
              next
          end
          config members
              edit 4
                  set interface "H1_T11"
                  set zone "overlay"
                  set source 172.31.0.66
                  set priority 10
                  set priority-in-sla 30
                  set priority-out-sla 40
              next
              edit 5
                  set interface "H1_T22"
                  set zone "overlay"
                  set source 172.31.0.66
                  set priority 10
                  set priority-in-sla 35
                  set priority-out-sla 45
              next
          end
          config health-check
              edit "HUB"
                  set server "172.31.100.100"
                  set embed-measured-health enable
                  set sla-id-redistribute 1
                  set members 4 5
                  config sla
                      edit 1
                          set link-cost-factor latency
                          set latency-threshold 70
                      next
                  end
              next
          end
      end

   2. Configure BGP:

      config router bgp
          set as 65001
          set router-id 172.31.0.66
          ......
          config neighbor
              edit "172.31.0.1"
                  ......
                  set remote-as 65001        
                  set update-source "Loopback0"
              next
          end
          config network
              edit 1
                  set prefix 10.0.4.0 255.255.255.0
              next
          end
          ......
      end

   3. View the health-check settings:

      # diagnose sys sdwan health-check
      Health Check(HUB):
      Seq(4 H1_T11): state(alive), packet-loss(0.000%) latency(0.230), jitter(0.040), mos(4.404), bandwidth-up(999998), bandwidth-dw(999996), bandwidth-bi(1999994) sla_map=0x1
      Seq(5 H1_T22): state(alive), packet-loss(0.000%) latency(0.188), jitter(0.007), mos(4.404), bandwidth-up(999998), bandwidth-dw(999996), bandwidth-bi(1999994) sla_map=0x1

3. Configure SD-WAN and BGP on the Hub (DC1_A_FGT):

   1. Configure SD-WAN:

      config system sdwan
          set status enable
          config zone
              edit "overlay"
              next
          end
          config members
              edit 1
                  set interface "EDGE_T1"
                  set zone "overlay"
              next
              edit 2
                  set interface "EDGE_T2"
                  set zone "overlay"
              next
          end
          config health-check
              edit "Remote_HC"
                  set detect-mode remote
                  set sla-id-redistribute 1
                  set members 1 2
                  config sla
                      edit 1
                          set link-cost-factor remote           
                      next
                  end
              next
          end
      end

   2. Configure BGP:

      config router bgp
          set as 65001
          set router-id 172.31.0.1
          set recursive-inherit-priority enable
          ......
          config neighbor-group
              edit "EDGE"
                  set remote-as 65001
                  set update-source "Loopback0"
                  set route-reflector-client enable
              next
          end
          config neighbor-range
              edit 1
                  set prefix 172.31.0.64 255.255.255.192
                  set neighbor-group "EDGE"
              next
          end
      ......
      end

   3. View the health-check settings:

      The following example shows:

      • rmt_ver=2 indicates that SLA information, SLA status, and overlay priority have been received.

      • rmt_sla=out indicates that received SLA status is out of SLA.

      • rmt_sla=in indicates that received SLA status is in SLA.

      • rmt_prio indicates the received overlay priority value.

      • EDGE_T1_0 is to H1_T11 on Spoke-1, and EDGE_T1_1 is to H1_T11 on Spoke-2.

      • EDGE_T2_1 is to H1_T22 on Spoke-1, and EDGE_T1_0 is to H1_T22 on Spoke-2.

      # diagnose sys sdwan health-check remote
      
      Remote Health Check: Remote_HC(1)
        Passive remote statistics of EDGE_T1(46):
          EDGE_T1_0(10.0.0.20): timestamp=06-12 14:04:29, src=172.31.0.65, latency=0.247, jitter=0.028, pktloss=0.000%, mos=4.404, SLA id=1(pass), rmt_ver=2, rmt_sla=in, rmt_prio=10
          EDGE_T1_1(172.31.0.66): timestamp=06-12 14:04:30, src=172.31.0.66, latency=0.246, jitter=0.031, pktloss=0.000%, mos=4.404, SLA id=1(pass), rmt_ver=2, rmt_sla=in, rmt_prio=30
      
      Remote Health Check: Remote_HC(2)
        Passive remote statistics of EDGE_T2(47):
          EDGE_T2_0(10.0.0.15): timestamp=06-12 14:04:30, src=172.31.0.66, latency=0.191, jitter=0.008, pktloss=0.000%, mos=4.404, SLA id=1(pass), rmt_ver=2, rmt_sla=in, rmt_prio=35
          EDGE_T2_1(172.31.0.65): timestamp=06-12 14:04:29, src=172.31.0.65, latency=0.201, jitter=0.008, pktloss=0.000%, mos=4.404, SLA id=1(pass), rmt_ver=2, rmt_sla=in, rmt_prio=15

4. After the spokes' overlay priorities are embedded in ICMP probes and transported to the hub, view the routing tables on the hub.

   The hub sets overlay priorities on IKE routes over EDGE_T1 and EDGE_T2. Meanwhile, recursively resolved BGP routes inherit the priorities from those IKE routes.

   1. On the hub, get the static routing table:

      # get router info routing-table static
      Routing table for VRF=0
      S       172.31.0.65/32 [15/0] via EDGE_T1 tunnel 10.0.0.20, [10/0]
                             [15/0] via EDGE_T2 tunnel 172.31.0.65, [15/0]
      S       172.31.0.66/32 [15/0] via EDGE_T1 tunnel 172.31.0.66, [30/0]
                             [15/0] via EDGE_T2 tunnel 10.0.0.15, [35/0]

   2. On the hub, get the BGP routing table:

      # get router info routing-table bgp
      Routing table for VRF=0
      B       10.0.3.0/24 [200/0] via 172.31.0.65 (recursive via EDGE_T1 tunnel 10.0.0.20 [10]), 01:15:46
                                                  (recursive via EDGE_T2 tunnel 172.31.0.65 [15]), 01:15:46, [1/0]
      B       10.0.4.0/24 [200/0] via 172.31.0.66 (recursive via EDGE_T1 tunnel 172.31.0.66 [30]), 01:13:46
                                                  (recursive via EDGE_T2 tunnel 10.0.0.15 [35]), 01:13:46, [1/0]

5. Change the latency on Spoke-1 and Spoke-2, and view the results.

   1. On Spoke-1, increase H1_T11's latency to 60 to make it out of SLA.

   2. On Spoke-2, increase H1_T11's latency to 80 to make it out of SLA.

   3. On Spoke-1, run a health check:

      Branch1_A_FGT (root) (Interim)# diagnose sys sdwan  health-check
      Health Check(HUB):
      Seq(4 H1_T11): state(alive), packet-loss(0.000%) latency(60.247), jitter(0.036), mos(4.373), bandwidth-up(999998), bandwidth-dw(999997), bandwidth-bi(1999995) sla_map=0x0
      Seq(5 H1_T22): state(alive), packet-loss(0.000%) latency(0.218), jitter(0.016), mos(4.404), bandwidth-up(999998), bandwidth-dw(999998), bandwidth-bi(1999996) sla_map=0x
      

   4. On Spoke-2, run a health check:

      Branch2_FGT (root) (Interim)# diagnose sys sdwan  health-check
      Health Check(HUB):
      Seq(4 H1_T11): state(alive), packet-loss(0.000%) latency(80.217), jitter(0.022), mos(4.361), bandwidth-up(999998), bandwidth-dw(999998), bandwidth-bi(1999996) sla_map=0x0
      Seq(5 H1_T22): state(alive), packet-loss(0.000%) latency(0.202), jitter(0.016), mos(4.404), bandwidth-up(999998), bandwidth-dw(999997), bandwidth-bi(1999995) sla_map=0x1

6. After the hub receives the updated overlay priorities, run a health check on the hub and view the routing tables.

   The hub has updated the route priorities.

   1. Run a health check:

      # diagnose sys sdwan health-check remote
      
      Remote Health Check: Remote_HC(1)
        Passive remote statistics of EDGE_T1(46):
        EDGE_T1_0(10.0.0.20): timestamp=06-12 14:19:26, src=172.31.0.65, latency=60.249, jitter=0.031, pktloss=0.000%, mos=4.373, SLA id=1(remote), rmt_ver=2, rmt_sla=out, rmt_prio=20
      EDGE_T1_1(172.31.0.66): timestamp=06-12 14:19:26, src=172.31.0.66, latency=80.222, jitter=0.021, pktloss=0.000%, mos=4.361, SLA id=1(remote), rmt_ver=2, rmt_sla=out, rmt_prio=40
      
      Remote Health Check: Remote_HC(2)
        Passive remote statistics of EDGE_T2(47):
        EDGE_T2_0(10.0.0.15): timestamp=06-12 14:19:26, src=172.31.0.66, latency=0.205, jitter=0.011, pktloss=0.000%, mos=4.404, SLA id=1(pass), rmt_ver=2, rmt_sla=in, rmt_prio=35
        EDGE_T2_1(172.31.0.65): timestamp=06-12 14:19:26, src=172.31.0.65, latency=0.215, jitter=0.009, pktloss=0.000%, mos=4.404, SLA id=1(pass), rmt_ver=2, rmt_sla=in, rmt_prio=15

   2. View the static routing table:

      For EDGE_T1, the priority changed from 10 to 20 and from 30 to 40 because it is out of SLA.

      DC1_A_FGT (root) (Interim)# get router info routing-table static
      Routing table for VRF=0
      S       172.31.0.65/32 [15/0] via EDGE_T2 tunnel 172.31.0.65, [15/0]
                                        [15/0] via EDGE_T1 tunnel 10.0.0.20, [20/0]
      S       172.31.0.66/32 [15/0] via EDGE_T2 tunnel 10.0.0.15, [35/0]
                                        [15/0] via EDGE_T1 tunnel 172.31.0.66, [40/0]

   3. View the BGP routing table:

      • For 10.0.3.0/24, EDGE_T2 is preferred. Priority for EDGE_T1 changed from 10 to 20.

      • For 10.0.4.0/24, EDGE_T2 is preferred. Priority for EDGE_T1 changed from 30 to 40.

      DC1_A_FGT (root) (Interim)# get router info routing-table  bgp
      Routing table for VRF=0
      B       10.0.3.0/24 [200/0] via 172.31.0.65 (recursive via EDGE_T2 tunnel 172.31.0.65 [15]), 00:07:28 
                                                  (recursive via EDGE_T1 tunnel 10.0.0.20 [20]), 00:07:28, [1/0]      
      B       10.0.4.0/24 [200/0] via 172.31.0.66 (recursive via EDGE_T2 tunnel 10.0.0.15 [35]), 00:07:28                                                          (recursive via EDGE_T1 tunnel 172.31.0.66 [40]), 00:07:28, [1/0]   

Speed test rerouting based on embedded SLA priorities

When spoke-initiated speed tests are enabled for this configuration, the out-of SLA priority is used by the hub to choose other routes during the speed test.

To configure speed tests:

1. On the hub, enable speed tests and allow them on the underlays and overlays.

   1. Enable speed tests:

      config system global
          set speedtest-server enable
          set speedtestd-ctrl-port 6000
          set speedtestd-server-port 7000
      end

   2. Allow speed tests on the underlay:

      config system interface
          edit "port1"
             ...
              set allowaccess ping speed-test
            ...
          next
      end

   3. Allow speed tests on the underlay:

      config system interface
          edit "port2"
              ...
              set allowaccess ping speed-test
             ...
          next
      end

   4. Allow speed tests on the overlay, and specify a shaping profile:

      config system interface
          edit "EDGE_T1"
              ...
              set allowaccess ping speed-test 
              set type tunnel
              set egress-shaping-profile "profile_1"
              ...
              set interface "port1"
          next
      end

   5. Allow speed tests on the overlay, and specify a shaping profile:

      config system interface
          edit "EDGE_T2"
              ...
              set allowaccess ping speed-test
              set type tunnel
              set egress-shaping-profile "profile_1"
              ...
              set interface "port2"
          next
      end

   6. View the shaping profile:

      config firewall shaping-profile
          edit "profile_1"                                                   
              set default-class-id 2
              config shaping-entries
                  edit 1
                      set class-id 2
                      set priority low
                      set guaranteed-bandwidth-percentage 10
                      set maximum-bandwidth-percentage 10
                  next
                  edit 2
                      set class-id 3
                      set priority medium
                      set guaranteed-bandwidth-percentage 30
                      set maximum-bandwidth-percentage 40
                  next
                  edit 3
                      set class-id 4
                      set guaranteed-bandwidth-percentage 20
                      set maximum-bandwidth-percentage 50
                  next
              end
          next
      end

2. On Spoke-1, schedule speed tests:

   config system speed-test-schedule
       edit "H1_T11"                                              
           set mode TCP
           set schedules "speed-test"
           set dynamic-server enable
           set ctrl-port 6000
           set server-port 7000
           set update-shaper remote
       next
       edit "H1_T22"  
           set mode UDP
           set schedules "speed-test"
           set dynamic-server enable
           set ctrl-port 6000
           set server-port 7000
           set update-shaper remote
       next
   end

Before starting the speed test on Spoke-1, route priorities are based on the received overlay priorities on both H1_T11 and H1_T22

DC1_A_FGT (root) (Interim)# get router info routing-table bgp
Routing table for VRF=0 
B       10.0.3.0/24 [200/0] via 172.31.0.65 (recursive via EDGE_T1 tunnel 10.0.0.20 [10]), 00:11:56 
                                            (recursive via EDGE_T2 tunnel 172.31.0.65 [15]), 00:11:56, [1/0]

While the speed test is running on H1_T11 of Spoke-1, Spoke-1 will constantly embed out-of SLA overlay priority into probes on H1_T11 and send them to the hub. Then the Hub updates route priorities accordingly and detours hub-to-spoke traffic to H1_T22 to avoid the impact on speed test of H1_T11. EDGE_T2 is preferred, and the EDGE_T1 priority changed from 10 to 20.

DC1_A_FGT (root) (Interim)# get router info routing-table  bgp
Routing table for VRF=0Routing table for VRF=0
B       10.0.3.0/24 [200/0] via 172.31.0.65 (recursive via EDGE_T2 tunnel 172.31.0.65 [15]), 00:03:49            
                                            (recursive via EDGE_T1 tunnel 10.0.0.20 [20]), 00:03:49, [1/0]

During the speed test on H1_T22 of Spoke-1, Spoke-1 constantly embeds out-of SLA overlay priority into probes on H1_T22 and sends them to the hub. Then the Hub updates route priorities accordingly and detours hub-to-spoke traffic to H1_T11 to avoid the impact on speed test of H1_T22. EDGE_T1 is preferred, and the EDGE_T2 priority changed from 15 to 25.

DC1_A_FGT (root) (Interim)# get router info routing-table  bgp
Routing table for VRF=0
B       10.0.3.0/24 [200/0] via 172.31.0.65 (recursive via EDGE_T1 tunnel 10.0.0.20 [10]), 00:04:06
                                                                  (recursive via EDGE_T2 tunnel 172.31.0.65 [25]), 00:04:06, [1/0]

Once speed test completes, the results are applied on child tunnels as egress-shaping-profile on the hub.

DC1_A_FGT (root) (Interim)# diagnose vpn tunnel  list
list all ipsec tunnel in vd 0
------------------------------------------------------
name=EDGE_T1_0 ver=2 serial=22 172.31.1.1:0->172.31.3.1:0 nexthop=172.31.1.2 tun_id=10.0.0.20 tun_id6=::10.0.0.31 status=up dst_mtu=1500 weight=1
.....
  dec: spi=9932cb6a esp=aes-gcm key=36 0819373cc74d0eb2dae8ac559519b756010fb42d2453b1c046429f8aad4d7dcbb507b990
       ah=null key=0
  enc: spi=dea67361 esp=aes-gcm key=36 c2ac45b9c1144d6728b5a7a542d1a35c86d018af35f6f661888dc39f771dc4820e2a40ff
       ah=null key=0
  dec:pkts/bytes=0/0, enc:pkts/bytes=3789/517724
  npu_flag=03 npu_rgwy=172.31.3.1 npu_lgwy=172.31.1.1 npu_selid=26 dec_npuid=1 enc_npuid=1 npu_isaidx=719 npu_osaidx=39
egress traffic control:
        bandwidth=711495(kbps) lock_hit=0 default_class=2 n_active_class=3
        class-id=2      allocated-bandwidth=71149(kbps)         guaranteed-bandwidth=71149(kbps)
                        max-bandwidth=71149(kbps)       current-bandwidth=1(kbps)
                        priority=low    forwarded_bytes=82K
                        dropped_packets=0       dropped_bytes=0
        class-id=3      allocated-bandwidth=284597(kbps)        guaranteed-bandwidth=213448(kbps)
                        max-bandwidth=284597(kbps)      current-bandwidth=0(kbps)
                        priority=medium         forwarded_bytes=0
                        dropped_packets=0       dropped_bytes=0
        class-id=4      allocated-bandwidth=355747(kbps)        guaranteed-bandwidth=142298(kbps)
                        max-bandwidth=355747(kbps)      current-bandwidth=0(kbps)
                        priority=high   forwarded_bytes=0
                        dropped_packets=0       dropped_bytes=0
------------------------------------------------------
name=EDGE_T2_1 ver=2 serial=1c 172.31.1.5:0->172.31.3.5:0 nexthop=172.31.1.6 tun_id=172.31.0.65 tun_id6=::10.0.0.25 status=up dst_mtu=1500 weight=1
......
  dec: spi=9932cb69 esp=aes-gcm key=36 01842dbbe1fe98fc6503b491c0768a844d815074950589b48db8a9a00c7505e73a96dc84
       ah=null key=0
  enc: spi=dea67360 esp=aes-gcm key=36 c2e4cb1065325de7970d6f8edb3cac8829a33420ff79acedd6dcf4012e4614dd92d7b16b
       ah=null key=0
  dec:pkts/bytes=0/0, enc:pkts/bytes=3518/499252
  npu_flag=03 npu_rgwy=172.31.3.5 npu_lgwy=172.31.1.5 npu_selid=27 dec_npuid=1 enc_npuid=1 npu_isaidx=718 npu_osaidx=40
egress traffic control:
        bandwidth=374876(kbps) lock_hit=0 default_class=2 n_active_class=3
        class-id=2      allocated-bandwidth=37487(kbps)         guaranteed-bandwidth=37487(kbps)
                        max-bandwidth=37487(kbps)       current-bandwidth=2(kbps)
                        priority=low    forwarded_bytes=78K
                        dropped_packets=0       dropped_bytes=0
        class-id=3      allocated-bandwidth=149950(kbps)        guaranteed-bandwidth=112462(kbps)
                        max-bandwidth=149950(kbps)      current-bandwidth=0(kbps)
                        priority=medium         forwarded_bytes=0
                        dropped_packets=0       dropped_bytes=0
        class-id=4      allocated-bandwidth=187438(kbps)        guaranteed-bandwidth=74975(kbps)
                        max-bandwidth=187438(kbps)      current-bandwidth=0(kbps)
                        priority=high   forwarded_bytes=0
                        dropped_packets=0       dropped_bytes=0