Embedded SD-WAN SLA information in ICMP probes

In the hub and spoke SD-WAN design, in order for traffic to pass symmetrically from spoke to hub and hub to spoke, it is essential for the hub to know which IPsec overlay is in SLA and out of SLA. Prior to introducing embedded SLA information in ICMP probes, it is common practice for spokes to use the SD-WAN neighbor feature and route-map-out-preferable setting to signal the health of each overlay to the hub. However, this requires BGP to be configured per overlay, and to manipulate BGP routes using custom BGP communities.

With embedded SLA information in ICMP probes, spokes can communicate their SLA for each overlay directly through ICMP probes to the hub. The hub learns these SLAs and maps the status for each spoke and its corresponding overlays.

The hub uses the SLA status to apply priorities to the IKE routes, giving routes over IPsec overlays that are within SLAs a lower priority value and routes over overlays out of SLAs a higher priority value. If BGP is used, recursively resolved BGP routes can inherit the priority from its parent.

Embedded SLA information in ICMP probes allows hub and spoke SD-WAN to be designed with a BGP on loopback topology, or without BGP at all. The following topology outlines an example of the BGP on loopback design where each spoke is peered with the hub and route reflector on the loopback interface.

In this topology, each FortiGate’s BGP router ID is based on its Loopback0 interface. Each spoke has SLA health checks defined to send ICMP probes to the server’s Lo_HC interface on 172.31.100.100. The ICMP probes include embedded SLA information for each SD-WAN overlay member.

Related SD-WAN settings:

config system sdwan
    config health-check
        edit <name>
            set detect-mode {active | passive | prefer-passive | remote}
            set embed-measured-health {enable | disable}
            config sla
                edit <id>
                    set priority-in-sla <integer>
                    set priority-out-sla <integer>
                next
            end
            set sla-id-redistribute <id>
        next
    end
end

detect-mode {active | passive | prefer-passive | remote}	Set the mode that determines how to detect the server: active: the probes are sent actively (default). passive: the traffic measures health without probes. prefer-passive: the probes are sent in case of no new traffic. remote: the link health is obtained from remote peers.
embed-measured-health {enable | disable}	Enable/disable embedding SLA information in ICMP probes (default = disable).
set priority-in-sla <integer>	Set the priority that will be set to the IKE route when the corresponding overlay is in SLA (0 - 65535).
set priority-out-sla <integer>	Set the priority that will be set to the IKE route when the corresponding overlay is out of SLA (0 - 65535).
sla-id-redistribute <id>	Set the SLA entry (ID) that will be applied to the IKE routes (0 - 32, default = 0).

Related BGP setting:

config router bgp
    set recursive-inherit-priority {enable | disable}
end

recursive-inherit-priority {enable | disable}

Enable/disable allowing recursive resolved BGP routes to inherit priority from its parent (default = disable).

Example with BGP on loopback SD-WAN

This example demonstrates the configurations needed to configure the SD-WAN and BGP settings for the preceding topology. It is assumed that IPsec VPN overlays are already configured per the topology, and that loopback interfaces are already configured on each FortiGate.

Configuring the Spoke_1 FortiGate

In the SD-WAN settings, note the following requirements:

1. Configure the SD-WAN zones and members. For each SD-WAN member, define the source of its probes to be the Loopback0 interface IP.
2. Configure the SLA health check to point to the Hub’s Lo_HC interface and IP. Enable embed-measured-health.
3. Configure an SD-WAN service rule to route traffic based on the maximize bandwidth (SLA) algorithm to prefer member H1_T11 over H1_T22.
4. Configure set exchange-interface-ip enable and set exchange-ip-addr4 to the Loopback0 interface IP. The exchange-interface-ip option is automatically turned on when ADVPN has already been configured. If ADVPN has not been configured, then set exchange-interface-ip enable must be configured before set exchange-ip-addr4 can be configured.

To configure the SD-WAN settings:

config system sdwan
    set status enable
    config zone
        edit "virtual-wan-link"
        next
        edit "overlay"
        next
    end
    config members
        edit 1
            set interface "H1_T11"
            set zone "overlay"
            set source 172.31.0.65
        next
        edit 4
            set interface "H1_T22"
            set zone "overlay"
            set source 172.31.0.65
        next
    end
    config health-check
        edit "HUB"
            set server "172.31.100.100"
            set embed-measured-health enable                   
            set members 0
            config sla
                edit 1
                    set link-cost-factor latency
                    set latency-threshold 100
                next
            end
        next
    end
    config service
        edit 1
            set mode sla
            set dst "CORP_LAN"
            set src "CORP_LAN"
            config sla
                edit "HUB"
                    set id 1
                next
            end
            set priority-members 1 4
        next
    end
end

To configure the BGP settings:

config router bgp
    set as 65001
    set router-id 172.31.0.65
    config neighbor
        edit "172.31.0.1"
            set remote-as 65001
            set update-source "Loopback0"
        next
    end
    config network
        edit 1
            set prefix 10.0.3.0 255.255.255.0
        next
    end
end

To add the loopback IP to the IPsec interface settings:

config vpn ipsec phase1-interface
    edit "H1_T11"
        set exchange-interface-ip enable
        set exchange-ip-addr4 172.31.0.65
    next
    edit "H1_T22"
        set exchange-interface-ip enable
        set exchange-ip-addr4 172.31.0.65
    next
end

Configuring the hub FortiGate

In the SD-WAN settings, note the following requirements:

1. Configure the SD-WAN zone and members.
2. Configure the SLA health checks to detect SLAs based on the remote site (spoke). This must be defined for each SD-WAN member:
   1. For the SLA, specify the same link cost factor and metric as the spoke (100).
   2. Define the IKE route priority for in and out of SLA. Lower priority values have higher priority than higher priority values.
3. Define the SLA entry ID that will be applied to the IKE routes.
4. Configure set exchange-interface-ip enable and set exchange-ip-addr4 to the Loopback0 interface IP. The exchange-interface-ip option is automatically turned on when ADVPN has already been configured. If ADVPN has not been configured, then set exchange-interface-ip enable must be configured before set exchange-ip-addr4 can be configured.

To configure the SD-WAN settings:

config system sdwan
    set status enable
    config zone
        edit "virtual-wan-link"
        next
    end
    config members
        edit 1
            set interface "EDGE_T1"
        next
        edit 2
            set interface "EDGE_T2"
        next
    end
    config health-check
        edit "1"
            set detect-mode remote
            set sla-id-redistribute 1
            set members 1
            config sla
                edit 1
                    set link-cost-factor latency
                    set latency-threshold 100
                    set priority-in-sla 10
                    set priority-out-sla 20
                next
            end
        next
        edit "2"
            set detect-mode remote
            set sla-id-redistribute 1
            set members 2
            config sla
                edit 1
                    set link-cost-factor latency
                    set latency-threshold 100
                    set priority-in-sla 15
                    set priority-out-sla 25
                next
            end
        next
    end
end

In the BGP settings, note the following requirements:

1. Enable recursive-inherit-priority to inherit the route priority from its parent, which is the priority defined in the health check SLA settings.
2. Configure the other BGP settings similar to a regular BGP hub.

To configure the BGP settings:

config router bgp
    set as 65001
    set router-id 172.31.0.1
    set recursive-inherit-priority enable
    config neighbor-group
        edit "EDGE"
            set remote-as 65001
            set update-source "Loopback0"
            set route-reflector-client enable
        next
    end
    config neighbor-range
        edit 1
            set prefix 172.31.0.64 255.255.255.192
            set neighbor-group "EDGE"
        next
    end
end

To add the loopback IP to the IPsec interface settings:

config vpn ipsec phase1-interface
    edit "EDGE_T1"
        set exchange-interface-ip enable
        set exchange-ip-addr4 172.31.0.1
    next
    edit "EDGE_T2"
        set exchange-interface-ip enable
        set exchange-ip-addr4 172.31.0.1
    next
end

Testing and verification

Once the hub and spokes are configured, verify that SLA statuses are passed from the spoke to the hub.

To verify that the SLA statuses are passed from the spoke to the hub:

1. On Spoke_1, display the status of the health-checks for H1_T11 and H1_T22:

   # diagnose sys sdwan  health-check
   Health Check(HUB):
   Seq(1 H1_T11): state(alive), packet-loss(0.000%) latency(0.228), jitter(0.018), mos(4.404), bandwidth-up(999999), bandwidth-dw(1000000), bandwidth-bi(1999999) sla_map=0x1
   Seq(4 H1_T22): state(alive), packet-loss(0.000%) latency(0.205), jitter(0.007), mos(4.404), bandwidth-up(999998), bandwidth-dw(1000000), bandwidth-bi(1999998) sla_map=0x1

2. On Spoke_1, display the status and order of the overlays in the SD-WAN service rule:

   # diagnose sys sdwan service
   Service(1): Address Mode(IPV4) flags=0x200 use-shortcut-sla
    Tie break: cfg
     Gen(1), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(sla), sla-compare-order
     Members(2):
       1: Seq_num(1 H1_T11), alive, sla(0x1), gid(0), cfg_order(0), local cost(0), selected                 
       2: Seq_num(4 H1_T22), alive, sla(0x1), gid(0), cfg_order(3), local cost(0), selected                 
     Src address(1):
           10.0.0.0-10.255.255.255
     Dst address(1):
           10.0.0.0-10.255.255.255

   Both overlays are within SLA, so H1_T11 is preferred due to its cfg-order.

   Spoke_1’s SLA information for H1_T11 and H1_T22 is embedded into the ICMP probes destined for the hub’s Lo_HC interface. The hub receives this information and maps the SLAs correspondingly per spoke and overlay based on the same SLA targets.

   As a result, since all SLAs are within target, the hub sets the routes over each overlay as follows:

   Hub SD-WAN member	Overlay	SLA status	Priority for IKE routes
   1	EDGE_T1	0x1 – within SLA	10
   2	EDGE_T2	0x1 – within SLA	15

3. Verify that the spoke has sent its health check result to hub.

   1. On the hub, display the status of the health checks for EDGE_T1 and EDGE_T2:

      # diagnose sys sdwan health-check remote 
      Remote Health Check: 2(2)
        Passive remote statistics of EDGE_T2(22):
      EDGE_T2_0(172.31.3.5): timestamp=02-09 16:19:11, latency=1.056, jitter=0.582, pktloss=0.000%
      Remote Health Check: 1(1)
        Passive remote statistics of EDGE_T1(21):
      EDGE_T1_0(172.31.3.1): timestamp=02-09 16:19:11, latency=1.269, jitter=0.675, pktloss=0.000%

4. When there are multiple spokes, additional options can be used to filter a spoke by health check name, or health check name and the member's sequence number (diagnose system sdwan health-check remote <hc_name> <seq_num>).

   1. To filter the health check by health check name:

      # diagnose sys sdwan health-check remote 1
      Remote Health Check: 1(1)
        Passive remote statistics of EDGE_T1(21):
      EDGE_T1_0(172.31.3.1): timestamp=02-09 16:43:37, latency=1.114, jitter=0.473, pktloss=0.000%

      When this method is used, the output displays all the members of the specified health check name.

   2. To filter the health check by health check name and the member's sequence number:

      # diagnose sys sdwan health-check remote 1 1
      Remote Health Check: 1(1)
        Passive remote statistics of EDGE_T1(21):
      EDGE_T1_0(172.31.3.1): timestamp=02-09 16:43:41, latency=1.178, jitter=0.497, pktloss=0.000%

      When this method is used, the output displays the specified member of the specified health check name.

   If the detect-mode is set to remote, use diagnose sys sdwan health-check remote in lieu of diagnose sys sdwan health-check.

5. Simultaneously, BGP recursive routes inherit the priority based on the parent IKE routes. The recursively resolved BGP routes that pass through EDGE_T1 will have a priority of 10, and routes that pass through EDGE_T2 will have a priority of 15. Therefore, traffic from the hub to the spoke will be routed to EDGE_T1.

   Verify the routing tables.

   1. Static:

      # get router info routing-table static
      Routing table for VRF=0
      S       172.31.0.65/32 [15/0] via EDGE_T1 tunnel 10.0.0.69 vrf 0, [10/0]
                                        [15/0] via EDGE_T2 tunnel 172.31.0.65 vrf 0, [15/0]

   2. BGP:

      # get router info routing-table bgp
      Routing table for VRF=0
      B       10.0.3.0/24 [200/0] via 172.31.0.65 (recursive via EDGE_T1 tunnel 10.0.0.69 vrf 0 [10]), 04:32:53
                                                  (recursive via EDGE_T2 tunnel 172.31.0.65 vrf 0 [15]), 04:32:53, [1/0]

Next, test by making the health checks over the spokes' H1_T11 tunnel out of SLA. This should trigger traffic to start flowing from the spokes' H1_T22 tunnel. Consequently, the SLA statuses are passed from the spoke to the hub, and the hub will start routing traffic to EDGE_T2.

To verify that the hub will start routing traffic to EDGE_T2 when the spoke H1_T11 tunnel is out of SLA:

1. On Spoke_1, display the status of the health checks for H1_T11 and H1_T22:

   # diagnose sys sdwan health-check
   Health Check(HUB):
   Seq(1 H1_T11): state(alive), packet-loss(0.000%) latency(120.228), jitter(0.013), mos(4.338), bandwidth-up(999999), bandwidth-dw(1000000), bandwidth-bi(1999999) sla_map=0x0
   Seq(4 H1_T22): state(alive), packet-loss(0.000%) latency(0.220), jitter(0.008), mos(4.404), bandwidth-up(999998), bandwidth-dw(1000000), bandwidth-bi(1999998) sla_map=0x1

2. Verify the routing tables.

   1. Static:

      # get router info routing-table static
      Routing table for VRF=0
      S       172.31.0.65/32 [15/0] via EDGE_T2 tunnel 172.31.0.65 vrf 0, [15/0]
                             [15/0] via EDGE_T1 tunnel 10.0.0.69 vrf 0, [20/0]

      The priority for EDGE_T1 has changed from 10 to 20.

   2. BGP:

      # get router info routing-table bgp
      Routing table for VRF=0
      B       10.0.3.0/24 [200/0] via 172.31.0.65 (recursive via EDGE_T2 tunnel 172.31.0.65 vrf 0 [15]), 00:01:19
                                                  (recursive via EDGE_T1 tunnel 10.0.0.69 vrf 0 [20]), 00:01:19, [1/0]

      EDGE_T2 is now preferred. The priority for EDGE_T1 has changed from 10 to 20.

Spoke_1’s SLA information for H1_T11 embedded into the ICMP probes has now changed.

As a result, the hub sets the routes over each overlay as follows:

Hub SD-WAN member	Overlay	SLA status	Priority for IKE routes
1	EDGE_T1	0x0 – out of SLA	20
2	EDGE_T2	0x1 – within SLA	15

The BGP recursive routes inherit the priority based on the parent IKE routes. Since priority for IKE routes on EDGE_T1 has changed to 20, recursively resolved BGP routes passing through EDGE_T1 has also dropped to 20. As a result, hub to spoke_1 traffic will go over EDGE_T2.