Exchange underlay link cost property with remote peer in IPsec VPN phase 1 negotiation 7.2.1

This information is also available in the FortiOS 7.2 Administration Guide: SD-WAN in large scale deployments

The underlay link cost property has been added to the IPsec VPN tunnel phase 1 configuration and enhances the IPsec VPN to exchange the link cost with a remote peer as a private notify payload in IKEv1 and IKEv2 phase 1 negotiations. This avoids possible health check daemon process load issues in the previous implementation of the link cost exchange feature, and it improves network scalability in a large-scale SD-WAN network with ADVPN.

config vpn ipsec phase1-interface
    edit <name>
        set link-cost <integer>
    next
end

link-cost <integer>

Set the VPN underlay link cost (0 - 255, default = 0).

If multiple shortcuts originate from the same SD-WAN member to different members on the same remote spoke, learned remote IPsec link costs on shortcuts will be used as a tie-breaker to decide which shortcut is preferred.

In this example, SD-WAN is configured on an ADVPN network with a BGP neighbor per overlay.

Instead of reflecting BGP routes with the route-reflector on the hub, when the shortcuts are triggered, IKE routes on the shortcuts are directly injected based on the configured phase 2 selectors to allow routes to be exchanged between spokes.

Routes between the hub and the spokes are exchanged by BGP, and the spokes use the default route to send spoke-to-spoke traffic to the hub and trigger the shortcuts.

To configure Spoke 1:

1. Configure the VPN remote gateway:

   config vpn ipsec phase1-interface
       edit "spoke11-p1"
           ...
           set mode-cfg-allow-client-selector enable 
           set link-cost 11
       next
       edit "spoke12-p1"
           ...
           set mode-cfg-allow-client-selector enable 
           set link-cost 21
       next
   end

2. Configure the SD-WAN settings:

   config system sdwan
       set status enable
       config zone
           edit "virtual-wan-link"
           next
       end
       config members
           edit 1
               set interface "spoke11-p1"
               set cost 10
           next
           edit 2
               set interface "spoke12-p1"
               set cost 20
           next
       end
       config health-check
           edit "1"
               set server "9.0.0.1"
               set members 0
               config sla
                   edit 1
                   next
               end
           next
       end
       config service
           edit 1
               set name "1"
               set mode sla
               set dst "all"
               set src "10.1.100.0"
               config sla
                   edit "1"
                       set id 1
                   next
               end
               set priority-members 1 2
           next
       end
   end

To configure Spoke 2:

1. Configure the VPN remote gateway:

   config vpn ipsec phase1-interface
       edit "spoke21-p1"
           ...
           set link-cost 101
       next
       edit "spoke22-p1"
           ...
           set link-cost 201
       next
   end

2. Configure the SD-WAN settings:

   config system sdwan
       set status enable
       config zone
           edit "virtual-wan-link"
           next
       end
       config members
           edit 1
               set interface "spoke21-p1"
               set cost 10
           next
           edit 2
               set interface "spoke22-p1"
               set cost 20
           next
       end
       config health-check
           edit "1"
               set server "9.0.0.1"
               set members 0
               config sla
                   edit 1
                   next
               end
           next
       end
       config service
           edit 1
               set name "1"
               set mode sla
               set dst "all"
               set src "192.168.5.0"
               config sla
                   edit "1"
                       set id 1
                   next
               end
               set priority-members 1 2
           next
       end
   end

To test the configuration:

1. Verify the service diagnostics on Spoke 1:

   # diagnose sys sdwan service
   
   Service(1): Address Mode(IPV4) flags=0x200 use-shortcut-sla
    Tie break: cfg
     Gen(4), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(sla), sla-compare-order
     Members(2):
       1: Seq_num(1 spoke11-p1), alive, sla(0x1), gid(0), cfg_order(0), local cost(10), selected
       2: Seq_num(2 spoke12-p1), alive, sla(0x1), gid(0), cfg_order(1), local cost(20), selected
     Src address(1):
           10.1.100.0-10.1.100.255
   
     Dst address(1):
           0.0.0.0-255.255.255.255

2. Verify the service diagnostics on Spoke 2:

   # diagnose sys sdwan service
   
   Service(1): Address Mode(IPV4) flags=0x200 use-shortcut-sla
    Tie break: cfg
     Gen(2), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(sla), sla-compare-order
     Members(2):
       1: Seq_num(1 spoke21-p1), alive, sla(0x1), gid(0), cfg_order(0), local cost(10), selected
       2: Seq_num(2 spoke22-p1), alive, sla(0x1), gid(0), cfg_order(1), local cost(20), selected
     Src address(1):
           192.168.5.0-192.168.5.255
   
     Dst address(1):
           0.0.0.0-255.255.255.255

3. Trigger shortcuts between Spoke 1 and Spoke 2:

   • Shortcuts spoke11-p1_1 and spoke11-p1_0 originate from spoke11-p1.

   • spoke11-p1_1 corresponds to spoke21-p1_0 on Spoke 2.

   • spoke11-p1_0 corresponds to spoke22-p1_0 on Spoke 2.

   Spoke 1:

   # diagnose sys sdwan service
   
   Service(1): Address Mode(IPV4) flags=0x200 use-shortcut-sla
    Tie break: cfg
     Gen(11), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(sla), sla-compare-order
     Member sub interface(4):
       3: seq_num(1), interface(spoke11-p1):
          1: spoke11-p1_0(80)
          2: spoke11-p1_1(81)
     Members(4):
       1: Seq_num(1 spoke11-p1_1), alive, sla(0x1), gid(0), remote cost(101), cfg_order(0), local cost(10), selected
       2: Seq_num(1 spoke11-p1_0), alive, sla(0x1), gid(0), remote cost(201), cfg_order(0), local cost(10), selected
       3: Seq_num(1 spoke11-p1), alive, sla(0x1), gid(0), cfg_order(0), local cost(10), selected
       4: Seq_num(2 spoke12-p1), alive, sla(0x1), gid(0), cfg_order(1), local cost(20), selected
     Src address(1):
           10.1.100.0-10.1.100.255
   
     Dst address(1):
           0.0.0.0-255.255.255.255

   Spoke 2:

   # diagnose sys sdwan service
   
   Service(1): Address Mode(IPV4) flags=0x200 use-shortcut-sla
    Tie break: cfg
     Gen(15), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(sla), sla-compare-order
     Member sub interface(4):
       2: seq_num(1), interface(spoke21-p1):
          1: spoke21-p1_0(75)
       4: seq_num(2), interface(spoke22-p1):
          1: spoke22-p1_0(74)
     Members(4):
       1: Seq_num(1 spoke21-p1_0), alive, sla(0x1), gid(0), remote cost(11), cfg_order(0), local cost(10), selected
       2: Seq_num(1 spoke21-p1), alive, sla(0x1), gid(0), cfg_order(0), local cost(10), selected
       3: Seq_num(2 spoke22-p1_0), alive, sla(0x1), gid(0), remote cost(11), cfg_order(1), local cost(20), selected
       4: Seq_num(2 spoke22-p1), alive, sla(0x1), gid(0), cfg_order(1), local cost(20), selected
     Src address(1):
           192.168.5.0-192.168.5.255
   
     Dst address(1):
           0.0.0.0-255.255.255.255

   The spoke11-p1_1 shortcut on Spoke 1 is preferred over spoke11-p1_0 due to the lower remote link cost of 101 when they have the same local SD-WAN member cost of 10.

4. Verify the policy route list on Spoke 1:

   # diagnose firewall proute list
   list route policy info(vf=root):
   
   id=2131755009(0x7f100001) vwl_service=1(1) vwl_mbr_seq=1 1 1 2 dscp_tag=0xfc 0xfc flags=0x0 tos=0x00 tos_mask=0x00 protocol=0 sport=0-65535 iif=0(any) dport=1-65535 path(4) oif=81(spoke11-p1_1) oif=80(spoke11-p1_0) oif=54(spoke11-p1) oif=55(spoke12-p1)
   source(1): 10.1.100.0-10.1.100.255
   destination(1): 0.0.0.0-255.255.255.255
   hit_count=176 last_used=2022-07-12 11:56:08