Example SD-WAN overlay placeholders using ADVPN 2.0

In a typical hub and spoke SD-WAN configuration, the hub uses the same underlay transports as the spokes. However, in unique circumstances, spokes may use different transports than the hub. For example, spokes may have direct MPLS connections that are not connected to the hub. Hubs are not necessarily connected to all the same underlay transports as spokes. In these instances, ADVPN 2.0 can use overlay placeholders to trigger shortcuts between spokes over transports to which hubs are not connected. Once the regular shortcut and placeholder shortcuts are established, SD-WAN will steer traffic based on the service rule, such as which link has the best quality.

In this SD-WAN example, the Hub, Spoke-1, and Spoke-2 all have broadband WAN connections. However, Spoke-1 and Spoke-2 have an additional MPLS connection between them. With ADVPN 2.0 enabled, Spoke-1 and Spoke-2 have regular parent tunnels (H1_T11) to the Hub. Spoke-1 and Spoke-2 also have placeholder parent tunnels configured (Placeholder_MPLS_1), where auto-discovery-dialup-placeholder is enabled and remote-gateway isn't statically specified. As a placeholder tunnel, no connection is established between the spoke and the hub. Instead, the tunnel interface is utilized in the SD-WAN configurations as if it were a regular overlay because shortcuts can be established between spokes in a placeholder overlay.

Traffic is sent from PC-1 to PC-2. This traffic matches an SD-WAN rule that include two interfaces: a VPN tunnel to the Hub, and a placeholder tunnel. As only the first tunnel is alive, the first bit of traffic goes through this tunnel to the Hub and triggers SHORTCUT_QUERY/SHORTCUT_REPLY exchange with the Hub. When Spoke-1 receives SHORTCUT_REPLY message, which includes WAN and SD-WAN information about Spoke-2's eligible shortcut interfaces, Spoke-1 calculates and creates the shortcut between regular parent tunnels and triggers a shortcut between the placeholder parent tunnels too.

As the SHORTCUT_REPLY from the Hub contains additional information for a second WAN interface on Spoke-2, a SHORTCUT_QUERY is triggered on Spoke-1 to see if another shortcut can be established. This SHORTCUT_QUERY utilizes the H1_T11 tunnel to facilitate the SHORTCUT_QUERY/SHORTCUT_REPLY exchange with the Hub.

Health-checks, which are automatically running on the regular shortcut and the placeholder shortcut, help decide which interface to use for forwarding the remaining traffic.

Settings on the spoke relevant to the example:

config vpn ipsec phase1-interface
    edit <placeholder_phase1_interface_name>
        set type dynamic
        ...
        set net-device enable
        ...
        set auto-discovery-dialup-placeholder enable
    next
end

To enable overlay placeholder tunnels on Spoke-1 and Spoke-2:

1. Configure key components on Spoke-1:

   1. Configure a parent tunnel and a placeholder tunnel:

      Tunnel H1_T11 is the parent tunnel, and tunnel Placeholder_MPLS_1 is the placeholder tunnel:

      config vpn ipsec phase1-interface
          edit "H1_T11"                                                       
              set interface "port1"
              set ike-version 2
              set keylife 28800
              set peertype any
              set net-device enable
              set exchange-ip-addr4 172.31.0.65
              set proposal aes256gcm-prfsha384
              set add-route disable
              set dpd on-idle
              set idle-timeout enable
              set idle-timeoutinterval 5
              set auto-discovery-receiver enable
              set encapsulation vpn-id-ipip
              set network-overlay enable
              set network-id 11
              set transport udp
              set remote-gw 172.31.1.1
              set psksecret ENC fRkYaHGx3MoAASxMsHoJSObbOfphOmBntEpKhZAY8/9OjjqJxbmTehe4Z7LOGTvjFRXTULiPBkfPupnr0JtBo0KJb0IoUfJ8zWU3mJttCyfHNaDBXG/t5d0D93iu3ZlRoSs1EBRD7KX5fO1QnSHm0maXTJZzrKLWVSaRztdOF246z/RbEXFzlm4+64ccLHG/3X/Ha1lmMjY3dkVA
              set dpd-retryinterval 5
          next
          edit "Placeholder_MPLS_1"                                              
              set type dynamic
              set interface "port3"
              set ike-version 2
              set keylife 28800
              set peertype any
              set net-device enable                                                 
              set exchange-ip-addr4 172.31.0.65
              set proposal aes128-sha256 aes256-sha256 aes128gcm-prfsha256 aes256gcm-prfsha384 chacha20poly1305-prfsha256
              set add-route disable
              set dpd on-idle
              set idle-timeout enable
              set idle-timeoutinterval 5
              set auto-discovery-receiver enable
              set auto-discovery-dialup-placeholder enable 
              set encapsulation vpn-id-ipip
              set network-overlay enable
              set network-id 250
              set transport udp
              set psksecret ENC e3XlBFTNA+fk4Kn7NMLfzJLgU0Pk6ssX9oJ7VSA2Q371x+oqIRSomFhRyZiErb1j07FmGT/lPUFHAB2BJG7vhNIh0SSjEqThai8rIKWqHsIpjHwUAM0/maBzH1iTXZAyjEX0vFiu65QSLAoQzfgXkfX16P82Q1gjIim/GX4phzA+cIYJnfegbqXylFg4fxkqCrYFGFlmMjY3dkVA
              set dpd-retryinterval 60
          next
      end

   2. Configure SD-WAN:

      Enable ADVPN 2.0 and health-checks for the overlay. Configure the SD-WAN members and their transport groups.

      config system sdwan
          set status enable
          config zone
              edit "overlay"
                  set advpn-select enable                                    
                  set advpn-health-check "HUB"                                
              next
          end
          config members
              edit 4
                  set interface "H1_T11"                                      
                  set zone "overlay"
                  set source 172.31.0.65
                  set priority 10
                  set transport-group 1                                      
              next
              edit 13
                  set interface "Placeholder_MPLS_1"                            
                  set zone "overlay"
                  set source 172.31.0.65
                  set priority 10
                  set transport-group 2                                    
              next
          end
          config health-check
              edit "HUB"
                  set server "172.31.100.100"
                  set members 4 13
                  config sla
                      edit 1
                          set link-cost-factor latency
                          set latency-threshold 100
                      next
                  end
              next
          end
          config service
              edit 1
                  set name "1"
                  set mode sla
                  set dst "spoke-2_LAN-1"
                  set src "spoke-1_LAN-1"
                  config sla
                      edit "HUB"
                          set id 1
                      next
                  end
                  set priority-members 4 13
              next
          end
      end

2. Configure key components on Spoke-2:

   1. Configure a parent tunnel and a placeholder tunnel:

      Tunnel H1_T11 is the parent tunnel, and tunnel Placeholder_MPLS_1 is the placeholder tunnel:

         config vpn ipsec phase1-interface
         edit "H1_T11"
              set interface "port1"
              set ike-version 2
              set keylife 28800
              set peertype any
              set net-device enable
              set exchange-ip-addr4 172.31.0.66
              set proposal aes256gcm-prfsha384
              set add-route disable
              set dpd on-idle
              set idle-timeout enable
              set idle-timeoutinterval 5
              set auto-discovery-receiver enable
              set encapsulation vpn-id-ipip
              set network-overlay enable
              set network-id 11
              set transport udp
              set remote-gw 172.31.1.1
              set psksecret ENC 7tP6FktH3z3nIpiGoANS/PbxfMOeLf0KCp65MbG2/yBmeAZz0XXcF9XMbnfBtaWhZhoTnGx086ST1aRmLDxCfVu/BM6S+j1XOUSxSZb6xgwRoaQsFTFaXq/8PYjUzJK7SdPZiJDsCrGaslMR/mTLzFTasA4y9YGteJM+tELW5K0C6Ntrwlq8UnAtrdFMGN/3BrLNyllmMjY3dkVA
              set dpd-retryinterval 5
          next
          edit "Placeholder_MPLS_1"                                              
              set type dynamic
              set interface "port3"
              set ike-version 2
              set keylife 28800
              set peertype any
              set net-device enable
              set exchange-ip-addr4 172.31.0.66
              set proposal aes128-sha256 aes256-sha256 aes128gcm-prfsha256 aes256gcm-prfsha384 chacha20poly1305-prfsha256
              set add-route disable
              set dpd on-idle
              set idle-timeout enable
              set idle-timeoutinterval 5
              set auto-discovery-receiver enable
              set auto-discovery-dialup-placeholder enable
              set encapsulation vpn-id-ipip
              set network-overlay enable
              set network-id 250
              set transport udp
              set psksecret ENC Un++iggqXo1wU8fAArjS9JY5rsAcd8+Xw3WR7MtRVLU0BpXT56bAwrP6wQ4gpMt32ABTQ/mAddlq1Mq5LxCV0X+lYaxj2vVIulTb1n71Wn1WeTbDXjiTzDbrMnp4fM9+1uBKwy+RJ+u782IpI9WRhCWE/H4LNgZhYzUmyluTCouNoBpJyzdJrekAXOtSvxLJfju/7FlmMjY3dkVA
              set dpd-retryinterval 60
          next
      end

   2. Configure SD-WAN:

      Enable ADVPN 2.0 and health-checks for the overlay. Configure the SD-WAN members and their transport groups. The internet overlay (H1_T11) is added to transport group 1, and the MPLS overlay (Placeholder_MPLS_1) is added to transport group 2.

      config system sdwan
          set status enable
          config zone
              edit "overlay"
                  set advpn-select enable                                  
                  set advpn-health-check "HUB"                            
              next
          end
          config members
              edit 4
                  set interface "H1_T11"                                  
                  set zone "overlay"
                  set source 172.31.0.66
                  set priority 10
                  set transport-group 1                                                    
              next
              edit 13
                  set interface "Placeholder_MPLS_1"                            
                  set zone "overlay"
                  set source 172.31.0.66
                  set priority 10
                  set transport-group 2
              next
          end
          config health-check
              edit "HUB"
                  set server "172.31.100.100"
                  set members 4 13
                  config sla
                      edit 1
                          set link-cost-factor latency
                          set latency-threshold 100
                      next
                  end
              next
          end
      end

To check the health status and traffic:

1. Check the health status on Spoke-1 and Spoke-2, and check the SD-WAN status of Spoke-1:

   1. Check the health of Spoke-1:

      The placeholder tunnel (Placeholder_MPLS_1) is dead.

      # diagnose sys sdwan  health-check 
      Health Check(HUB):
      Seq(4 H1_T11): state(alive), packet-loss(0.000%), latency(0.235), jitter(0.011), mos(4.404), bandwidth-up(999998), bandwidth-dw(999998), bandwidth-bi(1999996), sla_map=0x1
      Seq(13 Placeholder_MPLS_1): state(dead), packet-loss(100.000%), sla_map=0x0

   2. Check the health of Spoke-2:

      The placeholder tunnel (Placeholder_MPLS_1) is dead.

      # diagnose sys sdwan  health-check
      Health Check(HUB): 
      Seq(4 H1_T11): state(alive), packet-loss(0.000%), latency(0.250), jitter(0.041), mos(4.404), bandwidth-up(999998), bandwidth-dw(999998), bandwidth-bi(1999996), sla_map=0x1
      Seq(13 Placeholder_MPLS_1): state(dead), packet-loss(100.000%), sla_map=0x0 

   3. Check the SD-WAN status of Spoke-1:

      # diagnose sys sdwan service4
      Service(1): Address Mode(IPV4) flags=0x4200 use-shortcut-sla use-shortcut
       Tie break: cfg
       Shortcut priority: 3
        Gen(21), TOS(0x0/0x0), Protocol(0): src(1->65535):dst(1->65535), Mode(sla), sla-compare-order
        Members(2):
          1: Seq_num(4 H1_T11 overlay), alive, sla(0x1), gid(0), cfg_order(0), local cost(0), selected
          2: Seq_num(13 Placeholder_MPLS_1 overlay), dead, sla(0x0), gid(0), cfg_order(1), local cost(0)  
        Src address(1):
              10.0.3.0-10.0.3.255
        Dst address(1):
              10.0.4.0-10.0.4.255

2. Send traffic from PC-1 to PC-2:

   When Spoke-1 receives SHORTCUT_REPLY message, which includes SD-WAN information about Spoke-2, Spoke-1 calculates and creates the shortcut between regular parent tunnels, and it triggers a shortcut between the placeholder tunnels too.

   1. Check the health of Spoke-1:

      The health-check is automatically running on the regular shortcut (H1_T11_0) and on the placeholder shortcut (Placeholder_MPLS_1_0).

      # diagnose sys sdwan  health-check 
      Health Check(HUB):
      Seq(4 H1_T11): state(alive), packet-loss(0.000%), latency(0.239), jitter(0.017), mos(4.404), bandwidth-up(999998), bandwidth-dw(999998), bandwidth-bi(1999996), sla_map=0x1
      Seq(4 H1_T11_0): state(alive), packet-loss(0.000%), latency(0.260), jitter(0.010), mos(4.404), bandwidth-up(999999), bandwidth-dw(999999), bandwidth-bi(1999998), sla_map=0x1                        
      Seq(13 Placeholder_MPLS_1): state(dead), packet-loss(100.000%), sla_map=0x0
      Seq(13 Placeholder_MPLS_1_0): state(alive), packet-loss(0.000%), latency(0.139), jitter(0.006), mos(4.404), bandwidth-up(1000000), bandwidth-dw(1000000), bandwidth-bi(2000000), sla_map=0x1

   2. Check the health of Spoke-2:

      The health-check is automatically running on the regular shortcut (H1_T11_0) and on the placeholder shortcut (Placeholder_MPLS_1_0).

      # diagnose sys sdwan  health-check
      Health Check(HUB):
      Seq(4 H1_T11): state(alive), packet-loss(0.000%), latency(0.214), jitter(0.012), mos(4.404), bandwidth-up(999998), bandwidth-dw(999997), bandwidth-bi(1999995), sla_map=0x1
      Seq(4 H1_T11_0): state(alive), packet-loss(0.000%), latency(0.296), jitter(0.030), mos(4.404), bandwidth-up(999999), bandwidth-dw(999998), bandwidth-bi(1999997), sla_map=0x1                        
      Seq(13 Placeholder_MPLS_1): state(dead), packet-loss(100.000%), sla_map=0x0
      Seq(13 Placeholder_MPLS_1_0): state(alive), packet-loss(0.000%), latency(0.159), jitter(0.023), mos(4.404), bandwidth-up(1000000), bandwidth-dw(1000000), bandwidth-bi(2000000), sla_map=0x1

   3. Check the SD-WAN status of Spoke-1:

      The regular shortcut (H1_T11_0 overlay) is preferred, because it is the better performing link, as determined by SD-WAN intelligence.

      # diagnose sys sdwan service4
      
      Service(1): Address Mode(IPV4) flags=0x4200 use-shortcut-sla use-shortcut
       Tie break: cfg
       Shortcut priority: 3
        Gen(33), TOS(0x0/0x0), Protocol(0): src(1->65535):dst(1->65535), Mode(sla), sla-compare-order
        Member sub interface(4):
          3: seq_num(4), interface(H1_T11):
             1: H1_T11_0(1152)
          4: seq_num(13), interface(dummy_MPLS_1):
             1: dummy_MPLS_1_0(1153)
        Members(4):
          1: Seq_num(4 H1_T11_0 overlay), alive, sla(0x1), gid(0), cfg_order(0), local cost(0), selected 
          2: Seq_num(13 Placeholder_MPLS_1_0 overlay), alive, sla(0x1), gid(0), cfg_order(1), local cost(0), selected
          3: Seq_num(4 H1_T11 overlay), alive, sla(0x1), gid(0), cfg_order(0), local cost(0), selected
          4: Seq_num(13 Placeholder_MPLS_1 overlay), dead, sla(0x0), gid(0), cfg_order(1), local cost(0)
        Src address(1):
              10.0.3.0-10.0.3.255
        Dst address(1):
              10.0.4.0-10.0.4.255

3. Diagnose the sniffer packet:

   The first few packets travel over the regular parent tunnel (H1_T11), and then the packets switch to the regular shortcut tunnel (H1_T11_0).

   # diagnose sniffer packet any 'host 10.0.4.2' 4
   interfaces=[any]
   filters=[host 10.0.4.2]
   4.967575 port4 in 10.0.3.2 -> 10.0.4.2: icmp: echo request
   4.967976 H1_T11 out 10.0.3.2 -> 10.0.4.2: icmp: echo request
   4.969587 H1_T11 in 10.0.4.2 -> 10.0.3.2: icmp: echo reply
   4.969629 port4 out 10.0.4.2 -> 10.0.3.2: icmp: echo reply
   5.968744 port4 in 10.0.3.2 -> 10.0.4.2: icmp: echo request
   5.968838 H1_T11 out 10.0.3.2 -> 10.0.4.2: icmp: echo request
   5.969785 H1_T11 in 10.0.4.2 -> 10.0.3.2: icmp: echo reply
   5.969842 port4 out 10.0.4.2 -> 10.0.3.2: icmp: echo reply
   6.969970 port4 in 10.0.3.2 -> 10.0.4.2: icmp: echo request
   6.970187 H1_T11 out 10.0.3.2 -> 10.0.4.2: icmp: echo request        
   9.969251 port4 in 10.0.3.2 -> 10.0.4.2: icmp: echo request
   9.969366 H1_T11_0 out 10.0.3.2 -> 10.0.4.2: icmp: echo request      
   9.970229 H1_T11_0 in 10.0.4.2 -> 10.0.3.2: icmp: echo reply
   9.970278 port4 out 10.0.4.2 -> 10.0.3.2: icmp: echo reply
   10.969982 port4 in 10.0.3.2 -> 10.0.4.2: icmp: echo request
   10.970199 H1_T11_0 out 10.0.3.2 -> 10.0.4.2: icmp: echo request

4. When the regular shortcut tunnel (H1_T11_0) is out of SLA, traffic switches to the placeholder shortcut tunnel (Placeholder_MPLS_1).

   1. Diagnose the SD-WAN service:

      The placeholder shortcut tunnel (Placeholder_MPLS_1) is preferred, and the regular shortcut tunnel (H1_T11_0 overlay) is out of SLA.

      # diagnose sys sdwan service4
      
      Service(1): Address Mode(IPV4) flags=0x4200 use-shortcut-sla use-shortcut
       Tie break: cfg
       Shortcut priority: 3
        Gen(87), TOS(0x0/0x0), Protocol(0): src(1->65535):dst(1->65535), Mode(sla), sla-compare-order
        Member sub interface(4):
          2: seq_num(4), interface(H1_T11):
             1: H1_T11_0(1167)
          4: seq_num(13), interface(Placeholder_MPLS_1):
             1: Placeholder_MPLS_1_0(1168)
        Members(4):
          1: Seq_num(13 Placeholder_MPLS_1_0 overlay), alive, sla(0x1), gid(0), cfg_order(1), local cost(0), selected
          2: Seq_num(4 H1_T11 overlay), alive, sla(0x1), gid(0), cfg_order(0), local cost(0), selected
          3: Seq_num(4 H1_T11_0 overlay), alive, sla(0x0), gid(0), cfg_order(0), local cost(0), selected 
          4: Seq_num(13 Placeholder_MPLS_1 overlay), dead, sla(0x0), gid(0), cfg_order(1), local cost(0)
        Src address(1):
              10.0.3.0-10.0.3.255
        Dst address(1):
              10.0.4.0-10.0.4.255

   2. Sniff the packet to see the traffic switch to the placeholder shortcut tunnel (Placeholder_MPLS_1_0):

      # diagnose sniffer packet any 'host 10.0.4.2' 4
      interfaces=[any]
      filters=[host 10.0.4.2]
      
      17.356165 port4 in 10.0.3.2 -> 10.0.4.2: icmp: echo request
      17.356261 Placeholder_MPLS_1_0 out 10.0.3.2 -> 10.0.4.2: icmp: echo request
      17.356857 Placeholder_MPLS_1_0 in 10.0.4.2 -> 10.0.3.2: icmp: echo reply
      17.356913 port4 out 10.0.4.2 -> 10.0.3.2: icmp: echo reply
      18.361038 port4 in 10.0.3.2 -> 10.0.4.2: icmp: echo request
      18.361247 Placeholder_MPLS_1_0 out 10.0.3.2 -> 10.0.4.2: icmp: echo request