Use SD-WAN rules to steer multicast traffic

SD-WAN rules can now steer multicast traffic. When an SD-WAN member is out of SLA, multicast traffic can fail over to another SD-WAN member, and switch back when SLA recovers.

The new pim-use-sdwan option enables or disables the use of SD-WAN for PIM (Protocol Independent Multicast) when checking RP (Rendezvous Point) neighbors and sending packets.

config router multicast
    config pim-sm-global
        set pim-use-sdwan {enable | disable}
    end
end

When SD-WAN steers multicast traffic, ADVPN is not supported. Use the set shortcut option to disable shortcuts for the service: config system sdwan config service edit <id> set shortcut {enable | disable} next end end

Example 1

In this hub and spoke example, the PIM source is behind the hub FortiGate, and the RP is set to internal port (port2) of the hub firewall. Each spoke connects to the two WAN interfaces on the hub by using an overlay tunnel. The overlay tunnels are members of SD-WAN.

Receivers behind the spoke FortiGates request a stream from the source to receive traffic on tunnel1 by default. When the overlay tunnel goes out of SLA, the multicast traffic fails over to tunnel2 and continues to flow.

Following is an overview of how to configure the topology:

1. Configure the hub FortiGate in front of the PIM source. The RP is configured on internal port (port2) of the hub FortiGate.

2. Configure the spoke FortiGates.

3. Verify traffic failover.

To configure the hub:

1. On the hub, enable multicast routing, configure the multicast RP, and enable PIM sparse mode on each interface:

   config router multicast
       set multicast-routing enable
       config pim-sm-global
           config rp-address
               edit 1
                   set ip-address 172.16.205.1
               next
           end
       end
       config interface
           edit "tport1"
               set pim-mode sparse-mode
           next
           edit "tagg1"
               set pim-mode sparse-mode
           next
           edit "port2"
               set pim-mode sparse-mode
           next
       end
   end

To configure each spoke:

1. Enable SD-WAN with the following settings:

   • Configure the overlay tunnels as member of the SD-WAN zone.

   • Configure a performance SLA health-check using ping.

   • Configure a service rule for the PIM protocol with the following settings:

     • Use the lowest cost (SLA) strategy.

     • Monitor with the ping health-check.

   • Disable ADVPN shortcut.

   config system sdwan
       set status enable
       config zone
           edit "virtual-wan-link"
           next
       end
       config members
           edit 1
               set interface "tunnel1"
           next
           edit 2
               set interface "tunnel2"
           next
       end
       config health-check
           edit "ping"
               set server "172.16.205.1"
               set update-static-route disable
               set members 0
               config sla
                   edit 1
                   next
               end
           next
       end
       config service
           edit 1
               set mode sla
               set protocol 103
               set dst "all"
               config sla
                   edit "ping"
                       set id 1
                   next
               end
               set priority-members 1 2
               set use-shortcut-sla disable
               set shortcut disable
           next
           edit 2
               set mode sla
               set dst "all"
               config sla
                   edit "ping"
                       set id 1
                   next
               end
               set priority-members 1 2
           next
       end
   end

2. Enable multicast routing and configure the multicast RP. Enable PIM sparse-mode on each interface:

   config router multicast
       set multicast-routing enable
       config pim-sm-global
           set spt-threshold disable
           set pim-use-sdwan enable
           config rp-address
               edit 1
                   set ip-address 172.16.205.1
               next
           end
       end
       config interface
           edit "tunnel1"
               set pim-mode sparse-mode
           next
           edit "tunnel2"
               set pim-mode sparse-mode
           next
           edit "port4"
               set pim-mode sparse-mode
           next
       end
   end

To verify traffic failover:

With this configuration, multicast traffic starts on tunnel1. When tunnel1 becomes out of SLA, traffic switches to tunnel2. When tunnel1 is in SLA again, the traffic switches back to tunnel1.

The following health-check capture on the spokes shows tunnel1 in SLA with packet-loss (1.000%):

# diagnose sys sdwan health-check
Health Check(ping):
Seq(1 tunnel1): state(alive), packet-loss(0.000%) latency(0.056), jitter(0.002), mos(4.404), bandwidth-up(999999), bandwidth-dw(1000000), bandwidth-bi(1999999) sla_map=0x1
Seq(2 tunnel2): state(alive), packet-loss(0.000%) latency(0.100), jitter(0.002), mos(4.404), bandwidth-up(0), bandwidth-dw(0), bandwidth-bi(0) sla_map=0x1

# diagnose sys sdwan health-check
Health Check(ping):
Seq(1 tunnel1): state(alive), packet-loss(1.000%) latency(0.056), jitter(0.002), mos(4.404), bandwidth-up(999999), bandwidth-dw(1000000), bandwidth-bi(1999999) sla_map=0x1
Seq(2 tunnel2): state(alive), packet-loss(0.000%) latency(0.100), jitter(0.002), mos(4.404), bandwidth-up(0), bandwidth-dw(0), bandwidth-bi(0) sla_map=0x1

The following example shows tunnel1 out of SLA with packet-loss (3.000%):

# diagnose sys sdwan health-check
Health Check(ping):
Seq(1 tunnel1): state(alive), packet-loss(3.000%) latency(0.057), jitter(0.003), mos(4.403), bandwidth-up(999999), bandwidth-dw(1000000), bandwidth-bi(1999999) sla_map=0x0
Seq(2 tunnel2): state(alive), packet-loss(0.000%) latency(0.101), jitter(0.002), mos(4.404), bandwidth-up(0), bandwidth-dw(0), bandwidth-bi(0) sla_map=0x1

The following example shows tunnel1 back in SLA again:

# diagnose sys sdwan health-check
Health Check(ping):
Seq(1 tunnel1): state(alive), packet-loss(1.000%) latency(0.061), jitter(0.004), mos(4.404), bandwidth-up(999999), bandwidth-dw(1000000), bandwidth-bi(1999999) sla_map=0x0
Seq(2 tunnel2): state(alive), packet-loss(0.000%) latency(0.102), jitter(0.002), mos(4.404), bandwidth-up(0), bandwidth-dw(0), bandwidth-bi(0) sla_map=0x1

# diagnose sys sdwan health-check
Health Check(ping):
Seq(1 tunnel1): state(alive), packet-loss(0.000%) latency(0.061), jitter(0.004), mos(4.404), bandwidth-up(999999), bandwidth-dw(1000000), bandwidth-bi(1999999) sla_map=0x0
Seq(2 tunnel2): state(alive), packet-loss(0.000%) latency(0.102), jitter(0.002), mos(4.404), bandwidth-up(0), bandwidth-dw(0), bandwidth-bi(0) sla_map=0x1

The following example how traffic switches to tunnel2 while tunnel1 health-check is out of SLA. Source (172.16.205.11) sends traffic to the multicast group. Later the traffic switches back to tunnel1 once SLA returns to normal:

195.060797 tunnel1 in 172.16.205.11 -> 225.1.1.1: icmp: echo request
195.060805 port4 out 172.16.205.11 -> 225.1.1.1: icmp: echo request
196.060744 tunnel1 in 172.16.205.11 -> 225.1.1.1: icmp: echo request
196.060752 port4 out 172.16.205.11 -> 225.1.1.1: icmp: echo request
197.060728 tunnel1 in 172.16.205.11 -> 225.1.1.1: icmp: echo request
197.060740 port4 out 172.16.205.11 -> 225.1.1.1: icmp: echo request
198.060720 tunnel2 in 172.16.205.11 -> 225.1.1.1: icmp: echo request        
198.060736 port4 out 172.16.205.11 -> 225.1.1.1: icmp: echo request
199.060647 tunnel2 in 172.16.205.11 -> 225.1.1.1: icmp: echo request
199.060655 port4 out 172.16.205.11 -> 225.1.1.1: icmp: echo request
200.060598 tunnel2 in 172.16.205.11 -> 225.1.1.1: icmp: echo request
200.060604 port4 out 172.16.205.11 -> 225.1.1.1: icmp: echo request
... ...
... ...
264.060974 port4 out 172.16.205.11 -> 225.1.1.1: icmp: echo request
265.060950 tunnel2 in 172.16.205.11 -> 225.1.1.1: icmp: echo request
265.060958 port4 out 172.16.205.11 -> 225.1.1.1: icmp: echo request
266.060867 tunnel2 in 172.16.205.11 -> 225.1.1.1: icmp: echo request
266.060877 port4 out 172.16.205.11 -> 225.1.1.1: icmp: echo request
267.060828 tunnel2 in 172.16.205.11 -> 225.1.1.1: icmp: echo request
267.060835 port4 out 172.16.205.11 -> 225.1.1.1: icmp: echo request
268.060836 tunnel1 in 172.16.205.11 -> 225.1.1.1: icmp: echo request          
268.060854 port4 out 172.16.205.11 -> 225.1.1.1: icmp: echo request
269.060757 tunnel1 in 172.16.205.11 -> 225.1.1.1: icmp: echo request
269.060767 port4 out 172.16.205.11 -> 225.1.1.1: icmp: echo request
270.060645 tunnel1 in 172.16.205.11 -> 225.1.1.1: icmp: echo request
270.060653 port4 out 172.16.205.11 -> 225.1.1.1: icmp: echo request

Example 2

In this hub and spoke example, the PIM source is behind spoke 1, and the RP is configured on the hub FortiGate. BGP is used for routing. The hub uses embedded SLA in ICMP probes to determine the health of each tunnel, allowing it to prioritize healthy IKE routes.

The receiver is on another spoke. Upon requesting a stream, source passes the traffic to the RP on the hub FortiGate, and routes the traffic to the receiver over tunnel1. If a tunnel falls out of SLA, the multicast traffic fails over to the other tunnel.

In this configuration, SD-WAN steers multicast traffic by using embedded SLA information in ICMP probes. See also Embedded SD-WAN SLA information in ICMP probes. With this feature, the hub FortiGate can use the SLA information of the spoke's health-check to control BGP and IKE routes over tunnels.

Following is an overview of how to configure the topology:

1. Configure the hub FortiGate. The RP is configured on the hub FortiGate.

2. Configure the spoke FortiGate in front of the traffic receiver.

3. Configure the spoke FortiGate in front of the PIM source.

To configure the hub:

1. Configure loopbacks hub-lo1 172.31.0.1 for BGP and hub-lo100 172.31.100.100 for health-check:

   config system interface
       edit "hub-lo1"
           set vdom "hub"
           set ip 172.31.0.1 255.255.255.255
           set allowaccess ping
           set type loopback
           set snmp-index 82
       next
       edit "hub-lo100"
           set vdom "hub"
           set ip 172.31.100.100 255.255.255.255
           set allowaccess ping
           set type loopback
           set snmp-index 81
       next
   end

2. Enable multicast routing with the following settings:

   • Configure internal interface p25-v90 as RP.
   • Enable interfaces for PIM sparse-mode.

   config router multicast
       set multicast-routing enable
       config pim-sm-global
           config rp-address
               edit 1
                   set ip-address 192.90.1.11
               next
           end
       end
       config interface
           edit "p11"
               set pim-mode sparse-mode
           next
           edit "p101"
               set pim-mode sparse-mode
           next
           edit "p25-v90"
               set pim-mode sparse-mode
           next
       end
   end

3. Enable SD-WAN with the following settings:

   • Add interfaces p11 and p101 as members.
   • Configure embedded SLA health-checks to detect ICMP probes from each overlay tunnel. Prioritize based on the health of each tunnel.

   config system sdwan
       set status enable
       config zone
           edit "virtual-wan-link"
           next
       end
       config members
           edit 1
               set interface "p11"
           next
           edit 2
               set interface "p101"
           next
       end
       config health-check
           edit "1"
               set detect-mode remote
               set probe-timeout 60000
               set recoverytime 1
               set sla-id-redistribute 1
               set members 1
               config sla
                   edit 1
                       set link-cost-factor latency
                       set latency-threshold 100
                       set priority-in-sla 10
                       set priority-out-sla 20
                   next
               end
           next
           edit "2"
               set detect-mode remote
               set probe-timeout 60000
               set recoverytime 1
               set sla-id-redistribute 1
               set members 2
               config sla
                   edit 1
                       set link-cost-factor latency
                       set latency-threshold 100
                       set priority-in-sla 15
                       set priority-out-sla 25
                   next
               end
           next
       end
   end

4. Configure BGP to peer with neighbors. Neighbor group is configured for tunnel interface IP addresses:

   config router bgp
       set as 65505
       set router-id 172.31.0.1
       set ibgp-multipath enable
       set additional-path enable
       set recursive-inherit-priority enable
       config neighbor-group
           edit "gr1"
               set remote-as 65505
               set update-source "hub-lo1"
               set additional-path both
               set route-reflector-client enable
           next
       end
       config neighbor-range
           edit 1
               set prefix 10.10.0.0 255.255.0.0
               set neighbor-group "gr1"
           next
           edit 66
               set prefix 172.31.0.66 255.255.255.255
               set neighbor-group "gr1"
           next
       end
       config network
           ....
           edit 90
               set prefix 192.90.0.0 255.255.0.0
           next
       end
   end

To configure the spoke (in front of the receiver):

1. Enable multicast routing to use SD-WAN. Configure the RP address. Enable interfaces for PIM sparse-mode.

   config router multicast
       set multicast-routing enable
       config pim-sm-global
           set spt-threshold disable
           set pim-use-sdwan enable
           config rp-address
               edit 1
                   set ip-address 192.90.1.11
               next
           end
       end
       config interface
           edit "p195"
               set pim-mode sparse-mode
           next
           edit "p196"
               set pim-mode sparse-mode
           next
           edit "internal4"
               set pim-mode sparse-mode
               set static-group "225-1-1-122"
           next
       end
   end

2. Configure SD-WAN with the following settings:

   • Add overlay tunnel interfaces as members.

   • Configure a performance SLA health-check to send ping probes to the hub.

   • Configure a service rule for the PIM protocol. Use the lowest cost (SLA) strategy, and monitor with the ping health-check.

   • Disable ADVPN shortcuts.

   config system sdwan
       set status enable
       config zone
           edit "virtual-wan-link"
           next
       end
       config members
           edit 6
               set interface "p196"
           next
           edit 5
               set interface "p195"
           next
       end
       config health-check
           edit "ping"
               set server "172.31.100.100"
               set update-static-route disable
               set members 0
               config sla
                   edit 1
                       set link-cost-factor latency
                       set latency-threshold 100
                   next
               end
           next
       end
       config service
           edit 1
               set mode sla
               set protocol 103
               set dst "all"
               config sla
                   edit "ping"
                       set id 1
                   next
               end
               set priority-members 5 6
               set use-shortcut-sla disable
               set shortcut disable
           next
           edit 2
               set mode sla
               set dst "all"
               config sla
                   edit "ping"
                       set id 1
                   next
               end
               set priority-members 5 6
           next
       end
   end

3. Configure BGP and set neighbors to the overlay gateway IP address on the hub:

   config router bgp
       set as 65505
       set router-id 122.1.1.122
       set ibgp-multipath enable
       set additional-path enable
       config neighbor
           edit "10.10.100.254"
               set soft-reconfiguration enable
               set remote-as 65505
               set connect-timer 10
               set additional-path both
           next
           edit "10.10.101.254"
               set soft-reconfiguration enable
               set remote-as 65505
               set connect-timer 10
               set additional-path both
           next
       end
       config network
           edit 3
               set prefix 192.84.0.0 255.255.0.0
           next
       end
   end

4. Configure the default gateway to use the SD-WAN zone. Other routes are for the underlay to route traffic to the hub's WAN interfaces:

   config router static
       edit 10
           set distance 1
           set sdwan-zone "virtual-wan-link"
       next
       ....
       next
   end

To configure the spoke (in front of the source):

1. Enable multicast routing to use SD-WAN. Configure the RP address. Enable interfaces for PIM sparse-mode:

   config router multicast
       set multicast-routing enable
       config pim-sm-global
           set pim-use-sdwan enable
           config rp-address
               edit 1
                   set ip-address 192.90.1.11
               next
           end
       end
       config interface
           edit "p198"
               set pim-mode sparse-mode
           next
           edit "p200"
               set pim-mode sparse-mode
           next
           edit "npu0_vlink0"
               set pim-mode sparse-mode
           next
       end
   end

2. Configure loopback interface lo66 for BGP and sourcing SD-WAN traffic:

   config system interface
       edit "lo66"
           set vdom "root"
           set ip 172.31.0.66 255.255.255.255
           set allowaccess ping
           set type loopback
           set snmp-index 21
       next
   end

3. Configure SD-WAN:

   • Add overlay tunnel interfaces as members.
   • Configure a performance SLA health-check to send ping probes to the hub.
   • Configure a service rule for the PIM protocol. Use the lowest cost (SLA) strategy, and monitor with the ping health-check.
   • Disable the use of an ADVPN shortcut.

   In the following example, 11.11.11.11 is the underlay address for one of the WAN links on the hub, and 172.31.100.100 is the loopback address on the server.

   config system sdwan
       set status enable
       config zone
           edit "virtual-wan-link"
           next
           edit "overlay"
           next
       end
       config members
           edit 1
               set interface "p198"
               set zone "overlay"
               set source 172.31.0.66
           next
           edit 2
               set interface "p200"
               set zone "overlay"
               set source 172.31.0.66
           next
       end
       config health-check
           edit "ping"
               set server "11.11.11.11"            
               set members 0
               config sla
                   edit 1
                       set link-cost-factor latency
                       set latency-threshold 100
                   next
               end
           next
           edit "HUB"
               set server "172.31.100.100"        
               set embed-measured-health enable
               set members 0
               config sla
                   edit 1
                       set link-cost-factor latency
                       set latency-threshold 100
                   next
               end
           next
       end
        config service
           edit 1
               set mode sla
               set protocol 103
               set dst "all"
               config sla
                   edit "ping"
                       set id 1
                   next
               end
               set priority-members 1 2
               set use-shortcut-sla disable
               set shortcut disable
           next
           edit 2
               set mode sla
               set dst "all"
               config sla
                   edit "ping"
                       set id 1
                   next
               end
               set priority-members 1 2
           next
       end
   end

4. Configure BGP:

   config router bgp
       set as 65505
       set router-id 123.1.1.123
       set ibgp-multipath enable
       set additional-path enable
       config neighbor
           edit "172.31.0.1"
               set next-hop-self enable
               set soft-reconfiguration enable
               set remote-as 65505
               set update-source "lo66"
           next
       end
       config network
           edit 3
               set prefix 192.87.0.0 255.255.0.0
           next
       end
   end

5. Configure the default gateway to use the SD-WAN zone. Other routes are for the underlay to route to the hub's WAN interfaces:

   config router static
       edit 10
           set distance 1
           set sdwan-zone "virtual-wan-link" "overlay"
       next
       ...
       next
   end