Using the Fabric Overlay Orchestrator

If you cannot view the VPN > Fabric Overlay Orchestrator tree menu, configure the FortiGate as a root or a downstream device in the Security Fabric. See Configuring the root FortiGate and downstream FortiGates for more details.

The Fabric Overlay Orchestrator does not work when VDOM mode is enabled.

The following steps should be used to configure a self-orchestrated SD-WAN overlay within a single Security Fabric. These steps must be followed in order, and assume that the prerequisites and network topology are in place.

1. Configure the root FortiGate using the Fabric Overlay Orchestrator.

2. Configure one or more downstream FortiGates using the Fabric Overlay Orchestrator.

3. Configure an overlay on the spoke for an additional incoming interface on the hub (if applicable).

4. Verify the firewall policies on the hub FortiGate.

5. Verify the Fabric Overlay created by the Fabric Overlay Orchestrator:

   1. Verify the IPsec VPN tunnels on the hub FortiGate.

   2. Verify BGP routing on the hub FortiGate.

   3. Verify the performance SLAs on the hub FortiGate.

   4. Verify the firewall policies on a spoke FortiGate.

   5. Verify the IPsec VPN tunnels on a spoke FortiGate.

   6. Verify BGP routing on a spoke FortiGate.

   7. Verify the performance SLAs on a spoke FortiGate.

   8. Verify the spoke-to-spoke ADVPN communication.

6. Configure SD-WAN rules on the hub FortiGate.

7. Configure SD-WAN rules on the spoke FortiGates.

When configuring the root and downstream FortiGates, the Fabric Overlay Orchestrator configures the following settings in the background:

• IPsec overlay configuration (hub and spoke ADVPN tunnels)

• BGP configuration

• Policy routing

• SD-WAN zones

• SD-WAN performance SLAs

The FortiGate’s role in the SD-WAN overlay is automatically determined by its role in the Security Fabric. The Fabric root will be the hub, and any first-level downstream devices from the Fabric root will be spokes.

After using the Fabric Overlay Orchestrator on all FortiGates and verifying the overlay settings, complete the SD-WAN deployment configuration using steps 3 (if applicable), and steps 6 and 7. See SD-WAN rules for more information.

Creating firewall policies

The Fabric Overlay Orchestrator can create firewall policies to allow all traffic through the SD-WAN overlay, or firewall policies to just allow health check traffic through it instead. When the Fabric Overlay Orchestrator is enabled on the root FortiGate, there are three Policy creation options:

• Automatic: automatically create policies for the loopback interface and tunnel overlays.

• Health check: automatically create a policy for the loopback interface so the SD-WAN health checks are functional.

• Manual: no policies are automatically created.

The Automatic policy creation option creates wildcard allow policies for the tunnel overlays. For some cases, these policies do not provide the necessary granularity to restrict overlay traffic to specific subnets or hosts.

When the Fabric Overlay Orchestrator is configured on a device, changing the policy creation rule will create new policies based on the rule, but it will not delete existing policies. Deleting existing policies must be performed manually.

Configuring the root FortiGate using the Fabric Overlay Orchestrator

To configure the root FortiGate using the Fabric Overlay Orchestrator:

1. Go to VPN > Fabric Overlay Orchestrator.

2. Set the Status to Enabled. The Role is automatically selected based on the FortiGate's role in the Security Fabric. Ensure that Hub is selected. The Fabric root must always be the hub.

3. Set Policy creation to Automatic.

   

4. Click Next. The Overlay settings appear.

5. Select one or more interfaces as the Incoming interface or the underlay link over which the VPN overlay will be built (two incoming interfaces are selected in this example).

6. Enter the Pre-shared key.

   

7. Click Next. The Local Network settings appear.

8. Configure routing and local subnets to share the following with the VPN network:

   BGP AS	Optional setting to configure the BGP AS number. By default, this is set to 65400.
   Loopback address block	Optional setting to configure the loopback IP address. By default, this is set to 10.20.1.1/255.255.255.0.
   Shared interfaces	Select the interface of the local network to share with the VPN network.

   

9. Click Next. The Summary page appears.

10. Review the settings, then click Apply.

    

    An updated Summary page appears with all the settings.

    

    Note the following settings in this example:

    SD-WAN zone	Located in the Status section: fabric_vpn_sdwan.
    VPN tunnels	Located in the Overlay section in the Incoming interface table under the Phase 1 Interface column: fabric_vpn1 and fabric_vpn2.
    BGP	Located in the Local Network section. The BGP AS is 65400. The Shared subnets are 10.20.1.1/32 and 172.16.1.0/30.
    Loopback interface	Located in the Local Network section: F_Hub_loop.
    Firewall policies	Located in two sections: Overlay section in the Incoming interface table under the Policy column: Fabric_overlay_0 and Fabric_overlay_1 Local Network section in the Shared subnets table under the Policies column: fabric_vpn_1_in, fabric_vpn_0_out, and fabric_vpn_0_in

Configuring a downstream FortiGate using the Fabric Overlay Orchestrator

To configure a downstream FortiGate using the Fabric Overlay Orchestrator:

1. Go to VPN > Fabric Overlay Orchestrator.

2. Set the Status to Enabled. The Role is automatically selected based on the FortiGate's role in the Security Fabric. Ensure that Spoke is selected. Only downstream first-level FortiGates can be spokes.

   

3. Click Next. The Local Network settings appear.

4. Configure the routing and local subnets to share with the VPN network in the following fields:

   Shared interfaces

   Select the interface of the local network to share with the VPN network.

   

5. Click Next. As the downstream FortiGate updates, a Configuring spoke Fabric VPN from root FortiGate message appears The Summary page appears once the update is complete.

   

6. Review the settings, then click Apply.

   An updated Summary page appears with all the settings.

   

   Note the following settings in this example:

   SD-WAN zone	Located in the Status section: fabric_vpn_sdwan.
   VPN tunnels	Located in the Overlay section in the Incoming interface table under the Phase 1 Interface column: fabric_vpn1.
   BGP	Located in the Local Network section. The BGP AS is 65400. The Shared subnets are 10.1.1.0/24 and 10.20.1.2/32.
   Loopback interface	Located in the Local Network section: F_Hub_loop.
   Firewall policies	Located in the Local Network section in the Shared subnets table under the Policies column: fabric_vpn_0_out, fabric_vpn_0_in, and fabric_vpn_1_in.

The loopback IP addresses for the branches are generated based on the index number of the trusted device in the root FortiGate's Security Fabric (HUB) configuration.

config system csf
    set status enable
    set group-name "fabric"
    config trusted-list
        edit "FGVM02TM22000001"
            set serial "FGVM02TM22000001"
            set index 1
        next
        edit "FGVM02TM22000002"
            set serial "FGVM02TM22000002"
            set index 2
        next
    end
end

For example, if Branch1 (index 1) is the first FortiGate and Branch2 (index 2) is the second FortiGate authorized on the root FortiGate, the loopback addresses are generated as follows:

• Branch1 loopback IP: 10.20.1.2
• Branch2 loopback IP: 10.20.1.3

Configuring an overlay on the spoke for an additional incoming interface on the hub

A hub typically includes two incoming interfaces, but additional interfaces can be configured if needed. On downstream devices, the following warning is displayed on the Fabric Overlay Orchestrator page that The hub has multiple overlays configured but only one of the overlays on this device have been configured. Please manually select which interface to use for the other overlays.

To configure an additional incoming interface on a spoke:

1. Go to VPN > Fabric Overlay Orchestrator.

2. Click Configure Overlays in the warning box.

3. Navigate to the Overlay section, click the + in the Incoming interface field, and select WAN2 (port2) to add it to the overlay.

   

4. Click Next, then complete the remaining steps in the GUI wizard. On the Summary page, the additional interface WAN2 (port2) appears in the Incoming interfaces table.

   

Verifying the firewall policies on the hub FortiGate

Different policies are created on the hub FortiGate based on the Policy creation setting in the Fabric Overlay Orchestrator configuration (Automatic, Health check, or Manual).

Automatic

Go to Policy & Objects > Firewall Policy to verify that wildcard firewall policies have been configured on the hub FortiGate. This Fabric Overlay Orchestrator configuration example uses automatic policy creation, and the following firewall policies are configured:

The Automatic policy creation option creates wildcard allow policies for the tunnel overlays. For some cases, these policies do not provide the necessary granularity to restrict overlay traffic to specific subnets or hosts.

Health check

Go to Policy & Objects > Firewall Policy to verify that a single firewall policy allowing health check traffic to the hub’s loopback has been configured on the hub FortiGate. For example:

Manual

Go to Policy & Objects > Firewall Policy to verify that no firewall policies have been created by the Fabric Overlay Orchestrator. If desired, firewall policies must be manually configured on the hub FortiGate to allow traffic to the loopback interface for health checks and the overlays.

Verifying the Fabric Overlay created by the Fabric Overlay Orchestrator

To verify the IPsec VPN tunnels on the hub:

1. Go to Dashboard > Network and click the IPsec widget to expand it.

2. Verify that there are two tunnels established for each phase 1 interface.

   

   The naming convention <tunnel_name>_<number> indicates the relative order in which the tunnels were established:

   fabric_vpn_1_0	VPN tunnel listening on the hub’s WAN1 incoming interface; established with spoke 1 using its WAN1 interface
   fabric_vpn_1_1	VPN tunnel listening on the hub’s WAN1 incoming interface; established with spoke 2 using its WAN1 interface
   fabric_vpn_2_0	VPN tunnel listening on the hub’s WAN2 incoming interface; established with spoke 1 using its WAN2 interface
   fabric_vpn_2_1	VPN tunnel listening on the hub’s WAN2 incoming interface; established with spoke 2 using its WAN2 interface

Verify the BGP routing on the hub:

1. In the CLI, check the BGP peering status:

   HUB # get router info bgp summary
   
   VRF 0 BGP router identifier 10.20.1.1, local AS number 65400
   BGP table version is 11
   1 BGP AS-PATH entries
   0 BGP community entries
   Next peer check timer due in 43 seconds
   
   Neighbor   V         AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
   10.10.10.1 4      65400      23      27       11    0    0 00:09:28        2
   10.10.10.2 4      65400      16      16       11    0    0 00:06:36        2
   10.10.11.1 4      65400      14      20       11    0    0 00:09:22        2
   10.10.11.2 4      65400       7      11       11    0    0 00:03:22        2
   
   Total number of neighbors 4

2. Check the BGP advertised routes:

   HUB # get router info bgp neighbors 10.10.10.1 advertised-routes
   VRF 0 BGP table version is 11, local router ID is 10.20.1.1
   Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
   Origin codes: i - IGP, e - EGP, ? - incomplete
   
      Network          Next Hop            Metric     LocPrf Weight RouteTag Path
   *>i10.1.1.0/24      10.10.11.1                    100      0        0 i <-/2>
   *>i10.1.2.0/24      10.10.11.2                    100      0        0 i <-/2>
   *>i10.1.2.0/24      10.10.10.2                    100      0        0 i <-/1>
   *>i10.10.10.0/24    10.10.10.253                  100  32768        0 i <-/1>
   *>i10.10.11.0/24    10.10.10.253                  100  32768        0 i <-/1>
   *>i10.20.1.1/32     10.10.10.253                  100  32768        0 i <-/1>
   *>i10.20.1.2/32     10.10.11.1                    100      0        0 i <-/2>
   *>i10.20.1.3/32     10.10.11.2                    100      0        0 i <-/2>
   *>i10.20.1.3/32     10.10.10.2                    100      0        0 i <-/1>
   *>i172.16.1.0/30    10.10.10.253                  100  32768        0 i <-/1>
   
   Total number of prefixes 10

3. In the GUI, go to Dashboard > Network and click the Routing widget to expand it.

4. In the dropdown, select BGP Neighbors.

   

To verify the performance SLAs on the hub:

1. Go to Network > SD-WAN and select the Performance SLAs tab.

2. Verify that the performance SLAs are automatically created for each spoke. The performance SLA naming uses the serial number of the spoke FortiGate. There are two new entries.

   

To verify the firewall policies on a spoke FortiGate:

Different policies are created on the spoke FortiGates based on the hub's Policy creation setting in the Fabric Overlay Orchestrator configuration (Automatic, Health check, or Manual). The Automatic setting is used in this example.

1. Go to Policy & Objects > Firewall Policy.

2. Verify that wildcard firewall policies have been configured.

   

The Automatic policy creation option creates wildcard allow policies for the tunnel overlays. For some cases, these policies do not provide the necessary granularity to restrict overlay traffic to specific subnets or hosts.

If the hub's Policy creationsetting is Health Check, a single firewall policy that allows health check traffic to the spoke’s loopback should be configured on the spoke FortiGates:

If the hub's Policy creationsetting is Manual, there should be no new policies created by the Fabric Overlay Orchestrator. If desired, firewall policies must be manually configured on the spoke FortiGates to allow traffic to the loopback interface for health checks and the overlays.

To verify the IPsec VPN tunnels on a spoke:

1. Go to Dashboard > Network and click the IPsec widget to expand it.

2. Verify the IPsec tunnels that go back to the hub.

   

To verify BGP routing on a spoke:

1. In the CLI, check the BGP peering status:

   Branch1 # get router info bgp summary
   
   VRF 0 BGP router identifier 10.20.1.2, local AS number 65400
   BGP table version is 5
   1 BGP AS-PATH entries
   0 BGP community entries
   
   Neighbor     V         AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
   10.10.10.253 4      65400      41      37        4    0    0 00:23:39        8
   10.10.11.253 4      65400      38      34        4    0    0 00:23:33        8
   
   Total number of neighbors 2

2. Check the BGP advertised routes:

   Branch1 # get router info bgp neighbors 10.10.10.253 advertised-routes
   VRF 0 BGP table version is 5, local router ID is 10.20.1.2
   Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
   Origin codes: i - IGP, e - EGP, ? - incomplete
   
      Network          Next Hop            Metric     LocPrf Weight RouteTag Path
   *>i10.1.1.0/24      10.10.10.1                    100  32768        0 i <0/->
   *>i10.20.1.2/32     10.10.10.1                    100  32768        0 i <0/->
   
   Total number of prefixes 2

3. In the GUI, go to Dashboard > Network and click the Routing widget to expand it.

4. In the dropdown, select BGP Neighbors.

   

To verify the performance SLAs on a spoke:

1. Go to Network > SD-WAN and select the Performance SLAs tab.
2. Verify that the performance SLA is automatically created for the hub FortiGate. There is a new entry (FABRIC_VPN_HUB).

   

To verify the spoke-to-spoke ADVPN communication:

1. From Branch1, ping Branch2 (10.20.1.3):

   Branch1 # exec ping-options source 10.20.1.2
   Branch1 # exec ping 10.20.1.3
   PING 10.20.1.3 (10.20.1.3): 56 data bytes
   64 bytes from 10.20.1.3: icmp_seq=0 ttl=254 time=27.7 ms
   64 bytes from 10.20.1.3: icmp_seq=2 ttl=255 time=17.4 ms
   64 bytes from 10.20.1.3: icmp_seq=3 ttl=255 time=17.5 ms
   64 bytes from 10.20.1.3: icmp_seq=4 ttl=255 time=17.4 ms
   
   --- 10.20.1.3 ping statistics ---
   5 packets transmitted, 4 packets received, 20% packet loss
   round-trip min/avg/max = 17.4/20.0/27.7 ms

2. Verify the IPsec tunnel summary.

   • In the CLI, enter the following:

     Branch1 # get vpn ipsec tunnel summary
     'fabric_vpn_1_0' 10.198.3.2:0  selectors(total,up): 1/1  rx(pkt,err): 25/0  tx(pkt,err): 26/2
     'fabric_vpn_1' 10.198.5.2:0  selectors(total,up): 1/1  rx(pkt,err): 8032/0  tx(pkt,err): 8022/2
     'fabric_vpn_2' 10.198.6.2:0  selectors(total,up): 1/1  rx(pkt,err): 7462/0  tx(pkt,err): 7478/1

     The fabric_vpn_1_0 tunnel was created for spoke 1-to-spoke 2 communication.

   • In the GUI, go to Dashboard > Network and click the IPsec widget to expand it.

     

3. Verify that the performance SLA was updated. Go to Network > SD-WAN and select the Performance SLAs tab.

   

   The first performance SLA, fabric_vpn_1, that corresponds to the spoke-to-hub VPN tunnel is shown as up. The second one, fabric_vpn_1 that corresponds to the spoke-to-spoke VPN tunnel (fabric_vpn_1_0) is shown as down since 10.20.1.1 is the IP address corresponding to the hub’s loopback interface that is not present on another spoke.

Configuring SD-WAN rules on the hub FortiGate

On the hub, the Fabric Overlay Orchestrator automatically creates a performance SLA that corresponds to each spoke FortiGate using the serial number as the name of the performance SLA. SD-WAN rules must be configured on the hub FortiGate to direct traffic to each of the spokes using these performance SLAs.

To configure SD-WAN rules on the hub FortiGate:

1. Go to Network > SD-WAN, select the SD-WAN Rules tab, and click Create New.

2. Enter a name (such as Hub-To-Br1).

3. In the Source section, set the Address to the local subnet of the hub.

4. Configure the following in the Destination section:

   1. Set the Address to the local subnet of the spoke. If an address object does not exist yet, click Create in the slide-out pane and configure the address.

   2. Set the Protocol number as needed (default = ANY).

5. Configure the following in the Outgoing Interfaces section:

   1. Set the Interface selection strategy to Lowest cost (SLA).

   2. Set the Interface preference to the SD-WAN members.

   3. Set Required SLA target to the corresponding performance SLA created by the Fabric Overlay Orchestrator for this the spoke. The name is based on the spoke FortiGate's serial number (FGVM0XXX00000000 #1).

6. Click OK.

7. Repeat these steps for the other spoke. Ensure the Name is unique, and that the Destination address corresponds to the local subnet behind the spoke.

If you need to disable the Fabric Overlay Orchestrator on the hub FortiGate by setting the Status to Disabled, you must first delete any SD-WAN rules on the hub FortiGate created using this procedure to ensure the added configuration does not block the clean-up process.

Configuring SD-WAN rules on the spoke FortiGates

On each spoke, the Fabric Overlay Orchestrator automatically creates a performance SLA that corresponds to the hub FortiGate. An SD-WAN rule must be configured on the spoke FortiGates to direct traffic to the hub FortiGate using this performance SLA.

To configure an SD-WAN rule on a spoke FortiGate:

1. Go to Network > SD-WAN, select the SD-WAN Rules tab, and click Create New.

2. Enter a name (such as LAN-to-HUB).

3. In the Source section, set the Address to the local subnet of the spoke.

4. Configure the following in the Destination section:

   1. Set the Address to the local subnet of the hub. If an address object does not exist yet, click Create in the slide-out pane and configure the address.

   2. Set the Protocol number as needed (default = ANY).

5. Configure the following in the Outgoing Interfaces section:

   1. Set the Interface selection strategy to Lowest cost (SLA).

   2. Set the Interface preference to the SD-WAN members.

   3. Set Required SLA target to the corresponding performance SLA created by the Fabric Overlay Orchestrator, which is named FABRIC_VPN_HUB#1 by default.

6. Click OK.

If you need to disable the Fabric Overlay Orchestrator on a spoke FortiGate by setting the Status to Disabled, you must first delete any SD-WAN rules on the spoke FortiGate created using this procedure to ensure the added configuration does not block the clean-up process.