HA virtual cluster setup

Virtual clustering is an extension of FGCP HA to allow multiple clusters to be formed between your HA members. In effect, each cluster consists of the same HA members, with the option to prioritize different members as the primary unit. Each cluster operates as its on active-passive FGCP HA cluster, with different virtual domains residing in the virtual cluster. The following custom settings can be configured per cluster:

config system ha
    set vcluster-status enable
    config vcluster
        edit <id>
            set override {enable | disable}			
            set priority <integer>
            set vdom <vdom_1>, ... [vdom_n]
            set monitor <interface_1>, ... [interface_n]
            set pingserver-monitor-interface <interface_1>, ... [interface_n]
        next
    end
end

override {enable | disable}	Enable/disable override and increase the priority of the unit that should always be the primary.
priority <integer>	Increase the priority to select the primary unit (0 - 255, default = 128).
vdom <vdom_1>, ... [vdom_n]	Set the virtual domains in the virtual cluster.
monitor <interface_1>, ... [interface_n]	Set the interfaces to check for port monitoring (or link failure).
pingserver-monitor-interface <interface_1>, ... [interface_n]	Set the interfaces to check for remote IP monitoring.

Active-passive virtual clustering uses VDOM partitioning to send traffic for some VDOMs to the primary FortiGate and traffic for other VDOMs to the secondary FortiGates. Traffic distribution between FortiGates can potentially improve throughput. If a failure occurs and only one FortiGate continues to operate, all traffic fails over to that FortiGate, similar to normal HA. If the failed FortiGates rejoin the cluster, the configured traffic distribution is restored.

In an active-passive virtual cluster of two FortiGates, the first and second FortiGates share traffic processing according to the VDOM partitioning configuration. The following is an example of two virtual clusters, with each member acting as primary for different vclusters.

If you add a third or fourth FortiGate, the first and second FortiGates process all traffic and the other one or two FortiGates operate in standby mode. If the first or second FortiGate fails, one of the other FortiGates becomes the new primary or secondary FortiGate and begins processing traffic.

For better load balancing, it is recommended to have as many vclusters as there are HA members. This way, each HA member can be a primary unit for each cluster, thereby processing traffic while standing by for the other vcluster as secondary. The following is an example of four FortiGates in an FGCP cluster, with four vclusters and four VDOMs. Each FortiGate is the primary unit for a vcluster and actively processes traffic as the primary member.

Virtual clustering and heartbeat interfaces

The HA heartbeat provides the same HA services in a virtual clustering configuration as in a standard HA configuration. One set of HA heartbeat interfaces provides HA heartbeat services for all of the VDOMs in the cluster. You do not have to add a heartbeat interface for each VDOM.

Special considerations for NPU-based VLANs in a virtual cluster

In an FGCP cluster, the primary FortiGate uses virtual MAC addresses when forwarding traffic, and the secondary uses the physical MAC addresses when forwarding traffic. In a virtual cluster, packets are sent with the cluster’s virtual MAC addresses. However, in the case of NPU offloading on a non-root VDOM, traffic that leaves an NPU-based VLAN will use the physical MAC address of its parent interface rather than the virtual MAC address. If this behavior is not desired, disable auto-asic-offload in the firewall policy where the VLAN interface is used.

Support up to 30 virtual clusters

FortiOS supports up to 30 virtual clusters, which allows more VDOMs to be spread across different virtual clusters without overlapping. Each virtual cluster supports its own failover conditions. Prior to 7.2.0, only two virtual clusters were supported.

When configuring virtual clusters, the group-id is limited to a value from 0 to 7. If the HA group-id is greater than 7, use the command line first to change the group-id before enabling virtual clusters.

config system ha
    set group-id <integer>
end

When upgrading from 7.0 or earlier, old virtual clusters will be lost if the group-id is larger than 7.

Basic configuration

This example shows a virtual cluster configuration consisting of two FortiGates. The virtual cluster has two VDOMs, root and eng_vdm.

The root VDOM can only be associated with virtual cluster 1. The VDOM that is assigned as the management VDOM can also only be associated with virtual cluster 1.

To set up an HA virtual cluster using the GUI:

1. Make all the necessary connections as shown in the topology diagram.
2. Configure a regular A-P cluster:
   1. Log in to one of the FortiGates.
   2. Go to System > HA and set the following options:

      Mode	Active-Passive
      Device priority	128 or higher
      Group name	Example_cluster
      Heartbeat interfaces	ha1 and ha2

      Except for the device priority, these settings must be the same on all FortiGates in the cluster.

   3. Leave the remaining settings as their default values. They can be changed after the cluster is in operation.
   4. Click OK.

      The FortiGate negotiates to establish an HA cluster. Connectivity with the FortiGate may be temporarily lost as the HA cluster negotiates and the FGCP changes the MAC addresses of the FortiGate's interfaces.

   5. Factory reset the other FortiGate that will be in the cluster, configure GUI access, then repeat step 2 (omitting setting the device priority) to join the cluster.
3. On the primary FortiGate, go to System > Settings and enable Virtual Domains.
4. Click Apply. You will be logged out of the FortiGate.
5. Log back in to the FortiGate, and ensure that you are in the global VDOM.
6. Create the eng_vdm VDOM:
   1. Go to System > VDOM and click Create New. The New Virtual Domain pane opens.
   2. Enter the name in the Virtual Domain field, then click OK.
7. Implement a virtual cluster by moving the new VDOM to virtual cluster 2:
   1. Go to System > HA and enable VDOM Partitioning.

   2. In the table, click Create New. The New Virtual Cluster pane opens.

   3. Click the + and add the eng_vdm VDOM.

   4. Click OK to save the virtual cluster.

      

   5. Click OK to save the HA configuration.

To set up an HA virtual cluster using the CLI:

1. Make all the necessary connections as shown in the topology diagram.
2. Configure a regular A-P cluster. See HA active-passive cluster setup.
3. Enable VDOMs:

   config system global
       set vdom-mode multi-vdom
   end

   You will be logged out of the FortiGate.

4. Create the eng_vdm VDOM:

   config vdom
       edit eng_vdm
       next
   end

5. Reconfigure the HA settings to be a virtual cluster:

   config system ha 
       set vcluster-status enable 
       config vcluster 
           edit 1 
               set vdom root 
               set override disable 
           next 
           edit 2
               set vdom eng_vdm 
               set override disable 
           next 
       end 
   end

Configuration with 30 virtual clusters

In this example, there are 30 customers managed by an MSSP on an HA cluster, and each customer VDOM needs to failover independently of other customer VDOMs. Each customer is assigned to a different virtual cluster with its own virtual cluster configurations. This may include different monitored interfaces, ping servers, and priority for the primary and secondary cluster members. Each virtual cluster will fail over according to their own virtual cluster configurations.

This example assumes an A-P cluster and VDOMs have already been configured. See HA active-passive cluster setup and Virtual Domains for more information.

For each virtual cluster, this example assumes that unit 1 has an HA priority of 200, while unit 2 has an HA priority of 100. By default, unit 1 will be the primary cluster member of all the virtual clusters.

To configure multiple virtual clusters in the GUI:

1. Go to System > HA and enable VDOM Partitioning.
2. Create a virtual cluster:
   1. In the table, click Create New. The New Virtual Cluster pane opens.
   2. Set the Device priority to 200.
   3. Click the + and add the Virtual domains.
   4. Optionally, click the + and add the Monitor interfaces.
   5. Click OK.
3. Repeat step 2 to create the remaining virtual clusters.
4. Click OK to save the HA configuration. The HA page summary displays the multiple virtual clusters, each with a Primary and Secondary HA member.
5. Edit the priority settings for the secondary members to be 100:
   1. Select the Secondary member in the table, and click Edit.
   2. Set the Priority to 100.
   3. Click OK.
6. Repeat step 5 for the remaining secondary members.

To configure multiple virtual clusters in the CLI:

1. Configure the primary FortiGate:

   config system ha
       set vcluster-status enable
       config vcluster
           edit 1
               set override disable
               set priority 200
               set vdom "vdom1"
           next
           edit 2
               set override disable
               set priority 200
               set vdom "vdom2"
           next
           ...
           edit 30
               set override disable
               set priority 200
               set vdom "vdom30"
           next
       end
   end

2. Configure the secondary FortiGate:

   config system ha
       set vcluster-status enable
       config vcluster
           edit 1
               set override disable
               set priority 100
               set vdom "vdom1"
           next
           edit 2
               set override disable
               set priority 100
               set vdom "vdom2"
           next
           ...
           edit 30
               set override disable
               set priority 100
               set vdom "vdom30"
           next
       end
   end