Configuring High Availability (HA) basic settings

If you want to deploy the FortiWeb appliances in HA mode, it's recommended to first complete the HA basic settings introduced in this topic before you start setting other configurations.

When basic settings are done, there will be heartbeat links between the HA member to synchronize configuration. The active unit’s configuration is almost entirely synchronized to the passive appliance, so that changes made to the active appliance are propagated to the standby or slave appliance, ensuring that it is prepared for a failover. See Synchronization for configurations and data that are synchronized in HA group.

HA requirements

For active-passive HA, you need two identical physical FortiWeb appliances; for standard or high volume active-active HA, you need two or more (up to eight) identical physical FortiWeb appliances and firmware versions. For introductions on the HA modes, see FortiWeb high availability (HA) .
Redundant network topology: if the active or master appliance fails, physical network cabling and routes must be able to redirect web traffic to the standby or slave appliances. For details, see Topologies for high availability (HA) clustering.
At least one physical port on each HA appliance connected via crossover cables, or through switches. For details, see HA heartbeat.
For FortiWeb-VM:
- A valid license for all HA members. You cannot configure HA with trial licenses.
- Ensure the HA members have the same number of ports and are configured with the same amount of memory and vCPUs.

FortiWeb-VM supports HA. However, if you do not wish to use the native HA, you can use your hypervisor or VM environment manager to install your virtual appliances over a hardware cluster to improve availability. For example, VMware clusters can use vMotion or VMware HA.

Basic settings

Basic settings apply for all the HA modes, including active-passive, standard active-active, and high volume active-active modes.

To configure HA:

If the HA group will use FortiGuard services, license all FortiWeb appliances in the HA group, and register them with the Fortinet Customer Service & Support website:

https://support.fortinet.com/

FortiWebs in an HA group use the FortiGuard Distribution Server (FDS) to validate licenses and contracts. The master appliance maintains a connection with the FDS, and each slave appliance verifies its license status via the master appliance's connection. The master appliance will also use the connection with the FDS to forward contract information to each slave appliance.

If you license only the primary appliance in an HA group, after a failover, the secondary appliance will not be able to use the FortiGuard service. This could cause traffic to be scanned with out-of-date definitions, potentially allowing newer attacks.

Cable both appliances into a redundant network topology.
For details, see Configuring redundant interfaces.
Physically link the FortiWeb appliances that will be members of the HA group.

For the HA group, you must link at least one of their ports (e.g. port4 to port4) for heartbeat and synchronization traffic between members of the HA group. You can either:

Link two appliances directly via a crossover cable (for only two appliances in a group)
Link the appliances through a switch (for more than two appliances in a group)

If a switch is used to connect the heartbeat interfaces, the heartbeat interfaces must be reachable by Layer 2 multicast. To improve fault tolerance and reliability, link the ports through two separate switches. Do not connect these switches to your overall network, which could introduce a potential attack point, and could also allow network load to cause latency in the heartbeat, which could cause an unintentional failover.

Note: If the heartbeat is accidentally interrupted for an active-passive HA group, such as when a network cable is temporarily disconnected, the secondary appliance will assume that the primary unit has failed, and become the new primary appliance. If no failure has actually occurred, both FortiWeb appliances will be operating as primary appliances simultaneously.

To avoid unintentional failovers due to accidental detachment or hardware failure of a single heartbeat link, make two heartbeat links.

For example, you might link port3 to port3 on the other appliance, and link port4 to port4 on the other appliance, then configure both appliances to use those network interfaces for heartbeat and synchronization.

Log in to all the appliances as the admin administrator account.
Accounts whose access profile includes Read and Write permissions to the System Configuration area can configure HA, but may not be able to use features that may be necessary when using HA, such as logs and network configuration.

On all the appliances, go to System > High Availability > Settings.
To access this part of the web UI, your administrator's account access profile must have Read and Write permission to items in the System Configuration category. For details, see Permissions.

By default, each FortiWeb appliance operates as a single, standalone appliance: only the Configured HA mode drop-down list appears, with the Standalone option selected.

For Mode, select Active-Passive, Active-Active-Standard, or Active-Active-High Volume as desired.

Fail-open is disabled when the FortiWeb appliance is configured as part of an HA pair. For details about fail-to-wire, see Fail-to-wire for power loss/reboots.

Additional options appear that enable you to configure HA.

Configure these settings:

Device Priority	Type the priority of the appliance when selecting the active-passive primary (or active-active master) appliance in the HA group. On active-passive standby or active-active slave devices, this setting can be reconfigured using the CLI command `execute ha manage <serial-number_str> <priority_int>`. For details, see FortiWeb CLI Reference. This setting is optional. The smaller the number, the higher the priority. The valid range is 0 to 9. The default is 5. Note: By default, unless you enable Override, uptime is more important than this setting. For details, see How HA chooses the active appliance.
Override	Enable to make Device Priority a more important factor than uptime when selecting the main appliance. See How HA chooses the active appliance.
Group-name	Type a name to identify the HA pair if you have more than one. This setting is optional, and does not affect HA function. The maximum length is 63 characters.
Group ID	Type a number that identifies the HA group. All the members of the HA group must have the same group ID. If you have more than one HA group on the same network, each HA group must have a different group ID. Changing the group ID changes the group’s virtual MAC address. The valid range is 0 to 63. The default value is 0.
Session Pickup	Available only in Active-Active-Standard mode. Enable so that the master unit in the HA group synchronizes the session table with all group units. If a group unit fails, the HA session table information is available to the remaining group units which can use the session table to resume connections without interruption. Enable for session fail-over protection. If this is not required, disabling may reduce CPU usage and reduce HA heartbeat network bandwidth usage. Note: Only sessions that have been established for longer than 30 seconds will be synchronized.
Layer 7 Persistence Synchronization	Enable so that FortiWeb enforces session persistence between the master and slave appliances at the application layer. Note: This option is available only when the Mode is Active-Passive.
Monitor Interface	Select one or more network interfaces that each directly correlate with a physical link. These ports will be monitored for link failure. Port monitoring (also called interface monitoring) monitors physical network ports to verify that they are functioning properly and linked to their networks. If the physical port fails or the cable becomes disconnected, a failover occurs. You can monitor physical interfaces, but not VLAN subinterfaces or 4-port switches. If you select a link aggregate interface, failover occurs only if all the physical network interfaces in the logical interface fail. For details, see Link aggregation. Note: To prevent an unintentional failover, do not configure port monitoring until you configure HA on all the appliances in the HA group, and have plugged in the cables to link the physical network ports that will be monitored.
Heartbeat Interface	Select which port(s) on this appliance that all the appliances will use to send heartbeat signals and synchronization data (configuration synchronization for active-passive HA, or configuration and session synchronization for active-active HA) between each other (i.e. the HA heartbeat link). Connect this port to the same port number on the other HA group members. (e.g., If you select port3 for the primary heartbeat link, connect port3 on this appliance to port3 on the other appliances.) At least one heartbeat interface must be selected on each appliance in the HA group. Ports that currently have an IP address assigned for other purposes (that is, virtual servers or bridges) cannot be re-used as a heartbeat link. If a switch is used to connect the heartbeat interfaces, the heartbeat interfaces must be reachable by Layer 2 multicast. If a port is selected as the heartbeat interface, then MTU will be automatically changed from the default 1500 to 1400 to establish HA connection in VXLAN environments. Tip: If enough ports are available, you can select both a primary heartbeat interface and a secondary heartbeat interface on each appliance in the HA pair to provide heartbeat link redundancy. (You cannot use the same port as both the primary and secondary heartbeat interface on the same appliance, as this is incompatible with the purpose of link redundancy.) Note: The master appliance uses the heartbeat interface to synchronize its session table to other appliances in an Active-Active-Standard HA group by default. However, you can use extra interfaces for the session synchronization by configuring `set session-sync-dev <port_number>` in CLI command `config system ha`. Moreover, the appliance synchronizes sessions to others in unicast by default, but you can choose to synchronize sessions via broadcasting by configuring `set session-sync-broadcast {enable\|disable}` in the CLI command `config system ha`. Broadcasting is recommended if an Active-Active-Standard HA group contains many appliances. For details, see FortiWeb CLI Reference.
Reserved Management Interface	This option applies to active-passive and standard active-active modes. Enable to reserve network interfaces for this HA member. The configurations of the reserved interfaces, including the IP address and other settings, are not synchronized with other HA members. The reserved network interface can be used for the administrative access to the GUI and CLI of this member. You can also use it to connect this member to back-end servers that are not in the server pool of the HA group. If the reserved network interfaces are not in the same subnet with the management computer or the back-end servers, you need to configure the next-hop gateways in HA Static Route or HA Policy route. The configurations in the Static Route and Policy Route (System > Network > Route) are synchronized by all the HA members, but the configurations in HA Static Route or HA Policy route are applied only to this specific member. For details on the static route and policy route, see Adding a gateway and Creating a policy route.
Interface	Specifies the network interfaces to be reserved. The interfaces that are already used in the HA group configuration are excluded from the list.
HA Health Check	Enable to check whether the server policies are running properly on the HA group. Available only if the HA mode is Active-Active-Standard.

Click Apply.

All the appliances join the HA group by matching their Group ID. They begin to send heartbeat and synchronization traffic to each other through their heartbeat links.

To determine which appliance currently has the role of the main appliance, on System > High Availability > Settings, in the HA Member table, view the HA Role column:

main/master—The appliance in this row is currently active. The active appliance applies policies to govern the traffic passing to your web servers. Also called the primary, master, or main appliance.
standby—The appliance in this row is currently passive, and is not actively applying policies. The passive appliance listens to heartbeat traffic and port monitoring for signs that the main appliance may have become unresponsive, at which point it will assume the role of the main appliance. Also called the secondary or standby appliance.
slave—The appliance in this row is the slave node in active-active modes.

If both appliances believe that they are the main:

Test the cables and/or switches in the heartbeat link to verify that the link is functional.
Verify that you have selected the heartbeat port or ports in Heartbeat Interface. Make sure that the primary and secondary link is not crossed (that is, the primary heartbeat interface is not connected to the secondary heartbeat interface on the other appliance).
Verify that the Group ID matches on both appliances.
Verify that the ports on Monitor Interface are linked and up (available).
If the heartbeat link passes through switches and/or routers, you may need to adjust the time required after a reboot to assess network availability before electing the main appliance. To do this, use the the boot-time <seconds_int> command. For details, see FortiWeb CLI Reference.

For debugging logs, use the diagnose system ha status and diagnose debug application hatalk level commands. For details, see FortiWeb CLI Reference.

To monitor the HA group for failover, you can use SNMP (see Configuring an SNMP community), log messages (see Configuring logging), and alert email (see Alert email).

If the failover time is too long, from the CLI, enter config system ha and configure these settings:

arps <arp_int>

Enter the number of times that the FortiWeb appliance will broadcast address resolution protocol (ARP) packets (IPv4 environment) or Neighbor Solicitation (NS) packets (IPv6 environment) when it takes on the main role. Even though a new NIC has not actually been connected to the network, FortiWeb does this to notify the network that a different physical port has become associated with the IP address and virtual MAC of the HA pair.

This is sometimes called “using gratuitous ARP packets to train the network,” and can occur when the main appliance is starting up, or during a failover. Also configure arp-interval <seconds_int>.

Normally, you do not need to change this setting. Exceptions include:

Increase the number of times the main appliance sends gratuitous ARP packets if your HA pair takes a long time to fail over or to train the network. Sending more gratuitous ARP packets may help the failover to happen faster.
Decrease the number of times the main appliance sends gratuitous ARP packets if your HA pair has a large number of VLAN interfaces and virtual domains. Because gratuitous ARP packets are broadcast, sending them may generate a large amount of network traffic. As long as the HA pair still fails over successfully, you could reduce the number of times gratuitous ARP packets are sent to reduce the amount of traffic produced by a failover.

The valid range is 1–16. The default value is 10.

arp-interval <seconds_int>

Enter the number of seconds to wait between each broadcast of ARP/NS packets.

Normally, you do not need to change this setting. Exceptions include:

Decrease the interval if your HA pair takes a long time to fail over or to train the network. Sending ARP packets more frequently may help the failover to happen faster.
Increase the interval if your HA pair has a large number of VLAN interfaces and virtual domains. Because gratuitous ARP packets are broadcast, sending them may generate a large amount of network traffic. As long as the HA pair still fails over successfully, you could increase the interval between when gratuitous ARP packets are sent to reduce the rate of traffic produced by a failover.

The valid range is 1–20. The default value is 3.

Even when a FortiWeb appliance broadcasts gratuitous ARP/NS packets once it takes on the master role after a failover occurs, some equipment in the network may not immediately detect that there is a new primary unit in the group. To make sure that all equipment defects the failover, you can use the following CLI command:

config system ha

set link-failed-signal enable

end

For details, see FortiWeb CLI Reference.

If your HA link passes through switches and/or routers, and inadvertent failovers occur when rebooting the HA pair, you can increase the maximum time to wait for a heartbeat signal after a reboot by configuring boot-time <limit_int>. See FortiWeb CLI Reference.

Configuring redundant interfaces in HA

You can create an HA group with redundant interfaces that eliminate potential single points of failure. Redundant interfaces consist of at least two physical interfaces. At any given time, only one of the physical interfaces has traffic going through it; the other interfaces act as backups in the event that the active interface fails.

This is an example of an HA group with redundant interfaces:

For details, see Configuring redundant interfaces.

Checking your HA topology information and statistics

After completing your HA deployment, you can manage the HA topology and view information and statistics for each HA unit.

Go to System > High Availability > HA Topology. From here, you can select the master unit or slaves in the group, and a pop-up window will appear with the option to disconnect them. If you select a slave in the group, the pop-up will also provide options to view its attack logs, event logs, and traffic logs. On the log page, you can click the Download button to download the logs of the slaves. To view logs for the master unit in the group, go to Log&Report > Log Access and select the log(s) you want to view.

From System > High Availability > HA Topology, click View HA Statistics in the top right corner of the window. The following information about each unit in the group is displayed:

For best fault tolerance, make sure that your topology is fully redundant, with no single points of failure.

For example, in the above image, the switch, firewall, and Internet connection are all single points of failure. If any should fail, websites would be unavailable despite the HA group. To prevent this, you would add a dual ISP connection to separate service providers, preferably with their own redundant pathways upstream. You would also add a standby firewall, and a standby switch. For details, see Configuring redundant interfaces.