Fortinet black logo

Administration Guide

HA heartbeat & synchronization

HA heartbeat & synchronization

You can group multiple FortiWeb appliances together as a high availability (HA) cluster (see Configuring a high availability (HA) FortiWeb cluster). The heartbeat traffic indicates to other appliances in the HA cluster that the appliance is up and “alive.” Synchronization ensures that all appliances in the cluster remain ready to process traffic, even if you only change one of the appliances.

Heartbeat and synchronization traffic between cluster appliances occurs over the physical network ports selected in Heartbeat Interface. Heartbeat traffic uses multicast on port number 6065 and the IP address 239.0.0.1. Synchronization traffic uses TCP on port number 6010 and a reserved IP address. The HA IP addresses are hard-coded and cannot be modified.

Ensure that switches and routers that connect to heartbeat interfaces are configured to allow level2 frames. See Heartbeat packet Ethertypes.

Failover is triggered by any interruption to either the heartbeat or a port monitored network interface whose length of time exceeds your configured limits (Configuring a high availability (HA) FortiWeb cluster and Configuring a high availability (HA) FortiWeb cluster). When the active (“main”) appliance becomes unresponsive, the standby appliance:

  1. Assumes the virtual MAC address of the failed primary unit and broadcasts ARP/NS packets so that other equipment in the network will refresh their MAC forwarding tables and detect the new primary unit
  2. Assumes the role of the active appliance and scans network traffic

To keep the standby appliance ready in case of a failover, HA pairs also use the heartbeat link to automatically synchronize most of their configuration. Synchronization includes:

  • Core CLI-style configuration file (fwb_system.conf)
  • X.509 certificates, certificate request files (CSR), and private keys
  • HTTP error pages
  • FortiGuard IRIS Service database
  • FortiGuard Security Service files (attack signatures, predefined data types & suspicious URLs, known web crawlers & content scrapers, global white list, vulnerability scan signatures)
  • FortiGuard Antivirus signatures
  • Geography-to-IP database

and occurs immediately when an appliance joins the cluster, and thereafter every 30 seconds.

Although they are not automatically synchronized for performance reasons due to large size and frequent updates, you can manually force HA to synchronize. For instructions, see execute ha synchronize in the FortiWeb CLI Reference (http://docs.fortinet.com/fortiweb/reference). For a list of settings and data that is not synchronized, see Data that is not synchronized by HA and Configuration settings that are not synchronized by HA.

If you do not want to configure HA (perhaps you have a separate network appliance implementing HA externally), you can still replicate the FortiWeb’s configuration on another FortiWeb appliance. For details, see Replicating the configuration without FortiWeb HA (external HA)
See also

Data that is not synchronized by HA

In addition to the HA configuration, some data is also not synchronized.

  • FortiWeb HTTP sessionsFortiWeb appliances can use cookies to add and track its own sessions, functionality that is not inherently provided by HTTP. For details, see HTTP sessions & security. This state-tracking data corresponds in a 1:1 ratio to request volume, and therefore can change very rapidly. To minimize the performance impact on an HA cluster, this data is not synchronized.

Failover will not break web applications’ existing sessions, which do not reside on the FortiWeb, and are not the same thing as FortiWeb’s own HTTP sessions. The new active appliance will allow existing web application sessions to continue. For details, see FortiWeb sessions vs. web application sessions.

FortiWeb sessions are used by some FortiWeb features. After a failover, these features may not work, or may work differently, for existing sessions. (New sessions are not affected.) See the description for each setting that uses session cookies. For details, see Sessions & FortiWeb HA.

Note: All sessions that are shorter than 30 seconds will not be synchronized. Only sessions that have been established for longer than 30 seconds will be synchronized.

  • SSL/TLS sessions—HTTPS connections are stateful in that they must be able to remember states such as the security associations from the SSL/TLS handshake: the mutually supported cipher suite, the agreed parameters, and any certificates involved. Encryption and authentication in SSL/TLS cannot function without this. However, a new primary FortiWeb’s lack of existing HTTPS session information is gracefully handled by re-initializing the SSL/TLS session with the client.This does not impact to the encapsulated HTTP application, has only an initial failover impact during re-negotiation, and therefore is not synchronized.
  • Log messages—These describe events that happened on that specific appliance. After a failover, you may notice that there is a gap in the original active appliance’s log files that corresponds to the period of its downtime. Log messages created during the time when the standby was acting as the active appliance (if you have configured local log storage) are stored there, on the original standby appliance. For details about configuring local log storage, see Configuring logging.
  • Generated reports—Like the log messages that they are based upon, PDF, HTML, RTF, and plain text reports also describe events that happened on that specific appliance. As such, report settings are synchronized, but report output is not. For details about this feature, see Reports.
  • Machine learning data—Machine learning database is synchronized from the master node to the slave node only in Active-Passive mode. The data is synchronized every 10 minutes.
    In Active-Active mode, the database is not synchronized.
See also

Configuration settings that are not synchronized by HA

All configuration settings on the active FortiWeb are synchronized to the standby FortiWeb except these settings:

Operation mode You must set the operation mode of each HA group member before configuring HA. For details, see Setting the operation mode.
Host name The host name distinguishes each member of the FortiWeb HA cluster. For details, see Changing the FortiWeb appliance’s host name.
Network interfaces

(Reverse Proxy or Offline Protection mode only)

or

Bridge

(True Transparent Proxy or Transparent Inspection mode only)

In Active-Passsive mode, only the FortiWeb appliance acting as the main appliance, actively scanning web traffic, is configured with IP addresses on its network interfaces (or bridge). The standby appliance only uses the configured IP addresses if a failover occurs, and the standby appliance therefore assumes the role of the main appliance.

In Active-Active mode, all the cluster members actively scan web traffic. The IP address configured for the master applicance is synchronized to and used by all the cluster members.

For details, see Configuring the network interfaces or Configuring a bridge (V-zone).

If you have configured reserved management ports for a cluster member, that configuration, including administrative access and other settings, is not synchronized.

RAID level RAID settings are hardware-dependent and determined at boot time by looking at the drives (for software RAID) or the controller (hardware RAID), and are not stored in the system configuration. Therefore, they are not synchronized. For details, see RAID level & disk statuses.
HA active status and priority The HA configuration, which includes Device Priority, is not synchronized because this configuration must be different on the primary and secondary appliances.
See also

How HA chooses the active appliance

An HA pair may or may not resume their active and standby roles when the failed appliance resumes responsiveness to the heartbeat.

Since the current active appliance will by definition have a greater uptime than a failed previous active appliance that has just returned online, assuming each has the same number of available ports, the current active appliance usually retains its status as the active appliance, unless Override is enabled. If Override is enabled, and if Device Priority of the returning appliance is higher, it will be elected as the active appliance in the HA cluster.

If Override is disabled, HA considers (in order):
  1. The most available ports
  2. For example, if two FortiWeb appliances, FWB1 and FWB2, were configured to monitor two ports each, and FWB2 has just one port currently available according to Monitor Interface, FWB1 would become the active appliance, regardless of uptime or priority. But if both had 2 available ports, this factor alone would not be able to determine which appliance should be active, and the HA cluster would proceed to the next consideration.

  3. The highest uptime value
  4. Uptime is reset to zero if an appliance fails, or the status of any monitored port (per Monitor Interface) changes.

  5. The smallest Device Priority number (that is, 0 has the highest priority)
  6. The highest-sorting serial number
Serial numbers are sorted by comparing each character from left to right, where 9 and z are the greatest values, and result in highest placement in the sorted list.
If Override is enabled, HA considers (in order):
  1. The most available ports
  2. The smallest Device Priority number (that is, 0 has the highest priority)
  3. The highest uptime value
  4. The highest-sorting serial number
  5. If the heartbeat link occurs through switches or routers, and the active appliance is very busy, it might require more time to establish a heartbeat link through which it can negotiate to elect the active appliance. You can configure the amount of time that a FortiWeb appliance will wait after it boots to establish this connection before assuming that the other appliance is unresponsive, and that it should become the active appliance. For details, see the boot-time <seconds_int> setting in the FortiWeb CLI Reference:

http://docs.fortinet.com/fortiweb/reference

See also

Heartbeat packet Ethertypes

Normal IP packets are 802.3 packets that have an Ethernet type (Ethertype) field value of 0x0800. Ethertype values other than 0x0800 are understood as level2 frames rather than IP packets.

By default, HA heartbeat packets use the following Ethertypes, which are hard-coded and cannot be configured:

  • Ethertype 0x8890—For HA heartbeat packets that cluster members use to find other cluster member and to verify the status of other cluster members while the cluster is operating.
  • Ethertype 0x8893—For HA sessions that synchronize the cluster configurations.

Because heartbeat packets are recognized as level2 frames, the switches and routers that connect to heartbeat interfaces require a configuration that allows them. If these network devices drop level2 frames, they prevent heartbeat traffic between the members of the cluster.

In some cases, if you connect and configure the heartbeat interfaces so that regular traffic flows but heartbeat traffic is not forwarded, you can change the configuration of the switch that connects the HA heartbeat interfaces to allow level2 frames with Ethertypes 0x8890 and 0x8893 to pass.

For HA Ethertype, only numbers between 0x8890–0x889f can be used; also, different HA Ethertype shall use different numbers.

HA heartbeat & synchronization

You can group multiple FortiWeb appliances together as a high availability (HA) cluster (see Configuring a high availability (HA) FortiWeb cluster). The heartbeat traffic indicates to other appliances in the HA cluster that the appliance is up and “alive.” Synchronization ensures that all appliances in the cluster remain ready to process traffic, even if you only change one of the appliances.

Heartbeat and synchronization traffic between cluster appliances occurs over the physical network ports selected in Heartbeat Interface. Heartbeat traffic uses multicast on port number 6065 and the IP address 239.0.0.1. Synchronization traffic uses TCP on port number 6010 and a reserved IP address. The HA IP addresses are hard-coded and cannot be modified.

Ensure that switches and routers that connect to heartbeat interfaces are configured to allow level2 frames. See Heartbeat packet Ethertypes.

Failover is triggered by any interruption to either the heartbeat or a port monitored network interface whose length of time exceeds your configured limits (Configuring a high availability (HA) FortiWeb cluster and Configuring a high availability (HA) FortiWeb cluster). When the active (“main”) appliance becomes unresponsive, the standby appliance:

  1. Assumes the virtual MAC address of the failed primary unit and broadcasts ARP/NS packets so that other equipment in the network will refresh their MAC forwarding tables and detect the new primary unit
  2. Assumes the role of the active appliance and scans network traffic

To keep the standby appliance ready in case of a failover, HA pairs also use the heartbeat link to automatically synchronize most of their configuration. Synchronization includes:

  • Core CLI-style configuration file (fwb_system.conf)
  • X.509 certificates, certificate request files (CSR), and private keys
  • HTTP error pages
  • FortiGuard IRIS Service database
  • FortiGuard Security Service files (attack signatures, predefined data types & suspicious URLs, known web crawlers & content scrapers, global white list, vulnerability scan signatures)
  • FortiGuard Antivirus signatures
  • Geography-to-IP database

and occurs immediately when an appliance joins the cluster, and thereafter every 30 seconds.

Although they are not automatically synchronized for performance reasons due to large size and frequent updates, you can manually force HA to synchronize. For instructions, see execute ha synchronize in the FortiWeb CLI Reference (http://docs.fortinet.com/fortiweb/reference). For a list of settings and data that is not synchronized, see Data that is not synchronized by HA and Configuration settings that are not synchronized by HA.

If you do not want to configure HA (perhaps you have a separate network appliance implementing HA externally), you can still replicate the FortiWeb’s configuration on another FortiWeb appliance. For details, see Replicating the configuration without FortiWeb HA (external HA)
See also

Data that is not synchronized by HA

In addition to the HA configuration, some data is also not synchronized.

  • FortiWeb HTTP sessionsFortiWeb appliances can use cookies to add and track its own sessions, functionality that is not inherently provided by HTTP. For details, see HTTP sessions & security. This state-tracking data corresponds in a 1:1 ratio to request volume, and therefore can change very rapidly. To minimize the performance impact on an HA cluster, this data is not synchronized.

Failover will not break web applications’ existing sessions, which do not reside on the FortiWeb, and are not the same thing as FortiWeb’s own HTTP sessions. The new active appliance will allow existing web application sessions to continue. For details, see FortiWeb sessions vs. web application sessions.

FortiWeb sessions are used by some FortiWeb features. After a failover, these features may not work, or may work differently, for existing sessions. (New sessions are not affected.) See the description for each setting that uses session cookies. For details, see Sessions & FortiWeb HA.

Note: All sessions that are shorter than 30 seconds will not be synchronized. Only sessions that have been established for longer than 30 seconds will be synchronized.

  • SSL/TLS sessions—HTTPS connections are stateful in that they must be able to remember states such as the security associations from the SSL/TLS handshake: the mutually supported cipher suite, the agreed parameters, and any certificates involved. Encryption and authentication in SSL/TLS cannot function without this. However, a new primary FortiWeb’s lack of existing HTTPS session information is gracefully handled by re-initializing the SSL/TLS session with the client.This does not impact to the encapsulated HTTP application, has only an initial failover impact during re-negotiation, and therefore is not synchronized.
  • Log messages—These describe events that happened on that specific appliance. After a failover, you may notice that there is a gap in the original active appliance’s log files that corresponds to the period of its downtime. Log messages created during the time when the standby was acting as the active appliance (if you have configured local log storage) are stored there, on the original standby appliance. For details about configuring local log storage, see Configuring logging.
  • Generated reports—Like the log messages that they are based upon, PDF, HTML, RTF, and plain text reports also describe events that happened on that specific appliance. As such, report settings are synchronized, but report output is not. For details about this feature, see Reports.
  • Machine learning data—Machine learning database is synchronized from the master node to the slave node only in Active-Passive mode. The data is synchronized every 10 minutes.
    In Active-Active mode, the database is not synchronized.
See also

Configuration settings that are not synchronized by HA

All configuration settings on the active FortiWeb are synchronized to the standby FortiWeb except these settings:

Operation mode You must set the operation mode of each HA group member before configuring HA. For details, see Setting the operation mode.
Host name The host name distinguishes each member of the FortiWeb HA cluster. For details, see Changing the FortiWeb appliance’s host name.
Network interfaces

(Reverse Proxy or Offline Protection mode only)

or

Bridge

(True Transparent Proxy or Transparent Inspection mode only)

In Active-Passsive mode, only the FortiWeb appliance acting as the main appliance, actively scanning web traffic, is configured with IP addresses on its network interfaces (or bridge). The standby appliance only uses the configured IP addresses if a failover occurs, and the standby appliance therefore assumes the role of the main appliance.

In Active-Active mode, all the cluster members actively scan web traffic. The IP address configured for the master applicance is synchronized to and used by all the cluster members.

For details, see Configuring the network interfaces or Configuring a bridge (V-zone).

If you have configured reserved management ports for a cluster member, that configuration, including administrative access and other settings, is not synchronized.

RAID level RAID settings are hardware-dependent and determined at boot time by looking at the drives (for software RAID) or the controller (hardware RAID), and are not stored in the system configuration. Therefore, they are not synchronized. For details, see RAID level & disk statuses.
HA active status and priority The HA configuration, which includes Device Priority, is not synchronized because this configuration must be different on the primary and secondary appliances.
See also

How HA chooses the active appliance

An HA pair may or may not resume their active and standby roles when the failed appliance resumes responsiveness to the heartbeat.

Since the current active appliance will by definition have a greater uptime than a failed previous active appliance that has just returned online, assuming each has the same number of available ports, the current active appliance usually retains its status as the active appliance, unless Override is enabled. If Override is enabled, and if Device Priority of the returning appliance is higher, it will be elected as the active appliance in the HA cluster.

If Override is disabled, HA considers (in order):
  1. The most available ports
  2. For example, if two FortiWeb appliances, FWB1 and FWB2, were configured to monitor two ports each, and FWB2 has just one port currently available according to Monitor Interface, FWB1 would become the active appliance, regardless of uptime or priority. But if both had 2 available ports, this factor alone would not be able to determine which appliance should be active, and the HA cluster would proceed to the next consideration.

  3. The highest uptime value
  4. Uptime is reset to zero if an appliance fails, or the status of any monitored port (per Monitor Interface) changes.

  5. The smallest Device Priority number (that is, 0 has the highest priority)
  6. The highest-sorting serial number
Serial numbers are sorted by comparing each character from left to right, where 9 and z are the greatest values, and result in highest placement in the sorted list.
If Override is enabled, HA considers (in order):
  1. The most available ports
  2. The smallest Device Priority number (that is, 0 has the highest priority)
  3. The highest uptime value
  4. The highest-sorting serial number
  5. If the heartbeat link occurs through switches or routers, and the active appliance is very busy, it might require more time to establish a heartbeat link through which it can negotiate to elect the active appliance. You can configure the amount of time that a FortiWeb appliance will wait after it boots to establish this connection before assuming that the other appliance is unresponsive, and that it should become the active appliance. For details, see the boot-time <seconds_int> setting in the FortiWeb CLI Reference:

http://docs.fortinet.com/fortiweb/reference

See also

Heartbeat packet Ethertypes

Normal IP packets are 802.3 packets that have an Ethernet type (Ethertype) field value of 0x0800. Ethertype values other than 0x0800 are understood as level2 frames rather than IP packets.

By default, HA heartbeat packets use the following Ethertypes, which are hard-coded and cannot be configured:

  • Ethertype 0x8890—For HA heartbeat packets that cluster members use to find other cluster member and to verify the status of other cluster members while the cluster is operating.
  • Ethertype 0x8893—For HA sessions that synchronize the cluster configurations.

Because heartbeat packets are recognized as level2 frames, the switches and routers that connect to heartbeat interfaces require a configuration that allows them. If these network devices drop level2 frames, they prevent heartbeat traffic between the members of the cluster.

In some cases, if you connect and configure the heartbeat interfaces so that regular traffic flows but heartbeat traffic is not forwarded, you can change the configuration of the switch that connects the HA heartbeat interfaces to allow level2 frames with Ethertypes 0x8890 and 0x8893 to pass.

For HA Ethertype, only numbers between 0x8890–0x889f can be used; also, different HA Ethertype shall use different numbers.