Using high availability (HA)

Storing mail data from HA groups on a NAS server

About HA port numbers and protocols

About the HA heartbeat and synchronization

Heartbeat and synchronization through the primary and secondary heartbeat network interfaces:

monitors other units in the FortiMail HA group for failure
synchronizes configuration changes from the primary unit to the secondary units

For exceptions, see Settings that are not synchronized by HA.
(active-passive only) synchronizes the mail queue, FortiMail system mail directory, and user home directories

For exceptions, see Storing mail data from HA groups on a NAS server.

Synchronization intervals vary.

FortiGuard Antispam and FortiGuard Antivirus packages: Not synchronized.
Mail queue: Up to 20 minutes (not real time).
Configuration: Real time.

If configuration synchronization did not occur when expected, or if you have inadvertently de-synchronized the secondary unit’s configuration (for example, if a cable was accidentally disconnected), then you can manually initiate synchronization via GUI or the CLI command diagnose system ha sync on either the primary unit or the secondary unit.

Periodically, the secondary unit verifies that all configuration changes have been synchronized. If they have not, then the secondary unit will pull the configuration changes from the primary unit and reload the new configuration.

Secondary units also can push any changes made to its block and safe lists back to the primary unit. In active-active HA, these changes are then synchronized to all other secondary units.

The secondary unit expects to constantly receive heartbeat traffic from the primary unit. Loss of the heartbeat signal detects failure of the primary unit, and triggers the action that you select in On failure. For details, see Example: Failover scenarios.

Exceptions include system restarts and the execute reload CLI command. If the primary unit reboots or reloads its configuration, then it signals to the secondary unit to wait for the primary unit to complete the restart or reload. For details, see Failover scenario 2: System reboot or reload of the primary unit.

Behavior when the heartbeat signal is lost varies by HA mode and On failure:

Active-passive: The secondary unit becomes the new primary unit and starts receiving email connections. Some in-progress email connections may be interrupted and must be restarted, but most email clients and servers can gracefully handle this.
Active-active: If Primary backup has been selected, then your preferred backup unit will take over the role of the primary unit (Effective role becomes Primary).

If a specific Primary backup is not selected, then each secondary unit continues to operate as a secondary unit. However, with no primary unit, changes to the configuration are not synchronized anymore.

For failover examples and steps required to restore the initially configured roles in each case, see Example: Failover scenarios.

Interface monitoring, hard drive monitoring, and remote service monitoring do not provide configuration and data synchronization, and therefore they are not a complete replacement for the heartbeat. However you can use them as another way to detect failure. See Interface section and Service Monitor section.

See also

About HA modes

Settings that are not synchronized by HA

Storing mail data from HA groups on a NAS server

Synchronization of MTA queue directories after a failover

Storing mail data from HA groups on a NAS server

About HA port numbers and protocols

The default protocol and port numbers for HA heartbeat, synchronization, and service monitoring communications are configurable. See HA base port, the control-packet-option setting in the FortiMail CLI Reference, and Appendix C: Port Numbers.

If a firewall is between the primary and secondary FortiMail unit, then verify that the firewall policy allows HA port numbers. Blocked HA ports can cause incorrect failover and synchronization failure.

Settings that are not synchronized by HA

All settings on the primary unit are synchronized to the secondary unit, except the following:

Settings

Explanation

Operation mode

You must set the operation mode (gateway, transparent, or server) of each HA group member before configuring HA. Many settings vary by operation mode, and therefore configurations cannot be synchronized if the operation mode is different.

Host name

Different host names are used to distinguish members of the HA cluster when connecting to the GUI and to indicate which unit failed. For details, see Hostname.

Static route

Static routes are not synchronized because some or all in the network interfaces on each FortiMail unit in the HA cluster may be connected to different subnets. See also Configuring static routes .

Interface configuration

(gateway and server mode only)

Administrator connections to the GUI/CLI, alert email, and many other features require that you configure at least one network interface with an IP address. For details, see Configuring the network interfaces.

Exceptions include virtual IP addresses on active-passive HA. Virtual IP addresses are synchronized because, upon failover, the secondary unit must starts to use them. This mechanism allows traffic to receive connections instead of the failed primary unit. See Virtual IP address (or Virtual IPv6 address).

Management IP address

(transparent mode only)

Each FortiMail unit in the HA cluster should be configured with different management IP addresses for GUI and CLI connectivity purposes. For details, see About the management IP.

SNMP system information

Each FortiMail unit in the HA cluster will have its own SNMP system information, including the Description, Location, and Contact. For details, see Configuring SNMP queries and traps.

RAID configuration

RAID settings are hardware-dependent and determined at boot time by looking at the drives (for software RAID) or the controller (hardware RAID), and are not stored in the system configuration. Therefore, they are not synchronized.

Some HA settings

Shared password
Role

Product name and icon

The product name and icon under System > Customization > Appearance are not synchronized. All other appearance settings are synchronized.

Miscellaneous settings
(active-active HA only)

In active-active HA, the following settings are not synchronized:

local domain name (see Local domain name)
default certificate (see Managing local certificates)
iSCSI initiator name
iSCSI ID for remote storage (see NAS server)
SNMP settings (see Configuring SNMP queries and traps)
IP pools (see Configuring IP pools)
quarantine report host name (see Web release host name/IP)
IBE settings of base URL, Help content URL, and About content URL (see Configuring IBE encryption)
centralized IBE client IP address (see Centralized IBE)
centralized quarantine client IP address (see Centralized Quarantine)

All system, domain, and user level block/safe lists are synchronized.

User data is synchronized at predefined time intervals, not in real time.

Synchronization of MTA queue directories after a failover

During normal operation in active-passive HA, email messages are either:

being received or sent by the primary FortiMail unit
waiting to be delivered in the mail queue
stored in the primary unit’s mail data directories (email quarantines, email archives, and email inboxes of server mode)

When a failure occurs, sending and receiving is interrupted. The delivery attempt fails, and the sender usually retries to send the email message. However, stored messages remain in the primary unit’s mail data directories.

To prevent data loss when a primary unit fails, you usually should enable Synchronize mail data directory (unless NAS storage is used), but do not need to enable Synchronize MTA queue directory. This is because of an automatic recovery mechanism in FortiMail HA failover.

The secondary or primary backup unit detects that the primary unit has failed, and becomes the new primary unit.

If the former primary unit can reboot, it detects the new primary unit, and becomes a secondary unit.

Depending on the On failure setting, you may be required to click Restart HA on a failed primary unit.

The former primary unit pushes its mail queue to the new primary unit.

This synchronization occurs through the heartbeat link between the primary and secondary units, and prevents duplicate email messages from forming in the primary unit’s mail queue.
The new primary unit delivers email in its mail queues, including email messages synchronized from the new secondary unit.

As a result, if the failed primary unit can restart, no email is lost from the mail queue.

Even if you choose to synchronize the mail queue, because its contents change very rapidly and synchronization is periodic, there is a chance that some email will not have not been synchronized when a failover occurs.

If you have FortiMail units operating in server mode and in an active-active HA group, you must store mail data centrally on a network attached storage (NAS) server — not on each FortiMail unit. Otherwise email users’ messages and other mail data could be scattered across multiple FortiMail units.

For other HA and operating modes, however, it still may be better to store mail data on a NAS server.

For example, regular NAS server backups help to prevent mail data loss, even if a FortiMail unit has hardware failure. Also, during a temporary failure of a FortiMail unit, you can still access the mail data on the NAS server. When the FortiMail unit restarts, it can usually continue to access and use the mail data stored on the NAS server.

For active-active HA with a NAS server, only the primary unit sends quarantine reports to email users. The primary unit also acts as a proxy between email users and the NAS server when email users use FortiMail webmail to access quarantined email and to configure their own Bayesian filters.

For active-passive HA groups, the primary unit reads and writes all mail data to and from the NAS server in the same way as a standalone unit. If a failover occurs, the new primary unit uses the same NAS server for mail data. The new primary unit can access all mail data that the original primary unit stored on the NAS server. So if you are using a NAS server to store mail data, after a failover, the new primary unit continues operating with no loss of mail data.

If the FortiMail unit is a member of an active-passive HA group, and the HA group stores mail data on a remote NAS server, disable mail data synchronization to prevent duplicate mail data traffic.

For instructions on storing mail data on a NAS server, see Selecting the mail data storage location.

Synchronization of MTA queue directories after a failover

About logging, alert email, and SNMP for HA

For faster discovery and diagnosis of network problems that have caused an HA failover, you can configure SNMP, Syslog, and/or alert email to monitor the HA cluster.

To configure logging and alert email, configure the primary unit and enable HA events. When the configuration changes are synchronized to the secondary units, all FortiMail units in the HA group record their own separate log messages and send separate alert email messages. Log data is not synchronized.

To distinguish alert email from each member of the HA cluster, configure a different host name for each member. For details, see Hostname.

To use SNMP to monitor HA failover, configure each cluster member to enable HA events for the SNMP community, such as:

See also

Configuring SNMP queries and traps

Logs, reports, and alerts

About the HA heartbeat and synchronization

Configuring an HA group

To deploy FortiMail units as a high availability (HA) cluster, perform the following steps in order.

To deploy an HA group

Register all FortiMail units in the HA cluster with the Fortinet Technical Support web site:

https://support.fortinet.com/

If you use licensed features such as centralized HA monitoring, FortiGuard Antivirus, and/or FortiGuard Antispam, also purchase and register licenses for all units.

Connect the network interfaces that will be used for the heartbeat and synchronization between FortiMail units in the HA cluster. At least one heartbeat link is required.

For example, you could use a network cable to connect FortiMail A's port2 to FortiMail B's port2.

Don't disconnect the heartbeat once HA is enabled. If the heartbeat is accidentally interrupted for an active-passive HA group, such as when a network cable is temporarily disconnected, the secondary unit will assume that the primary unit has failed, and become the new primary unit. If no failure has actually occurred, both FortiMail units will be operating as primary units at the same time. This can cause an IP address conflict. In active-active HA groups, configuration synchronization can be disrupted. For details on correcting this, see Restore to configured role.

For better heartbeat reliability, create two heartbeat links: a primary and a secondary. Directly link the pair of heartbeat ports with an Ethernet crossover cable, or connect them through a dedicated local switch that is not connected to your overall network. This ensures enough bandwidth and low latency for the synchronization and heartbeat. If the heartbeat is interrupted, then a failover may occur. See also About the HA heartbeat and synchronization.

If you are making an active-passive HA group, and the operation mode is gateway or server, add a Virtual IP address (or Virtual IPv6 address) and Virtual hostname to the network interface that will receive email connections. Update DNS records to use this virtual IP address, not the physical IP address. Wait for the DNS records to propagate to non-authoritative DNS servers before you enable HA.

If you are making an active-active HA group, configure storage of mail data on a NAS server. See Storing mail data from HA groups on a NAS server.(Active-passive members can also benefit from a NAS server, but do not require it.)

For active-active HA, if the FortiMail unit is operating in server mode, you must store mail data externally on a NAS server. Failure to store mail data externally could result in mailboxes and other data scattered over multiple FortiMail units.

On each member of the HA group, go to System > High Availability > Configuration and:

Configure the following:

GUI item	Description
State	Enable or disable HA.
HA mode	Select either Active-Active or Active-Passive. For details, see About HA modes.
On failure	Select what the HA group will do when it detects a failure, either: Switch off immediately: On recovery, do not process email or join the HA group until you manually select the Effective role (see Restart HA and Restore to configured role). Wait for recovery: On recovery, the failed primary unit’s Effective role becomes Secondary. To manually restore the FortiMail unit to acting in its configured Role, see Restore to configured role. Wait for recovery and switch to configured role: On recovery, the failed primary unit's Effective role automatically becomes Primary again, and the secondary unit that was temporarily acting as primary automatically becomes Secondary again. This option may be useful if the cause of failure is temporary and rare, but may cause problems if the cause of failure is recurring, resulting in many extra role changes. Tip: In most cases, you should select Wait for recovery.
Shared password	Enter an HA password for the HA group members. Before HA group members synchronize with each other, they verify that they have the same shared password. This prevents them from accidentally synchronizing with FortiMail units that do not belong to the same cluster. Therefore you must add the shared HA password to each unit in the HA group.

Expand the Member section. For each FortiMail unit in the HA group, click New and configure the following:

GUI item	Description
Role	Select the role of the FortiMail unit in the HA group, either Primary or Secondary Each HA group member's role is not synchronized because this distinguishes the primary and secondary units. Effects of the role vary by HA mode. See About HA modes.
IPv4 address (or IPv6 address)	Enter the IP address of the network interface that will listen for the heartbeat and synchronization on the primary or secondary (depending on which entry you are currently configuring in the table). If you want more heartbeat interfaces, click + and then add those IP addresses. Alternatively, if you are currently configuring the device that you are adding to the table, click Use Current Device. Note: You must also bring up and then enable Heartbeat status on the interface. If it is disabled, but the IP address is configured here, then HA will detect that the heartbeat link has failed.
Hostname	Displays the hostname of the primary or secondary (depending on which entry you are currently configuring in the table). Note: Do not configure the hostname here. It will not update the hostname used by the FortiMail unit's SMTP relay/proxy. Instead, configure Host name in the mail settings and Virtual hostname, and then click Use Current Device to automatically paste the hostname into this field.
Primary backup (Active-active secondary units only)	If HA mode is Active-Active, then there can be many secondary units. Enable this setting if Role is Secondary, and you want to select this member to become the new primary when a failure is detected. Note: Usually you should have a primary backup. Otherwise configuration synchronization will be interrupted upon failure. See About the HA heartbeat and synchronization.
Comment	Optional. Enter a descriptive comment.

If the HA group is active-passive, configure the Virtual IP address (or Virtual IPv6 address) that will transfer upon failover.
If the HA group stores mail data on NAS, disable Synchronize mail data directory.
Optionally, configure:
Click Apply on the primary unit, and then on the secondary units.

If the HA group is active-active, configure the load balancer with either remote service monitoring or interface monitoring to detect failed FortiMail units, and to redirect connections to available FortiMail units.
Monitor the status of each cluster member. For details, see Monitoring HA status, Logs, reports, and alerts, and Centrally monitoring the HA cluster.

See also

About HA modes

About the HA heartbeat and synchronization

Settings that are not synchronized by HA

Advanced Option section

Go to System > High Availability > Configuration.
Expand the Advanced Option section.

Configure the following and then click Apply:

GUI item

Description

Synchronize mail data directory

(Active-Passive only)

Enable if the HA group does not store its mail data on a NAS server, in order to synchronize system quarantine, per-recipient quarantines, email archives, email users’ preferences, and (server mode only) mailboxes with the HA group members.See Storing mail data from HA groups on a NAS server.

If mail data changes frequently, you can manually initiate a data synchronization when significant changes are complete. For details, see Start configuration sync.

Synchronize MTA queue directory

(Active-Passive only)

Enable if you want to synchronize the mail queue with the HA group members.

Caution: If the primary unit experiences a hardware failure and you cannot restart it, and if this option is disabled, MTA queue directory data could be lost.

Note: Enabling this option can affect the FortiMail unit’s performance, because periodic synchronization of the mail queue can be processor and bandwidth-intensive. Additionally, because the content of the MTA queue directories is very dynamic, periodically synchronizing MTA queue directories between FortiMail units may not guarantee against loss of all email in those directories. Even if MTA queue directory synchronization is disabled, after a failover, a separate synchronization mechanism may successfully prevent loss of MTA queue data. For details, see Synchronization of MTA queue directories after a failover and Managing the mail queue.

Enabling this option can affect the FortiMail unit’s performance, because periodic synchronization of the mail queue can be processor and bandwidth-intensive. Additionally, because the content of the MTA queue directories is very dynamic, periodically synchronizing MTA queue directories between FortiMail units may not guarantee against loss of all email in those directories. Even if MTA queue directory synchronization is disabled, after a failover, a separate synchronization mechanism may successfully prevent loss of MTA queue data. For details, see Synchronization of MTA queue directories after a failover and Managing the mail queue.

HA base port

Enter the first of multiple port numbers (see Appendix C: Port Numbers) that will be used for:

heartbeat signals
synchronization control
data synchronization
configuration synchronization

For both active-active and active-passive HA, in addition or alternatively to configuring the heartbeat, you can configure service monitoring. For details, see Service Monitor section and About the HA heartbeat and synchronization.

In addition to automatic immediate and periodic configuration synchronization, you can also manually initiate synchronization. For details, see Start configuration sync.

Heartbeat lost threshold

Enter the total amount of time, in seconds, that a FortiMail unit can be unresponsive until and HA detects a failure and performs the action in On failure.

Tip: The heartbeat verifies availability every1 second. To prevent unnecessary failover when the primary unit is temporarily experiencing very heavy load and therefore heartbeat responses are slow, configure a longer threshold (for example, 3 seconds or more) to allow the secondary unit enough time to send more heartbeat signals to confirm unresponsiveness. To determine the best heartbeat threshold, it is useful to know your FortiMail unit's performance baseline and peaks. See also Establish a system baseline and Troubleshoot resource issues.

If you have service level agreements (SLA), then you may be required to keep this time short. If the failure detection time is too long, email delivery could be delayed or fail until HA detects the failure. This reduces service uptime.

Remote services as heartbeat

Enable to avoid the the On failure action if both the primary and secondary heartbeat links temporarily fail, but remote service monitoring detects that the FortiMail unit is still available.

The On failure action can still occur if the HA process restarts due to system reboot or HA daemon restart. Then it examines the physical heartbeat links first. If they are not found, then failure is detected.

This setting provides an extra HA heartbeat only, not synchronization. To avoid synchronization problems, do not use remote service monitoring as a heartbeat for a long time. This feature is intended only as a temporary heartbeat until you reestablish a normal primary or secondary heartbeat link.

Interface section

In a basic HA deployment, the heartbeat interface provides a basic signal to other HA group members about the health of the primary FortiMail unit. However, you can use an additional signals. Interface monitoring periodically tests the local network interfaces on the primary unit . If a malfunctioning interface is detected, HA performs the action configured in On failure.

Optionally, configure the interface monitoring interval and failure detection threshold. See Service Monitor section.
Go to System > High Availability > Configuration.
Expand the Interface section.
Select a row for a network interface in the table, and then click Edit.

Configure the following settings:

GUI item

Description

Heartbeat status

Enable if this interface will listen for HA heartbeat and synchronization communications.

You must enable at least one of the heartbeat interfaces that you defined in IPv4 address (or IPv6 address). Otherwise HA will detect a failure.

Port

Displays the name of the network interface that you are configuring.

Optionally, you can click the name to view or configure its settings. See also Configuring the network interfaces.

Virtual IP address (or Virtual IPv6 address)

Enter a virtual IP address that the primary unit will have on this network interface. Upon failure detection, the secondary will become the new primary and start to use the virtual IP address.

For gateway mode and server mode deployments, DNS records should be configured to point to the virtual IP address, not physical IP addresses.See also About HA modes, Configuring the network interfaces, About IPv6 Support.

This setting is available only if HA mode is Active-Passive.

The interface IP address must be different from, but on the same subnet as, the IP addresses of the other heartbeat network interfaces of other members in the HA group.

When configuring other FortiMail units in the HA group, use this value as the:

Remote peer IP (for active-passive HA)
Primary configuration (for secondary units in active-active HA)
Peer systems (for the primary unit in active-active HA)

Virtual hostname

Enter a virtual hostname.

Similar to behavior with the virtual IP address, the virtual hostname belongs to the current primary unit. Upon failover, the secondary unit becomes the new primary unit, and so it starts to use the virtual hostname instead.

This setting is available only if HA mode is Active-Passive.

Enable port monitor

Enable to monitor a physical network port for failure. If the port fails, a failure is detected by the HA cluster.

Service Monitor section

Failed FortiMail units, in the simplest HA deployments, are detected by an interrupted heartbeat. However HA can also detect failure of hardware and network services. Heartbeats detect the general responsiveness of a primary unit, but do not test each daemon (for example, POP3 or webmail service), hard drive, and physical network ports used by non-heartbeat traffic. Therefore you can add hardware and service monitoring to be more specific. Alternatively, if the heartbeat link is briefly disconnected, remote services monitoring can prevent an unnecessary failover by temporarily acting as a secondary heartbeat.

With remote service monitoring, the secondary unit connects to the SMTP, POP3, and/or web service (HTTP) on the primary unit to detect failure. For server mode, IMAP service can also be monitored.

With local network interface monitoring and hard drive monitoring, the primary unit monitors its own network interfaces and hard drives.Hard drive monitoring tests that the local hard drive is still accessible, and disk space exists for mail data. If the hard disk is not responsive, or if the mail data disk is 95% full, then a failure is detected.

Network interface monitoring tests all network interfaces where:

Status is enabled (the network interface is up)
Enable port monitor is enabled

Alert email, log messages, and SNMP traps (if configured) indicate the specific cause.

For example, if service monitoring detects failure of port2 on the primary unit, it records this log message:

date=2005-11-18 time=18:20:31 device_id=FE-4002905500194 log_id=0107000000 type=event subtype=ha pri=notice user=ha ui=ha action=unknown status=success msg="monitord: local problem detected (port2), shutting down"

and sends this alert email:

Subject: monitord: local problem detected (port2), shutting down [primary-host-name]

This is the FortiMail HA unit at 10.0.0.1.

A local problem (port2) has been detected, telling remote to take over and shutting down.

To configure hardware and service monitoring

Go to System > High Availability > Configuration.
Expand the Service Monitor section.

Select a row in the table and click Edit.

For Remote SMTP, Remote IMAP, Remote POP, and Remote HTTP services, configure the following and click OK:

GUI item	Description
Enable	Enable or disable monitoring for the selected service.
Name	Displays the service name.
Port	Enter the listening port number of the service on the primary FortiMail and (active-active HA only) secondary. See also Appendix C: Port Numbers.
Timeout	Enter the amount of time in seconds to wait for a response to the connection.
Interval	Enter the time in seconds between each test.
Retries	Enter the number of consecutively failed tests that indicate a failure.

For interface monitoring, configure the following and click OK (to configure which ports are monitored, see Interface section):

GUI item	Description
Interval	Enter the time in seconds between each test.
Retries	Enter the number of consecutively failed tests that indicate a failure.

For local hard drive monitoring, configure the following and click OK:

GUI item	Description
Enable	Enable or disable monitoring that the local hard drive.
Interval	Enter the time in seconds between each test.
Retries	Enter the number of consecutively failed tests that indicate a failure.

Monitoring HA status

After you configure HA (see Configuring an HA group), to view the roles and synchronization status of the HA group, go System > High Availability > Status. You can also manually initiate synchronization and reset the current Effective role to match the initial Configured role.

GUI item	Description
State	Displays the configured HA mode.
Configured role	Displays the configured Role. In active-active HA, the secondary unit that is the primary backup (if configured) will display Secondary, like other secondary units. After a failure has been detected, the FortiMail unit may not be acting in the role that it was initially configured for, and then this will not match Effective role. For details, see Combinations of configured and effective HA role.
Effective role	Displays the role that this FortiMail unit is currently operating in, either: Primary: Acting as primary unit. Secondary: Acting as secondary unit. Off: For primary units, this indicates that interface or remote service monitoring has detected a failure and therefore the primary unit went offline and halted HA processes. For secondary units, this indicates that it detected an HA synchronization failure; if sync immediately fails again, then the action in On failure will occur. See also Restart HA. Failed: Service monitoring or network interface monitoring has detected a failure and the diagnostic connection is currently determining if the problem has been corrected or it must perform the action in On failure. Holdoff: For secondary units, this indicates that the primary unit is rebooting and asked to wait longer than the usual Heartbeat lost threshold so that the reboot can complete. If the primary does not return, then a failure is detected and it must perform the action in On failure. After a failure has been detected, the FortiMail unit may not be acting in the role that it was initially configured for, and then this will not match Configured role. For details, see Combinations of configured and effective HA role. For information on restoring the FortiMail unit to the initially configured role, in Action, click Restore to configured role.
Member Status	A table with some basic statuses about all FortiMail units that belong to the HA group, including: SN: Serial number. IP: IPv4 address (or IPv6 address) of the network interface for the primary heartbeat. Version: Firmware version. A FortiMail unit must run the same firmware version in order to join the HA group, so that the configuration can be synchronized. Configured: Configured role. In addition, if a secondary unit has been configured as the Primary Backup, it is denoted with an icon. Effective: Effective role. Status: Whether or not the HA cluster is synchronized. Up Time: Duration of time that the HA cluster member has been operational. Last Seen: When this FortiMail unit’s HA daemon last communicated with the others in the HA group to make sure that they are available. See also Heartbeat lost thresholdand HA base port.
Action	Depending on the context, one or more the following actions may be available: Start configuration sync: Click to manually initiate configuration synchronization with other FortiMail units in the HA cluster. See also Settings that are not synchronized by HA. Restore to configured role: Click to manually reset the Effective role to match the unit's Configured role. Restart HA: If the primary unit's Effective role is Off, and then you have fixed the cause of the failure, click to restart HA processes.

Configuring an HA group

Service Monitor section

Recovering from a heartbeat link failure

Combinations of configured and effective HA role

Role	Effective role	Result
Primary	Primary	Normal for the primary unit of an HA group.
Secondary	Secondary	Normal for the secondary unit of an HA group. In active-active HA, this can also occur if the primary unit has failed. Most of the secondary units continue to be secondary. If you selected one of them to be the primary backup, however, then its Effective role becomes Primary.
Primary	Off	Either the: primary unit failed, and On failure is Switch off immediately FortiMail unit is starting to operate in HA mode and its HA processes such as configuration synchronization are stopped. To return it to the originally configured role, see Recovering from a heartbeat link failure. Note: This is caused by a stopped heartbeat, not remote service monitoring or hardware/interface monitoring.
Secondary	Off	The secondary unit has detected a failure, or the FortiMail unit is starting to operate in HA mode. After the secondary unit starts and connects with the primary unit to form an HA group, the first configuration synchronization may fail. To prevent both the secondary and primary units from simultaneously acting as primary units, the Effective role becomes Off. If the next synchronization fails, then the secondary unit’s Effective role becomes Primary.
Primary	Failed	Remote service monitoring or local network interface monitoring on the primary unit has detected a failure. Once the problem that caused the failure has been corrected, the Effective role changes from Failed to either Secondary or Primary, depending on the On failure setting.
Primary	Secondary	The primary unit failed. A secondary unit automatically became the new primary unit. When the failed unit restarted, it detected that there was already a primary unit in the HA group, and so now the failed unit is the new secondary unit. If you want the failed unit to return to acting as the primary unit, in Action, you must manually select Restore to configured role.
Secondary	Primary	The secondary unit detected that the primary unit failed, and then the secondary unit became the new primary unit. If you want it to return to acting as the secondary unit, in Action, you must manually select Restore to configured role.

Monitoring HA status

Configuring an HA group

Service Monitor section