Fortinet black logo

Administration Guide

What happens during a failover

Copy Link
Copy Doc ID 457be585-ead3-11e9-8977-00505692583a:358779
Download PDF

What happens during a failover

The Master node and Primary Slave nodes send heartbeats to each other to detect if its peers are alive. If the Master node is not accessible, such as during a reboot, a failover will occur. You can also configure a Ping server to frequently check the unit's network condition and downgrade itself to Primary Slave type when the condition is appropriate to trigger a failover. As highlighted by the boxes in the lower image below, the Primary Slave and Master switch roles after the failover, and the cluster IP changes accordingly:

The failover logic handles two different scenarios:

Objective node available

The Objective node is a slave (either Primary or Regular) that can decide the new Master. For example, if a cluster consists of one Master node, one Primary Slave node, and one Regular Slave node, the Regular Slave node is the objective node.

After a Primary Slave node takes over the Master role, the original Master node will accept the decision when it is back online.

After the original Master is back online, it will become a Primary Slave node.

No Objective node available

When there is no Objective Node in the cluster, the cluster topography is not stable and the failover process may take several rounds of role changes. This occurs when there is no communication between nodes because the cluster's internal communication is down . During the failover process, the final roles of master and primary slave are decided by three principal factors: the internal connections, the health check and the serial number.

Internal Connections

The internal connections in a cluster involve two ports: Port1 and the cluster internal port, which is typically Port2 depending on your configuration.

• Port1 is used when a node prompts itself to be the master and needs confirmation from other nodes.

• The cluster internal port is used for cluster nodes to detect whether its connection to other nodes in the cluster is available or not, and is used to ask the primary slave to failover when its health check fails.

Health Check

• The health check is used to check the connection with the ping server. If this connection fails in the Master Slave, it triggers a failover.

Serial Number

• Once the port1 connection is recovered, the unit with the newer Serial Number will keep the Master role and the unit with the older Serial Number will become the Primary Slave.

When the new Master is decided, it will:

  1. Build up the scan environment.
  2. Apply all the settings synchronized from the original Master except the port3 IP and the internal communication port IP of the original Master.

After a failover occurs, the original Master might become a Primary Slave node.

It keeps its original Port3 IP and internal cluster communication IP. All other interface ports will be shutdown as it becomes a slave node. Some functionalities will be turned off such as Email Alerts. If the user wants to re-configure its settings, such as the interface IP, the user must do that through the CLI command or the Master's Central Management page.

tooltip icon

Do not change the new master configuration before the old master has returned online, because there is a risk the configuration could be lost. If It is absolutely necessary to reconfigure the new master, it is recommended to first remove the old master from the cluster using the CLI command hc-master -r.

As the new Master takes over the port that client devices communicate with will switch to it. As the new Master needs time to start up all the services, clients may experience a temporary service interruption.

What happens during a failover

The Master node and Primary Slave nodes send heartbeats to each other to detect if its peers are alive. If the Master node is not accessible, such as during a reboot, a failover will occur. You can also configure a Ping server to frequently check the unit's network condition and downgrade itself to Primary Slave type when the condition is appropriate to trigger a failover. As highlighted by the boxes in the lower image below, the Primary Slave and Master switch roles after the failover, and the cluster IP changes accordingly:

The failover logic handles two different scenarios:

Objective node available

The Objective node is a slave (either Primary or Regular) that can decide the new Master. For example, if a cluster consists of one Master node, one Primary Slave node, and one Regular Slave node, the Regular Slave node is the objective node.

After a Primary Slave node takes over the Master role, the original Master node will accept the decision when it is back online.

After the original Master is back online, it will become a Primary Slave node.

No Objective node available

When there is no Objective Node in the cluster, the cluster topography is not stable and the failover process may take several rounds of role changes. This occurs when there is no communication between nodes because the cluster's internal communication is down . During the failover process, the final roles of master and primary slave are decided by three principal factors: the internal connections, the health check and the serial number.

Internal Connections

The internal connections in a cluster involve two ports: Port1 and the cluster internal port, which is typically Port2 depending on your configuration.

• Port1 is used when a node prompts itself to be the master and needs confirmation from other nodes.

• The cluster internal port is used for cluster nodes to detect whether its connection to other nodes in the cluster is available or not, and is used to ask the primary slave to failover when its health check fails.

Health Check

• The health check is used to check the connection with the ping server. If this connection fails in the Master Slave, it triggers a failover.

Serial Number

• Once the port1 connection is recovered, the unit with the newer Serial Number will keep the Master role and the unit with the older Serial Number will become the Primary Slave.

When the new Master is decided, it will:

  1. Build up the scan environment.
  2. Apply all the settings synchronized from the original Master except the port3 IP and the internal communication port IP of the original Master.

After a failover occurs, the original Master might become a Primary Slave node.

It keeps its original Port3 IP and internal cluster communication IP. All other interface ports will be shutdown as it becomes a slave node. Some functionalities will be turned off such as Email Alerts. If the user wants to re-configure its settings, such as the interface IP, the user must do that through the CLI command or the Master's Central Management page.

tooltip icon

Do not change the new master configuration before the old master has returned online, because there is a risk the configuration could be lost. If It is absolutely necessary to reconfigure the new master, it is recommended to first remove the old master from the cluster using the CLI command hc-master -r.

As the new Master takes over the port that client devices communicate with will switch to it. As the new Master needs time to start up all the services, clients may experience a temporary service interruption.