FPC failure

If one or more FPCs in the primary FortiGate-6000 fails, the cluster renegotiates and the FortiGate-6000 with the most operating FPCs becomes the primary FortiGate-6000. An FPC failure can occur if an FPC shuts down due to a software crash or hardware problem, or if the FPC is manually shut down.

FPCs also shut down if two of the three FortiGate-6000 power supply units (PSUs) become disconnected from their power source. The FortiGate-6000 includes three hot-swappable PSUs in a 2+1 redundant configuration. At least two of the PSUs must be operating to provide power to the FortiGate-6000. If only one PSU is operating, only four of the FPCs will continue running (usually the FPCs in slots 1 to 4). For more information about FPC failure with power loss, see AC power supply units (PSUs).

From the management board GUI dashboard, the Sensor Information dashboard widget displays information about the status of the power supplies. If all power supplies are operating, the widget displays their Status as Normal.

From the management board CLI, you can use the execute sensor list command to verify if the power supplies are operating. The command displays the current status of all FortiGate-6000 sensors including the power supply sensors. Power supply sensor entries should be similar to the following (shown for a FortiGate-6301E). The power supply sensor lines start with PS{1|2|3}:

65 PS1 VIN           alarm=0  value=122  threshold_status=0
66 PS1 VOUT_12V      alarm=0  value=12.032  threshold_status=0
67 PS1 Temp 1        alarm=0  value=24  threshold_status=0
68 PS1 Temp 2        alarm=0  value=36  threshold_status=0
69 PS1 Fan 1         alarm=0  value=8832  threshold_status=0
70 PS1 Status        alarm=0
71 PS2 VIN           alarm=0  value=122  threshold_status=0
72 PS2 VOUT_12V      alarm=0  value=12.032  threshold_status=0
73 PS2 Temp 1        alarm=0  value=24  threshold_status=0
74 PS2 Temp 2        alarm=0  value=37  threshold_status=0
75 PS2 Fan 1         alarm=0  value=9088  threshold_status=0
76 PS2 Status        alarm=0
77 PS3 VIN           alarm=0  value=122  threshold_status=0
78 PS3 VOUT_12V      alarm=0  value=12.032  threshold_status=0
79 PS3 Temp 1        alarm=0  value=23  threshold_status=0
80 PS3 Temp 2        alarm=0  value=37  threshold_status=0
81 PS3 Fan 1         alarm=0  value=9088  threshold_status=0
82 PS3 Status        alarm=0

Any non zero alarm or threshold_status values indicate a possible problem with that power supply.

After the primary FortiGate-6000 in an HA cluster experiences an FPC failure, the cluster negotiates and the FortiGate-6000 with the most operating FPCs becomes the new primary FortiGate-6000. The new primary FortiGate-6000 sends gratuitous arp packets out all of its connected interfaces to inform attached switches to send traffic to it. Sessions then resume with the new primary FortiGate-6000.

If the secondary FortiGate-6000 experiences an FPC failure, its status in the cluster does not change. In future cluster negotiations the FortiGate-6000 with an FPC failure is less likely to become the primary FortiGate-6000.

To prevent multiple failovers, if an FPC failure occurs in an HA cluster with override enabled, you should disable override until you can fix the problems and get all the FPCs up and running and synchronized.

After an FPC failure, sessions and configuration changes are not synchronized to the failed FPCs.

If failed FPCs recover in the secondary FortiGate-6000, it will continue to operate as the secondary FortiGate-6000 and will attempt to re-synchronize the FPCs with the management board. This process may take a few minutes, but if it is successful, the secondary FortiGate-6000 can return to fully participate in the cluster.

If there have been many configuration changes, the FPCs need to be manually synchronized with the management board. Log into the CLI of each out of synch FPC and enter the execute factoryreset command to reset the configuration. After the FPC restarts, the management board will attempt to synchronize its configuration. If the configuration synchronization is successful, the FPC can start processing traffic again.

If there has been a firmware upgrade, and the firmware running on the failed FPC is out of date, you can upgrade the firmware of the FPC as described in the section: Installing firmware on an individual FPC.

You can optionally use the following command to make sure the sessions on the FPCs in the secondary FortiGate-6000 are synchronized with the sessions on the FPCs in the primary FortiGate-6000.

diagnose test application chlbd 10

Once all of the FPCs are operating and synchronized, the secondary FortiGate-6000 can fully participate with the cluster.

For more information about troubleshooting FPC failures, see Troubleshooting an FPC failure.