Troubleshooting an FPC failure

This section describes some steps you can use to troubleshoot an FPC failure or to help provide information about the failure to Fortinet Support.

Displaying FPC link and heartbeat status

Start by running the diagnose load-balance status command from the management board CLI to check the status of the FPCs. The following output shows the FPC in slot 1 operating normally and a problem with the FPC in slot 2:

diagnose load-balance status
==========================================================================
MBD SN: F6KF31T018900143
  Master FPC Blade: slot-1

     Slot  1: FPC6KFT018901327
       Status:Working   Function:Active
       Link:      Base: Up          Fabric: Up
       Heartbeat: Management: Good   Data: Good
       Status Message:"Running"
     Slot  2:
       Status:Dead      Function:Active
       Link:      Base: Up          Fabric: Down
       Heartbeat: Management: Failed Data: Failed
       Status Message:"Waiting for management heartbeat."
       ...

If both the base and fabric links are down

If the diagnose load-balance status command shows that both the base and fabric links are down, the FPC may be powered off or shut down.

From the management board CLI, run the execute sensor list command to check the status of the power supplies. Look for the PS1, PS2, and PS3 output lines.

For example, for PS1:

...
65 PS1 VIN           alarm=0  value=122  threshold_status=0
66 PS1 VOUT_12V      alarm=0  value=12.032  threshold_status=0
67 PS1 Temp 1        alarm=0  value=26  threshold_status=0
68 PS1 Temp 2        alarm=0  value=38  threshold_status=0
69 PS1 Fan 1         alarm=0  value=8832  threshold_status=0
70 PS1 Status        alarm=0
...

If the power supplies are all OK, the output for all of the PS lines should include Alarm=0 and Status=0.

If the command output indicates problems with the power supplies, make sure they are all connected to power.

If they are connected, there may be a hardware problem. Contact Fortinet Support for assistance.
If the power supplies are connected and operating normally, set up two SSH sessions to the management board.
From SSH session 1, enter the following command to connect to the FPC console:

execute system console-server connect <slot_id>
Press Enter to see if there is any response.
From SSH session 2, use the following commands to power the FPC off and back on:

execute load-balance slot power-off <slot_id>

execute load-balance slot power-on <slot_id>
From SSH session1, check to see if the FPC starts up normally after running the power-on command.
If SSH session 1 shows the FPC starting up, when it has fully started, use the get system status command to compare the FPC and management board FortiOS versions.

If the versions don't match, see Updating FPC firmware to match the management board
If the FPC doesn't start up there may be a hardware problem, contact Fortinet Support for assistance.

If only one link is down

If the base or fabric link is up, then check the Heartbeat line of the diagnose load-balance status output. The following conditions on the FPC can cause the management heartbeat to fail:

The FPC did not start up correctly.
The FPC software may have stopped operating because a process has stopped.
The FPC may have experienced a kernel panic.
The FPC may have experienced a daemon or processes panic.

To get more information about the cause:

Set up two SSH sessions to the management board.
From SSH session 1, enter the following command to connect to the FPC console:

execute system console-server connect <slot_id>
Press Enter to see if there is any response.
If there is a response to SSH session 1 and if you can log into the FPC from SSH session 1:
1. Dump the crash log by entering:
  
  diagnose debug crashlog read
2. Use the get system status command to compare the FPC and management board FortiOS versions.
  
  If the versions don't match, see Updating FPC firmware to match the management board.
If there is no response to SSH session1, or if you cannot log into the FPC from SSH session 1, switch to SSH session 2.
1. From SSH session 2, run the NMI reset command:
  
  execute load-balance slot nmi-reset <slot_id>
2. From SSH session 1, check to see if any messages appear.
3. If a kernel panic stack trace is displayed, save it.
  
  The FPC should automatically reboot after displaying the stack trace.
4. If nothing happens on SSH session 1, go back to SSH session 2, and run the following commands to power off and power on the FPC:
  
  execute load-balance slot power-off <slot_id>
  
  execute load-balance slot power-on <slot_id>
5. If SSH session 1 shows the FPC starting up, when it has fully started, use the get system status command to compare the FPC and management board FortiOS versions.
6. If the versions match, start an SSH session to log into the FPC, and dump the comlog by entering:
  
  diagnose debug comlog read
  
  If the comlog was not enabled, it will be empty.
7. Also dump the crash log if you haven't been able to do so by entering:
  
  diagnose debug crashlog read
8. Contact Fortinet Support for assistance.
  
  If requested you can provide the comlog and crashlog to help determine the cause of the problem.

Updating FPC firmware to match the management board

Use the following steps to update the firmware running on the FPC to match the firmware running on the management board.

Obtain a FortiGate-6000 firmware image file that matches the version running on the management board and add it to an FTP or TFTP server or a to USB key.
Use the following command to upload the firmware image file to the internal FortiGate-6000 TFTP server:

execute upload image {ftp | tftp | usb}
Then from management board CLI, use the following command to upgrade the firmware running on the FPC:

execute load-balance update image <slot_id>
After the firmware has upgraded, use get system status on the FPC to confirm it is running the same firmware version as the management board.

Troubleshooting configuration synchronization issues

After confirming that the management board and the FPC are running the same firmware build, use the following command to determine if configuration synchronization errors remain:

diagnose sys confsync status

In the command output, in_sync=1 means the FPC is synchronized and can operate normally, in_sync=0 means the FPC is not synchronized. If the FPC is up but not synchronized, see Troubleshooting Tip: FortiGate 7000 Series blade config synchronization issues (confsync) for help troubleshooting configuration synchronization issues.