Fortinet white logo
Fortinet white logo

Handbook

Solutions by issue type

Solutions by issue type

This section includes the following topics:

Management Port Connectivity issues

One of your first tests when configuring a new FortiDDoS should be to determine whether management device can reach FortiDDoS management ports.

If your management device cannot reach FortiDDoS and you have followed the Quick Start Guide instructions to set Management Port IP addresses, static-routes/gateway and DNS, first check hardware connectivity.

  • Ensure the network cables are properly plugged into the correct management port(s) on the FortiDDoS appliance.
  • Ensure there are connection lights for the management port(s) on the appliance.
    • If not, change the cable if the cable or its connector are damaged or you are unsure about the cable’s type or quality.
    • Ensure the attached device Ethernet port is set to AUTO
    • Connect the FortiDDoS-F appliance to different hardware to see if that makes a difference.
  • In the web UI, select Network > Interface > Management Ports tab and ensure that the Config Status and Link Status indicators for the ports that are in use are green (indicating that physical connections are present

If the hardware connections are correct, and the appliance is powered on but you cannot connect using the CLI or web UI, check routing policies.

You can ping and traceroute to FortiDDoS or from a FortiDDoS management port(s) to check routing paths.

You can also capture packets in pcap format from the Management ports. See Packet Capture.

CLI commands:

execute ping | ping6 | ping-option | ping6-option

  • Use ping-option to set the management port IP you wish to ping from. By default system will ping from management port 1.

execute traceroute

diagnose sniffer packet {any|mgmt1|mgmt2}

  • Allows you to watch packets to and from the management ports

diagnose netlink route list

  • Show you current routing table.

Capturing Management Port packets

You can capture management port activity using the system's packet capture facility.

Go to Network>Packet Capture

Select mgmt1 or mgmt2 as appropriate. Further instructions for Packet Capture can be found here.

Data path connectivity issues

FortiDDoS data ports have no IP nor MAC addresses in the data path. Ping and Traceroute will not work to or from these ports.

Ping or Traceroute through the FortiDDoS from an outside client to inside server (or the reverse) should be totally transparent. (See bypass/fail-open below.)

For connectivity issues, first check that front panel port LEDs are green and flashing(?) when connected. If not, there is a physical connectivity issue or the appliance is in forced bypass/fail-open mode. Use CLI execute bypass-traffic disable to ensure the appliance is inline.

Check the FortiDDoS GUI Network > Interfaces > Traffic Ports to confirm that the traffic ports are configured “up” and the link status is “up” (connected).

Check that FortiDDoS and the connected devices are all set the same where there are speed/duplex settings. For example, the upstream and downstream devices and both traffic ports on the FortiDDoS must be set to Auto for 1000BT copper ports. If an attached device port is set to 1000Full, for example, the connection will revert to half-duplex, which may not exhibit problems until traffic becomes high. Also, remember that in bypass/fail-open mode the attached equipment is now negotiating directly, without FortiDDoS in the path. To test this and to test that FortiDDoS is not impeding traffic use CLI execute-bypass enable to remove FortiDDoS from that path of 1000BT GE links and the GE Optical LC links. If connectivity still fails the issue is in the physical cabling or the port settings on the connected devices.

Resource issues

If the system resource usage appears to be abnormally high according to the Dashboard > Status: System Resource widget or the CLI command get system status, you can view the current consumption by each process by entering this CLI command:diagnose system top delay 10.

The above command generates a list of processes every 10 seconds. It includes the process names, their process ID (pid), status, CPU usage, and memory usage.

The report continues to refresh and display in the CLI until you press q (quit).

If the issue recurs, and corresponds with a hardware or configuration change, you might need to change the configuration. Look especially into reducing frequent logging. If the issue persists, contact Fortinet Technical Support.

Service Protection Policy (SPP) issues

In early FortiDDoS-F-Series releases, the Round-Robin Databases (RRDs) were created automatically for each SPP whenever the user created a new SPP via the GUI or CLI. However, if the user makes a configuration change to the SPP while the RRD creation was in progress, then the process could be interrupted in the background. This will result in incomplete RRDs with missing information for logging and graphing of traffic and drops.

In later FortiDDoS-F-Series releases, the SPPs and RRDs for all possible SPPs are created during the upgrade process. However, existing incomplete RRDs will not be repaired. Checks of RRDs and SPPs are required if you are upgrading from 6.1.0, 6.1.4 or 6.2.0.

Check the integrity of the system Service Protection Policies (SPPs) using the following CLI commands.

diagnose debug rrd_files_check

Output:

Global expected:5, found:5 (this is the global SPP)

SPP:0 expected:1857, found:1857 (this SPP is used internally)

SPP:1 expected:1857, found:1857 (this is the default SPP)

SPP:2 expected:1857, found:1857

SPP:3 expected:1857, found:1857

SPP:4 expected:1857, found:1857 (Limit for VM-04)

SPP:5 expected:1857, found:1857

SPP:6 expected:1857, found:1857

SPP:7 expected:1857, found:1857

SPP:8 expected:1857, found:1857 (Limit for 200F/VM08)

SPP:9 expected:1857, found:1857

SPP:10 expected:1857, found:1857

SPP:11 expected:1857, found:1857

SPP:12 expected:1857, found:1857

SPP:13 expected:1857, found:1857

SPP:14 expected:1857, found:1857

SPP:15 expected:1857, found:1857

SPP:16 expected:1857, found:1857 (Limit for 1500F/VM16)


If the expected and found numbers above do not match (they may not be 1857 as above, but must match), you must follow the directions below to recreate/reset the RRDs.


Note: Recreating/resetting the SPP RRDs removes all previous traffic and drop graphing information for that SPP. However, Logs are retained. If you are unsure on how to proceed, contact FortiCare for support.

Repair the SPP using the following CLI commands.

If one or a few SPPs from 1-4/8/16 are missing RRDs:

execute spp-rrd-reset spp <rule_name> (where rule_name is the textual name from the GUI)


If many SPPs are missing RRDs:

execute rrd-reset All

Note: All is case-sensitive.


If Global is missing RRDs:

execute global-rrd-reset


If any SPP is missing, contact FortiCare for support.

GUI lockup

Fortinet is receiving occasional reports of the GUI locking up at the login prompt after the correct credentials are applied while the SSH login continues to operate. If this occurs:

  1. Access the system using SSH.
  2. At the prompt, enter #execute recover-gui
  3. Refresh the browser.

This restarts several GUI processes. If this does not provide GUI access, contact Fortinet support.

Solutions by issue type

Solutions by issue type

This section includes the following topics:

Management Port Connectivity issues

One of your first tests when configuring a new FortiDDoS should be to determine whether management device can reach FortiDDoS management ports.

If your management device cannot reach FortiDDoS and you have followed the Quick Start Guide instructions to set Management Port IP addresses, static-routes/gateway and DNS, first check hardware connectivity.

  • Ensure the network cables are properly plugged into the correct management port(s) on the FortiDDoS appliance.
  • Ensure there are connection lights for the management port(s) on the appliance.
    • If not, change the cable if the cable or its connector are damaged or you are unsure about the cable’s type or quality.
    • Ensure the attached device Ethernet port is set to AUTO
    • Connect the FortiDDoS-F appliance to different hardware to see if that makes a difference.
  • In the web UI, select Network > Interface > Management Ports tab and ensure that the Config Status and Link Status indicators for the ports that are in use are green (indicating that physical connections are present

If the hardware connections are correct, and the appliance is powered on but you cannot connect using the CLI or web UI, check routing policies.

You can ping and traceroute to FortiDDoS or from a FortiDDoS management port(s) to check routing paths.

You can also capture packets in pcap format from the Management ports. See Packet Capture.

CLI commands:

execute ping | ping6 | ping-option | ping6-option

  • Use ping-option to set the management port IP you wish to ping from. By default system will ping from management port 1.

execute traceroute

diagnose sniffer packet {any|mgmt1|mgmt2}

  • Allows you to watch packets to and from the management ports

diagnose netlink route list

  • Show you current routing table.

Capturing Management Port packets

You can capture management port activity using the system's packet capture facility.

Go to Network>Packet Capture

Select mgmt1 or mgmt2 as appropriate. Further instructions for Packet Capture can be found here.

Data path connectivity issues

FortiDDoS data ports have no IP nor MAC addresses in the data path. Ping and Traceroute will not work to or from these ports.

Ping or Traceroute through the FortiDDoS from an outside client to inside server (or the reverse) should be totally transparent. (See bypass/fail-open below.)

For connectivity issues, first check that front panel port LEDs are green and flashing(?) when connected. If not, there is a physical connectivity issue or the appliance is in forced bypass/fail-open mode. Use CLI execute bypass-traffic disable to ensure the appliance is inline.

Check the FortiDDoS GUI Network > Interfaces > Traffic Ports to confirm that the traffic ports are configured “up” and the link status is “up” (connected).

Check that FortiDDoS and the connected devices are all set the same where there are speed/duplex settings. For example, the upstream and downstream devices and both traffic ports on the FortiDDoS must be set to Auto for 1000BT copper ports. If an attached device port is set to 1000Full, for example, the connection will revert to half-duplex, which may not exhibit problems until traffic becomes high. Also, remember that in bypass/fail-open mode the attached equipment is now negotiating directly, without FortiDDoS in the path. To test this and to test that FortiDDoS is not impeding traffic use CLI execute-bypass enable to remove FortiDDoS from that path of 1000BT GE links and the GE Optical LC links. If connectivity still fails the issue is in the physical cabling or the port settings on the connected devices.

Resource issues

If the system resource usage appears to be abnormally high according to the Dashboard > Status: System Resource widget or the CLI command get system status, you can view the current consumption by each process by entering this CLI command:diagnose system top delay 10.

The above command generates a list of processes every 10 seconds. It includes the process names, their process ID (pid), status, CPU usage, and memory usage.

The report continues to refresh and display in the CLI until you press q (quit).

If the issue recurs, and corresponds with a hardware or configuration change, you might need to change the configuration. Look especially into reducing frequent logging. If the issue persists, contact Fortinet Technical Support.

Service Protection Policy (SPP) issues

In early FortiDDoS-F-Series releases, the Round-Robin Databases (RRDs) were created automatically for each SPP whenever the user created a new SPP via the GUI or CLI. However, if the user makes a configuration change to the SPP while the RRD creation was in progress, then the process could be interrupted in the background. This will result in incomplete RRDs with missing information for logging and graphing of traffic and drops.

In later FortiDDoS-F-Series releases, the SPPs and RRDs for all possible SPPs are created during the upgrade process. However, existing incomplete RRDs will not be repaired. Checks of RRDs and SPPs are required if you are upgrading from 6.1.0, 6.1.4 or 6.2.0.

Check the integrity of the system Service Protection Policies (SPPs) using the following CLI commands.

diagnose debug rrd_files_check

Output:

Global expected:5, found:5 (this is the global SPP)

SPP:0 expected:1857, found:1857 (this SPP is used internally)

SPP:1 expected:1857, found:1857 (this is the default SPP)

SPP:2 expected:1857, found:1857

SPP:3 expected:1857, found:1857

SPP:4 expected:1857, found:1857 (Limit for VM-04)

SPP:5 expected:1857, found:1857

SPP:6 expected:1857, found:1857

SPP:7 expected:1857, found:1857

SPP:8 expected:1857, found:1857 (Limit for 200F/VM08)

SPP:9 expected:1857, found:1857

SPP:10 expected:1857, found:1857

SPP:11 expected:1857, found:1857

SPP:12 expected:1857, found:1857

SPP:13 expected:1857, found:1857

SPP:14 expected:1857, found:1857

SPP:15 expected:1857, found:1857

SPP:16 expected:1857, found:1857 (Limit for 1500F/VM16)


If the expected and found numbers above do not match (they may not be 1857 as above, but must match), you must follow the directions below to recreate/reset the RRDs.


Note: Recreating/resetting the SPP RRDs removes all previous traffic and drop graphing information for that SPP. However, Logs are retained. If you are unsure on how to proceed, contact FortiCare for support.

Repair the SPP using the following CLI commands.

If one or a few SPPs from 1-4/8/16 are missing RRDs:

execute spp-rrd-reset spp <rule_name> (where rule_name is the textual name from the GUI)


If many SPPs are missing RRDs:

execute rrd-reset All

Note: All is case-sensitive.


If Global is missing RRDs:

execute global-rrd-reset


If any SPP is missing, contact FortiCare for support.

GUI lockup

Fortinet is receiving occasional reports of the GUI locking up at the login prompt after the correct credentials are applied while the SSH login continues to operate. If this occurs:

  1. Access the system using SSH.
  2. At the prompt, enter #execute recover-gui
  3. Refresh the browser.

This restarts several GUI processes. If this does not provide GUI access, contact Fortinet support.