Troubleshooting high CPU usage

Connection-related problems may occur when FortiGate's CPU resources are over extended. This occurs when you deploy too many FortiOS features at the same time.

Examples of CPU intensive features:

• VPN high-level encryption
• Intensive scanning of all traffic
• Logging all traffic and packets
• Dashboard widgets that frequently perform data updates

For information on customizing the CPU use threshold, see Execute a CLI script based on CPU and memory thresholds.

Determining the current level of CPU usage

You can view CPU usage levels in the GUI or CLI. For precise usage values for both overall usage and specific processes, use the CLI.

To view CPU usage in the GUI:

Go to Dashboard > Status. Real-time CPU usage information is located in the CPU widget.

To view CPU usage in the CLI:

• Show top processes information:

  diagnose sys top

• Show top threads information:

  diagnose sys top-all

Sample output:

Run Time: 86 days, 0 hours and 10 minutes

0U, 0N, 0S, 100I, 0WA, 0HI, 0SI, 0ST; 3040T, 2437F

bcm.user 93 S < 3.1 0.4

httpsd 18922 S 1.5 0.5

httpsd 19150 S 0.3 0.5

newcli 20195 R 0.1 0.1

cmdbsvr 115 S 0.0 0.8

pyfcgid 20107 S 0.0 0.6

forticron 146 S 0.0 0.5

httpsd 139 S 0.0 0.5

cw_acd 166 S 0.0 0.5

miglogd 136 S 0.0 0.5

pyfcgid 20110 S 0.0 0.4

pyfcgid 20111 S 0.0 0.4

pyfcgid 20109 S 0.0 0.4

httpsd 20192 S 0.0 0.4

miglogd 174 S 0.0 0.4

miglogd 175 S 0.0 0.4

fgfmd 165 S 0.0 0.3

newcli 20191 S 0.0 0.3

initXXXXXXXXXXX 1 S 0.0 0.3

httpsd 184 s 0.0 0.3

The following table explains the codes in the second line of the output:

Code	Description
U	Percentage of user space applications that are currently using the CPU
N	Percentage of time that the CPU spent on low priority processes since the last shutdown
S	Percentage of system processes (or kernel processes) that are using the CPU
I	Percentage of idle CPU resources
WA	Percentage of time that the CPU spent waiting on IO peripherals since the last shutdown
HI	Percentage of time that the CPU spent handling hardware interrupt routines since the last shutdown
SI	Percentage of time that the CPU spent handling software interrupt routines since the last shutdown
ST	Steal time: Percentage of time a virtual CPU waits for the physical CPU when the hypervisor is servicing another virtual processor
T	Total FortiOS system memory in MB
F	Free memory in MB

Each additional line of the command output displays information specific to processes or threads that are running on the FortiGate unit. For example, the sixth line of the output is: newcli 20195 R 0.1 0.1

The following table describes the data in the sixth line of the output:

Item	Description
newcli	The process (or thread) name. Duplicate process or thread names indicate that separate instances of that process or thread are running.
20195	The process or thread ID, which can be any number.
R	Current state of the process or thread. The process or thread state can be: R - running S - sleep Z - zombie D- disk sleep
0.1	The percentage of CPU capacity that the process or thread is using. CPU usage can range from 0.0 for a process or thread that is sleeping to higher values for a process or thread that's taking a lot of CPU time.
0.1	The amount of memory that the process or thread is using. Memory usage can range from 0.1 to 5.5 and higher.

You can use the following single-key commands when running diagnose sys top or diagnose sys top-all:

• q to quit and return to the normal CLI prompt.
• p to sort the processes by the amount of CPU that the processes are using.
• m to sort the processes by the amount of memory that the processes are using.

The output only displays the top processes or threads that are running. For example, if 20 are listed, they are the top 20 currently running, sorted by either CPU or memory usage. You can configure the number of processes or threads displayed, using the following CLI commands:

diagnose sys top <integer_seconds> <integer_maximum_lines>

diagnose sys top-all <integer_seconds> <integer_maximum_lines>

Where:

• <integer_seconds> is the delay in seconds (default is 5)
• <integer_maximum_lines> is the maximum number of lines (or processes) to list (default is 20)

Determining which features are using the most CPU resources

You can use the CLI to view the top few processes that are currently running and using the most CPU resources.

To view processes using the most CPU resources:

get system performance top

The entries at the top are using the most CPU resources. The second column from the right shows CPU usage by percentage. Note which processes are using the most resources and try to reduce their CPU load.

Processes you will see include:

• ipsengine: the IPS engine that scans traffic for intrusions
• scanunitd: antivirus scanner
• httpsd: secure HTTP
• iked: internet key exchange (IKE) in use with IPsec VPN tunnels
• newcli: active whenever you're accessing the CLI
• sshd: there are active secure socket connections
• cmdbsrv: the command database server application

Go to the features that are at the top of the list and look for evidence of CPU overuse. Generally, the monitor for a feature is a good place to start.

Checking for unnecessary CPU “wasters”

These are some best practices that will reduce your CPU usage, even if the FortiGate is not experiencing high CPU usage. Note that if the following information instructs you to turn off a feature that you require, disregard that part of the instructions.

• Use hardware acceleration wherever possible to offload tasks from the CPU. Offloading tasks, such as encryption, frees up the CPU for other tasks.
• Avoid the use of GUI widgets that require computing cycles, such as the Top Sessions widget. These widgets constantly pol the system for information, which uses CPU and other resources.
• Schedule antivirus, IPS, and firmware updates during off-peak hours. These updates do not usually consume CPU resources but they can disrupt normal operation.
• Check the log levels and which events are being logged. This is the severity of the messages that are recorded. Consider going up one level to reduce the amount of logging. Also, if there are events you do not need to monitor, remove them from the list.
• Log to FortiCloud instead of logging to memory or disk. Logging to memory quickly uses up resources and logging to local disk impacts overall performance and reduces the lifetime of the unit.

  Fortinet recommends logging to FortiCloud to avoid using too much CPU.

• If the disk is almost full, transfer the logs or data off the disk to free up space. When a disk is almost full it consumes a lot of resources to find free space and organize the files.
• If packet logging is enabled on the FortiGate, consider disabling it. When packet logging is enabled, it records every packet that comes through that policy.
• Halt all sniffers and traces.
• Ensure the FortiGate isn't scanning traffic twice. Traffic does not need to be rescanned if it enters the FortiGate on one interface, goes out another, and then comes back in again. Doing so is a waste of resources. However, ensure that traffic truly is being scanned once.

• Reduce the session timers to close unused sessions faster. Enter the following CLI commands, which reduce the default values. Note that, by default, the system adds 10 seconds to tcp-timewait.

  config system global

  set tcp-halfclose-timer 30

  set tcp-halfopen-timer 30

  set tcp-timewait-timer 0

  set udp-idle-timer 60

  end

• Go to System > Feature Visibility, and enable only features that you need.

SNMP monitoring

When CPU usage is under control, use SNMP to monitor CPU usage. Alternatively, use logging to record CPU and memory usage every 5 minutes.

Once the system is back to normal, you should set up a warning system that sends alerts when CPU resources are used excessively. A common method to do this is using SNMP. SNMP monitors many values in FortiOS and allows you to set high water marks that generate events. You can run an application on your computer to watch for and record these events.

To enable SNMP:

1. Go to System > SNMP.
2. Configure an SNMP community.

See SNMP.

You can use the System Resources widget to record CPU usage if SNMP is too complicated. However, the widget only records problems as they happen and will not send you alerts for problems.