Fortinet black logo

Cookbook

Troubleshooting CPU and network resources

Copy Link
Copy Doc ID 5be0d1a4-3f0d-11eb-96b9-00505692583a:152469
Download PDF

Checking CPU and memory resources

Check the CPU and memory resources when the FortiGate is not working, the network is slow, or there is a reduced firewall session setup rate. All processes share the system resources in FortiOS, including CPU and memory.

To view system resources in the GUI:

Go to Dashboard > Status.

The resource information is located in the CPU and Memory widgets. For information, see Dashboard.

To view system resources in the CLI:

get system performance status

Sample output:

FGT# get system performance status

CPU states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq

CPU0 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq

CPU1 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq

CPU2 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq

CPU3 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq

Memory: 4050332k total, 527148k used (13%), 3381312k free (83%), 141872k freeable (3%)

Average network usage: 41 / 28 kbps in 1 minute, 54 / 44 kbps in 10 minutes, 42 / 34 kbps in 30 minutes

Average sessions: 33 sessions in 1 minute, 48 sessions in 10 minutes, 38 sessions in 30 minutes

Average session setup rate: 0 sessions per second in last 1 minute, 0 sessions per second in last 10 minutes, 0 sessions per second in last 30 minutes

Virus caught: 0 total in 1 minute

IPS attacks blocked: 0 total in 1 minute

Uptime: 0 days, 22 hours, 59 minutes

The first line of the output shows the CPU usage by category:

CPU states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq

The second line of the output shows the memory usage:

Memory: 4050332k total, 527148k used (13%), 3381312k free (83%), 141872k freeable (3%)

Memory usage should not exceed 90%. Using too much memory prevents some processes from functioning properly. For example, if the system is running low on memory, antivirus scanning enters into failopen mode where it drops connections or bypasses the antivirus system.

Other lines of output, such as average network usage, average session setup rate, viruses caught, and IPS attacks blocked, help determine why system resource usage is high.

For example:

  • A high average network usage may indicate high traffic processing on the FortiGate,
  • A very low or zero, average session setup rate may indicate the proxy is overloaded and unable to do its job.

Troubleshooting CPU and network resources

FortiGate has stopped working

If the FortiGate has stopped working, the first line of the output will look similar to this:

CPU states: 0% user 0% system 0% nice 100% idle

Network is slow

If your network is running slow, the first line of the output will look similar to this:

CPU states: 1% user 98% system 0% nice 1% idle

This example shows that all of the CPU is being used by system processes, and the FortiGate is overloaded. When overloading occurs, it is possible a process such as scanunitid is using all the resources to scan traffic. In this case you need to reduce the amount of traffic being scanned by blocking unwanted protocols, configuring more security policies to limit scanning to certain protocols, or similar actions.

It is also possible a hacker has accessed your network and is overloading it with malicious activity, such as running a spam server or using zombie PCs to attack other networks on the Internet.

You can use the following command to investigate the problem with the CPU:

get system performance top

This command shows all of the top processes that are running on the FortiGate and their CPU usage. The process names are on the left. If a process is using most of the CPU cycles, investigate it to determine whether the activity is normal.

Reduced firewall session setup rate

A reduced firewall session setup rate can be caused by a lack of system resources on the FortiGate, or reaching the session count limit for a VDOM.

Tooltip

As a best practice, administrators should record the session setup rate during normal operation to establish a baseline to help define a problem when your are troubleshooting.

The session setup rate appears in the average sessions section of the output.

A reduced firewall session setup rate will look similar to this:

Average sessions: 80 sessions in 1 minute, 30 sessions in 10 minutes, 42 sessions in 30 minutes

Average session setup rate: 3 sessions per second in last 1 minute, 0 sessions per second in last 10 minutes, 0 sessions per second in last 30 minutes

In the example above, there were 80 sessions in 1 minute, or an average of 3 sessions per second.

The values for 10 minutes and 30 minutes allow you to take a longer average for a more reliable value if your FortiGate is working at maximum capacity. The smallest FortiGate can have 1,000 sessions established per second across the unit.

Note

The session setup rate is a global command. If you have multiple VDOMs configured with many sessions in each VDOM, the session setup rate per VDOM will be slower than if there are no VDOMs configured.

High memory usage

As with any system, a FortiGate has limited hardware resources, such as memory, and all processes running on the FortiGate share the memory. Each process uses more or less memory, depending on its workload. For example, a process usually uses more memory in high traffic situations. If some processes use all of the available memory, other processes will not be able to run.

When high memory usage occurs, the services may freeze up, connections may be lost, or new connections may be refused.

If you see high memory usage in the Memory widget, the FotiGate may be handling high traffic volumes. Alternatively, the FortiGate may have problems with connection pool limits that are affecting a single proxy. If the FortiGate receives large volumes of traffic on a specific proxy, the unit may exceed the connection pool limit. If the number of free connections within a proxy connection pool reaches zero, issues may occur.

To view current memory usage information in the CLI:

diagnose hardware sysinfo memory

Sample output:

total: used: free: shared: buffers: cached: shm:

Mem: 2074185728 756936704 1317249024 0 20701184 194555904 161046528

Swap: 0 0 0

MemTotal: 2025572 kB

MemFree: 1286376 kB

MemShared: 0 kB

Buffers: 20216 kB

Cached: 189996 kB

SwapCached: 0 kB

Active: 56644 kB

Inactive: 153648 kB

HighTotal: 0 kB

HighFree: 0 kB

LowTotal: 2025572 kB

LowFree: 1286376 kB

SwapTotal: 0 kB

SwapFree: 0 kB

Checking CPU and memory resources

Check the CPU and memory resources when the FortiGate is not working, the network is slow, or there is a reduced firewall session setup rate. All processes share the system resources in FortiOS, including CPU and memory.

To view system resources in the GUI:

Go to Dashboard > Status.

The resource information is located in the CPU and Memory widgets. For information, see Dashboard.

To view system resources in the CLI:

get system performance status

Sample output:

FGT# get system performance status

CPU states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq

CPU0 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq

CPU1 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq

CPU2 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq

CPU3 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq

Memory: 4050332k total, 527148k used (13%), 3381312k free (83%), 141872k freeable (3%)

Average network usage: 41 / 28 kbps in 1 minute, 54 / 44 kbps in 10 minutes, 42 / 34 kbps in 30 minutes

Average sessions: 33 sessions in 1 minute, 48 sessions in 10 minutes, 38 sessions in 30 minutes

Average session setup rate: 0 sessions per second in last 1 minute, 0 sessions per second in last 10 minutes, 0 sessions per second in last 30 minutes

Virus caught: 0 total in 1 minute

IPS attacks blocked: 0 total in 1 minute

Uptime: 0 days, 22 hours, 59 minutes

The first line of the output shows the CPU usage by category:

CPU states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq

The second line of the output shows the memory usage:

Memory: 4050332k total, 527148k used (13%), 3381312k free (83%), 141872k freeable (3%)

Memory usage should not exceed 90%. Using too much memory prevents some processes from functioning properly. For example, if the system is running low on memory, antivirus scanning enters into failopen mode where it drops connections or bypasses the antivirus system.

Other lines of output, such as average network usage, average session setup rate, viruses caught, and IPS attacks blocked, help determine why system resource usage is high.

For example:

  • A high average network usage may indicate high traffic processing on the FortiGate,
  • A very low or zero, average session setup rate may indicate the proxy is overloaded and unable to do its job.

Troubleshooting CPU and network resources

FortiGate has stopped working

If the FortiGate has stopped working, the first line of the output will look similar to this:

CPU states: 0% user 0% system 0% nice 100% idle

Network is slow

If your network is running slow, the first line of the output will look similar to this:

CPU states: 1% user 98% system 0% nice 1% idle

This example shows that all of the CPU is being used by system processes, and the FortiGate is overloaded. When overloading occurs, it is possible a process such as scanunitid is using all the resources to scan traffic. In this case you need to reduce the amount of traffic being scanned by blocking unwanted protocols, configuring more security policies to limit scanning to certain protocols, or similar actions.

It is also possible a hacker has accessed your network and is overloading it with malicious activity, such as running a spam server or using zombie PCs to attack other networks on the Internet.

You can use the following command to investigate the problem with the CPU:

get system performance top

This command shows all of the top processes that are running on the FortiGate and their CPU usage. The process names are on the left. If a process is using most of the CPU cycles, investigate it to determine whether the activity is normal.

Reduced firewall session setup rate

A reduced firewall session setup rate can be caused by a lack of system resources on the FortiGate, or reaching the session count limit for a VDOM.

Tooltip

As a best practice, administrators should record the session setup rate during normal operation to establish a baseline to help define a problem when your are troubleshooting.

The session setup rate appears in the average sessions section of the output.

A reduced firewall session setup rate will look similar to this:

Average sessions: 80 sessions in 1 minute, 30 sessions in 10 minutes, 42 sessions in 30 minutes

Average session setup rate: 3 sessions per second in last 1 minute, 0 sessions per second in last 10 minutes, 0 sessions per second in last 30 minutes

In the example above, there were 80 sessions in 1 minute, or an average of 3 sessions per second.

The values for 10 minutes and 30 minutes allow you to take a longer average for a more reliable value if your FortiGate is working at maximum capacity. The smallest FortiGate can have 1,000 sessions established per second across the unit.

Note

The session setup rate is a global command. If you have multiple VDOMs configured with many sessions in each VDOM, the session setup rate per VDOM will be slower than if there are no VDOMs configured.

High memory usage

As with any system, a FortiGate has limited hardware resources, such as memory, and all processes running on the FortiGate share the memory. Each process uses more or less memory, depending on its workload. For example, a process usually uses more memory in high traffic situations. If some processes use all of the available memory, other processes will not be able to run.

When high memory usage occurs, the services may freeze up, connections may be lost, or new connections may be refused.

If you see high memory usage in the Memory widget, the FotiGate may be handling high traffic volumes. Alternatively, the FortiGate may have problems with connection pool limits that are affecting a single proxy. If the FortiGate receives large volumes of traffic on a specific proxy, the unit may exceed the connection pool limit. If the number of free connections within a proxy connection pool reaches zero, issues may occur.

To view current memory usage information in the CLI:

diagnose hardware sysinfo memory

Sample output:

total: used: free: shared: buffers: cached: shm:

Mem: 2074185728 756936704 1317249024 0 20701184 194555904 161046528

Swap: 0 0 0

MemTotal: 2025572 kB

MemFree: 1286376 kB

MemShared: 0 kB

Buffers: 20216 kB

Cached: 189996 kB

SwapCached: 0 kB

Active: 56644 kB

Inactive: 153648 kB

HighTotal: 0 kB

HighFree: 0 kB

LowTotal: 2025572 kB

LowFree: 1286376 kB

SwapTotal: 0 kB

SwapFree: 0 kB