Checking CPU and memory resources
Check the CPU and memory resources when the FortiGate is not working, the network is slow, or there is a reduced firewall session setup rate. All processes share the system resources in FortiOS, including CPU and memory.
To view system resources in the GUI:
Go to Dashboard > Status. The resource information is located in the CPU and Memory widgets. For information, see Dashboards and Monitors.
To view system resources in the CLI:
# get system performance status
Sample output:
# get system performance status CPU states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq CPU0 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq CPU1 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq CPU2 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq CPU3 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq Memory: 2039608k total, 1154872k used (56.6%), 571856k free (28.0%), 312880k freeable (15.4%) Average network usage: 0 / 0 kbps in 1 minute, 1 / 9 kbps in 10 minutes, 2 / 45 kbps in 30 minutes Maximal network usage: 1 / 0 kbps in 1 minute, 36 / 1630 kbps in 10 minutes, 655 / 6758 kbps in 30 minutes Average sessions: 7 sessions in 1 minute, 7 sessions in 10 minutes, 6 sessions in 30 minutes Maximal sessions: 9 sessions in 1 minute, 13 sessions in 10 minutes, 36 sessions in 30 minutes Average session setup rate: 0 sessions per second in last 1 minute, 0 sessions per second in last 10 minutes, 0 sessions per second in last 30 minutes Maximal session setup rate: 0 sessions per second in last 1 minute, 5 sessions per second in last 10 minutes, 14 sessions per second in last 30 minutes Average NPU sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last 30 minutes Maximal NPU sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last 30 minutes Virus caught: 0 total in 1 minute IPS attacks blocked: 0 total in 1 minute Uptime: 9 days, 0 hours, 38 minutes
-
The first lines of the output show the CPU usage by category for each CPU core:
CPU states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq CPU0 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq CPU1 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq CPU2 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq CPU3 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
-
The next line of the output shows the memory usage:
Memory: 2039608k total, 1154872k used (56.6%), 571856k free (28.0%), 312880k freeable (15.4%)
Memory usage should not exceed 90%. Using too much memory prevents some processes from functioning properly. For example, if the system is running low on memory, antivirus scanning enters into failopen mode where it drops connections or bypasses the antivirus system.
-
Network usage, sessions, session setup rate, and NPU sessions are shown next, summarizing the network and session usage. Samples are taken every three seconds. The
Average
values are the average of all of the samples taken during the sample period (1, 10, 30 minutes, and so on). TheMaximal
values are the maximum values recorded during the sample period. -
For example, a high
average network usage
may indicate high traffic processing on the FortiGate, while a very low or zeroaverage session setup rate
may indicate the proxy is overloaded and unable to function.
-
The
viruses caught
andIPS attacks blocked
lines help determine why system resource usage is high.
Troubleshooting CPU and network resources
FortiGate has stopped working
If the FortiGate has stopped working, the first line of the output will look similar to this:
CPU states: 0% user 0% system 0% nice 100% idle
Network is slow
If your network is running slow, the first line of the output will look similar to this:
CPU states: 1% user 98% system 0% nice 1% idle
This example shows that all of the CPU is being used by system processes, and the FortiGate is overloaded. When overloading occurs, it is possible a process such as scanunitid
is using all the resources to scan traffic. In this case you need to reduce the amount of traffic being scanned by blocking unwanted protocols, configuring more security policies to limit scanning to certain protocols, or similar actions.
It is also possible a hacker has accessed your network and is overloading it with malicious activity, such as running a spam server or using zombie PCs to attack other networks on the Internet.
You can use the following commands to investigate the problem with the CPU:
# get system performance top <delay> <lines>
# diagnose sys top <delay> <lines> <repeat>
These commands show all of the top processes that are running on the FortiGate and their CPU usage; the process names are on the left. If a process is using most of the CPU cycles, investigate it to determine whether that activity is normal.
Reduced firewall session setup rate
A reduced firewall session setup rate can be caused by a lack of system resources on the FortiGate, or reaching the session count limit for a VDOM.
As a best practice, administrators should record the session setup rate during normal operation to establish a baseline to help define a problem when your are troubleshooting. |
The session setup rate appears in the average sessions
section of the output.
A reduced firewall session setup rate will look similar to this:
Average sessions: 80 sessions in 1 minute, 30 sessions in 10 minutes, 42 sessions in 30 minutes Average session setup rate: 3 sessions per second in last 1 minute, 0 sessions per second in last 10 minutes, 0 sessions per second in last 30 minutes
In the example above, there were 80 sessions in 1 minute, or an average of 3 sessions per second.
The values for 10 minutes
and 30 minutes
allow you to take a longer average for a more reliable value if your FortiGate is working at maximum capacity. The smallest FortiGate can have 1,000 sessions established per second across the unit.
The session setup rate is a global command. If you have multiple VDOMs configured with many sessions in each VDOM, the session setup rate per VDOM will be slower than if there are no VDOMs configured. |
High memory usage
As with any system, a FortiGate has limited hardware resources, such as memory, and all processes running on the FortiGate share the memory. Each process uses more or less memory, depending on its workload. For example, a process usually uses more memory in high traffic situations. If some processes use all of the available memory, other processes will not be able to run.
When high memory usage occurs, the services may freeze up, connections may be lost, or new connections may be refused.
If you see high memory usage in the Memory widget, the FotiGate may be handling high traffic volumes. Alternatively, the FortiGate may have problems with connection pool limits that are affecting a single proxy. If the FortiGate receives large volumes of traffic on a specific proxy, the unit may exceed the connection pool limit. If the number of free connections within a proxy connection pool reaches zero, issues may occur.
To view current memory usage information in the CLI:
# diagnose hardware sysinfo memory
Sample output:
# diagnose hardware sysinfo memory MemTotal: 3075676 kB MemFree: 1067428 kB MemAvailable: 1490772 kB Buffers: 193700 kB Cached: 802828 kB SwapCached: 0 kB Active: 1015412 kB Inactive: 266168 kB Active(anon): 745256 kB Inactive(anon): 72208 kB Active(file): 270156 kB Inactive(file): 193960 kB Unevictable: 207480 kB Mlocked: 0 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 16 kB Writeback: 0 kB AnonPages: 492532 kB Mapped: 257632 kB Shmem: 339868 kB Slab: 161308 kB SReclaimable: 42236 kB SUnreclaim: 119072 kB KernelStack: 3872 kB PageTables: 31948 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 1537836 kB Committed_AS: 22223044 kB VmallocTotal: 34359738367 kB VmallocUsed: 0 kB VmallocChunk: 0 kB Percpu: 308 kB AnonHugePages: 0 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB CmaTotal: 0 kB CmaFree: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 0 kB DirectMap4k: 51136 kB DirectMap2M: 3094528 kB