Checking CPU information&Issues

1. Check CPU information

   FortiWeb# diagnose hardware cpu list #show the detail info for all CPU/vCPU

   FortiWeb-AWS-M01 # diagnose hardware cpu list

   processor : 0

   vendor_id : GenuineIntel

   cpu family : 6

   model : 79

   model name : Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz

   stepping : 1

   microcode : 0xb000038

   cpu MHz : 2300.049

   cache size : 46080 KB

   physical id : 0

   siblings : 2

   core id : 0

   cpu cores : 2

   apicid : 0

   initial apicid : 0

   fpu : yes

   fpu_exception : yes

   cpuid level : 13

   wp : yes

   …

2. CPU & processor numbers

   FortiWeb # fn cat /proc/cpuinfo | grep "cpu cores" #Check physical CPU cores

   cpu cores : 16

   /# cat /proc/cpuinfo |grep "processor" | sort -u | wc -l #Check logical CPU cores when hyperthread is enabled

   32 cat /proc/cpuinfo |grep "processor

3. Check which daemon or process consuming the most CPU usage

   To determine if high load is frequently a problem, you can display the average load level by using these CLI commands:

   FortiWeb # get system performance

   CPU states: 5% used, 95% idle

   Memory states: 29% used

   Up: 9 days, 12 hours, 52 minutes.

   top

   Use the CLI to view the per-CPU/core process load level and a list of the most system-intensive processes. This may show processes that are consuming resources unusually.

   While the command is running, you can press Shift + P to sort the five columns of data by CPU usage (the default) or Shift + M to sort by memory usage.

   FortiWeb# diagnose system top 10

   Mem: 4867300K used, 126120392K free, 16536K shrd, 10792K buff, 117620K cached

   CPU: 0.1% usr 0.1% sys 0.0% nic 99.6% idle 0.0% io 0.0% irq 0.0% sirq

   Load average: 1.71 1.55 1.49 2/953 52110

   PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND

   6262 1 root S 9582m 7.4 31 0.3 /bin/proxyd

   6264 1 root S 6539m 5.1 29 0.0 /bin/bot_daemon

   6273 1 root S 2498m 1.9 21 0.0 /bin/garbage -o standalone

   6316 6238 root S 2098m 1.6 24 0.0 /bin/mysqld --defaults-file=/data/e

   6251 1 root S 803m 0.6 10 0.0 /bin/monitord

   6269 1 root S 411m 0.3 21 0.0 /bin/sandboxd

   6271 1 root S 400m 0.3 43 0.0 /bin/shibd -F -f -p /var/run/shibd.

   6287 1 root S 256m 0.2 59 0.0 /bin/statusd

   The above command generates a report of processes every 10 seconds. The report provides the process names, their process ID (pid), status, CPU usage, and memory usage.

   The report continues to refresh and display in the CLI until you press q (quit).

   perf top

   The perf top command is used for real time system profiling and functions similarly to the top utility. However, where the top utility generally shows you how much CPU time a given process or thread is using, perf top shows you how much CPU time each specific function uses. In its default state, perf top tells you about functions being used across all CPUs in both the user-space and the kernel-space.

   FortiWeb# diagnose system perf # or “perf top” in backend shell

   FortiWeb# diagnose system perf

   PerfTop: 69182 irqs/sec kernel:96.4% exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles], (all, 64 CPUs)

   --------------------------------------------------------------------------------

   13.50% [kernel] [k] find_busiest_group

   3.20% [kernel] [k] idle_cpu

   3.15% [kernel] [k] _raw_spin_lock

   2.44% [kernel] [k] __schedule

   2.42% [kernel] [k] rcu_sched_clock_irq

   2.07% [kernel] [k] _raw_spin_trylock

   1.95% [kernel] [k] native_irq_return_iret

4. Kill processes

   Once you locate an offending PID from “diagnose system top”, you may want to terminate it. For example, in a test environment or when you fail to locate the cause when access to a server-policy always fails, you may try to kill proxyd or dnsproxyd.

   Under normal conditions, killing a process is not recommended.

   diagnose system kill 9 <pid>

   or

   Fn kill 9 <pid>

   On some 7.0.x builds, you can execute “fn kill <pid>” on the front-end CLI, or need to login to the back-end shell and then execute kill:

   /# kill 9 <pid>

   Please refer to Run backend-shell commands to learn how to configure shell-access.

5. Check if high CPU usage is caused by heavy traffic load

   Heavy traffic loads can cause sustained high CPU or RAM usage. If this is unusual, no action may be required, unless you are being subject to a DoS attack. Sustained heavy traffic load may indicate that you need a more powerful model of FortiWeb.

   You can check traffic load via GUI or debug logs in several ways:

   1) Monitor Total Connection per Second, Total Connections and Total HTTP Transaction， Throughput on the GUI dashboard.

   Total Connection per Second, Total Connections (also Concurrent Connection) are displayed directly in the widgets “System Resource” and “Policy Sessions”, whereas the current HTTP transaction per second is not displayed directly on GUI. You need to enable/add a widget named “HTTP Transactions” and calculate the TPS by dividing the total transaction in 5 minutes.

   Taking the screenshot below for example, the concurrent connection is 100000 and there are no new connections established per second, whereas there are nearly 6000000 transactions in the past 5 minutes - equal to 20000 transactions per second (TPS), so this might be the main cause why CPU usage reaches 10%.

   

   2)Some of these four real-time performance numbers can be also obtained via CLIs:

   • Total Connection per Second: diagnose policy total-conn-psec list

   • Total Connections: diagnose policy total-session list

   • Total Throughput in HTTP level: diagnose policy total-traffic http list
     This statistics from CLI only includes HTTP payload, does not include L2 & L3 headers

   • HTTP Transaction per Second: diagnose policy total-detail-stats list <server-policy>
     No total statistics in CLI

   3) Check TCP connections in TIME_WAIT status

   TIME_WAIT connections cannot be displayed in dashboard widgets but also consume system connection/memory resources. You can also check connection in backend shell:

   /# netstat -nat | awk '{print $6}' | sort | uniq -c | sort -r

   199101 ESTABLISHED #Concurrent connections

   251 LISTEN

   7 TIME_WAIT

   1 established)

   1 Foreign

   4) Examine traffic history in the traffic log. Go to Logs&Report > Log Access > Traffic.

   If massive traffic logs are generated in a short period, it indicates heavy traffic load.

6. Check if high CPU usage is caused by Attacks

   A prolonged denial of service (DoS) or brute-force login attack (to name just a few) can bring your web servers to a standstill, if your FortiWeb appliance is not configured for it.

   In the FortiWeb appliance's web UI, you can watch for attacks in two ways:

   1) Monitor current HTTP traffic on the dashboard. Go to System > Status > Status and examine the attack event history graph in the Policy Summary widget.

   2) Examine attack history in the traffic log. Go to Logs&Report > Log Access > Attack.

   Before attacks occur, use the FortiWeb appliance's rich feature set to configure attack defenses.

7. 7. Check if high CPU usage is caused by port exhaustion

   If there is only one interface IP address and only one back-end server, the CPU tends to be relatively high because too many ports will be occupied for building up connections with the back-end server..

   You will see many connections are in the TIME_WAIT state and being closed after 2 MSL (Maximum Segment life) time. To check the connection in the TIME_WAIT state, you can run dia network netstat -nat | grep TIME_WAIT.

   If you run “diagnose system perf top”, you can see the following perf top info:

   

   you have three options to avoid the port exhaustion.

   • set ip-local-port-assign-ex

   
   
   FortiWeb # config system network-option
   FortiWeb (network-option) # sh
   config system network-option
     set ip-local-port-assign-ex enable
   end
   

   • Add more IP addresses of the interface to real server and enable the ip-src-balance option.

     To add more ip addresses of the interface:

     config system interface
       edit "port3"
         set type physical
         set ip 40.40.40.1/16
         config  secondaryip
           edit 1
             set ip 40.40.40.2/16
           next
         end
         config  classless_static_route
         end
       next
     end

     To enable the ip-src-balance option:

        config system network-option
          set ip-src-balance enable
          set ip6-src-balance enable
        end

   • Applying client real IP in busiest server policy:

     config server-policy policy
        edit policy_name
          set client-real-ip enable
     	 ip range: ip-range 
        next
     end
     

8. Check system and debug logs to see CPU resource status:

   1) Log&Report > Event > Filter > Action > check-resource

   Log example:

   CPU usage too high,CPU usage is 95, process proxyd

   2) Analyze NMON files with all relevant statistics

   NMON files include CPU, Mem, I/O statistics, you can do a comprehensive analysis from these relevant information.