Fortinet black logo

Administration Guide

Execute a CLI script based on memory and CPU thresholds

Execute a CLI script based on memory and CPU thresholds

Automation stitches can be created to run a CLI script and send an email message when memory or CPU usage exceeds specified thresholds.

Note

The maximum size of the CLI script action output is 16K characters. In cases where the output exceeds 16K, the email received will contain truncated output. To avoid this, it is recommended to limit the number of commands per script.

Note

Automation stitches that use Conserve Mode and High CPU triggers can only be created in the CLI. Once created, they can be edited in the GUI.

To define CPU and memory usage thresholds:
config system global
    set cpu-use-threshold <percent>
    set memory-use-threshold-extreme <percent>
    set memory-use-threshold-green <percent>
    set memory-use-threshold-red <percent>
end

Where:

cpu-use-threshold

Threshold at which CPU usage is reported, in percent of total possible CPU utilization (default = 90).

memory-use-threshold-extreme

Threshold at which memory usage is considered extreme, and new sessions are dropped, in percent of total RAM (default = 95).

memory-use-threshold-green

Threshold at which memory usage forces the FortiGate to exit conserve mode, in percent of total RAM (default = 82).

memory-use-threshold-red

Threshold at which memory usage forces the FortiGate to enter conserve mode, in percent of total RAM (default = 88).

Configuring a high memory usage stitch

In this example, an automation stitch is created that runs two CLI scripts to collect debug information, and then two email messages will be received with CLI output to a specified email address when the memory usage causes the FortiGate to enter conserve mode.

Note

Since the output in this example will exceed 16K, two scripts are used. The CLI scripts are run sequentially, and an email is sent out each time a script runs.

To create an automation stitch for high memory usage:
  1. Create the automation trigger:
    config system automation-trigger
        edit "auto_high_memory"
            set event-type low-memory
        next
    end
  2. Create the automation actions:
    config system automation-action
        edit "high_memory_debug1"
            set action-type cli-script
            set script "diagnose sys top 5 20 5
    diagnose sys session full-stat 
    get system performance status"
            set output-size 10
            set timeout 0	
            set accprofile "super_admin"	
        next
        edit "auto_high_memory_email1"
            set action-type email
            set email-to "person@fortinet.com"
            set email-subject "CSF stitch alert: high_memory1"
            set message "%%results%%"
        next
           edit "high_memory_debug2"
            set action-type cli-script
            set script "diagnose sys session full-stat
    diagnose hardware sysinfo shm
    diagnose hardware sysinfo memory"
            set accprofile "super_admin"
        next
        edit "auto_high_memory_email2"
            set action-type email
            set email-to "person@fortinet.com"
            set email-subject "CSF stitch alert: high_memory2"
            set message "%%results%%"
        next
    end
  3. Create the automation stitch:
    config system automation-stitch
        edit "auto_high_memory"
            set trigger "auto_high_memory"
            set action "high_memory_debug1" "auto_high_memory_email1" "high_memory_debug2" "auto_high_memory_email2"
        next
    end

Results

When the FortiGate enters conserve mode due to the memory-use-threshold-red being exceeded, the GUI displays a notice, and the auto_high_memoryautomation stitch is triggered. This causes the CLI scripts to run and the scripts' results are emailed to the specified address.

Here is sample text from the email message:

CSF stitch alert: high_memory1
DoNotReply@notification.fortinet.net
Tue 05/16/2023 5:34 PM
########## script name: autod.0 ########## 
========== #1, 2023-05-16 17:34:17 ========== 
Client_Fgt $  diagnose sys top 5 20 3 
Run Time:  0 days, 0 hours and 0 minutes 61U, 0N, 6S, 33I, 0WA, 0HI, 0SI, 0ST; 1356T, 129F
       ipshelper     2601      S <    61.6     8.0    0
       ipsengine     2745      S <     4.9     8.5    0
         cmdbsvr     2528      S N     0.0     7.9    0
         cmdbsvr     2529      S       0.0     5.0    0
       scanunitd     2610      S <     0.0     3.8    0
         miglogd     2603      S       0.0     3.6    0
          cw_acd     2634      S       0.0     3.4    0
            node     2574      S       0.0     3.3    0
       forticron     2584      S       0.0     2.9    0
         miglogd     2693      S       0.0     2.8    0
         reportd     2604      S       0.0     2.5    0
          httpsd     2573      S       0.0     2.4    0

Configuring a high CPU usage stitch

Similar to the previous example, an automation stitch can be created that runs a CLI script to collect debug information, and then email the results of the script to a specified email address when CPU usage threshold is exceeded (High CPU trigger type).

The following commands are recommended for collecting debug information, but they are not the only options. Other commands can be used.

diagnose sys cmdb info
diagnose sys vd list | grep fib
diagnose sys top 5 20 2
diagnose sys session full-stat
diagnose sys session list | grep "\<dirty\>" –c
get system performance status
diagnose sys session full-stat
diagnose hardware sysinfo memory
diagnose sys cmdb info
diagnose sys vd list | grep fib

Execute a CLI script based on memory and CPU thresholds

Automation stitches can be created to run a CLI script and send an email message when memory or CPU usage exceeds specified thresholds.

Note

The maximum size of the CLI script action output is 16K characters. In cases where the output exceeds 16K, the email received will contain truncated output. To avoid this, it is recommended to limit the number of commands per script.

Note

Automation stitches that use Conserve Mode and High CPU triggers can only be created in the CLI. Once created, they can be edited in the GUI.

To define CPU and memory usage thresholds:
config system global
    set cpu-use-threshold <percent>
    set memory-use-threshold-extreme <percent>
    set memory-use-threshold-green <percent>
    set memory-use-threshold-red <percent>
end

Where:

cpu-use-threshold

Threshold at which CPU usage is reported, in percent of total possible CPU utilization (default = 90).

memory-use-threshold-extreme

Threshold at which memory usage is considered extreme, and new sessions are dropped, in percent of total RAM (default = 95).

memory-use-threshold-green

Threshold at which memory usage forces the FortiGate to exit conserve mode, in percent of total RAM (default = 82).

memory-use-threshold-red

Threshold at which memory usage forces the FortiGate to enter conserve mode, in percent of total RAM (default = 88).

Configuring a high memory usage stitch

In this example, an automation stitch is created that runs two CLI scripts to collect debug information, and then two email messages will be received with CLI output to a specified email address when the memory usage causes the FortiGate to enter conserve mode.

Note

Since the output in this example will exceed 16K, two scripts are used. The CLI scripts are run sequentially, and an email is sent out each time a script runs.

To create an automation stitch for high memory usage:
  1. Create the automation trigger:
    config system automation-trigger
        edit "auto_high_memory"
            set event-type low-memory
        next
    end
  2. Create the automation actions:
    config system automation-action
        edit "high_memory_debug1"
            set action-type cli-script
            set script "diagnose sys top 5 20 5
    diagnose sys session full-stat 
    get system performance status"
            set output-size 10
            set timeout 0	
            set accprofile "super_admin"	
        next
        edit "auto_high_memory_email1"
            set action-type email
            set email-to "person@fortinet.com"
            set email-subject "CSF stitch alert: high_memory1"
            set message "%%results%%"
        next
           edit "high_memory_debug2"
            set action-type cli-script
            set script "diagnose sys session full-stat
    diagnose hardware sysinfo shm
    diagnose hardware sysinfo memory"
            set accprofile "super_admin"
        next
        edit "auto_high_memory_email2"
            set action-type email
            set email-to "person@fortinet.com"
            set email-subject "CSF stitch alert: high_memory2"
            set message "%%results%%"
        next
    end
  3. Create the automation stitch:
    config system automation-stitch
        edit "auto_high_memory"
            set trigger "auto_high_memory"
            set action "high_memory_debug1" "auto_high_memory_email1" "high_memory_debug2" "auto_high_memory_email2"
        next
    end

Results

When the FortiGate enters conserve mode due to the memory-use-threshold-red being exceeded, the GUI displays a notice, and the auto_high_memoryautomation stitch is triggered. This causes the CLI scripts to run and the scripts' results are emailed to the specified address.

Here is sample text from the email message:

CSF stitch alert: high_memory1
DoNotReply@notification.fortinet.net
Tue 05/16/2023 5:34 PM
########## script name: autod.0 ########## 
========== #1, 2023-05-16 17:34:17 ========== 
Client_Fgt $  diagnose sys top 5 20 3 
Run Time:  0 days, 0 hours and 0 minutes 61U, 0N, 6S, 33I, 0WA, 0HI, 0SI, 0ST; 1356T, 129F
       ipshelper     2601      S <    61.6     8.0    0
       ipsengine     2745      S <     4.9     8.5    0
         cmdbsvr     2528      S N     0.0     7.9    0
         cmdbsvr     2529      S       0.0     5.0    0
       scanunitd     2610      S <     0.0     3.8    0
         miglogd     2603      S       0.0     3.6    0
          cw_acd     2634      S       0.0     3.4    0
            node     2574      S       0.0     3.3    0
       forticron     2584      S       0.0     2.9    0
         miglogd     2693      S       0.0     2.8    0
         reportd     2604      S       0.0     2.5    0
          httpsd     2573      S       0.0     2.4    0

Configuring a high CPU usage stitch

Similar to the previous example, an automation stitch can be created that runs a CLI script to collect debug information, and then email the results of the script to a specified email address when CPU usage threshold is exceeded (High CPU trigger type).

The following commands are recommended for collecting debug information, but they are not the only options. Other commands can be used.

diagnose sys cmdb info
diagnose sys vd list | grep fib
diagnose sys top 5 20 2
diagnose sys session full-stat
diagnose sys session list | grep "\<dirty\>" –c
get system performance status
diagnose sys session full-stat
diagnose hardware sysinfo memory
diagnose sys cmdb info
diagnose sys vd list | grep fib