Execute a CLI script based on memory and CPU thresholds

Automation stitches can be created to run a CLI script and send an email message when memory or CPU usage exceeds specified thresholds.

The maximum size of the CLI script action output is 192K characters.

To define memory and CPU usage thresholds:

config system global
    set cpu-use-threshold <percent>
    set memory-use-threshold-extreme <percent>
    set memory-use-threshold-green <percent>
    set memory-use-threshold-red <percent>
end

Where:

cpu-use-threshold	Threshold at which CPU usage is reported, in percent of total possible CPU utilization (default = 90).
memory-use-threshold-extreme	Threshold at which memory usage is considered extreme, and new sessions are dropped, in percent of total RAM (default = 95).
memory-use-threshold-green	Threshold at which memory usage forces the FortiGate to exit conserve mode, in percent of total RAM (default = 82).
memory-use-threshold-red	Threshold at which memory usage forces the FortiGate to enter conserve mode, in percent of total RAM (default = 88).

Configuring a high memory usage stitch

In this example, an automation stitch is created that runs a CLI script to collect debug information, and then email the results of the script to a specified email address when the memory usage causes the FortiGate to enter conserve mode.

To create an automation stitch for high memory usage in the GUI:

1. Go to Security Fabric > Automation and click Create New.
2. Enter the stitch name (auto_high_memory).
3. Configure the trigger:
   1. Click Add Trigger.
   2. Click Create and select Conserve Mode.
   3. Enter the name, auto_high_memory.
   4. Click OK.
   5. Select the trigger in the list and click Apply.
4. Configure the CLI Script action:
   1. Click Add Action.
   2. Click Create and select CLI Script.
   3. Enter the following:

      Name	high_memory_debug
      Script	diagnose sys top 5 20 3 diagnose sys session full-stat get system performance status diagnose sys session full-stat diagnose hardware sysinfo shm diagnose hardware sysinfo memory
      Administrator profile	Select a profile

      

   4. Click OK.
   5. Select the action in the list and click Apply.
5. Configure the Email notification action:
   1. Click Add Action.
   2. Click Create and select Email.
   3. Enter the following:

      Name	auto_high_memory_email
      To	Enter an email address
      Subject	CSF stitch alert: high_memory
      Body	%%results%%

   4. Click OK.
   5. Select the action in the list and click Apply.
6. Click OK.

To create an automation stitch for high memory usage in the CLI:

1. Create the automation trigger:

   config system automation-trigger
       edit "auto_high_memory"
           set event-type low-memory
       next
   end

2. Create the automation actions:

   config system automation-action
       edit "high_memory_debug"
           set action-type cli-script
           set script "diagnose sys top 5 20 3
   diagnose sys session full-stat
   get system performance status
   diagnose sys session full-stat
   diagnose hardware sysinfo shm
   diagnose hardware sysinfo memory"
           set output-size 10
           set timeout 0	
           set accprofile "super_admin"	
       next
       edit "auto_high_memory_email"
           set action-type email
           set email-to "person@fortinet.com"
           set email-subject "CSF stitch alert: high_memory"
           set message "%%results%%"
       next
   end

3. Create the automation stitch:

   config system automation-stitch
       edit "auto_high_memory"
           set trigger "auto_high_memory"
           config actions
               edit 1
                   set action "high_memory_debug"
                   set required enable
               next
               edit 2
                   set action "auto_high_memory_email"
                   set required enable
               next
           end
       next
   end

Results

When the FortiGate enters conserve mode due to the memory-use-threshold-red being exceeded, the GUI displays a notice, and the auto_high_memory automation stitch is triggered. This causes the CLI scripts to run and the scripts' results are emailed to the specified address.

Here is sample text from the email message:

CSF stitch alert: high_memory
DoNotReply@notification.fortinet.net
Tue 10/03/2023 5:34 PM
########## script name: autod.0 ########## 
========== #1, 2023-10-03 17:34:17 ========== 
Client_Fgt $  diagnose sys top 5 20 3 
Run Time:  0 days, 0 hours and 0 minutes 61U, 0N, 6S, 33I, 0WA, 0HI, 0SI, 0ST; 1356T, 129F
       ipshelper     2601      S <    61.6     8.0    0
       ipsengine     2745      S <     4.9     8.5    0
         cmdbsvr     2528      S N     0.0     7.9    0
         cmdbsvr     2529      S       0.0     5.0    0
       scanunitd     2610      S <     0.0     3.8    0
         miglogd     2603      S       0.0     3.6    0
          cw_acd     2634      S       0.0     3.4    0
            node     2574      S       0.0     3.3    0
       forticron     2584      S       0.0     2.9    0
         miglogd     2693      S       0.0     2.8    0
         reportd     2604      S       0.0     2.5    0
          httpsd     2573      S       0.0     2.4    0
...

Configuring a high CPU usage stitch

Similar to the previous example, an automation stitch can be created that runs a CLI script to collect debug information, and then email the results of the script to a specified email address when CPU usage threshold is exceeded (High CPU trigger type).

The following commands are recommended for collecting debug information, but they are not the only options. Other commands can be used.

diagnose sys cmdb info
diagnose sys vd list | grep fib
diagnose sys top 5 20 2
diagnose sys session full-stat
diagnose sys session list | grep "\<dirty\>" –c
get system performance status
diagnose sys session full-stat
diagnose hardware sysinfo memory
diagnose sys cmdb info
diagnose sys vd list | grep fib