Fortinet black logo

Administration Guide

Execute a CLI script based on memory and CPU thresholds

Execute a CLI script based on memory and CPU thresholds

Automation stitches can be created to run a CLI script and send an email message when memory or CPU usage exceeds specified thresholds.

Note

The maximum size of the CLI script action output is 192K characters.

To define memory and CPU usage thresholds:
config system global
    set cpu-use-threshold <percent>
    set memory-use-threshold-extreme <percent>
    set memory-use-threshold-green <percent>
    set memory-use-threshold-red <percent>
end

Where:

cpu-use-threshold

Threshold at which CPU usage is reported, in percent of total possible CPU utilization (default = 90).

memory-use-threshold-extreme

Threshold at which memory usage is considered extreme, and new sessions are dropped, in percent of total RAM (default = 95).

memory-use-threshold-green

Threshold at which memory usage forces the FortiGate to exit conserve mode, in percent of total RAM (default = 82).

memory-use-threshold-red

Threshold at which memory usage forces the FortiGate to enter conserve mode, in percent of total RAM (default = 88).

Configuring a high memory usage stitch

In this example, an automation stitch is created that runs a CLI script to collect debug information, and then email the results of the script to a specified email address when the memory usage causes the FortiGate to enter conserve mode.

To create an automation stitch for high memory usage in the GUI:
  1. Go to Security Fabric > Automation and click Create New.
  2. Enter the stitch name (auto_high_memory).
  3. Configure the trigger:
    1. Click Add Trigger.
    2. Select Conserve Mode.
    3. Click Apply.
  4. Configure the CLI Script action:
    1. Click Add Action.
    2. Click Create and select CLI Script.
    3. Enter the following:

      Name

      high_memory_debug

      Script

      diagnose sys top 5 20 3
      diagnose sys session full-stat
      get system performance status
      diagnose sys session full-stat
      diagnose hardware sysinfo shm
      diagnose hardware sysinfo memory
      

      Administrator profile

      Select a profile

    4. Click OK.
    5. Select the action in the list and click Apply.
  5. Configure the Email notification action:
    1. Click Add Action.
    2. Click Create and select Email.
    3. Enter the following:

      Name

      auto_high_memory_email

      To

      Enter an email address

      Subject

      CSF stitch alert: high_memory

      Body

      %%results%%

    4. Click OK.
    5. Select the action in the list and click Apply.
  6. Click OK.
To create an automation stitch for high memory usage in the CLI:
  1. Create the automation trigger:
    config system automation-trigger
        edit "Conserve Mode"
            set event-type low-memory
        next
    end
  2. Create the automation actions:
    config system automation-action
        edit "high_memory_debug"
            set action-type cli-script
            set script "diagnose sys top 5 20 3
    diagnose sys session full-stat
    get system performance status
    diagnose sys session full-stat
    diagnose hardware sysinfo shm
    diagnose hardware sysinfo memory"
            set output-size 10
            set timeout 0	
            set accprofile "super_admin"	
        next
        edit "auto_high_memory_email"
            set action-type email
            set email-to "person@fortinet.com"
            set email-subject "CSF stitch alert: high_memory"
            set message "%%results%%"
        next
    end
  3. Create the automation stitch:
    config system automation-stitch
        edit "auto_high_memory"
            set trigger "Conserve Mode"
            config actions
                edit 1
                    set action "high_memory_debug"
                    set required enable
                next
                edit 2
                    set action "auto_high_memory_email"
                    set required enable
                next
            end
        next
    end

Results

When the FortiGate enters conserve mode due to the memory-use-threshold-red being exceeded, the GUI displays a notice, and the auto_high_memory automation stitch is triggered. This causes the CLI script to run and the script results are emailed to the specified address.

Here is sample text from the email message:

CSF stitch alert: high_memory
DoNotReply@notification.fortinet.net
Tue 05/16/2023 5:34 PM
########## script name: autod.0 ########## 
========== #1, 2023-05-16 17:34:17 ========== 
Client_Fgt $  diagnose sys top 5 20 3 
Run Time:  0 days, 0 hours and 0 minutes 61U, 0N, 6S, 33I, 0WA, 0HI, 0SI, 0ST; 1356T, 129F
       ipshelper     2601      S <    61.6     8.0    0
       ipsengine     2745      S <     4.9     8.5    0
         cmdbsvr     2528      S N     0.0     7.9    0
         cmdbsvr     2529      S       0.0     5.0    0
       scanunitd     2610      S <     0.0     3.8    0
         miglogd     2603      S       0.0     3.6    0
          cw_acd     2634      S       0.0     3.4    0
            node     2574      S       0.0     3.3    0
       forticron     2584      S       0.0     2.9    0
         miglogd     2693      S       0.0     2.8    0
         reportd     2604      S       0.0     2.5    0
          httpsd     2573      S       0.0     2.4    0

...

Configuring a high CPU usage stitch

Similar to the previous example, an automation stitch can be created that runs a CLI script to collect debug information, and then email the results of the script to a specified email address when CPU usage threshold is exceeded (High CPU trigger).

The following commands are recommended for collecting debug information, but they are not the only options. Other commands can be used.

diagnose sys cmdb info
diagnose sys vd list | grep fib
diagnose sys top 5 20 2
diagnose sys session full-stat
diagnose sys session list | grep "\<dirty\>" –c
get system performance status
diagnose sys session full-stat
diagnose hardware sysinfo memory
diagnose sys cmdb info
diagnose sys vd list | grep fib

Execute a CLI script based on memory and CPU thresholds

Automation stitches can be created to run a CLI script and send an email message when memory or CPU usage exceeds specified thresholds.

Note

The maximum size of the CLI script action output is 192K characters.

To define memory and CPU usage thresholds:
config system global
    set cpu-use-threshold <percent>
    set memory-use-threshold-extreme <percent>
    set memory-use-threshold-green <percent>
    set memory-use-threshold-red <percent>
end

Where:

cpu-use-threshold

Threshold at which CPU usage is reported, in percent of total possible CPU utilization (default = 90).

memory-use-threshold-extreme

Threshold at which memory usage is considered extreme, and new sessions are dropped, in percent of total RAM (default = 95).

memory-use-threshold-green

Threshold at which memory usage forces the FortiGate to exit conserve mode, in percent of total RAM (default = 82).

memory-use-threshold-red

Threshold at which memory usage forces the FortiGate to enter conserve mode, in percent of total RAM (default = 88).

Configuring a high memory usage stitch

In this example, an automation stitch is created that runs a CLI script to collect debug information, and then email the results of the script to a specified email address when the memory usage causes the FortiGate to enter conserve mode.

To create an automation stitch for high memory usage in the GUI:
  1. Go to Security Fabric > Automation and click Create New.
  2. Enter the stitch name (auto_high_memory).
  3. Configure the trigger:
    1. Click Add Trigger.
    2. Select Conserve Mode.
    3. Click Apply.
  4. Configure the CLI Script action:
    1. Click Add Action.
    2. Click Create and select CLI Script.
    3. Enter the following:

      Name

      high_memory_debug

      Script

      diagnose sys top 5 20 3
      diagnose sys session full-stat
      get system performance status
      diagnose sys session full-stat
      diagnose hardware sysinfo shm
      diagnose hardware sysinfo memory
      

      Administrator profile

      Select a profile

    4. Click OK.
    5. Select the action in the list and click Apply.
  5. Configure the Email notification action:
    1. Click Add Action.
    2. Click Create and select Email.
    3. Enter the following:

      Name

      auto_high_memory_email

      To

      Enter an email address

      Subject

      CSF stitch alert: high_memory

      Body

      %%results%%

    4. Click OK.
    5. Select the action in the list and click Apply.
  6. Click OK.
To create an automation stitch for high memory usage in the CLI:
  1. Create the automation trigger:
    config system automation-trigger
        edit "Conserve Mode"
            set event-type low-memory
        next
    end
  2. Create the automation actions:
    config system automation-action
        edit "high_memory_debug"
            set action-type cli-script
            set script "diagnose sys top 5 20 3
    diagnose sys session full-stat
    get system performance status
    diagnose sys session full-stat
    diagnose hardware sysinfo shm
    diagnose hardware sysinfo memory"
            set output-size 10
            set timeout 0	
            set accprofile "super_admin"	
        next
        edit "auto_high_memory_email"
            set action-type email
            set email-to "person@fortinet.com"
            set email-subject "CSF stitch alert: high_memory"
            set message "%%results%%"
        next
    end
  3. Create the automation stitch:
    config system automation-stitch
        edit "auto_high_memory"
            set trigger "Conserve Mode"
            config actions
                edit 1
                    set action "high_memory_debug"
                    set required enable
                next
                edit 2
                    set action "auto_high_memory_email"
                    set required enable
                next
            end
        next
    end

Results

When the FortiGate enters conserve mode due to the memory-use-threshold-red being exceeded, the GUI displays a notice, and the auto_high_memory automation stitch is triggered. This causes the CLI script to run and the script results are emailed to the specified address.

Here is sample text from the email message:

CSF stitch alert: high_memory
DoNotReply@notification.fortinet.net
Tue 05/16/2023 5:34 PM
########## script name: autod.0 ########## 
========== #1, 2023-05-16 17:34:17 ========== 
Client_Fgt $  diagnose sys top 5 20 3 
Run Time:  0 days, 0 hours and 0 minutes 61U, 0N, 6S, 33I, 0WA, 0HI, 0SI, 0ST; 1356T, 129F
       ipshelper     2601      S <    61.6     8.0    0
       ipsengine     2745      S <     4.9     8.5    0
         cmdbsvr     2528      S N     0.0     7.9    0
         cmdbsvr     2529      S       0.0     5.0    0
       scanunitd     2610      S <     0.0     3.8    0
         miglogd     2603      S       0.0     3.6    0
          cw_acd     2634      S       0.0     3.4    0
            node     2574      S       0.0     3.3    0
       forticron     2584      S       0.0     2.9    0
         miglogd     2693      S       0.0     2.8    0
         reportd     2604      S       0.0     2.5    0
          httpsd     2573      S       0.0     2.4    0

...

Configuring a high CPU usage stitch

Similar to the previous example, an automation stitch can be created that runs a CLI script to collect debug information, and then email the results of the script to a specified email address when CPU usage threshold is exceeded (High CPU trigger).

The following commands are recommended for collecting debug information, but they are not the only options. Other commands can be used.

diagnose sys cmdb info
diagnose sys vd list | grep fib
diagnose sys top 5 20 2
diagnose sys session full-stat
diagnose sys session list | grep "\<dirty\>" –c
get system performance status
diagnose sys session full-stat
diagnose hardware sysinfo memory
diagnose sys cmdb info
diagnose sys vd list | grep fib