Fortinet black logo

Administration Guide

Execute a CLI script based on memory and CPU thresholds

Execute a CLI script based on memory and CPU thresholds

Automation stitches can be created to run a CLI script and send an email message when memory or CPU usage exceeds specified thresholds.

Note

The maximum size of the CLI script action output is 16K characters. In cases where the output exceeds 16K, the email received will contain truncated output. To avoid this, it is recommended to limit the number of commands per script.

To define memory and CPU usage thresholds:
config system global
    set cpu-use-threshold <percent>
    set memory-use-threshold-extreme <percent>
    set memory-use-threshold-green <percent>
    set memory-use-threshold-red <percent>
end

Where:

cpu-use-threshold

Threshold at which CPU usage is reported, in percent of total possible CPU utilization (default = 90).

memory-use-threshold-extreme

Threshold at which memory usage is considered extreme, and new sessions are dropped, in percent of total RAM (default = 95).

memory-use-threshold-green

Threshold at which memory usage forces the FortiGate to exit conserve mode, in percent of total RAM (default = 82).

memory-use-threshold-red

Threshold at which memory usage forces the FortiGate to enter conserve mode, in percent of total RAM (default = 88).

Configuring a high memory usage stitch

In this example, an automation stitch is created that runs two CLI scripts to collect debug information, and then two email messages will be received with CLI output to a specified email address when the memory usage causes the FortiGate to enter conserve mode.

Note

Since the output in this example will exceed 16K, two scripts are used. The CLI scripts are run sequentially, and an email is sent out each time a script runs.

To create an automation stitch for high memory usage in the GUI:
  1. Go to Security Fabric > Automation and click Create New.
  2. Enter the stitch name (auto_high_memory).
  3. Configure the trigger:
    1. Click Add Trigger.
    2. Click Create and select Conserve Mode.
    3. Enter the name, auto_high_memory.
    4. Click OK.
    5. Select the trigger in the list and click Apply.
  4. Configure the first CLI Script action:
    1. Click Add Action.
    2. Click Create and select CLI Script.
    3. Enter the following:

      Name

      high_memory_debug1

      Script

      diagnose sys top 5 20 5
      diagnose sys session full-stat
      get system performance status
      

      Administrator profile

      Select a profile

    4. Click OK.
    5. Select the action in the list and click Apply.
  5. Configure the first Email notification action:
    1. Click Add Action.
    2. Click Create and select Email.
    3. Enter the following:

      Name

      auto_high_memory_email1

      To

      Enter an email address

      Subject

      CSF stitch alert: high_memory1

      Body

      %%results%%

    4. Click OK.
    5. Select the action in the list and click Apply.
  6. Configure the second CLI Script action:
    1. Click Add Action.
    2. Click Create and select CLI Script.
    3. Enter the following:

      Name

      high_memory_debug2

      Script

      diagnose sys session full-stat
      diagnose hardware sysinfo shm
      diagnose hardware sysinfo memory
      

      Administrator profile

      Select a profile

    4. Click OK.
    5. Select the action in the list and click Apply.
  7. Configure the second Email notification action:
    1. Click Add Action.
    2. Click Create and select Email.
    3. Enter the following:

      Name

      auto_high_memory_email2

      To

      Enter an email address

      Subject

      CSF stitch alert: high_memory2

      Body

      %%results%%

    4. Click OK.
    5. Select the action in the list and click Apply.
  8. Click OK.
To create an automation stitch for high memory usage in the CLI:
  1. Create the automation trigger:
    config system automation-trigger
        edit "auto_high_memory"
            set event-type low-memory
        next
    end
  2. Create the automation actions:
    config system automation-action
        edit "high_memory_debug1"
            set action-type cli-script
            set script "diagnose sys top 5 20 5
    diagnose sys session full-stat 
    get system performance status"
            set output-size 10
            set timeout 0	
            set accprofile "super_admin"	
        next
        edit "auto_high_memory_email1"
            set action-type email
            set email-to "person@fortinet.com"
            set email-subject "CSF stitch alert: high_memory1"
            set message "%%results%%"
        next
           edit "high_memory_debug2"
            set action-type cli-script
            set script "diagnose sys session full-stat
    diagnose hardware sysinfo shm
    diagnose hardware sysinfo memory"
            set accprofile "super_admin"
        next
        edit "auto_high_memory_email2"
            set action-type email
            set email-to "person@fortinet.com"
            set email-subject "CSF stitch alert: high_memory2"
            set message "%%results%%"
        next
    end
  3. Create the automation stitch:
    config system automation-stitch
        edit "auto_high_memory"
            set trigger "auto_high_memory"
            config actions
                edit 1
                    set action "high_memory_debug1"
                    set required enable
                next
                edit 2
                    set action "auto_high_memory_email1"
                    set required enable
                next
                edit 3
                    set action "high_memory_debug2"
                    set required enable
                next
                edit 4
                    set action "auto_high_memory_email2"
                    set required enable
                next
            end
        next
    end

Results

When the FortiGate enters conserve mode due to the memory-use-threshold-red being exceeded, the GUI displays a notice, and the auto_high_memory automation stitch is triggered. This causes the CLI scripts to run and the scripts' results are emailed to the specified address.

Here is sample text from the email message:

CSF stitch alert: high_memory1
DoNotReply@notification.fortinet.net
Tue 05/16/2023 5:34 PM
########## script name: autod.0 ########## 
========== #1, 2023-05-16 17:34:17 ========== 
Client_Fgt $  diagnose sys top 5 20 3 
Run Time:  0 days, 0 hours and 0 minutes 61U, 0N, 6S, 33I, 0WA, 0HI, 0SI, 0ST; 1356T, 129F
       ipshelper     2601      S <    61.6     8.0    0
       ipsengine     2745      S <     4.9     8.5    0
         cmdbsvr     2528      S N     0.0     7.9    0
         cmdbsvr     2529      S       0.0     5.0    0
       scanunitd     2610      S <     0.0     3.8    0
         miglogd     2603      S       0.0     3.6    0
          cw_acd     2634      S       0.0     3.4    0
            node     2574      S       0.0     3.3    0
       forticron     2584      S       0.0     2.9    0
         miglogd     2693      S       0.0     2.8    0
         reportd     2604      S       0.0     2.5    0
          httpsd     2573      S       0.0     2.4    0
...

Configuring a high CPU usage stitch

Similar to the previous example, an automation stitch can be created that runs a CLI script to collect debug information, and then email the results of the script to a specified email address when CPU usage threshold is exceeded (High CPU trigger type).

The following commands are recommended for collecting debug information, but they are not the only options. Other commands can be used.

diagnose sys cmdb info
diagnose sys vd list | grep fib
diagnose sys top 5 20 2
diagnose sys session full-stat
diagnose sys session list | grep "\<dirty\>" –c
get system performance status
diagnose sys session full-stat
diagnose hardware sysinfo memory
diagnose sys cmdb info
diagnose sys vd list | grep fib

Execute a CLI script based on memory and CPU thresholds

Automation stitches can be created to run a CLI script and send an email message when memory or CPU usage exceeds specified thresholds.

Note

The maximum size of the CLI script action output is 16K characters. In cases where the output exceeds 16K, the email received will contain truncated output. To avoid this, it is recommended to limit the number of commands per script.

To define memory and CPU usage thresholds:
config system global
    set cpu-use-threshold <percent>
    set memory-use-threshold-extreme <percent>
    set memory-use-threshold-green <percent>
    set memory-use-threshold-red <percent>
end

Where:

cpu-use-threshold

Threshold at which CPU usage is reported, in percent of total possible CPU utilization (default = 90).

memory-use-threshold-extreme

Threshold at which memory usage is considered extreme, and new sessions are dropped, in percent of total RAM (default = 95).

memory-use-threshold-green

Threshold at which memory usage forces the FortiGate to exit conserve mode, in percent of total RAM (default = 82).

memory-use-threshold-red

Threshold at which memory usage forces the FortiGate to enter conserve mode, in percent of total RAM (default = 88).

Configuring a high memory usage stitch

In this example, an automation stitch is created that runs two CLI scripts to collect debug information, and then two email messages will be received with CLI output to a specified email address when the memory usage causes the FortiGate to enter conserve mode.

Note

Since the output in this example will exceed 16K, two scripts are used. The CLI scripts are run sequentially, and an email is sent out each time a script runs.

To create an automation stitch for high memory usage in the GUI:
  1. Go to Security Fabric > Automation and click Create New.
  2. Enter the stitch name (auto_high_memory).
  3. Configure the trigger:
    1. Click Add Trigger.
    2. Click Create and select Conserve Mode.
    3. Enter the name, auto_high_memory.
    4. Click OK.
    5. Select the trigger in the list and click Apply.
  4. Configure the first CLI Script action:
    1. Click Add Action.
    2. Click Create and select CLI Script.
    3. Enter the following:

      Name

      high_memory_debug1

      Script

      diagnose sys top 5 20 5
      diagnose sys session full-stat
      get system performance status
      

      Administrator profile

      Select a profile

    4. Click OK.
    5. Select the action in the list and click Apply.
  5. Configure the first Email notification action:
    1. Click Add Action.
    2. Click Create and select Email.
    3. Enter the following:

      Name

      auto_high_memory_email1

      To

      Enter an email address

      Subject

      CSF stitch alert: high_memory1

      Body

      %%results%%

    4. Click OK.
    5. Select the action in the list and click Apply.
  6. Configure the second CLI Script action:
    1. Click Add Action.
    2. Click Create and select CLI Script.
    3. Enter the following:

      Name

      high_memory_debug2

      Script

      diagnose sys session full-stat
      diagnose hardware sysinfo shm
      diagnose hardware sysinfo memory
      

      Administrator profile

      Select a profile

    4. Click OK.
    5. Select the action in the list and click Apply.
  7. Configure the second Email notification action:
    1. Click Add Action.
    2. Click Create and select Email.
    3. Enter the following:

      Name

      auto_high_memory_email2

      To

      Enter an email address

      Subject

      CSF stitch alert: high_memory2

      Body

      %%results%%

    4. Click OK.
    5. Select the action in the list and click Apply.
  8. Click OK.
To create an automation stitch for high memory usage in the CLI:
  1. Create the automation trigger:
    config system automation-trigger
        edit "auto_high_memory"
            set event-type low-memory
        next
    end
  2. Create the automation actions:
    config system automation-action
        edit "high_memory_debug1"
            set action-type cli-script
            set script "diagnose sys top 5 20 5
    diagnose sys session full-stat 
    get system performance status"
            set output-size 10
            set timeout 0	
            set accprofile "super_admin"	
        next
        edit "auto_high_memory_email1"
            set action-type email
            set email-to "person@fortinet.com"
            set email-subject "CSF stitch alert: high_memory1"
            set message "%%results%%"
        next
           edit "high_memory_debug2"
            set action-type cli-script
            set script "diagnose sys session full-stat
    diagnose hardware sysinfo shm
    diagnose hardware sysinfo memory"
            set accprofile "super_admin"
        next
        edit "auto_high_memory_email2"
            set action-type email
            set email-to "person@fortinet.com"
            set email-subject "CSF stitch alert: high_memory2"
            set message "%%results%%"
        next
    end
  3. Create the automation stitch:
    config system automation-stitch
        edit "auto_high_memory"
            set trigger "auto_high_memory"
            config actions
                edit 1
                    set action "high_memory_debug1"
                    set required enable
                next
                edit 2
                    set action "auto_high_memory_email1"
                    set required enable
                next
                edit 3
                    set action "high_memory_debug2"
                    set required enable
                next
                edit 4
                    set action "auto_high_memory_email2"
                    set required enable
                next
            end
        next
    end

Results

When the FortiGate enters conserve mode due to the memory-use-threshold-red being exceeded, the GUI displays a notice, and the auto_high_memory automation stitch is triggered. This causes the CLI scripts to run and the scripts' results are emailed to the specified address.

Here is sample text from the email message:

CSF stitch alert: high_memory1
DoNotReply@notification.fortinet.net
Tue 05/16/2023 5:34 PM
########## script name: autod.0 ########## 
========== #1, 2023-05-16 17:34:17 ========== 
Client_Fgt $  diagnose sys top 5 20 3 
Run Time:  0 days, 0 hours and 0 minutes 61U, 0N, 6S, 33I, 0WA, 0HI, 0SI, 0ST; 1356T, 129F
       ipshelper     2601      S <    61.6     8.0    0
       ipsengine     2745      S <     4.9     8.5    0
         cmdbsvr     2528      S N     0.0     7.9    0
         cmdbsvr     2529      S       0.0     5.0    0
       scanunitd     2610      S <     0.0     3.8    0
         miglogd     2603      S       0.0     3.6    0
          cw_acd     2634      S       0.0     3.4    0
            node     2574      S       0.0     3.3    0
       forticron     2584      S       0.0     2.9    0
         miglogd     2693      S       0.0     2.8    0
         reportd     2604      S       0.0     2.5    0
          httpsd     2573      S       0.0     2.4    0
...

Configuring a high CPU usage stitch

Similar to the previous example, an automation stitch can be created that runs a CLI script to collect debug information, and then email the results of the script to a specified email address when CPU usage threshold is exceeded (High CPU trigger type).

The following commands are recommended for collecting debug information, but they are not the only options. Other commands can be used.

diagnose sys cmdb info
diagnose sys vd list | grep fib
diagnose sys top 5 20 2
diagnose sys session full-stat
diagnose sys session list | grep "\<dirty\>" –c
get system performance status
diagnose sys session full-stat
diagnose hardware sysinfo memory
diagnose sys cmdb info
diagnose sys vd list | grep fib