Execute a CLI script based on CPU and memory thresholds

Automation stitches can be created to run a CLI script and send an email message when CPU or memory usage exceeds specified thresholds.

In this example, two automation stitches are created that run a CLI script to collect debug information, and then email the results of the script to a specified email address when CPU usage threshold is exceeded or memory usage causes the FortiGate to enter conserve mode.

Automation stitches that use High CPU and Conserve Mode triggers can only be created in the CLI. Once create, they can be edited in the GUI.

To define CPU and memory usage thresholds:

config system global
    set cpu-use-threshold <percent>
    set memory-use-threshold-extreme <percent>
    set memory-use-threshold-green <percent>
    set memory-use-threshold-red <percent>
end

Where:

cpu-use-threshold	Threshold at which CPU usage is reported, in percent of total possible CPU utilization (default = 90).
memory-use-threshold-extreme	Threshold at which memory usage is considered extreme, and new sessions are dropped, in percent of total RAM (default = 95).
memory-use-threshold-green	Threshold at which memory usage forces the FortiGate to exit conserve mode, in percent of total RAM (default = 82).
memory-use-threshold-red	Threshold at which memory usage forces the FortiGate to enter conserve mode, in percent of total RAM (default = 88).

Configure the automation stitches

High CPU usage stitch

To create an automation stitch for high CPU usage:

1. Create an automation action to run a CLI script:

   config system automation-action
       edit "high_cpu_debug"
           set action-type cli-script
           set required enable
           set script "diagnose debug cli 8
   diagnose debug console timestamp enable
   diagnose debug enable
   diagnose debug crashlog read
   get system performance status
   get system session status
   diagnose sys session full-stat
   diagnose firewall iprope state
   diagnose sys flash list
   diagnose hardware sysinfo memory
   diagnose hardware sysinfo slab
   diagnose hardware sysinfo shm
   diagnose hardware deviceinfo disk
   get system arp
   diagnose ip arp list
   diagnose ip address list
   get router info routing-table all
   get router info kernel
   diagnose ip rtcache list
   diagnose sys top-summary
   diagnose sys top 9 99"
       next
   end

2. Create an automation action to send an email:

   config system automation-action
       edit "auto_high_cpu_email"
           set action-type email
           set email-to "person@fortinet.com"
           set email-subject "CSF stitch alert: high_cpu"
           set message "%%results%%"
       next
   end

3. Create an automation trigger:

   config system automation-trigger
       edit "auto_high_cpu"
           set event-type high-cpu
       next
   end

4. Create an automation stitch:

   config system automation-stitch
       edit "auto_high_cpu"
           set trigger "auto_high_cpu"
           set action "high_cpu_debug" "auto_high_cpu_email"
       next
   end

To edit the automation stitch in the GUI:

1. Go to Security Fabric > Automation.
2. Double click the auto_high_cpu stitch.

   

3. Edit the stitch as required, then click OK.

High memory usage stitch

To create an automation stitch for high memory usage:

1. Create an automation action to run a CLI script:

   config system automation-action
       edit "high_memory_debug"
           set action-type cli-script
           set required enable
           set script "diagnose debug cli 8
   diagnose debug console timestamp enable
   diagnose debug enable
   diagnose debug crashlog read
   get system performance status
   get system session status
   diagnose sys session full-stat
   diagnose firewall iprope state
   diagnose sys flash list
   diagnose hardware sysinfo memory
   diagnose hardware sysinfo slab
   diagnose hardware sysinfo shm
   diagnose hardware deviceinfo disk
   get system arp
   diagnose ip arp list
   diagnose ip address list
   get router info routing-table all
   get router info kernel
   diagnose ip rtcache list
   diagnose sys top-summary
   diagnose sys top 9 99"
       next
   end

2. Create an automation action to send an email:

   config system automation-action
       edit "auto_high_memory_email"
           set action-type email
           set email-to "person@fortinet.com"
           set email-subject "CSF stitch alert: high_memory"
           set message "%%results%%"
       next
   end

3. Create an automation trigger:

   config system automation-trigger
       edit "auto_high_memory"
           set event-type low-memory
       next
   end

4. Create an automation stitch:

   config system automation-stitch
       edit "auto_high_memory"
           set trigger "auto_high_memory"
           set action "high_memory_debug" "auto_high_memory_email"
       next
   end

To edit the automation stitch in the GUI:

1. Go to Security Fabric > Automation.
2. Double click the auto_high_memory stitch.

   

3. Edit the stitch as required, then click OK.

Results

When FortiGate enters conserve mode due to the memory-use-threshold-red being exceeded, the GUI displays a notice, and the auto_high_memory automation stitch is triggered, causing the CLI script to run and the results of the script to be emailed to the specified address.

Here is an example of the email message:

CSF stitch alert: high_memory
noreply@notification.fortinet.net
Thu 11/21/2019 11:06 AM
James Li
FGT[FGVM16TM19000026] Automation Stitch:auto_high_memory is triggered.
########## script name: autod.47 ##########
========== #1, 2019-11-21 11:07:24 ==========
FGVM16TM19000026 $  diag deb cli 8
Debug messages will be on for 25 minutes.
FGVM16TM19000026 $  diag deb console timestamp enable
FGVM16TM19000026 $  diag deb enable
FGVM16TM19000026 $  diag deb crashlog read
1: 2019-08-08 11:35:25 the killed daemon is /bin/dhcpcd: status=0x0
2: 2019-08-08 17:52:47 the killed daemon is /bin/pyfcgid: status=0x0
3: 2019-08-23 11:32:31 from=license status=INVALID
4: 2019-08-23 11:32:32 from=license status=INVALID
5: 2019-11-21 09:53:31 from=license status=VALID
...