Fortinet Document Library

Version:

Version:


Table of Contents

Cookbook

Download PDF
Copy Link

Execute a CLI script based on CPU and memory thresholds

Automation stitches can be created to run a CLI script and send an email message when CPU or memory usage exceeds specified thresholds.

In this example, two automation stitches are created that run a CLI script to collect debug information, and then email the results of the script to a specified email address when CPU usage threshold is exceeded or memory usage causes the FortiGate to enter conserve mode.

Note

Automation stitches that use High CPU and Conserve Mode triggers can only be created in the CLI. Once create, they can be edited in the GUI.

To define CPU and memory usage thresholds:
config system global
    set cpu-use-threshold <percent>
    set memory-use-threshold-extreme <percent>
    set memory-use-threshold-green <percent>
    set memory-use-threshold-red <percent>
end

Where:

cpu-use-threshold

Threshold at which CPU usage is reported, in percent of total possible CPU utilization (default = 90).

memory-use-threshold-extreme

Threshold at which memory usage is considered extreme, and new sessions are dropped, in percent of total RAM (default = 95).

memory-use-threshold-green

Threshold at which memory usage forces the FortiGate to exit conserve mode, in percent of total RAM (default = 82).

memory-use-threshold-red

Threshold at which memory usage forces the FortiGate to enter conserve mode, in percent of total RAM (default = 88).

Configure the automation stitches

High CPU usage stitch

To create an automation stitch for high CPU usage:
  1. Create an automation action to run a CLI script:
    config system automation-action
        edit "high_cpu_debug"
            set action-type cli-script
            set required enable
            set script "diagnose debug cli 8
    diagnose debug console timestamp enable
    diagnose debug enable
    diagnose debug crashlog read
    get system performance status
    get system session status
    diagnose sys session full-stat
    diagnose firewall iprope state
    diagnose sys flash list
    diagnose hardware sysinfo memory
    diagnose hardware sysinfo slab
    diagnose hardware sysinfo shm
    diagnose hardware deviceinfo disk
    get system arp
    diagnose ip arp list
    diagnose ip address list
    get router info routing-table all
    get router info kernel
    diagnose ip rtache list
    diagnose sys top-summary
    diagnose sys top 9 99"
        next
    end
  2. Create an automation action to send an email:
    config system automation-action
        edit "auto_high_cpu_email"
            set action-type email
            set email-to "person@fortinet.com"
            set email-subject "CSF stitch alert: high_cpu"
            set email-body "%%results%%"
        next
    end
  3. Create an automation trigger:
    config system automation-trigger
        edit "auto_high_cpu"
            set event-type high-cpu
        next
    end
  4. Create an automation stitch:
    config system automation-stitch
        edit "auto_high_cpu"
            set trigger "auto_high_cpu"
            set action "high_cpu_debug" "auto_high_cpu_email"
        next
    end
To edit the automation stitch in the GUI:
  1. Go to Security Fabric > Automation.
  2. Double click on the auto_high_cpu stitch.

  3. Edit the stitch as required, then click OK.

High memory usage stitch

To create an automation stitch for high memory usage:
  1. Create an automation action to run a CLI script:
    config system automation-action
        edit "high_memory_debug"
            set action-type cli-script
            set required enable
            set script "diagnose debug cli 8
    diagnose debug console timestamp enable
    diagnose debug enable
    diagnose debug crashlog read
    get system performance status
    get system session status
    diagnose sys session full-stat
    diagnose firewall iprope state
    diagnose sys flash list
    diagnose hardware sysinfo memory
    diagnose hardware sysinfo slab
    diagnose hardware sysinfo shm
    diagnose hardware deviceinfo disk
    get system arp
    diagnose ip arp list
    diagnose ip address list
    get router info routing-table all
    get router info kernel
    diagnose ip rtache list
    diagnose sys top-summary
    diagnose sys top 9 99"
        next
    end
  2. Create an automation action to send an email:
    config system automation-action
        edit "auto_high_memory_email"
            set action-type email
            set email-to "person@fortinet.com"
            set email-subject "CSF stitch alert: high_memory"
            set email-body "%%results%%"
        next
    end
  3. Create an automation trigger:
    config system automation-trigger
        edit "auto_high_memory"
            set event-type low-memory
        next
    end
  4. Create an automation stitch:
    config system automation-stitch
        edit "auto_high_memory"
            set trigger "auto_high_memory"
            set action "high_memory_debug" "auto_high_memory_email"
        next
    end
To edit the automation stitch in the GUI:
  1. Go to Security Fabric > Automation.
  2. Double click on the auto_high_memory stitch.

  3. Edit the stitch as required, then click OK.

Results

When FortiGate enters conserve mode due to the memory-use-threshold-red being exceeded, the GUI displays a notice, and the auto_high_memory automation stitch is triggered, causing the CLI script to run and the results of the script to be emailed to the specified address.

Here is an example of the email message:

CSF stitch alert: high_memory
noreply@notification.fortinet.net
Thu 11/21/2019 11:06 AM
James Li
FGT[FGVM16TM19000026] Automation Stitch:auto_high_memory is triggered.
########## script name: autod.47 ##########
========== #1, 2019-11-21 11:07:24 ==========
FGVM16TM19000026 $  diag deb cli 8
Debug messages will be on for 25 minutes.
FGVM16TM19000026 $  diag deb console timestamp enable
FGVM16TM19000026 $  diag deb enable
FGVM16TM19000026 $  diag deb crashlog read
1: 2019-08-08 11:35:25 the killed daemon is /bin/dhcpcd: status=0x0
2: 2019-08-08 17:52:47 the killed daemon is /bin/pyfcgid: status=0x0
3: 2019-08-23 11:32:31 from=license status=INVALID
4: 2019-08-23 11:32:32 from=license status=INVALID
5: 2019-11-21 09:53:31 from=license status=VALID
...

Execute a CLI script based on CPU and memory thresholds

Automation stitches can be created to run a CLI script and send an email message when CPU or memory usage exceeds specified thresholds.

In this example, two automation stitches are created that run a CLI script to collect debug information, and then email the results of the script to a specified email address when CPU usage threshold is exceeded or memory usage causes the FortiGate to enter conserve mode.

Note

Automation stitches that use High CPU and Conserve Mode triggers can only be created in the CLI. Once create, they can be edited in the GUI.

To define CPU and memory usage thresholds:
config system global
    set cpu-use-threshold <percent>
    set memory-use-threshold-extreme <percent>
    set memory-use-threshold-green <percent>
    set memory-use-threshold-red <percent>
end

Where:

cpu-use-threshold

Threshold at which CPU usage is reported, in percent of total possible CPU utilization (default = 90).

memory-use-threshold-extreme

Threshold at which memory usage is considered extreme, and new sessions are dropped, in percent of total RAM (default = 95).

memory-use-threshold-green

Threshold at which memory usage forces the FortiGate to exit conserve mode, in percent of total RAM (default = 82).

memory-use-threshold-red

Threshold at which memory usage forces the FortiGate to enter conserve mode, in percent of total RAM (default = 88).

Configure the automation stitches

High CPU usage stitch

To create an automation stitch for high CPU usage:
  1. Create an automation action to run a CLI script:
    config system automation-action
        edit "high_cpu_debug"
            set action-type cli-script
            set required enable
            set script "diagnose debug cli 8
    diagnose debug console timestamp enable
    diagnose debug enable
    diagnose debug crashlog read
    get system performance status
    get system session status
    diagnose sys session full-stat
    diagnose firewall iprope state
    diagnose sys flash list
    diagnose hardware sysinfo memory
    diagnose hardware sysinfo slab
    diagnose hardware sysinfo shm
    diagnose hardware deviceinfo disk
    get system arp
    diagnose ip arp list
    diagnose ip address list
    get router info routing-table all
    get router info kernel
    diagnose ip rtache list
    diagnose sys top-summary
    diagnose sys top 9 99"
        next
    end
  2. Create an automation action to send an email:
    config system automation-action
        edit "auto_high_cpu_email"
            set action-type email
            set email-to "person@fortinet.com"
            set email-subject "CSF stitch alert: high_cpu"
            set email-body "%%results%%"
        next
    end
  3. Create an automation trigger:
    config system automation-trigger
        edit "auto_high_cpu"
            set event-type high-cpu
        next
    end
  4. Create an automation stitch:
    config system automation-stitch
        edit "auto_high_cpu"
            set trigger "auto_high_cpu"
            set action "high_cpu_debug" "auto_high_cpu_email"
        next
    end
To edit the automation stitch in the GUI:
  1. Go to Security Fabric > Automation.
  2. Double click on the auto_high_cpu stitch.

  3. Edit the stitch as required, then click OK.

High memory usage stitch

To create an automation stitch for high memory usage:
  1. Create an automation action to run a CLI script:
    config system automation-action
        edit "high_memory_debug"
            set action-type cli-script
            set required enable
            set script "diagnose debug cli 8
    diagnose debug console timestamp enable
    diagnose debug enable
    diagnose debug crashlog read
    get system performance status
    get system session status
    diagnose sys session full-stat
    diagnose firewall iprope state
    diagnose sys flash list
    diagnose hardware sysinfo memory
    diagnose hardware sysinfo slab
    diagnose hardware sysinfo shm
    diagnose hardware deviceinfo disk
    get system arp
    diagnose ip arp list
    diagnose ip address list
    get router info routing-table all
    get router info kernel
    diagnose ip rtache list
    diagnose sys top-summary
    diagnose sys top 9 99"
        next
    end
  2. Create an automation action to send an email:
    config system automation-action
        edit "auto_high_memory_email"
            set action-type email
            set email-to "person@fortinet.com"
            set email-subject "CSF stitch alert: high_memory"
            set email-body "%%results%%"
        next
    end
  3. Create an automation trigger:
    config system automation-trigger
        edit "auto_high_memory"
            set event-type low-memory
        next
    end
  4. Create an automation stitch:
    config system automation-stitch
        edit "auto_high_memory"
            set trigger "auto_high_memory"
            set action "high_memory_debug" "auto_high_memory_email"
        next
    end
To edit the automation stitch in the GUI:
  1. Go to Security Fabric > Automation.
  2. Double click on the auto_high_memory stitch.

  3. Edit the stitch as required, then click OK.

Results

When FortiGate enters conserve mode due to the memory-use-threshold-red being exceeded, the GUI displays a notice, and the auto_high_memory automation stitch is triggered, causing the CLI script to run and the results of the script to be emailed to the specified address.

Here is an example of the email message:

CSF stitch alert: high_memory
noreply@notification.fortinet.net
Thu 11/21/2019 11:06 AM
James Li
FGT[FGVM16TM19000026] Automation Stitch:auto_high_memory is triggered.
########## script name: autod.47 ##########
========== #1, 2019-11-21 11:07:24 ==========
FGVM16TM19000026 $  diag deb cli 8
Debug messages will be on for 25 minutes.
FGVM16TM19000026 $  diag deb console timestamp enable
FGVM16TM19000026 $  diag deb enable
FGVM16TM19000026 $  diag deb crashlog read
1: 2019-08-08 11:35:25 the killed daemon is /bin/dhcpcd: status=0x0
2: 2019-08-08 17:52:47 the killed daemon is /bin/pyfcgid: status=0x0
3: 2019-08-23 11:32:31 from=license status=INVALID
4: 2019-08-23 11:32:32 from=license status=INVALID
5: 2019-11-21 09:53:31 from=license status=VALID
...