Fortinet black logo

Administration Guide

Execute a CLI script based on CPU and memory thresholds

Execute a CLI script based on CPU and memory thresholds

Automation stitches can be created to run a CLI script and send an email message when CPU or memory usage exceeds specified thresholds.

In this example, two automation stitches are created that run a CLI script to collect debug information, and then email the results of the script to a specified email address when the CPU usage threshold is exceeded, or memory usage causes the FortiGate to enter conserve mode.

Note

The maximum size of the CLI script action output is 16K characters.

To define CPU and memory usage thresholds:
config system global
    set cpu-use-threshold <percent>
    set memory-use-threshold-extreme <percent>
    set memory-use-threshold-green <percent>
    set memory-use-threshold-red <percent>
end

Where:

cpu-use-threshold

Threshold at which CPU usage is reported, in percent of total possible CPU utilization (default = 90).

memory-use-threshold-extreme

Threshold at which memory usage is considered extreme, and new sessions are dropped, in percent of total RAM (default = 95).

memory-use-threshold-green

Threshold at which memory usage forces the FortiGate to exit conserve mode, in percent of total RAM (default = 82).

memory-use-threshold-red

Threshold at which memory usage forces the FortiGate to enter conserve mode, in percent of total RAM (default = 88).

Configuring the automation stitches

High CPU usage stitch

To create an automation stitch for high CPU usage in the GUI:
  1. Go to Security Fabric > Automation and click Create New.
  2. Enter the stitch name (auto_high_cpu).
  3. Configure the trigger:
    1. Click Add Trigger.
    2. Click Create and select High CPU.
    3. Enter the name, auto_high_cpu.
    4. Click OK.
    5. Select the trigger in the list and click Apply.
  4. Configure the CLI Script action:
    1. Click Add Action.
    2. Click Create and select CLI Script.
    3. Enter the following:

      Name

      high_cpu_debug

      Script

      diagnose debug cli 8
      diagnose debug console timestamp enable
      diagnose debug enable
      diagnose debug crashlog read
      get system performance status
      get system session status
      diagnose sys session full-stat
      diagnose firewall iprope state
      diagnose sys flash list
      diagnose hardware sysinfo memory
      diagnose hardware sysinfo slab
      diagnose hardware sysinfo shm
      diagnose hardware deviceinfo disk
      get system arp
      diagnose ip arp list
      diagnose ip address list
      get router info routing-table all
      get router info kernel
      diagnose ip rtcache list
      diagnose sys top-summary
      diagnose sys top 9 99

      Administrator profile

      Select a profile

    4. Click OK.
    5. Select the action in the list and click Apply.
  5. Configure the Email notification action:
    1. Click Add Action.
    2. Click Create and select Email.
    3. Enter the following:

      Name

      auto_high_cpu_email

      To

      Enter an email address

      Subject

      CSF stitch alert: high_cpu

      Body

      %%results%%

    4. Click OK.
    5. Select the action in the list and click Apply.
  6. Click OK.
To create an automation stitch for high CPU usage in the CLI:
  1. Create the automation trigger:
    config system automation-trigger
        edit "auto_high_cpu"
            set event-type high-cpu
        next
    end
  2. Create the automation actions:
    config system automation-action
        edit "high_cpu_debug"
            set action-type cli-script
            set script "diagnose debug cli 8
    diagnose debug console timestamp enable
    diagnose debug enable
    diagnose debug crashlog read
    get system performance status
    get system session status
    diagnose sys session full-stat
    diagnose firewall iprope state
    diagnose sys flash list
    diagnose hardware sysinfo memory
    diagnose hardware sysinfo slab
    diagnose hardware sysinfo shm
    diagnose hardware deviceinfo disk
    get system arp
    diagnose ip arp list
    diagnose ip address list
    get router info routing-table all
    get router info kernel
    diagnose ip rtcache list
    diagnose sys top-summary
    diagnose sys top 9 99"
            set output-size 10
            set timeout 0
            set accprofile "super_admin"
        next
        edit "auto_high_cpu_email"
            set action-type email
            set email-to "person@fortinet.com"
            set email-subject "CSF stitch alert: high_cpu"
            set message "%%results%%"
        next
    end
  3. Create the automation stitch:
    config system automation-stitch
        edit "auto_high_cpu"
            set trigger "auto_high_cpu"
            config actions
                edit 1
                    set action "high_cpu_debug"
                    set required enable
                next
                edit 2
                    set action "auto_high_cpu_email"
                    set required enable
                next
            end
        next
    end

High memory usage stitch

To create an automation stitch for high memory usage in the GUI:
  1. Go to Security Fabric > Automation and click Create New.
  2. Enter the stitch name (auto_high_memory).
  3. Configure the trigger:
    1. Click Add Trigger.
    2. Click Create and select Conserve Mode.
    3. Enter the name, auto_high_memory.
    4. Click OK.
    5. Select the trigger in the list and click Apply.
  4. Configure the CLI Script action:
    1. Click Add Action.
    2. Click Create and select CLI Script.
    3. Enter the following:

      Name

      high_memory_debug

      Script

      diagnose debug cli 8
      diagnose debug console timestamp enable
      diagnose debug enable
      diagnose debug crashlog read
      get system performance status
      get system session status
      diagnose sys session full-stat
      diagnose firewall iprope state
      diagnose sys flash list
      diagnose hardware sysinfo memory
      diagnose hardware sysinfo slab
      diagnose hardware sysinfo shm
      diagnose hardware deviceinfo disk
      get system arp
      diagnose ip arp list
      diagnose ip address list
      get router info routing-table all
      get router info kernel
      diagnose ip rtcache list
      diagnose sys top-summary
      diagnose sys top 9 99

      Administrator profile

      Select a profile

    4. Click OK.
    5. Select the action in the list and click Apply.
  5. Configure the Email notification action:
    1. Click Add Action.
    2. Click Create and select Email.
    3. Enter the following:

      Name

      auto_high_memory_email

      To

      Enter an email address

      Subject

      CSF stitch alert: high_memory

      Body

      %%results%%

    4. Click OK.
    5. Select the action in the list and click Apply.
  6. Click OK.
To create an automation stitch for high memory usage in the CLI:
  1. Create the automation trigger:
    config system automation-trigger
        edit "auto_high_memory"
            set event-type low-memory
        next
    end
  2. Create the automation actions:
    config system automation-action
        edit "high_memory_debug"
            set action-type cli-script
            set script "diagnose debug cli 8
    diagnose debug console timestamp enable
    diagnose debug enable
    diagnose debug crashlog read
    get system performance status
    get system session status
    diagnose sys session full-stat
    diagnose firewall iprope state
    diagnose sys flash list
    diagnose hardware sysinfo memory
    diagnose hardware sysinfo slab
    diagnose hardware sysinfo shm
    diagnose hardware deviceinfo disk
    get system arp
    diagnose ip arp list
    diagnose ip address list
    get router info routing-table all
    get router info kernel
    diagnose ip rtcache list
    diagnose sys top-summary
    diagnose sys top 9 99"
            set output-size 10
            set timeout 0	
            set accprofile "super_admin"	
        next
        edit "auto_high_memory_email"
            set action-type email
            set email-to "person@fortinet.com"
            set email-subject "CSF stitch alert: high_memory"
            set message "%%results%%"
        next
    end
  3. Create the automation stitch:
    config system automation-stitch
        edit "auto_high_memory"
            set trigger "auto_high_memory"
            config actions
                edit 1
                    set action "high_memory_debug"
                    set required enable
                next
                edit 2
                    set action "auto_high_memory_email"
                    set required enable
                next
            end
        next
    end

Results

When the FortiGate enters conserve mode due to the memory-use-threshold-red being exceeded, the GUI displays a notice, and the auto_high_memory automation stitch is triggered. This causes the CLI script to run and the script results are emailed to the specified address.

Here is sample text from the email message:

CSF stitch alert: high_memory
noreply@notification.fortinet.net
Thu 11/21/2019 11:06 AM
John Doe
FGT[FGVM16TM19000000] Automation Stitch:auto_high_memory is triggered.
########## script name: autod.47 ##########
========== #1, 2019-11-21 11:07:24 ==========
FGVM16TM19000000 $  diag deb cli 8
Debug messages will be on for 25 minutes.
FGVM16TM19000000 $  diag deb console timestamp enable
FGVM16TM19000000 $  diag deb enable
FGVM16TM19000000 $  diag deb crashlog read
1: 2019-08-08 11:35:25 the killed daemon is /bin/dhcpcd: status=0x0
2: 2019-08-08 17:52:47 the killed daemon is /bin/pyfcgid: status=0x0
3: 2019-08-23 11:32:31 from=license status=INVALID
4: 2019-08-23 11:32:32 from=license status=INVALID
5: 2019-11-21 09:53:31 from=license status=VALID
...

Execute a CLI script based on CPU and memory thresholds

Automation stitches can be created to run a CLI script and send an email message when CPU or memory usage exceeds specified thresholds.

In this example, two automation stitches are created that run a CLI script to collect debug information, and then email the results of the script to a specified email address when the CPU usage threshold is exceeded, or memory usage causes the FortiGate to enter conserve mode.

Note

The maximum size of the CLI script action output is 16K characters.

To define CPU and memory usage thresholds:
config system global
    set cpu-use-threshold <percent>
    set memory-use-threshold-extreme <percent>
    set memory-use-threshold-green <percent>
    set memory-use-threshold-red <percent>
end

Where:

cpu-use-threshold

Threshold at which CPU usage is reported, in percent of total possible CPU utilization (default = 90).

memory-use-threshold-extreme

Threshold at which memory usage is considered extreme, and new sessions are dropped, in percent of total RAM (default = 95).

memory-use-threshold-green

Threshold at which memory usage forces the FortiGate to exit conserve mode, in percent of total RAM (default = 82).

memory-use-threshold-red

Threshold at which memory usage forces the FortiGate to enter conserve mode, in percent of total RAM (default = 88).

Configuring the automation stitches

High CPU usage stitch

To create an automation stitch for high CPU usage in the GUI:
  1. Go to Security Fabric > Automation and click Create New.
  2. Enter the stitch name (auto_high_cpu).
  3. Configure the trigger:
    1. Click Add Trigger.
    2. Click Create and select High CPU.
    3. Enter the name, auto_high_cpu.
    4. Click OK.
    5. Select the trigger in the list and click Apply.
  4. Configure the CLI Script action:
    1. Click Add Action.
    2. Click Create and select CLI Script.
    3. Enter the following:

      Name

      high_cpu_debug

      Script

      diagnose debug cli 8
      diagnose debug console timestamp enable
      diagnose debug enable
      diagnose debug crashlog read
      get system performance status
      get system session status
      diagnose sys session full-stat
      diagnose firewall iprope state
      diagnose sys flash list
      diagnose hardware sysinfo memory
      diagnose hardware sysinfo slab
      diagnose hardware sysinfo shm
      diagnose hardware deviceinfo disk
      get system arp
      diagnose ip arp list
      diagnose ip address list
      get router info routing-table all
      get router info kernel
      diagnose ip rtcache list
      diagnose sys top-summary
      diagnose sys top 9 99

      Administrator profile

      Select a profile

    4. Click OK.
    5. Select the action in the list and click Apply.
  5. Configure the Email notification action:
    1. Click Add Action.
    2. Click Create and select Email.
    3. Enter the following:

      Name

      auto_high_cpu_email

      To

      Enter an email address

      Subject

      CSF stitch alert: high_cpu

      Body

      %%results%%

    4. Click OK.
    5. Select the action in the list and click Apply.
  6. Click OK.
To create an automation stitch for high CPU usage in the CLI:
  1. Create the automation trigger:
    config system automation-trigger
        edit "auto_high_cpu"
            set event-type high-cpu
        next
    end
  2. Create the automation actions:
    config system automation-action
        edit "high_cpu_debug"
            set action-type cli-script
            set script "diagnose debug cli 8
    diagnose debug console timestamp enable
    diagnose debug enable
    diagnose debug crashlog read
    get system performance status
    get system session status
    diagnose sys session full-stat
    diagnose firewall iprope state
    diagnose sys flash list
    diagnose hardware sysinfo memory
    diagnose hardware sysinfo slab
    diagnose hardware sysinfo shm
    diagnose hardware deviceinfo disk
    get system arp
    diagnose ip arp list
    diagnose ip address list
    get router info routing-table all
    get router info kernel
    diagnose ip rtcache list
    diagnose sys top-summary
    diagnose sys top 9 99"
            set output-size 10
            set timeout 0
            set accprofile "super_admin"
        next
        edit "auto_high_cpu_email"
            set action-type email
            set email-to "person@fortinet.com"
            set email-subject "CSF stitch alert: high_cpu"
            set message "%%results%%"
        next
    end
  3. Create the automation stitch:
    config system automation-stitch
        edit "auto_high_cpu"
            set trigger "auto_high_cpu"
            config actions
                edit 1
                    set action "high_cpu_debug"
                    set required enable
                next
                edit 2
                    set action "auto_high_cpu_email"
                    set required enable
                next
            end
        next
    end

High memory usage stitch

To create an automation stitch for high memory usage in the GUI:
  1. Go to Security Fabric > Automation and click Create New.
  2. Enter the stitch name (auto_high_memory).
  3. Configure the trigger:
    1. Click Add Trigger.
    2. Click Create and select Conserve Mode.
    3. Enter the name, auto_high_memory.
    4. Click OK.
    5. Select the trigger in the list and click Apply.
  4. Configure the CLI Script action:
    1. Click Add Action.
    2. Click Create and select CLI Script.
    3. Enter the following:

      Name

      high_memory_debug

      Script

      diagnose debug cli 8
      diagnose debug console timestamp enable
      diagnose debug enable
      diagnose debug crashlog read
      get system performance status
      get system session status
      diagnose sys session full-stat
      diagnose firewall iprope state
      diagnose sys flash list
      diagnose hardware sysinfo memory
      diagnose hardware sysinfo slab
      diagnose hardware sysinfo shm
      diagnose hardware deviceinfo disk
      get system arp
      diagnose ip arp list
      diagnose ip address list
      get router info routing-table all
      get router info kernel
      diagnose ip rtcache list
      diagnose sys top-summary
      diagnose sys top 9 99

      Administrator profile

      Select a profile

    4. Click OK.
    5. Select the action in the list and click Apply.
  5. Configure the Email notification action:
    1. Click Add Action.
    2. Click Create and select Email.
    3. Enter the following:

      Name

      auto_high_memory_email

      To

      Enter an email address

      Subject

      CSF stitch alert: high_memory

      Body

      %%results%%

    4. Click OK.
    5. Select the action in the list and click Apply.
  6. Click OK.
To create an automation stitch for high memory usage in the CLI:
  1. Create the automation trigger:
    config system automation-trigger
        edit "auto_high_memory"
            set event-type low-memory
        next
    end
  2. Create the automation actions:
    config system automation-action
        edit "high_memory_debug"
            set action-type cli-script
            set script "diagnose debug cli 8
    diagnose debug console timestamp enable
    diagnose debug enable
    diagnose debug crashlog read
    get system performance status
    get system session status
    diagnose sys session full-stat
    diagnose firewall iprope state
    diagnose sys flash list
    diagnose hardware sysinfo memory
    diagnose hardware sysinfo slab
    diagnose hardware sysinfo shm
    diagnose hardware deviceinfo disk
    get system arp
    diagnose ip arp list
    diagnose ip address list
    get router info routing-table all
    get router info kernel
    diagnose ip rtcache list
    diagnose sys top-summary
    diagnose sys top 9 99"
            set output-size 10
            set timeout 0	
            set accprofile "super_admin"	
        next
        edit "auto_high_memory_email"
            set action-type email
            set email-to "person@fortinet.com"
            set email-subject "CSF stitch alert: high_memory"
            set message "%%results%%"
        next
    end
  3. Create the automation stitch:
    config system automation-stitch
        edit "auto_high_memory"
            set trigger "auto_high_memory"
            config actions
                edit 1
                    set action "high_memory_debug"
                    set required enable
                next
                edit 2
                    set action "auto_high_memory_email"
                    set required enable
                next
            end
        next
    end

Results

When the FortiGate enters conserve mode due to the memory-use-threshold-red being exceeded, the GUI displays a notice, and the auto_high_memory automation stitch is triggered. This causes the CLI script to run and the script results are emailed to the specified address.

Here is sample text from the email message:

CSF stitch alert: high_memory
noreply@notification.fortinet.net
Thu 11/21/2019 11:06 AM
John Doe
FGT[FGVM16TM19000000] Automation Stitch:auto_high_memory is triggered.
########## script name: autod.47 ##########
========== #1, 2019-11-21 11:07:24 ==========
FGVM16TM19000000 $  diag deb cli 8
Debug messages will be on for 25 minutes.
FGVM16TM19000000 $  diag deb console timestamp enable
FGVM16TM19000000 $  diag deb enable
FGVM16TM19000000 $  diag deb crashlog read
1: 2019-08-08 11:35:25 the killed daemon is /bin/dhcpcd: status=0x0
2: 2019-08-08 17:52:47 the killed daemon is /bin/pyfcgid: status=0x0
3: 2019-08-23 11:32:31 from=license status=INVALID
4: 2019-08-23 11:32:32 from=license status=INVALID
5: 2019-11-21 09:53:31 from=license status=VALID
...