Fortinet black logo

User Guide

Custom Linux CounterMeasure actions

24.2.0
Copy Link
Copy Doc ID af1daa65-c273-11ec-9fd1-fa163e15d75b:690306
Download PDF

Custom Linux CounterMeasure actions (FortiMonitor Agent)

You can create custom CounterMeasure actions for Linux using JSON or Python. In most cases, a CounterMeasure created using JSON will be sufficient. For more advanced or complex use cases, you have the option to use Python.

Here are some basic CounterMeasure actions that you can create using JSON:

  • Information gathering commands (top, netstat, reading a log file, etc.)

  • Restarting a service or server

  • Deleting files

For advanced use cases such as the following, use Python:

  • Authentication

  • Interacting with APIs

  • Managing state over multiple steps

Create CounterMeasure actions using JSON

To create custom Linux CounterMeasures using JSON, create a new JSON file in the /usr/share/fm-agent/countermeasures directory. The command provided in the `command` key-value will be executed when the CounterMeasure is triggered. The command's output (if any) will be returned and available in the FortiMonitor Control Panel. The following example shows a JSON CounterMeasure that returns the output of a netstat command.

{
"name": "netstat",
"textkey": "info.netstat",
"description": "Gather most recent netstat output",
"max_frequency": 60,
"max_runtime": null,
"author": "testing@fortinet.com",
"wall_announce_delay": null,
"command": "netstat -ant"
}

For more details, see Implementation Reference. Keep in mind that the command will be run as the fm-agent user. Any elevated privileges will need to be configured as described in the sudo privileges section.

Create CounterMeasure actions using Python

To create your own CounterMeasure:

  1. Create a new python file in the /usr/share/fm-agent/countermeasures directory and ensure your implementation subclasses the CountermeasurePlugin class.
    A full reference is available in the Implementation Reference section. Largely, implementing the run() method and providing a few instance variables is all that is required. You can see a basic example in CounterMeasure Action metadata example.

The name of your custom class needs to end with Countermeasure.

2. Run the following command, where my_script.py is the name of your CounterMeasure script:
python3 -m py_compile my_script.py

Ensure that there are no syntax errors. If your CounterMeasure script has syntax errors, it will not be added to the control panel.

3. Rebuild your Agent metadata with the following command:
python3 /usr/bin/fm-agent/fm_agent.py --rebuild-metadata

Your CounterMeasure will then be available in the FortiMonitor control panel.

Logging

If you need to log information in your CounterMeasure plugin, you can use self.log.info("your message here"), which will log to /var/log/fm-agent/countermeasure.log

Leverage incident data

When the agent is notified that it should run a local CounterMeasure action, it also receives metadata about the underlying incident triggering the CounterMeasure. This JSON object is available to you in your code via the metadata property. For example, the below code returns the incident metadata right back to FortiMonitor.

CounterMeasure Action metadata example

from CountermeasurePlugin import CountermeasurePlugin
class TestMetaCountermeasure(CountermeasurePlugin):
name = "Test Metadata"
author = "FortiMonitor User"
textkey = "test.metadata"
description = "Returns passed metadata"
def run(self):
output = str(self.metadata)
self.save_text_output(output)
self.save_return_code(0)

This is helpful as it allows you to take action based on certain criteria, such as which application or metrics triggered the incident. The payload schema is detailed in the following table.

Key

Description

Outage

id

The ID number of the associated incident.

alert_label

Alert label of the incident/anomaly.

timestamp

UTC timestamp of when the incident/clear occurred.

severity

The severity of the outage/anomaly, either "critical" or "warning".

reasons

The reasons for the network service incidents or the details for anomalies.

Server

id

The ID number of the server experiencing the incident/clear.

server_key

The server key for the server.

fqdn

The fully qualified domain name of the server experiencing the incident/clear.

name

Name of the server experiencing the incident/clear.

tags

The tags for the server.

partner_server_id

The partner server id for the server.

Customer attributes

metrics

Services experiencing the incident/clear or resources experiencing the anomaly/clear.

metric_tags

The tags for all of the metrics involved in the outage.

resource

For resource anomalies: resources experiencing the anomaly/clear.

services

For service incident: services experiencing the incident/clear.

Limit plugin execution time

You may optionally set the maximum time a CounterMeasure plugin may run using the max_runtime property. You may supply any number of seconds, represented as an integer. If it exceeds the allotted time, the CounterMeasure driver will attempt to kill it, though it is not guaranteed

View plugins on an instance

To view all the plugins available to use on an instance, use the following command:

python /usr/bin/fm-agent/countermeasure.py list_plugins

If you're not seeing an expected plugin, ensure that it is in /usr/share/fm-agent/countermeasures.

sudo privileges

CounterMeasure plugins are executed by the fm-agent user, which is created at the time of agent installation. The fm-agent user itself does not have elevated privileges and does not require them to perform it's normal tasks. There may be times, however, when creating custom CounterMeasure plugins that you need the fm-agent user to have elevated privileges - this requires passwordless access to be configured for the fm-agent user (an example is below). As well, one out-of-the-box CounterMeasure plugin requires elevated permissions - reboot. If you attempt to run this CounterMeasure before you've configured permissions, it will fail.

Verifying sudo privileges

The Agent ships with a CLI helper method to allows you to validate sudo privileges for your CounterMeasure plugins. It will help you identify plugins that the fm-agent user does not have the proper privileges to run.

First, we need to enable switching to the fm-agent user.

  • Open /etc/passwd. At the end of the fm-agent line, remove /usr/sbin/nologin and replace it with /bin/bash

  • Save the file.

Now we can switch to the fm-agent user:

su fm-agent

Then, we can run the following command, which will highlight the plugins that require sudo privileges and whether or not those privileges are configured correctly.

python3 /usr/bin/fm-agent/countermeasure.py validate_sudo

If you've properly configured sudoers, you'll see an output like below. Otherwise, you'll see Missing Permissions.

Verifying sudo requirements for reboot
reboot:shutdown                 Pass

Implementation reference

Instance Variable

Type

Description

Required

name

String

Human-readable name for the Countermeasure, will be displayed in the control panel and alerts

Yes

author

String

Identifier of the author (recommended to be your email address)

Yes

textkey

String

Unique identifier for the countermeasure, should be lowercase letters, numbers, underscores, and periods. No spaces allowed

Yes

description

String

Description of the countermeasure, for display at command line and in the FortiMonitor control panel

Yes

wall_announce_delay

Int

How long to pause execution of the countermeasure after announcing it as a wall message. Set to None to disable wall announcements for this countermeasure

No

max_frequency

Int

The shortest allowed time between two executions of this plugin, in seconds. If less than that time has elapsed, the second execution won't be performed. Leave set to None to disable frequency checks

No

max_runtime

Int

The longest amount of time that the plugin should be allowed to run. The Countermeasures driver will attempt to kill the execution when it exceeds this time, although due to the condition that may not be guaranteed

No

sudo_requirements

[Command Lines]

List of full command lines this plugin requires sudo access for, which is used for validating sudo configurations

No


Method

Parameters

Description

Required

run

none

Execute the countermeasure action

Yes

validate

none

Method to perform validation on the plugin's setup. This is called by the command-line tool's "validate-plugins" command. Mainly used by helper subclasses that intend to have some additional properties overridden. Should return nothing if the plugin is valid, or a string describing validation issues if there are problems.

No

prepare

none

Method to be run before execution, for any initial setup or validation that the countermeasure action needs to perform

No

save_text_output

output (String)

Save countermeasure output as plain text for later publishing up to the FortiMonitor cloud

No

save_html_output

output (String)

Save countermeasure output as formatted HTML for later publishing up to the FortiMonitor cloud

No

save_return_code

return_code

Save the return code from the countermeasure execution

No

Custom Linux CounterMeasure actions (FortiMonitor Agent)

You can create custom CounterMeasure actions for Linux using JSON or Python. In most cases, a CounterMeasure created using JSON will be sufficient. For more advanced or complex use cases, you have the option to use Python.

Here are some basic CounterMeasure actions that you can create using JSON:

  • Information gathering commands (top, netstat, reading a log file, etc.)

  • Restarting a service or server

  • Deleting files

For advanced use cases such as the following, use Python:

  • Authentication

  • Interacting with APIs

  • Managing state over multiple steps

Create CounterMeasure actions using JSON

To create custom Linux CounterMeasures using JSON, create a new JSON file in the /usr/share/fm-agent/countermeasures directory. The command provided in the `command` key-value will be executed when the CounterMeasure is triggered. The command's output (if any) will be returned and available in the FortiMonitor Control Panel. The following example shows a JSON CounterMeasure that returns the output of a netstat command.

{
"name": "netstat",
"textkey": "info.netstat",
"description": "Gather most recent netstat output",
"max_frequency": 60,
"max_runtime": null,
"author": "testing@fortinet.com",
"wall_announce_delay": null,
"command": "netstat -ant"
}

For more details, see Implementation Reference. Keep in mind that the command will be run as the fm-agent user. Any elevated privileges will need to be configured as described in the sudo privileges section.

Create CounterMeasure actions using Python

To create your own CounterMeasure:

  1. Create a new python file in the /usr/share/fm-agent/countermeasures directory and ensure your implementation subclasses the CountermeasurePlugin class.
    A full reference is available in the Implementation Reference section. Largely, implementing the run() method and providing a few instance variables is all that is required. You can see a basic example in CounterMeasure Action metadata example.

The name of your custom class needs to end with Countermeasure.

2. Run the following command, where my_script.py is the name of your CounterMeasure script:
python3 -m py_compile my_script.py

Ensure that there are no syntax errors. If your CounterMeasure script has syntax errors, it will not be added to the control panel.

3. Rebuild your Agent metadata with the following command:
python3 /usr/bin/fm-agent/fm_agent.py --rebuild-metadata

Your CounterMeasure will then be available in the FortiMonitor control panel.

Logging

If you need to log information in your CounterMeasure plugin, you can use self.log.info("your message here"), which will log to /var/log/fm-agent/countermeasure.log

Leverage incident data

When the agent is notified that it should run a local CounterMeasure action, it also receives metadata about the underlying incident triggering the CounterMeasure. This JSON object is available to you in your code via the metadata property. For example, the below code returns the incident metadata right back to FortiMonitor.

CounterMeasure Action metadata example

from CountermeasurePlugin import CountermeasurePlugin
class TestMetaCountermeasure(CountermeasurePlugin):
name = "Test Metadata"
author = "FortiMonitor User"
textkey = "test.metadata"
description = "Returns passed metadata"
def run(self):
output = str(self.metadata)
self.save_text_output(output)
self.save_return_code(0)

This is helpful as it allows you to take action based on certain criteria, such as which application or metrics triggered the incident. The payload schema is detailed in the following table.

Key

Description

Outage

id

The ID number of the associated incident.

alert_label

Alert label of the incident/anomaly.

timestamp

UTC timestamp of when the incident/clear occurred.

severity

The severity of the outage/anomaly, either "critical" or "warning".

reasons

The reasons for the network service incidents or the details for anomalies.

Server

id

The ID number of the server experiencing the incident/clear.

server_key

The server key for the server.

fqdn

The fully qualified domain name of the server experiencing the incident/clear.

name

Name of the server experiencing the incident/clear.

tags

The tags for the server.

partner_server_id

The partner server id for the server.

Customer attributes

metrics

Services experiencing the incident/clear or resources experiencing the anomaly/clear.

metric_tags

The tags for all of the metrics involved in the outage.

resource

For resource anomalies: resources experiencing the anomaly/clear.

services

For service incident: services experiencing the incident/clear.

Limit plugin execution time

You may optionally set the maximum time a CounterMeasure plugin may run using the max_runtime property. You may supply any number of seconds, represented as an integer. If it exceeds the allotted time, the CounterMeasure driver will attempt to kill it, though it is not guaranteed

View plugins on an instance

To view all the plugins available to use on an instance, use the following command:

python /usr/bin/fm-agent/countermeasure.py list_plugins

If you're not seeing an expected plugin, ensure that it is in /usr/share/fm-agent/countermeasures.

sudo privileges

CounterMeasure plugins are executed by the fm-agent user, which is created at the time of agent installation. The fm-agent user itself does not have elevated privileges and does not require them to perform it's normal tasks. There may be times, however, when creating custom CounterMeasure plugins that you need the fm-agent user to have elevated privileges - this requires passwordless access to be configured for the fm-agent user (an example is below). As well, one out-of-the-box CounterMeasure plugin requires elevated permissions - reboot. If you attempt to run this CounterMeasure before you've configured permissions, it will fail.

Verifying sudo privileges

The Agent ships with a CLI helper method to allows you to validate sudo privileges for your CounterMeasure plugins. It will help you identify plugins that the fm-agent user does not have the proper privileges to run.

First, we need to enable switching to the fm-agent user.

  • Open /etc/passwd. At the end of the fm-agent line, remove /usr/sbin/nologin and replace it with /bin/bash

  • Save the file.

Now we can switch to the fm-agent user:

su fm-agent

Then, we can run the following command, which will highlight the plugins that require sudo privileges and whether or not those privileges are configured correctly.

python3 /usr/bin/fm-agent/countermeasure.py validate_sudo

If you've properly configured sudoers, you'll see an output like below. Otherwise, you'll see Missing Permissions.

Verifying sudo requirements for reboot
reboot:shutdown                 Pass

Implementation reference

Instance Variable

Type

Description

Required

name

String

Human-readable name for the Countermeasure, will be displayed in the control panel and alerts

Yes

author

String

Identifier of the author (recommended to be your email address)

Yes

textkey

String

Unique identifier for the countermeasure, should be lowercase letters, numbers, underscores, and periods. No spaces allowed

Yes

description

String

Description of the countermeasure, for display at command line and in the FortiMonitor control panel

Yes

wall_announce_delay

Int

How long to pause execution of the countermeasure after announcing it as a wall message. Set to None to disable wall announcements for this countermeasure

No

max_frequency

Int

The shortest allowed time between two executions of this plugin, in seconds. If less than that time has elapsed, the second execution won't be performed. Leave set to None to disable frequency checks

No

max_runtime

Int

The longest amount of time that the plugin should be allowed to run. The Countermeasures driver will attempt to kill the execution when it exceeds this time, although due to the condition that may not be guaranteed

No

sudo_requirements

[Command Lines]

List of full command lines this plugin requires sudo access for, which is used for validating sudo configurations

No


Method

Parameters

Description

Required

run

none

Execute the countermeasure action

Yes

validate

none

Method to perform validation on the plugin's setup. This is called by the command-line tool's "validate-plugins" command. Mainly used by helper subclasses that intend to have some additional properties overridden. Should return nothing if the plugin is valid, or a string describing validation issues if there are problems.

No

prepare

none

Method to be run before execution, for any initial setup or validation that the countermeasure action needs to perform

No

save_text_output

output (String)

Save countermeasure output as plain text for later publishing up to the FortiMonitor cloud

No

save_html_output

output (String)

Save countermeasure output as formatted HTML for later publishing up to the FortiMonitor cloud

No

save_return_code

return_code

Save the return code from the countermeasure execution

No