You can create custom CounterMeasure actions for Linux using JSON or Python. In most cases, a CounterMeasure created using JSON will be sufficient. For more advanced or complex use cases, you have the option to use Python.
Here are some basic CounterMeasure actions that you can create using JSON:
-
Information gathering commands (top, netstat, reading a log file, etc.)
-
Restarting a service or server
-
Deleting files
For advanced use cases such as the following, use Python:
-
Authentication
-
Interacting with APIs
-
Managing state over multiple steps
Create CounterMeasure actions using JSON
To create custom Linux CounterMeasures using JSON, create a new JSON file in the /usr/share/fm-agent/countermeasures
directory. The command provided in the `command` key-value will be executed when the CounterMeasure is triggered. The command's output (if any) will be returned and available in the FortiMonitor Control Panel. The following example shows a JSON CounterMeasure that returns the output of a netstat command.
{
"name": "netstat",
"textkey": "info.netstat",
"description": "Gather most recent netstat output",
"max_frequency": 60,
"max_runtime": null,
"author": "testing@fortinet.com",
"wall_announce_delay": null,
"command": "netstat -ant"
}
For more details, see Implementation Reference. Keep in mind that the command will be run as the fm-agent user. Any elevated privileges will need to be configured as described in the sudo privileges section.
Create CounterMeasure actions using Python
To create your own CounterMeasure:
-
Create a new python file in the
/usr/share/fm-agent/countermeasures
directory and ensure your implementation subclasses theCountermeasurePlugin
class.
A full reference is available in the Implementation Reference section. Largely, implementing the run() method and providing a few instance variables is all that is required. You can see a basic example in CounterMeasure Action metadata example.
The name of your custom class needs to end with Countermeasure.
2. Run the following command, where my_script.py is the name of your CounterMeasure script:python3 -m py_compile my_script.py
Ensure that there are no syntax errors. If your CounterMeasure script has syntax errors, it will not be added to the control panel.
3. Rebuild your Agent metadata with the following command: python3 /usr/bin/fm-agent/fm_agent.py --rebuild-metadata
Your CounterMeasure will then be available in the FortiMonitor control panel.
Logging
If you need to log information in your CounterMeasure plugin, you can use self.log.info("your message here")
, which will log to /var/log/fm-agent/countermeasure.log
Leverage incident data
When the agent is notified that it should run a local CounterMeasure action, it also receives metadata about the underlying incident triggering the CounterMeasure. This JSON object is available to you in your code via the metadata property. For example, the below code returns the incident metadata right back to FortiMonitor.
CounterMeasure Action metadata example
from CountermeasurePlugin import CountermeasurePlugin
class TestMetaCountermeasure(CountermeasurePlugin):
name = "Test Metadata"
author = "FortiMonitor User"
textkey = "test.metadata"
description = "Returns passed metadata"
def run(self):
output = str(self.metadata)
self.save_text_output(output)
self.save_return_code(0)
This is helpful as it allows you to take action based on certain criteria, such as which application or metrics triggered the incident. The payload schema is detailed in the following table.
Key |
Description |
---|---|
Outage |
|
id |
The ID number of the associated incident. |
alert_label |
Alert label of the incident/anomaly. |
timestamp |
UTC timestamp of when the incident/clear occurred. |
severity |
The severity of the outage/anomaly, either "critical" or "warning". |
reasons |
The reasons for the network service incidents or the details for anomalies. |
Server |
|
id |
The ID number of the server experiencing the incident/clear. |
server_key |
The server key for the server. |
fqdn |
The fully qualified domain name of the server experiencing the incident/clear. |
name |
Name of the server experiencing the incident/clear. |
tags |
The tags for the server. |
partner_server_id |
The partner server id for the server. |
Customer attributes |
|
metrics |
Services experiencing the incident/clear or resources experiencing the anomaly/clear. |
metric_tags |
The tags for all of the metrics involved in the outage. |
resource |
For resource anomalies: resources experiencing the anomaly/clear. |
services |
For service incident: services experiencing the incident/clear. |
Limit plugin execution time
You may optionally set the maximum time a CounterMeasure plugin may run using the max_runtime
property. You may supply any number of seconds, represented as an integer. If it exceeds the allotted time, the CounterMeasure driver will attempt to kill it, though it is not guaranteed
View plugins on an instance
To view all the plugins available to use on an instance, use the following command:
python /usr/bin/fm-agent/countermeasure.py list_plugins
If you're not seeing an expected plugin, ensure that it is in /usr/share/fm-agent/countermeasures.
sudo privileges
CounterMeasure plugins are executed by the fm-agent user, which is created at the time of agent installation. The fm-agent user itself does not have elevated privileges and does not require them to perform it's normal tasks. There may be times, however, when creating custom CounterMeasure plugins that you need the fm-agent user to have elevated privileges - this requires passwordless access to be configured for the fm-agent user (an example is below). As well, one out-of-the-box CounterMeasure plugin requires elevated permissions - reboot. If you attempt to run this CounterMeasure before you've configured permissions, it will fail.
Verifying sudo privileges
The Agent ships with a CLI helper method to allows you to validate sudo privileges for your CounterMeasure plugins. It will help you identify plugins that the fm-agent user does not have the proper privileges to run.
First, we need to enable switching to the fm-agent
user.
-
Open
/etc/passwd
. At the end of thefm-agent
line, remove/usr/sbin/nologin
and replace it with/bin/bash
-
Save the file.
Now we can switch to the fm-agent
user:
su fm-agent
Then, we can run the following command, which will highlight the plugins that require sudo privileges and whether or not those privileges are configured correctly.
python3 /usr/bin/fm-agent/countermeasure.py validate_sudo
If you've properly configured sudoers, you'll see an output like below. Otherwise, you'll see Missing Permissions.
Verifying sudo requirements for reboot
reboot:shutdown Pass
Implementation reference
Instance Variable |
Type |
Description |
Required |
---|---|---|---|
name |
String |
Human-readable name for the Countermeasure, will be displayed in the control panel and alerts |
Yes |
author |
String |
Identifier of the author (recommended to be your email address) |
Yes |
textkey |
String |
Unique identifier for the countermeasure, should be lowercase letters, numbers, underscores, and periods. No spaces allowed |
Yes |
description |
String |
Description of the countermeasure, for display at command line and in the FortiMonitor control panel |
Yes |
wall_announce_delay |
Int |
How long to pause execution of the countermeasure after announcing it as a wall message. Set to None to disable wall announcements for this countermeasure |
No |
max_frequency |
Int |
The shortest allowed time between two executions of this plugin, in seconds. If less than that time has elapsed, the second execution won't be performed. Leave set to None to disable frequency checks |
No |
max_runtime |
Int |
The longest amount of time that the plugin should be allowed to run. The Countermeasures driver will attempt to kill the execution when it exceeds this time, although due to the condition that may not be guaranteed |
No |
sudo_requirements |
[Command Lines] |
List of full command lines this plugin requires sudo access for, which is used for validating sudo configurations |
No |
Method |
Parameters |
Description |
Required |
---|---|---|---|
run |
none |
Execute the countermeasure action |
Yes |
validate |
none |
Method to perform validation on the plugin's setup. This is called by the command-line tool's "validate-plugins" command. Mainly used by helper subclasses that intend to have some additional properties overridden. Should return nothing if the plugin is valid, or a string describing validation issues if there are problems. |
No |
prepare |
none |
Method to be run before execution, for any initial setup or validation that the countermeasure action needs to perform |
No |
save_text_output |
output (String) |
Save countermeasure output as plain text for later publishing up to the FortiMonitor cloud |
No |
save_html_output |
output (String) |
Save countermeasure output as formatted HTML for later publishing up to the FortiMonitor cloud |
No |
save_return_code |
return_code |
Save the return code from the countermeasure execution |
No |