The FortiMonitor Agent supports generating custom incidents, which provides a mechanism for your own applications to push event data to our system and leverage FortiMonitor’s alerting and dashboard functionality. To use the custom incidents, you need to have the latest version of the Agent and have custom incidents enabled on your account. Please contact our support team to have that enabled.

To use the custom incident support, you create a JSON file each time there is an incident to report and place it in a special agent directory. The next time the agent syncs, it picks up that file and sends it to the FortiMonitor Monitoring Cloud where it is processed and begins alerting.

Incident File Generation

The incident files are JSON files that get placed in the following directories:

Linux

/usr/share/fm-agent/incoming

Windows

C:\Program Files (x86)\FortimonitorAgent\custom_metric

Additional directories can be specified by setting a metric_incoming_directory variable with a comma-separated list of directories in your Agent configuration file (Linux only) located in /etc/fm-agent/panopta_agent.cfg.

Each file should look like the following:

{"agent_payload_version": 1.0,

"plugin_textkey": "sample_custom_plugin",

"plugin_category_name": "Sample Application Plugin",

"incidents": [

{"textkey": "billing.batch_error",

"timestamp": "2017-08-17 10:50:00",

"description": "Error processing billing batch #23",

"action": "open",

"metadata": {

"lines_processed": 100,

"error_count": 27

}

},

{"textkey": "billing.batch_error",

"timestamp": "2017-08-17 11:50:00",

"description": "Error processing billing batch #24",

"action": "resolve",

"match_key": "billing-batch-24",

"metadata": {

"lines_processed": 100,

"error_count": 27

}

}

]

}

The top-level keys in the document set the agent version, which should be left as 1.0, and the plugin textkey and category name. The textkey is used to reference this application internally, and can be any string of letters, numbers, periods, and underscores. The plugin category name is used for display in the control panel’s metric catalog and for alert descriptions.

The incidents list contains a separate entry for each incident that you wish to report. The keys in each entry are:

• textkey: an internal reference for the resource the incident applies to. Follows the same character convention as the plugin textkey.

• timestamp: The time of the event is starting or being resolved. The timestamps are in YYYY-MM-DD HH:MM:SS format, and need to be in the UTC timezone.

• description: A human readable description of the event, which is shown in the control panel and alerts.

• action: A value of open or resolve, which indicates if the event is being created or resolved.

• match_key: An optional field, if provided it is used to connect open and resolve events that relate to the same issue. These are arbitrary strings, so they can be any identifier that makes sense for your application. If not provided, incidents can be closed through the control panel or will automatically get closed after 60 minutes.

• metadata: An optional dictionary of key/value fields that will get added to the incident as additional data that is then shown in the control panel. Use this to pass in any additional details that your users will want to see when diagnosing the issue.

You can bundle together multiple incidents into one file, or create each one as a separate file. The agent will pick up and process each file found in the import directory. We recommend naming files based on the application name and current timestamp to avoid running into conflicts with previous files that are waiting to be picked up.

Control Panel configuration

Before you can start sending up custom incidents, you need to enable support on the server whose agent will be sending up the incident data. Navigate to the server’s Instance Details page in the control panel, where you should see a Custom Incidents tab:

Here you will need to set up filtering rules that map incident plugin and resource textkeys to the type of alerting that they will trigger. Click on the Add Incident Type button, then you will see the following modal:

Enter the plugin textkey you are using in the JSON files, and optionally specific resource textkeys (either as exact matches or using a * for partial matches. Once the filter is in place, any custom incident data that is sent through the agent will create incidents that will appear in your dashboards and begin sending alerts using the specified timeline.

As a quick way to get started, you can create a single filter that matches all resource textkeys, and later can refine this with more specific resource textkey filters if you wish to use different alert timelines for different type of incidents.