Fortinet black logo

Administration Guide

FortiSOAR integration with FortiMonitor

Copy Link
Copy Doc ID f50e6507-ba25-11ec-9fd1-fa163e15d75b:253964
Download PDF

FortiSOAR integration with FortiMonitor

From release 7.2.0 onwards, FortiSOAR has been integrated with FortiMonitor to enable monitoring including CPU, RAM, Disk monitoring, network card bandwidth, Nginx, PostgreSQL monitoring etc., of your FortiSOAR instances using FortiMonitor. FortiMonitor is used for managing large-scale infrastructure monitoring from single pane of glass, regardless of infrastructure deployment, and it empowers monitoring, incident management, and automated remediation.

Setting up a FortiSOAR instance to be monitored using FortiMonitor

Note

You cannot install FortiMonitor if you have set up a proxy environment. This is a known limitation with FortiMonitor as it uses python2 that does not automatically honor OS configured proxy.

To monitor FortiSOAR using FortiMonitor, you need to install an agent on the instance that needs to be monitored.

Note

You can set up monitoring of an external secure message exchange instance by FortiMonitor; however, monitoring in the case of an air gapped environment is not supported.

If you are existing FortiMonitor customer, then log onto your FortiMonitor instance and click Monitoring > Monitoring Policies. Next, click Default Monitoring and select the FortiSOAR checkbox in the drawer that appears before installing the FortiMonitor agent.

Steps to install the agent is as follows:

  1. You must have a FortiMonitor account. If you do not have a FortiMonitor account, you can create an account with a 30-day free trial on https://www.fortinet.com/offers/fortimonitor-free-trial.
  2. Once you have created your account, you need to get your customer key. You can get your customer key by logging into FortiMonitor (https://fortimonitor.forticloud.com/login) using your credentials. Then, click on your profile and select the My Account option. In the Edit Account dialog, from the Customer Key field, copy your customer key.
  3. Ensure that the following URLs are reachable from your instance:
    • packages.panopta.com
    • aggregator2.panopta.com
  4. SSH to the FortiSOAR instance that you want to FortiMonitor to monitor and run the csadm system fortimonitor agent install --customer-key <your customer key> command.
    This installs the agent on the instance that requires to be monitored using FortiMonitor.
    For details on managing the FortiMonitor agent using csadm, see the FortiSOAR Admin CLI chapter.

Monitoring FortiSOAR using FortiMonitor

Log into FortiMonitor (https://fortimonitor.forticloud.com/login) and you will see the FortiSOAR dashboard as displayed in the following sample image:

FortiMontior - FortiSOAR Dashboard

Click the Instances button and then click FortiSOAR to view all the FortiSOAR instances that are being monitored by FortiMonitor.

Components that are monitored out-of-the box by FortiMonitor on FortiSOAR

  • CPU (Usage in %)
  • Disk (Usage in % for every logical volume and also for /boot partition)
  • I/O (read and write requests/sec for disks)
  • Network card bandwidth including loopback(lo) interface (kb/sec)
  • RAM usage (%)
  • NTP (Difference between NTP and machine in sec)
  • Nginx (dropped connections, requests per second, handled connections, etc)
  • PostgreSQL for every database (connectors, das, gateway, notifier, postman, sealab, venom) includes active connections, blocks read from disk(blocks/min), buffer cache hit rate(%), total transactions(tx/min), etc.
  • Postfix (number of requests, postfix queue size)

For information on FortiMonitor, see the FortiMonitor Documentation.

FortiSOAR-specific components that are monitored by FortiMonitor

  • Services status
  • License Expiry
  • Root Certificate Expiry
  • Audit logs database size
  • Cluster data replication lag (Applicable only for HA)
  • Cluster heartbeat lag (Applicable only for HA)
  • Connector health
  • Nginx certificate expiry
  • Playbook logs database size
  • Primary database size
  • Queued playbooks
  • RabbitMQ certificate expiry
  • Self heartbeat lag
  • Tenancy data replication lag (Applicable only for MSSP)

To filter FortiSOAR metrics, click the FortiSOAR metric:

FortiMontior - FortiSOAR Dashboard - FortiSOAR Metrics

High Availability clusters are not monitored by default, and require some additional steps to be performed, which is mentioned in the following topic.

Enabling High Availability Cluster Monitoring

To monitor an HA cluster, you must perform the steps mentioned in the Setting up a FortiSOAR instance to be monitored using FortiMonitor topic for each node in the cluster. You also have to set up attributes for the nodes on FortiMonitor for the takeover operation as described in the following topic.

Performing Takeover

In an HA cluster all secondary nodes monitor the heartbeat of the primary node. If the primary node is down for more than 20 minutes, then FortiMonitor creates an incident for takeover.

Note

If there is more than one secondary node in the HA cluster, then each of the nodes creates separate incidents asking for approval for this countermeasure (takeover). However, you should approve this countermeasure only for the node that you want to create as the new primary node of the cluster, and ignore the approval requests from the remaining secondary nodes.

To perform takeover to make a secondary node as the primary cluster node, click the Notifications icon and view the active incidents. Click the incident from the secondary node that you want to create as the new primary node of the cluster, then click on the countermeasure icon and then Approve the takeover only for that node. Ignore the approval requests from the remaining secondary nodes.

By default, all other nodes join the new primary node as secondary passive nodes. However, you can choose to change this by adding attributes for the nodes on the server page of the new primary node.

Example of adding attributes on the server page of the new primary node:
node1.example.com --old primary
node2.example.com --new primary
node3.example.com --this will join the new primary node (node2) by default, as a secondary passive node.
However, if we want to make node3 as active secondary, then on the server page of the new primary node (node2), in the Instance Group section, click Attributes. In the Instance Configuration dialog, in the Attributes section, click Add Attribute to add a Key and Value pair. For example, in the Key field, enter node3.example.com, and in the Value field, enter active and click Save, and then click Save Changes. In the Value field, you can add 'active' that means that the specified node will join the HA cluster as an "active secondary node", or 'do-not-join' that means that the specified node will not join the HA cluster.

Note

It is recommended that attributes for the nodes that are part of an HA cluster should be set when the HA cluster is created.

An example screenshot of a FortiMonitor page containing a FortiSOAR group whose nodes that are part of an HA cluster have their attributes set:

FortiMonitor page containing FortiSOAR group with HA cluster nodes having attributes set

Skipping monitoring of connectors

All the configurations of all the connectors that are configured on the FortiSOAR node (not agent nodes) are monitored. However, If you want to skip monitoring for a connector, then perform the following steps:

  1. SSH to the FortiSOAR instance that is being monitored by FortiMonitor.
  2. Open the panopta_agent cfg file:
    vi /etc/panopta-agent/panopta_agent.cfg.
  3. In the panopta_agent cfg file, in the [fortisoar] section, add the following:
    fortisoar_connector_exclusion_list = <comma-separated-connector-list>
    For example, if you want to skip monitoring for the IMAP and SMTP connectors, add the following line to the panopta_agent cfg file:
    fortisoar_connector_exclusion_list = smtp,imap
  4. Save and quit the panopta_agent cfg file.
  5. Rebuilt the metadata for the FortiMonitor agent using the following command:
    csadm system fortimonitor agent rebuild-metadata
    Running this command, enables the connector monitoring changes to be updated in the agent, and the Connector Health metrics get immediately reflected.
    If you do not run this command, then the connector monitoring changes get reflected after one hour.

OnSight Collector

The OnSight collector is a lightweight virtual appliance that sits on your internal private network and allows you to securely monitor infrastructure, which is protected behind your firewall. The collector has many capabilities, including: network level probing (ICMP, jitter, packet loss, port level), uptime checks, synthetic monitoring checks, network device metric collection, and many other telemetry integrations.

More information about the OnSight vCollector can be found in "FortiMonitor User Guide > OnSight vCollector" at https://docs.fortinet.com/fortimonitor.

Frequently Asked Questions

Q. How to add queued playbooks as a metric for monitoring using the FortiSOAR template

A. To add a queued playbook as a metric, do the following:

  1. Log into FortiMonitor.
  2. Navigate to the Monitoring Config tab.
  3. Select the FortiSOAR tag.
  4. Click on the Add Metric link:
    FortiMonitor - Adding a Metric
  5. Select the Queued playbooks metric:
    FortiMonitor - Adding the Queued pb metric
  6. In Settings from the Frequency drop-down list select the frequency of monitoring the queued playbook:
    Queue pb metric - Setting the frequency of monitoring
  7. Click the Thresholds & CounterMeasures tab and add the threshold counts as per your requirements:
    Queue pb metrics- Setting the thresholds
  8. Click Save to save the queued playbooks as a metric for monitoring.

Q. How to change the monitoring interval for the License expiry metric

A. To change the monitoring interval for the License expiry metric, do the following:

  1. Log into FortiMonitor.
  2. Navigate to the Monitoring Config tab.
  3. Select the FortiSOAR tag.
  4. Click the hamburger menu in the License expiry metric row and select the Edit option:
    Fortimonitor - Editing the license expiry metric
  5. In Settings from the Frequency drop-down list, select the frequency of monitoring the license expiry:
    License expiry metric - Setting the frequency of monitoring
  6. Click Save to update the monitoring interval for the License expiry metric.

Q. How to modify the timings of generating a Takeover Incident

A. To change the timings of generating a Takeover incident from the default of 20 minutes too, for example, 5 minutes, i.e., to generate a Takeover if the cluster heartbeat lag is greater than 5 minutes, do the following:

  1. Log into FortiMonitor.
  2. Click on a secondary node.
  3. Navigate to the Monitoring Config tab.
  4. Select the FortiSOAR tag.
  5. Click the hamburger menu in the Cluster heartbeat lag metric row and select the Edit option:
    Fortimonitor - Editing the Cluster heartbeat lag metric
  6. Click the Thresholds & CounterMeasures tab and modify the Generate Incident when cluster heartbeat lag is parameter as per your requirements. For our example, set it to greater than 5 minutes: Cluster heartbeat lag metric - Setting the thresholds
  7. Click Save to update to the timing of generating a Takeover Incident.

Q. How to remove stale entries on FortiMonitor for FortiSOAR metrics after a takeover

A. Post-takeover, you might observe stale entries for FortiSOAR metrics on Fortimontior as FortiMonitor does not remove the stale entries on its own when the metadata is rebuilt. The stale metrics require to be removed manually as follows:
For the Primary Node:

  1. Log into FortiMonitor.
  2. Click on a primary node.
  3. Navigate to the Monitoring Config tab.
  4. Select the FortiSOAR tag.
  5. Click the hamburger menu in a FortiSOAR metric row, for example, the Cluster data replication lag metric row and select the Delete option:
    FortiMonitor: Deleting Stale FSR entries on the primary node
    This deletes the stale entry for this particular FortiSOAR metric. You can do the same for the other FortiSOAR metrics that display stale entries.

For the Secondary Node:

  1. Log into FortiMonitor.
  2. Click on a secondary node.
  3. Navigate to the Monitoring Config tab.
  4. Select the FortiSOAR tag.
  5. Click the hamburger menu in a FortiSOAR metric row, for example, the Cluster heartbeat lag metric row and select the Delete option:

    FortiMonitor: Deleting Stale FSR entries on the secondary nodeThis deletes the stale entry for this particular FortiSOAR metric on this secondary node. You can do the same for the other FortiSOAR metrics that display stale entries on other secondary nodes.


FortiSOAR integration with FortiMonitor

From release 7.2.0 onwards, FortiSOAR has been integrated with FortiMonitor to enable monitoring including CPU, RAM, Disk monitoring, network card bandwidth, Nginx, PostgreSQL monitoring etc., of your FortiSOAR instances using FortiMonitor. FortiMonitor is used for managing large-scale infrastructure monitoring from single pane of glass, regardless of infrastructure deployment, and it empowers monitoring, incident management, and automated remediation.

Setting up a FortiSOAR instance to be monitored using FortiMonitor

Note

You cannot install FortiMonitor if you have set up a proxy environment. This is a known limitation with FortiMonitor as it uses python2 that does not automatically honor OS configured proxy.

To monitor FortiSOAR using FortiMonitor, you need to install an agent on the instance that needs to be monitored.

Note

You can set up monitoring of an external secure message exchange instance by FortiMonitor; however, monitoring in the case of an air gapped environment is not supported.

If you are existing FortiMonitor customer, then log onto your FortiMonitor instance and click Monitoring > Monitoring Policies. Next, click Default Monitoring and select the FortiSOAR checkbox in the drawer that appears before installing the FortiMonitor agent.

Steps to install the agent is as follows:

  1. You must have a FortiMonitor account. If you do not have a FortiMonitor account, you can create an account with a 30-day free trial on https://www.fortinet.com/offers/fortimonitor-free-trial.
  2. Once you have created your account, you need to get your customer key. You can get your customer key by logging into FortiMonitor (https://fortimonitor.forticloud.com/login) using your credentials. Then, click on your profile and select the My Account option. In the Edit Account dialog, from the Customer Key field, copy your customer key.
  3. Ensure that the following URLs are reachable from your instance:
    • packages.panopta.com
    • aggregator2.panopta.com
  4. SSH to the FortiSOAR instance that you want to FortiMonitor to monitor and run the csadm system fortimonitor agent install --customer-key <your customer key> command.
    This installs the agent on the instance that requires to be monitored using FortiMonitor.
    For details on managing the FortiMonitor agent using csadm, see the FortiSOAR Admin CLI chapter.

Monitoring FortiSOAR using FortiMonitor

Log into FortiMonitor (https://fortimonitor.forticloud.com/login) and you will see the FortiSOAR dashboard as displayed in the following sample image:

FortiMontior - FortiSOAR Dashboard

Click the Instances button and then click FortiSOAR to view all the FortiSOAR instances that are being monitored by FortiMonitor.

Components that are monitored out-of-the box by FortiMonitor on FortiSOAR

  • CPU (Usage in %)
  • Disk (Usage in % for every logical volume and also for /boot partition)
  • I/O (read and write requests/sec for disks)
  • Network card bandwidth including loopback(lo) interface (kb/sec)
  • RAM usage (%)
  • NTP (Difference between NTP and machine in sec)
  • Nginx (dropped connections, requests per second, handled connections, etc)
  • PostgreSQL for every database (connectors, das, gateway, notifier, postman, sealab, venom) includes active connections, blocks read from disk(blocks/min), buffer cache hit rate(%), total transactions(tx/min), etc.
  • Postfix (number of requests, postfix queue size)

For information on FortiMonitor, see the FortiMonitor Documentation.

FortiSOAR-specific components that are monitored by FortiMonitor

  • Services status
  • License Expiry
  • Root Certificate Expiry
  • Audit logs database size
  • Cluster data replication lag (Applicable only for HA)
  • Cluster heartbeat lag (Applicable only for HA)
  • Connector health
  • Nginx certificate expiry
  • Playbook logs database size
  • Primary database size
  • Queued playbooks
  • RabbitMQ certificate expiry
  • Self heartbeat lag
  • Tenancy data replication lag (Applicable only for MSSP)

To filter FortiSOAR metrics, click the FortiSOAR metric:

FortiMontior - FortiSOAR Dashboard - FortiSOAR Metrics

High Availability clusters are not monitored by default, and require some additional steps to be performed, which is mentioned in the following topic.

Enabling High Availability Cluster Monitoring

To monitor an HA cluster, you must perform the steps mentioned in the Setting up a FortiSOAR instance to be monitored using FortiMonitor topic for each node in the cluster. You also have to set up attributes for the nodes on FortiMonitor for the takeover operation as described in the following topic.

Performing Takeover

In an HA cluster all secondary nodes monitor the heartbeat of the primary node. If the primary node is down for more than 20 minutes, then FortiMonitor creates an incident for takeover.

Note

If there is more than one secondary node in the HA cluster, then each of the nodes creates separate incidents asking for approval for this countermeasure (takeover). However, you should approve this countermeasure only for the node that you want to create as the new primary node of the cluster, and ignore the approval requests from the remaining secondary nodes.

To perform takeover to make a secondary node as the primary cluster node, click the Notifications icon and view the active incidents. Click the incident from the secondary node that you want to create as the new primary node of the cluster, then click on the countermeasure icon and then Approve the takeover only for that node. Ignore the approval requests from the remaining secondary nodes.

By default, all other nodes join the new primary node as secondary passive nodes. However, you can choose to change this by adding attributes for the nodes on the server page of the new primary node.

Example of adding attributes on the server page of the new primary node:
node1.example.com --old primary
node2.example.com --new primary
node3.example.com --this will join the new primary node (node2) by default, as a secondary passive node.
However, if we want to make node3 as active secondary, then on the server page of the new primary node (node2), in the Instance Group section, click Attributes. In the Instance Configuration dialog, in the Attributes section, click Add Attribute to add a Key and Value pair. For example, in the Key field, enter node3.example.com, and in the Value field, enter active and click Save, and then click Save Changes. In the Value field, you can add 'active' that means that the specified node will join the HA cluster as an "active secondary node", or 'do-not-join' that means that the specified node will not join the HA cluster.

Note

It is recommended that attributes for the nodes that are part of an HA cluster should be set when the HA cluster is created.

An example screenshot of a FortiMonitor page containing a FortiSOAR group whose nodes that are part of an HA cluster have their attributes set:

FortiMonitor page containing FortiSOAR group with HA cluster nodes having attributes set

Skipping monitoring of connectors

All the configurations of all the connectors that are configured on the FortiSOAR node (not agent nodes) are monitored. However, If you want to skip monitoring for a connector, then perform the following steps:

  1. SSH to the FortiSOAR instance that is being monitored by FortiMonitor.
  2. Open the panopta_agent cfg file:
    vi /etc/panopta-agent/panopta_agent.cfg.
  3. In the panopta_agent cfg file, in the [fortisoar] section, add the following:
    fortisoar_connector_exclusion_list = <comma-separated-connector-list>
    For example, if you want to skip monitoring for the IMAP and SMTP connectors, add the following line to the panopta_agent cfg file:
    fortisoar_connector_exclusion_list = smtp,imap
  4. Save and quit the panopta_agent cfg file.
  5. Rebuilt the metadata for the FortiMonitor agent using the following command:
    csadm system fortimonitor agent rebuild-metadata
    Running this command, enables the connector monitoring changes to be updated in the agent, and the Connector Health metrics get immediately reflected.
    If you do not run this command, then the connector monitoring changes get reflected after one hour.

OnSight Collector

The OnSight collector is a lightweight virtual appliance that sits on your internal private network and allows you to securely monitor infrastructure, which is protected behind your firewall. The collector has many capabilities, including: network level probing (ICMP, jitter, packet loss, port level), uptime checks, synthetic monitoring checks, network device metric collection, and many other telemetry integrations.

More information about the OnSight vCollector can be found in "FortiMonitor User Guide > OnSight vCollector" at https://docs.fortinet.com/fortimonitor.

Frequently Asked Questions

Q. How to add queued playbooks as a metric for monitoring using the FortiSOAR template

A. To add a queued playbook as a metric, do the following:

  1. Log into FortiMonitor.
  2. Navigate to the Monitoring Config tab.
  3. Select the FortiSOAR tag.
  4. Click on the Add Metric link:
    FortiMonitor - Adding a Metric
  5. Select the Queued playbooks metric:
    FortiMonitor - Adding the Queued pb metric
  6. In Settings from the Frequency drop-down list select the frequency of monitoring the queued playbook:
    Queue pb metric - Setting the frequency of monitoring
  7. Click the Thresholds & CounterMeasures tab and add the threshold counts as per your requirements:
    Queue pb metrics- Setting the thresholds
  8. Click Save to save the queued playbooks as a metric for monitoring.

Q. How to change the monitoring interval for the License expiry metric

A. To change the monitoring interval for the License expiry metric, do the following:

  1. Log into FortiMonitor.
  2. Navigate to the Monitoring Config tab.
  3. Select the FortiSOAR tag.
  4. Click the hamburger menu in the License expiry metric row and select the Edit option:
    Fortimonitor - Editing the license expiry metric
  5. In Settings from the Frequency drop-down list, select the frequency of monitoring the license expiry:
    License expiry metric - Setting the frequency of monitoring
  6. Click Save to update the monitoring interval for the License expiry metric.

Q. How to modify the timings of generating a Takeover Incident

A. To change the timings of generating a Takeover incident from the default of 20 minutes too, for example, 5 minutes, i.e., to generate a Takeover if the cluster heartbeat lag is greater than 5 minutes, do the following:

  1. Log into FortiMonitor.
  2. Click on a secondary node.
  3. Navigate to the Monitoring Config tab.
  4. Select the FortiSOAR tag.
  5. Click the hamburger menu in the Cluster heartbeat lag metric row and select the Edit option:
    Fortimonitor - Editing the Cluster heartbeat lag metric
  6. Click the Thresholds & CounterMeasures tab and modify the Generate Incident when cluster heartbeat lag is parameter as per your requirements. For our example, set it to greater than 5 minutes: Cluster heartbeat lag metric - Setting the thresholds
  7. Click Save to update to the timing of generating a Takeover Incident.

Q. How to remove stale entries on FortiMonitor for FortiSOAR metrics after a takeover

A. Post-takeover, you might observe stale entries for FortiSOAR metrics on Fortimontior as FortiMonitor does not remove the stale entries on its own when the metadata is rebuilt. The stale metrics require to be removed manually as follows:
For the Primary Node:

  1. Log into FortiMonitor.
  2. Click on a primary node.
  3. Navigate to the Monitoring Config tab.
  4. Select the FortiSOAR tag.
  5. Click the hamburger menu in a FortiSOAR metric row, for example, the Cluster data replication lag metric row and select the Delete option:
    FortiMonitor: Deleting Stale FSR entries on the primary node
    This deletes the stale entry for this particular FortiSOAR metric. You can do the same for the other FortiSOAR metrics that display stale entries.

For the Secondary Node:

  1. Log into FortiMonitor.
  2. Click on a secondary node.
  3. Navigate to the Monitoring Config tab.
  4. Select the FortiSOAR tag.
  5. Click the hamburger menu in a FortiSOAR metric row, for example, the Cluster heartbeat lag metric row and select the Delete option:

    FortiMonitor: Deleting Stale FSR entries on the secondary nodeThis deletes the stale entry for this particular FortiSOAR metric on this secondary node. You can do the same for the other FortiSOAR metrics that display stale entries on other secondary nodes.