Fortinet black logo

Administration Guide

Monitoring FortiSOAR

Copy Link
Copy Doc ID b60b554b-4ea9-11ed-9d74-fa163e15d75b:737579
Download PDF

Monitoring FortiSOAR

Administrators can monitor various important aspects of their FortiSOAR system such as uptime (availability) of FortiSOAR, monitoring databases, services, disk space utilization, CPU and Memory utilization, etc.

The "System Health Status" Dashboard, monitoring playbooks, and the High Availability (HA) notifications that FortiSOAR sends already monitor various elements of the FortiSOAR system and also send appropriate notifications to users.

You can also use the csadm ha show-health command to get the heath information for FortiSOAR nodes. For more information, see the High Availability support in FortiSOAR chapter.

This chapter intends to provide more details on what are the various aspects that can be monitored in case you want to fine tune the monitoring and/or setup monitoring using custom tools.

You can set up the system monitoring and purging of audit and playbook logs as part of your initial deployment and configuration process. For more information, see the Setting up monitoring for your FortiSOAR system topic in the Additional configuration settings for FortiSOAR chapter of the "Deployment Guide."

You can also set up system monitoring for FortiSOAR, both in case of a single node system and High Availability (HA) clusters. You can also receive monitor the system and get email notifications for failures of any FortiSOAR service, or if any monitored thresholds exceed the set threshold. In case of HA clusters, in addition, you can also monitor and get notified in case of heartbeat failures and high replication lags between nodes of your HA cluster. For more information, see the Configuring System and Cluster Health Monitoring topic in the System Configuration chapter.

For the list of logs that you can use for troubleshooting FortiSOAR, see the Debugging, Troubleshooting, and optimizing FortiSOAR chapter.

For information on monitoring the secure message exchange, see the Monitoring the connectivity of the different nodes at the secure message exchange topic in the "Multi-tenancy support in FortiSOAR Guide."

Benefits of monitoring

Implementing effective application monitoring offers the following benefits:

  • Increased server, services, and application availability.
  • Faster detection of network outages and protocol failures.
  • Faster detection of failed services, processes, and scheduled jobs.

Manually setting up monitoring for each FortiSOAR component

To monitor various components of your FortiSOAR system, you need to SSH to your FortiSOAR VM and login as a user who has root or sudo permissions and then run the commands mentioned in the following sections.

Monitoring uptime of FortiSOAR

To monitor availability of FortiSOAR, run the uptime command.

You can also get the latest health check details for your FortiSOAR system. By default, the FortiSOAR health check runs every 5 minutes. For more information, see the System Configuration chapter.

Monitoring FortiSOAR services

To know the status of all FortiSOAR services run the # csadm services --status command.

To view the status of individual FortiSOAR services, run the # systemctl status <service_name> command. For example, to see the status of the nginx service, use the # systemctl status nginx command.

When you run # csadm services --status command the status of FortiSOAR services are displayed with a background color so that you can quickly and easily identify which services are running and which are not running. The status of services that are running are displayed in a Green background, and the status of services that are not running are displayed in a Red background.

Following image displays how the statuses of FortiSOAR services are displayed when some services are running, and some are not running:

Status of FortiSOAR™ services

You can also use the "System Monitoring" widget to monitor various FortiSOAR system resources such as CPU, Disk Space and memory utilization and status of various FortiSOAR services. FortiSOAR includes a default system monitoring dashboard, the "System Health Status" Dashboard, that displays the usage and health of various components in your FortiSOAR system. For more information on the System Monitoring widget, see the Dashboards, Templates, and Widgets chapter in the "User Guide."

Monitoring databases

To know the status of your PostgreSQL database run the systemctl status postgresql-$(psql --version |egrep -o '[0-9]{1,}\.'| cut -d'.' -f1) -l command.

To know the status of your Elasticsearch database run the systemctl status elasticsearch command.

Monitoring Disk Space Utilization

			$ sudo df -H | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{ print $5 " " $1 }'
			43% /dev/mapper/vgos-root
			13% /dev/sda1
			2% /dev/mapper/vgos-tmp
			10% /dev/mapper/vgos-var
			1% /dev/mapper/vgos-rabbitmq
			1% /dev/mapper/vgos-home
			1% /dev/mapper/vgdata-relations
			1% /dev/mapper/vgsearch-search
			3% /dev/mapper/vgos-log
			1% /dev/mapper/vgcoredump-coredump
			7% /dev/mapper/vgos-audit
			22% /dev/mapper/vgapp-csapps
		$

You can also use the System Monitoring widget and the "System Health Status" Dashboard to monitor the disk space utilization.

Monitoring CPU and Memory Utilization

# Top 50 process with memory and CPU usage
	$sudo ps -eo pid,cmd,%mem,%cpu --sort=-%mem | head -50
	# RAM and Swap memory usage command
	free -m

You can also use the System Monitoring widget and the "System Health Status" Dashboard to monitor the CPU and memory utilization.

Monitoring connectors

Use the "Connector Health" widget to track the health of all the configurations of all your configured connectors. You can view the status of your connector configurations using the "System Health Status" Dashboard.

You can also retrieve the health status of any connector configuration using the following API call: POST /api/integration/connectors/healthcheck/<name>/<version>/?config=<config_id>
For example, POST /api/integration/connectors/healthcheck/smtp/2.3.3/?config=88c3d39c-2fa9-4731-b00d-29815008f17c

Following are additional APIs around connectors and configurations:

  • POST /api/integration/connectors: Use this API to list all connectors installed on a FortiSOAR instance.
  • POST /api/integration/connectors/<name>/<version>: Use this API to list all configurations for a connector.

For information on authenticating and invoking FortiSOAR APIs, see the "API Guide."

Monitoring workflows

To know the number of workflows that are queued and not yet picked up for execution use the following command:
rabbitmqctl list_queues -p fsr-cluster --no-table-headers --silent | grep -E "^\s*celery\s+" | awk '{print $2}'

The number returned by this command should be 0, or at the maximum within two digits. If this number remains high for long, it means that the workflow engine is not able to cope up with the requests and requires to be tuned or you need to scale horizontally. From version 7.0.0 onwards, you can set the threshold for the workflow queue (default is 100) and if the threshold set is reached or crossed for this parameter, an email notification is sent to the specified email addresses, see the System Configuration chapter for more information. For information on tuning workflows, see the Debugging, Troubleshooting, and optimizing FortiSOAR chapter and Debugging and Optimizing Playbooks in the "Playbooks Guide."

Monitoring FortiSOAR

Administrators can monitor various important aspects of their FortiSOAR system such as uptime (availability) of FortiSOAR, monitoring databases, services, disk space utilization, CPU and Memory utilization, etc.

The "System Health Status" Dashboard, monitoring playbooks, and the High Availability (HA) notifications that FortiSOAR sends already monitor various elements of the FortiSOAR system and also send appropriate notifications to users.

You can also use the csadm ha show-health command to get the heath information for FortiSOAR nodes. For more information, see the High Availability support in FortiSOAR chapter.

This chapter intends to provide more details on what are the various aspects that can be monitored in case you want to fine tune the monitoring and/or setup monitoring using custom tools.

You can set up the system monitoring and purging of audit and playbook logs as part of your initial deployment and configuration process. For more information, see the Setting up monitoring for your FortiSOAR system topic in the Additional configuration settings for FortiSOAR chapter of the "Deployment Guide."

You can also set up system monitoring for FortiSOAR, both in case of a single node system and High Availability (HA) clusters. You can also receive monitor the system and get email notifications for failures of any FortiSOAR service, or if any monitored thresholds exceed the set threshold. In case of HA clusters, in addition, you can also monitor and get notified in case of heartbeat failures and high replication lags between nodes of your HA cluster. For more information, see the Configuring System and Cluster Health Monitoring topic in the System Configuration chapter.

For the list of logs that you can use for troubleshooting FortiSOAR, see the Debugging, Troubleshooting, and optimizing FortiSOAR chapter.

For information on monitoring the secure message exchange, see the Monitoring the connectivity of the different nodes at the secure message exchange topic in the "Multi-tenancy support in FortiSOAR Guide."

Benefits of monitoring

Implementing effective application monitoring offers the following benefits:

  • Increased server, services, and application availability.
  • Faster detection of network outages and protocol failures.
  • Faster detection of failed services, processes, and scheduled jobs.

Manually setting up monitoring for each FortiSOAR component

To monitor various components of your FortiSOAR system, you need to SSH to your FortiSOAR VM and login as a user who has root or sudo permissions and then run the commands mentioned in the following sections.

Monitoring uptime of FortiSOAR

To monitor availability of FortiSOAR, run the uptime command.

You can also get the latest health check details for your FortiSOAR system. By default, the FortiSOAR health check runs every 5 minutes. For more information, see the System Configuration chapter.

Monitoring FortiSOAR services

To know the status of all FortiSOAR services run the # csadm services --status command.

To view the status of individual FortiSOAR services, run the # systemctl status <service_name> command. For example, to see the status of the nginx service, use the # systemctl status nginx command.

When you run # csadm services --status command the status of FortiSOAR services are displayed with a background color so that you can quickly and easily identify which services are running and which are not running. The status of services that are running are displayed in a Green background, and the status of services that are not running are displayed in a Red background.

Following image displays how the statuses of FortiSOAR services are displayed when some services are running, and some are not running:

Status of FortiSOAR™ services

You can also use the "System Monitoring" widget to monitor various FortiSOAR system resources such as CPU, Disk Space and memory utilization and status of various FortiSOAR services. FortiSOAR includes a default system monitoring dashboard, the "System Health Status" Dashboard, that displays the usage and health of various components in your FortiSOAR system. For more information on the System Monitoring widget, see the Dashboards, Templates, and Widgets chapter in the "User Guide."

Monitoring databases

To know the status of your PostgreSQL database run the systemctl status postgresql-$(psql --version |egrep -o '[0-9]{1,}\.'| cut -d'.' -f1) -l command.

To know the status of your Elasticsearch database run the systemctl status elasticsearch command.

Monitoring Disk Space Utilization

			$ sudo df -H | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{ print $5 " " $1 }'
			43% /dev/mapper/vgos-root
			13% /dev/sda1
			2% /dev/mapper/vgos-tmp
			10% /dev/mapper/vgos-var
			1% /dev/mapper/vgos-rabbitmq
			1% /dev/mapper/vgos-home
			1% /dev/mapper/vgdata-relations
			1% /dev/mapper/vgsearch-search
			3% /dev/mapper/vgos-log
			1% /dev/mapper/vgcoredump-coredump
			7% /dev/mapper/vgos-audit
			22% /dev/mapper/vgapp-csapps
		$

You can also use the System Monitoring widget and the "System Health Status" Dashboard to monitor the disk space utilization.

Monitoring CPU and Memory Utilization

# Top 50 process with memory and CPU usage
	$sudo ps -eo pid,cmd,%mem,%cpu --sort=-%mem | head -50
	# RAM and Swap memory usage command
	free -m

You can also use the System Monitoring widget and the "System Health Status" Dashboard to monitor the CPU and memory utilization.

Monitoring connectors

Use the "Connector Health" widget to track the health of all the configurations of all your configured connectors. You can view the status of your connector configurations using the "System Health Status" Dashboard.

You can also retrieve the health status of any connector configuration using the following API call: POST /api/integration/connectors/healthcheck/<name>/<version>/?config=<config_id>
For example, POST /api/integration/connectors/healthcheck/smtp/2.3.3/?config=88c3d39c-2fa9-4731-b00d-29815008f17c

Following are additional APIs around connectors and configurations:

  • POST /api/integration/connectors: Use this API to list all connectors installed on a FortiSOAR instance.
  • POST /api/integration/connectors/<name>/<version>: Use this API to list all configurations for a connector.

For information on authenticating and invoking FortiSOAR APIs, see the "API Guide."

Monitoring workflows

To know the number of workflows that are queued and not yet picked up for execution use the following command:
rabbitmqctl list_queues -p fsr-cluster --no-table-headers --silent | grep -E "^\s*celery\s+" | awk '{print $2}'

The number returned by this command should be 0, or at the maximum within two digits. If this number remains high for long, it means that the workflow engine is not able to cope up with the requests and requires to be tuned or you need to scale horizontally. From version 7.0.0 onwards, you can set the threshold for the workflow queue (default is 100) and if the threshold set is reached or crossed for this parameter, an email notification is sent to the specified email addresses, see the System Configuration chapter for more information. For information on tuning workflows, see the Debugging, Troubleshooting, and optimizing FortiSOAR chapter and Debugging and Optimizing Playbooks in the "Playbooks Guide."