Debugging, Troubleshooting, and optimizing FortiSOAR

Administrators can use various logs that FortiSOAR generates to troubleshoot FortiSOAR issues. This chapter lists the key FortiSOAR services and processes and also provides some troubleshooting tips. This chapter also provides some additional configuration settings so that you can tune the results that get displayed by FortiSOAR for record similarity and field prediction. For more information on record similarity and field prediction, see the Working with Modules - Alerts & Incidents chapter in the "User Guide."

If you face any issues while deploying or upgrading FortiSOAR, see the Troubleshooting FortiSOAR chapter in the "Deployment Guide." If you face deployment or upgrade failures due to insufficient space, or if you face issues while using FortiSOAR that might be caused due to insufficient space, like you are unable to log into FortiSOAR or FortiSOAR services stop working, then see the Issues occurring in FortiSOAR due to insufficient space section in the Troubleshooting FortiSOAR chapter in the "Deployment Guide."

For information about monitoring various important aspects of your FortiSOAR system, see the Monitoring FortiSOAR chapter.

List of logs used for troubleshooting FortiSOAR

FortiSOAR log files are stored in the following location: /var/log/cyops. You will find the following directories in the /var/log/cyops location:

Log Name	Purpose
cyops-api/ssl_cyops_api_access.log	Used for troubleshooting web (nginx) UI or API access issues.
cyops-api/ssl_cyops_api_error.log	Used for troubleshooting API errors.
cyops-api/prod.log	Used for troubleshooting FortiSOAR PHP related issues.
cyops-api/last_system_publish.log	Used for troubleshooting publishing issues. It captures the output when a publish is fired from the UI after MMD changes.
cyops-auth/das.log	Used for troubleshooting FortiSOAR authentication issues.
cyops-auth/fdn.log	Used for troubleshooting FortiSOAR license synchronization issues with FortiGuard Distribution Network (FDN).
cyops-auth/ha.log	Used for troubleshooting FortiSOAR High Availability issues.
cyops-gateway/auditlog.log	Used for troubleshooting FortiSOAR audit log issues.
cyops-gateway/etl.log	Used for troubleshooting FortiSOAR application and system-level issues.
cyops-gateway/saml.log	Used for troubleshooting FortiSOAR SAML issues.
cyops-search/falcon.log	Used for troubleshooting FortiSOAR Search issues. If you get any error when you are indexing or searching for a record in FortiSOAR, you can use the falcon.log file to troubleshoot ElasticSearch issues. This log is also used for checking the status of ElasticSearch indexing.
cyops-gateway/gateway.log	Used for troubleshooting FortiSOAR Gateway issues such as, audit log page failing to load or the SSO configuration page failing to load.
cyops-notifier/notifier.log	Used for troubleshooting FortiSOAR Web Socket issues.
cyops-workflow/beat.log	Used for troubleshooting issues of the FortiSOAR Scheduler.
cyops-workflow/celeryd.log	Used for troubleshooting FortiSOAR playbook runtime issues.
cyops-workflow/sealab.log	Used for troubleshooting FortiSOAR playbook framework issues.
cyops-workflow/ssl_cyops_workflow_access.log	Used for troubleshooting playbook access issues.
cyops-workflow/ssl_cyops_workflow_error.log	Used for troubleshooting playbook errors.
cyops-workflow/uwsgi.log	Used for troubleshooting FortiSOAR playbook and connector issues.
cyops-integrations/connectors.log	Used for troubleshooting FortiSOAR connector related issues.
cyops-integrations/integrations.log	Used for troubleshooting FortiSOAR connector framework issues.
cyops-integrations/integrations/imap/listener.log	Used for troubleshooting the IMAP connector issues.
cyops-integrations/ssl_cyops_integrations_access.log	Used for troubleshooting connector access issues.
cyops-integrations/ssl_cyops_integrations_error.log	Used for troubleshooting connector errors.
csadm/db.log	Used for troubleshooting database externalization errors.
install For example, 6.4.1-2133.log.	Used for troubleshooting FortiSOAR installation issues. Install logs are named according to the FortiSOAR version and build number.
install/connectors.log	Used for troubleshooting connector installation issues.
upgrade_fortisoar_<version_number>-<timestamp>.log For example, upgrade_fortisoar_6.0.0-2020-02-18-1558604373.log	Stores upgrade console log and you can use it to troubleshoot FortiSOAR upgrade issues.
install/config-vm-<timestamp>.log For example, install/config-vm-27_Nov_2018_18h_14m_34s.log	Used for troubleshooting FortiSOAR Configuration Wizard issues.

For troubleshooting FortiSOAR audit log issues use the tomcat.log located at /opt/cyops-tomcat/tomcat.log.

For Centos OS level errors, use the Messages logs located at /var/log/messages.

For troubleshooting issues related to dedicated tenant nodes or FortiSOAR agents, see the following logs:
var/log/cyops/cyops-routing-agent/postman.log, var/log/cyops/cyops-routing-agent/uwsgi.log, var/log/cyops/cyops-routing-agent/ssl_cyops_routing_agent_error.log, and var/log/cyops/cyops-routing-agent/ssl_cyops_routing_agent_access.log.

Logging Levels

You can set the following logging levels in the log files:

• DEBUG: Low-level system information for debugging purposes.
• INFO: General system information.
• WARNING: Information describing a minor problem that has occurred.
• ERROR: Information describing a major problem that has occurred.
• CRITICAL: Information describing a critical problem that has occurred.

Changing the logging levels

• For sealab or workflow:
  1. Open the /opt/cyops-workflow/sealab/sealab/config.ini file and set the WORKFLOW_LOG_LEVEL parameter to the required logging level. For example, WORKFLOW_LOG_LEVEL = 'INFO'
  2. Restart the uwsgi service.
• For integrations:
  1. Open the /opt/cyops-integrations/integrations/configs/config.ini file and set the connector_logger_level parameter to the required logging level. For example, connector_logger_level= 'INFO'
  2. Restart the uwsgi service.
• For celeryd:
  1. Open the /etc/celery/celeryd.conf file and set the CELERYD_LOG_LEVEL parameter to the required logging level. For example, CELERYD_LOG_LEVEL = 'INFO'
  2. Restart the celeryd service.
• For nginx (UI), API, or php:
  1. Open the /opt/cyops-api/app/config/config_prod.yml file and set the level parameter to the required logging level. For example, level = 'INFO'
  2. Run the # systemctl restart php-fpm nginx command.

List of key FortiSOAR services and processes

Name of Services/Processes	Description
postgresql-12	Service for all application data stored in postgresql DB. To know the status of this service, use the # systemctl status postgresql-12 command.
elasticsearch	Service to bring up the elasticsearch service. To know the status of this service, use the # systemctl status elasticsearch command.
php-fpm	Service for PHP FastCGI implementation. To know the status of this service, use the # systemctl status php-fpm command.
uwsgi	Software application that aims at developing a full stack for building hosting services. uWSGI is named after the Web Server Gateway Interface. We host our playbook execution engine application and connector integrations applications on a uWSGI server. To know the status of uwsgi use the # systemctl status uwsgi command.
celeryd	celeryd is used to run the playbooks asynchronously in the FortiSOAR playbook execution engine. To know the status of celeryd use the # systemctl status celeryd command.
celerybeatd	celerybeatd is a playbook scheduler; used to kick off tasks at regular intervals, that are then executed by available worker nodes in the cluster. To know the status of use the # systemctl status celerybeatd command.
cyops-auth	Service used for FortiSOAR authentications. To know the status of this services, use the # systemctl status cyops-auth command.
cyops-tomcat	Service used for SSO, auditing, and websocket. To know the status of this services, use the # systemctl status tomcat command.
cyops-search	Service used for full-text searching, finding similar records and predicting the value of fields based on similarity.
cyops-ha	Service is responsible for setting up High Availability in the FortiSOAR environment
cyops-postman	Service is responsible for setting up and managing FortiSOAR's distributed multi-tenant setup.
cyops-integrations-agent	Service is responsible for running actions on remote FortiSOAR agents.
rabbitmq-server	Service is responsible to send audit and live sync notifications. This service is also responsible for data transfer in a distributed environment.
nginx	Service used for Web UI. To know the status of this services, use the # systemctl status nginx command.

If you want to restart, start, or stop all the services together, use FortiSOAR Admin CLI (csadm). For more information on csadm, see the FortiSOAR Admin CLI chapter.

You can run the csadm command on any FortiSOAR machine using any terminal. Any user who has root or sudo permissions can run the csadm command.

To restart FortiSOAR services, type: # csadm services --restart

To start FortiSOAR services, type: # csadm services --start

To stop FortiSOAR services, type: # csadm services --stop

To know the status of all FortiSOAR services type: # csadm services --status

To know more about monitoring services, see the Monitoring FortiSOAR chapter.

Additional settings for record similarity and field predictions

FortiSOAR supports "Record Similarity" i.e., FortiSOAR displays records that are similar to the record on which you are working. FortiSOAR also supports "Record Field Value Prediction" i.e., FortiSOAR predicts values of fields of your choice within a record from the values of fields of existing records based on the criteria you have defined, making it easier for analysts to make informed decisions. For more information, see the Working with Modules - Alerts & Incidents chapter in the "User Guide."

This section provides information on how you can tune the results that are displayed by FortiSOAR for record similarity and field predictions using the following parameters in the /opt/cyops/config/cyops-search/config.yml file:

• minimum_should_match: <percentageValue>: This setting defines that a record will be considered similar only if there is a match of at least the percentage value that you have specified on the related fields. This is especially true for similarity based on related records. For example, if you set this parameter as minimum_should_match: 10% (default), then if you have defined similarity for alerts based on related indicators, then FortiSOAR will display only those records as similar that match a minimum of 10% of the indicators. Therefore, for an alert that has 10 related indicators, FortiSOAR similarity results will display alerts that even have one common indicator; but if an alert has 20 related indicators, then FortiSOAR similarity results will display only those alerts that have at least 2 indicators in common.
• max_query_terms: <numberOfItems>: This setting defines how many terms of the parent record will be looked up for similarity in other records. Continuing the same example as above, if you set this parameter as max_query_terms: 25 (default), then if an alert has more than 25 indicators, only 25 of them will be checked for similarity in other records. Note that increasing the value of this setting will increase the time FortiSOAR takes to return similarity and suggestion results.

For more information on the above parameters and other parameters, refer to the ElastiSearch reference at: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-mlt-query.html

Change the default value of some of the user profile parameters

An administrator with CRUD permissions on the Security module can change the default value of the following user-profile related parameters:

Parameter Name	Description	Default Value
max_reset_attempts	Maximum number of times users' can click the Reset Password link before actually resetting their password. If the user exceeds the value set in this parameter, then users' will not get a new link to reset their password based on the number of hours specified in the reset_locktime parameter. By default, the max_reset_attempts is set to 10 times and the reset_locktime is set to 12 hours, therefore, if a user clicks the Reset Password 11 times without actually resetting their password, then the user will not get a new link to reset their password for 12 hours.	10 times
reset_locktime	Number of hours that users' will not get a new link to reset their password if they exceed the value set in the max_reset_attempts parameter.	12 hours
max_failed	Number of times that users' can enter an incorrect password, while logging into FortiSOAR, before their account gets locked. If the user exceeds the value set in this parameter, then the user will get locked out based on the number of minutes specified in the lock_minutes parameter. By default, the max_failed is set to 5 times and the lock_minutes is set to 30 mins, therefore, if a user enters an incorrect password 6 times, then their account gets locked for 30 mins.	5 times
lock_minutes	Number of minutes that users' account gets locked if they exceed the value set in the max_failed parameter.	30 mins

To change the value of the max_reset_attempts parameter, the administrator should run the following curl command on their FortiSOAR system:

curl -X PUT \
  https://<FORTISOAR HOSTNAME/IP>/api/auth/config \
  -H 'Authorization: <Bearer Token>' \
  -d '{
	"option":"max_reset_attempts",
	"value":5
}'

The above command changes the number of times users' can click the Reset Password link to 5 times, i.e., a user can click the Reset Password link 5 times without actually setting the new password. However, if the user clicks the Reset Password link for the 6th time, the user will be blocked.

Similarly, to change the value of the reset_locktime, max_failed, and lock_minutes parameters, the administrator should run the same curl command on their FortiSOAR system, but after changing the option and value parameter values:

curl -X PUT \
  https://<FORTISOAR HOSTNAME/IP>/api/auth/config \
  -H 'Authorization: <Bearer Token>' \
  -d '{
	"option":"reset_locktime",
	"value":2
}'

The above command changes the number of hours that users' will not get a new link to reset their password to 2 hours if they have exceeded the value set in the max_reset_attempts parameter.

curl -X PUT \
  https://<FORTISOAR HOSTNAME/IP>/api/auth/config \
  -H 'Authorization: <Bearer Token>' \
  -d '{
	"option":"max_failed",
	"value":3
}'

The above command changes the number of times that users' can enter an incorrect password while logging into FortiSOAR before their account gets locked to 3, i.e., users' account will be locked if they enter an incorrect password 4 times while logging into FortiSOAR.

curl -X PUT \
  https://<FORTISOAR HOSTNAME/IP>/api/auth/config \
  -H 'Authorization: <Bearer Token>' \
  -d '{
	"option":"lock_minutes",
	"value":15
}'

The above command changes the number of minutes that users' account will be locked to 15 minutes if they have exceeded the value set in the max_reset_attempts parameter.

Security considerations for Websockets

The Websocket server allows connection from any hostname URL UI. To restrict connections to the Websocket server, you can implement Cross-Origin Resource Sharing (CORS) restrictions in WebSocket.

To implement CORS in WebSocket, you have to add a comma-separated list of allowed hosts (IP address or domain name) in the /etc/cyops/config.yml file and the hostname should start with "http://" or "https://". Dummy entries have already been added in the /etc/cyops/config.yml file to help you in this process. If you have added domain names to the /etc/cyops/config.yml file, then you must also restart the 'tomcat' service, using the systemctl restart cyops-tomcat command.

Once the hostname entries have been added, the WebSocket will only establish a connection with those UIs whose URLs match the specified hostnames.

Troubleshooting Tips

Your Workflow data size has increased

Increase in your Workflow data can cause performance bottlenecks.

Resolution:

You can purge Executed Playbook Logs using the Purge Logs button on the top-right of the Executed Playbook Logs dialog. For more information on purging, see the Debugging and Optimizing Playbooks chapter in the "Playbooks Guide."

Error displayed while performing a search operation in FortiSOAR

Resolution:

If you get any error while performing a global search in FortiSOAR, check that the elasticsearch.service and the cyops-search.service are running.

If these are not running, then start these services using the following commands:

# systemctl start elasticsearch

# systemctl start cyops-search

For more information, see the FortiSOAR Search Errors topic in the Elasticsearch Configuration chapter.

Reindexing FortiSOAR modules for search

Partial indexing of a module, or when a module does not get indexed, can lead to errors in FortiSOAR search. You can manually reindex any skipped or unsuccessfully indexed modules. For more information, see the FortiSOAR Search Errors topic in the Elasticsearch Configuration chapter.

Resolution:

To reindex all the FortiSOAR modules, run the following command:

$ sudo -u nginx php /opt/cyops-api/app/console app:elastic:create --env="prod"

To reindex specific FortiSOAR modules, run the following command:

$ sudo -u nginx php /opt/cyops-api/app/console app:elastic:create --env="prod" --index='{"type":"type of the module(s)"}'

For example:

$ sudo -u nginx php /opt/cyops-api/app/console app:elastic:create --env="prod" --index='{"type":"indicators", "tasks"}

FortiSOAR crashing with "out of memory" errors

By default, FortiSOAR configures Elaticsearch to use 4 GB of RAM. If there are too many records or any very heavy records (such as large files uploaded) created per day on the system, it might crash with "out of memory" errors. To fix this, you must increase the memory allocated to Elasticsearch.

Resolution

1. Change the following entry in /etc/elasticsearch/jvm.options to a higher value based on memory available on your server:
   -Xms4g
   -Xmx4g
2. Restart Elasticsearch using the following command:
   systemctl restart elasticsearch

Changing Postgres worker memory

When the primary data in the system becomes large (eg, over million alerts) and you notice that the system is slow to respond. The slowness could be caused due to database queries taking longer with the increased database size. You can fine tune this behavior by increasing the following Postgres settings based on the available free memory on the system:

1. Increase the shared buffer and worker memory in the /var/lib/pgsql/12/data/postgresql.conf file: shared_buffers = 2048MB
work_mem = 16MB
2. Restart Postgres using the following command:
   systemctl restart postgresql-12

Changing the maximum number of records that can be linked in one call

The maximum number of records that can be linked in one call should be 99.

To change this limit, you can change the max_relation_count parameter value in the /opt/cyops-api/configs/parameters_prod.yml file:

# max numnber of relations to normalize
   max_relation_count: 100