Version:

Version:

Version:

Version:


Table of Contents

Administration Guide

Download PDF
Copy Link

Debugging, Troubleshooting, and Optimizing FortiSOAR

Administrators can use various logs that FortiSOAR generates to troubleshoot FortiSOAR issues. This chapter lists the key FortiSOAR services and processes and also provides some troubleshooting tips. This chapter also provides some additional configuration settings so that you can tune the results that get displayed by FortiSOAR for record similarity and field prediction. For more information on record similarity and field prediction, see the Working with Modules - Alerts & Incidents chapter in the "User Guide."

If you face any issues while deploying or upgrading FortiSOAR, see the Troubleshooting FortiSOAR chapter in the "Deployment Guide." If you face deployment or upgrade failures due to insufficient space, or if you face issues while using FortiSOAR that might be caused due to insufficient space, like you are unable to log into FortiSOAR or FortiSOAR services stop working, then see the Issues occurring in FortiSOAR due to insufficient space section in the Troubleshooting FortiSOAR chapter in the "Deployment Guide."

For information about monitoring various important aspects of your FortiSOAR system, see the Monitoring FortiSOAR chapter.

List of logs used for troubleshooting FortiSOAR

FortiSOAR log files are stored in the following location: /var/log/cyops. You will find the following directories in the /var/log/cyops location:

Log Name Purpose
cyops-api/ssl_cyops_api_access.log Used for troubleshooting web (nginx) UI or API access issues.
cyops-api/ssl_cyops_api_error.log Used for troubleshooting API errors.
cyops-api/prod.log Used for troubleshooting FortiSOAR PHP related issues.
cyops-api/last_system_publish.log Used for troubleshooting publishing issues. It captures the output when a publish is fired from the UI after MMD changes.
cyops-auth/das.log Used for troubleshooting FortiSOAR authentication issues.
cyops-auth/fdn.log Used for troubleshooting FortiSOAR license synchronization issues with FortiGuard Distribution Network (FDN).
cyops-auth/ha.log Used for troubleshooting FortiSOAR High Availability issues.
cyops-gateway/auditlog.log Used for troubleshooting FortiSOAR audit log issues.
cyops-gateway/saml.log Used for troubleshooting FortiSOAR SAML issues.
cyops-search/falcon.log Used for troubleshooting FortiSOAR Search issues. If you get any error when you are indexing or searching for a record in FortiSOAR, you can use the falcon.log file to troubleshoot ElasticSearch issues.
This log is also used for checking the status of ElasticSearch indexing.
cyops-gateway/gateway.log Used for troubleshooting FortiSOAR Gateway issues such as, audit log page failing to load or the SSO configuration page failing to load.
cyops-notifier/notifier.log Used for troubleshooting FortiSOAR Web Socket issues.
cyops-workflow/beat.log Used for troubleshooting issues of the FortiSOAR Scheduler.
cyops-workflow/celeryd.log Used for troubleshooting FortiSOAR playbook runtime issues.
cyops-workflow/sealab.log Used for troubleshooting FortiSOAR playbook framework issues.
cyops-workflow/ssl_cyops_workflow_access.log Used for troubleshooting playbook access issues.
cyops-workflow/ssl_cyops_workflow_error.log Used for troubleshooting playbook errors.
cyops-workflow/uwsgi.log Used for troubleshooting FortiSOAR playbook and connector issues.
cyops-integrations/connectors.log Used for troubleshooting FortiSOAR connector related issues.
cyops-integrations/integrations.log Used for troubleshooting FortiSOAR connector framework issues.
cyops-integrations/integrations/imap/listener.log Used for troubleshooting the IMAP connector issues.
cyops-integrations/ssl_cyops_integrations_access.log Used for troubleshooting connector access issues.
cyops-integrations/ssl_cyops_integrations_error.log Used for troubleshooting connector errors.
csadm/db.log Used for troubleshooting database externalization errors.
install
For example, 6.4.1-2133.log.
Used for troubleshooting FortiSOAR installation issues.
Install logs are named according to the FortiSOAR version and build number.
install/connectors.log Used for troubleshooting connector installation issues.
upgrade_fortisoar_<version_number>-<timestamp>.log
For example, upgrade_fortisoar_6.0.0-2020-02-18-1558604373.log
Stores upgrade console log and you can use it to troubleshoot FortiSOAR upgrade issues.
install/config-vm-<timestamp>.log
For example, install/config-vm-27_Nov_2018_18h_14m_34s.log
Used for troubleshooting FortiSOAR Configuration Wizard issues.

For troubleshooting FortiSOAR audit log issues use the auditlog.log located at /var/log/cyops/cyops-gateway/auditlog.log.

For Centos OS level errors, use the Messages logs located at /var/log/messages.

For troubleshooting issues related to dedicated tenant nodes or FortiSOAR agents, see the following logs:

  • var/log/cyops/cyops-routing-agent/postman.log
  • var/log/cyops/cyops-routing-agent/uwsgi.log
  • var/log/cyops/cyops-routing-agent/ssl_cyops_routing_agent_error.log
  • var/log/cyops/cyops-routing-agent/ssl_cyops_routing_agent_access.log.

For troubleshooting issues related to FortiMonitor, see the following logs:

  • var/log/panopta-agent/: For troubleshooting issues related to the FortiMonitor Agent
  • var/log/cyops/fmonitor/: For troubleshooting issues related to the csadm CLI for FortiMonitor.

For troubleshooting issues related to the separate runtime user for integrations:

  • var/log/cyops/cyops-integrations/integration.log
  • var/log/cyops/cyops-integrations/connectors.log
  • getfacl var/log/cyops/cyops-integrations/
  • getfacl opt/cyops-integrations/integrations
  • getfacl var/run/uwsgi
  • getfacl opt/cyops-integrations/.env

Logging Levels

You can set the following logging levels in the log files:

  • DEBUG: Low-level system information for debugging purposes.
  • INFO: General system information.
  • WARNING: Information describing a minor problem that has occurred.
  • ERROR: Information describing a major problem that has occurred.
  • CRITICAL: Information describing a critical problem that has occurred.

Changing the logging levels

  • For sealab or workflow:
    1. Open the /opt/cyops-workflow/sealab/sealab/config.ini file and set the WORKFLOW_LOG_LEVEL parameter to the required logging level. For example, WORKFLOW_LOG_LEVEL = 'INFO'
    2. Restart the uwsgi service.
  • For integrations:
    1. Open the /opt/cyops-integrations/integrations/configs/config.ini file and set the connector_logger_level parameter to the required logging level. For example, connector_logger_level= 'INFO'
    2. Restart the uwsgi service.
  • For celeryd:
    1. Open the /etc/celery/celeryd.conf file and set the CELERYD_LOG_LEVEL parameter to the required logging level. For example, CELERYD_LOG_LEVEL = 'INFO'
    2. Restart the celeryd service.
  • For nginx (UI), API, or php:
    1. Open the /opt/cyops-api/config/packages/prod/monolog.yaml file and set the level parameter to the required logging level. For example, level = 'INFO'
    2. Run the # systemctl restart php-fpm nginx command.
  • To enable all console errors in your browser, add DEBUG = true in local storage as follows:
    1. Open your browser and click select More Tools > Developer Tools.
    2. Click the Application tab and then click Storage.
    3. In Local Storage, type DEBUG as the key and true as the value.

List of key FortiSOAR services and processes

Name of Services/Processes Description
postgresql-12 Service for all application data stored in postgresql DB.
To know the status of this service, use the # systemctl status postgresql-12 command.
elasticsearch Service to bring up the elasticsearch service.
To know the status of this service, use the # systemctl status elasticsearch command.
php-fpm Service for PHP FastCGI implementation.
To know the status of this service, use the # systemctl status php-fpm command.
uwsgi Software application that aims at developing a full stack for building hosting services. uWSGI is named after the Web Server Gateway Interface. We host our playbook execution engine application and connector integrations applications on a uWSGI server.
To know the status of uwsgi use the # systemctl status uwsgi command.
celeryd celeryd is used to run the playbooks asynchronously in the FortiSOAR playbook execution engine.
To know the status of celeryd use the # systemctl status celeryd command.
celerybeatd celerybeatd is a playbook scheduler; used to kick off tasks at regular intervals, that are then executed by available worker nodes in the cluster.
To know the status of use the # systemctl status celerybeatd command.
cyops-auth Service used for FortiSOAR authentications.
To know the status of this services, use the # systemctl status cyops-auth command.
cyops-tomcat Service used for SSO, auditing, and websocket.
To know the status of this services, use the # systemctl status tomcat command.
cyops-search Service used for full-text searching, finding similar records and predicting the value of fields based on similarity.
cyops-ha Service is responsible for setting up High Availability in the FortiSOAR environment
cyops-postman Service is responsible for setting up and managing FortiSOAR's distributed multi-tenant setup.
cyops-integrations-agent Service is responsible for running actions on remote FortiSOAR agents.
rabbitmq-server Service is responsible to send audit and live sync notifications. This service is also responsible for data transfer in a distributed environment.
nginx Service used for Web UI.
To know the status of this services, use the # systemctl status nginx command.

If you want to restart, start, or stop all the services together, use FortiSOAR Admin CLI (csadm). For more information on csadm, see the FortiSOAR Admin CLI chapter.

You can run the csadm command on any FortiSOAR machine using any terminal. Any user who has root or sudo permissions can run the csadm command.

To restart FortiSOAR services, type: # csadm services --restart

To start FortiSOAR services, type: # csadm services --start

To stop FortiSOAR services, type: # csadm services --stop

To know the status of all FortiSOAR services type: # csadm services --status

To know more about monitoring services, see the Monitoring FortiSOAR chapter.

Configurations required for exporting of records with unsupported character sets in the PDF format

You can export records to PDF, by clicking the 'Hamburger' menu on the grid view and selecting the Export All Visible Columns to PDF option, or you can open the record and in the detail view of the record, and click the Export Record > Export as PDF option. The FortiSOAR UI uses the 'pdfmake' library, to export the record data from the record's grid and view panel to the PDF format. The 'pdfmake' library by default comes with the ROBOTO font that does not support Korean, Chinese, and other character sets. Therefore, to export the record data that contains unsupported character sets from the record's grid and view panel in FortiSOAR to the PDF format, you can use one of the following options:

Option 1:

Use SSH and update the /opt/cyops-ui/vendor/config.json to provide the CDN URLs for a specific font:

"pdfmake": {
    "font": {
         "normal": "https://cdnjs.cloudflare.com/ajax/libs/pdfmake/0.1.66/fonts/Roboto/Roboto-Regular.ttf",
         "bold": "https://cdnjs.cloudflare.com/ajax/libs/pdfmake/0.1.66/fonts/Roboto/Roboto-Medium.ttf",
         "italics": "https://cdnjs.cloudflare.com/ajax/libs/pdfmake/0.1.66/fonts/Roboto/Roboto-Italic.ttf",
         "bolditalics": "https://cdnjs.cloudflare.com/ajax/libs/pdfmake/0.1.66/fonts/Roboto/Roboto-MediumItalic.ttf"
    }
 }

Option 2

Build and use the custom font for 'pdfmake' as follows:

  1. Copy font files into the /opt/cyops-ui/vendor/pdfmake directory.
  2. Build the vfs_fonts.js file with new fonts. For example to build the vfs_fonts.js file with the 'Roboto' font family use the following command:
    # sh build.sh Roboto-Regular.ttf Roboto-Medium.ttf Roboto-Italic.ttf Roboto-MediumItalic.ttf
  3. Update the /opt/cyops-ui/vendor/config.json file with the font references:
    "pdfmake": {
       "font": {
         "normal": "Roboto-Regular.ttf",
         "bold": "Roboto-Medium.ttf"
         "italics": "Roboto-Italics.ttf"
         "bolditalics": "Roboto-MediumItalic.ttf"
       }
    }

    You can also refer to the /opt/cyops-ui/vendor/pdfmake/README.txt file for instructions.

Additional settings for record similarity and field predictions

FortiSOAR supports "Record Similarity" i.e., FortiSOAR displays records that are similar to the record on which you are working. FortiSOAR also supports "Record Field Value Prediction" i.e., FortiSOAR predicts values of fields of your choice within a record from the values of fields of existing records based on the criteria you have defined, making it easier for analysts to make informed decisions. For more information, see the Working with Modules - Alerts & Incidents chapter in the "User Guide."

This section provides information on how you can tune the results that are displayed by FortiSOAR for record similarity and field predictions using the following parameters in the /opt/cyops/config/cyops-search/config.yml file:

  • minimum_should_match: <percentageValue>: This setting defines that a record will be considered similar only if there is a match of at least the percentage value that you have specified on the related fields. This is especially true for similarity based on related records. For example, if you set this parameter as minimum_should_match: 10% (default), then if you have defined similarity for alerts based on related indicators, then FortiSOAR will display only those records as similar that match a minimum of 10% of the indicators. Therefore, for an alert that has 10 related indicators, FortiSOAR similarity results will display alerts that even have one common indicator; but if an alert has 20 related indicators, then FortiSOAR similarity results will display only those alerts that have at least 2 indicators in common.
  • max_query_terms: <numberOfItems>: This setting defines how many terms of the parent record will be looked up for similarity in other records. Continuing the same example as above, if you set this parameter as max_query_terms: 25 (default), then if an alert has more than 25 indicators, only 25 of them will be checked for similarity in other records. Note that increasing the value of this setting will increase the time FortiSOAR takes to return similarity and suggestion results.

For more information on the above parameters and other parameters, refer to the ElastiSearch reference at: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-mlt-query.html

Change the default value of some of the user profile parameters

An administrator with CRUD permissions on the Security module can change the default value of the following user-profile related parameters:

Parameter Name Description Default Value
max_reset_attempts Maximum number of times users' can click the Reset Password link before actually resetting their password.
If the user exceeds the value set in this parameter, then users' will not get a new link to reset their password based on the number of hours specified in the reset_locktime parameter.
By default, the max_reset_attempts is set to 10 times and the reset_locktime is set to 12 hours, therefore, if a user clicks the Reset Password 11 times without actually resetting their password, then the user will not get a new link to reset their password for 12 hours.
10 times
reset_locktime Number of hours that users' will not get a new link to reset their password if they exceed the value set in the max_reset_attempts parameter. 12 hours
max_failed Number of times that users' can enter an incorrect password, while logging into FortiSOAR, before their account gets locked.
If the user exceeds the value set in this parameter, then the user will get locked out based on the number of minutes specified in the lock_minutes parameter.
By default, the max_failed is set to 5 times and the lock_minutes is set to 30 mins, therefore, if a user enters an incorrect password 6 times, then their account gets locked for 30 mins.
5 times
lock_minutes Number of minutes that users' account gets locked if they exceed the value set in the max_failed parameter. 30 mins

To change the value of the max_reset_attempts parameter, the administrator should run the following curl command on their FortiSOAR system:

curl -X PUT \
  https://<FORTISOAR HOSTNAME/IP>/api/auth/config \
  -H 'Authorization: <Bearer Token>' \
  -d '{
	"option":"max_reset_attempts",
	"value":5
}'

The above command changes the number of times users' can click the Reset Password link to 5 times, i.e., a user can click the Reset Password link 5 times without actually setting the new password. However, if the user clicks the Reset Password link for the 6th time, the user will be blocked.

Similarly, to change the value of the reset_locktime, max_failed, and lock_minutes parameters, the administrator should run the same curl command on their FortiSOAR system, but after changing the option and value parameter values:

curl -X PUT \
  https://<FORTISOAR HOSTNAME/IP>/api/auth/config \
  -H 'Authorization: <Bearer Token>' \
  -d '{
	"option":"reset_locktime",
	"value":2
}'

The above command changes the number of hours that users' will not get a new link to reset their password to 2 hours if they have exceeded the value set in the max_reset_attempts parameter.

curl -X PUT \
  https://<FORTISOAR HOSTNAME/IP>/api/auth/config \
  -H 'Authorization: <Bearer Token>' \
  -d '{
	"option":"max_failed",
	"value":3
}'

The above command changes the number of times that users' can enter an incorrect password while logging into FortiSOAR before their account gets locked to 3, i.e., users' account will be locked if they enter an incorrect password 4 times while logging into FortiSOAR.

curl -X PUT \
  https://<FORTISOAR HOSTNAME/IP>/api/auth/config \
  -H 'Authorization: <Bearer Token>' \
  -d '{
	"option":"lock_minutes",
	"value":15
}'

The above command changes the number of minutes that users' account will be locked to 15 minutes if they have exceeded the value set in the max_reset_attempts parameter.

Security considerations for Websockets

The Websocket server allows connection from any hostname URL UI. To restrict connections to the Websocket server, you can implement Cross-Origin Resource Sharing (CORS) restrictions in WebSocket.

To implement CORS in WebSocket, you have to add a comma-separated list of allowed hosts (IP address or domain name) in the /etc/cyops/config.yml file and the hostname should start with "http://" or "https://". Dummy entries have already been added in the /etc/cyops/config.yml file to help you in this process. If you have added domain names to the /etc/cyops/config.yml file, then you must also restart the 'tomcat' service, using the systemctl restart cyops-tomcat command.

Once the hostname entries have been added, the WebSocket will only establish a connection with those UIs whose URLs match the specified hostnames.

Troubleshooting Tips

Your Workflow data size has increased

Increase in your Workflow data can cause performance bottlenecks.

Resolution:

You can purge Executed Playbook Logs using the Purge Logs button on the top-right of the Executed Playbook Logs dialog. For more information on purging, see the Debugging and Optimizing Playbooks chapter in the "Playbooks Guide."

Error displayed while performing a search operation in FortiSOAR

Resolution:

If you get any error while performing a global search in FortiSOAR, check that the elasticsearch.service and the cyops-search.service are running.

If these are not running, then start these services using the following commands:

# systemctl start elasticsearch

# systemctl start cyops-search

For more information, see the FortiSOAR Search Errors topic in the Elasticsearch Configuration chapter.

Reindexing FortiSOAR modules for search

Partial indexing of a module, or when a module does not get indexed, can lead to errors in FortiSOAR search. You can manually reindex any skipped or unsuccessfully indexed modules. For more information, see the FortiSOAR Search Errors topic in the Elasticsearch Configuration chapter.

Resolution:

To reindex all the FortiSOAR modules, run the following command:

$ sudo -u nginx php /opt/cyops-api/bin/console app:elastic:create --env="prod"

To reindex specific FortiSOAR modules, run the following command:

$ sudo -u nginx php /opt/cyops-api/bin/console app:elastic:create --env="prod" --index='{"type":"type of the module(s)"}'

For example:

$ sudo -u nginx php /opt/cyops-api/bin/console app:elastic:create --env="prod" --index='{"type":"indicators", "tasks"}

FortiSOAR crashing with "out of memory" errors

By default, FortiSOAR configures Elaticsearch to use 4 GB of RAM. If there are too many records or any very heavy records (such as large files uploaded) created per day on the system, it might crash with "out of memory" errors. To fix this, you must increase the memory allocated to Elasticsearch.

Resolution

  1. Change the following entry in /etc/elasticsearch/jvm.options to a higher value based on memory available on your server:
    -Xms4g
    -Xmx4g
  2. Restart Elasticsearch using the following command:
    systemctl restart elasticsearch

Changing Postgres worker memory

When the primary data in the system becomes large (eg, over million alerts) and you notice that the system is slow to respond. The slowness could be caused due to database queries taking longer with the increased database size.

Resolution

You can fine tune this behavior by increasing the following Postgres settings based on the available free memory on the system:

  1. Increase the shared buffer and worker memory in the /var/lib/pgsql/12/data/postgresql.conf file: shared_buffers = 2048MB
    work_mem = 16MB
  2. Restart Postgres using the following command:
    systemctl restart postgresql-12

Changing the maximum number of records that can be linked in one call

The maximum number of records that can be linked in one call should be 99.

To change this limit, you can change the max_relation_count parameter value in the /opt/cyops-api/configs/parameters_prod.yml file:

# max numnber of relations to normalize
max_relation_count: 100

Crashing of the Tomcat server when the 'Detailed' mode for Log Forwarding is enabled

If you have enabled the 'Detailed' mode for Log Forwarding in Settings > Log Forwarding > Audit Logs, then your Tomcat server might crash and the 'Live Sync' might continuously fail.

Resolution

To avoid crashing of your Tomcat server and you do not want to increase the memory assigned to the Tomcat service, you need to disable log forwarding and then set log forwarding to the 'Basic' mode as follows:

  1. SSH to your FortiSOAR server and login as a root user.
  2. List the current log forward settings using the following command:
    # csadm log forward show-config
  3. Copy the UUID from the output of the show-config command.
  4. Disable forwarding of the audit logs to the syslog server using the following command:
    # csadm log forward update-config --uuid <uuid> --filter application
    or
    # csadm log forward update-config --uuid <uuid> --filter none
  5. Restart the Tomcat service using the following command:
    # systemctl restart cyops-tomcat
  6. Log on to FortiSOAR, and click Settings > System Configuration.
  7. Click the Log Forwarding tab.
  8. From the Specify Audit Log Detail Level drop-down list, select Basic, and click Save.

If you for some reason you require to use the 'Detailed' mode for log forwarding, you can increase the memory assigned to the Tomcat service running the log forwarding process using the following steps:

  1. In /usr/lib/systemd/system/cyops-tomcat.service change the Xms value as follows:
    Environment="CATALINA_OPTS=-Xms4096M -Xmx4096M -server -XX:+UseParallelGC"
  2. Run systemctl daemon-reload
  3. Run systemctl restart cyops-tomcat

FortiSOAR displaying errors such as app.ERROR: Unable to load API credentials from cache or DAS

FortiSOAR might display errors such as:
[2022-03-10 22:58:21] app.ERROR: Unable to load API credentials from cache or DAS. [] []
[2022-03-10 22:58:21] app.ERROR: /opt/cyops-api/src/Encoder/Hmac/DasHmacSecretKeyProvider.php at line 114 [] []

If you have not regenerated the default FortiSOAR self-signed certificates for more than one year.

Resolution

To resolve this issue, run the following command as a root user (using 'sudo su' and using the csadmin password) using a SSH session:
csadm certs --generate `hostname`

Once this command is run successfully, you require to restart all services using the following command:
csadm services --restart

Recommendation Engine does not work and displays an "ML Service not running" error

If you have upgraded FortiSOAR or due to some other reason, you might observe that the Recommendation Engine is not working and not displaying any suggestions. You can also get an error such as "ML Service is not running." This issue can occur if the ml service fails to start on port 10449.

Resolution

Restart the uwsgi service and you will observe that the ml service is running on port 10449. Once the ml service starts, you will observe that the recommendation engine begins to work and displays appropriate suggestions like phishing classification.

Debugging, Troubleshooting, and Optimizing FortiSOAR

Administrators can use various logs that FortiSOAR generates to troubleshoot FortiSOAR issues. This chapter lists the key FortiSOAR services and processes and also provides some troubleshooting tips. This chapter also provides some additional configuration settings so that you can tune the results that get displayed by FortiSOAR for record similarity and field prediction. For more information on record similarity and field prediction, see the Working with Modules - Alerts & Incidents chapter in the "User Guide."

If you face any issues while deploying or upgrading FortiSOAR, see the Troubleshooting FortiSOAR chapter in the "Deployment Guide." If you face deployment or upgrade failures due to insufficient space, or if you face issues while using FortiSOAR that might be caused due to insufficient space, like you are unable to log into FortiSOAR or FortiSOAR services stop working, then see the Issues occurring in FortiSOAR due to insufficient space section in the Troubleshooting FortiSOAR chapter in the "Deployment Guide."

For information about monitoring various important aspects of your FortiSOAR system, see the Monitoring FortiSOAR chapter.

List of logs used for troubleshooting FortiSOAR

FortiSOAR log files are stored in the following location: /var/log/cyops. You will find the following directories in the /var/log/cyops location:

Log Name Purpose
cyops-api/ssl_cyops_api_access.log Used for troubleshooting web (nginx) UI or API access issues.
cyops-api/ssl_cyops_api_error.log Used for troubleshooting API errors.
cyops-api/prod.log Used for troubleshooting FortiSOAR PHP related issues.
cyops-api/last_system_publish.log Used for troubleshooting publishing issues. It captures the output when a publish is fired from the UI after MMD changes.
cyops-auth/das.log Used for troubleshooting FortiSOAR authentication issues.
cyops-auth/fdn.log Used for troubleshooting FortiSOAR license synchronization issues with FortiGuard Distribution Network (FDN).
cyops-auth/ha.log Used for troubleshooting FortiSOAR High Availability issues.
cyops-gateway/auditlog.log Used for troubleshooting FortiSOAR audit log issues.
cyops-gateway/saml.log Used for troubleshooting FortiSOAR SAML issues.
cyops-search/falcon.log Used for troubleshooting FortiSOAR Search issues. If you get any error when you are indexing or searching for a record in FortiSOAR, you can use the falcon.log file to troubleshoot ElasticSearch issues.
This log is also used for checking the status of ElasticSearch indexing.
cyops-gateway/gateway.log Used for troubleshooting FortiSOAR Gateway issues such as, audit log page failing to load or the SSO configuration page failing to load.
cyops-notifier/notifier.log Used for troubleshooting FortiSOAR Web Socket issues.
cyops-workflow/beat.log Used for troubleshooting issues of the FortiSOAR Scheduler.
cyops-workflow/celeryd.log Used for troubleshooting FortiSOAR playbook runtime issues.
cyops-workflow/sealab.log Used for troubleshooting FortiSOAR playbook framework issues.
cyops-workflow/ssl_cyops_workflow_access.log Used for troubleshooting playbook access issues.
cyops-workflow/ssl_cyops_workflow_error.log Used for troubleshooting playbook errors.
cyops-workflow/uwsgi.log Used for troubleshooting FortiSOAR playbook and connector issues.
cyops-integrations/connectors.log Used for troubleshooting FortiSOAR connector related issues.
cyops-integrations/integrations.log Used for troubleshooting FortiSOAR connector framework issues.
cyops-integrations/integrations/imap/listener.log Used for troubleshooting the IMAP connector issues.
cyops-integrations/ssl_cyops_integrations_access.log Used for troubleshooting connector access issues.
cyops-integrations/ssl_cyops_integrations_error.log Used for troubleshooting connector errors.
csadm/db.log Used for troubleshooting database externalization errors.
install
For example, 6.4.1-2133.log.
Used for troubleshooting FortiSOAR installation issues.
Install logs are named according to the FortiSOAR version and build number.
install/connectors.log Used for troubleshooting connector installation issues.
upgrade_fortisoar_<version_number>-<timestamp>.log
For example, upgrade_fortisoar_6.0.0-2020-02-18-1558604373.log
Stores upgrade console log and you can use it to troubleshoot FortiSOAR upgrade issues.
install/config-vm-<timestamp>.log
For example, install/config-vm-27_Nov_2018_18h_14m_34s.log
Used for troubleshooting FortiSOAR Configuration Wizard issues.

For troubleshooting FortiSOAR audit log issues use the auditlog.log located at /var/log/cyops/cyops-gateway/auditlog.log.

For Centos OS level errors, use the Messages logs located at /var/log/messages.

For troubleshooting issues related to dedicated tenant nodes or FortiSOAR agents, see the following logs:

  • var/log/cyops/cyops-routing-agent/postman.log
  • var/log/cyops/cyops-routing-agent/uwsgi.log
  • var/log/cyops/cyops-routing-agent/ssl_cyops_routing_agent_error.log
  • var/log/cyops/cyops-routing-agent/ssl_cyops_routing_agent_access.log.

For troubleshooting issues related to FortiMonitor, see the following logs:

  • var/log/panopta-agent/: For troubleshooting issues related to the FortiMonitor Agent
  • var/log/cyops/fmonitor/: For troubleshooting issues related to the csadm CLI for FortiMonitor.

For troubleshooting issues related to the separate runtime user for integrations:

  • var/log/cyops/cyops-integrations/integration.log
  • var/log/cyops/cyops-integrations/connectors.log
  • getfacl var/log/cyops/cyops-integrations/
  • getfacl opt/cyops-integrations/integrations
  • getfacl var/run/uwsgi
  • getfacl opt/cyops-integrations/.env

Logging Levels

You can set the following logging levels in the log files:

  • DEBUG: Low-level system information for debugging purposes.
  • INFO: General system information.
  • WARNING: Information describing a minor problem that has occurred.
  • ERROR: Information describing a major problem that has occurred.
  • CRITICAL: Information describing a critical problem that has occurred.

Changing the logging levels

  • For sealab or workflow:
    1. Open the /opt/cyops-workflow/sealab/sealab/config.ini file and set the WORKFLOW_LOG_LEVEL parameter to the required logging level. For example, WORKFLOW_LOG_LEVEL = 'INFO'
    2. Restart the uwsgi service.
  • For integrations:
    1. Open the /opt/cyops-integrations/integrations/configs/config.ini file and set the connector_logger_level parameter to the required logging level. For example, connector_logger_level= 'INFO'
    2. Restart the uwsgi service.
  • For celeryd:
    1. Open the /etc/celery/celeryd.conf file and set the CELERYD_LOG_LEVEL parameter to the required logging level. For example, CELERYD_LOG_LEVEL = 'INFO'
    2. Restart the celeryd service.
  • For nginx (UI), API, or php:
    1. Open the /opt/cyops-api/config/packages/prod/monolog.yaml file and set the level parameter to the required logging level. For example, level = 'INFO'
    2. Run the # systemctl restart php-fpm nginx command.
  • To enable all console errors in your browser, add DEBUG = true in local storage as follows:
    1. Open your browser and click select More Tools > Developer Tools.
    2. Click the Application tab and then click Storage.
    3. In Local Storage, type DEBUG as the key and true as the value.

List of key FortiSOAR services and processes

Name of Services/Processes Description
postgresql-12 Service for all application data stored in postgresql DB.
To know the status of this service, use the # systemctl status postgresql-12 command.
elasticsearch Service to bring up the elasticsearch service.
To know the status of this service, use the # systemctl status elasticsearch command.
php-fpm Service for PHP FastCGI implementation.
To know the status of this service, use the # systemctl status php-fpm command.
uwsgi Software application that aims at developing a full stack for building hosting services. uWSGI is named after the Web Server Gateway Interface. We host our playbook execution engine application and connector integrations applications on a uWSGI server.
To know the status of uwsgi use the # systemctl status uwsgi command.
celeryd celeryd is used to run the playbooks asynchronously in the FortiSOAR playbook execution engine.
To know the status of celeryd use the # systemctl status celeryd command.
celerybeatd celerybeatd is a playbook scheduler; used to kick off tasks at regular intervals, that are then executed by available worker nodes in the cluster.
To know the status of use the # systemctl status celerybeatd command.
cyops-auth Service used for FortiSOAR authentications.
To know the status of this services, use the # systemctl status cyops-auth command.
cyops-tomcat Service used for SSO, auditing, and websocket.
To know the status of this services, use the # systemctl status tomcat command.
cyops-search Service used for full-text searching, finding similar records and predicting the value of fields based on similarity.
cyops-ha Service is responsible for setting up High Availability in the FortiSOAR environment
cyops-postman Service is responsible for setting up and managing FortiSOAR's distributed multi-tenant setup.
cyops-integrations-agent Service is responsible for running actions on remote FortiSOAR agents.
rabbitmq-server Service is responsible to send audit and live sync notifications. This service is also responsible for data transfer in a distributed environment.
nginx Service used for Web UI.
To know the status of this services, use the # systemctl status nginx command.

If you want to restart, start, or stop all the services together, use FortiSOAR Admin CLI (csadm). For more information on csadm, see the FortiSOAR Admin CLI chapter.

You can run the csadm command on any FortiSOAR machine using any terminal. Any user who has root or sudo permissions can run the csadm command.

To restart FortiSOAR services, type: # csadm services --restart

To start FortiSOAR services, type: # csadm services --start

To stop FortiSOAR services, type: # csadm services --stop

To know the status of all FortiSOAR services type: # csadm services --status

To know more about monitoring services, see the Monitoring FortiSOAR chapter.

Configurations required for exporting of records with unsupported character sets in the PDF format

You can export records to PDF, by clicking the 'Hamburger' menu on the grid view and selecting the Export All Visible Columns to PDF option, or you can open the record and in the detail view of the record, and click the Export Record > Export as PDF option. The FortiSOAR UI uses the 'pdfmake' library, to export the record data from the record's grid and view panel to the PDF format. The 'pdfmake' library by default comes with the ROBOTO font that does not support Korean, Chinese, and other character sets. Therefore, to export the record data that contains unsupported character sets from the record's grid and view panel in FortiSOAR to the PDF format, you can use one of the following options:

Option 1:

Use SSH and update the /opt/cyops-ui/vendor/config.json to provide the CDN URLs for a specific font:

"pdfmake": {
    "font": {
         "normal": "https://cdnjs.cloudflare.com/ajax/libs/pdfmake/0.1.66/fonts/Roboto/Roboto-Regular.ttf",
         "bold": "https://cdnjs.cloudflare.com/ajax/libs/pdfmake/0.1.66/fonts/Roboto/Roboto-Medium.ttf",
         "italics": "https://cdnjs.cloudflare.com/ajax/libs/pdfmake/0.1.66/fonts/Roboto/Roboto-Italic.ttf",
         "bolditalics": "https://cdnjs.cloudflare.com/ajax/libs/pdfmake/0.1.66/fonts/Roboto/Roboto-MediumItalic.ttf"
    }
 }

Option 2

Build and use the custom font for 'pdfmake' as follows:

  1. Copy font files into the /opt/cyops-ui/vendor/pdfmake directory.
  2. Build the vfs_fonts.js file with new fonts. For example to build the vfs_fonts.js file with the 'Roboto' font family use the following command:
    # sh build.sh Roboto-Regular.ttf Roboto-Medium.ttf Roboto-Italic.ttf Roboto-MediumItalic.ttf
  3. Update the /opt/cyops-ui/vendor/config.json file with the font references:
    "pdfmake": {
       "font": {
         "normal": "Roboto-Regular.ttf",
         "bold": "Roboto-Medium.ttf"
         "italics": "Roboto-Italics.ttf"
         "bolditalics": "Roboto-MediumItalic.ttf"
       }
    }

    You can also refer to the /opt/cyops-ui/vendor/pdfmake/README.txt file for instructions.

Additional settings for record similarity and field predictions

FortiSOAR supports "Record Similarity" i.e., FortiSOAR displays records that are similar to the record on which you are working. FortiSOAR also supports "Record Field Value Prediction" i.e., FortiSOAR predicts values of fields of your choice within a record from the values of fields of existing records based on the criteria you have defined, making it easier for analysts to make informed decisions. For more information, see the Working with Modules - Alerts & Incidents chapter in the "User Guide."

This section provides information on how you can tune the results that are displayed by FortiSOAR for record similarity and field predictions using the following parameters in the /opt/cyops/config/cyops-search/config.yml file:

  • minimum_should_match: <percentageValue>: This setting defines that a record will be considered similar only if there is a match of at least the percentage value that you have specified on the related fields. This is especially true for similarity based on related records. For example, if you set this parameter as minimum_should_match: 10% (default), then if you have defined similarity for alerts based on related indicators, then FortiSOAR will display only those records as similar that match a minimum of 10% of the indicators. Therefore, for an alert that has 10 related indicators, FortiSOAR similarity results will display alerts that even have one common indicator; but if an alert has 20 related indicators, then FortiSOAR similarity results will display only those alerts that have at least 2 indicators in common.
  • max_query_terms: <numberOfItems>: This setting defines how many terms of the parent record will be looked up for similarity in other records. Continuing the same example as above, if you set this parameter as max_query_terms: 25 (default), then if an alert has more than 25 indicators, only 25 of them will be checked for similarity in other records. Note that increasing the value of this setting will increase the time FortiSOAR takes to return similarity and suggestion results.

For more information on the above parameters and other parameters, refer to the ElastiSearch reference at: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-mlt-query.html

Change the default value of some of the user profile parameters

An administrator with CRUD permissions on the Security module can change the default value of the following user-profile related parameters:

Parameter Name Description Default Value
max_reset_attempts Maximum number of times users' can click the Reset Password link before actually resetting their password.
If the user exceeds the value set in this parameter, then users' will not get a new link to reset their password based on the number of hours specified in the reset_locktime parameter.
By default, the max_reset_attempts is set to 10 times and the reset_locktime is set to 12 hours, therefore, if a user clicks the Reset Password 11 times without actually resetting their password, then the user will not get a new link to reset their password for 12 hours.
10 times
reset_locktime Number of hours that users' will not get a new link to reset their password if they exceed the value set in the max_reset_attempts parameter. 12 hours
max_failed Number of times that users' can enter an incorrect password, while logging into FortiSOAR, before their account gets locked.
If the user exceeds the value set in this parameter, then the user will get locked out based on the number of minutes specified in the lock_minutes parameter.
By default, the max_failed is set to 5 times and the lock_minutes is set to 30 mins, therefore, if a user enters an incorrect password 6 times, then their account gets locked for 30 mins.
5 times
lock_minutes Number of minutes that users' account gets locked if they exceed the value set in the max_failed parameter. 30 mins

To change the value of the max_reset_attempts parameter, the administrator should run the following curl command on their FortiSOAR system:

curl -X PUT \
  https://<FORTISOAR HOSTNAME/IP>/api/auth/config \
  -H 'Authorization: <Bearer Token>' \
  -d '{
	"option":"max_reset_attempts",
	"value":5
}'

The above command changes the number of times users' can click the Reset Password link to 5 times, i.e., a user can click the Reset Password link 5 times without actually setting the new password. However, if the user clicks the Reset Password link for the 6th time, the user will be blocked.

Similarly, to change the value of the reset_locktime, max_failed, and lock_minutes parameters, the administrator should run the same curl command on their FortiSOAR system, but after changing the option and value parameter values:

curl -X PUT \
  https://<FORTISOAR HOSTNAME/IP>/api/auth/config \
  -H 'Authorization: <Bearer Token>' \
  -d '{
	"option":"reset_locktime",
	"value":2
}'

The above command changes the number of hours that users' will not get a new link to reset their password to 2 hours if they have exceeded the value set in the max_reset_attempts parameter.

curl -X PUT \
  https://<FORTISOAR HOSTNAME/IP>/api/auth/config \
  -H 'Authorization: <Bearer Token>' \
  -d '{
	"option":"max_failed",
	"value":3
}'

The above command changes the number of times that users' can enter an incorrect password while logging into FortiSOAR before their account gets locked to 3, i.e., users' account will be locked if they enter an incorrect password 4 times while logging into FortiSOAR.

curl -X PUT \
  https://<FORTISOAR HOSTNAME/IP>/api/auth/config \
  -H 'Authorization: <Bearer Token>' \
  -d '{
	"option":"lock_minutes",
	"value":15
}'

The above command changes the number of minutes that users' account will be locked to 15 minutes if they have exceeded the value set in the max_reset_attempts parameter.

Security considerations for Websockets

The Websocket server allows connection from any hostname URL UI. To restrict connections to the Websocket server, you can implement Cross-Origin Resource Sharing (CORS) restrictions in WebSocket.

To implement CORS in WebSocket, you have to add a comma-separated list of allowed hosts (IP address or domain name) in the /etc/cyops/config.yml file and the hostname should start with "http://" or "https://". Dummy entries have already been added in the /etc/cyops/config.yml file to help you in this process. If you have added domain names to the /etc/cyops/config.yml file, then you must also restart the 'tomcat' service, using the systemctl restart cyops-tomcat command.

Once the hostname entries have been added, the WebSocket will only establish a connection with those UIs whose URLs match the specified hostnames.

Troubleshooting Tips

Your Workflow data size has increased

Increase in your Workflow data can cause performance bottlenecks.

Resolution:

You can purge Executed Playbook Logs using the Purge Logs button on the top-right of the Executed Playbook Logs dialog. For more information on purging, see the Debugging and Optimizing Playbooks chapter in the "Playbooks Guide."

Error displayed while performing a search operation in FortiSOAR

Resolution:

If you get any error while performing a global search in FortiSOAR, check that the elasticsearch.service and the cyops-search.service are running.

If these are not running, then start these services using the following commands:

# systemctl start elasticsearch

# systemctl start cyops-search

For more information, see the FortiSOAR Search Errors topic in the Elasticsearch Configuration chapter.

Reindexing FortiSOAR modules for search

Partial indexing of a module, or when a module does not get indexed, can lead to errors in FortiSOAR search. You can manually reindex any skipped or unsuccessfully indexed modules. For more information, see the FortiSOAR Search Errors topic in the Elasticsearch Configuration chapter.

Resolution:

To reindex all the FortiSOAR modules, run the following command:

$ sudo -u nginx php /opt/cyops-api/bin/console app:elastic:create --env="prod"

To reindex specific FortiSOAR modules, run the following command:

$ sudo -u nginx php /opt/cyops-api/bin/console app:elastic:create --env="prod" --index='{"type":"type of the module(s)"}'

For example:

$ sudo -u nginx php /opt/cyops-api/bin/console app:elastic:create --env="prod" --index='{"type":"indicators", "tasks"}

FortiSOAR crashing with "out of memory" errors

By default, FortiSOAR configures Elaticsearch to use 4 GB of RAM. If there are too many records or any very heavy records (such as large files uploaded) created per day on the system, it might crash with "out of memory" errors. To fix this, you must increase the memory allocated to Elasticsearch.

Resolution

  1. Change the following entry in /etc/elasticsearch/jvm.options to a higher value based on memory available on your server:
    -Xms4g
    -Xmx4g
  2. Restart Elasticsearch using the following command:
    systemctl restart elasticsearch

Changing Postgres worker memory

When the primary data in the system becomes large (eg, over million alerts) and you notice that the system is slow to respond. The slowness could be caused due to database queries taking longer with the increased database size.

Resolution

You can fine tune this behavior by increasing the following Postgres settings based on the available free memory on the system:

  1. Increase the shared buffer and worker memory in the /var/lib/pgsql/12/data/postgresql.conf file: shared_buffers = 2048MB
    work_mem = 16MB
  2. Restart Postgres using the following command:
    systemctl restart postgresql-12

Changing the maximum number of records that can be linked in one call

The maximum number of records that can be linked in one call should be 99.

To change this limit, you can change the max_relation_count parameter value in the /opt/cyops-api/configs/parameters_prod.yml file:

# max numnber of relations to normalize
max_relation_count: 100

Crashing of the Tomcat server when the 'Detailed' mode for Log Forwarding is enabled

If you have enabled the 'Detailed' mode for Log Forwarding in Settings > Log Forwarding > Audit Logs, then your Tomcat server might crash and the 'Live Sync' might continuously fail.

Resolution

To avoid crashing of your Tomcat server and you do not want to increase the memory assigned to the Tomcat service, you need to disable log forwarding and then set log forwarding to the 'Basic' mode as follows:

  1. SSH to your FortiSOAR server and login as a root user.
  2. List the current log forward settings using the following command:
    # csadm log forward show-config
  3. Copy the UUID from the output of the show-config command.
  4. Disable forwarding of the audit logs to the syslog server using the following command:
    # csadm log forward update-config --uuid <uuid> --filter application
    or
    # csadm log forward update-config --uuid <uuid> --filter none
  5. Restart the Tomcat service using the following command:
    # systemctl restart cyops-tomcat
  6. Log on to FortiSOAR, and click Settings > System Configuration.
  7. Click the Log Forwarding tab.
  8. From the Specify Audit Log Detail Level drop-down list, select Basic, and click Save.

If you for some reason you require to use the 'Detailed' mode for log forwarding, you can increase the memory assigned to the Tomcat service running the log forwarding process using the following steps:

  1. In /usr/lib/systemd/system/cyops-tomcat.service change the Xms value as follows:
    Environment="CATALINA_OPTS=-Xms4096M -Xmx4096M -server -XX:+UseParallelGC"
  2. Run systemctl daemon-reload
  3. Run systemctl restart cyops-tomcat

FortiSOAR displaying errors such as app.ERROR: Unable to load API credentials from cache or DAS

FortiSOAR might display errors such as:
[2022-03-10 22:58:21] app.ERROR: Unable to load API credentials from cache or DAS. [] []
[2022-03-10 22:58:21] app.ERROR: /opt/cyops-api/src/Encoder/Hmac/DasHmacSecretKeyProvider.php at line 114 [] []

If you have not regenerated the default FortiSOAR self-signed certificates for more than one year.

Resolution

To resolve this issue, run the following command as a root user (using 'sudo su' and using the csadmin password) using a SSH session:
csadm certs --generate `hostname`

Once this command is run successfully, you require to restart all services using the following command:
csadm services --restart

Recommendation Engine does not work and displays an "ML Service not running" error

If you have upgraded FortiSOAR or due to some other reason, you might observe that the Recommendation Engine is not working and not displaying any suggestions. You can also get an error such as "ML Service is not running." This issue can occur if the ml service fails to start on port 10449.

Resolution

Restart the uwsgi service and you will observe that the ml service is running on port 10449. Once the ml service starts, you will observe that the recommendation engine begins to work and displays appropriate suggestions like phishing classification.