Debugging, Troubleshooting, and optimizing FortiSOAR
Administrators can use various logs that FortiSOAR generates to troubleshoot FortiSOAR issues. This chapter lists the key FortiSOAR services and processes and also provides some troubleshooting tips. This chapter also provides some additional configuration settings so that you can tune the results that get displayed by FortiSOAR for record similarity and field prediction. For more information on record similarity and field prediction, see the Working with Modules - Alerts & Incidents chapter in the "User Guide."
If you face any issues while deploying or upgrading FortiSOAR, see the Troubleshooting FortiSOAR chapter in the "Deployment Guide." If you face deployment or upgrade failures due to insufficient space, or if you face issues while using FortiSOAR that might be caused due to insufficient space, like you are unable to log into FortiSOAR or FortiSOAR services stop working, then see the Issues occurring in FortiSOAR due to insufficient space
section in the Troubleshooting FortiSOAR chapter in the "Deployment Guide."
For information about monitoring various important aspects of your FortiSOAR system, see the Monitoring FortiSOAR chapter.
List of logs used for troubleshooting FortiSOAR
FortiSOAR log files are stored in the following location: /var/log/cyops
. You will find the following directories in the /var/log/cyops
location:
Log Name | Purpose |
---|---|
cyops-api/ssl_cyops_api_access.log
|
Used for troubleshooting web (nginx) UI or API access issues. |
cyops-api/ssl_cyops_api_error.log
|
Used for troubleshooting API errors. |
cyops-api/prod.log
|
Used for troubleshooting FortiSOAR PHP related issues. |
cyops-api/last_system_publish.log
|
Used for troubleshooting publishing issues. It captures the output when a publish is fired from the UI after MMD changes. |
cyops-auth/das.log
|
Used for troubleshooting FortiSOAR authentication issues. |
cyops-auth/fdn.log
|
Used for troubleshooting FortiSOAR license synchronization issues with FortiGuard Distribution Network (FDN). |
cyops-auth/ha.log
|
Used for troubleshooting FortiSOAR High Availability issues. |
cyops-gateway/auditlog.log
|
Used for troubleshooting FortiSOAR audit log issues. |
cyops-gateway/etl.log
|
Used for troubleshooting FortiSOAR application and system-level issues. |
cyops-gateway/saml.log
|
Used for troubleshooting FortiSOAR SAML issues. |
cyops-search/falcon.log
|
Used for troubleshooting FortiSOAR Search issues. If you get any error when you are indexing or searching for a record in FortiSOAR, you can use the falcon.log file to troubleshoot ElasticSearch issues. This log is also used for checking the status of ElasticSearch indexing. |
cyops-gateway/gateway.log
|
Used for troubleshooting FortiSOAR Gateway issues such as, audit log page failing to load or the SSO configuration page failing to load. |
cyops-notifier/notifier.log
|
Used for troubleshooting FortiSOAR Web Socket issues. |
cyops-workflow/beat.log
|
Used for troubleshooting issues of the FortiSOAR Scheduler. |
cyops-workflow/celeryd.log
|
Used for troubleshooting FortiSOAR playbook runtime issues. |
cyops-workflow/sealab.log
|
Used for troubleshooting FortiSOAR playbook framework issues. |
cyops-workflow/ssl_cyops_workflow_access.log
|
Used for troubleshooting playbook access issues. |
cyops-workflow/ssl_cyops_workflow_error.log
|
Used for troubleshooting playbook errors. |
cyops-workflow/uwsgi.log
|
Used for troubleshooting FortiSOAR playbook and connector issues. |
cyops-integrations/connectors.log
|
Used for troubleshooting FortiSOAR connector related issues. |
cyops-integrations/integrations.log
|
Used for troubleshooting FortiSOAR connector framework issues. |
cyops-integrations/integrations/imap/listener.log
|
Used for troubleshooting the IMAP connector issues. |
cyops-integrations/ssl_cyops_integrations_access.log
|
Used for troubleshooting connector access issues. |
cyops-integrations/ssl_cyops_integrations_error.log
|
Used for troubleshooting connector errors. |
csadm/db.log
|
Used for troubleshooting database externalization errors. |
install
For example, 6.4.1-2133.log . |
Used for troubleshooting FortiSOAR installation issues. Install logs are named according to the FortiSOAR version and build number. |
install/connectors.log
|
Used for troubleshooting connector installation issues. |
upgrade_fortisoar_<version_number>-<timestamp>.log
For example, upgrade_fortisoar_6.0.0-2020-02-18-1558604373.log
|
Stores upgrade console log and you can use it to troubleshoot FortiSOAR upgrade issues. |
install/config-vm-<timestamp>.log
For example, install/config-vm-27_Nov_2018_18h_14m_34s.log
|
Used for troubleshooting FortiSOAR Configuration Wizard issues. |
For troubleshooting FortiSOAR audit log issues use the tomcat.log
located at /opt/cyops-tomcat/tomcat.log
.
For Centos OS level errors, use the Messages
logs located at /var/log/messages
.
For troubleshooting issues related to dedicated tenant nodes or FortiSOAR agents, see the following logs:var/log/cyops/cyops-routing-agent/postman.log
, var/log/cyops/cyops-routing-agent/uwsgi.log
, var/log/cyops/cyops-routing-agent/ssl_cyops_routing_agent_error.log
, and var/log/cyops/cyops-routing-agent/ssl_cyops_routing_agent_access.log
.
Logging Levels
You can set the following logging levels in the log files:
- DEBUG: Low-level system information for debugging purposes.
- INFO: General system information.
- WARNING: Information describing a minor problem that has occurred.
- ERROR: Information describing a major problem that has occurred.
- CRITICAL: Information describing a critical problem that has occurred.
Changing the logging levels
- For sealab or workflow:
- Open the
/opt/cyops-workflow/sealab/sealab/config.ini
file and set theWORKFLOW_LOG_LEVEL
parameter to the required logging level. For example,WORKFLOW_LOG_LEVEL = 'INFO'
- Restart the
uwsgi
service.
- Open the
- For integrations:
- Open the
/opt/cyops-integrations/integrations/configs/config.ini
file and set theconnector_logger_level
parameter to the required logging level. For example,connector_logger_level= 'INFO'
- Restart the
uwsgi
service.
- Open the
- For celeryd:
- Open the
/etc/celery/celeryd.conf
file and set theCELERYD_LOG_LEVEL
parameter to the required logging level. For example,CELERYD_LOG_LEVEL = 'INFO'
- Restart the
celeryd
service.
- Open the
- For nginx (UI), API, or php:
- Open the
/opt/cyops-api/app/config/config_prod.yml
file and set thelevel
parameter to the required logging level. For example,level = 'INFO'
- Run the
# systemctl restart php-fpm nginx
command.
- Open the
List of key FortiSOAR services and processes
Name of Services/Processes | Description |
---|---|
postgresql-12 | Service for all application data stored in postgresql DB. To know the status of this service, use the # systemctl status postgresql-12 command. |
elasticsearch | Service to bring up the elasticsearch service. To know the status of this service, use the # systemctl status elasticsearch command. |
php-fpm | Service for PHP FastCGI implementation. To know the status of this service, use the # systemctl status php-fpm command. |
uwsgi | Software application that aims at developing a full stack for building hosting services. uWSGI is named after the Web Server Gateway Interface. We host our playbook execution engine application and connector integrations applications on a uWSGI server. To know the status of uwsgi use the # systemctl status uwsgi command. |
celeryd | celeryd is used to run the playbooks asynchronously in the FortiSOAR playbook execution engine. To know the status of celeryd use the # systemctl status celeryd command. |
celerybeatd | celerybeatd is a playbook scheduler; used to kick off tasks at regular intervals, that are then executed by available worker nodes in the cluster. To know the status of use the # systemctl status celerybeatd command. |
cyops-auth | Service used for FortiSOAR authentications. To know the status of this services, use the # systemctl status cyops-auth command. |
cyops-tomcat | Service used for SSO, auditing, and websocket. To know the status of this services, use the # systemctl status tomcat command. |
cyops-search | Service used for full-text searching, finding similar records and predicting the value of fields based on similarity. |
cyops-ha | Service is responsible for setting up High Availability in the FortiSOAR environment |
cyops-postman | Service is responsible for setting up and managing FortiSOAR's distributed multi-tenant setup. |
cyops-integrations-agent | Service is responsible for running actions on remote FortiSOAR agents. |
rabbitmq-server | Service is responsible to send audit and live sync notifications. This service is also responsible for data transfer in a distributed environment. |
nginx | Service used for Web UI. To know the status of this services, use the # systemctl status nginx command. |
If you want to restart, start, or stop all the services together, use FortiSOAR Admin CLI (csadm
). For more information on csadm
, see the FortiSOAR Admin CLI chapter.
You can run the csadm
command on any FortiSOAR machine using any terminal. Any user who has root
or sudo
permissions can run the csadm
command.
To restart FortiSOAR services, type: # csadm services --restart
To start FortiSOAR services, type: # csadm services --start
To stop FortiSOAR services, type: # csadm services --stop
To know the status of all FortiSOAR services type: # csadm services --status
To know more about monitoring services, see the Monitoring FortiSOAR chapter.
Additional settings for record similarity and field predictions
FortiSOAR supports "Record Similarity" i.e., FortiSOAR displays records that are similar to the record on which you are working. FortiSOAR also supports "Record Field Value Prediction" i.e., FortiSOAR predicts values of fields of your choice within a record from the values of fields of existing records based on the criteria you have defined, making it easier for analysts to make informed decisions. For more information, see the Working with Modules - Alerts & Incidents chapter in the "User Guide."
This section provides information on how you can tune the results that are displayed by FortiSOAR for record similarity and field predictions using the following parameters in the /opt/cyops/config/cyops-search/config.yml
file:
-
minimum_should_match
:<percentageValue>
: This setting defines that a record will be considered similar only if there is a match of at least the percentage value that you have specified on the related fields. This is especially true for similarity based on related records. For example, if you set this parameter asminimum_should_match
:10%
(default), then if you have defined similarity for alerts based on related indicators, then FortiSOAR will display only those records as similar that match a minimum of 10% of the indicators. Therefore, for an alert that has 10 related indicators, FortiSOAR similarity results will display alerts that even have one common indicator; but if an alert has 20 related indicators, then FortiSOAR similarity results will display only those alerts that have at least 2 indicators in common. -
max_query_terms
:<numberOfItems>
: This setting defines how many terms of the parent record will be looked up for similarity in other records. Continuing the same example as above, if you set this parameter asmax_query_terms
:25
(default), then if an alert has more than 25 indicators, only 25 of them will be checked for similarity in other records. Note that increasing the value of this setting will increase the time FortiSOAR takes to return similarity and suggestion results.
For more information on the above parameters and other parameters, refer to the ElastiSearch reference at: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-mlt-query.html
Change the default value of some of the user profile parameters
An administrator with CRUD permissions on the Security
module can change the default value of the following user-profile related parameters:
Parameter Name | Description | Default Value |
---|---|---|
max_reset_attempts
|
Maximum number of times users' can click the Reset Password link before actually resetting their password. If the user exceeds the value set in this parameter, then users' will not get a new link to reset their password based on the number of hours specified in the reset_locktime parameter. By default, the max_reset_attempts is set to 10 times and the reset_locktime is set to 12 hours, therefore, if a user clicks the Reset Password 11 times without actually resetting their password, then the user will not get a new link to reset their password for 12 hours. |
10 times |
reset_locktime
|
Number of hours that users' will not get a new link to reset their password if they exceed the value set in the max_reset_attempts parameter. |
12 hours |
max_failed
|
Number of times that users' can enter an incorrect password, while logging into FortiSOAR, before their account gets locked. If the user exceeds the value set in this parameter, then the user will get locked out based on the number of minutes specified in the lock_minutes parameter. By default, the max_failed is set to 5 times and the lock_minutes is set to 30 mins, therefore, if a user enters an incorrect password 6 times, then their account gets locked for 30 mins. |
5 times |
lock_minutes
|
Number of minutes that users' account gets locked if they exceed the value set in the max_failed parameter. |
30 mins |
To change the value of the max_reset_attempts
parameter, the administrator should run the following curl command on their FortiSOAR system:
curl -X PUT \
https://<FORTISOAR HOSTNAME/IP>/api/auth/config \
-H 'Authorization: <Bearer Token>' \
-d '{
"option":"max_reset_attempts",
"value":5
}'
The above command changes the number of times users' can click the Reset Password link to 5 times, i.e., a user can click the Reset Password link 5 times without actually setting the new password. However, if the user clicks the Reset Password link for the 6th time, the user will be blocked.
Similarly, to change the value of the reset_locktime
, max_failed
, and lock_minutes
parameters, the administrator should run the same curl command on their FortiSOAR system, but after changing the option and value parameter values:
curl -X PUT \
https://<FORTISOAR HOSTNAME/IP>/api/auth/config \
-H 'Authorization: <Bearer Token>' \
-d '{
"option":"reset_locktime",
"value":2
}'
The above command changes the number of hours that users' will not get a new link to reset their password to 2 hours if they have exceeded the value set in the max_reset_attempts
parameter.
curl -X PUT \
https://<FORTISOAR HOSTNAME/IP>/api/auth/config \
-H 'Authorization: <Bearer Token>' \
-d '{
"option":"max_failed",
"value":3
}'
The above command changes the number of times that users' can enter an incorrect password while logging into FortiSOAR before their account gets locked to 3, i.e., users' account will be locked if they enter an incorrect password 4 times while logging into FortiSOAR.
curl -X PUT \
https://<FORTISOAR HOSTNAME/IP>/api/auth/config \
-H 'Authorization: <Bearer Token>' \
-d '{
"option":"lock_minutes",
"value":15
}'
The above command changes the number of minutes that users' account will be locked to 15 minutes if they have exceeded the value set in the max_reset_attempts
parameter.
Security considerations for Websockets
The Websocket server allows connection from any hostname URL UI. To restrict connections to the Websocket server, you can implement Cross-Origin Resource Sharing (CORS) restrictions in WebSocket.
To implement CORS in WebSocket, you have to add a comma-separated list of allowed hosts (IP address or domain name) in the /etc/cyops/config.yml
file and the hostname should start with "http://
" or "https://
". Dummy entries have already been added in the /etc/cyops/config.yml
file to help you in this process. If you have added domain names to the /etc/cyops/config.yml
file, then you must also restart the 'tomcat' service, using the systemctl restart cyops-tomcat
command.
Once the hostname entries have been added, the WebSocket will only establish a connection with those UIs whose URLs match the specified hostnames.
Troubleshooting Tips
Your Workflow data size has increased
Increase in your Workflow data can cause performance bottlenecks.
Resolution:
You can purge Executed Playbook Logs using the Purge Logs button on the top-right of the Executed Playbook Logs
dialog. For more information on purging, see the Debugging and Optimizing Playbooks chapter in the "Playbooks Guide."
Error displayed while performing a search operation in FortiSOAR
Resolution:
If you get any error while performing a global search in FortiSOAR, check that the elasticsearch.service
and the cyops-search.service
are running.
If these are not running, then start these services using the following commands:
# systemctl start elasticsearch
# systemctl start cyops-search
For more information, see the FortiSOAR Search Errors
topic in the Elasticsearch Configuration chapter.
Reindexing FortiSOAR modules for search
Partial indexing of a module, or when a module does not get indexed, can lead to errors in FortiSOAR search. You can manually reindex any skipped or unsuccessfully indexed modules. For more information, see the FortiSOAR Search Errors
topic in the Elasticsearch Configuration chapter.
Resolution:
To reindex all the FortiSOAR modules, run the following command:
$ sudo -u nginx php /opt/cyops-api/app/console app:elastic:create --env="prod"
To reindex specific FortiSOAR modules, run the following command:
$ sudo -u nginx php /opt/cyops-api/app/console app:elastic:create --env="prod" --index='{"type":"type of the module(s)"}'
For example:
$ sudo -u nginx php /opt/cyops-api/app/console app:elastic:create --env="prod" --index='{"type":"indicators", "tasks"}
FortiSOAR crashing with "out of memory" errors
By default, FortiSOAR configures Elaticsearch to use 4 GB of RAM. If there are too many records or any very heavy records (such as large files uploaded) created per day on the system, it might crash with "out of memory" errors. To fix this, you must increase the memory allocated to Elasticsearch.
Resolution
- Change the following entry in
/etc/elasticsearch/jvm.options
to a higher value based on memory available on your server:
-Xms4g
-Xmx4g - Restart Elasticsearch using the following command:
systemctl restart elasticsearch
Changing Postgres worker memory
When the primary data in the system becomes large (eg, over million alerts) and you notice that the system is slow to respond. The slowness could be caused due to database queries taking longer with the increased database size. You can fine tune this behavior by increasing the following Postgres settings based on the available free memory on the system:
- Increase the shared buffer and worker memory in the
/var/lib/pgsql/12/data/postgresql.conf
file:shared_buffers = 2048MB
work_mem = 16MB
- Restart Postgres using the following command:
systemctl restart postgresql-12
Changing the maximum number of records that can be linked in one call
The maximum number of records that can be linked in one call should be 99.
To change this limit, you can change the max_relation_count
parameter value in the /opt/cyops-api/configs/parameters_prod.yml
file:
# max numnber of relations to normalize
max_relation_count: 100