Debugging, Troubleshooting, and optimizing FortiSOAR
Administrators can use various logs that FortiSOAR generates to troubleshoot FortiSOAR issues. This chapter lists the key FortiSOAR services and processes and also provides some troubleshooting tips. This chapter also provides some additional configuration settings so that you can tune the results that get displayed by FortiSOAR for record similarity and field prediction. For more information on record similarity and field prediction, see the Working with Modules - Alerts & Incidents chapter in the "User Guide."
If you face any issues while deploying or upgrading FortiSOAR, see the Troubleshooting FortiSOAR chapter in the "Deployment Guide." If you face deployment or upgrade failures due to insufficient space, or if you face issues while using FortiSOAR that might be caused due to insufficient space, like you are unable to log into FortiSOAR or FortiSOAR services stop working, then see the Issues occurring in FortiSOAR due to insufficient space
section in the Troubleshooting FortiSOAR chapter in the "Deployment Guide."
For information about monitoring various important aspects of your FortiSOAR system, see the Monitoring FortiSOAR chapter.
List of logs used for troubleshooting FortiSOAR
FortiSOAR log files are stored in the following location: /var/log/cyops
. You will find the following directories in the /var/log/cyops
location:
Log Name | Purpose |
---|---|
cyops-api/ssl_cyops_api_access.log
|
Used for troubleshooting web (nginx) UI or API access issues. |
cyops-api/ssl_cyops_api_error.log
|
Used for troubleshooting API errors. |
cyops-api/prod.log
|
Used for troubleshooting FortiSOAR PHP related issues. |
cyops-api/last_system_publish.log
|
Used for troubleshooting publishing issues. It captures the output when a publish is fired from the UI after MMD changes. |
cyops-auth/das.log
|
Used for troubleshooting FortiSOAR authentication issues. |
cyops-auth/fdn.log
|
Used for troubleshooting FortiSOAR license synchronization issues with FortiGuard Distribution Network (FDN). |
cyops-auth/ha.log
|
Used for troubleshooting FortiSOAR High Availability issues. |
cyops-gateway/auditlog.log
|
Used for troubleshooting FortiSOAR audit log issues. |
cyops-gateway/etl.log
|
Used for troubleshooting FortiSOAR application and system-level issues. |
cyops-gateway/saml.log
|
Used for troubleshooting FortiSOAR SAML issues. |
cyops-search/falcon.log
|
Used for troubleshooting FortiSOAR Search issues. If you get any error when you are indexing or searching for a record in FortiSOAR, you can use the falcon.log file to troubleshoot ElasticSearch issues. This log is also used for checking the status of ElasticSearch indexing. |
cyops-gateway/gateway.log
|
Used for troubleshooting FortiSOAR Gateway issues such as, audit log page failing to load or the SSO configuration page failing to load. |
cyops-notifier/notifier.log
|
Used for troubleshooting FortiSOAR Web Socket issues. |
cyops-workflow/beat.log
|
Used for troubleshooting issues of the FortiSOAR Scheduler. |
cyops-workflow/celeryd.log
|
Used for troubleshooting FortiSOAR playbook runtime issues. |
cyops-workflow/sealab.log
|
Used for troubleshooting FortiSOAR playbook framework issues. |
cyops-workflow/ssl_cyops_workflow_access.log
|
Used for troubleshooting playbook access issues. |
cyops-workflow/ssl_cyops_workflow_error.log
|
Used for troubleshooting playbook errors. |
cyops-workflow/uwsgi.log
|
Used for troubleshooting FortiSOAR playbook and connector issues. |
cyops-integrations/connectors.log
|
Used for troubleshooting FortiSOAR connector related issues. |
cyops-integrations/integrations.log
|
Used for troubleshooting FortiSOAR connector framework issues. |
cyops-integrations/integrations/imap/listener.log
|
Used for troubleshooting the IMAP connector issues. |
cyops-integrations/ssl_cyops_integrations_access.log
|
Used for troubleshooting connector access issues. |
cyops-integrations/ssl_cyops_integrations_error.log
|
Used for troubleshooting connector errors. |
csadm/db.log
|
Used for troubleshooting database externalization errors. |
install
For example, 6.4.1-2133.log . |
Used for troubleshooting FortiSOAR installation issues. Install logs are named according to the FortiSOAR version and build number. |
install/connectors.log
|
Used for troubleshooting connector installation issues. |
upgrade_fortisoar_<version_number>-<timestamp>.log
For example, upgrade_fortisoar_6.0.0-2020-02-18-1558604373.log
|
Stores upgrade console log and you can use it to troubleshoot FortiSOAR upgrade issues. |
install/config-vm-<timestamp>.log
For example, install/config-vm-27_Nov_2018_18h_14m_34s.log
|
Used for troubleshooting FortiSOAR Configuration Wizard issues. |
For troubleshooting FortiSOAR audit log issues use the tomcat.log
located at /opt/cyops-tomcat/tomcat.log
.
For Centos OS level errors, use the Messages
logs located at /var/log/messages
.
For troubleshooting issues related to dedicated tenant nodes or FortiSOAR agents, see the following logs:
-
var/log/cyops/cyops-routing-agent/postman.log
-
var/log/cyops/cyops-routing-agent/uwsgi.log
-
var/log/cyops/cyops-routing-agent/ssl_cyops_routing_agent_error.log
-
var/log/cyops/cyops-routing-agent/ssl_cyops_routing_agent_access.log
.
For troubleshooting issues related to FortiMonitor, see the following logs:
-
var/log/panopta-agent/
: For troubleshooting issues related to the FortiMonitor Agent -
var/log/cyops/fmonitor/
: For troubleshooting issues related to the csadm CLI for FortiMonitor.
For troubleshooting issues related to the separate runtime user for integrations:
-
var/log/cyops/cyops-integrations/integration.log
-
var/log/cyops/cyops-integrations/connectors.log
-
getfacl var/log/cyops/cyops-integrations/
-
getfacl opt/cyops-integrations/integrations
-
getfacl var/run/uwsgi
-
getfacl opt/cyops-integrations/.env
Logging Levels
You can set the following logging levels in the log files:
- DEBUG: Low-level system information for debugging purposes.
- INFO: General system information.
- WARNING: Information describing a minor problem that has occurred.
- ERROR: Information describing a major problem that has occurred.
- CRITICAL: Information describing a critical problem that has occurred.
Changing the logging levels
- For sealab or workflow:
- Open the
/opt/cyops-workflow/sealab/sealab/config.ini
file and set theWORKFLOW_LOG_LEVEL
parameter to the required logging level. For example,WORKFLOW_LOG_LEVEL = 'INFO'
- Restart the
uwsgi
service.
- Open the
- For integrations:
- Open the
/opt/cyops-integrations/integrations/configs/config.ini
file and set theconnector_logger_level
parameter to the required logging level. For example,connector_logger_level= 'INFO'
- Restart the
uwsgi
service.
- Open the
- For celeryd:
- Open the
/etc/celery/celeryd.conf
file and set theCELERYD_LOG_LEVEL
parameter to the required logging level. For example,CELERYD_LOG_LEVEL = 'INFO'
- Restart the
celeryd
service.
- Open the
- For nginx (UI), API, or php:
- Open the
/opt/cyops-api/app/config/config_prod.yml
file and set thelevel
parameter to the required logging level. For example,level = 'INFO'
- Run the
# systemctl restart php-fpm nginx
command.
- Open the
- To enable all console errors in your browser, add
DEBUG = true
in local storage as follows:- Open your browser and click select More Tools > Developer Tools.
- Click the Application tab and then click Storage.
- In Local Storage, type
DEBUG
as the key andtrue
as the value.
List of key FortiSOAR services and processes
Name of Services/Processes | Description |
---|---|
postgresql-12 | Service for all application data stored in postgresql DB. To know the status of this service, use the # systemctl status postgresql-12 command. |
elasticsearch | Service to bring up the elasticsearch service. To know the status of this service, use the # systemctl status elasticsearch command. |
php-fpm | Service for PHP FastCGI implementation. To know the status of this service, use the # systemctl status php-fpm command. |
uwsgi | Software application that aims at developing a full stack for building hosting services. uWSGI is named after the Web Server Gateway Interface. We host our playbook execution engine application and connector integrations applications on a uWSGI server. To know the status of uwsgi use the # systemctl status uwsgi command. |
celeryd | celeryd is used to run the playbooks asynchronously in the FortiSOAR playbook execution engine. To know the status of celeryd use the # systemctl status celeryd command. |
celerybeatd | celerybeatd is a playbook scheduler; used to kick off tasks at regular intervals, that are then executed by available worker nodes in the cluster. To know the status of use the # systemctl status celerybeatd command. |
cyops-auth | Service used for FortiSOAR authentications. To know the status of this services, use the # systemctl status cyops-auth command. |
cyops-tomcat | Service used for SSO, auditing, and websocket. To know the status of this services, use the # systemctl status tomcat command. |
cyops-search | Service used for full-text searching, finding similar records and predicting the value of fields based on similarity. |
cyops-ha | Service is responsible for setting up High Availability in the FortiSOAR environment |
cyops-postman | Service is responsible for setting up and managing FortiSOAR's distributed multi-tenant setup. |
cyops-integrations-agent | Service is responsible for running actions on remote FortiSOAR agents. |
rabbitmq-server | Service is responsible to send audit and live sync notifications. This service is also responsible for data transfer in a distributed environment. |
nginx | Service used for Web UI. To know the status of this services, use the # systemctl status nginx command. |
If you want to restart, start, or stop all the services together, use FortiSOAR Admin CLI (csadm
). For more information on csadm
, see the FortiSOAR Admin CLI chapter.
You can run the csadm
command on any FortiSOAR machine using any terminal. Any user who has root
or sudo
permissions can run the csadm
command.
To restart FortiSOAR services, type: # csadm services --restart
To start FortiSOAR services, type: # csadm services --start
To stop FortiSOAR services, type: # csadm services --stop
To know the status of all FortiSOAR services type: # csadm services --status
To know more about monitoring services, see the Monitoring FortiSOAR chapter.
Configurations required for exporting of records with unsupported character sets in the PDF format
You can export records to PDF, by clicking the 'Hamburger' menu on the grid view and selecting the Export All Visible Columns to PDF option, or you can open the record and in the detail view of the record, and click the Export Record > Export as PDF option. The FortiSOAR UI uses the 'pdfmake' library, to export the record data from the record's grid and view panel to the PDF format. The 'pdfmake' library by default comes with the ROBOTO font that does not support Korean, Chinese, and other character sets. Therefore, to export the record data that contains unsupported character sets from the record's grid and view panel in FortiSOAR to the PDF format, you can use one of the following options:
Option 1:
Use SSH
and update the /opt/cyops-ui/vendor/config.json
to provide the CDN URLs for a specific font:
"pdfmake": { "font": { "normal": "https://cdnjs.cloudflare.com/ajax/libs/pdfmake/0.1.66/fonts/Roboto/Roboto-Regular.ttf", "bold": "https://cdnjs.cloudflare.com/ajax/libs/pdfmake/0.1.66/fonts/Roboto/Roboto-Medium.ttf", "italics": "https://cdnjs.cloudflare.com/ajax/libs/pdfmake/0.1.66/fonts/Roboto/Roboto-Italic.ttf", "bolditalics": "https://cdnjs.cloudflare.com/ajax/libs/pdfmake/0.1.66/fonts/Roboto/Roboto-MediumItalic.ttf" } }
Option 2
Build and use the custom font for 'pdfmake' as follows:
-
Copy font files into the
/opt/cyops-ui/vendor/pdfmake
directory. - Build the
vfs_fonts.js
file with new fonts. For example to build thevfs_fonts.js
file with the 'Roboto' font family use the following command:# sh build.sh Roboto-Regular.ttf Roboto-Medium.ttf Roboto-Italic.ttf Roboto-MediumItalic.ttf
- Update the
/opt/cyops-ui/vendor/config.json
file with the font references:"pdfmake": {
"font": {
"normal": "Roboto-Regular.ttf",
"bold": "Roboto-Medium.ttf"
"italics": "Roboto-Italics.ttf"
"bolditalics": "Roboto-MediumItalic.ttf"
}
}
You can also refer to the/opt/cyops-ui/vendor/pdfmake/README.txt
file for instructions.
Additional settings for record similarity and field predictions
FortiSOAR supports "Record Similarity" i.e., FortiSOAR displays records that are similar to the record on which you are working. FortiSOAR also supports "Record Field Value Prediction" i.e., FortiSOAR predicts values of fields of your choice within a record from the values of fields of existing records based on the criteria you have defined, making it easier for analysts to make informed decisions. For more information, see the Working with Modules - Alerts & Incidents chapter in the "User Guide."
This section provides information on how you can tune the results that are displayed by FortiSOAR for record similarity and field predictions using the following parameters in the /opt/cyops/config/cyops-search/config.yml
file:
-
minimum_should_match
:<percentageValue>
: This setting defines that a record will be considered similar only if there is a match of at least the percentage value that you have specified on the related fields. This is especially true for similarity based on related records. For example, if you set this parameter asminimum_should_match
:10%
(default), then if you have defined similarity for alerts based on related indicators, then FortiSOAR will display only those records as similar that match a minimum of 10% of the indicators. Therefore, for an alert that has 10 related indicators, FortiSOAR similarity results will display alerts that even have one common indicator; but if an alert has 20 related indicators, then FortiSOAR similarity results will display only those alerts that have at least 2 indicators in common. -
max_query_terms
:<numberOfItems>
: This setting defines how many terms of the parent record will be looked up for similarity in other records. Continuing the same example as above, if you set this parameter asmax_query_terms
:25
(default), then if an alert has more than 25 indicators, only 25 of them will be checked for similarity in other records. Note that increasing the value of this setting will increase the time FortiSOAR takes to return similarity and suggestion results.
For more information on the above parameters and other parameters, refer to the ElastiSearch reference at: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-mlt-query.html
Change the default value of some of the user profile parameters
An administrator with CRUD permissions on the Security
module can change the default value of the following user-profile related parameters:
Parameter Name | Description | Default Value |
---|---|---|
max_reset_attempts
|
Maximum number of times users' can click the Reset Password link before actually resetting their password. If the user exceeds the value set in this parameter, then users' will not get a new link to reset their password based on the number of hours specified in the reset_locktime parameter. By default, the max_reset_attempts is set to 10 times and the reset_locktime is set to 12 hours, therefore, if a user clicks the Reset Password 11 times without actually resetting their password, then the user will not get a new link to reset their password for 12 hours. |
10 times |
reset_locktime
|
Number of hours that users' will not get a new link to reset their password if they exceed the value set in the max_reset_attempts parameter. |
12 hours |
max_failed
|
Number of times that users' can enter an incorrect password, while logging into FortiSOAR, before their account gets locked. If the user exceeds the value set in this parameter, then the user will get locked out based on the number of minutes specified in the lock_minutes parameter. By default, the max_failed is set to 5 times and the lock_minutes is set to 30 mins, therefore, if a user enters an incorrect password 6 times, then their account gets locked for 30 mins. |
5 times |
lock_minutes
|
Number of minutes that users' account gets locked if they exceed the value set in the max_failed parameter. |
30 mins |
To change the value of the max_reset_attempts
parameter, the administrator should run the following curl command on their FortiSOAR system:
curl -X PUT \
https://<FORTISOAR HOSTNAME/IP>/api/auth/config \
-H 'Authorization: <Bearer Token>' \
-d '{
"option":"max_reset_attempts",
"value":5
}'
The above command changes the number of times users' can click the Reset Password link to 5 times, i.e., a user can click the Reset Password link 5 times without actually setting the new password. However, if the user clicks the Reset Password link for the 6th time, the user will be blocked.
Similarly, to change the value of the reset_locktime
, max_failed
, and lock_minutes
parameters, the administrator should run the same curl command on their FortiSOAR system, but after changing the option and value parameter values:
curl -X PUT \
https://<FORTISOAR HOSTNAME/IP>/api/auth/config \
-H 'Authorization: <Bearer Token>' \
-d '{
"option":"reset_locktime",
"value":2
}'
The above command changes the number of hours that users' will not get a new link to reset their password to 2 hours if they have exceeded the value set in the max_reset_attempts
parameter.
curl -X PUT \
https://<FORTISOAR HOSTNAME/IP>/api/auth/config \
-H 'Authorization: <Bearer Token>' \
-d '{
"option":"max_failed",
"value":3
}'
The above command changes the number of times that users' can enter an incorrect password while logging into FortiSOAR before their account gets locked to 3, i.e., users' account will be locked if they enter an incorrect password 4 times while logging into FortiSOAR.
curl -X PUT \
https://<FORTISOAR HOSTNAME/IP>/api/auth/config \
-H 'Authorization: <Bearer Token>' \
-d '{
"option":"lock_minutes",
"value":15
}'
The above command changes the number of minutes that users' account will be locked to 15 minutes if they have exceeded the value set in the max_reset_attempts
parameter.
Security considerations for Websockets
The Websocket server allows connection from any hostname URL UI. To restrict connections to the Websocket server, you can implement Cross-Origin Resource Sharing (CORS) restrictions in WebSocket.
To implement CORS in WebSocket, you have to add a comma-separated list of allowed hosts (IP address or domain name) in the /etc/cyops/config.yml
file and the hostname should start with "http://
" or "https://
". Dummy entries have already been added in the /etc/cyops/config.yml
file to help you in this process. If you have added domain names to the /etc/cyops/config.yml
file, then you must also restart the 'tomcat' service, using the systemctl restart cyops-tomcat
command.
Once the hostname entries have been added, the WebSocket will only establish a connection with those UIs whose URLs match the specified hostnames.
Troubleshooting Tips
Your Workflow data size has increased
Increase in your Workflow data can cause performance bottlenecks.
Resolution:
You can purge Executed Playbook Logs using the Purge Logs button on the top-right of the Executed Playbook Logs
dialog. For more information on purging, see the Debugging and Optimizing Playbooks chapter in the "Playbooks Guide."
Error displayed while performing a search operation in FortiSOAR
Resolution:
If you get any error while performing a global search in FortiSOAR, check that the elasticsearch.service
and the cyops-search.service
are running.
If these are not running, then start these services using the following commands:
# systemctl start elasticsearch
# systemctl start cyops-search
For more information, see the FortiSOAR Search Errors
topic in the Elasticsearch Configuration chapter.
Reindexing FortiSOAR modules for search
Partial indexing of a module, or when a module does not get indexed, can lead to errors in FortiSOAR search. You can manually reindex any skipped or unsuccessfully indexed modules. For more information, see the FortiSOAR Search Errors
topic in the Elasticsearch Configuration chapter.
Resolution:
To reindex all the FortiSOAR modules, run the following command:
$ sudo -u nginx php /opt/cyops-api/bin/console app:elastic:create --env="prod"
To reindex specific FortiSOAR modules, run the following command:
$ sudo -u nginx php /opt/cyops-api/bin/console app:elastic:create --env="prod" --index='{"type":"type of the module(s)"}'
For example:
$ sudo -u nginx php /opt/cyops-api/bin/console app:elastic:create --env="prod" --index='{"type":"indicators", "tasks"}
FortiSOAR crashing with "out of memory" errors
By default, FortiSOAR configures Elaticsearch to use 4 GB of RAM. If there are too many records or any very heavy records (such as large files uploaded) created per day on the system, it might crash with "out of memory" errors. To fix this, you must increase the memory allocated to Elasticsearch.
Resolution
- Change the following entry in
/etc/elasticsearch/jvm.options
to a higher value based on memory available on your server:
-Xms4g
-Xmx4g - Restart Elasticsearch using the following command:
systemctl restart elasticsearch
Changing Postgres worker memory
When the primary data in the system becomes large (eg, over million alerts) and you notice that the system is slow to respond. The slowness could be caused due to database queries taking longer with the increased database size.
Resolution
You can fine tune this behavior by increasing the following Postgres settings based on the available free memory on the system:
- Increase the shared buffer and worker memory in the
/var/lib/pgsql/12/data/postgresql.conf
file:shared_buffers = 2048MB
work_mem = 16MB
- Restart Postgres using the following command:
systemctl restart postgresql-12
Changing the maximum number of records that can be linked in one call
The maximum number of records that can be linked in one call should be 99.
To change this limit, you can change the max_relation_count
parameter value in the /opt/cyops-api/configs/parameters_prod.yml
file:
# max numnber of relations to normalize
max_relation_count: 100
Crashing of the Tomcat server when the 'Detailed' mode for Log Forwarding is enabled
If you have enabled the 'Detailed' mode for Log Forwarding in Settings > Log Forwarding > Audit Logs, then your Tomcat server might crash and the 'Live Sync' might continuously fail.
Resolution
To avoid crashing of your Tomcat server and you do not want to increase the memory assigned to the Tomcat service, you need to disable log forwarding and then set log forwarding to the 'Basic' mode as follows:
- SSH to your FortiSOAR server and login as a root user.
- List the current log forward settings using the following command:
# csadm log forward show-config
- Copy the UUID from the output of the
show-config
command. - Disable forwarding of the audit logs to the syslog server using the following command:
# csadm log forward update-config --uuid <uuid> --filter application
or# csadm log forward update-config --uuid <uuid> --filter none
- Restart the Tomcat service using the following command:
# systemctl restart cyops-tomcat
- Log on to FortiSOAR, and click Settings > System Configuration.
- Click the Log Forwarding tab.
- From the Specify Audit Log Detail Level drop-down list, select Basic, and click Save.
If you for some reason you require to use the 'Detailed' mode for log forwarding, you can increase the memory assigned to the Tomcat service running the log forwarding process using the following steps:
- In
/usr/lib/systemd/system/cyops-tomcat.service
change the Xms value as follows:Environment="CATALINA_OPTS=-Xms4096M -Xmx4096M -server -XX:+UseParallelGC"
- Run
systemctl daemon-reload
- Run
systemctl restart cyops-tomcat
FortiSOAR displaying errors such as app.ERROR: Unable to load API credentials from cache or DAS
FortiSOAR might display errors such as:[2022-03-10 22:58:21] app.ERROR: Unable to load API credentials from cache or DAS. [] []
[2022-03-10 22:58:21] app.ERROR: /opt/cyops-api/src/Encoder/Hmac/DasHmacSecretKeyProvider.php at line 114 [] []
If you have not regenerated the default FortiSOAR self-signed certificates for more than one year.
Resolution
To resolve this issue, run the following command as a root user (using 'sudo su
' and using the csadmin
password) using a SSH session:csadm certs --generate `hostname`
Once this command is run successfully, you require to restart all services using the following command:csadm services --restart
Recommendation Engine does not work and displays an "ML Service not running" error
If you have upgraded FortiSOAR or due to some other reason, you might observe that the Recommendation Engine is not working and not displaying any suggestions. You can also get an error such as "ML Service is not running." This issue can occur if the ml
service fails to start on port 10449.
Resolution
Restart the uwsgi
service and you will observe that the ml
service is running on port 10449. Once the ml
service starts, you will observe that the recommendation engine begins to work and displays appropriate suggestions like phishing classification.