Elasticsearch Configuration
FortiSOAR leverages the fast search capability of Elasticsearch for quick text search across all records and files in the FortiSOAR database. FortiSOAR supports externalization of Elasticsearch data. Externalization is indexing of data to an Elasticsearch instance that has the same or higher version of Elasticsearch outside of the FortiSOAR virtual appliance; the steps for which are covered in this chapter.
The minimum version of your Elasticsearch cluster must be 7.0.2, if you want to externalize your ElasticticSearch data. |
If you want to externalize your other FortiSOAR PostgreSQL database, see the Externalization of your FortiSOAR PostgreSQL database chapter.
Externalization and Authentication of Elasticsearch
If you require to change the location of your Elasticsearch instance from your local instance to a remote machine, run the following steps on your externalized Elasticseach machine:
- Use the Elasticsearch documentation to install elasticsearch.
Important: Ensure that you install the same version of elasticsearch that is currently installed on your FortiSOAR instance on the externalized Elasticseach machine. For example, if you have version elasticsearch v7.2.0 currently on your FortiSOAR instance, then u must install elasticsearch v7.2.0 on the remote machine on which you want to externalize elasticseach. - Configure elasticsearch to accept connections from outside 'localhost' by updating the '
network.host
' and 'discovery.type
' variables tonetwork.host: 0.0.0.0
' and 'discovery.type: single-node
' in the/etc/elasticsearch/elasticsearch.yml
file.
You must comment thecluster.initial_master_nodes
line in theelasticsearch.yml
file. For example,#cluster.initial_master_nodes: ["example_hostname"]
- Run the following commands:
# mkdir -p /opt/cyops-search
# chmod 755 /opt/cyops-search - Copy the
/opt/cyops-search/exclude.list
file from your FortiSOAR machine to the/opt/cyops-search/exclude.list
file on your externalized elasticsearch host. - Copy the
/etc/elasticsearch/security.policy
file from your FortiSOAR machine to the/etc/elasticsearch/security.policy
file on your externalized elasticsearch host. - Run the following commands:
# chmod 644 /opt/cyops-search/exclude.list
# chmod 660 /etc/elasticsearch/security.policy
# chown root:elasticsearch /etc/elasticsearch/security.policy - Append the '
-Djava.security.policy=/etc/elasticsearch/security.policy
' line to the/etc/elasticsearch/jvm.options
file. - Start elasticsearch using the following command:
# systemctl start elasticsearch
- Open the elasticsearch port in your firewall, using the following command:
# firewall-cmd --permanent --add-port=9200/tcp
# firewall-cmd --reload - (Optional) To enable SSL for elasticsearch, see the https://www.elastic.co/blog/configuring-ssl-tls-and-https-to-secure-elasticsearch-kibana-beats-and-logstash blog.
- Once elasticsearch is externalized, it is recommended that you stop and mask 'localhost' elasticsearch on your FortiSOAR machine, using the following commands:
# systemctl stop elasticsearch
# systemctl disable elasticsearch
# systemctl mask elasticsearch
Once elasticsearch is externalized, update the db_config.yml
file, which is located at: /opt/cyops/configs/database/db_config.yml
In the db_config.yml
file, you require to update the host and port (if needed) in the elasticsearch
section that appears as follows:
elasticsearch:
es_external: false
es_host: localhost
es_port: 9200
es_user: elastic
initial_backoff: 60
max_backoff: 6000
secret: None
ssl_cert_path: ""
use_ssl: false
To change the location of your Elasticsearch instance from your local instance to a remote machine:
es_external: false
> Set the value ofes_external
totrue
to externalization of elasticsearch.es_host: localhost
> Update host value with the hostname or IP address of the remote Elasticsearch machine.es_port: 9200
> Update the port required to access the remote Elasticsearch machine, if required.
For authentication of Elasticsearch:
es_user: elastic
> Update the username that is used to access the remote Elasticsearch machine, if Authentication is enabled on the remote Elasticsearch machine.secret: None
> Update the secret (password) that is used to access the remote Elasticsearch machine, if Authentication is enabled on the remote Elasticsearch machine.
You also require to assign ngnix
permission to the SSL certificate that you have specified in the db_config.yml
file using the following command:
chown nginx:nginx filename.pem
Externalized elasticseach must have SSL enabled for use in the FortiSOAR high availability cluster. Also, ensure that you set the use_ssl flag to 'true' and specify the ssl_cert_path as the path of your external elasticsearch CA certificate. |
Migration of Elasticsearch data
Once you complete the externalization of Elasticsearch, you will require to migrate your data from your local instance to the remote Elasticsearch machine.
To migrate the remote Elasticsearch machine run the following command on your FortiSOAR instance as a root
user after changing the directory to /opt/cyops-api/
:
$ sudo -u nginx php /opt/cyops-api/bin/console app:elastic:create --env="prod"
Troubleshooting Tips
FortiSOAR Search Errors
FortiSOAR Search performs indexing in an asynchronous fashion in the backend. Users could be faced with certain scenarios that could lead to a restart of services, which can cause indexing to stop. In this case, FortiSOAR might display any of the following errors when users are performing a search operation on FortiSOAR:
Search indexing is in progress. Partial results are returned.
Search indexing has stopped. You must manually rerun indexing (see product documentation for instructions) or raise a support ticket for the same.
We are sorry, but the server encountered an error while handling your search request. Please contact your administrator for assistance.
In this case, use the /var/log/cyops/cyops-search/falcon.log
log file to check which modules are published and indexed and which modules are yet to be published (pending).
For example, the /var/log/cyops/cyops-search/falcon.log
log file will display results as follows:
2019-02-13,11:00:44 INFO blocking_connection: _dispatch_events(): 1445: Module Currently Getting Published: ['attachments']
2019-02-13,11:00:44 INFO blocking_connection: _dispatch_events(): 1445: Indexing for Module: 'attachments' started Total Records to be indexed: '1'
2019-02-13,11:00:49 INFO blocking_connection: _dispatch_events(): 1445: Module: 'attachments' Successful Total Records indexed: '1'
2019-02-13,11:00:49 INFO blocking_connection: _dispatch_events(): 1445: on_publish_message called
2019-02-13,11:00:53 INFO blocking_connection: _dispatch_events(): 1445: creating index with mapping
2019-02-13,11:01:00 INFO blocking_connection: _dispatch_events(): 1445: Module Currently Getting Published: ['emails']
2019-02-13,11:01:02 INFO blocking_connection: _dispatch_events(): 1445: Indexing for Module: 'emails' started Total Records to be indexed: '1'
2019-02-13,11:01:04 INFO blocking_connection: _dispatch_events(): 1445: Module: 'emails' Successful Total Records indexed: '1'
The above example shows the attachments
and emails
modules currently being indexed and its total number of records. Any failure in indexing any modules will be logged here. You can monitor the progress of this file while the indexing is in progress.
If any module(s) are missing from the published list or if any module has the Publish Module: '<name of module>' Unsuccessful
listed in the /var/log/cyops/cyops-search/falcon.log
log file; the indicators
and tasks
modules in our example, then you must manually run the indexing for those module(s) using the following command:
$ sudo -u nginx php bin/console app:elastic:create --env="prod" --index='{"type":["<list of comma-seperated module names that require to be indexed>"]}
For our example, run the following command:
$ sudo -u nginx php bin/console app:elastic:create --env="prod" --index='{"type":["indicators","tasks"]}'
Optimizing the reindexing of Elasticsearch data
If reindexing is required in your environment for syncing that latest Elasticsearch data, FortiSOAR reindexes only the latest 50,000 (maximum) records. This is because reindexing the entire set of records makes the system sluggish and results in high CPU utilization.
You can customize the maximum number of records to be reindexed by editing the /opt/cyops/configs/cyops-search/config.yml
file and changing the value of the following parameter: max_indexing_limit: 50000
.
You need to explicitly add the "max_indexing_limit
" parameter to the /opt/cyops/configs/cyops-search/config.yml
file if you have upgraded your instance to FortiSOAR release 7.4.0 or later.