Fortinet black logo

Administration Guide

Elasticsearch Configuration

Copy Link
Copy Doc ID 4631dc5b-1d0e-11ec-8c53-00505692583a:907294
Download PDF

Elasticsearch Configuration

FortiSOAR leverages the fast search capability of Elasticsearch for quick text search across all records and files in the FortiSOAR database. FortiSOAR supports externalization of Elasticsearch data. Externalization is indexing of data to an Elasticsearch instance that has the same or higher version of Elasticsearch outside of the FortiSOAR virtual appliance; the steps for which are covered in this chapter.

Tooltip

The minimum version of your Elasticsearch cluster must be 7.0.2, if you want to externalize your ElasticticSearch data.

If you want to externalize your other FortiSOAR PostgreSQL database, see the Externalization of your FortiSOAR PostgreSQL database chapter.

Externalization and Authentication of Elasticsearch

If you require to change the location of your Elasticsearch instance from your local instance to a remote machine, you need to update the db_config.yml file, which is located at: /opt/cyops/configs/database/db_config.yml

In the db_config.yml file, you require to update the host and port (if needed) in the elasticsearch section that appears as follows:

elasticsearch: 
  es_host: localhost
  es_port: 9200
  es_user: None
  initial_backoff: 60
  max_backoff: 6000
  secret: None
  ssl_cert_path: ""
  use_ssl: false

To change the location of your Elasticsearch instance from your local instance to a remote machine:

es_host: localhost > Update host value with the hostname or IP address of the remote Elasticsearch machine.

es_port: 9200 > Update the port required to access the remote Elasticsearch machine, if required.

For authentication of Elasticsearch (require X-Pack License):

es_user: None > Update the username that is used to access the remote Elasticsearch machine, if Authentication is enabled on the remote Elasticsearch machine

secret: None > Update the secret (password) that is used to access the remote Elasticsearch machine, if Authentication is enabled on the remote Elasticsearch machine.

You also require to assign ngnix permission to the SSL certificate that you have specified in the db_config.yml file using the following command:

chown nginx:nginx filename.pem

Migration of Elasticsearch data

Once you complete the externalization of Elasticsearch, you will require to migrate your data from your local instance to the remote Elasticsearch machine.

To migrate the remote Elasticsearch machine run the following command on your FortiSOAR instance as a root user:

$ sudo -u nginx php /opt/cyops-api/bin/console app:elastic:create --env="prod"

Troubleshooting

FortiSOAR Search Errors

FortiSOAR Search performs indexing in an asynchronous fashion in the backend. Users could be faced with certain scenarios that could lead to a restart of services, which can cause indexing to stop. In this case, FortiSOAR might display any of the following errors when users are performing a search operation on FortiSOAR:

  • Search indexing is in progress. Partial results are returned.
  • Search indexing has stopped. You must manually rerun indexing (see product documentation for instructions) or raise a support ticket for the same.
  • We are sorry, but the server encountered an error while handling your search request. Please contact your administrator for assistance.

In this case, use the /var/log/cyops/cyops-search/falcon.log log file to check which modules are published and indexed and which modules are yet to be published (pending).

For example, the /var/log/cyops/cyops-search/falcon.log log file will display results as follows:

2019-02-13,11:00:44 INFO blocking_connection: _dispatch_events(): 1445: Module Currently Getting Published: ['attachments']
2019-02-13,11:00:44 INFO blocking_connection: _dispatch_events(): 1445: Indexing for Module: 'attachments' started Total Records to be indexed: '1'
2019-02-13,11:00:49 INFO blocking_connection: _dispatch_events(): 1445: Module: 'attachments' Successful Total Records indexed: '1'
2019-02-13,11:00:49 INFO blocking_connection: _dispatch_events(): 1445: on_publish_message called
2019-02-13,11:00:53 INFO blocking_connection: _dispatch_events(): 1445: creating index with mapping
2019-02-13,11:01:00 INFO blocking_connection: _dispatch_events(): 1445: Module Currently Getting Published: ['emails']
2019-02-13,11:01:02 INFO blocking_connection: _dispatch_events(): 1445: Indexing for Module: 'emails' started Total Records to be indexed: '1'
2019-02-13,11:01:04 INFO blocking_connection: _dispatch_events(): 1445: Module: 'emails' Successful Total Records indexed: '1'

The above example shows the attachments and emails modules currently being indexed and its total number of records. Any failure in indexing any modules will be logged here. You can monitor the progress of this file while the indexing is in progress.

If any module(s) are missing from the published list or if any module has the Publish Module: '<name of module>' Unsuccessful listed in the /var/log/cyops/cyops-search/falcon.log log file; the indicators and tasks modules in our example, then you must manually run the indexing for those module(s) using the following command:

$ sudo -u nginx php app/console app:elastic:create --env="prod" --index='{"type":["<list of comma-seperated module names that require to be indexed>"]}

For our example, run the following command:

$ sudo -u nginx php app/console app:elastic:create --env="prod" --index='{"type":["indicators","tasks"]}'

Elasticsearch Configuration

FortiSOAR leverages the fast search capability of Elasticsearch for quick text search across all records and files in the FortiSOAR database. FortiSOAR supports externalization of Elasticsearch data. Externalization is indexing of data to an Elasticsearch instance that has the same or higher version of Elasticsearch outside of the FortiSOAR virtual appliance; the steps for which are covered in this chapter.

Tooltip

The minimum version of your Elasticsearch cluster must be 7.0.2, if you want to externalize your ElasticticSearch data.

If you want to externalize your other FortiSOAR PostgreSQL database, see the Externalization of your FortiSOAR PostgreSQL database chapter.

Externalization and Authentication of Elasticsearch

If you require to change the location of your Elasticsearch instance from your local instance to a remote machine, you need to update the db_config.yml file, which is located at: /opt/cyops/configs/database/db_config.yml

In the db_config.yml file, you require to update the host and port (if needed) in the elasticsearch section that appears as follows:

elasticsearch: 
  es_host: localhost
  es_port: 9200
  es_user: None
  initial_backoff: 60
  max_backoff: 6000
  secret: None
  ssl_cert_path: ""
  use_ssl: false

To change the location of your Elasticsearch instance from your local instance to a remote machine:

es_host: localhost > Update host value with the hostname or IP address of the remote Elasticsearch machine.

es_port: 9200 > Update the port required to access the remote Elasticsearch machine, if required.

For authentication of Elasticsearch (require X-Pack License):

es_user: None > Update the username that is used to access the remote Elasticsearch machine, if Authentication is enabled on the remote Elasticsearch machine

secret: None > Update the secret (password) that is used to access the remote Elasticsearch machine, if Authentication is enabled on the remote Elasticsearch machine.

You also require to assign ngnix permission to the SSL certificate that you have specified in the db_config.yml file using the following command:

chown nginx:nginx filename.pem

Migration of Elasticsearch data

Once you complete the externalization of Elasticsearch, you will require to migrate your data from your local instance to the remote Elasticsearch machine.

To migrate the remote Elasticsearch machine run the following command on your FortiSOAR instance as a root user:

$ sudo -u nginx php /opt/cyops-api/bin/console app:elastic:create --env="prod"

Troubleshooting

FortiSOAR Search Errors

FortiSOAR Search performs indexing in an asynchronous fashion in the backend. Users could be faced with certain scenarios that could lead to a restart of services, which can cause indexing to stop. In this case, FortiSOAR might display any of the following errors when users are performing a search operation on FortiSOAR:

  • Search indexing is in progress. Partial results are returned.
  • Search indexing has stopped. You must manually rerun indexing (see product documentation for instructions) or raise a support ticket for the same.
  • We are sorry, but the server encountered an error while handling your search request. Please contact your administrator for assistance.

In this case, use the /var/log/cyops/cyops-search/falcon.log log file to check which modules are published and indexed and which modules are yet to be published (pending).

For example, the /var/log/cyops/cyops-search/falcon.log log file will display results as follows:

2019-02-13,11:00:44 INFO blocking_connection: _dispatch_events(): 1445: Module Currently Getting Published: ['attachments']
2019-02-13,11:00:44 INFO blocking_connection: _dispatch_events(): 1445: Indexing for Module: 'attachments' started Total Records to be indexed: '1'
2019-02-13,11:00:49 INFO blocking_connection: _dispatch_events(): 1445: Module: 'attachments' Successful Total Records indexed: '1'
2019-02-13,11:00:49 INFO blocking_connection: _dispatch_events(): 1445: on_publish_message called
2019-02-13,11:00:53 INFO blocking_connection: _dispatch_events(): 1445: creating index with mapping
2019-02-13,11:01:00 INFO blocking_connection: _dispatch_events(): 1445: Module Currently Getting Published: ['emails']
2019-02-13,11:01:02 INFO blocking_connection: _dispatch_events(): 1445: Indexing for Module: 'emails' started Total Records to be indexed: '1'
2019-02-13,11:01:04 INFO blocking_connection: _dispatch_events(): 1445: Module: 'emails' Successful Total Records indexed: '1'

The above example shows the attachments and emails modules currently being indexed and its total number of records. Any failure in indexing any modules will be logged here. You can monitor the progress of this file while the indexing is in progress.

If any module(s) are missing from the published list or if any module has the Publish Module: '<name of module>' Unsuccessful listed in the /var/log/cyops/cyops-search/falcon.log log file; the indicators and tasks modules in our example, then you must manually run the indexing for those module(s) using the following command:

$ sudo -u nginx php app/console app:elastic:create --env="prod" --index='{"type":["<list of comma-seperated module names that require to be indexed>"]}

For our example, run the following command:

$ sudo -u nginx php app/console app:elastic:create --env="prod" --index='{"type":["indicators","tasks"]}'