Fortinet black logo

Data Ingestion

Copy Link
Copy Doc ID 055c1301-9e84-11eb-b70b-00505692583a:682058
Download PDF

Data Ingestion

FortiSOAR has a dedicated data ingestion wizard that facilitates data ingestion from external SIEM solutions and other third-party sources like threat intelligence platforms, email solutions, etc. The wizard also takes care of scheduling of data ingestion into FortiSOAR, if the connector is enabled for scheduling.

The Data Ingestion wizard eases data ingestion configuration and mapping of fields between the two systems. The wizard assists in fetching sample data from the source, mapping fields from the source data into a FortiSOAR module and sets up an ingestion schedule if required and if the connector is enabled for scheduling. Based on the inputs provided in the wizard, the system dynamically generates the ingestion workflows without the user being exposed to the details of playbooks and easing the process of configuring ingestion.

The Data Ingestion wizard allows mapping of dropdown (picklist) fields and items such that you can map picklist items to a range of values, lesser than, and greater than values. The Data Ingestion wizard allows you to define the frequency at which you want to pull content into FortiSOAR from third-party integration, such as SIEMs that allow you to define the polling frequency. By default, the frequency of pulling content is set at five minutes.

FortiSOAR and its connectors come with sample playbooks for ingesting data from various SIEM sources, Threat Intelligence Platforms, Vulnerability Management Tools, Configuration Management Databases, etc. Connectors that you can use to ingest data through this wizard are enabled by default in FortiSOAR.

FortiSOAR 6.0.0 has enhanced the data ingestion as follows:

  • Support for fetching data for each configuration of your connector, i.e., if you have two configurations present in your connector, then you can pull ingestion data for each configuration.
  • Support for multiple queries for pulling data based on your requirement; however, you must specify one query to schedule pulling of data from the connector into FortiSOAR.
  • Enhanced the UI of the data ingestion wizard to make it easier to work with the wizard.
  • Added a Data Ingestion tab on the Connectors page that monitors your data ingestion and provides information on which connectors are configured for using the Data Ingestion Wizard, as well as other information such as what is the status of a configuration, what is the schedule for ingestion, when data was last pulled using that configuration, etc.
  • Added support for inserting or upserting records in bulk during data ingestion, which improves the performance since all the records are created or upserted in one request.

Modes of Data Ingestion

Data Ingestion can work in any of the following three modes:

  • Notification Based: Some connectors such as, IMAP, Exchange, Syslog, etc have a Notification Service that gets instantly notified when a message arrives on the server. The notification service, in turn, triggers a FortiSOAR playbook to create FortiSOAR records from the fetched data.
  • Schedule Based: Some connectors such as QRadar, Anomali, ServiceNow, etc use the Fetch APIs of the integration, i.e., for example, the fetch API of Anomali, along with a user-defined query to fetch data from the product into FortiSOAR. These fetch playbooks are scheduled to run by default, every 5 mins or can also be scheduled to run according to a user-defined interval.
    Note that most integrations that support a Notification Based ingestion also support a Schedule Based ingestion. You must, however, configure only one of the two since otherwise both would pull the data from the same source and there might be data loss due to conflicts.
  • App Push: Some connectors such as Splunk have an add-on that is provided by FortiSOAR and which can be installed on the server side to push data from the integration to FortiSOAR. The add-on is configured with a user-password or appliance-based authentication in FortiSOAR and it triggers FortiSOAR playbooks to create the records in FortiSOAR.

Permissions required for using the Data Ingestion Wizard

To use data ingestion, you must be assigned a role that has Read and Update permissions on the Connectors module and Create, Read, Update, and Execute permissions on the Playbooks module. If you want to schedule the data ingestion, then you also require to have Create, Update and Read permissions on the Schedules module.

Data Ingestion Wizard

The Data Ingestion tab displays connectors that are configured for using the Data Ingestion Wizard, i.e., you can use the data ingestion wizard to ingest data into FortiSOAR for these connectors.

To view the data ingestion tab, log on to FortiSOAR and on the left navigation pane, click Automation > Connectors. Click the Data Ingestion tab, which will display all the connectors that are enabled for data ingestion, along with the number of configurations available for that connector.

Data Ingestion tab on the Connectors page

You can filter the display on the Data Ingestion page to show all the connectors that are enabled for data ingestion, whether configured or not, by clicking the All (default) filter. To view connectors that are enabled for data ingestion, but not configured, click the Not Configured filter, and to view connectors that are enabled for data ingestion that are configured, click the Configured filter. You can search for a connector by connector name in the Search by connector name field. In case of connectors that are not configured, you can click the Add Configuration link to open the Connector Configuration page of that connector enabling you to configure the connector.

From version 6.4.3 onwards, support is added for running the data ingestion wizard on agent configurations. To display the details of the configurations, click the <number of configurations> configurations available link. To view the configuration details of the current node, click Self. To view the agent configuration details, click Agent and then from the drop-down list, select the name of the agent whose configuration details you want to view, as shown in the following image:

Data Ingestion - Agent Configuration Details page

Details that are displayed for the configurations are as following:

  • Status of a configuration, whether it is Available or Disconnected.
  • Schedule for running the ingestion, i.e., what is the frequency at which the third-party integration will be polled to ingest data into FortiSOAR.
  • Last Data Pulled on that contains the datetime when the data was last pulled into FortiSOAR using this configuration.
  • Actions contains the following options:
    Configure Ingestion: Clicking this link displays the Data Ingestion Wizard. This option is displayed when you are configuring data ingestion for the first time for a particular ingestion.
    Ingestion Settings: Clicking this link displays the Data Ingestion Wizard with the configured ingestion settings. This option is displayed when you have already configured data ingestion for a particular configuration.
    Ingestion Playbooks: Clicking this link opens the ingestion playbooks collection in a new window.
    Start Ingestion / Stop Ingestion: Clicking this link either starts an ingestion for a newly-added configuration or stops an ingestion for a particular configuration based on the ingestion settings you have configured.
    Data Ingestion - Configuration Details
Caution

If you have upgraded to FortiSOAR 6.0.0 or later from an earlier version, then before you reconfigure ingestion for the same connector configuration, you must deactivate the earlier ingestion playbooks that are present in the ingestion collection for the connector. The links to the ingestion playbooks that were created prior to the upgrade will be present on the System Fixtures page in the "Ingestion Playbooks" section will not be visible in the Data Ingestion tab (new in version 6.0.0) of the "Connectors" page. If your data ingestion is schedule-based, then you must also stop or delete the earlier schedules for the connector.

Process of ingesting data using the Data Ingestion Wizard

  1. Log on to FortiSOAR.
  2. On the left navigation pane, click Automation > Connectors > Installed.
    On Installed Connectors page, you will see the list of installed connectors, either in grid/list view or in the card view. You will also see an Data Ingestion tab on the Connectors page, for more information, see Data Ingestion.
    Following is an image of the Installed Connectors page in the Card view:
    Connector Page -  Card View
  3. Click a connector using which you want to ingest data. For our example, we have chosen Anomali ThreatStream.
  4. On the Connector Configuration pane, from the Select Configuration drop-down list select the configuration for which you want to configure the ingestion, and then click Configure Data Ingestion.
    Connector Configuration Pane with the Data Ingestion Configuration Button
    Only connectors that have been enabled for data ingestion will have the Configure Data Ingestion button. To ingest data, your connector must be in the Available state.
  5. Click Configure Data Ingestion to display the Start screen of the Data Ingestion Wizard.
    Data Ingestion Wizard - Start Screen
  6. Click the Let's Start.. button to display the Fetch Sample Data screen.
  7. Sample data is required to create a field mapping between your connector data and FortiSOAR. The sample data is pulled from connector actions or ingestion playbooks. By default, FortiSOAR ingests data using an "Ingestion Playbook" that is included by default with each connector that is enabled for data ingestion.
    A connector would generally require some configuration details, which you will require to fill in on this screen. For example, in the case of Anomali ThreatStream, in the Search Query field, enter the query based on which you want to fetch data into FortiSOAR. In the Number of Incidents to Fetch field, enter the number of incidents that will be fetched from Anomali ThreatStream, for example 10. In the Fetch Incidents in Last X Min, enter the number of minutes for which indicators that will be fetched from Anomali ThreatStream, for example 10, in this case the data ingestion wizard will fetch indicators created in the last 10 minutes in Anomali ThreatStream, and then click Fetch Data.
    Data Ingestion Wizard - Fetch Sample Data Screen: Anomali ThreatStream connector
    Another example is Syslog where you will see the Message_samples field in which you can add samples of messages in the CEF Format using which you want to map the fields:
    Data Ingestion Wizard - Fetch Sample Data Screen: Syslog
    Or, as is the case of HP ArcSight, you will see the Event IDs field, in which you require to specify a comma-separated list of valid Event IDs using which you want to map the fields.
    Data Ingestion Wizard - Fetch Sample Data Screen: HP ArcSight
  8. On the Field Mapping screen, map the fields of the sample data to the fields present in FortiSOAR as follows:
    1. The Field Mapping screen displays the Sample Data on the right side of the screen and the Field Mapping (FortiSOAR fields) on the left side of the screen. The sample data is in the form of a Key-Value pair.
      From the Module drop-down list that appears next to Field Mapping, select the FortiSOAR module for which you want to map the fields. The default module will be already be selected, for example, Indicators.
      Note: If you select any module other than the default module, you will require to remap all the fields.
    2. From the Sample Data fields, map the fields onto the fields present in the Indicators module as follows:
      Important: Some fields such as Name and some picklists can come pre-mapped with their keys. You do not require to re-map these fields unless you want to override their default values. You can search for fields in the record and in the sample data.
      To map a field, click the key in the sample data to add jinja for the field. For example, description from the Anomali ThreatStream sample data can be mapped to Description field in FortiSOAR by clicking description. Once you click description in Sample Data, its key will be added to the Description field:
      Data Ingestion Wizard - Mapping Text fields
      To view the key of a field, double click in the field, for example if you click the newly-mapped description field, its key {{vars.sourcedata["description"]}} will be displayed. Once the key (which is in jinja) is displayed, this field will always be displayed with the key value.
      Note: If bulk insert or upsert is supported for the connector, for example, IMAP, then the key value changes from {{vars.sourcedata["name of the field"]}} to {{vars.item["name of the field"]}}. For example, {{vars.item["description"]}}.
      From version 7.0.0 onwards, if your administrator has enabled any 'Lookup' or 'Picklist' type of field to accept the values generated from the recommendation engine, then you will see an Auto populate checkbox appearing beside this field. For example, the 'Type' field in the following image:
      Auto Populate option for lookup or picklist type fields
      If you select the Auto populate checkbox, and users have not specified any values for such fields, then the value of such fields get auto-populated with the values from the recommendation engine that is based on learning from past similar records.
      Picklists can come pre-populated with their defined defaults, if they are available. You can also map picklists using the wizard.
      To map a picklist, such as Reputation, select the picklist whose value you want to map. For example, confidence from the Anomali ThreatStream sample data can be mapped to the Reputation picklist in FortiSOAR by clicking confidence. FortiSOAR would already have mapped this picklist by default, and it will appear as confidence whose key value is {{vars.sourcedata["confidence"]}}. Next, you must map the items of the Reputation picklists. To view and map the items of the Reputation picklist in FortiSOAR, click the down arrow (v). The Reputation picklist has items such as Good, Malicious, Suspicious, TBD, and No Reputation Available. These picklist items can be mapped as per the values defined in the source. You can map more than one picklist item in FortiSOAR can map to a single value in the source. For example, you can map both TBD and No Reputation Available to 0 in FortiSOAR. You can also map one picklist item in FortiSOAR to two values or more values in the source, for example, Suspicious can be 1,2,3,4,5. In this case, if the confidence field in Anomali ThreatStream has value 1, 2, 3, 4, or 5, then all cases with confidence 1 to 5 will be mapped to Suspicious reputation in FortiSOAR. You can also specify mapping of a range of values, lesser than, and greater than values. For example, Good can be mapped 91..100 in FortiSOAR, which means that if the confidence field in Anomali ThreatStream contains any value between 91 to 100 they will be mapped to Good in FortiSOAR. Similarly, if you map Malicious to >5<91 in FortiSOAR this means that if the confidence field in Anomali ThreatStream contains any value greater than 5 and less than 91, then they will be mapped to Malicious reputation in FortiSOAR.
      Field Mapping - Mapping the reputation field
      Note: Multiple expressions of type "number" must be separated by a space. Also, in case of a range there must be two dots between the numbers.
      You can also specify string values in a picklist such as Open, In Progress, Closed, etc. Multiple expressions of type "string" must be comma-separated.
      In case you want to use another query to fetch additional data in order to create comprehensive mapping, click the + Fetch additional sample data link that appears in the Sample Data header. Clicking the + Fetch additional sample data link opens the Configurations dialog in which you can change the configuration such as, updating the Search_query and click Fetch Data. The data ingestion wizard fetches the data based on the updated configuration in a new page in the Sample Data section. You can continue mapping based on the newly fetched data.
      Once you are satisfied with the mappings, click Save Mapping & Continue.
  9. (Optional) If your connector is enabled for scheduling, such as Anomali ThreatStream, then you will be shown the Scheduling screen, using which you can specify the schedule for data ingestion from the connector into FortiSOAR, i.e., you can specify the polling frequency to a third-party integration, such as SIEM, so that the content gets pulled from the third-party integration into FortiSOAR. By default, scheduling is set to pull data every 5 minutes.
    To configure scheduling, from the Do you want to schedule the ingestion? drop-down list, select Yes.
    In the Configure Schedule Settings section, specify the Cron expression for the schedule. For example, if you want to pull data from Anomali ThreatStream every 15 minutes, click Every X Minute and in the minute box enter */15.
    Data Ingestion Wizard - Scheduling
    Note: If you select the Limit execution to one active instance at a time checkbox, then do not rerun the workflow (schedule) if the previous schedule is still running. Some connectors such as IMAP support data ingestion using both a scheduled-based pull as well as instant notification using the Notification Service. You must, however, configure only one of the two, since both would otherwise pull the data from the same source and there could be data loss due to conflicts. Therefore, the Scheduling screen for the IMAP connector displays the following warning:
    Data Ingestion Wizard - Scheduling for IMAP
    Only if you want to ingest data using a scheduled-based pull, then you should use the Scheduling screen to specify the schedule for data ingestion from the IMAP connector into FortiSOAR.
    Once you are satisfied with the scheduling, click Save Settings & Continue.
  10. The Summary screen displays a summary of the mapping done, and it also contains links to the Ingestion playbooks.
    Data Ingestion Wizard - Summary for McAfee ESM Connector
    Click Done to complete the data ingestion and exit the Data Ingestion Wizard.

Notes for Data Ingestion Playbooks:

  • The Summary step contains links to playbooks that contain the data mapping based on the mappings you have specified in the wizard. You can click the links and open the playbook in the Playbook Designer.
    For example, to open the playbook that fetches data from the Anomali ThreatStream connector, click Anomali ThreatStream > Ingest link, which opens the Anomali ThreatStream > Ingest playbook in the Playbook Designer.
  • You can open the Data Ingestion Playbooks collection by clicking here in the Summary step.
  • You can modify these data ingestion playbooks or the mappings in the playbooks using the Playbook Designer and those modifications are reflected back in the mappings, i.e., if you open the "Data Ingestion Wizard" again you will see the modified content.

Notes for Data Ingestion

  • If you are using the IMAP connector for data ingestion, then you must ensure that you configure either a scheduled-based pull, using Scheduling in the Data Ingestion Wizard, or the instant notification method using the Email Notification Service, by clicking the Enable Email Notification Service in the IMAP connector configuration pane. If you configure both then both the methods would pull the data from the same source and there could be data loss due to conflicts.
  • If you are using the IMAP connector for data ingestion and at the field mapping stage you change the default module for data ingestion, which is set as Alerts to any other module, for example, Emails, then the playbook used for ingesting data remain active and no records will be created. To get the data ingestion to work correctly, do the following:
    Open the > IMAP > Create Record playbook and click the Upsert File Indicators step. In the Upsert File Indicators step, you need to initialize the IRI value of the module that you want to use for data ingestion instead of the alert record IRI. To achieve this, click the emailRecordIRI field, and use "Dynamic Values" to enter the IRI value of the required module or enter the following Jinja value: {{vars.steps.Create_Record['@id']}}.
    For example, if you have changed the module used for data ingestion from Alerts to Emails, then to initialize the emailRecordIRI variable instead of alertRecordIRI, in the emailRecordIRI field, enter the {{vars.steps.Create_Record['@id']}} value.
    Also, note that If you select any module other than the default module, you will require to remap all the fields.
  • If you are using the Syslog connector for data ingestion, then you must ensure playbooks used for data ingestion provide input in the CEF format, otherwise you require to modify the playbooks. Since CEF is the most common format for Syslog messages, the default data ingestion playbooks parse the message as CEF. If the Syslog messages from your server are non-CEF complaint, you can modify the "Parse CEF" step in the > Syslog> Fetch playbook that is part of the playbooks shipped with the Syslog connector, before you configure Data Ingestion. The Syslog connector also provides "Parse Message" action to parse RFC 3164 and RFC 5424 formatted messages and convert these message formats to CEF format, instead of using the "Parse CEF" step.

Troubleshooting

The Fetch Data screen in case of IMAP displays the`ERROR :: create failed: [ALREADYEXISTS] Folder name ... error

This error occurs when you are running the Data Ingestion Wizard using the IMAP connector. Click Fetch Data on the Fetch Data screen and an error such as "> IMAP > Fetch has failed because CS_INTEGRATION-5: Error occured while executing the connector action ERROR:: create failed [ALREADYEXISTS] Folder name conflicts with existing folder name........" is displayed.

The above error can occur due to the following reasons:

  • This issue occurs if the Email destination folder specified on server is similar to an existing folder. Mailbox names are case sensitive on creation of folders using the IMAP client, for example, if you specify MyFolder in the in the Email destination field and a folder named myfolder already exists on the server, then this error will be displayed on the Fetch Data screen when you are running the Data Ingestion Wizard.
  • This error also occurs if the folder that you have specified does not have permission for the IMAP client.

Resolution

  • In the IMAP configuration pane, update the folder name in the Email destination field and save the configuration:
    IMAP Configuration: Email destination field
  • Ensure that you provide appropriate permissions to the email destination folder for the IMAP client.

Post upgrade to FortiSOAR 6.0.0 or later from version 5.1.0 does not save any data ingestion configurations

If you have upgraded your FortiSOAR version from 5.1.0 to 6.0.0 or later, you will see data ingestion configurations that you had saved in version 5.1.0 are not available in version 6.0.0. You will not see any schedules, ingestion playbook link, or actions button on the Data Ingestion page. However, your data ingestion will continue to run. This is because support for fetching data for each configuration of your connector has been introduced in version 6.0.0 and therefore, the older data ingestion configurations and playbooks have been deprecated.

Resolution

Rerun the data ingestion wizard, i.e., reconfigure your data ingestion if required. Note that your previously configured data ingestion will continue to run, however, you will not be able to see the historical data. If you do require to reconfigure the data ingestion, then you must ensure that you stop your previous configured data ingestion, otherwise both the current and the old data ingestion will continue to run.

Data Ingestion

FortiSOAR has a dedicated data ingestion wizard that facilitates data ingestion from external SIEM solutions and other third-party sources like threat intelligence platforms, email solutions, etc. The wizard also takes care of scheduling of data ingestion into FortiSOAR, if the connector is enabled for scheduling.

The Data Ingestion wizard eases data ingestion configuration and mapping of fields between the two systems. The wizard assists in fetching sample data from the source, mapping fields from the source data into a FortiSOAR module and sets up an ingestion schedule if required and if the connector is enabled for scheduling. Based on the inputs provided in the wizard, the system dynamically generates the ingestion workflows without the user being exposed to the details of playbooks and easing the process of configuring ingestion.

The Data Ingestion wizard allows mapping of dropdown (picklist) fields and items such that you can map picklist items to a range of values, lesser than, and greater than values. The Data Ingestion wizard allows you to define the frequency at which you want to pull content into FortiSOAR from third-party integration, such as SIEMs that allow you to define the polling frequency. By default, the frequency of pulling content is set at five minutes.

FortiSOAR and its connectors come with sample playbooks for ingesting data from various SIEM sources, Threat Intelligence Platforms, Vulnerability Management Tools, Configuration Management Databases, etc. Connectors that you can use to ingest data through this wizard are enabled by default in FortiSOAR.

FortiSOAR 6.0.0 has enhanced the data ingestion as follows:

  • Support for fetching data for each configuration of your connector, i.e., if you have two configurations present in your connector, then you can pull ingestion data for each configuration.
  • Support for multiple queries for pulling data based on your requirement; however, you must specify one query to schedule pulling of data from the connector into FortiSOAR.
  • Enhanced the UI of the data ingestion wizard to make it easier to work with the wizard.
  • Added a Data Ingestion tab on the Connectors page that monitors your data ingestion and provides information on which connectors are configured for using the Data Ingestion Wizard, as well as other information such as what is the status of a configuration, what is the schedule for ingestion, when data was last pulled using that configuration, etc.
  • Added support for inserting or upserting records in bulk during data ingestion, which improves the performance since all the records are created or upserted in one request.

Modes of Data Ingestion

Data Ingestion can work in any of the following three modes:

  • Notification Based: Some connectors such as, IMAP, Exchange, Syslog, etc have a Notification Service that gets instantly notified when a message arrives on the server. The notification service, in turn, triggers a FortiSOAR playbook to create FortiSOAR records from the fetched data.
  • Schedule Based: Some connectors such as QRadar, Anomali, ServiceNow, etc use the Fetch APIs of the integration, i.e., for example, the fetch API of Anomali, along with a user-defined query to fetch data from the product into FortiSOAR. These fetch playbooks are scheduled to run by default, every 5 mins or can also be scheduled to run according to a user-defined interval.
    Note that most integrations that support a Notification Based ingestion also support a Schedule Based ingestion. You must, however, configure only one of the two since otherwise both would pull the data from the same source and there might be data loss due to conflicts.
  • App Push: Some connectors such as Splunk have an add-on that is provided by FortiSOAR and which can be installed on the server side to push data from the integration to FortiSOAR. The add-on is configured with a user-password or appliance-based authentication in FortiSOAR and it triggers FortiSOAR playbooks to create the records in FortiSOAR.

Permissions required for using the Data Ingestion Wizard

To use data ingestion, you must be assigned a role that has Read and Update permissions on the Connectors module and Create, Read, Update, and Execute permissions on the Playbooks module. If you want to schedule the data ingestion, then you also require to have Create, Update and Read permissions on the Schedules module.

Data Ingestion Wizard

The Data Ingestion tab displays connectors that are configured for using the Data Ingestion Wizard, i.e., you can use the data ingestion wizard to ingest data into FortiSOAR for these connectors.

To view the data ingestion tab, log on to FortiSOAR and on the left navigation pane, click Automation > Connectors. Click the Data Ingestion tab, which will display all the connectors that are enabled for data ingestion, along with the number of configurations available for that connector.

Data Ingestion tab on the Connectors page

You can filter the display on the Data Ingestion page to show all the connectors that are enabled for data ingestion, whether configured or not, by clicking the All (default) filter. To view connectors that are enabled for data ingestion, but not configured, click the Not Configured filter, and to view connectors that are enabled for data ingestion that are configured, click the Configured filter. You can search for a connector by connector name in the Search by connector name field. In case of connectors that are not configured, you can click the Add Configuration link to open the Connector Configuration page of that connector enabling you to configure the connector.

From version 6.4.3 onwards, support is added for running the data ingestion wizard on agent configurations. To display the details of the configurations, click the <number of configurations> configurations available link. To view the configuration details of the current node, click Self. To view the agent configuration details, click Agent and then from the drop-down list, select the name of the agent whose configuration details you want to view, as shown in the following image:

Data Ingestion - Agent Configuration Details page

Details that are displayed for the configurations are as following:

  • Status of a configuration, whether it is Available or Disconnected.
  • Schedule for running the ingestion, i.e., what is the frequency at which the third-party integration will be polled to ingest data into FortiSOAR.
  • Last Data Pulled on that contains the datetime when the data was last pulled into FortiSOAR using this configuration.
  • Actions contains the following options:
    Configure Ingestion: Clicking this link displays the Data Ingestion Wizard. This option is displayed when you are configuring data ingestion for the first time for a particular ingestion.
    Ingestion Settings: Clicking this link displays the Data Ingestion Wizard with the configured ingestion settings. This option is displayed when you have already configured data ingestion for a particular configuration.
    Ingestion Playbooks: Clicking this link opens the ingestion playbooks collection in a new window.
    Start Ingestion / Stop Ingestion: Clicking this link either starts an ingestion for a newly-added configuration or stops an ingestion for a particular configuration based on the ingestion settings you have configured.
    Data Ingestion - Configuration Details
Caution

If you have upgraded to FortiSOAR 6.0.0 or later from an earlier version, then before you reconfigure ingestion for the same connector configuration, you must deactivate the earlier ingestion playbooks that are present in the ingestion collection for the connector. The links to the ingestion playbooks that were created prior to the upgrade will be present on the System Fixtures page in the "Ingestion Playbooks" section will not be visible in the Data Ingestion tab (new in version 6.0.0) of the "Connectors" page. If your data ingestion is schedule-based, then you must also stop or delete the earlier schedules for the connector.

Process of ingesting data using the Data Ingestion Wizard

  1. Log on to FortiSOAR.
  2. On the left navigation pane, click Automation > Connectors > Installed.
    On Installed Connectors page, you will see the list of installed connectors, either in grid/list view or in the card view. You will also see an Data Ingestion tab on the Connectors page, for more information, see Data Ingestion.
    Following is an image of the Installed Connectors page in the Card view:
    Connector Page -  Card View
  3. Click a connector using which you want to ingest data. For our example, we have chosen Anomali ThreatStream.
  4. On the Connector Configuration pane, from the Select Configuration drop-down list select the configuration for which you want to configure the ingestion, and then click Configure Data Ingestion.
    Connector Configuration Pane with the Data Ingestion Configuration Button
    Only connectors that have been enabled for data ingestion will have the Configure Data Ingestion button. To ingest data, your connector must be in the Available state.
  5. Click Configure Data Ingestion to display the Start screen of the Data Ingestion Wizard.
    Data Ingestion Wizard - Start Screen
  6. Click the Let's Start.. button to display the Fetch Sample Data screen.
  7. Sample data is required to create a field mapping between your connector data and FortiSOAR. The sample data is pulled from connector actions or ingestion playbooks. By default, FortiSOAR ingests data using an "Ingestion Playbook" that is included by default with each connector that is enabled for data ingestion.
    A connector would generally require some configuration details, which you will require to fill in on this screen. For example, in the case of Anomali ThreatStream, in the Search Query field, enter the query based on which you want to fetch data into FortiSOAR. In the Number of Incidents to Fetch field, enter the number of incidents that will be fetched from Anomali ThreatStream, for example 10. In the Fetch Incidents in Last X Min, enter the number of minutes for which indicators that will be fetched from Anomali ThreatStream, for example 10, in this case the data ingestion wizard will fetch indicators created in the last 10 minutes in Anomali ThreatStream, and then click Fetch Data.
    Data Ingestion Wizard - Fetch Sample Data Screen: Anomali ThreatStream connector
    Another example is Syslog where you will see the Message_samples field in which you can add samples of messages in the CEF Format using which you want to map the fields:
    Data Ingestion Wizard - Fetch Sample Data Screen: Syslog
    Or, as is the case of HP ArcSight, you will see the Event IDs field, in which you require to specify a comma-separated list of valid Event IDs using which you want to map the fields.
    Data Ingestion Wizard - Fetch Sample Data Screen: HP ArcSight
  8. On the Field Mapping screen, map the fields of the sample data to the fields present in FortiSOAR as follows:
    1. The Field Mapping screen displays the Sample Data on the right side of the screen and the Field Mapping (FortiSOAR fields) on the left side of the screen. The sample data is in the form of a Key-Value pair.
      From the Module drop-down list that appears next to Field Mapping, select the FortiSOAR module for which you want to map the fields. The default module will be already be selected, for example, Indicators.
      Note: If you select any module other than the default module, you will require to remap all the fields.
    2. From the Sample Data fields, map the fields onto the fields present in the Indicators module as follows:
      Important: Some fields such as Name and some picklists can come pre-mapped with their keys. You do not require to re-map these fields unless you want to override their default values. You can search for fields in the record and in the sample data.
      To map a field, click the key in the sample data to add jinja for the field. For example, description from the Anomali ThreatStream sample data can be mapped to Description field in FortiSOAR by clicking description. Once you click description in Sample Data, its key will be added to the Description field:
      Data Ingestion Wizard - Mapping Text fields
      To view the key of a field, double click in the field, for example if you click the newly-mapped description field, its key {{vars.sourcedata["description"]}} will be displayed. Once the key (which is in jinja) is displayed, this field will always be displayed with the key value.
      Note: If bulk insert or upsert is supported for the connector, for example, IMAP, then the key value changes from {{vars.sourcedata["name of the field"]}} to {{vars.item["name of the field"]}}. For example, {{vars.item["description"]}}.
      From version 7.0.0 onwards, if your administrator has enabled any 'Lookup' or 'Picklist' type of field to accept the values generated from the recommendation engine, then you will see an Auto populate checkbox appearing beside this field. For example, the 'Type' field in the following image:
      Auto Populate option for lookup or picklist type fields
      If you select the Auto populate checkbox, and users have not specified any values for such fields, then the value of such fields get auto-populated with the values from the recommendation engine that is based on learning from past similar records.
      Picklists can come pre-populated with their defined defaults, if they are available. You can also map picklists using the wizard.
      To map a picklist, such as Reputation, select the picklist whose value you want to map. For example, confidence from the Anomali ThreatStream sample data can be mapped to the Reputation picklist in FortiSOAR by clicking confidence. FortiSOAR would already have mapped this picklist by default, and it will appear as confidence whose key value is {{vars.sourcedata["confidence"]}}. Next, you must map the items of the Reputation picklists. To view and map the items of the Reputation picklist in FortiSOAR, click the down arrow (v). The Reputation picklist has items such as Good, Malicious, Suspicious, TBD, and No Reputation Available. These picklist items can be mapped as per the values defined in the source. You can map more than one picklist item in FortiSOAR can map to a single value in the source. For example, you can map both TBD and No Reputation Available to 0 in FortiSOAR. You can also map one picklist item in FortiSOAR to two values or more values in the source, for example, Suspicious can be 1,2,3,4,5. In this case, if the confidence field in Anomali ThreatStream has value 1, 2, 3, 4, or 5, then all cases with confidence 1 to 5 will be mapped to Suspicious reputation in FortiSOAR. You can also specify mapping of a range of values, lesser than, and greater than values. For example, Good can be mapped 91..100 in FortiSOAR, which means that if the confidence field in Anomali ThreatStream contains any value between 91 to 100 they will be mapped to Good in FortiSOAR. Similarly, if you map Malicious to >5<91 in FortiSOAR this means that if the confidence field in Anomali ThreatStream contains any value greater than 5 and less than 91, then they will be mapped to Malicious reputation in FortiSOAR.
      Field Mapping - Mapping the reputation field
      Note: Multiple expressions of type "number" must be separated by a space. Also, in case of a range there must be two dots between the numbers.
      You can also specify string values in a picklist such as Open, In Progress, Closed, etc. Multiple expressions of type "string" must be comma-separated.
      In case you want to use another query to fetch additional data in order to create comprehensive mapping, click the + Fetch additional sample data link that appears in the Sample Data header. Clicking the + Fetch additional sample data link opens the Configurations dialog in which you can change the configuration such as, updating the Search_query and click Fetch Data. The data ingestion wizard fetches the data based on the updated configuration in a new page in the Sample Data section. You can continue mapping based on the newly fetched data.
      Once you are satisfied with the mappings, click Save Mapping & Continue.
  9. (Optional) If your connector is enabled for scheduling, such as Anomali ThreatStream, then you will be shown the Scheduling screen, using which you can specify the schedule for data ingestion from the connector into FortiSOAR, i.e., you can specify the polling frequency to a third-party integration, such as SIEM, so that the content gets pulled from the third-party integration into FortiSOAR. By default, scheduling is set to pull data every 5 minutes.
    To configure scheduling, from the Do you want to schedule the ingestion? drop-down list, select Yes.
    In the Configure Schedule Settings section, specify the Cron expression for the schedule. For example, if you want to pull data from Anomali ThreatStream every 15 minutes, click Every X Minute and in the minute box enter */15.
    Data Ingestion Wizard - Scheduling
    Note: If you select the Limit execution to one active instance at a time checkbox, then do not rerun the workflow (schedule) if the previous schedule is still running. Some connectors such as IMAP support data ingestion using both a scheduled-based pull as well as instant notification using the Notification Service. You must, however, configure only one of the two, since both would otherwise pull the data from the same source and there could be data loss due to conflicts. Therefore, the Scheduling screen for the IMAP connector displays the following warning:
    Data Ingestion Wizard - Scheduling for IMAP
    Only if you want to ingest data using a scheduled-based pull, then you should use the Scheduling screen to specify the schedule for data ingestion from the IMAP connector into FortiSOAR.
    Once you are satisfied with the scheduling, click Save Settings & Continue.
  10. The Summary screen displays a summary of the mapping done, and it also contains links to the Ingestion playbooks.
    Data Ingestion Wizard - Summary for McAfee ESM Connector
    Click Done to complete the data ingestion and exit the Data Ingestion Wizard.

Notes for Data Ingestion Playbooks:

  • The Summary step contains links to playbooks that contain the data mapping based on the mappings you have specified in the wizard. You can click the links and open the playbook in the Playbook Designer.
    For example, to open the playbook that fetches data from the Anomali ThreatStream connector, click Anomali ThreatStream > Ingest link, which opens the Anomali ThreatStream > Ingest playbook in the Playbook Designer.
  • You can open the Data Ingestion Playbooks collection by clicking here in the Summary step.
  • You can modify these data ingestion playbooks or the mappings in the playbooks using the Playbook Designer and those modifications are reflected back in the mappings, i.e., if you open the "Data Ingestion Wizard" again you will see the modified content.

Notes for Data Ingestion

  • If you are using the IMAP connector for data ingestion, then you must ensure that you configure either a scheduled-based pull, using Scheduling in the Data Ingestion Wizard, or the instant notification method using the Email Notification Service, by clicking the Enable Email Notification Service in the IMAP connector configuration pane. If you configure both then both the methods would pull the data from the same source and there could be data loss due to conflicts.
  • If you are using the IMAP connector for data ingestion and at the field mapping stage you change the default module for data ingestion, which is set as Alerts to any other module, for example, Emails, then the playbook used for ingesting data remain active and no records will be created. To get the data ingestion to work correctly, do the following:
    Open the > IMAP > Create Record playbook and click the Upsert File Indicators step. In the Upsert File Indicators step, you need to initialize the IRI value of the module that you want to use for data ingestion instead of the alert record IRI. To achieve this, click the emailRecordIRI field, and use "Dynamic Values" to enter the IRI value of the required module or enter the following Jinja value: {{vars.steps.Create_Record['@id']}}.
    For example, if you have changed the module used for data ingestion from Alerts to Emails, then to initialize the emailRecordIRI variable instead of alertRecordIRI, in the emailRecordIRI field, enter the {{vars.steps.Create_Record['@id']}} value.
    Also, note that If you select any module other than the default module, you will require to remap all the fields.
  • If you are using the Syslog connector for data ingestion, then you must ensure playbooks used for data ingestion provide input in the CEF format, otherwise you require to modify the playbooks. Since CEF is the most common format for Syslog messages, the default data ingestion playbooks parse the message as CEF. If the Syslog messages from your server are non-CEF complaint, you can modify the "Parse CEF" step in the > Syslog> Fetch playbook that is part of the playbooks shipped with the Syslog connector, before you configure Data Ingestion. The Syslog connector also provides "Parse Message" action to parse RFC 3164 and RFC 5424 formatted messages and convert these message formats to CEF format, instead of using the "Parse CEF" step.

Troubleshooting

The Fetch Data screen in case of IMAP displays the`ERROR :: create failed: [ALREADYEXISTS] Folder name ... error

This error occurs when you are running the Data Ingestion Wizard using the IMAP connector. Click Fetch Data on the Fetch Data screen and an error such as "> IMAP > Fetch has failed because CS_INTEGRATION-5: Error occured while executing the connector action ERROR:: create failed [ALREADYEXISTS] Folder name conflicts with existing folder name........" is displayed.

The above error can occur due to the following reasons:

  • This issue occurs if the Email destination folder specified on server is similar to an existing folder. Mailbox names are case sensitive on creation of folders using the IMAP client, for example, if you specify MyFolder in the in the Email destination field and a folder named myfolder already exists on the server, then this error will be displayed on the Fetch Data screen when you are running the Data Ingestion Wizard.
  • This error also occurs if the folder that you have specified does not have permission for the IMAP client.

Resolution

  • In the IMAP configuration pane, update the folder name in the Email destination field and save the configuration:
    IMAP Configuration: Email destination field
  • Ensure that you provide appropriate permissions to the email destination folder for the IMAP client.

Post upgrade to FortiSOAR 6.0.0 or later from version 5.1.0 does not save any data ingestion configurations

If you have upgraded your FortiSOAR version from 5.1.0 to 6.0.0 or later, you will see data ingestion configurations that you had saved in version 5.1.0 are not available in version 6.0.0. You will not see any schedules, ingestion playbook link, or actions button on the Data Ingestion page. However, your data ingestion will continue to run. This is because support for fetching data for each configuration of your connector has been introduced in version 6.0.0 and therefore, the older data ingestion configurations and playbooks have been deprecated.

Resolution

Rerun the data ingestion wizard, i.e., reconfigure your data ingestion if required. Note that your previously configured data ingestion will continue to run, however, you will not be able to see the historical data. If you do require to reconfigure the data ingestion, then you must ensure that you stop your previous configured data ingestion, otherwise both the current and the old data ingestion will continue to run.