Polling Detections

Detections are the primary mechanism by which the network event data is searched on a continuous basis for anomalies or suspicious behaviors. For this reason, in order to integrate with the FortiNDR Cloud Service, we need to be able to poll and to import all the detections regularly reporting any detections within a short period of time. This makes the polling strategy we follow extremely important since it not only needs to avoid duplication but it must ensure that no detection is missing.

In this section, we provide a quick overview of the detection object and the polling strategy used in the Client Library. We then show how to use the continuous polling method to pull and enrich the detections information.

Detections

A detection is an alert mechanism that notifies you of events impacting your device and network. Detections allow you to quickly identify and respond to suspicious or known malicious activity in your network.

Detections are identified based on both the IP address and the Sensor ID to avoid issues with overlapping IP space. A duplicate detection is not generated if an active detection already exists for the IP address and sensor ID pair. Instead, the Last Seen timestamp is updated and the event is added to the rule's Latest Events. Detections can be resolved either manually or automatically to show that the underlying cause of the events should no longer be triggering.

Detection Object

There are many fields in a Detection object. In this document we will focus on only a few of them. The Detections API documentation can be reviewed for additional details.

• Account uuid:

  The account_uuid field, is a unique identifier of the account owning this detection. Users can only see detections owned by or shared with the account they have access to.

• Device ip:

  The device_ip field contains the IP address or CIDR of the devices associated to this particular detection.

• Muted:

  Detections, as well as rules and devices can be muted to avoid receiving notifications whenever they are created or updated. The muted, muted_rule and muted_device_uuid are boolean fields that show if the detection is muted.

  • Muted_rule: All instances of that detection are muted regardless of device
  • Muted: This specific detection with this device is muted
  • Muted_device_uuid This IP is muted across all detections
• Timestamps:

  There are four timestamps on a detection object (created, first_seen, last_seen and updated). These timestamps are stored in UTC with microseconds precision (yyyy-mm-ddThh:mm:ss.000000Z). Of all of these timestamps, the created timestamp is the only one that is immutable.

Polling Strategy

The polling strategy we follow while pulling detections is very important since it not only needs to avoid duplication but it must ensure that no detection is missing. Below we show the key aspects for the polling strategy used in the FncApiClient’s continuous_polling method:

• Search for detections regularly:

  Since the network’s events are constantly being processed, we need to search for new detections using the GET Detections endpoint. This search needs to be performed regularly so we get the detections as fast as possible. The recommended search interval is 5 minutes.

• Use of a fixed window:

  It is recommended to fix the end date as well as the start date. In this way we can ensure there is no overlap in the search window the next time the search is performed.

• Allow delay (recommended 10 minutes):

  While processing the detections, there is a time difference between when the detection is created and when the detection can be retrieved by the API. This delay allows some time for the detection to be processed, avoiding missing any detection. It is recommended to make this delay configurable so it can be updated if needed.

• Use created_or_shared_start_date and created_or_shared_end_date:

  The created timestamp is the only immutable one in the detection object. For this reason, it is recommended you use it for the fixed search window. However, there could be a time difference between when the detection is created and when it is shared with the specific account_uuid. The created_or_shared_start_date and created_or_shared_end_date filters in the Get detections endpoint take both into account. Using these timestamps helps avoid missing detections.

Taking into account the above key aspects, the recommended polling strategy would be:

• Get the last checkpoint or the configured start date if it is the first time the search is called.
• Set the fix window as:
  • start_date = last_checkpoint or the configured start date
  • end_date = utc_now() - 10min

    Ensure start_date is before the end_date.

• Use the Get Detections endpoint to search for detections incrementally with the following arguments:
  • created_or_shared_start_date = start_date
  • created_or_shared_end_date = end_date
  • sort_by = device_ip (This is required to ensure order is kept for every page)
  • sort_order = asc
  • limit = 10000 (This is the maximum page size)
  • include = rules (Use the include argument to provide extra information for the rules and/or indicators. See, GET_DETECTIONS
  • offset = (Started in 0 and incremented 10000 each time until no detection is received)
• Import each detection using the detection['create'] as the timestamp.
• Update the last_checkpoint to be the end_date of the fixed window
• Repeat every 5 minutes

Important: It is important to know that all the timestamps handled by FortiNDR Cloud Service APIs do not have any time zone information and are assumed to be in UTC. Therefore, you must ensure that any timestamp you provide is in UTC. Also, if you need to use any retrieved timestamp, the time zone information needs to be added prior to using it.