Fetch Events History

The historical event data can be retrieved by calling the fetch_events_by_days method for each of the last 7 days. However, given the large amount of information being retrieved it could take a long time. For that reason, the FncMetastreamClient exposes a poll_history method that allows you to retrieve historical data in smaller chunks. This method requires a context with the history value set to pull data from the history[‘start_date’] up to history [‘end_date’]. To learn how to split the context into history and current, see Fetch Events History

Once the context is split, we are ready to start pulling the historical data using the FncMetastreamClient.poll_history method. The steps involved are:

1. Imports

The main classes required while calling specific endpoint are shown below.

from fnc.errors import FncClientError

from fnc.fnc_client import FncClient

from fnc.metastream.metastream_client import FncMetastreamClient

from fnc.metastream.s3_client import MetastreamContext

2. Get the Client

The FncMetastreamClient is created using the FncClient class’s method and providing the required arguments. For a detailed description, see Getting the client.

3. Split the Context

We need to pass the context with the history value to the poll_history method.

# Split the poling interval in history and current

h_context, context = client.get_splitted_context(start_date_str=start_date)

The history field in the context contains the start and end date to be pulled it can be manually created but using the above method, ensure duplications will be avoided since the end date of the history context will be the checkpoint in the current one. However, this method can only be used once

4. Fetch a piece of the historical data

The FncMetastreamClient.poll_history method is used to fetch apiece of the events for previous days. The method requires two arguments the endpoint to be called and a dictionary with its required arguments.

client.call_endpoint(day: datetime, event_type: str, limit: int = 0, context: MetastreamContext = None)

Property	Type	Required	Description
context	MetastreamContext	false	An object that stores specific session wide data such as metrics and checkpoint. It needs to contain the history value set.
event_type	string	true	The event type to download. Possible values are observation, suricata.
interval	timedelta	false	The maximum interval of time to search each time. It defaults to one day.

5. Handle Errors

Any exception occurring while fetching the events will be raised as a FncClientError exception. The specific problem can be identified by using the FncClientError fields.

Each call to this method retrieves a piece of the historical data. To fetch the whole history, it needs to be called while the history[‘start_date’] is before the history[‘end_date’].

Example

from fnc.errors import FncClientError
from fnc.fnc_client import FncClient
from fnc.metastream.metastream_client import FncMetastreamClient
from fnc.metastream.s3_client import MetastreamContext
 
api_token = ''
access_key = ''
secret_key = ''
account_code = ''
bucket = '' 
event_type = ''
client = None
 
try:
    client: FncMetastreamClient = FncClient.get_metastream_client(
      name= client_name, 
      access_key= access_key, 
      secret_key= secret_key, 
      account_code = account_code, 
      bucket_name= bucket_name
    )
    client.get_logger().set_level(level=log_level)
 
    interval = timedelta(hours=1) # By default the size of the interval will be 1 day
    # 7 days is the maximum of events historical data that can be retrieve
    h_context, context = client.get_splitted_context(
      start_date_str=star_date_str
    )
 
    history = h_context.get_history(event_type=event_type)
    while h_context.history.get('start_date') < h_context.history.get('end_date'):
    for events in client.poll_history(
      context=h_context,
      event_type=event_type,
      interval=interval
    ): 
      # Process events
      print(f'num events: {len(events)}')
except FncClientError as e:
    client.get_logger().error(e)