Fetch Events

Given the large amount of information that might be stored in AWS S3 Buckets, it is recommended to retrieve events for one event type at a time and search for a period no longer than a day. This recommendation is enforced by the fetch_events() function. This function allows you to fetch all raw events for the specified event type that were observed since the specified start date.

Keep the following considerations in mind:

• The start date must be less than a day before and it must have the timezone information or UTC will be assumed by default.
• This is a generator function that produces a series of events usable in a for-loop or that can be retrieved one at a time with the next() function.

In this section, we show how to use the FncMetastreamClient to fetch current events. The steps involved are:

1. Imports

The main classes required while calling specific endpoint are shown below.

from fnc.fnc_client import FncClient

from fnc.metastream import FncMetastreamClient

from fnc.errors import FncClientError

2. Get the Client

The FncMetastreamClient is created using the FncClient class’s method and providing the required arguments. For a detailed description, see Getting the client.

3. Fetch events

The FncMetastreamClient.fetch_events method is used to fetch events for specific interval. The method requires five arguments the endpoint to be called and a dictionary with its required arguments.

Property	Type	Required	Description
event_type	string	true	The event type to download. Possible values are observation, suricata.
start_date	datetime	true	The start time to restrict results based on their timestamp. Value must have timezone information or UTC will be assumed.
end_date	datetime	false	The end time to restrict results based on their timestamp. Value must have timezone information or UTC will be assumed. It is set to now by default
limit	int	false	The maximum number of events to fetch. Must be between 1 and 10000.
context	MetastreamContext	false	An object that stores specific session wide data such as metrics and checkpoint.

for events in client.fetch_events(event_type=…, limit=…, start_date=…, context=…):
    # Do something …

4. Handle Errors

Any exception occurring while fetching the events will be raised as a FncClientError exception. The specific problem can be identified by using the FncClientError fields.

Example

The code below will retrieve Observations from the previous two days until all the Observations have been retrieved.

from fnc.metastream import FncMetastreamClient
from fnc.errors import FncClientError
from fnc.fnc_client import FncClient 
from fnc.metastream.s3_client import MetastreamContext

client_name = ''
access_key = ''
secret_key = ''
account_code = ''
bucket_name = ''
log_level = None
client = None

try:
    client: FncMetastreamClient = FncClient.get_metastream_client(
      name= client_name, 
      access_key= access_key, 
      secret_key= secret_key, 
      account_code = account_code, 
      bucket_name= bucket_name
    )
    client.get_logger().set_level(level=log_level)

    evet_type = 'observation'
    start_date = datetime.now(timezone.utc) - timedelta(hours=2)
    end_date = datetime.now(timezone.utc)
    context = MetastreamContext()

    for events in client.fetch_events(
      event_type=event_type, 
      start_date= start_date, 
      end_date= end_date, 
      context= context
    ):
      #process events
      print(f'num events: {len(events)}')

except FncClientError as e:
    client.get_logger().error(e)