Generic Log API Poller (HTTPS_ADVANCED) Integration

Generic Log API Poller (HTTPS Advanced) Integration

This integration provides a low-code/no-code solution to integrating new logging APIs into FortiSIEM. Users can now directly define an API within FortiSIEM, which will then reach out to the target destination at the specified pull interval, and retrieve log events.

There are some substantial pre-requesites, you must obtain authentication credentials to your target API (basic auth, oauth2.0 client credentials, api key, or bearer token, etc), and you must understand the elements of the API call to make, such as hostname, URI stem, headers, parameters, and body in both the authentication phase if additional (e.g. oauth2.0 token authorization), and the API call's own header/parameters, pagination details to retrieve the event data.

You can ensure your examples work first by using the curl command. Each component of those curl commands you can define in a FortiSIEM credential using the walkthrough below.

Overview

The approximate steps are here:

Steps 1-5 are related to setting up the ingest of your data.
Steps 6+ are related to parsing, setting up rules and reports to make use of your ingested data.

Understand the API endpoint you'd like to ingest logs from, ensure you understand the authentication mechanism, API information such as headers, parameters, body if applicable.
Test your understanding with curl to make sure you can retrieve that data manually.
Define a device/application type in FortiSIEM -- Only do this if the device or app you are integrating with is not already defined under ADMIN > Device Support > Devices/Apps.
Create a FortiSIEM credential - Selecting your desired device type, selecting protocol "HTTPS Advanced", and plugging in the API information in the appropriate fields
Under Step 2: Enter IP Range to Credential Associations in Admin > Setup > Credentials, click Add, select the credential you just made.

Note: If you'd like a specific collector to make the API call instead of the Supervisor, there will be a collector drop-down to the right that you have to select first.

Click Test > Test connectivity without ping.

Check the pull events tab at the top - A yellow star will be present next to your new API job. Wait approximately 5 minutes for first real API call to be made. It should change to a green checkbox.
Go to Analytics tab, run query for your data, it will come by default as unknown events until you create a custom parser for the data. If your SIEM is configured to drop unknown events, disable that option temporarily.
Build a custom parser for your audit log data. See information on custom parser training here.
Define Event Types and map them to appropriate event type groups.
Define any necessary rules and reports for the new data.

Take the following general steps.

Define a New Device Type (if one does not exist)

Navigate to ADMIN > Device Support > Devices/Apps.
Click New.
From the Device/Application Type Definition window, take the following steps.
1. From the Category drop-down list, select Application.
2. In the Vendor field, enter the vendor name.
3. In the Model field, enter the model/service name.
4. In the Version field, enter the specific version of the API to be used, otherwise enter "ANY".
5. From the Device/App Group drop-down list, select the category group(s) that the new device type will fall under. For example, Firewall, Server, etc...
6. From the Access Protocol drop-down list, de-select any selected access protocols, then select HTTPS Advanced.
7. Leave the App Package Group field empty.
8. (Optional) In the Description field, enter any additional information about the new device type.
9. Click Save.

Set Up the Access Credentials for the Device Type

Navigate to ADMIN > Setup > Credentials.
Under Step 1: Enter Credentials, click New.

From the Access Method Definition window, take the following steps.

In the Name field, enter the name of the API.
From the Device Type drop-down list select the device type, or leave as Generic.
From the Access Protocol drop-down list, select HTTPS ADVANCED.
In the Pull Interval field, configure the pull interval rate. The default interval is 5 minutes.
Select Import Definition or Export Definition.

Import Definition - If someone else has created a logging API credential definition, they can export to a JSON file and provide it to you. You can upload this file to automatically fill in all details. The only items you need to provide yourself are the credentials/secrets required for the API to function.

Export Definition – If you have defined a working API integration, you can export the configuration to a JSON file. This JSON contains everything except the credential/secret/api key as configured under the Authentication tab. Note that this may still contain items containing user specifics such as username, or user specific URI stems, so you may need to remove sensitive data prior to sharing.
From the Authentication Type drop-down list, select one of the following configurations:
1. Basic Auth
2. Oauth2.0
3. API Key
4. Bearer Token
  
  The Authentication configuration is dependent on the Authentication Type selected.
Click on the Edit icon next to General Parameters to configure the logging API Endpoint's basic information.
1. In the Host Name field, enter the host name of the API endpoint. Examples: https://host.example.com or host.example.com
2. In the URI Stem field, enter the stem of the API endpoint, which is the part of the URI after the host, but before the arguments. Example: If the intended API call is https://host.example.com/v1/events?limit=10&eventType=test, the URI stem is the part that occurs after the hostname, and ends before the ? parameter separator. In this example, the URL stem would be /v1/events.
3. From the HTTP Method drop-down list, select GET or POST. Determine your configuration by consulting your vendor's API documentation.
4. Select the Disable SSL Certificate Check checkbox if your server does not have a valid SSL certificate. If checked, certificate validation is ignored.
  
  Note: Normally, this is never recommended, but in some cases, it may be necessary, hence this option is available.
5. In the JSON Response Log Key field, enter the response key that identifies the JSON key value pair containing an array of logs events from the logging API. The system will look under this JSON key, and the value will be treated as an array of log events.
  
  Most JSON APIs, when returning multiple event objects from one API call, will place them under a JSON key. See the following response objects from the API to identify where this value is important.
  
  Example - Response from API is just an array of log events, no other structures.
  
  Response from server:
```
[ 

{ ..event1 data…}, 

{ ..event2 data…} 

] 
```
  JSON Response Key: <leave blank>
  
  Example - Response from API puts array of events under a key named alerts.
```
{ 

  “metadata1”:”xxxx”, 

  “someotherdata”:”xxxx”, 

  “alerts”: [{…event1…},{…event2…},{…event3…}] 

} 
```
  JSON Response Key: alerts
6. In the Log Header field, enter the header string you want to prepend to each log event. The reasoning for this is because not every JSON log file cleanly identifies that it came from a given source. You can type any string here as a header. We recommend doing vendor + service.
  
  For example, if the log header is: “Vendor_Example.COM Event:” then every individual event will look like the following:
```
“Vendor_Example.COM Event: {..json event data here..}”
```
  The resulting log events ingested via Analytics will look like the following:
```
<date> <api hostname> <resolved hostname IP> <log header you typed in credential>: <json event>
Jul 27 15:43:03 2022 api.example.com 1.1.1.1 Vendor_Example.COM Event: {...json event1...}
Jul 27 15:43:03 2022 api.example.com 1.1.1.1 Vendor_Example.COM Event: {...json event2...}
```
  Note: The colon ':' is inserted after your header, so if your header contains "some header:", your result would be:
```
<date> <api hostname> <resolved hostname IP> some header:: <json event>
```
  So if you see double colons, that is the reason.
  
  The Log Header field makes building FortiSIEM parsers easy, as they only have to match the header which cleanly identifies the type of event it is and where it came from.

Click on the Edit icon next to Authentication Parameters to configure the logging API Endpoint's basic information. The options available in authentication depend on the selection from Authentication Type. Follow the configuration based on the Authentication Type you selected: Basic Auth, Oath 2.0, API Key, or Bearer Token.

If Basic Auth was selected for Authentication Type, take the following steps.

In the User Name field, enter the username for API authentication.
In the Password field, enter the password associated with the username.
In the Confirm Password field, re-enter the passwords associated with the username.
Click OK.
Continue to Log API Configuration.

If Oath 2.0 was selected for Authentication Type, you will have the following options to set up Oauth2.0 Client Credentials authorization flow, as well as the header, parameter, and body options of the request.

From the General Tab

In the Access Token URL field, enter the full URL to the oauth2.0 authorization endpoint.
From the Grant Type drop-down list, select Client Credentials. Currently, this is the only Oauth2.0 authorization flow supported.
In the Client Id field, enter the client Id. Normally, this is obtained from the API. You typically create a set of oauth credentials on the service, and download this.
In the Client Secret field, enter the client secret. Normally, this is obtained from the API. You typically create a set of oauth credentials on the service, and download this.
From the Send Method drop-down list, select Send in Body or Send as Basic Auth Header. Use the option specified by the API. The first option will send a pair of x-www-form-urlencoded key-value pairs in the POST body. The second option will base64 encode the ‘clientId:clientSecret’ into the Authorization header of the post request.
After configuration of the Header Tab, Parameter Tab, and Body Tab, continue to Log API Configuration.

From Header Tab

Note: You can add up to a total of 10 additional header, 10 parameter and 10 body key-value pairs. They can be of type string (pasted text) or of type date, that will be a calculated date value based on the execution time and/or last execution time.

For oauth2.0 POST request to the authorize endpoint, you can specify any additional headers here. Consult your vendor's API documentation on the expected additional headers if any.

From Parameter Tab

Parameters are any additional arguments accepted at the end of the URL.

Example: https://example.com/v1/events?arg1=value&arg2=value

For Key Type string, Key Name is the argument name, and Key Value is the literal/static value

So for Key Type string, Key Name format, and Key Value json, it would equate to : https://example.com/authorize?format=json

From Body Tab

Auth body contains the x-www-form-urlencoded body of key-value pairs that get sent with the oauth2.0 authorize request. The options you configure depend on the API you are using, and may not be needed at all.

For example, Microsoft graph requires the following POST Body Key-Value options

Key Type	Key Name	Key Value
string	grant_type	client_credentials
string	scope	https://graph.microsoft.com/.default

Note: FortiSIEM credential automatically adds the client_id and client_secret to the body.

Here is a CURL example of the Get Token endpoint for Graph:

Note: Replace {tenant} with your tenant!

curl -X POST -H "Content-Type: application/x-www-form-urlencoded" -d 'client_id=xxxx&scope=https://graph.microsoft.com/.default&client_secret=xxxx&grant_type=client_credentials' 'https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token'

Notice the –d switch contains the POST Body data. The key value pairs are:

client_id -> auto-populated.

client_secret -> auto-populated.

grant_type -> client_credentials

scope -> https://graph.microsoft.com/.default

The first two are handled by FortiSIEM. Additional Key Names such as grant_type and scope must be entered.

If API Key was selected for Authentication Type, an API key is supplied either as a header or parameter.

In the API Key Name field, enter the parameter name, or the header name.
In the API Key Value field, enter your API key secret value.
From the Send Method drop-down list, select the option appropriate for your API. Some API request this value be placed in the header of the request, or the query parameters. Check your API documentation for more information.

Examples:

If send method is Header: curl -H "authkey: secret1234" https://example.com/v1/events

If send method is Param: curl https://example.com/v1/events?authKey=secret1234
Continue to Log API Configuration.

If Bearer Token was selected for Authentication Type, an API key is supplied either as a header or parameter.

If this option was selected, your API has already generated a Bearer token that is permanent for use. You’ll notice that oauth2.0 also uses Bearer token, but the token is retrieved at runtime. In some APIs, the token is pre-generated and permanent.

Enter the token value into the Bearer Token field, and it will populate the Authorization header automatically.

Example:

A curl equivalent for your API will look like this:

curl -H "Authorization: Bearer secret1234" https://example.com/v1/events

Continue to Log API Configuration.

Log API Configuration

Click on the Edit icon next to Log API Parameters to configure the Log API. The Log API definition contains 4 sections - Header, Parameter, Body, and Pagination . The first 3 sections follow a set definition of key-value pairs representing either the header fields, URL parameter key-value pairs, or POST body key-value pairs.

Adding a String Key-Value Pair

Click New.
From the Key Type drop-down list, select String.
In the Key Name field, enter the header name, parameter name, or body key name depending on which tab (Log API Header, Log API Parameter, Log API Body) has been selected.
In the Key Value field, enter the string value.

Example: For the logging API, to set the Header “Content-Type” to application/json, you would configure the following:

Key Type: String

Key Name: Content-Type

Key Value: application/json

The curl equivalent would be:
```
curl –H “Content-Type:application/json” https://example.com/v1/events
```

Adding a Date Key-Value Pair

Most, if not all logging APIs will require a set of a date parameters. Some require a from (oldest) date to retrieve, or both a "from" and "to" parameter.

Since dates for a recurring poller can only be determined at run time, we define a facility to let you define these.

From the Log API Header, Log API Parameter, or Log API Body Tab, take the following steps.

Click New.
From the Key Type drop-down list, select Date.
In the Key Name field, enter the header name, parameter name, or body key name for the date parameter. Consult your API documentation to determine the proper parameter.
In the Key Date Format drop-down list, select the date format the API expects the date to be supplied in.

From the Date Use Type drop-down list, select FromOperator, ToOperator, or RangeOperator.

Date Use Type	Description
FromOperator	This is the oldest time interval. The script formats the last run time into this argument in the selected format.
ToOperator	If required by your API, this will be calculated as the latest date. The script sets this to the current run time in the selected format.
RangeOperator	Defines time interval date ranges within a single URL parameter instead of two parameters.

From the Date Operator Type drop-down list, select the operator used by your API.

Some APIs have date ranges that are inclusive vs exclusive. An API that has inclusive date ranges means that the date you specified includes all events up to and including that timestamp. E.g. >= or <=. Exclusive date ranges means the API will return events up to the provided timestamp, but not on that timestamp. In other words “>” or “<”. Consult your API documentation.
In the Key Value Prefix field, enter the Key Value Prefix if needed.

In some rare situations, you may need to prepend a string literal to your date format. In most cases you will leave this blank.
In the Key Value Postfix field, enter the Key Value Postfix field if needed.

In some rare situations, you may need to postfix a string literal to your date format. In most cases you will leave this blank.

If RangeOperator was selected, select the operator from the Left Date Operator Type drop-down list , and enter the Left Key Value Prefix, the Left Key Value Postfix, and Joiner. Also, from the Right Date Operator Type drop-down list, select the operator, and enter the Right Key Value Prefix and Right Key Value Postfix. The Range Operator is a special type of date parameter where the parameter contains both date ranges specifying the start and end times that logs should be returned.

Note: URL encoding not used here for readability / example purposes only.

Example of a range parameter in an API call.
```
curl https://graph.microsoft.com/beta/security/incidents?$filter=lastUpdateDateTime ge 2022-07-20T00:00:00.0Z and lastUpdateDateTime lt 2022-07-21T00:00:00.0Z
```
Notice that the date range is specified in one URL parameter called "$filter".

Some APIs such as Microsoft graph support this complex query logic in a single parameter. Most *other* APIs, have two separate date parameters to identify the start and end time ranges to collect events.

Example:
```
https://example.com/events?from=2022-07-20T00:00:00.0Z&to=2022-07-21T00:00:00.0Z
```
The from parameter is the start range, and the to parameter is the end range.

Example of a Date Parameter

If poll Interval is 5 minutes. Last Poll Time is used for the FromOperator, and ToOperator set to current execution time. (Now – 5 minutes)

curl https://example.com/v1/events?from=2022-01-01T00:00:00Z&to=2022-01-01T00:05:00Z

Example 2 for RangeOperator

curl https://example.com/v1/events?$filter=lastUpdateDateTime+ge+2022-06-06T15:20:50.000Z+and+lastUpdateDateTime+lt+2022-06-06T22:24:03.000Z

The Parameter Key Type would be date.

The Parameter Key Name would be "$filter".

Note: Do not include the quotations.

The Key Date Format would be ISO8601-YYYY-MM-DDThhh:mm:ss.sTZD.

The Date Use Type field would be RangeOperator.

The Left Date Operator Type would be >=.

The Left Key Value Prefix would be "lastUpdateDateTime+ge+"
Note: Do not include the quotations.

The Joiner field would be "+and+"
Note: Do not include the quotations.

The Right Date Operator Type would be <.

The Right Key Value Prefix would be "lastUpdateDateTime+lt+"

Note: Do not include the quotations.

Log API Pagination

This tab defines pagination within your API.

From the Pagination Method drop-down list, select the pagination method.

Currently, only the following pagination methods are supported:

Link Header – Uses Web Linking RFC 8288 next header to reference next page of events
Offset and Limit – uses a combination of offset and limit parameters to control pagination
- In the Limit Key Name field, enter the limit key name.
- In the Limit Value field, enter the limit value.
- In the Offset Key Name field, enter the offset key name.
- In the Offset Start Value field, enter the offset start value.
- In the Offset Increment Value field, enter the offset increment value.
- In the Offset Max Value field, enter the offset maximum value.
Custom Next Url – Check either response body or response header for a next URL for paged events.
- In the Key Name field, enter the key name.
- From the Key Location drop-down list, select body or header.
- From the Key Type drop-down list, select URL or URL_STEM.
Custom Next Cursor – Check response body for a cursor parameter to next set of events
- In the Cursor Key Name field, enter the cursor key name.
- From the Cursor Type drop-down list, select parameter.
- In the Next Cursor Name field, enter the next cursor name.
None – No pagination, entire request is returned in one API call. Repeat data possible

Set Up the IP/Host with Created Access Credentials

In Step 2: Enter IP Range to Credential Associations, if you have more than one FortiSIEM collector, select the collector that will do the polling from the drop-down list. Note: A drop-down list will not appear if you only have one collector.

Click New.
1. Select the credential name you created (during Set Up the Access Credentials for the Device Type) from the Credentials drop-down list. The IP/Host Name field should auto populate the URL.
2. Click Save.
Click the Test drop-down list and select Test Connectivity without Ping to test the connection.
If you encounter an error, refer to Common Errors for additional information that may help you resolve the issue.
Wait for approximately 5 minutes.
Navigate to ANALYTICS, and confirm that events appear.

Common Errors

After clicking on Test connectivity w/o Ping, if you receive an error message, review the following for possible causes.

401 - Unauthorized. The credentials you provided were invalid, or most likely not typed in correctly.

403 - General error observe error message.

404 - Not Found, the URI stem / api endpoint is likely not accurate, or the header/parameter values are not correct.

429 - Too Many API calls being made. You may have too many concurrent API endpoints configured to target API.

Contact Fortinet Support if you have issues resolving ay of these errors.