Ingesting JSON Formatted Events Received via HTTP(S) POST

FortiSIEM can receive, parse, and store JSON formatted events received via HTTP(S) POST. Follow these steps to implement this.

1. Configure the FortiSIEM node with the HTTPS credential for receiving the HTTP(S) POST event by taking the following steps.

   1. Identity the FortiSIEM node receiving the events. Most likely, this will be the Collector.

   2. SSH to the Collector and run the command.

      htpasswd -b /etc/httpd/accounts/passwds <user> '<password>'

      Note: If the password contains special characters, it is advisable to encode the password in single quotes.

2. Make sure the events are being pushed to the FortiSIEM node using the credentials in Step 1 via this REST API:
   

   https://<FSMNodeName>/rawupload?vendor=<vendor>&model=<model>&reptIp=<reptIp>&reptName=<reptHost>

   where FSNNodeName is the resolvable host name or FQDN in Step 1. The parameters Reporting Vendor (vendor), Reporting Model (model), Reporting Device (reptHost), and Reporting IP (reptIP) are needed to create a CMDB entry and populate events.

   Argument	Description
   vendor	The vendor of the product that the logs originated from.
   model	The model of the product that the logs originated from.
   reptIp	This is the reporting IP, or the source of the log. The value you specify here will populate the CMDB as a reporting device.
   reptName	This is the reporting device name, or the hostname of the device sending the logs.
   separator	(Optional) This is used to split the whole content of one file into multiple events. If it is omitted in the URL, it means one event one file. Its value only allows the following input characters [%, 0-9, a-Z]. If the value contains any other character, the file name will start with "invalid_event_" and the parser will not parse the file. For URL special character, please use its encode string. For example: '\n' ---> '%0A' ',' ---> '%2C'

   Note: If the Model contains whitespace, such as “Model 24”, you must correctly encode spaces and other special characters in the URL parameters.

   HTTP Method: POST

   HTTP Body: log in JSON format. The JSON file can be compressed into tar, tgz, gz, zip format, in addition to text support.

   Sample Curl to Send a JSON File

   This example is sending a SAP Enterprise Threat Detection log.

   curl -kv -u ‘user:password’ -d "@json_event.json" -X POST 'https://<FSMNodeName>/rawupload?vendor=SAP&model=ETD&reptIp=192.0.2.20&reptName=LogForwarder1'

   The above sends the JSON event stored in the file json_event.json to FortiSIEM. FortiSIEM then processes it, the resulting event should look like the following in Log Format, with an added header attached.

   Log Format

   [PH_DEV_MON_CUSTOM_JSON]:[reptVendor]=<vendor>,[reptModel]=<model>,[reptDevName]=<reptName>,[reptDevIpAddr]=<reptIp>,[json]=<JSON>

   Where <JSON> is the actual JSON log body posted to FortiSIEM.

3. Query the events by using the Reporting Device Name or IP in Step 2 and Event Type in step 4e, by taking the following steps.

   1. Go to the ANALYTICS tab.

   2. Run a query for the Reporting IP = ‘#.#.#.#’ for the last 10 minutes.

   3. Observe the raw event, it should be in the format of.

      

4. Create a new parser matching the header format with your provided vendor, model by taking the following steps.

   1. Login to the Supervisor.

   2. Navigate to ADMIN > Device Support > Parsers.

   3. From the Search... field, enter PHCustomJsonParser.

   4. Select it, and click Clone.

   5. Make any required changes so that additional event attributes are parsed.

   6. Name your parser appropriately, for example, <Vendor>_<Model>_Custom_Parser.

   7. Use a similar event format recognizer: <eventFormatRecognizer><![CDATA[\[PH_DEV_MON_CUSTOM_JSON]\:\[reptVendor\]=<vendor>,\[reptModel\]=<model>,]]></eventFormatRecognizer>

      Note: See the parser training documentation on making a custom parser for your event.
      

   8. Validate, Test, and Save the parser.

   9. Click Apply All to deploy the parser changes.

5. If your JSON log events are batched into a single HTTPS POST operation (JSON contains many distinct events), there is a methodology to split the events using the splitJsonEvent() function, and discard the original monolithic event. Examine the SAPEnterpriseThreatDetectionParser on how to use the function splitJsonEvent().