Exporting Events from FortiSIEM
The following tools are provided:
phExportESEvent Tool
Description: This tool exports events from Elasticsearch into a CSV file.
Usage: phExportESEvent <ESUrl> <ESPort> <ESDeploymentType> "<ESUser>" "<ESPassword>" <ESIndexName> <ReportingDevIp> <destDir> <splitThreads> <LogLevel>
|
Argument |
Description |
|---|---|
ESUrl
|
The Elasticsearch URL. Example, http://192.0.2.0. |
ESPort
|
The Elasticsearch coordinating node port, e.g. 9200. |
ESType
|
Provide the Elasticsearch type. 1: Native 2: AWS Elasticsearch Service 3: Elasticsearch Cloud |
ESUser
|
Provide the Elasticsearch username. "" means no username. |
ESPassword
|
Provide the Elasticsearch password. "" means no password. |
ESIndexName
|
The name of the Elasticsearch index to be exported, for example, fortisiem-event-2020.06.17-1. |
ReportDevIp
|
The IP address of the report device to be used to select events to export. "" means select all devices. |
destDir
|
The export directory: output_dir. |
|
|
The number of threads to be used for export, e.g., 10. |
|
|
The debug level for script output printing: |
Notes:
- Can be run from Supervisor or Worker.
- Can be run as admin user.
Examples:
Native Elasticsearch Deployment Example
phExportESEvent https://192.0.2.0 9200 1 "Joe.123--test" "password" fortisiem-event-2021.08.05-1-000001 "192.0.2.4" /archive/ 10 INFO
AWS Elasticsearch Service Deployment Example
phExportESEvent https://search-eesna78-aaaa4ysukru3ui4ayaz2yya3km.us-east-1.es.amazonaws.com 443 2 "key" "secret" fortisiem-event-2021.09.29-1 "" /archive/ 10 INFO
Elasticsearch Cloud Deployment Example
phExportESEvent https://cpaagg33-d11e01.es.us-central1.gcp.cloud.es.io 9243 3 "elastic" "password" fortisiem-event-2021.10.01-1-000001 "" /archive/ 10 INFO
phExportEvent Tool
Description: This tool exports events from EventDB into a CSV file. The CSV file contains the following columns:
- Customer Id (applicable to SP license)
- Reporting Device IP
- Reporting Device Name
- Event Received Time
- Raw Message
Usage: phExportEvent {--dest DESTINATION_DIR} {--starttime START_TIME | --relstarttime RELATIVE_START_TIME} {--endtime END_TIME | --relendtime RELATIVE_END_TIME} [--dev DEVICE_NAME] [--org ORGANIZATION_NAME] [-t TIME_ZONE]
| Argument | Description |
|---|---|
DESTINATION_DIR
|
Destination directory where the exported event files are saved. |
START_TIME
|
Starting time of events to be exported. The format is YYYY-MM-DD HH:MM:SS {+|-}
TZ. If TZ is not given, the local time zone of the machine where the script is running will
be used. Example: 2010-03-10 23:00:00 -8 means Pacific Standard Time,
23:00:00 03/10/2010. 2010-07-29 10:20:00 +5:30 means India Standard
Time 10:20:00 07/29/2010.
|
RELATIVE_
START_TIME
|
This must be used together with
where |
END_TIME
|
Ending time of events to be exported. The format is the same as described for START_TIME. |
RELATIVE_END_
TIME
|
This must be used together with START_TIME. Ending time of events to be exported
is relative forward to the start time, specified using START_TIME. The format is
the same that is used for RELATIVE_START_TIME.
|
DEVICE_NAME
|
Provide the host name or IP address of the device with the events to be exported. Use a
comma-separated list to specify multiple IPs or host names, for example, --dev
10.1.1.1,10.10.10.1,router1,router2. Host name is case insensitive. |
ORGANIZATION_
NAME
|
This is used only for Service Provider deployments. Provide the name of the organization with the
events to be exported. To specify multiple organizations, enter a command for each organization, for example, --org "Public Bank" --org "Private
Bank". The organization name is case insensitive.
|
TIME_ZONE
|
Specifies the time zone used to format the event received time in the exported event
files. The format is {+|-}TZ, for example, -8 means Pacific Standard Time,
+5:30 means India Standard Time. |
Notes:
- Can be run from Supervisor or Worker.
- Can be run as admin user.
TestESSplitter Tool
Description: This tool exports events from ElasticSearch to a directory in FortiSIEM EventDB format.
Usage: TestESSplitter <ESBroker> <ESPort> <ESClusterType> <ESUser> <ESPassword> <IndexName> <destDir> <splitThreads> <logLevel>
| Argument | Description |
|---|---|
ESBroker
|
The IP of ElasticSearch Co-ordinator node. |
|
|
The port used for ElasticSearch. |
|
|
The ElasticSearch Cluster type. Values are "1" for Native, "2" for Amazon OpenSearch Service (previously known as Amazon Elasticsearch Service), and "3" for Elastic Cloud. |
ESUser
|
The ElasticSearch username for authentication. |
ESPassword
|
The ElasticSearch password for authentication. |
IndexName
|
Provide an Index name. A new Index is created per day. Here is an example index name, fortisiem-event-2021.05.14-2000-000001 where“fortisiem-event-2021.05.14” is the day and “2000” is the Organization ID. To find a list of indexes, run this command:curl -XGET '10.10.2.5:9200/_cat/shards?v'replacing 10.10.2.5 with the IP of a Co-ordinator node. |
destDir
|
Destination directory where the exported events are saved in FortiSIEM eventDB format. Note: A trailing slash is mandatory. Example: |
|
|
Number of threads. |
|
|
INFO or DEBUG level log messages. |
Notes:
- Can be run from Supervisor or Worker.
- Can be run as admin user.
- This tool is located in
/opt/phoenix/bin/.
Example:
[root@fsm]# /opt/phoenix/bin/TestESSplitter 10.10.2.5 "" "" fortisiem-event-2021.05.14-2000-000001 /root/output 10 INFO [PH_MODULE_LOG_LEVEL_CHANGE]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=phBaseProcess.cpp,[lineNumber]=675,[oldLogLevel]=2047,[newLogLevel]=424,[phLogDetail]=Module received log level change [PH_MODULE_LOCAL_CONFIG_LOADED]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=phConfigLoader.cpp,[lineNumber]=166,[configName]=global,[phLogDetail]=Module loaded local config successfully [PH_MODULE_LOCAL_CONFIG_LOADED]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=phConfigLoader.cpp,[lineNumber]=166,[configName]=phdatamanager,[phLogDetail]=Module loaded local config successfully [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=phHttpClientPool.cpp,[lineNumber]=46,[phLogDetail]=phHttpClientPool: init hosts/port/auth/header=10.10.2.5/9200/:****/Content-Type: application/json * Trying 10.10.2.5... * TCP_NODELAY set * Connected to 10.10.2.5 (10.10.2.5) port 9200 (#0) > GET / HTTP/1.1 Host: 10.10.2.5:9200 Accept: */* Content-Type: application/json < HTTP/1.1 200 OK < content-type: application/json; charset=UTF-8 < content-length: 530 < * Connection #0 to host 10.10.2.5 left intact [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=1732,[phLogDetail]=Elastic init success: http://10.10.2.5:9200/ * Found bundle for host 10.10.2.5: 0x18f0870 [can pipeline] * Re-using existing connection! (#0) with host 10.10.2.5 * Connected to 10.10.2.5 (10.10.2.5) port 9200 (#0) > GET /_cat/indices/fortisiem-event-2021.05.14-2000-000001?h=pri,rep,docs.count HTTP/1.1 Host: 10.10.2.5:9200 Accept: */* Content-Type: application/json … … … … < [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=1974,[phLogDetail]=Elastic succeed hits total 66 for index fortisiem-event-2021.05.14-2000-000001 slice 1 max 10 * Connection #0 to host 10.10.2.5 left intact [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=1974,[phLogDetail]=Elastic succeed hits total 61 for index fortisiem-event-2021.05.14-2000-000001 slice 8 max 10 [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=2002,[phLogDetail]=Elastic index query completed 0 seconds [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=2002,[phLogDetail]=Elastic index query completed 0 seconds < HTTP/1.1 200 OK < content-type: application/json; charset=UTF-8 < content-length: 47737 < * Connection #0 to host 10.10.2.5 left intact [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=1974,[phLogDetail]=Elastic succeed hits total 53 for index fortisiem-event-2021.05.14-2000-000001 slice 3 max 10 < HTTP/1.1 200 OK < content-type: application/json; charset=UTF-8 < content-length: 47178 < * Connection #0 to host 10.10.2.5 left intact < HTTP/1.1 200 OK < content-type: application/json; charset=UTF-8 < content-length: 41910 < * Connection #0 to host 10.10.2.5 left intact < HTTP/1.1 200 OK < content-type: application/json; charset=UTF-8 < content-length: 53258 < * Connection #0 to host 10.10.2.5 left intact < HTTP/1.1 200 OK < content-type: application/json; charset=UTF-8 < content-length: 60587 < * Connection #0 to host 10.10.2.5 left intact [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=1974,[phLogDetail]=Elastic succeed hits total 59 for index fortisiem-event-2021.05.14-2000-000001 slice 4 max 10 [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=1974,[phLogDetail]=Elastic succeed hits total 53 for index fortisiem-event-2021.05.14-2000-000001 slice 7 max 10 [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=1974,[phLogDetail]=Elastic succeed hits total 68 for index fortisiem-event-2021.05.14-2000-000001 slice 6 max 10 [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=1974,[phLogDetail]=Elastic succeed hits total 46 for index fortisiem-event-2021.05.14-2000-000001 slice 2 max 10 [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=2002,[phLogDetail]=Elastic index query completed 0 seconds [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=2002,[phLogDetail]=Elastic index query completed 0 seconds [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=2002,[phLogDetail]=Elastic index query completed 0 seconds [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=2002,[phLogDetail]=Elastic index query completed 0 seconds [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=2002,[phLogDetail]=Elastic index query completed 0 seconds [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=TestESSplitter.cpp,[lineNumber]=82,[phLogDetail]=Events processed for split: 559 3.15
The result will be eventDB structured directories and files.
[root@fsm]# ls -l /root/output/ total 0 drwx------ 3 root root 22 May 14 15:25 CUSTOMER_2000 [root@fsm]# ls -l /root/output/CUSTOMER_2000/ total 0 drwx------ 3 root root 19 May 14 15:25 internal [root@fsm]# ls -l /root/output/CUSTOMER_2000/internal/ total 0 drwx------ 3 root root 37 May 14 15:25 18761 [root@fsm]# ls -l /root/output/CUSTOMER_2000/internal/18761/ total 4 drwx------ 12 root root 4096 May 14 15:25 450264-450287-168428094 [root@fsm]# ls -l /root/output/CUSTOMER_2000/internal/18761/450264-450287-168428094/ total 0 drwx------ 3 root root 18 May 14 15:25 seg-1-0-48-1620951010-1620971132 drwx------ 3 root root 18 May 14 15:25 seg-1-1-70-1620950470-1620971172 drwx------ 3 root root 18 May 14 15:25 seg-1-2-35-1620950916-1620971172 drwx------ 3 root root 18 May 14 15:25 seg-1-3-66-1620951819-1620969371 drwx------ 3 root root 18 May 14 15:25 seg-1-4-61-1620950830-1620970642 drwx------ 3 root root 18 May 14 15:25 seg-1-5-59-1620950830-1620971132 drwx------ 3 root root 18 May 14 15:25 seg-1-6-53-1620950482-1620970632 drwx------ 3 root root 18 May 14 15:25 seg-1-7-46-1620951278-1620971182 drwx------ 3 root root 18 May 14 15:25 seg-1-8-53-1620950470-1620970452 drwx------ 3 root root 18 May 14 15:25 seg-1-9-68-1620950650-1620971132
phClickHouseCSVExport Tool
Description: This tool exports events from ClickHouse into a CSV file. The file will contain these fields:
- Event Receive Time
- Reporting IP
- Event Type
- Raw Event Log.
Usage: phClickHouseCSVExport --starttime [Start Time] --endtime [End Time] --outfile [Output file] --deviceip [Reporting Device IP Address] --devicename [Reporting Device Name] --orgid [Organization ID (0 - 4294967295)] --orgname [Organization Name] --eventtype [Event Type]
|
Argument |
Description |
|---|---|
--starttime [Start Time]
|
Starting time of events to be exported. It must be in the following format: "YYYY-MM-DD hh:mm:ss". The supported time zone is GMT. Make sure to enclose the Start Time with quotation marks. Example: |
--endtime [End Time]
|
The end time of events to be exported. It must be in the following format: "YYYY-MM-DD hh:mm:ss". The supported time zone is GMT. Make sure to enclose the End Time with quotation marks. Example: |
--outfile [Output file]
|
The output file where the exported events are saved in FortiSIEM, CSV format. |
--deviceip [Reporting Device IP Address]
|
Provide the IP address of the device with the events to be exported. Only one reporting device IP address is supported. For example, |
--devicename [Reporting Device Name]
|
Provide the host name of the device with the events to be exported. For example, --devicename
router1. Host name is case insensitive. |
--orgid [Organization ID]
|
Provide the ID of the organization with the events to be exported. The number can be from 0 to 4294967295. |
--orgname [Organization Name]
|
This is used only for Service Provider deployments. Provide the name of the organization with the
events to be exported. To specify multiple organizations, enter a command for each organization, for example, --org "Public Bank" --orgname "Private
Bank". The organization name is case insensitive.
|
--eventtype [Event Type]
|
Specify the event types to be exported. |
Notes:
- Can be run from Supervisor or Worker.
- Can be run as admin user.