CSV Data Management

CSV Data Management v1.2.0

Home FortiSOAR Connectors CSV Data Management

1.2.0

CSV Data Management v1.2.0

About the connector

This document provides information about the CSV Data Management connector using which you can perform different operations on CSV files like reading the CSV file, performing deduplication, merging two CSV files, joining two CSV files, concatenating two CSV files, and returning well-formatted datasets.

This connector uses functionality from the 'Polars', 'Pandas', and 'Numpy' python modules for merging, joining, and concatenating CSV files. The CSV file data is converted into a data frame and then processed. To know more about these operations see the following:
https://pola-rs.github.io/polars-book/user-guide/
https://pandas.pydata.org/docs/user_guide/merging.html
https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.join.html

Version information

Connector Version: 1.2.0

FortiSOAR™ Version Tested on: 7.4.0-3024

Authored By: Fortinet

Certified: Yes

Release Notes for version 1.2.0

The following enhancements have been made to the CSV Data Management connector in version 1.2.0:

Added the ability to directly add JSON data as input to the "Convert JSON to CSV" action.
Migrated all actions, except the "Convert JSON to CSV" action, to use the 'Polars' library. Previously, actions were using the 'Pandas' library.
Reduced the connector's CPU usage and sped up the execution of its actions to improve the performance of the CSV Data Management connector.

Installing the connector

Use the Content Hub to install the connector. For the detailed procedure to install a connector, click here.

You can also use the following yum command as a root user to install connectors from an SSH session:
yum install cyops-connector-csv-data-management

Minimum Permissions Required

Not applicable

Configuring the connector

You do not require to configure this connector since it performs different operations on CSV files like reading files, performing deduplication, merging two CSV files, etc. For the description of the Content Hub and other details, click here.

Actions supported by the connector

The following automated operations can be included in playbooks and you can also use the annotations to access operations:

Function	Description	Annotation and Category
Extract Data from Single CSV	Extracts data from a CSV file based on the specified column names and other input parameters. Optionally, you can also select an option to deduplicate the resultant recordset based on the specified column(s).	read_csv_file Investigation
Merge and Extract Data from two CSV	Extracts data from CSV files based on the specified column names and other input parameters, by merging two CSV files. Optionally, you can also select an option to deduplicate the resultant recordset based on the specified column(s).	read_and_merge_csv_file Investigation
Concat and Extract Data from two CSV	Extracts data from CSV files based on the specified column names and other input parameters, by concatenating two CSV files. Optionally, you can also select an option to deduplicate the resultant recordset based on the specified column(s).	read_and_concat_csv_file Investigation
Join and Extract Data from two CSV	Extracts data from CSV files based on the specified column names and other input parameters, by joining two CSV files. Optionally, you can also select an option to deduplicate the resultant recordset based on the specified column(s).	read_and_join_csv_file Investigation
Convert JSON to CSV	Converts the output of a playbook step, which is in the JSON format, and converts it to the CSV file. NOTE: The JSON provided in the specified playbook step output must be simple JSON for this operation to work; if the input is a complex JSON this operation will fail to create the CSV file.	json_to_csv Investigation

Notes on the workings of the concat, join, or merge actions

All database operations such as filtering of datasets, deduplication of values, etc. occur on the dataset that is the result of the concat, join, or merge actions. The following points should be noted:

The 'Join' action appends '_FirstFile' and '_SecondFile' in cases where the same columns are not present in files that are being joined. In the case of the 'Join' action, the files are joined horizontally; therefore, columns from the first file and the second file are present but are suffixed with '_FirstFile' and '_SecondFile' to differentiate column names.
Therefore, in order to perform database operations, say filter datasets, you have to specify the correct column name that is part of the dataset that is the result of the concat, join, or merge actions.
The 'Concat' action concats records (vertically) from the second file to the records of the first file, therefore, both files should have the same columns name.
The 'Merge' action merges records of the first file and the second file based on the column that is common to both files. If any column does not have a value, the merged file displays 'NA' as the value of this column.

operation: Extract Data from Single CSV

Input parameters

Parameter	Description
Type	Select the method using which you want to submit the CSV file whose data you want to extract. You can choose between Attachment ID and File IRI.
Reference ID	Specify the reference ID of the file based on the 'Type' you have selected. If you have selected the type as the 'Attachment ID' option, then in this field specify the Attachment ID of the CSV file whose data you want to extract. If you have selected the type as the 'File IRI' option, then in this field specify the IRI of the CSV file whose data you want to extract.
Column Names	Specify the comma-separated column names that you want to extract from the specified CSV file.
Deduplicate Values on	Specify the column name using which you want to deduplicate data from the specified CSV file.
Number of rows to skip	Specify the number of rows you want to skip from the top of the first specified CSV file. Note: The first row is skipped even if it has column names.
Filter Dataset	(Optional) Select the method using which you want to filter data. You can choose to filter column data using regex as a filter (On Values Matching a Regex) or using the 'is in' filter (On Specified values). If you choose the 'On Values Matching a Regex' option, then you must specify the following parameters: Filter on Column: Specify the column name that you want to use to filter data. Regex Filter: Specify the regex expression using which you want to filter data. If you choose 'On Specified values', then you must specify the following parameters: Filter on Column: Specify the column name that you want to use to filter data. Filter String: Specify the string value using which you want to filter data.
Convert recordset into batch	Select this option to return rows as recordsets in a list of 20 batches. If this option is left cleared, then the complete result is returned in a single recordset.
Save as attachment	Select this option to save the resultant recordSet as an attachment in the CSV format.

Output

No output schema is available at this time.

operation: Merge and Extract Data from two CSV

Input parameters

Parameter	Description
Type	Select the method using which you want to submit the first CSV file that you want to merge and extract their data. You can choose between Attachment ID and File IRI.
First File Reference ID	Specify the reference ID of the first CSV file based on the 'Type' you have selected. If you have selected the type as the 'Attachment ID' option, then in this field specify the first Attachment ID of the first CSV file that you want to merge and extract their data. If you have selected the type as the 'File IRI' option, then in this field specify the IRI of the first CSV file that you want to merge and extract their data.
First File Column Names	Specify the comma-separated column names that you want to extract from the specified first CSV file.
Number of rows to skip from First File	Specify the number of rows you want to skip from the top of the first specified CSV file. Note: The first row is skipped even if it has column names.
Type	Select the method using which you want to submit the second CSV file that you want to merge and extract their data. You can choose between Attachment ID and File IRI.
Second File Reference ID	Specify the reference ID of the second CSV file based on the 'Type' you have selected. If you have selected the type as the 'Attachment ID' option, then in this field specify the second Attachment ID of the second CSV file that you want to merge and extract their data. If you have selected the type as the 'File IRI' option, then in this field specify the IRI of the second CSV file that you want to merge and extract their data.
Second File Column Names	Specify the comma-separated column names that you want to extract from the specified second CSV file.
Number of rows to skip from Second File	Specify the number of rows you want to skip from the top of the second specified CSV file. Note: The first row is skipped even if it has column names.
Merge on Column	Specify the column name that is common in the two specified CSV files using which you want to merge the data from both files.
Deduplicate Values on	Specify the column name using which you want to deduplicate data from the specified CSV files.
Filter Dataset	(Optional) Select the method using which you want to filter data. You can choose to filter column data using regex as a filter (On Values Matching a Regex) or using the 'is in' filter (On Specified values). If you choose the 'On Values Matching a Regex' option, then you must specify the following parameters: Filter on Column: Specify the column name that you want to use to filter data. Regex Filter: Specify the regex expression using which you want to filter data. If you choose 'On Specified values', then you must specify the following parameters: Filter on Column: Specify the column name that you want to use to filter data. Filter String: Specify the string value using which you want to filter data.
Convert recordset into batch	Select this option to return rows as recordsets in a list of 20 batches. If this option is left cleared, then the complete result is returned in a single recordset.
Save as attachment	Select this option to save the resultant recordSet as an attachment in the CSV format.

Output

No output schema is available at this time.

operation: Concat and Extract Data from two CSV

Input parameters

Parameter	Description
Type	Select the method using which you want to submit the first CSV file that you want to concat and extract their data. You can choose between Attachment ID and File IRI.
First File Reference ID	Specify the reference ID of the first CSV file based on the 'Type' you have selected. If you have selected the type as the 'Attachment ID' option, then in this field specify the first Attachment ID of the first CSV file that you want to concat and extract their data. If you have selected the type as the 'File IRI' option, then in this field specify the IRI of the first CSV file that you want to concat and extract their data.
First File Column Names	Specify the comma-separated column names that you want to extract from the specified first CSV file.
Number of rows to skip from First File	Specify the number of rows you want to skip from the top of the first specified CSV file. Note: The first row is skipped even if it has column names.
Type	Select the method using which you want to submit the second CSV file that you want to concat and extract their data. You can choose between Attachment ID and File IRI.
Second File Reference ID	Specify the reference ID of the second CSV file based on the 'Type' you have selected. If you have selected the type as the 'Attachment ID' option, then in this field specify the second Attachment ID of the second CSV file that you want to concat and extract their data. If you have selected the type as the 'File IRI' option, then in this field specify the IRI of the second CSV file that you want to concat and extract their data.
Second File Column Names	Specify the comma-separated column names that you want to extract from the specified second CSV file.
Number of rows to skip from Second File	Specify the number of rows you want to skip from the top of the second specified CSV file. Note: The first row is skipped even if it has column names.
Deduplicate Values on	Specify the column name using which you want to deduplicate data from the specified CSV files.
Filter Dataset	(Optional) Select the method using which you want to filter data. You can choose to filter column data using regex as a filter (On Values Matching a Regex) or using the 'is in' filter (On Specified values). If you choose the 'On Values Matching a Regex' option, then you must specify the following parameters: Filter on Column: Specify the column name that you want to use to filter data. Regex Filter: Specify the regex expression using which you want to filter data. If you choose 'On Specified values', then you must specify the following parameters: Filter on Column: Specify the column name that you want to use to filter data. Filter String: Specify the string value using which you want to filter data.
Convert recordset into batch	Select this option to return rows as recordsets in a list of 20 batches. If this option is left cleared, then the complete result is returned in a single recordset.
Save as attachment	Select this option to save the resultant recordSet as an attachment in the CSV format.

Output

No output schema is available at this time.

operation: Join and Extract Data from two CSV

Input parameters

Parameter	Description
Type	Select the method using which you want to submit the first CSV file that you want to join and extract their data. You can choose between Attachment ID and File IRI.
First File Reference ID	Specify the reference ID of the first CSV file based on the 'Type' you have selected. If you have selected the type as the 'Attachment ID' option, then in this field specify the first Attachment ID of the first CSV file that you want to join and extract their data. If you have selected the type as the 'File IRI' option, then in this field specify the IRI of the first CSV file that you want to join and extract their data.
First File Column Names	Specify the comma-separated column names that you want to extract from the specified first CSV file.
Number of rows to skip from First File	Specify the number of rows you want to skip from the top of the first specified CSV file. Note: The first row is skipped even if it has column names.
Type	Select the method using which you want to submit the second CSV file that you want to join and extract their data. You can choose between Attachment ID and File IRI.
Second File Reference ID	Specify the reference ID of the second CSV file based on the 'Type' you have selected. If you have selected the type as the 'Attachment ID' option, then in this field specify the second Attachment ID of the second CSV file that you want to join and extract their data. If you have selected the type as the 'File IRI' option, then in this field specify the IRI of the second CSV file that you want to join and extract their data.
Second File Column Names	Specify the comma-separated column names that you want to extract from the specified second CSV file.
Number of rows to skip from Second File	Specify the number of rows you want to skip from the top of the second specified CSV file. Note: The first row is skipped even if it has column names.
Merge on Column	Specify the column name that is common in the two specified CSV files using which you want to merge the data from both files.
Deduplicate Values on	Specify the column name using which you want to deduplicate data from the specified CSV files.
Filter Dataset	(Optional) Select the method using which you want to filter data. You can choose to filter column data using regex as a filter (On Values Matching a Regex) or using the 'is in' filter (On Specified values). If you choose the 'On Values Matching a Regex' option, then you must specify the following parameters: Filter on Column: Specify the column name that you want to use to filter data. Regex Filter: Specify the regex expression using which you want to filter data. If you choose 'On Specified values', then you must specify the following parameters: Filter on Column: Specify the column name that you want to use to filter data. Filter String: Specify the string value using which you want to filter data.
Convert recordset into batch	Select this option to return rows as recordsets in a list of 20 batches. If this option is left cleared, then the complete result is returned in a single recordset.
Save as attachment	Select this option to save the resultant recordSet as an attachment in the CSV format.

Output

No output schema is available at this time.

operation: Convert JSON to CSV

Input parameters

NOTE: The JSON provided in the specified playbook step output must be simple JSON for this operation to work; if the input is a complex JSON this operation will fail to create the CSV file.

Parameter	Description
Input Type	Select the method using which you want to submit the playbook whose step output you want to convert from the JSON format to the CSV format. You can choose between JSON, Attachment ID, or File IRI. Based on the Input Type you have selected, specify the following parameters: If you have selected the input type as the 'JSON', then specify the following: JSON: Enter the JSON Data that you want to convert to the CSV format. Provide CSV Filename: Specify the name to be given to the CSV file generated by this operation. The generated CSV file will be compressed. Record Path: (Optional) Specify the entry point of the JSON Field and the type of the specified field must be either a 'String' or a 'List'. If you do not specify an entry point, then, by default, all the JSON Data is converted to the CSV format. Additional Fields: (Optional) Specify any additional fields you want to add to the CSV. If there is more than one additional field, then fields must be specified in the comma-separated format without spaces. By default, this field is set to "None", which means that additional fields will not be merged. If you have selected the input type as the 'Attachment ID', then specify the following: Reference ID: Specify the Attachment ID of the playbook whose step output you want to convert from the JSON format to the CSV format. Provide CSV Filename: Specify the name to be given to the CSV file generated by this operation. The generated CSV file will be compressed. If you have selected the input type as the 'File IRI', then specify the following Reference ID: Specify the IRI of the playbook whose step output you want to convert from the JSON format to the CSV format. Provide CSV Filename: Specify the name to be given to the CSV file generated by this operation. The generated CSV file will be compressed.

Parameter

Description

Input Type

Select the method using which you want to submit the playbook whose step output you want to convert from the JSON format to the CSV format. You can choose between JSON, Attachment ID, or File IRI.
Based on the Input Type you have selected, specify the following parameters:

If you have selected the input type as the 'JSON', then specify the following:
- JSON: Enter the JSON Data that you want to convert to the CSV format.
- Provide CSV Filename: Specify the name to be given to the CSV file generated by this operation. The generated CSV file will be compressed.
- Record Path: (Optional) Specify the entry point of the JSON Field and the type of the specified field must be either a 'String' or a 'List'. If you do not specify an entry point, then, by default, all the JSON Data is converted to the CSV format.
- Additional Fields: (Optional) Specify any additional fields you want to add to the CSV. If there is more than one additional field, then fields must be specified in the comma-separated format without spaces. By default, this field is set to "None", which means that additional fields will not be merged.
If you have selected the input type as the 'Attachment ID', then specify the following:
- Reference ID: Specify the Attachment ID of the playbook whose step output you want to convert from the JSON format to the CSV format.
- Provide CSV Filename: Specify the name to be given to the CSV file generated by this operation. The generated CSV file will be compressed.
If you have selected the input type as the 'File IRI', then specify the following
- Reference ID: Specify the IRI of the playbook whose step output you want to convert from the JSON format to the CSV format.
- Provide CSV Filename: Specify the name to be given to the CSV file generated by this operation. The generated CSV file will be compressed.

Output

No output schema is available at this time.

Included playbooks

The Sample - CSV Data Management - 1.2.0 playbook collection comes bundled with the CSV Data Management connector. These playbooks contain steps using which you can perform all supported actions. You can see bundled playbooks in the Automation > Playbooks section in FortiSOAR™ after importing the CSV Data Management connector.

Convert JSON to CSV
Read CSV and Filter dataset using Regex
Read CSV and Filter dataset for specific value
Read two CSV and join into single data set
Read two CSV and concat in single data set
Read Single Column CSV
Read two CSV and merge in single data set
Read CSV with no headers
Read CSV and create dataset

Note: If you are planning to use any of the sample playbooks in your environment, ensure that you clone those playbooks and move them to a different collection since the sample playbook collection gets deleted during the connector upgrade and delete.

About the connector

Version information

Connector Version: 1.2.0

FortiSOAR™ Version Tested on: 7.4.0-3024

Authored By: Fortinet

Certified: Yes

Release Notes for version 1.2.0

The following enhancements have been made to the CSV Data Management connector in version 1.2.0:

Added the ability to directly add JSON data as input to the "Convert JSON to CSV" action.
Migrated all actions, except the "Convert JSON to CSV" action, to use the 'Polars' library. Previously, actions were using the 'Pandas' library.
Reduced the connector's CPU usage and sped up the execution of its actions to improve the performance of the CSV Data Management connector.

Installing the connector

Use the Content Hub to install the connector. For the detailed procedure to install a connector, click here.

You can also use the following yum command as a root user to install connectors from an SSH session:
yum install cyops-connector-csv-data-management

Minimum Permissions Required

Not applicable

Configuring the connector

Actions supported by the connector

The following automated operations can be included in playbooks and you can also use the annotations to access operations:

Function	Description	Annotation and Category
Extract Data from Single CSV	Extracts data from a CSV file based on the specified column names and other input parameters. Optionally, you can also select an option to deduplicate the resultant recordset based on the specified column(s).	read_csv_file Investigation
Merge and Extract Data from two CSV	Extracts data from CSV files based on the specified column names and other input parameters, by merging two CSV files. Optionally, you can also select an option to deduplicate the resultant recordset based on the specified column(s).	read_and_merge_csv_file Investigation
Concat and Extract Data from two CSV	Extracts data from CSV files based on the specified column names and other input parameters, by concatenating two CSV files. Optionally, you can also select an option to deduplicate the resultant recordset based on the specified column(s).	read_and_concat_csv_file Investigation
Join and Extract Data from two CSV	Extracts data from CSV files based on the specified column names and other input parameters, by joining two CSV files. Optionally, you can also select an option to deduplicate the resultant recordset based on the specified column(s).	read_and_join_csv_file Investigation
Convert JSON to CSV	Converts the output of a playbook step, which is in the JSON format, and converts it to the CSV file. NOTE: The JSON provided in the specified playbook step output must be simple JSON for this operation to work; if the input is a complex JSON this operation will fail to create the CSV file.	json_to_csv Investigation

Notes on the workings of the concat, join, or merge actions

The 'Join' action appends '_FirstFile' and '_SecondFile' in cases where the same columns are not present in files that are being joined. In the case of the 'Join' action, the files are joined horizontally; therefore, columns from the first file and the second file are present but are suffixed with '_FirstFile' and '_SecondFile' to differentiate column names.
Therefore, in order to perform database operations, say filter datasets, you have to specify the correct column name that is part of the dataset that is the result of the concat, join, or merge actions.
The 'Concat' action concats records (vertically) from the second file to the records of the first file, therefore, both files should have the same columns name.
The 'Merge' action merges records of the first file and the second file based on the column that is common to both files. If any column does not have a value, the merged file displays 'NA' as the value of this column.

operation: Extract Data from Single CSV

Input parameters

Parameter	Description
Type	Select the method using which you want to submit the CSV file whose data you want to extract. You can choose between Attachment ID and File IRI.
Reference ID	Specify the reference ID of the file based on the 'Type' you have selected. If you have selected the type as the 'Attachment ID' option, then in this field specify the Attachment ID of the CSV file whose data you want to extract. If you have selected the type as the 'File IRI' option, then in this field specify the IRI of the CSV file whose data you want to extract.
Column Names	Specify the comma-separated column names that you want to extract from the specified CSV file.
Deduplicate Values on	Specify the column name using which you want to deduplicate data from the specified CSV file.
Number of rows to skip	Specify the number of rows you want to skip from the top of the first specified CSV file. Note: The first row is skipped even if it has column names.
Filter Dataset	(Optional) Select the method using which you want to filter data. You can choose to filter column data using regex as a filter (On Values Matching a Regex) or using the 'is in' filter (On Specified values). If you choose the 'On Values Matching a Regex' option, then you must specify the following parameters: Filter on Column: Specify the column name that you want to use to filter data. Regex Filter: Specify the regex expression using which you want to filter data. If you choose 'On Specified values', then you must specify the following parameters: Filter on Column: Specify the column name that you want to use to filter data. Filter String: Specify the string value using which you want to filter data.
Convert recordset into batch	Select this option to return rows as recordsets in a list of 20 batches. If this option is left cleared, then the complete result is returned in a single recordset.
Save as attachment	Select this option to save the resultant recordSet as an attachment in the CSV format.

Output

No output schema is available at this time.

operation: Merge and Extract Data from two CSV

Input parameters

Parameter	Description
Type	Select the method using which you want to submit the first CSV file that you want to merge and extract their data. You can choose between Attachment ID and File IRI.
First File Reference ID	Specify the reference ID of the first CSV file based on the 'Type' you have selected. If you have selected the type as the 'Attachment ID' option, then in this field specify the first Attachment ID of the first CSV file that you want to merge and extract their data. If you have selected the type as the 'File IRI' option, then in this field specify the IRI of the first CSV file that you want to merge and extract their data.
First File Column Names	Specify the comma-separated column names that you want to extract from the specified first CSV file.
Number of rows to skip from First File	Specify the number of rows you want to skip from the top of the first specified CSV file. Note: The first row is skipped even if it has column names.
Type	Select the method using which you want to submit the second CSV file that you want to merge and extract their data. You can choose between Attachment ID and File IRI.
Second File Reference ID	Specify the reference ID of the second CSV file based on the 'Type' you have selected. If you have selected the type as the 'Attachment ID' option, then in this field specify the second Attachment ID of the second CSV file that you want to merge and extract their data. If you have selected the type as the 'File IRI' option, then in this field specify the IRI of the second CSV file that you want to merge and extract their data.
Second File Column Names	Specify the comma-separated column names that you want to extract from the specified second CSV file.
Number of rows to skip from Second File	Specify the number of rows you want to skip from the top of the second specified CSV file. Note: The first row is skipped even if it has column names.
Merge on Column	Specify the column name that is common in the two specified CSV files using which you want to merge the data from both files.
Deduplicate Values on	Specify the column name using which you want to deduplicate data from the specified CSV files.
Filter Dataset	(Optional) Select the method using which you want to filter data. You can choose to filter column data using regex as a filter (On Values Matching a Regex) or using the 'is in' filter (On Specified values). If you choose the 'On Values Matching a Regex' option, then you must specify the following parameters: Filter on Column: Specify the column name that you want to use to filter data. Regex Filter: Specify the regex expression using which you want to filter data. If you choose 'On Specified values', then you must specify the following parameters: Filter on Column: Specify the column name that you want to use to filter data. Filter String: Specify the string value using which you want to filter data.
Convert recordset into batch	Select this option to return rows as recordsets in a list of 20 batches. If this option is left cleared, then the complete result is returned in a single recordset.
Save as attachment	Select this option to save the resultant recordSet as an attachment in the CSV format.

Output

No output schema is available at this time.

operation: Concat and Extract Data from two CSV

Input parameters

Parameter	Description
Type	Select the method using which you want to submit the first CSV file that you want to concat and extract their data. You can choose between Attachment ID and File IRI.
First File Reference ID	Specify the reference ID of the first CSV file based on the 'Type' you have selected. If you have selected the type as the 'Attachment ID' option, then in this field specify the first Attachment ID of the first CSV file that you want to concat and extract their data. If you have selected the type as the 'File IRI' option, then in this field specify the IRI of the first CSV file that you want to concat and extract their data.
First File Column Names	Specify the comma-separated column names that you want to extract from the specified first CSV file.
Number of rows to skip from First File	Specify the number of rows you want to skip from the top of the first specified CSV file. Note: The first row is skipped even if it has column names.
Type	Select the method using which you want to submit the second CSV file that you want to concat and extract their data. You can choose between Attachment ID and File IRI.
Second File Reference ID	Specify the reference ID of the second CSV file based on the 'Type' you have selected. If you have selected the type as the 'Attachment ID' option, then in this field specify the second Attachment ID of the second CSV file that you want to concat and extract their data. If you have selected the type as the 'File IRI' option, then in this field specify the IRI of the second CSV file that you want to concat and extract their data.
Second File Column Names	Specify the comma-separated column names that you want to extract from the specified second CSV file.
Number of rows to skip from Second File	Specify the number of rows you want to skip from the top of the second specified CSV file. Note: The first row is skipped even if it has column names.
Deduplicate Values on	Specify the column name using which you want to deduplicate data from the specified CSV files.
Filter Dataset	(Optional) Select the method using which you want to filter data. You can choose to filter column data using regex as a filter (On Values Matching a Regex) or using the 'is in' filter (On Specified values). If you choose the 'On Values Matching a Regex' option, then you must specify the following parameters: Filter on Column: Specify the column name that you want to use to filter data. Regex Filter: Specify the regex expression using which you want to filter data. If you choose 'On Specified values', then you must specify the following parameters: Filter on Column: Specify the column name that you want to use to filter data. Filter String: Specify the string value using which you want to filter data.
Convert recordset into batch	Select this option to return rows as recordsets in a list of 20 batches. If this option is left cleared, then the complete result is returned in a single recordset.
Save as attachment	Select this option to save the resultant recordSet as an attachment in the CSV format.

Output

No output schema is available at this time.

operation: Join and Extract Data from two CSV

Input parameters

Parameter	Description
Type	Select the method using which you want to submit the first CSV file that you want to join and extract their data. You can choose between Attachment ID and File IRI.
First File Reference ID	Specify the reference ID of the first CSV file based on the 'Type' you have selected. If you have selected the type as the 'Attachment ID' option, then in this field specify the first Attachment ID of the first CSV file that you want to join and extract their data. If you have selected the type as the 'File IRI' option, then in this field specify the IRI of the first CSV file that you want to join and extract their data.
First File Column Names	Specify the comma-separated column names that you want to extract from the specified first CSV file.
Number of rows to skip from First File	Specify the number of rows you want to skip from the top of the first specified CSV file. Note: The first row is skipped even if it has column names.
Type	Select the method using which you want to submit the second CSV file that you want to join and extract their data. You can choose between Attachment ID and File IRI.
Second File Reference ID	Specify the reference ID of the second CSV file based on the 'Type' you have selected. If you have selected the type as the 'Attachment ID' option, then in this field specify the second Attachment ID of the second CSV file that you want to join and extract their data. If you have selected the type as the 'File IRI' option, then in this field specify the IRI of the second CSV file that you want to join and extract their data.
Second File Column Names	Specify the comma-separated column names that you want to extract from the specified second CSV file.
Number of rows to skip from Second File	Specify the number of rows you want to skip from the top of the second specified CSV file. Note: The first row is skipped even if it has column names.
Merge on Column	Specify the column name that is common in the two specified CSV files using which you want to merge the data from both files.
Deduplicate Values on	Specify the column name using which you want to deduplicate data from the specified CSV files.
Filter Dataset	(Optional) Select the method using which you want to filter data. You can choose to filter column data using regex as a filter (On Values Matching a Regex) or using the 'is in' filter (On Specified values). If you choose the 'On Values Matching a Regex' option, then you must specify the following parameters: Filter on Column: Specify the column name that you want to use to filter data. Regex Filter: Specify the regex expression using which you want to filter data. If you choose 'On Specified values', then you must specify the following parameters: Filter on Column: Specify the column name that you want to use to filter data. Filter String: Specify the string value using which you want to filter data.
Convert recordset into batch	Select this option to return rows as recordsets in a list of 20 batches. If this option is left cleared, then the complete result is returned in a single recordset.
Save as attachment	Select this option to save the resultant recordSet as an attachment in the CSV format.

Output

No output schema is available at this time.

operation: Convert JSON to CSV

Input parameters

NOTE: The JSON provided in the specified playbook step output must be simple JSON for this operation to work; if the input is a complex JSON this operation will fail to create the CSV file.

Parameter	Description
Input Type	Select the method using which you want to submit the playbook whose step output you want to convert from the JSON format to the CSV format. You can choose between JSON, Attachment ID, or File IRI. Based on the Input Type you have selected, specify the following parameters: If you have selected the input type as the 'JSON', then specify the following: JSON: Enter the JSON Data that you want to convert to the CSV format. Provide CSV Filename: Specify the name to be given to the CSV file generated by this operation. The generated CSV file will be compressed. Record Path: (Optional) Specify the entry point of the JSON Field and the type of the specified field must be either a 'String' or a 'List'. If you do not specify an entry point, then, by default, all the JSON Data is converted to the CSV format. Additional Fields: (Optional) Specify any additional fields you want to add to the CSV. If there is more than one additional field, then fields must be specified in the comma-separated format without spaces. By default, this field is set to "None", which means that additional fields will not be merged. If you have selected the input type as the 'Attachment ID', then specify the following: Reference ID: Specify the Attachment ID of the playbook whose step output you want to convert from the JSON format to the CSV format. Provide CSV Filename: Specify the name to be given to the CSV file generated by this operation. The generated CSV file will be compressed. If you have selected the input type as the 'File IRI', then specify the following Reference ID: Specify the IRI of the playbook whose step output you want to convert from the JSON format to the CSV format. Provide CSV Filename: Specify the name to be given to the CSV file generated by this operation. The generated CSV file will be compressed.

Parameter

Description

Input Type

If you have selected the input type as the 'JSON', then specify the following:
- JSON: Enter the JSON Data that you want to convert to the CSV format.
- Provide CSV Filename: Specify the name to be given to the CSV file generated by this operation. The generated CSV file will be compressed.
- Record Path: (Optional) Specify the entry point of the JSON Field and the type of the specified field must be either a 'String' or a 'List'. If you do not specify an entry point, then, by default, all the JSON Data is converted to the CSV format.
- Additional Fields: (Optional) Specify any additional fields you want to add to the CSV. If there is more than one additional field, then fields must be specified in the comma-separated format without spaces. By default, this field is set to "None", which means that additional fields will not be merged.
If you have selected the input type as the 'Attachment ID', then specify the following:
- Reference ID: Specify the Attachment ID of the playbook whose step output you want to convert from the JSON format to the CSV format.
- Provide CSV Filename: Specify the name to be given to the CSV file generated by this operation. The generated CSV file will be compressed.
If you have selected the input type as the 'File IRI', then specify the following
- Reference ID: Specify the IRI of the playbook whose step output you want to convert from the JSON format to the CSV format.
- Provide CSV Filename: Specify the name to be given to the CSV file generated by this operation. The generated CSV file will be compressed.

Output

No output schema is available at this time.

Included playbooks

Convert JSON to CSV
Read CSV and Filter dataset using Regex
Read CSV and Filter dataset for specific value
Read two CSV and join into single data set
Read two CSV and concat in single data set
Read Single Column CSV
Read two CSV and merge in single data set
Read CSV with no headers
Read CSV and create dataset

Secure Networking

Hybrid Mesh Firewall

NOC Management

LAN

WAN

Unified SASE

Single Vendor SASE

Cloud Network Security

Secure Endpoint Connectivity

Web Application / API Protection

Security Operations

Security Operations Automation

Identity

Early Detection & Prevention

Secure Networking

Hybrid Mesh Firewall

NOC Management

LAN

WAN

Communication & Surveillance

Unified SASE

Single Vendor SASE

Secure Endpoint Connectivity

Cloud Network Security

Cloud-Native Security

Web Application / API Protection

Security Operations

Security Operations Automation

Endpoint

Data Protection

Identity

Email

Early Detection & Prevention

Expert Services

Edge Firewall

Orchestration & management

SD Branch

Application Delivery

Single Vendor SASE

Secure Endpoint Connectivity

Secure Private Access

Thin Edge

Identity

Application Gateway

Enterprise Asset Management

Endpoint Agent

Agentless Security Posture

Identity

Wireless

Switching

Identity

Privilege Acccess Management

Next Generation Firewall

Orchestration & management

Expert Services

All

All

Account Management

SAAS Management

SAAS Application Security

Managed Services

Platform as a service (PAAS)

Other SAAS Services

4D Pillars

Cloud

Popular Solutions