Utilities Connector v3.2.1

Overview

FortiSOAR™ provides you with a number of pre-installed connectors or built-ins, such as the Utilities or Database connectors that you can use within FortiSOAR™ playbooks, as a connector step, and perform automated operations. These connectors are bundled and named based on the type of operations the connectors can perform. For example, the Database connector would contain actions that you can perform with respect to the database like querying the database. It is easy to extend and enhance these connectors.

Apart from the FortiSOAR™ Built-in connectors, Fortinet also provides a number of connectors for popular integrations like SIEMs, such as FortiSIEM, Splunk, etc., and Ticketing systems such as Jira. You can see a list of published connectors on the FortiSOAR Connectors Documentation site.

The process of installing, configuring, and using connectors is defined in the Introduction to connectors chapter in the "Connectors Guide", which is part of the FortiSOAR™ documentation or see the Installing a connector and Configuring a connector articles.

FortiSOAR™ Built-in connectors are upgraded by default with a FortiSOAR™ upgrade. Use the Content Hub to upgrade your connectors to the latest version. For more information on the connector store, see the Introduction to connectors chapter and see the FortiSOAR Built-in connectors article.

Important: Before you upgrade your FortiSOAR™ version, it is highly recommended that you take a backup of your FortiSOAR™ Built-in connector's (SSH, IMAP, Database, Utilities, etc.) configuration since the configuration of your FortiSOAR™ Built-in connectors might be reset if there are changes to the configuration parameters across versions.

FortiSOAR™ in version 6.0.0 has refactored the output of some operations of some built-in connectors such as the "Email: Extracts email's metadata from email file" operation of the Utilities connector. Due to refactoring, there have been some changes to the output of the Utilities connector which are not backward compatible. For example, the body key in the response now returns a dictionary with keys 'json', 'html' instead of an array of these. It is recommended to switch to the new format. However, if you want to retain the old output format, and you have only upgraded the connector version and not upgraded your FortiSOAR™ version, then you must do the following:

1. Append the following in the /opt/cyops-integrations/integrations/configs/config.ini file:
   [connector_configuration]
extract_email_metadata_legacy: true
   The output of the "Email: Extracts email's metadata from email file" operation is determined by the extract_email_metadata_legacy parameter. If the extract_email_metadata_legacy parameter is set as true then the output will be generated in the old format, and if it is set as false, then the output will be generated in the new format.
2. Add the following at the end of the /opt/cyops-integrations/integrations/integrations/settings.py file:
   APPLICATION_CONFIG = application_config
3. Restart the uswgi service using the following command:
   # systemctl restart uwsgi

Important: If you are upgrading to FortiSOAR™ 6.0.0, then you need to perform only steps 1 and 3.

Utilities

Use this connector for performing operations in FortiSOAR™, such as performing a FortiSOAR™ search using the Query API, updating a FortiSOAR™ resource, and creating a FortiSOAR™ resource. This connector also contains other useful utilities such as extracting email metadata such as indicators from an email file, uploading a file to FortiSOAR™ and associating that file with an attachment, i.e., providing the File IRI in the output, converting file formats, such as XML to JSON or CEF to JSON, and zipping and password protecting a file.

This connector is ready to use, and you do not need to configure this connector.

Notes on using actions in the Utilities connector

• If you use the download file action of the Utilities connector in a playbook, for example, the "File: Download File From URL" step, the "cyops_filepath" key in the response contains the name of the file under /tmp on a FortiSOAR instance. You can use this value as an input to any next step that needs to consume this file. For example, sending the file as an attachment using the corresponding action of the 'Exchange' connector.
• Any file that is downloaded on the agent using the download file action of the Utilities connector will not be available for any of the next steps in the playbook. For example, if you create a playbook add then add the Utilities connector operation "File: Download File From URL" step for an FSR Agent configuration, add the download URL, and save the step. Next, you add the "File: Create Attachment From File" step that provides the file reference from the "Download" step, and save and run the playbook. The playbook will fail with an error such as "Connector step is failing with error 'Invalid input :: Given filename/filepath /tmp/f68ab00fb7da4dfd9db4bb95abb1471e doesn't exists'". This is expected behavior since when a file download operation is performed on an FSR agent, the operation cleans the file when the response is returned to the base FortiSOAR™ node. Therefore, if any following step expects the downloaded file to be present at the agent will cause that step to fail. For more information on FSR agents, see the Segmented Network Support chapter in the "Administration Guide", which is part of the FortiSOAR™ documentation.

Example of using the Utilities connector

An example of the utilities that are included in the Utilities connector is the "Utils: Convert JSON into an HTML table" utility. This utility generates an HTML-formatted string based on the input JSON. The HTML-formatted string will appear as follows:

<table class="cs-data-table">
<tr>
<th>pid</th>
<th>path</th>
<th>username</th>
</tr>
<tr>
<td>4</td>
<td>c:\\windows\\system32\\ntoskrnl.exe</td>
<td>NT AUTHORITY\\SYSTEM</td>
</tr>
<tr>
<td>296</td>
<td>c:\\windows\\system32\\smss.exe</td>
<td>NT AUTHORITY\\SYSTEM</td>
</tr>
</table>

For the given input JSON:

{
"operation": "get_process_list",
"data": [
{
"path": "c:\\windows\\system32\\ntoskrnl.exe",
"create_time": 1529090266,
"command_line": "",
"parent_guid": "00000004-0000-0000-0000-000000000000",
"proc_guid": "00000004-0000-0004-01d4-04dd8adc9fb8",
"parent": 0,
"username": "NT AUTHORITY\\SYSTEM",
"pid": 4,
"sid": "s-1-5-18"
},
{
"path": "c:\\windows\\system32\\smss.exe",
"create_time": 1529090266,
"command_line": "\\SystemRoot\\System32\\smss.exe",
"parent_guid": "00000004-0000-0004-01d4-04dd8adc9fb8",
"proc_guid": "00000004-0000-0128-01d4-04dd8ae6291c",
"parent": 4,
"username": "NT AUTHORITY\\SYSTEM",
"pid": 296,
"sid": "s-1-5-18"
},
"status": "Success",
"message": ""
}

For some of the operations, enhancements, bug fixes, and various versions of the Utilities connector see the Utilities Connector Release Notes section.

Utilities Connector Release Notes

Version 3.2.1

• Enhanced the "Utils: Convert JSON into a HTML Table" operation by adding Display as a parameter to this operation. Using the Display parameter you can select the layout of the converted HTML table to be Vertical or Horizontal (default).
• Fixed the issue of retention of files that were generated by the playbook once the playbook execution is completed or the playbook fails. Now, the files generated by the playbook are deleted on completion or failure of the playbook.
• Fixed the issue with the "File: Unzip" operation where extraction was failing for files with long names.
• Updated the "File: Upload a file in the system and Create an Attachment" operation as follows:
  • Added a tooltip for the Filename parameter so the user gets to know that this will be the name of the created attachment.
  • Fixed an issue that caused this operation to fail if the length of the filename length was greater than 255 characters.
  • Fixed an issue where the file would not be uploaded to FortiSOAR if the filename contained special characters.

Version 3.2.0

• Froze the markdown package that was used by the Utilities connector to version 3.4.4, since there is an issue with the newer version of the markdown python library that was used by the Utilities connector.
• Removed the deprecated 'Email: Extracts email's metadata from email file (Deprecated)' action from the Utilities connector actions. This action was retained till this version to ensure backward compatibility.

Version 3.1.2

• Updated the "FSR: Create/Update Global Variables" operation to support record upsert, i.e., this operation would earlier update a current macro and fail if the macro did not exist. Now, if the macro does not exist, it gets created, and if the macro exists, then its value gets updated.
• Fixed an issue in the "FSR: Extract Artifacts from String" action where the device ID was being extracted as an IP indicator. Since the IPv4 regex is generic in FortiSOAR, therefore, now, support has been added to customize the regex used for extracting 'Host', 'IP (IPv4 and IPv6)', and 'Email' type artifacts.

Version 3.1.1

• Updated the "Utils: Converts XML, CSV, XLS or XLSX files to Dictionary" operation to fix a bug that failed to clean up downloaded files from tmp. This could lead to tmp getting filled up regularly leading to issues while running other operations.

Version 3.1.0

• Added a new operation named "Utils: Convert Markdown string to HTML" that converts markdown strings to HTML. These HTML strings can then be used in emails etc, with proper formatting such as working hyperlinks.
• Added a new operation named "File: Read PEM Certificate" that ingests PEM files and parses their content, i.e., you can specify the file path, file IRI, or attachment IRI of the certificate (.pem) file, whose contents is then read and parsed by this operation.
• Updated the "Utils: Format as RichText" operation to support both HTML and Markdown as formats for Rich Text Formatting. Earlier, only the Markdown Editor was supported for rich text. You can now also switch between the HTML and Markdown editors, with the Markdown editor set as the default option.
• Updated the "Utils: Converts XML, CSV, XLS or XLSX files to Dictionary" operation to fix a bug that failed to convert .xlsx files to the dictionary format. Now, you can specify .xlsx files and this operation will be able to successfully convert them to the dictionary format.

Version 3.0.5

• Added compatibility fixes to make the Utilities connector v3.0.5 compatible with FortiSOAR™ versions earlier to 6.4.3, i.e., version 3.0.5 of the connector is compatible with 6.4.1, 6.4.0, and earlier versions of FortiSOAR™.
• Added support for executing 'File-related' operations such as "File: Download File from URL", "File: Create Attachment from File, etc. on the "Agent" machine from FortiSOAR™ version 6.4.3 onwards.
• Updated the output schema of the "Email: Extracts email's metadata from email file" operation to generalize the Message-id as message-id. This also fixes the issue of data ingestion failing if the email has .msg files as attachments.

Version 3.0.4

• Updated the "File: Create File from String" operation to support binary data.
• Updated the "Email: Extracts email's metadata from email file" operation to solve the issue due to which this operation did not extract all the characters from an email attachment.
  This issue occurs due to a limit set on the content length of an email, which trimmed out content exceeding the limit. Now, this limit has been made configurable in the EMAIL_CHARACTER_COUNT_MAX parameter, and by default, it is set to 100000 characters. To change the limit, edit the EMAIL_CHARACTER_COUNT_MAX parameter that is present in the /opt/cyops-integrations/integrations/configs/config.ini file. For example:
  EMAIL_CHARACTER_COUNT_MAX = 100000 <- Change this value as per your requirement.
  If you have updated the value of the EMAIL_CHARACTER_COUNT_MAX parameter then you have to restart the uwsgi and celeryd services for the change to take effect.
  Use the # systemctl restart celeryd and # systemctl restart uwsgi commands to restart the celeryd and uswgi services.
  Important: FortiSOAR™ version 6.4.3 and later supports configuring the EMAIL_CHARACTER_COUNT_MAX parameter.

Version 3.0.3

• Updated the CSS of the "Utils: Convert JSON into a HTML Table" operation to display the Show More button only if there are more than "5" rows in the table. This enables effortless viewing and editing of large tables in the Collaboration Panel that has been converted from JSON to HTML using this operation and added into the alert using a playbook.

Version 3.0.2

• Optimized the output of the "Utils: Convert JSON into a HTML Table" operation so that the Show More button is not visible in case there are fewer than 5 records in the table.
• Fixed a bug in the "Email: Extracts email's metadata from email file" operation, which did not delete the attachment that is downloaded to the /tmp directory once the playbook completes its execution. Now, when the "Email: Extracts email's metadata from email file" operation extracts any eml file that contains an attachment, then the attachment is downloaded as a temporary file, and stored in the /tmp directory, which will now be deleted once the playbook completes its execution.

Version 3.0.1

• Updated the branding for the Utilities connector such as updating the connector publisher to "Fortinet" and updating the names, descriptions, and action names of playbooks based on the branding changes. For example, the "CyOPs: Extract Artifacts from String" action has been renamed to "FSR: Extract Artifacts from String", the "CyOPs: Make CyOPs API Call" action has been renamed to "FSR: Make FortiSOAR API Call", the "File: Upload a file to CyOPs and Create an Attachment" action has been renamed to "File: Upload a file in the system and create an attachment", etc. Also, updated parameters within actions to match the branding, for example in the "File: Zip" action, renamed the "CyOPs File IRI or Filename" field to "File IRI or Filename".

Version 3.0.0

• Added support for URL encoding and decoding in the "FSR: Extract Artifacts from String" operation.
• Added support to create a vertical table template, using the HTML template, in the "Utils: Convert JSON into a HTML table" operation. A Styling checkbox has also been added, which provides support for applying custom HTML styles to the table. If you select this checkbox, then you can add custom HTML styles in the Table Styling field.
• Refactored the output of the "Email: Extracts email's metadata from email file" operation.
• Due to refactoring, there have been some changes to the output of the Utilities connector which are not backward compatible. For example, the body key in the response now returns a dictionary with keys 'json', 'html' instead of an array of these. It is recommended to switch to the new format. If you want to retain the old output format, then you can update the extract_email_metadata_legacy to true in the /opt/cyops-integrations/integrations/configs/connectors.yml file. By default, extract_email_metadata_legacy is set as false.
  Once you have updated the key, restart the uwsgi service using the following command:
  # systemctl restart uwsgi
• Renamed the action "CyOPs: Update Macro" to "FSR: Create/Update Global Variables".
• The connector logo has also been updated in version 3.0.0 of the Utilities connector.

Version 2.7.0

• Added support for IPv6 address in the "FSR: Extract Artifacts from String" operation.
• Enhanced the "Email: Extracts email's metadata from email file" operation to allow the extraction of embedded emails or messages up to a depth of 4 levels.
• Enhanced the logic of extraction of emails and messages to make the extraction process more efficient. Also, fixed the issue of freezing of artifacts for long URLs using the Extract Artifacts operation.
• Added small fixes and enhancements to improve the functionality and readability of various actions.

Important: If you upgrade the Utilities connector to v2.7.0, then playbooks that contain the "Utils: Make REST API Call" and "FSR: Make FortiSOAR API Call" operations reset the get, put, post, and delete methods if they were not written in capital letters. This occurs due to the current enhancements in the Utilities connector v2.7.0. However, there will not be any functional impact due to this since the playbook will execute successfully after the upgrade, and you can correctly add the methods to the playbook post-upgrade.

Version 2.6.0

• Fixed important bugs such as modifying the URL regex in the "Extract Artifacts from String" operation.

Version 2.4.1

• Updated the output of the Utils: Is IP in CIDR operation so that consistency is maintained in the output format for both the matched IP result and the unmatched IP result. It also contains an enhanced logo for the Utilities connector.

Version 2.4.0

• Added the "File: Unzip" operation that you can use to unzip a file or a password-protected file that is present in your FortiSOAR™ instance, i.e., the file must either be present in the Attachment module in FortiSOAR™ or in the /tmp folder of your FortiSOAR™ instance.
• Added the "File: XOR Decryption" operation to perform XOR Bitwise operations on a file that is present in your FortiSOAR™ instance. This operation stores the output of the operation, i.e., the decrypted data in a file that you have specified.
• Updated the "Utils: Convert XML, CSV, XLS or XLSX Files to Dictionary" to add the support of converting files that are in XLS or XLSX format to a dictionary. Earlier only XML and CSV formats were supported.
• Updated the "FSR: Extract Artifacts from String" operation as follows:
  • Renamed the Whitelist field to Whitelist Artifacts. In this field, you can add a comma-separated list of artifacts such as, domains, CIDR Ranges, and IP addresses that will be omitted from the extraction results. For example, if you have specified "example.com" in the Whitelist Artifacts field, then all related artifacts such as "test@example.com", and "example.com/sub1/sub2" will be considered as whitelisted.
  • Added the Whitelist Private Addresses checkbox. Select this checkbox to allow private addresses to be whitelisted. If you do not select this checkbox, then private RFC1918 IPv4 addresses such as, 10.0.0.0/8 will not be whitelisted.
  • Added the Override URL Extraction Settings checkbox. Select this checkbox to edit the default regex expression or provide a custom regex that will override the default URL extraction settings.
    The RegEx Expression For URL Extraction field is displayed only when you select this option.
• Renamed the "File: Zip and Password Protect" operation to "File: Zip" and updated this operation as follows:
  • Updated the Filename field to File IRI or Filename since now you can either specify a file IRI or a file name as an input to this operation. Earlier you could only specify a filename.
  • Updated the Password field to be a not-required field, which means that you can now create the zip file with or without a password. Earlier you could only create the zip file with password protection.

Apart from the above enhancements, the Utilities connector also contains better error handling, which includes displaying enhanced, precise, and detailed error messages, making it easier for you to debug Utilities connector issues. For information on common Utilities connector issues, see the Common Utilities connector errors section in the Debugging common playbook and connector issues article present in the Fortinet Knowledge Base.

Version 2.3.0

• Enhanced the "FSR: Extract Artifacts from String" operation as follows:
  • Inclusion of a new parameter named Whitelist parameter. This operation extracts indicators from the specified string which are used for further processing. This enhancement provides you with the ability to add a comma-separated list of indicators that you want to whitelist, i.e., these indicators will not be included in the result of this operation. The output schema of this operation now displays only the results and the whitelisted_results fields; all the other output fields are not visible. However, the other output fields will yet be supported for a couple of more releases to ensure backward compatibility.
  • Extraction of URLs only if they are pre-pended with http, https, ftp, or sftp.
• Enhanced the "Email: Extract email's metadata from email file" operation to also extract attachments from the .eml or .msg files. Earlier only the metadata used to be extracted (and not the attachments) from the .eml or .msg files.

Version 2.2.0

• Updated the "Email: Extracts email's metadata from email file" step to allow users to specify a file path of the attachment as an input to this step. This action has also been enhanced to make the File Type an optional parameter since the action now automatically recognizes the file type of the attachment. Therefore, users do not require to specify the file type explicitly.
• Updated the Create Attachment checkbox to be an optional field in the "File: Upload a file in the system and Create an Attachment" operation, so that users can use this operation without mandatorily having to create an attachment record.

Version 2.1.3

• Fixed the "File: Create Attachment from File" step to not require authentication as a part of the header in the playbook step.
• Fixed the "FSR: Extract Artifacts from String" step in the Utilities connector to parse email addresses and URLs correctly, and this step is also enhanced to allow the format of the result to
• Better error handling: Display of enhanced, precise, and detailed error messages, making it easier be loopable, i.e., all results in unified_result.r for you to debug Utilities connector issues.

Version 2.1.2

• Added a new function named "Utils: Convert XML or CSV Files to Dictionary". This function converts XML or CSV files to JSON. The "Utils: Convert XML to Dictionary" function has changed to support converting an XML string to JSON. This version also includes some bug fixes such as marking the "Name" field as mandatory for the "Create Attachment From File" function. Earlier, you could leave the "Name" field empty and run the playbook, which would cause the playbook to fail.

Version 2.1.0

• Added new functions such as the "Zip and Password Protect" function and enhancements to a number of functions such as the "Compute Hash" function has been enhanced to contain the downloaded file path as a result in the file path.