Debugging and Optimizing Playbooks

This chapter explains how you can easily debug playbooks in FortiSOAR using execution history and executed playbooks logs. It also provides you information on how to tune various keys and troubleshoot playbook errors.

The Integrations API call has been changed in version 7.0.0 to support only POST calls; earlier GET calls were also supported. Therefore, if you have any existing playbooks that uses the GET calls, then that playbook will fail. To resolve this issue, you have to manually change the method from GET to POST in your playbooks.

From version 7.0.2 onwards, you can define the logging levels (INFO or DEBUG) for your playbook execution logs, both globally as well as at the individual playbook level. For more information on playbook logging levels, and how to set those levels on individual playbooks, see the Introduction to Playbooks chapter. Note that the 'Debugging Playbooks' content assumes that you are running the playbooks in the DEBUG mode.

Debugging Playbooks

As you develop more sophisticated Playbooks, the ability to easily debug playbooks becomes exceeding important. FortiSOAR has designed the Execution History to make it easier for you to see the results of your executed playbooks and for you to debug playbooks.

Use the Executed Playbook Logs icon () that appears on the top-right corner of the FortiSOAR screen to view the logs and results of your executed playbooks as soon as you log on to FortiSOAR. You can also use the executed playbook logs to debug your playbooks.

FortiSOAR implements Playbook RBAC, which means that you can view logs of only those playbooks of which you (your team) are the owner. For more information, see the Introduction to Playbooks chapter.

The Execution History provides the following details:

• Playbooks have been organized by the parent-child relationship.
  
• Playbooks have a console using which you can see debug messages with more significant details.
• Playbook designer includes the playbook execution history option.
• Playbooks can be filtered by Playbook Name or Record IRI, user, date range, or status.
• Playbook Execution History Log contains details of the playbook result, including information about the environment and the playbook steps, including which steps were completed, which steps are awaiting some action, which steps were failed, and which steps were skipped.

From version 7.0.2 onwards, users will not be able to view the execution history of 'legacy' playbooks, i.e. the execution history of playbooks that were run before release 6.0 will not be visible.

The Executed Playbook Logs do not display the Trace information from the error message so that the readability of the Executed Playbook Logs is enhanced since the clutter in the error details screen is reduced and you can directly view the exact error. The Trace information is yet present in the playbook logs.

FortiSOAR also contains enhanced the error messages that are precise and detailed making it easier for you to debug playbook issues. For information about the common playbook error messages and how to debug them, see the Debugging common playbook and connector errors article present in the Fortinet Knowledge Base.

You can access the playbook execution history as follows:

• Clicking the Executed Playbook Logs icon () in the upper right corner of the FortiSOAR screen.
  You have an option of purging executed playbook logs from the Executed Playbooks Log dialog. For more information see Purging Executed Playbook Logs.
• Clicking Tools > Execution History in the playbook designer to view the execution history associated with that particular playbook.
  
• Clicking the Executed Playbook Logs icon in the detail view of a record such as an alert record to view the playbooks that have been executed on that particular record in a flowchart format. This makes it easier for users to view the flow of playbooks, especially useful for viewing the parallel execution paths in playbooks.
  
  You can toggle between the Record view, which displays only the logs of the playbooks that are executed on that particular record and the Global view displays logs for all the playbooks that are executed on the FortiSOAR system. You can also purge executed playbook logs for a particular record by clicking the Settings icon on the top-right of the Executed Playbook Logs dialog in the 'Detail' view of that record, and then selecting the Purge Logs option. For more information see Purging Executed Playbook Logs.

Executed Playbook Logs

Click the Executed Playbook Logs icon in the upper-right corner of FortiSOAR to view the logs and results of your executed playbook. Clicking the Executed Playbook Logs icon displays the Executed Playbook Logs dialog as shown in the following image:

The Executed Playbook Logs displays the executed playbooks in the flowchart format, as is displayed in the playbook designer. This makes it easier for users to view the flow of playbooks, especially useful for viewing the parallel execution paths in playbooks.

Playbooks are sorted by chronological datetime, with the playbook that was executed last being displayed first. All playbooks are displayed with 10 playbooks being displayed per page. Click a playbook in the list to display it in the flowchart format and also see the details of the playbook result, the environment and the playbook steps, including which steps are completed, failed, awaiting or skipped.

The Execution Playbook Log dialog also displays a count of the total playbooks executed, the date and time of when the playbook was executed, the time taken for executing the playbook and the mode in which the playbook was run, i.e., Info (default) or Debug. For more information on the modes of running a playbook and how to change the modes on individual playbooks, see the Introduction to Playbooks chapter.

From release 7.2.2 onwards, the executed playbook log displays information about who has triggered or terminated the playbook in the 'Executed by' field as follows:

• Manually-triggered playbooks display the username of the user who has triggered or terminated the playbook in the 'Executed by' field.
• API playbooks that are triggered by a particular user's token display the username of that user in the 'Executed by' field.
• Playbooks that are triggered using the On Create, On Update, or On Delete triggers display 'Playbook' in the 'Executed by' field if the record creation, updation, or deletion is a result of an automated action using playbooks, for example, data ingestion, enriching indicators, etc.

• Playbooks executed by users from the playbook designer, irrespective of their trigger type, display the username of the user who has triggered or terminated the playbook in the 'Executed by' field.

When you click on playbook steps, you can toggle the ENV button to toggle between the environment of the step and the output of the step. You can also copy the environment, error, and step details to the clipboard by clicking the Copy 'Env' to Clipboard or Copy 'OUTPUT' to Clipboard button.

You can also open the playbook directly in the playbook designer from the Executed Playbook Logs dialog by clicking the Edit Playbook button that appears in the right section of the dialog.

You can collapse and expand the Executed Playbook Logs dialog by clicking the << or >> arrows as shown in the following image:

You can refresh the playbook logs and filter logs associated with playbooks using the Filter icon:

Clicking the Filter icon allows you to filter playbook logs using the following options:

• Playbook Name: In the Search by Playbook Name or Record IRI field, filter the log associated with a particular playbook, based on the playbook name or the record IRI associated with the playbook.
  Example of filtering logs using the Record IRI: /alerts/bd4bf0a6-b023-4bd7-a182-f6938fa37ada.
• From Date: You can filter the log based on the date from which the playbooks were executed.
• To Date: You can filter the log based on the date till when the playbooks were executed. Using the From Date and To Date fields, you can create a data range for retrieving the logs of playbook executed during that time period.
• Run By: From the Run By drop-down list, filter the log associated with a particular playbook, based on the user who has run the playbook.
• Status: From the Status drop-down list, filter the log associated with a particular playbook, based on the status of the playbook execution. You can choose from the following options: Incipient, Active, Awaiting, Paused, Failed, Finished, or Finished with error.

You will also see the timestamp when the playbook was executed and the time it took for the playbook to complete its execution.

To purge Executed Playbook Logs, click the Settings icon on the top-right of the Executed Playbook Logs dialog and select the Purge Logs option. For more information, see Purging Executed Playbook Logs.

To terminate a playbook that are in the Active, Incipient, or Awaiting state, click the Terminate button. To terminate all running instances of a particular type, click the Settings icon and select the Terminate Running Instances option. For more information, see Terminating playbooks.

Environment

Click Env to view the complete environmental context in which the playbook was executed, including the input-output and computed variables across all steps in the playbook, if your playbook is executed in the 'Debug' mode. If the playbook is executed in the 'Info' mode it displays the status. In release 7.2.1, the complete JSON tree has a reference of the "vars" root node (earlier ENV was written) at the top of JSON), making the writing of jinja in playbooks easier.

Running huge ingestions and other workflows that load several records into memory can cause the memory usage to be high. In case of connector actions, the 'env' was passed on to each action and it also used to get the 'env' triggered from the workflow (worker 'env'). This unnecessarily increases the memory requirement and limits workflow scaling. Also, connectors only need a minimal information such as requests headers, public-private key, and auth info from the 'env' that can be selectively passed. Therefore, to solve the memory consumption issue, connectors are passed only the required 'env' fields and not the complete environment information. However, if you observe any issues in a connector or you specifically require the complete environment to be passed to the connector, then you need to add the CONNECTOR_KEEP_COMPLETE_ENV variable at the end of the /opt/cyops-workflow/sealab/sealab/settings.py file, and save the file. Then, you must restart the celeryd service using the following command:
# systemctl restart celeryd

Playbook Steps

Clicking playbook steps display the input, output, and configuration for that step. You can toggle the ENV button to toggle between the environment in which the playbook was executed and the steps of the playbook. You can also copy the environment, error, and step details to the clipboard by clicking the Copy 'ENV' to Clipboard or Copy 'OUTPUT' to Clipboard button.

Clicking playbook Steps section lists all the steps that were part of the playbook and displays the status of each step using icons. The icons indicate whether the step was completed (green tick), skipped (grey skipped symbol), awaiting some action (orange hour glass symbol) or failed (red failed symbol).

For example, if a playbook is awaiting some action, such as waiting for approvals from a person or team who are specified as approvers, then the state of such playbooks is displayed as Awaiting.

The status of the playbook will display as "Awaiting" till the action for which the playbook execution halted is completed, after which the playbook will move ahead with the workflow as per the specified sequence.

You can click on a playbook step for which you want to view the details, and you will see tabs associated with the playbook step: Input, Pending Inputs (if the playbook is in the awaiting state), Output (if the playbook finishes) or Error (if the playbook fails), and Config.

Input Tab

The input tab displays data, in the case of the first step of the playbook such as the Start step, input arguments and evaluated arguments. The data displays the trigger information for the playbook. The input_args displays the input in the jinja format that the user has entered for this step. The evaluated_args displays what the user input was evaluated by the playbook once the step gets executed.

Pending Inputs Tab

If a playbook is in an "Awaiting" state, i.e., it requires some input or decision from users to continue with its workflow, then the Pending Inputs tab is displayed:

Once the user provides the required inputs and submits their action, the playbook continues its execution as per the defined workflow.

Output or Error Tab

If the playbook step finishes, then the Output tab displays the result/output of the playbook step.

From release 7.2.0 onwards, the contents of the output tab have been enhanced in case of manual input playbooks to include the name of the user (username) who has taken the decision on manual input leading to the resumption of the playbook:

If the playbook step fails, then the Error tab displays the Error message for that step. Click the step that has the error (step with a red cross icon) to view the error message, so that it becomes easier for you to know the cause of the error and debug the cause of the playbook failure.

FortiSOAR has enhanced error messages by making them more precise and thereby making it easier for you to debug the issues. Also, the Trace information has been removed from the executed playbook log to reduce the clutter in the error details screen and directly display the exact error. The Trace information will be present in the product logs located at:

• For Playbook runtime issues: /var/log/cyops/cyops-workflow/celeryd.log
• For connector issues in cases where playbooks have connectors: /var/log/cyops/cyops-integrations/connectors.log

For information about the common playbook error messages and how to debug them, see the Debugging common playbook and connector errors article present in the Fortinet Knowledge Base.

FortiSOAR also provides you with the option to resume the same running instance of a failed playbook from the step at which the playbook step failed, by clicking the Rerun From Last Failed Step button. This is useful in cases where the connector is not configured or you have network issues that causes the playbook to fail, since you can resume the same running instance of the playbook once you have configured the connector or resolved the network issues. However, if you change something in the playbook steps, then that would be a rerun of the playbook and not a resume or retry of that playbook.

Users who have Execute and Read permissions on the Playbooks module can rerun playbooks in their own instance. Administrative users who have Read permissions on the Security module and Execute and Read permissions on the Playbooks module can rerun their own playbooks and also playbooks belonging to users of the same team.

Notes:

• If you have upgraded your FortiSOAR system, then you can resume only those playbooks that were run after the upgrade.
• If you have a playbook that had failed before you upgrade your FortiSOAR system, and post-upgrade you try to resume the execution of that playbook, then that playbook fails to resume its execution.

To resume the running instance of a failed playbook, do the following:

1. Open the Executed Playbook Logs dialog.
2. Click the failed playbook that you want to resume, and then click the Rerun From Last Failed Step button.
   FortiSOAR displays the Playbook retriggered from last failed step message and the failed playbook resumes from the failed step:
   
   A playbook that has been rerun will display the Retriggered text.
   

Config Tab

The Config tab displays the step variables detailed entered by the user for the particular step and also includes information about whether other variables, such as ignore_errors, MockOutputUsed, or the when condition have been used (true/false) in the playbook step.

Instances Tab

The Instances tab is displayed in case of playbooks containing reference playbook steps. The "Instances" tab allows users to see details such as, name and status of all child instances in a single view.

For example, as displayed in the following image, the "Testing Reference Playbook" references "child 1", which in turn references 10 other child playbooks because of the loop applied on the reference playbook step. When you click the "child 1" step, you can see the "Instances" tab, containing the status of the step "Finished", names of all its child playbooks, and the name of the step that referenced the playbook, which is "Reference to Child 2":

Link to Child Playbooks

Playbooks have been organized by the parent-child relationship. The Parent playbook displays a link that lists the number of child playbook(s) associated with the parent playbook. Clicking the link displays the execution history for the child playbook(s).

The UI of the execution playbook log displays playbooks that contain various levels of child playbooks in the same visual execution log window. You can click the parent playbook and view its child playbooks, and similarly you can view the children of the child playbook by clicking the child playbook all without losing context of the playbook. You can also see the breadcrumb navigation from parent playbook to the child playbook at the top of the playbook log.

The Executed Playbook Logs displays the execution history of the child playbooks, i.e., you can search for the child playbook in Executed Playbooks Logs and the search results will display the child playbook and its execution history and you can also use the Load Env JSON feature in the Jinja Editor making debugging of the child playbooks easier.

Child playbooks inherit the logging level setting from their parent playbook, irrespective of their own setting. For example, if the child playbook's logging level is set as INFO and its parent's is set at DEBUG; the child playbook's logging level will automatically be set at the DEBUG level. For more information on playbook logging levels, and how to set those levels, see the Introduction to Playbooks chapter.

If the parent playbook has a number of child playbooks, you can also search for child playbooks, by clicking the search icon that is present beside the child playbook link and then entering the name of the playbook in the Search by Playbook Name field. You can also filter the child playbooks on its running status, such as Incipient, Active, Awaiting, etc. by selecting the status from the All Status drop-down list.

For example, in the following image, the Testing Reference Playbook playbook has 1 child playbook: child 1. You can click child 1 to view its execution history:

As displayed in the above image, you can also see the breadcrumb navigation from parent playbook to the child playbook at the top of the playbook log. You can also view the name of the step at which the child playbook is referenced in the navigator panel. If the playbook contains a reference playbook step then you can click through the child playbooks within the same visual execution log window, allowing you to navigate through the playbook, without losing the context of the playbook. You can also easily navigate back to the parent playbooks using the breadcrumbs present on top of the log.

For example, as displayed in the following image, the "Testing Reference Playbook" contains a child playbook "child 1" at step "call child 1", which in turn contains 10 other child playbooks because of a loop applied on the reference playbook step, such as "child 2 test for long text in playbook", which in turn calls "child 3":

Purging Executed Playbook Logs

You can purge Executed Playbook Logs by clicking the Settings icon on the top-right of the Executed Playbook Logs dialog, and then selecting the Purge Logs option. Purging executed playbook logs allows you to permanently delete old playbook history logs that you do not require and frees up space on your FortiSOAR instance. You can also schedule purging, on a global level, for both audit logs and executed playbook logs. For information on scheduling Audit Logs and Executed Playbook Logs, see the Purging of audit logs and executed playbook logs topic in the System Configuration chapter of the "Administration Guide."

To purge Executed Playbook Logs, you must be assigned a role that has a minimum of Read permission on the Security module and Delete permissions on the Playbooks module.

To purge Executed Playbook Logs, click the Settings icon and select the Purge Logs option, which displays the Purge Playbook Execution Logs dialog:

In the Purge All logs before, field, select the time frame (using the calendar widget) before which you want to clear all the executed playbook logs. For example, if you want to clear all executed playbook logs before December 01st, 2019, 9:00 AM, then select this date and time using the calendar widget.

Click the Exclude Awaiting Playbooks checkbox (default) to exclude the playbooks that are in the "Awaiting" state from the purging process.

To purge the logs, click the Purge Logs button, which displays a warning as shown in the following image:

Click the I Have Read the warning - Purge Logs to continue the purging process.

Filtering playbook logs by tags

You can filter playbook execution logs by tags or keywords that you have added in your playbooks.

A user who has a role with a minimum of Update permission on the Security module can save tags, which will be applied as a default filter for playbook execution logs to all other user. A user who does not have such a role can add a tag to filter playbook execution logs and view the filtered playbook execution logs but cannot save that filter.

Click the Settings icon on the top-right of the Executed Playbook Logs dialog to view tags that have been added by default to filter the playbook execution logs. You can see a message 1 Tags Excluded, which means that playbook logs with one specific tag is being excluded by default.

You can either click the 1 Tags Excluded link or the Filter Logs By Tags option to open the Filter Logs by Tags popup as shown in the following image:

To filter playbook logs based on tags, add a comma-separated list of tags in the Tags field.

In the Mode section, choose Exclude to exclude playbook logs with the specified tags. You will observe that the #system tag is already added as a tag in the Exclude mode, which means the any playbook with the system tag will be excluded from the playbook logs. To include only those playbook logs with the specified tags, click Only Include. For example, if you only want to view the logs of phishing playbooks, i.e., logs of playbooks that have phishing tag, click Only Include and type phishing in the Tags field. You must also remove the system tag from the Only Include mode, since otherwise playbook logs with both the phishing and system tags will be included.

You can specify a comma-separated list to Include all tags or Exclude all tags. You cannot have a mix of Include and Exclude tags.

Filters will apply from the time you have set the filter, i.e., if you have added a phishing tag in the Exclude list at 16/05/2019 17:00 hours, then the filter will apply only from this time. The historical logs, logs before 16/05/2019 17:00 hours will continue to display in the Executed Playbooks Logs.

From version 7.0.1 onwards, the settings of the playbook execution history logs that are filtered using tags have been updated as follows:

• The 'global' filter will be applicable on only on the global execution log list.
• Record level playbook execution history log will show all playbook logs by default; users can apply temporary filters to filter the result. The Set as default filter for all users option has been removed from the record level playbook execution log filter settings.
• Playbook-level playbook execution log will show all logs. When you open a playbook in the playbook designer and view its execution history, you will see all the logs. There is no UI option in playbook execution history to filter the logs. For example, the default exclude filter that is applied for 'system' playbooks will not be applicable when you open the playbook execution log for a specific playbook by clicking Tools > Execution History in the playbook designer.

An example of excluding playbook logs by tag follows:
If you have added tags such as dataIngestion in your playbooks, then you can filter out the data ingestion logs by clicking Exclude and typing dataIngestion in the Tags field. If an administrator with Update rights on the Security module wants this filter to be visible to all users, then the administrator can save this filter as a default for all users, by clicking the Set as default filter for all users checkbox and then clicking the Save & Apply Filter button.

From version 7.0.1, the Set as default filter for all users checkbox has been removed from the record level playbook execution log filter settings.

If you do not have appropriate rights, you can apply the filter for only yourself by clicking the Apply Filter button and view the filtered playbook executed logs.

This applies the filter and displays text such as 2 Tags Excluded on the top-right corner of the Executed Playbook Logs dialog. Now, the Executed Playbook Logs will not display logs for any system playbook or for any data ingestion playbook.

Users (without administrative rights) can remove filters by clicking Settings > Filter Logs by Tags or clicking the <number of Tags included> link to display the Filter Logs By Tags dialog and click Clear All Tags to remove the tags added and add their own tags. However, these changes will only be applicable till that instance of the log window is open. If the page refreshes or the window reloads, then the tags specified by the administrator will again be applied.

Terminating playbooks

You can terminate playbooks that are in the Active, Incipient, or Awaiting state. Users who have Read and Execute permissions on the Playbooks module can terminate a running instance of their own playbook instance. Administrators who have Read permissions on the Security module and Execute permissions on the Playbooks module can terminate running instances of any playbook.

To terminate a running playbook instance, open the Executed Playbook Logs and click the instance that you want to terminate and click Terminate as shown in the following image:

Once you click Terminate, the Terminate Execution dialog is displayed in which you can choose to either terminate only the particular running instance, by clicking Terminate Current Instance Only or terminate all running instances, by clicking Terminate All Running Instances.

If you click Terminate Running Instance Only, then the state of that playbook changes to Terminated:

You can also choose to terminate the running instances of all playbooks that are in the Active, Incipient, or Awaiting state.

To terminate the running instances of all playbooks based on the status of the playbooks, do the following:

1. Click the Settings icon on the top-right of the Executed Playbook Logs dialog.
2. Select the Terminate Running Instances option, which displays the Terminate Running Instances dialog.
3. In the Terminate Running Instances dialog, select the status (Active, Incipient, or Awaiting) whose running instances of Playbooks you want to terminate, and click Terminate.
   
   You can rerun the playbook from the step it was terminated by clicking the Rerun Pending Steps button on the terminated playbook.

Setting up auto-cleanup of workflow execution history

Workflow Execution history is extensively persisted in the database for debugging and validating the input and output of playbooks at each step. A very large execution history, however, causes overhead regarding consumption of extra disk space, increase in the time required for upgrading FortiSOAR, etc. Therefore, it highly recommended to set up an auto-cleanup of the workflow execution history using a weekly cron schedule.

To delete the workflow run history keeping the last 'X' entries, ssh to your FortiSOAR appliance as root and run the following command:

# /opt/cyops-workflow/.env/bin/python /opt/cyops-workflow/sealab/manage.py cleandb --keep X
For example, to delete all workflow run history, apart from the last 1000 entries, use the following command:
# /opt/cyops-workflow/.env/bin/python /opt/cyops-workflow/sealab/manage.py cleandb --keep 1000

To set up a weekly schedule delete workflow history, to the above command, add a cron expression entry in the /etc/crontab file that would schedule a workflow execution history cleanup as per your requirements. Command to edit a cron job is crontab -e.

For example, the command to add an entry in the /etc/crontab file that would schedule a workflow execution history cleanup to every Saturday night and delete all workflow run history, apart from the last 1000 entries, would be as follows:
# 0 0 * * SAT /opt/cyops-workflow/.env/bin/python /opt/cyops-workflow/sealab/manage.py cleandb --keep 1000

Note that running the above command deletes the workflow entries but does not release the disk space back to the OS, i.e., it keeps it reserved for the Postgres process. This is the desired behavior, and no further action is required if the execution history cleanup is scheduled because the Postgres process would need the freed-up disk space to store further workflows. If, however, you also wish to reclaim disk space for backup or restore or other activities, you would additionally need to run a "full vacuum" on the database after you ssh to your FortiSOAR appliance as root and run the following commands:

psql -U cyberpgsql sealab
psql (10.3)
Type "help" for help.

sealab=# vacuum full;
VACUUM
sealab=# \q

Known Issue: If you do not schedule the workflow execution cleanup, and you are deleting a very large set of entries in one go, then the db cleanup command might fail due to the failure in loading the large set of entries into memory. In this case, you will have to run the command in batches.
For example:
# /opt/cyops-workflow/.env/bin/python /opt/cyops-workflow/sealab/manage.py cleandb --keep 100000
# /opt/cyops-workflow/.env/bin/python /opt/cyops-workflow/sealab/manage.py cleandb --keep 90000

Disabling Playbook Priority

You can disable playbook priority, in case playbook priority queuing is hampering your system's performance. Version 7.0.0 onwards, FortiSOAR uses rabbitmq as the message broker, and because of this priorities of an already declared queue in rabbitmq cannot be changed dynamically; for more information, see https://www.rabbitmq.com/priority.html#using-policies.

Therefore, from version 7.0.0 onwards, use the ENABLE_PRIORITY setting in the workflow engine to enable/disable playbook priority. However, before changing the ENABLE_PRIORITY setting, first delete the celery queue, after you have ensured that no data is present in the celery queue.

To enable/disable the ENABLE_PRIORITY setting in /opt/cyops-workflow/sealab/config.ini, do the following:

1. List queues and check the celery queue and its count of messages, use the following command:
   rabbitmqctl list_queues -p intra-cyops
   If the celery queue has zero messages in it, then the output would be:
   celery 0
   This ensures that no data is present in the celery queue.
2. Delete the celery queue , using the following command:
   rabbitmqctl delete_queue celery -p intra-cyops
3. Change the ENABLE_PRIORITY flag to false to disable playbook priority, or true to enable playbook priority.
4. Restart celeryd using the # systemctl restart celeryd command.

Optimizing Playbooks

Playbook steps that were looped (using the Loop option in the playbook step) can be run either in a sequentially or in parallel. For more information, see the Loop topic in the Triggers & Step chapter.

You can tune the thread pool size and other settings for parallel execution using the settings that are mentioned in the following table:

Key name and location	Description	Default value
THREAD_POOL_WORKER /opt/cyops-workflow/sealab/sealab/config.ini	The thread pool size is used for parallel execution. The THREAD_POOL_WORKER variable is used to optimize the parallel execution and enhance performance. You can reduce the default value of the thread pool size from the default value if: 1. The number of cores on your FortiSOAR instance are lesser than the default recommended. 2. The task to be executed in the loop step is synchronous in nature and thread context switching would be an overhead.	8
SYNC_DELAY_LIMIT /opt/cyops-workflow/sealab/sealab/config.ini	If the delay specified in the playbook step is higher than this threshold, then the loop step will be decoupled from the main playbook and run asynchronously. For example if you set the SYNC_DELAY_LIMIT to 60, it means that a 60 seconds check is added, and after 60 seconds the playbooks should run in parallel. This works in parallel with your playbook soft limit time, CELERYD_TASK_SOFT_TIME_LIMIT parameter. The time set in the CELERYD_TASK_SOFT_TIME_LIMIT parameter must be greater than the time set in the SYNC_DELAY_LIMIT parameter.	60
CELERYD_TASK_SOFT_TIME_LIMIT /opt/cyops-workflow/sealab/sealab/config.ini	To change the soft time limit for playbooks. The soft time limit value is set in seconds. NOTE: The soft time limit allows the task to catch an exception to clean up before it is killed: the hard timeout is not catch-able and force terminates the task. So in the case the soft time limit is exceeded and cleanup is stuck then the hard time limit (expected to be higher than the soft time limit) will force terminate the playbook task as a safety net and to avoid forever running of task.	1800
CELERYD_TASK_TIME_LIMIT /opt/cyops-workflow/sealab/sealab/config.ini	To change the hard time limit for playbooks. The time limit value is set in seconds. Note: This value should always be higher than the SOFT_TIME_LIMIT. For more details, see the Celery 4.3.0 documentation >> User guide: task_soft_limit section.	2400
CELERYD_OPTS /etc/celery/celeryd.conf	To optimize the parallel running of threads in celery so that your overall playbook execution time is reduced. By default, the workflow engine spawns a separate process running a workflow. If the tasks in the workflow are asynchronous and short lived, the thread-based workers can be enabled. For more details, see the Celery 4.3.0 documentation > User guide > Concurrency >> Concurrency with Event section.	CELERYD_OPTS=“-P=eventlet -c=30"

These optimizations also help in scaling your playbooks by resolving bottleneck that slow down playbook execution and resolving internal timeout issues.

FortiSOAR supports parallel branch execution of playbooks. Parallel branch execution optimizes playbook execution by having the ability to execute two or more independent paths parallelly.

You can enable or disable parallel execution by changing the value (true/false) of the PARALLEL_PATH variable in the [Application] section in the /opt/cyops-workflow/sealab/sealab/config.ini file. By default, a fresh install of version 5.1.0 will have the PARALLEL_PATH variable set as true.

Optimized Workflow Runtime for Memory and CPU consumption

In release 7.2.0, FortiSOAR has optimized and greatly improved the playbook execution process. Tremendous improvements are observed for playbook execution times as well as OS resource consumption during playbook execution. Some notable improvements are:

• Reduction of Virtual Memory consumption during playbook execution by 50%.
• Reduction of CPU usage during playbook execution by 50%.
• Optimized execution of Jinja expressions.

• Database space utilization is optimized by reducing storing duplicate values used by the playbook framework.

Some test cases were run to compare playbook execution times across 7.0.2 and 7.2.0:

Comparison of Playbook Execution Times between 7.0.2 and 7.2.0
Workflow Type (creates 1240 records)	Time taken for execution in FortiSOAR 7.0.2	Time taken for execution in FortiSOAR 7.2.0	Improvement*
Playbook with create record using the 'For Loop'	1 minute 51 seconds 751 milliseconds	57 seconds 190 milliseconds	1.9x Faster
Playbook with a Create Record step using the 'For Loop with the Batch option'	1 minute 44 seconds 715 milliseconds	56 seconds 710 milliseconds	1.8x Faster
Playbook with a Create Record step using the 'For Loop with the Batch option and the Parallel options'	1 minute 21 seconds 132 millisecond	24 seconds 967 milliseconds	3.2x Faster
Playbook with a Create Record step using the 'For Loop with the Batch option and the Sequential options'	3 minutes 31 seconds 70 milliseconds	1 minute 47 seconds 143 milliseconds	1.9x Faster
Playbook with a Reference step using the 'For Loop with the Parallel option'	7 minutes 40 seconds 386 milliseconds	3 minutes 16 seconds 756 milliseconds	2.3x Faster
Playbook with a Reference step using the 'For Loop with the Parallel option that runs asynchronously' Note: In the case of parallel execution, even if the playbook is completed, record creation continues in the background.	1 minute 6 seconds 567 milliseconds	8 seconds 460 milliseconds	7x Faster
Playbook with a Reference step using the 'For Loop with the Sequential option'	10 minutes 43 seconds 133 milliseconds	6 minutes 1 second 350 milliseconds	1.7x Faster
Playbook with a Reference step using the 'For Loop with the Sequential option that runs asynchronously'	2 minutes 28 seconds 10 milliseconds	15 seconds 477 millisecond	9.24x Faster
Playbook using the json2html filter in the Update Record step	5 minutes 7 seconds 664 milliseconds	1 minute 44 seconds 236 milliseconds	2.9x Faster
* - Calculated using the formula: Previous execution time / Current execution time

Similarly, the data ingestion process was also tested to compare its execution times across 7.0.2 and 7.2.0. The following steps were performed in the tests:

1. Ingest 17000 records by reading a CSV file.
2. Create corresponding tickets in a ticketing platform using an integration (connector).
3. Playbook loops over a Reference step with each iteration consisting of 5000 records.

Comparison of Playbook Execution Times between 7.0.2 and 7.2.0
Time taken for execution in FortiSOAR 7.0.2	Time taken for execution in FortiSOAR 7.2.0	Improvement*
36 min 45 seconds 134 milliseconds	14 minutes 40 seconds 812 milliseconds	2.5x Faster
* - Calculated using the formula: Previous execution time / Current execution

Troubleshooting Playbooks

Very high CPU usage and/or memory usage by 'python' process

This issue could be caused by to some errors in workflows, like updating the same record in a post-update playbook; or a flood of events from a SIEM that causes massive workflow queuing on the FortiSOAR side.

Resolution

If the large queue build-up is due to a playbook error, or similar events' flood that need not be created as alerts or incidents into FortiSOAR, you can purge the workflow queue.

Run the following command on your FortiSOAR system to see the number of workflows queued:
rabbitmqctl list_queues -p fsr-cluster

To purge the queue run the following command:

systemctl stop celeryd
rabbitmqctl purge_queue celery -p fsr-cluster
systemctl start celeryd

Purging is an irreversible action, which should be exercised with caution in a production environment.

After upgrading to release 7.2.0 playbooks fail with the 'Access Denied' error for files downloaded while running playbooks

After you have upgraded your system to release 7.2.0 or later from a release prior to 7.2.0, and you execute playbooks that download files, you will observe that such playbooks might fail with the 'Access Denied' error. This is because in 7.2.0 fsr-integrations is used as the workflow service user to create files during playbook execution. As the releases prior to 7.2.0 used nginx as the user, files created by ngnix will not be deleted by new user (fsr-integrations).

Resolution

If there is are playbooks that create files using the Code Snippet step or any such step, where the playbook is not directly creating files, then you have to ensure that those files are deleted as part of playbook flow. If this is not done, then old files that were created before the upgrade are not deleted and such playbooks fail to delete the older files because of the difference in user permissions.

Jinja cannot handle integers that have more than 16 characters

This issue is caused due to the JavaScript max integer safe size, i.e., 2⁵³ - 1, i.e., 9007199254740991.

Resolution

You do not need to do anything since the difference is seen only while rendering the data in the browser, but not in the Database persistence.

Playbooks failing with the Picklist item:<name of picklist filter> error

If the picklist filter in the playbook has a value that does not exactly match with the existing actual value (case sensitive), then such playbooks will fail.

Resolution

Ensure that you have specified the value of the playbook filters exactly as its existing value. From release 7.2.0 onwards, an exact case match is required for playbook filters to work.
For example, if you have specified {"Severity" | "MEDIUM", "@id")}} in a playbook, then this playbook will fail if the actual value of Severity is Medium.

Filters in running playbooks do not work after you upgrade your system in case of pre-upgrade log records

You can apply filters on running playbooks using the Executed Playbook Logs. These filters will apply to log records that are created post-upgrade and will not apply to log records that were created pre-upgrade.

For log records that were created before the upgrade, use the playbook detail API:
GET: https://<FortiSOAR_HOSTNAME/IP>/api/wf/api/workflows/<playbook id>/?format=json

To get the playbook id, use the playbook list API:
GET: https://<FortiSOAR_HOSTNAME/IP>/api/wf/api/workflows/?depth=2&limit=30&ordering=-modified

Playbooks are failing, or you are getting a No Permission error

Resolution

When the Playbook does not have appropriate permissions, then playbooks fail. Playbook is the default appliance in FortiSOAR that gets included in a new team.

If you cannot access records, such as alerts, then you must ensure that you are part of the team or part of a sibling or a child team that can access the records, and you must have appropriate permissions on that module whose records you require to access or update. Only users with CRUD access to the Appliances module can update the Playbook assignment. For more information on teams and roles, see the Security Management chapter in the "Administration Guide."

Playbook fails after the ingestion is triggered

There are many reasons for a playbook failure, for example, if a required field is null in the target module record, or there are problems with the Playbook Appliance keys.

Resolution

Investigate the reason for failure using the Playbook Execution History tab (earlier known as Running Playbooks) in the Playbook Administration page. Review the step in which the failure is being generated and the result of the step, which should contain an appropriate error message with details. Once you have identified the error, and if you cannot troubleshoot the error, contact the FortiSOAR support team for further assistance using the Fortinet Customer Service & Support web portal at https://support.fortinet.com/.

Incorrect Hostname being displayed in links contained in emails sent by System Playbooks

When you are using a system playbook that sends an email, for example, when an alert is escalated to an incident, and an Incident Lead is assigned, then the system playbook sends an email to the Incident Lead specified. The email that is sent to the Incident Lead contains the link to the incident using the default hostname.

Resolution

To ensure that the correct hostname is displayed in the email, you must update the appropriate hostname as per your FortiSOAR instance, in the Playbook Designer as follows:

1. Open the Playbook Designer.
2. Click Tools > Global Variables to display a list of global variables.
3. Click the Edit icon in the Server_fqhn global variables, and in the Field Value field add the appropriate hostname value.
4. Click Submit.
   The system playbook will now send emails containing the updated hostname link.

In the system playbook (or any playbook) that is sending an email, ensure that you have used the Server_fqhn global variable in the Send Email step.

Purging executed playbook logs issues

If you are facing issues while purging of executed playbook logs such as, the purge activity is taking a long time or the purging activity seems to be halted, then you could check if the Soft time limit (600s) exceeded for workflow.task.clean_workflow_task[<taskid>] error is present in the /var/log/cyops/cyops-workflow/celeryd.log file. The Soft time limit error might occur if the amount of playbook logs to be purged is very large.

Resolution

Increase the value set for the LOG_PURGE_CHUNK_SIZE parameter, which is present in the [application] section, of the /opt/cyops-workflow/sealab/sealab/config.ini file.

By default, the LOG_PURGE_CHUNK_SIZE parameter is set to 1000.

Playbooks fails with the "Too many connections to database" error when using the "parallel" option for a loop step in Playbooks

Playbooks can fail with the Too many connections to database error when you have selected Parallel in a loop step to execute playbook steps in parallel.

Resolution

To resolve this issue, reduce the number of parallel threads. To reduce the number of parallel threads, you have to change the value of the THREAD_POOL_WORKER variable. The THREAD_POOL_WORKER variable is present in the /opt/cyops-workflow/sealab/sealab/config.ini file, and by default the value of this variable is set to 8.

Playbooks fails with the "Picklist item not found" error

Your playbooks fails with an error such as:
Picklist Item: /api/3/picklists/a1bac09b-1441-45aa-ad1b-c88744e48e72 not found URL: https://localhost/api/3/alerts

Resolution

This issue is caused when you have specified an incorrect value for a picklist item. To resolve this issue, check and correct the values of the picklist item in the current step or the previous step of the playbook.

Correcting the server address for the manual input endpoints sent in emails

FortiSOAR uses the dynamic variable 'Server_fqhn' to construct the server url for all links in emails. The default value of this variable is the hostname of the machine set at the time the VM Configuration wizard is run. If your deployment is in a NATed or cloud environment, the public address of the instance will not match the hostname, and the manual input email might contain a wrong server address.

Resolution

To rectify this issue, you must update the 'Server_fqhn' dynamic variable and provide the DNS resolvable FQHN of your FortiSOAR instance.

Playbooks fail if any of their steps attempt to connect to a database directly, without a valid password, from FortiSOAR release 7.3.0 onwards

In release 7.3.0, access to PostgreSQL without a password has been restricted. This causes a failure in the case of any playbook step that uses the 'Database' connector to access a database directly, without a valid password.

Resolution

Update the playbook step to use a read-only database user to access the database.

Frequently Asked Questions

Q: Is there a way to force variables set in a reference playbook to carry over into the parent playbook? I rather not put a group of steps I need in the parent if I can avoid it, as I am using the child playbook as an action itself, so would it duplicate the functions?

A: In general, variables set in child playbooks do not carry over to the parent playbook. The one exception is that the Reference a Playbook step will return (in vars.result) the return value of the last executed step in the child playbook. For instance, if the last step in the child playbook is Find Record, then the Reference a Playbook step will populate vars.result with the records that have been found using the Find Record step.

If u want to define the playbooks result as a combination of results of previous steps or sub-steps, you can use the Set Variable step at the end of the playbook and define variables that would contain data that you require to be returned.

Q: How do I convert Epoch time returned by a SIEM to a datetime format?

A: If you have a playbook, which has a connector step that connects to a SIEM, such as ArcSight or QRadar, and the SIEM returns the result in Epoch time (milliseconds), then you can convert Epoch time to the datetime format using the following command:
# arrow.get(1531937147932/1000).to(‘Required Timezone’).strftime(“%Y-%m-%d %H:%M:%S %Z%z”)
or
# arrow.get(1531937147932/1000).to(‘Required’).format(‘YYYY-MM-DD HH:mm:ss ZZ’)
For example,
# arrow.get(1531937147932/1000).to(‘EST’).format(‘YYYY-MM-DD HH:mm:ss ZZ’)
Will return the following output:
2018-07-18 14:05:47 EDT-0400
For more examples on dates and times used in Python, see http://arrow.readthedocs.io/en/latest/.

Q: How do I change the timeout limit for playbooks?

A: To change the time limit or soft time limit for playbooks you must edit the CELERYD_TASK_TIME_LIMIT and CELERYD_TASK_SOFT_TIME_LIMIT parameters in the /opt/cyops-workflow/sealab/sealab/config.ini file. By default, these parameters are set in seconds, as follows:
CELERYD_TASK_TIME_LIMIT = 2400
CELERYD_TASK_SOFT_TIME_LIMIT = 1800
Once you have made the change you must restart all the FortiSOAR services by using csadm and running the following command as a root user: :
# csadm services --restart