Fortinet black logo

Administration Guide

Backing up and Restoring FortiSOAR

Copy Link
Copy Doc ID db0133d0-f2c6-11ec-bb32-fa163e15d75b:804260
Download PDF

Backing up and Restoring FortiSOAR

This chapter describes the process of backing up and restoring FortiSOAR, whether or not you have not externalized your PostgreSQL database.

Prerequisites

You must have the root or sudo permissions to perform backup and restore.

Note

Ensure that you have enough disk space available to perform backup and restore tasks. It is recommended that you have available disk space of around 3X of the data size, for example, if your data size is 2GB, then you should have around 6GB of available disk space, to ensure that the processes do not stop or fail.

Backup Process

Use the FortiSOAR Admin CLI (csadm) db subcommand to regularly perform backups and restore, which restores the data seamlessly to a new FortiSOAR environment. To perform backup and restore, you must have root access on your FortiSOAR system. For more information on csadm, see the FortiSOAR Admin CLI chapter.

The FortiSOAR Admin CLI performs a full database backup of your FortiSOAR server each time. There is no provision of incremental backups. Backups are performed for a particular version of FortiSOAR, and backups should be restored on the exact versions of FortiSOAR. If a newer version of FortiSOAR is available and you want to move to that newer version of FortiSOAR, you must restore the backed-up version only and then upgrade to the latest FortiSOAR version. This is to ensure that all the new changes will be present.

Tooltip

The FortiSOAR Admin CLI backs up the latest three backups every time it creates a new backup. Any backups older than the latest three backups are deleted.

Data that is backed up during the backup process

The FortiSOAR Admin CLI backs up the following files, configurations, and data during the backup process:

  • site-packages
  • connectors
  • playbooks
  • application.conf
  • db_config.yml
  • pg_hba.conf
  • Syslog forwarding configuration
  • All major configuration files such as das.ini, postgresql.conf
  • PostgreSQL database backups as per requirements
  • User-defined custom expressions
Note

Backup of the configuration files are taken only in case of localized databases.

Prerequisites to running the backup process

You must have the NFS or local backup storage path.

Performing a backup

To perform a backup run the csadm command on any FortiSOAR machine using any terminal. A user who has root or sudo permissions can run the csadm command.

  1. SSH to your FortiSOAR VM and login as a root user.
  2. To perform a backup, type the following command:
    # csadm db --backup [<backup_dir_path>]
    [<backup_dir_path>] is the directory where backup files will be created. If you do not specify any path, then by default, the backup file is stored in the current working directory.
    Optionally, from version 6.4.3 onwards, you can specify the --exclude-workflow argument to exclude all the "Executed Playbook Logs" from the backup. Executed playbook logs are primarily meant for debugging so they are not a very critical component to be backed up. However, they constitute a major part of the database size, so excluding them from the backup reduces time and space needed for the backup. To exclude all the "Executed Playbook Logs" from the backup, type the command as follows:
    # csadm db --backup [<backup_dir_path>] --exclude-workflow
    Important: FortiSOAR backs up the latest three backups every time it creates a new backup. Any backups older than the latest three backups are deleted.
  3. (Optional) If you only want to backup only your configuration files, then type the following command:
    # csadm db --backup-config [<backup_dir_path>]
    Once you run the above command, you will be asked to provide the path of the configuration backup file. If you do not specify any path, then by default, the backup file is stored in the current working directory.

Running a backup as a scheduled job

Following is an example of running a backup as a scheduled cron job, on your FortiSOAR system or external Secure Message Exchange, that will run at 12:30 am every day. You can schedule the backup process based on your requirements.

Add the cron job to run at 12:30 am every day as follows:

$ sudo crontab –e
30 00 * * * csadm db --backup <backup_dir_path>

Once the backup process is successfully completed, the final DR_BACKUP_<FortiSOAR_version>_timestamp.tgz file is located in the directory where the backup files are created. It would be the same directory that you have specified when you ran the csadm db --backup <backup_dir_path> command. The DR_BACKUP_<FortiSOAR_version>_timestamp.tgz file includes the timestamp on when the backup is created.

The DR_BACKUP_<FortiSOAR_version>_timestamp.tgz file includes all the backup files. You can run the following command to check the contents of the DR_BACKUP_<FortiSOAR_version>_timestamp.tgz file :
# tar -tvf <DR_BACKUP_<FortiSOAR_version>_timestamp.tgz>

Restore Process

To restore the data on a new FortiSOAR server run the csadm command on any FortiSOAR machine using any terminal. A user who has root or sudo permissions can run the csadm.

Restoring data

  1. Move the backup file to the new FortiSOAR VM.
  2. SSH to the new FortiSOAR VM and login as a root user.
  3. To restore the data, type the following command:
    # csadm db --restore <backup_file_path>
    [<backup_file_path>] is the directory where you have saved the backed up files. Note that the backup process, by default stores the backup in a locally saved file: /home/csadmin/db_backup/DR_BACKUP_<yyyymmdd_hhmmss>.tgz
    Important: Once you have restored FortiSOAR, you are required to get and deploy a new license for this FortiSOAR instance. Your existing license will not work on the restored instance. For the procedure to get a new license, see the Licensing FortiSOAR chapter in the "Deployment Guide."

Notes for the Restore process

  • If you have backed up a FortiSOAR instance that has a Secure Message Exchange enabled and is using a signed certificate, then you would need to re-apply the signed certificate on the new instance. For steps on how to replace certificates, see the Replacing the self-signed certificates on the secure message exchange with signed certificates topic in the "Multi-tenancy support in FortiSOAR Guide."
    If the restore is done on a machine that has a different FQHN, you will need to update the master FQHN at any tenant nodes or agents, which are connected through this secure message exchange. You can either update the tenant nodes or agents manually on the respective remote nodes or download the installer file again from the master and apply it on the remote nodes. For more information, see the Segmented Network support in FortiSOAR chapter.
  • If you have backed up only the configuration, using the csadm db --backup-config command, of your FortiSOAR instance that has an externalized database, then once you have restored FortiSOAR on another instance, you need to run the following commands to point FortiSOAR to the external database:
    sudo -u nginx php /opt/cyops-api/bin/console cache:clear
    systemctl restart php-fpm
    For information on database externalization, see the Externalization of your FortiSOAR PostgreSQL database chapter.

Backup and Restore process for FortiSOAR High Availability systems

You can backup and restore your High Availability (HA) systems, if for example, your primary database gets corrupted, using the following steps:

  1. Configure backup on your primary database. Steps for which are mentioned in the Backup Process section.
  2. Remove the secondary nodes from the HA cluster using the csadm ha leave-cluster command. For more information on the HA and the csadm ha CLI, see the High Availability support in FortiSOAR chapter.
  3. Restore the primary database. Steps for which are mentioned in the Restore Process section.
  4. Add the secondary nodes back to the HA cluster using the csadm ha join-cluster command.

Regenerating RabbitMQ certificates when you are creating an HA cluster using a restored node

In addition to the above steps, if you are creating an HA cluster after you have taken a backup of an enterprise node, say T1 and restored this backup on another enterprise node, say T2, you also need to regenerate RabbitMQ certificates on the restored node. This is required as FortiSOAR restores the TLS certificate of T1 on the T2 node, causing hostname validation failure when you try to join another node to the T2 node in a HA cluster. Therefore, once you have restored your backup of the T1 node to the T2 node, then you must regenerate MQ certificates on the T2 node using the following command:
csadm mq certs --generate

Backup and Restore process for the External Secure Message Exchange systems

From version 7.0.0 onwards, you can also backup and restore the data of your external Secure Message Exchange (SME) system, by using the following commands:

  • csadm db --backup [<backup_dir_path>]: Performs a backup of your external SME system. Steps for which are mentioned in the Backup Process section.
    Note: The --exclude-workflow argument is not applicable to the external SME.

  • csadm db --restore [<backup_file_path>: Performs data restore for your external SME system from a locally stored file, whose path you have specified. The default location of the backup file is (/home/csadmin/db_backup/DR_BACKUP_<yyyymmdd_hhmmss>.tgz). Steps for which are mentioned in the Restore Process section.

In case of your configured MSSP setup, if you have taken a backup of your external SME instance 'A' and restored that backup on instance 'B', then in case of environments that use signed certificates you should not have any issues since the CLI copies over the certificates from the backed up to the restored node. However, if the certificates are the default self-signed certificates that are auto-generated by the platform, then these are not restored as they are tied to the hostname, and the hostname of the node 'B' might be different from node 'A'. So if you have a development environment with self-signed certs, after the SME is restored to the new node 'B', you have to perform different set of steps when the restored SME hostname is same and when it is different. Steps for both these scenarios follow:

Steps to be run on the external SME after it is restored when the external SME hostname is the same post restore:

  1. Replace the certificate on the external SME with the certificate copied from <Extracted Backup Folder Name>/secure_message_exchange/ssl to cp -rp ssl/ /opt/cyops/configs/rabbitmq/ssl
  2. Restart the services on the external SME using the following command: csadm services --restart
  3. Restart the cyops-postman service on the tenant node.
  4. Restart the csagent service on SNS Agents using the following command: csagent --option services --action restart

Steps to be run on the external SME after it is restored when the external SME hostname is changed post restore:

  1. Update the certificate content on the master node's external SME. Path from where you need to take the certificate's content is /opt/cyops/configs/rabbitmq/ssl/cyopsca/cacert.pem
  2. On the master node, export and download the configuration of the tenant node.
  3. Import the same configuration (json) file on each tenant node to setup replication.

Troubleshooting backup and Restore Issues

Post-restore the FSR agent status is not updated as "Remote Node Unreachable"

When you have restored FSR agents using a backup with agents and self-SME, the agent status does not get updated as "Remote Node Unreachable".

Resolution

  1. Navigate to the FSR agent and disable the FSR agent and then enable it again to reflect the correct connectivity status of the FSR agent, which is "Remote Node Unreachable".
  2. Ensure that the restored node is resolvable from the FSR agent system.
  3. Click the Download Installer link for each FSR agent with the Include Pre-existing Connectors on Agent checkbox selected.
  4. Install the downloaded bin file, for example <agent-name>-install.bin, on the FSR agent.
  5. Restart the cyops-integrations-agent service on the FSR agent.
    Now check the FSR agent status; it should display as "Available".

Backing up and Restoring FortiSOAR

This chapter describes the process of backing up and restoring FortiSOAR, whether or not you have not externalized your PostgreSQL database.

Prerequisites

You must have the root or sudo permissions to perform backup and restore.

Note

Ensure that you have enough disk space available to perform backup and restore tasks. It is recommended that you have available disk space of around 3X of the data size, for example, if your data size is 2GB, then you should have around 6GB of available disk space, to ensure that the processes do not stop or fail.

Backup Process

Use the FortiSOAR Admin CLI (csadm) db subcommand to regularly perform backups and restore, which restores the data seamlessly to a new FortiSOAR environment. To perform backup and restore, you must have root access on your FortiSOAR system. For more information on csadm, see the FortiSOAR Admin CLI chapter.

The FortiSOAR Admin CLI performs a full database backup of your FortiSOAR server each time. There is no provision of incremental backups. Backups are performed for a particular version of FortiSOAR, and backups should be restored on the exact versions of FortiSOAR. If a newer version of FortiSOAR is available and you want to move to that newer version of FortiSOAR, you must restore the backed-up version only and then upgrade to the latest FortiSOAR version. This is to ensure that all the new changes will be present.

Tooltip

The FortiSOAR Admin CLI backs up the latest three backups every time it creates a new backup. Any backups older than the latest three backups are deleted.

Data that is backed up during the backup process

The FortiSOAR Admin CLI backs up the following files, configurations, and data during the backup process:

  • site-packages
  • connectors
  • playbooks
  • application.conf
  • db_config.yml
  • pg_hba.conf
  • Syslog forwarding configuration
  • All major configuration files such as das.ini, postgresql.conf
  • PostgreSQL database backups as per requirements
  • User-defined custom expressions
Note

Backup of the configuration files are taken only in case of localized databases.

Prerequisites to running the backup process

You must have the NFS or local backup storage path.

Performing a backup

To perform a backup run the csadm command on any FortiSOAR machine using any terminal. A user who has root or sudo permissions can run the csadm command.

  1. SSH to your FortiSOAR VM and login as a root user.
  2. To perform a backup, type the following command:
    # csadm db --backup [<backup_dir_path>]
    [<backup_dir_path>] is the directory where backup files will be created. If you do not specify any path, then by default, the backup file is stored in the current working directory.
    Optionally, from version 6.4.3 onwards, you can specify the --exclude-workflow argument to exclude all the "Executed Playbook Logs" from the backup. Executed playbook logs are primarily meant for debugging so they are not a very critical component to be backed up. However, they constitute a major part of the database size, so excluding them from the backup reduces time and space needed for the backup. To exclude all the "Executed Playbook Logs" from the backup, type the command as follows:
    # csadm db --backup [<backup_dir_path>] --exclude-workflow
    Important: FortiSOAR backs up the latest three backups every time it creates a new backup. Any backups older than the latest three backups are deleted.
  3. (Optional) If you only want to backup only your configuration files, then type the following command:
    # csadm db --backup-config [<backup_dir_path>]
    Once you run the above command, you will be asked to provide the path of the configuration backup file. If you do not specify any path, then by default, the backup file is stored in the current working directory.

Running a backup as a scheduled job

Following is an example of running a backup as a scheduled cron job, on your FortiSOAR system or external Secure Message Exchange, that will run at 12:30 am every day. You can schedule the backup process based on your requirements.

Add the cron job to run at 12:30 am every day as follows:

$ sudo crontab –e
30 00 * * * csadm db --backup <backup_dir_path>

Once the backup process is successfully completed, the final DR_BACKUP_<FortiSOAR_version>_timestamp.tgz file is located in the directory where the backup files are created. It would be the same directory that you have specified when you ran the csadm db --backup <backup_dir_path> command. The DR_BACKUP_<FortiSOAR_version>_timestamp.tgz file includes the timestamp on when the backup is created.

The DR_BACKUP_<FortiSOAR_version>_timestamp.tgz file includes all the backup files. You can run the following command to check the contents of the DR_BACKUP_<FortiSOAR_version>_timestamp.tgz file :
# tar -tvf <DR_BACKUP_<FortiSOAR_version>_timestamp.tgz>

Restore Process

To restore the data on a new FortiSOAR server run the csadm command on any FortiSOAR machine using any terminal. A user who has root or sudo permissions can run the csadm.

Restoring data

  1. Move the backup file to the new FortiSOAR VM.
  2. SSH to the new FortiSOAR VM and login as a root user.
  3. To restore the data, type the following command:
    # csadm db --restore <backup_file_path>
    [<backup_file_path>] is the directory where you have saved the backed up files. Note that the backup process, by default stores the backup in a locally saved file: /home/csadmin/db_backup/DR_BACKUP_<yyyymmdd_hhmmss>.tgz
    Important: Once you have restored FortiSOAR, you are required to get and deploy a new license for this FortiSOAR instance. Your existing license will not work on the restored instance. For the procedure to get a new license, see the Licensing FortiSOAR chapter in the "Deployment Guide."

Notes for the Restore process

  • If you have backed up a FortiSOAR instance that has a Secure Message Exchange enabled and is using a signed certificate, then you would need to re-apply the signed certificate on the new instance. For steps on how to replace certificates, see the Replacing the self-signed certificates on the secure message exchange with signed certificates topic in the "Multi-tenancy support in FortiSOAR Guide."
    If the restore is done on a machine that has a different FQHN, you will need to update the master FQHN at any tenant nodes or agents, which are connected through this secure message exchange. You can either update the tenant nodes or agents manually on the respective remote nodes or download the installer file again from the master and apply it on the remote nodes. For more information, see the Segmented Network support in FortiSOAR chapter.
  • If you have backed up only the configuration, using the csadm db --backup-config command, of your FortiSOAR instance that has an externalized database, then once you have restored FortiSOAR on another instance, you need to run the following commands to point FortiSOAR to the external database:
    sudo -u nginx php /opt/cyops-api/bin/console cache:clear
    systemctl restart php-fpm
    For information on database externalization, see the Externalization of your FortiSOAR PostgreSQL database chapter.

Backup and Restore process for FortiSOAR High Availability systems

You can backup and restore your High Availability (HA) systems, if for example, your primary database gets corrupted, using the following steps:

  1. Configure backup on your primary database. Steps for which are mentioned in the Backup Process section.
  2. Remove the secondary nodes from the HA cluster using the csadm ha leave-cluster command. For more information on the HA and the csadm ha CLI, see the High Availability support in FortiSOAR chapter.
  3. Restore the primary database. Steps for which are mentioned in the Restore Process section.
  4. Add the secondary nodes back to the HA cluster using the csadm ha join-cluster command.

Regenerating RabbitMQ certificates when you are creating an HA cluster using a restored node

In addition to the above steps, if you are creating an HA cluster after you have taken a backup of an enterprise node, say T1 and restored this backup on another enterprise node, say T2, you also need to regenerate RabbitMQ certificates on the restored node. This is required as FortiSOAR restores the TLS certificate of T1 on the T2 node, causing hostname validation failure when you try to join another node to the T2 node in a HA cluster. Therefore, once you have restored your backup of the T1 node to the T2 node, then you must regenerate MQ certificates on the T2 node using the following command:
csadm mq certs --generate

Backup and Restore process for the External Secure Message Exchange systems

From version 7.0.0 onwards, you can also backup and restore the data of your external Secure Message Exchange (SME) system, by using the following commands:

  • csadm db --backup [<backup_dir_path>]: Performs a backup of your external SME system. Steps for which are mentioned in the Backup Process section.
    Note: The --exclude-workflow argument is not applicable to the external SME.

  • csadm db --restore [<backup_file_path>: Performs data restore for your external SME system from a locally stored file, whose path you have specified. The default location of the backup file is (/home/csadmin/db_backup/DR_BACKUP_<yyyymmdd_hhmmss>.tgz). Steps for which are mentioned in the Restore Process section.

In case of your configured MSSP setup, if you have taken a backup of your external SME instance 'A' and restored that backup on instance 'B', then in case of environments that use signed certificates you should not have any issues since the CLI copies over the certificates from the backed up to the restored node. However, if the certificates are the default self-signed certificates that are auto-generated by the platform, then these are not restored as they are tied to the hostname, and the hostname of the node 'B' might be different from node 'A'. So if you have a development environment with self-signed certs, after the SME is restored to the new node 'B', you have to perform different set of steps when the restored SME hostname is same and when it is different. Steps for both these scenarios follow:

Steps to be run on the external SME after it is restored when the external SME hostname is the same post restore:

  1. Replace the certificate on the external SME with the certificate copied from <Extracted Backup Folder Name>/secure_message_exchange/ssl to cp -rp ssl/ /opt/cyops/configs/rabbitmq/ssl
  2. Restart the services on the external SME using the following command: csadm services --restart
  3. Restart the cyops-postman service on the tenant node.
  4. Restart the csagent service on SNS Agents using the following command: csagent --option services --action restart

Steps to be run on the external SME after it is restored when the external SME hostname is changed post restore:

  1. Update the certificate content on the master node's external SME. Path from where you need to take the certificate's content is /opt/cyops/configs/rabbitmq/ssl/cyopsca/cacert.pem
  2. On the master node, export and download the configuration of the tenant node.
  3. Import the same configuration (json) file on each tenant node to setup replication.

Troubleshooting backup and Restore Issues

Post-restore the FSR agent status is not updated as "Remote Node Unreachable"

When you have restored FSR agents using a backup with agents and self-SME, the agent status does not get updated as "Remote Node Unreachable".

Resolution

  1. Navigate to the FSR agent and disable the FSR agent and then enable it again to reflect the correct connectivity status of the FSR agent, which is "Remote Node Unreachable".
  2. Ensure that the restored node is resolvable from the FSR agent system.
  3. Click the Download Installer link for each FSR agent with the Include Pre-existing Connectors on Agent checkbox selected.
  4. Install the downloaded bin file, for example <agent-name>-install.bin, on the FSR agent.
  5. Restart the cyops-integrations-agent service on the FSR agent.
    Now check the FSR agent status; it should display as "Available".