Upgrading a FortiSOAR Distributed Multi-Tenancy Configuration

This section describes the procedure to upgrade a FortiSOAR distributed multi-tenant configuration for managed security services providers (MSSPs) or Distributed SOC configuration.

You must first upgrade the master node of your FortiSOAR distributed multi-tenant configuration and only then upgrade the tenant nodes of your FortiSOAR multi-tenancy setup.

In case of a distributed deployment, both the master and the tenant nodes must be upgraded. A version mismatch will not work if either of them upgrades to 7.4.3.

In release 7.4.2, the default behavior of the On Create, On Update, and On Delete playbooks is changed to run only on the instance where the record is created. If you want to retain the previous behavior for these playbooks, see the Post-Upgrade Tasks chapter.

Upgrading a FortiSOAR master node

Before you upgrade your FortiSOAR master node, ensure the following:

• All playbooks have completed their execution on the master.
• The tenant node(s) are deactivated from the master node before upgrading the master node, and tenant nodes have disabled communication to the master node from the "Master Configuration" page.

If the master node of your multi-tenant configuration is part of an HA setup, i.e., MSSP +HA, then follow the steps mentioned in the Upgrading a FortiSOAR High Availability Cluster chapter.

If the master node of your multi-tenant configuration is not part of an HA setup, then follow the steps mentioned in the Upgrading a FortiSOAR Enterprise Instance chapter.

Upgrading a FortiSOAR Tenant node

Before you upgrade your FortiSOAR tenant node, ensure the following:

• Data replication from the tenant node to the master node is stopped. You can stop data replication by logging on to the tenant node and clicking Settings to open the System page, then in the Multi Tenancy section, click the Master Configuration menu item and then in the Communication With Master Node section, toggle the Enabled button to NO.
  Once you have completed the upgrade process, i.e., upgrading both your master and tenant nodes from a release prior to 7.4.2 to release 7.4.2 or later, you must restart the cyops-integrations-agent service on tenant nodes using the following command:
  systemctl restart cyops-integrations-agent
  You must restart the cyops-integrations-agent service at tenant nodes before you download the agent's or tenant's logs, from the master node's console..
• All playbooks have completed their execution on the tenant.
• All schedule playbooks that fetch data from data sources to the tenant are stopped.
• Any application that pushes data from data sources to the tenant is stopped.

If the tenant node of your multi-tenant configuration is part of an HA setup, i.e., MSSP +HA, then follow the steps mentioned in the Upgrading a FortiSOAR High Availability Cluster section.

If the tenant node of your multi-tenant configuration is not part of an HA setup, then follow the steps mentioned in the Upgrading a FortiSOAR Enterprise Instance section.

After the tenant node has been successfully upgraded, you must toggle the Allow Module Management setting to NO and then back to YES. This is needed only if you were already using the 'Allow Module Management' feature and is required to synchronize the tenant module metadata with the master instance. You can ignore this step, if your 'Allow Module Management' setting was already disabled before the upgrade.

Upgrading a FortiSOAR Secure Message Exchange

A secure message exchange establishes a secure channel that is used to relay information to the agents or tenant nodes. To create a dedicated secure channel, you are required to add the reference of the installed and configured secure message exchange, when you add agent or tenant nodes to your environment. For information on agents see the Segmented Network support in FortiSOAR chapter in the "Administration Guide," and for more information on secure message exchange and tenants, see the "Multi-Tenancy support in FortiSOAR Guide".

1. Ensure that you stop data replication between the master and the tenant nodes. You can stop data replication by logging on to the tenant node and clicking Settings to open the System page, then in the Multi Tenancy section, click the Master Configuration menu item and then in the Communication With Master Node section, toggle the Enabled button to NO.
2. Upgrade the SSH to the secure message exchange VM that you want to upgrade.
3. Check that you are connected to a tmux session. A tmux session is needed for situations where network connectivity is less than favorable. You can check your tmux session using the following command:
   # tmux ls
   This command returns an output such as the following example:
   0: 1 windows (created Thu Nov 24 09:37:47 2022) [170x47]
   Log back into the SSH console and run the following command to reattach the tmux session:
   tmux attach-session -t 0
   
4. Run the following command to download the upgrade installer:
   # wget https://repo.fortisoar.fortinet.com/7.4.3/upgrade-fortisoar-7.4.3.bin
5. Run the upgrade installer using the following command:
   # sh upgrade-fortisoar-7.4.3.bin
   OR
   # chmod +x upgrade-fortisoar-7.4.3.bin
# ./upgrade-fortisoar-7.4.3.bin
   Notes: The FortiSOAR upgrade installer checks for the following:
   • The space available in /tmp or /var/temp (if exist in /etc/fstab), which must be at least 2GB. If the space available in in /tmp or /var/temp is less than 2GB, then the upgrade installer exits after displaying an appropriate error message. If you want to skip this space validation check, you can use the --skip-tmp-validation option while running the upgrade script:
     # ./upgrade-fortisoar-7.4.3.bin --skip-tmp-validation
   • The disk space available in /boot, and if the /boot has insufficient space, then the upgrade installer exits after displaying an appropriate error message. Use standard methods to remove old kernels and free up space in /boot.
   Once you complete cleaning up /boot and/or increasing disk space and space in /tmp (as per the messages provided by the upgrade installer) and you have sufficient space for upgrading FortiSOAR, you must re-run the upgrade installer to continue the process of upgrading FortiSOAR.
6. Once you have successfully upgraded the secure message exchange, start the data replication between the master and the tenant nodes again by toggling the Data Replication button to ON, and then verify the replication.

Upgrading a FortiSOAR Secure Message Exchange Cluster

RabbitMQ supports clustering, that in conjunction with Queue Mirroring can be used for an Active-Active configuration as explained in the Clustering Guide and in the Highly Available (Mirrored) Queues article, which includes steps on how to set up the clusters and monitor queues. The clustered instances should be fronted by a TCP Load Balancer such as HAProxy, and clients should connect to the cluster using the address of the proxy. For more information, see the Multi-tenancy support in FortiSOAR guide.

For the purpose of the following procedure, we are considering a two-node MQ mirrored queue clusters that are both added to the Reverse Proxy.

1. Configure the Reverse Proxy to pass requests only to Node1, which is the primary node of the MQ cluster.
   Therefore, now all requests will be handled by Node1 and Node2 will be available for maintenance.
2. Log on to the Node2 terminal session as a root user, and upgrade Node2 by following the steps mentioned in the Upgrading a FortiSOAR Secure Message Exchange section.
3. Configure the Reverse Proxy to route requests through Node2. Therefore, now all requests will be handled by Node2 and Node1 will be available for maintenance.
4. Login to Node1, and upgrade Node1 as per the procedure mentioned in step 2.
5. Reconfigure the Reverse Proxy to load balance both Node1 and Node2.

Troubleshooting upgrade issues for MSSP setups

Replication from tenant to master stops once you upgrade an MSSP with an HA setup

If you have upgraded an MSSP+HA setup, then post-upgrade the replication from tenant nodes to the master node stopped.

Resolution

To resolve this issue, once you have upgraded your MSSP setup and created the HA cluster, you must restart all services on the primary master node and the primary tenant node using the following command:
csadm services --restart