Upgrading a FortiSIEM Cluster Deployment

• Overview
• Upgrade Supervisor
• Upgrade Worker
• Migrating Elasticsearch data from 5.2.1 or earlier to 5.2.6
• Upgrade Report Server
• Upgrade the Collector Image From the Supervisor
• Troubleshooting a FortiSIEM Upgrade

Overview

Follow these steps while upgrading a VA cluster.

1. Shut down all Workers. Collectors can be up and running.
2. Upgrade Super first (while all workers are shutdown).
3. After Super is up and running, upgrade worker one by one.
4. Upgrade Collectors.

Step #1 prevents the accumulation of Report files while Super is not available during upgrade (#2). If these steps are not followed, Supervisor may not be able to come up after upgrade because of excessive unprocessed report file accumulation.

Note: Both Super and Worker MUST be on the same FortiSIEM version, else various software modules may not work properly. However, Collectors can be in an older version (one version older) - they will work except that they may not have the latest discovery and performance monitoring features in the Super/Worker versions. So FortiSIEM recommends that you also upgrade Collectors within a short period of time. If you have Collectors in your deployment, make sure you have configured an image server to use as a repository for the Collector.

Upgrade Supervisor

Supervisor must be upgraded first, before Workers and Collectors and Report Server.

1. Download the image from the Fortinet Support website https://support.fortinet.com. See "Downloading FortiSIEM Products" for more information on downloading products from the support website.
2. Unzip the zip file to get the tar file.
3. Copy the .tar file to the Supervisor:
   1. Copy the va-5.2.6.1623.tar file, using SCP (for example), from your system to the Supervisor.
   2. Make sure this file is in a directory named 5.2.6.1623.
4. Using SSH, log in to the FortiSIEM virtual appliance as the root user. To avoid issues with SSH connection timeouts, disconnects etc., run the upgrade in screen using the following command:

   screen -S upgrade

   To connect the screen after failure:

   run screen –r

5. Run the phdownloadimage script and point to your directory:
   

   # cd /pbin
# ./phdownloadimage file:///root/5.2.6.1623

6. Run the phupgradeimage script to actually upgrade:

   # cd /pbin
# ./phupgradeimage

Upgrade Worker

Workers must be upgraded after Super.

1. Download the image from Fortinet Support Site to your system and unzip to get the tar file.
2. Copy the .tar file to the Worker:
   1. Copy the va-5.2.6.1623.tar file, using SCP (for example), from your system to the Worker.
   2. Make sure this file is in a directory named 5.2.6.1623.
3. Using SSH, log in to the FortiSIEM virtual appliance as the root user. To avoid issues with SSH connection timeouts, disconnects etc., run the upgrade in screen using the following command:

   screen -S upgrade

   To connect the screen after failure:

   run screen –r

4. Run the phdownloadimage script and point to your directory:

   # cd /pbin
# ./phdownloadimage file:///root/5.2.6.1623

5. Run the phupgradeimage script:

   # cd /pbin
# ./phupgradeimage

Migrating Elasticsearch data from 5.2.1 or earlier to 5.2.6

In 5.2.4, Elasticsearch query behavior changed from case-sensitive to case-insensitive. Therefore, Elasticsearch event data format has changed. After upgrade, data will be written in the new format starting new day UTC time. FortiSIEM can only query data in the new format. For existing customers that are already running Elasticsearch, older data must be re-indexed for searches to work, after upgrading to 5.2.6. Exact steps are as follows. It is advisable to start the upgrade with a few hours to go before new day in UTC time. Here is a PST example: a new day in UTC time format begins at 5pm PST. The customer can begin the upgrade at 12 PM PST.

1. Upgrade FortiSIEM Supervisor and Workers to 5.2.6.
2. Go to Admin > Setup > Storage. Click Test and Save.
3. Re-index earlier days – do not re-index today's data as new data is being written.
4. After a new day in UTC time, re-index yesterday’s index. See Re-indexing.
5. Delete all old indices. See Delete old index.
6. Create an alias. See Creating alias.

Data will be queryable after steps 4 and 6 are complete.

Re-indexing:

curl -X POST "X.X.X.X:9200/_reindex" -H 'Content-Type: application/json' -d'
{
"source": {
"index": "fortisiem-event-2019.04.22"
},
"dest": {
"index": "fortisiem-event-upgrade-2019.04.22"
}
}'

Delete old index:

curl -XDELETE http://X.X.X.X:9200/fortisiem-event-2019.04.22

Creating alias:

curl -X POST "X.X.X.X:9200/_aliases" -H 'Content-Type: application/json' -d'
{
"actions" : [
{ "add" : { "index" : "fortisiem-event-upgrade-2019.04.22", "alias" : "fortisiem-event-2019.04.22" } }
]
}
'

Upgrade Report Server

Complete the following steps to upgrade the Report Server. Because the upgrade is not working properly, you will have to complete additional steps here.

1. Download the files from image server to your system and unzip to get the tar file.
2. Copy the .tar file to the Report Server.
   1. Copy the rs-5.2.6.1623.tar file, using SCP (for example), from your system to the Report Server.
   2. Make sure this file is in a directory named 5.2.6.1623.
3. Using SSH, log in to the FortiSIEM virtual appliance as the root user. To avoid issues with SSH connection timeouts, disconnects etc., run the upgrade in screen using the following command:

   screen -S upgrade

   To connect the screen after failure:

   run screen –r

4. Log in as root to Report Server via SSH and move the tar file to that directory and open the tar file:

   # mkdir /root/5.2.6.1623
# mv rs-5.2.6.1623.tar /root/5.2.6.1623 /
# cd 5.2.6.1623 /
# tar xf rs-5.2.6.1623.tar

5. Obtain the phdownloadimage script. You can do this in either of the following ways:
   1. Upgrade Super to 5.2.6. Then copy the Super’s /pbin/phdownloadimage and replace the Report Server’s /pbin/phdownloadimage script.
   2. Contact Fortinet Support: https://support.fortinet.com.
6. Replace the phdownloadimage script in the /opt/phoenix/deployment/jumpbox folder with the copy you just obtained.
7. Run the phdownloadimage script and point to your directory:

   # cd /pbin
# ./phdownloadimage file:///root/5.2.6.1623
   

8. Run the phupgradeimage script.

   # cd /pbin
# ./phupgradeimage

Report Server upgrade to 5.2.6 is not working properly. If you are running Report Server, then complete these steps to upgrade to 5.2.6:

1. Upgrade Super, Worker, Collector, and Report Server as described above.
2. Archive the Report Server event database. Run this command:
   /opt/phoenix/deployment/reportdb_archiver.sh
3. The report db backup is under /data/archive/reportdb/reportdb_2019-09-09T14-33-26.
4. Delete the Report Server from Super.
5. Add the Report Server back to Super.
6. Restore Report Server event database from Archive. Run this command:
   /opt/phoenix/deployment/reportdb_restore.sh/data/archive/reportdb/reportdb_2019-09-09T14-33-26.
   

Upgrade the Collector Image From the Supervisor

Follow these steps to download the Collector image files from the support site:

1. Download the Collector upgrade file from the Fortinet Support site and copy it to a location on the Supervisor.
2. Check the MD5 checksum with the one published on the Support site to make sure the image is correctly downloaded.
3. Log in to the Supervisor as root user.
4. Check whether the Collector package from a previous upgrade is present in the Supervisor. If it is, delete it.
5. Prepare the upgrade file for Collector download:
   1. Go to /opt/phoenix/phscripts/bin/.
   2. Run the command:
      phSetupCollectorUpgrade.sh <coImageZipFile> <superFQDN/IP>
      where coImageZipFile is the full path of the location of the Collector upgrade file in Step 1 and superFQDN/IP is the FQDN or IP that must be resolvable from Collectors
6. Go to Settings > System > Collector Image Server and make sure that the image download URL is displayed. This value is generated by the system and cannot be edited.
7. Go to ADMIN > Health > Collector Health
   1. Select a Collector and click Action > Download Image. This will cause the Collectors to download the upgrade images from the Supervisor.
   2. Select a Collector and click Action > Install Image. This will cause the Collectors to install the upgrade.

Troubleshooting a FortiSIEM Upgrade

FortiSIEM generates a number of log files to help you diagnose any problems you might encounter during the upgrade process.

Inspect this log file in the /tmp folder:

• dbschemaupgrade_1623x.log

and this log file in the /opt/phoenix/log folder:

• upgrade-populatedb_1623x.log