Miscellaneous Maintenance Operations

• Restarting Primary Leader

• Restarting Primary Follower

• Restarting a Process on any Supervisor

• Restarting Workers

• Changing IP on Primary Leader

• Changing IP on Primary Follower

• Changing IP on Secondary Supervisor

Restarting Primary Leader

This should be avoided as much as possible, since the Leader contains the Master PostGreSQL database. App Server or any other modules on the Leader can be restarted normally.

1. Check Cluster health to make sure that Leader and Follower health are good and replication is up to date.

2. Make sure users are logged out.

3. Shutdown or Reboot the Leader node.

4. After Leader is up (that is, all processes are up), then users can login to the Leader.

5. Log on to each Follower and restart the App Server process on that node. After the App Server process is up on a Follower, users can log on to the Follower node.

Restarting Primary Follower

This can be done normally. The user can login to any other Supervisor node. While the Follower node is down, the local PostGreSQL database will fall behind. If the Follower node comes back up within a reasonable time period, then PostGreSQL database replication will catch up and the system will become normal.

Restarting a Process on any Supervisor

For any process other than DB Server on the Primary Leader, this can be done normally, and the Cluster should be up and users should be able to login. To restart the DB Server on Primary Leader, follow these steps:

1. Check Cluster health to make sure that Leader and Follower health are good and replication is up to date.

2. Make sure users are logged out.

3. Restart the DB Server on the Leader node.

4. After the DB Server on the Leader is up, restart the App Server. After both App Server and DB Server are up, then users can login to the Leader.

5. Log on to each Follower and restart the App Server process on that node. After the App Server process is up on a Follower, users can log on to the Follower node.

Restarting Workers

This can be done normally.

Changing IP on Primary Leader

The Supervisor’s IP address is present in FortiSIEM CMDB as well as in Linux files. Additionally, the change needs to be propagated to the Follower Supervisor, otherwise the replication process will fail.

Follow these steps to complete the Changing IP process.

1. SSH to Primary Leader Supervisor as root.

2. Run configFSM.py.

   1. Choose 1 Supervisor.

   2. Choose 5 change_network_config.

   3. Choose 1 IPv4.

3. After the script finishes, you will see the following message:
   

   Operation change_ip completed successfully, Waiting For Reboot

   

4. The message is a bit misleading since the Supervisor will not reboot. To validate that the change IP task is completed, Login to GUI using the new IP.

5. Click on the Jobs/Errors icon in the upper right area of the GUI.

6. In the Search field, enter "Super Leader IP Change" to search for this job.
   

   In the Status column, the job must be Done, and in the Progress column, the progress must be 100%.

   

7. Some special steps are needed for ClickHouse environment since ClickHouse configuration files also contain the IP address. The steps are mentioned in Handling ClickHouse Node IP Change, located in the Online Help Appendix.
   

Changing IP on Primary Follower

The Supervisor’s IP address is present in FortiSIEM CMDB as well as in Linux files. Additionally, the change needs to be propagated to the Follower Supervisor, otherwise the replication process will fail.

Follow these steps to complete the Changing IP process.

1. SSH to Primary Leader Supervisor as root.

2. Run configFSM.py.

   1. Choose 1 Supervisor.

   2. Choose 5 change_network_config.

   3. Choose 1 IPv4.

3. After the script finishes, you will see the following message:
   

   Operation change_ip completed successfully, Waiting For Reboot

   

4. The message is a bit misleading since the Supervisor will not reboot. To validate that the change IP task is completed, Login to GUI using the new IP.

5. Click on the Jobs/Errors icon in the upper right area of the GUI.

6. In the Search field, enter "Super Follower IP Change" to search for this job.
   

   In the Status column, the job must be Done, and in the Progress column, the progress must be 100%.
   

   
   

7. Some special steps are needed for ClickHouse environment since ClickHouse configuration files also contain the IP address. The steps are mentioned in Handling ClickHouse Node IP Change, located in the Online Help Appendix.

Changing IP on Secondary Supervisor

If, for any reason, you need to change the IP address on the Secondary (Site 2) for Disaster Recovery, take the following steps:

1. Turn off Disaster Recovery, by following the instructions in Turning Off Disaster Recovery.

2. Change the IP of your Secondary (Site 2).

3. Re-enable Disaster Recovery by following the instructions in Configuring Disaster Recovery.