Fortinet Document Library

Version:

Version:

Version:


Table of Contents

Download PDF
Copy Link

FortiSIEM Disaster Recovery Procedures

This section provides details about setting up and managing FortiSIEM disaster recovery and failover.

Prerequisites

It is recommended to use DNS names for Supervisor to support this operation.

Note: You need two separate FortiSIEM licenses - one for each site.

  1. Create DNS Names for the Supervisor nodes at the two sites, for example:
    • Site1.fortisiem.acme.com
    • Site2.fortisiem.acme.com
  2. Install FortiSIEM on both sites (version 5.2.1 and later).
  3. The FortiSIEM setup at the two sites must be identical. (Note: Collectors are not part of replication as they are deployed close to the devices.)
    1. Number of Workers
    2. Storage type
    3. Archiving setup
    4. Report Server setup
    5. Hardware resources (CPU, Memory, Disk) of various FortiSIEM nodes
  4. Make sure the users and Collector nodes can access both Supervisor nodes by DNS names.
  5. Make sure that the two sites can communicate with each other using HTTPS, SSH and PostgreSQL.
  6. Log in to the Supervisor, go to ADMIN > Settings > Database > Replicate and make sure you have the required information for the two sites.

Set up Disaster Recovery and Failover

Before setting up Disaster Recovery, determine which of the two sites will be the Primary site to start with.

  1. Log in to the current Primary Supervisor node:
    1. Go to ADMIN > Settings > Database > Replicate.
    2. Enter the required information based on the event storage type:
    3. Click Save.
      The Replication process starts and runs in the background. A Job will be created – the status can be seen in the Job window.
    4. Go back to ADMIN > Settings > Database > Replicate page.
    5. Click Export and Save the Replication Configuration file on your local system.
  2. Log in to the current Secondary Supervisor node:
    1. Go to ADMIN > Settings > Database > Replicate.
    2. Select the exported Replication file from the current Primary site stored in your local computer. Click Import.
    3. The Replicate page opens. Make sure the information is correct. Click Save.
      This will continue the Replication process that started with the current Primary. A Job will also be created on the current Secondary – the status can be seen in the Job window.

The Replication process works as follows:

  1. The CMDB of the current Primary will be replicated to current Secondary.
  2. Once step 1 is complete, you will see that the two jobs created in Primary and Secondary will show complete in the Job window.
  3. Other replication will begin and continue until replication status changes:
    1. SVN replication – Secondary will pull in replication files from Primary
    2. Profile DB replication
    3. Local or NFS based replication
    4. Elasticsearch replication

Operating FortiSIEM in Replication mode

  1. Make sure DNS points to current Primary:
    1. DNS: fortisiem.acme.com (Site1.fortisiem.acme.com)
    2. Users log in to fortisiem.acme.com (Site1.fortisiem.acme.com)
    3. Collectors register to fortisiem.acme.com. This will cause the Worker Configuration at Site1.fortisiem.acme.com to be pushed to Collectors.
  2. Events and configurations will be sent to the Primary and replicated to Secondary.
  3. Profile Database and Incidents will be computed in the Primary and replicated to Secondary.
  4. Inline Reports and scheduled reports are not copied over from Primary to Secondary. Incident Notification and Scheduled Report delivery happens on Primary only.
  5. Incidents trigger on Primary and replicates to Secondary.
  6. Archiving and Report Server synching occur independently on the two sites.
  7. All processes will be up on Primary. However, on the Secondary node, only the processes required for user login (App Server, PostGreSQL, Query Master/Worker, Java Query Server) and replication (Data Purger) will be up.
  8. The Secondary site can be operated like a Primary except the events are delayed because of replication (see ADMIN > Settings > Database > Replicate > Replication Frequency).
  9. CMDB is set in a multi-master mode – so any changes on Secondary are replicated over to Primary. It is recommended to do all edits on the Primary site.

Handling Disaster

Assuming that disaster happens at the current Primary Site, make sure DNS points to current Secondary, that is, fortisiem.acme.com points to Site2.fortisiem.acme.com. Users log in to fortisiem.acme.com (now Site2.fortisiem.acme.com).

Log in to the current Secondary Supervisor node (Site2.fortisiem.acme.com) and:

  1. Go to ADMIN > Settings > Database > Replicate.
  2. Switch the roles – set Site2.fortisiem.acme.com as the Primary site.
  3. Click Save.
    A Replication Change will be created in the Secondary and it will finish (Progress 100%). All processes will come up on Supervisor and all Worker nodes.

Collectors will send to the new Primary (Site2.fortisiem.acme.com) as follows:

  1. Collectors will first fail to send to old Workers (part of Site1.fortisiem.acme.com).
  2. Collectors will request a new Worker list from Super (now Site2.fortisiem.acme.com because of DNS change).
  3. Collectors a new list of Workers from Site2.fortisiem.acme.com
  4. Collectors will start sending events to the new Primary FortiSIEM cluster

Handling Recovery

Once the old Primary (Site1.fortisiem.acme.com) recovers, you may want to switch back to that site.

First, make the old Primary (Site1.fortisiem.acme.com) a Secondary.

  1. Log on to Site1.fortisiem.acme.com and make it Secondary and follow these steps:
    1. Go to ADMIN > Settings > Database > Replicate.
    2. Make sure Site1.fortisiem.acme.com is Secondary and Site2.fortisiem.acme.com is Primary. This will happen because the CMDB replication will bring back changes from Site2 (Current Primary) to Site 1 (Current Secondary).
    3. Click Save.
      A Replication Change will be created in the Secondary and it will finish (Progress 100%).
    4. Only the processes required for user login (App Server, PostGreSQL, Query Master/Worker, Java Query Server) and replication (Data Purger) will be up.
    5. Replication will continue – all missing data during disaster will flow back in from Site2 to Site1.

Once Site1.fortisiem.acme.com has come up, make it Primary.

  1. Log in to Site1.fortisiem.acme.com and make it Primary:
    1. Go to ADMIN > Settings > Database > Replicate.
    2. Switch Roles.
    3. Click Save.
  2. Log in to Site2.fortisiem.acme.com and make it Secondary:
    1. Go to ADMIN > Settings > Database > Replicate.
    2. Verify Roles (verify since the changes in Site1 will be replicated).
    3. Click Save.

Disabling Disaster Recovery

If you do not want to enable the Disaster Recovery feature, you can turn off this.

Log in to the current Primary (note that this has to be done first):

  1. Go to ADMIN > Settings > Database > Replicate.
  2. Uncheck Enable Replication to disable Replication.
  3. Click Save and make sure Replication setting task is completed.
    A Replication Job will be created. Make sure that the Job is finished from the Jobs and Errors window.

Log in to the current Secondary site:

  1. Go to ADMIN > Settings > Database > Replicate.
  2. Make sure Replication is disabled.
  3. Click Save.
    A Replication Job will be created. Make sure that the Job is finished from the Jobs and Errors window.

FortiSIEM Disaster Recovery Procedures

This section provides details about setting up and managing FortiSIEM disaster recovery and failover.

Prerequisites

It is recommended to use DNS names for Supervisor to support this operation.

Note: You need two separate FortiSIEM licenses - one for each site.

  1. Create DNS Names for the Supervisor nodes at the two sites, for example:
    • Site1.fortisiem.acme.com
    • Site2.fortisiem.acme.com
  2. Install FortiSIEM on both sites (version 5.2.1 and later).
  3. The FortiSIEM setup at the two sites must be identical. (Note: Collectors are not part of replication as they are deployed close to the devices.)
    1. Number of Workers
    2. Storage type
    3. Archiving setup
    4. Report Server setup
    5. Hardware resources (CPU, Memory, Disk) of various FortiSIEM nodes
  4. Make sure the users and Collector nodes can access both Supervisor nodes by DNS names.
  5. Make sure that the two sites can communicate with each other using HTTPS, SSH and PostgreSQL.
  6. Log in to the Supervisor, go to ADMIN > Settings > Database > Replicate and make sure you have the required information for the two sites.

Set up Disaster Recovery and Failover

Before setting up Disaster Recovery, determine which of the two sites will be the Primary site to start with.

  1. Log in to the current Primary Supervisor node:
    1. Go to ADMIN > Settings > Database > Replicate.
    2. Enter the required information based on the event storage type:
    3. Click Save.
      The Replication process starts and runs in the background. A Job will be created – the status can be seen in the Job window.
    4. Go back to ADMIN > Settings > Database > Replicate page.
    5. Click Export and Save the Replication Configuration file on your local system.
  2. Log in to the current Secondary Supervisor node:
    1. Go to ADMIN > Settings > Database > Replicate.
    2. Select the exported Replication file from the current Primary site stored in your local computer. Click Import.
    3. The Replicate page opens. Make sure the information is correct. Click Save.
      This will continue the Replication process that started with the current Primary. A Job will also be created on the current Secondary – the status can be seen in the Job window.

The Replication process works as follows:

  1. The CMDB of the current Primary will be replicated to current Secondary.
  2. Once step 1 is complete, you will see that the two jobs created in Primary and Secondary will show complete in the Job window.
  3. Other replication will begin and continue until replication status changes:
    1. SVN replication – Secondary will pull in replication files from Primary
    2. Profile DB replication
    3. Local or NFS based replication
    4. Elasticsearch replication

Operating FortiSIEM in Replication mode

  1. Make sure DNS points to current Primary:
    1. DNS: fortisiem.acme.com (Site1.fortisiem.acme.com)
    2. Users log in to fortisiem.acme.com (Site1.fortisiem.acme.com)
    3. Collectors register to fortisiem.acme.com. This will cause the Worker Configuration at Site1.fortisiem.acme.com to be pushed to Collectors.
  2. Events and configurations will be sent to the Primary and replicated to Secondary.
  3. Profile Database and Incidents will be computed in the Primary and replicated to Secondary.
  4. Inline Reports and scheduled reports are not copied over from Primary to Secondary. Incident Notification and Scheduled Report delivery happens on Primary only.
  5. Incidents trigger on Primary and replicates to Secondary.
  6. Archiving and Report Server synching occur independently on the two sites.
  7. All processes will be up on Primary. However, on the Secondary node, only the processes required for user login (App Server, PostGreSQL, Query Master/Worker, Java Query Server) and replication (Data Purger) will be up.
  8. The Secondary site can be operated like a Primary except the events are delayed because of replication (see ADMIN > Settings > Database > Replicate > Replication Frequency).
  9. CMDB is set in a multi-master mode – so any changes on Secondary are replicated over to Primary. It is recommended to do all edits on the Primary site.

Handling Disaster

Assuming that disaster happens at the current Primary Site, make sure DNS points to current Secondary, that is, fortisiem.acme.com points to Site2.fortisiem.acme.com. Users log in to fortisiem.acme.com (now Site2.fortisiem.acme.com).

Log in to the current Secondary Supervisor node (Site2.fortisiem.acme.com) and:

  1. Go to ADMIN > Settings > Database > Replicate.
  2. Switch the roles – set Site2.fortisiem.acme.com as the Primary site.
  3. Click Save.
    A Replication Change will be created in the Secondary and it will finish (Progress 100%). All processes will come up on Supervisor and all Worker nodes.

Collectors will send to the new Primary (Site2.fortisiem.acme.com) as follows:

  1. Collectors will first fail to send to old Workers (part of Site1.fortisiem.acme.com).
  2. Collectors will request a new Worker list from Super (now Site2.fortisiem.acme.com because of DNS change).
  3. Collectors a new list of Workers from Site2.fortisiem.acme.com
  4. Collectors will start sending events to the new Primary FortiSIEM cluster

Handling Recovery

Once the old Primary (Site1.fortisiem.acme.com) recovers, you may want to switch back to that site.

First, make the old Primary (Site1.fortisiem.acme.com) a Secondary.

  1. Log on to Site1.fortisiem.acme.com and make it Secondary and follow these steps:
    1. Go to ADMIN > Settings > Database > Replicate.
    2. Make sure Site1.fortisiem.acme.com is Secondary and Site2.fortisiem.acme.com is Primary. This will happen because the CMDB replication will bring back changes from Site2 (Current Primary) to Site 1 (Current Secondary).
    3. Click Save.
      A Replication Change will be created in the Secondary and it will finish (Progress 100%).
    4. Only the processes required for user login (App Server, PostGreSQL, Query Master/Worker, Java Query Server) and replication (Data Purger) will be up.
    5. Replication will continue – all missing data during disaster will flow back in from Site2 to Site1.

Once Site1.fortisiem.acme.com has come up, make it Primary.

  1. Log in to Site1.fortisiem.acme.com and make it Primary:
    1. Go to ADMIN > Settings > Database > Replicate.
    2. Switch Roles.
    3. Click Save.
  2. Log in to Site2.fortisiem.acme.com and make it Secondary:
    1. Go to ADMIN > Settings > Database > Replicate.
    2. Verify Roles (verify since the changes in Site1 will be replicated).
    3. Click Save.

Disabling Disaster Recovery

If you do not want to enable the Disaster Recovery feature, you can turn off this.

Log in to the current Primary (note that this has to be done first):

  1. Go to ADMIN > Settings > Database > Replicate.
  2. Uncheck Enable Replication to disable Replication.
  3. Click Save and make sure Replication setting task is completed.
    A Replication Job will be created. Make sure that the Job is finished from the Jobs and Errors window.

Log in to the current Secondary site:

  1. Go to ADMIN > Settings > Database > Replicate.
  2. Make sure Replication is disabled.
  3. Click Save.
    A Replication Job will be created. Make sure that the Job is finished from the Jobs and Errors window.