Fortinet black logo

High Availability and Disaster Recovery Procedures - EventDB

Configuring Disaster Recovery

Configuring Disaster Recovery

Configuration

Ensure you have followed the Prerequisites for a Successful DR Implementation prior to this configuration.

Assume there are two sites, Site 1 needs to be set up as Primary, and Site 2 as Secondary.

Take the following steps to configure Disaster Recovery.

Step 1. Collect UUID and SSH Public Key from Primary (Site 1)

  1. For the UUID, obtain the Hardware ID value through an SSH session by running the following command on Site 1.

    /opt/phoenix/bin/phLicenseTool --show

    For example:

  2. Enter/paste the Hardware ID into the UUID field for the Site 1 FortiSIEM.

  3. Under Configuration and Profile Replication, generate the SSH Public Key and SSH Private Key Path by entering the following in your SSH session from Site 1:

    su – admin

    ssh-keygen -t rsa -b 4096

    Leave the file location as default, and press enter at the passphrase prompt.

    The output will appear similar to the following:

    Generating public/private rsa key pair.

    Enter file in which to save the key (/opt/phoenix/bin/.ssh/id_rsa):

    Created directory '/opt/phoenix/bin/.ssh'.

    Enter passphrase (empty for no passphrase):

    Enter same passphrase again:

    Your identification has been saved in /opt/phoenix/bin/.ssh/id_rsa.

    Your public key has been saved in /opt/phoenix/bin/.ssh/id_rsa.pub.

    The key fingerprint is:

    a9:43:88:d1:ed:b0:99:b5:bb:e7:6d:55:44:dd:3e:48 admin@site1.fsmtesting.com

    The key's randomart image is:

    +--[ RSA 4096]----+

    | ....|

    | . . E. o|

  4. For the SSH Public Key enter the following command, and copy all of the output.

    cat /opt/phoenix/bin/.ssh/id_rsa.pub

Step 2. Collect UUID and SSH Public Key from Secondary (Site 2)

  1. On the Site 2 FortiSIEM node, SSH as root.

  2. Run the following command to get the Hardware ID, also known as the UUID. Record this Site 2 Hardware ID, as you will need it later.

    /opt/phoenix/bin/phLicenseTool --show


  3. Generate a public key for Site 2 by running the following commands.

    su – admin

    ssh-keygen -t rsa -b 4096

    Leave the file location as default, and press enter at the passphrase prompt. Your output will appear similar to the following.

    Generating public/private rsa key pair.

    Enter file in which to save the key (/opt/phoenix/bin/.ssh/id_rsa):

    Created directory '/opt/phoenix/bin/.ssh'.

    Enter passphrase (empty for no passphrase):

    Enter same passphrase again:

    Your identification has been saved in /opt/phoenix/bin/.ssh/id_rsa.

    Your public key has been saved in /opt/phoenix/bin/.ssh/id_rsa.pub.

    The key fingerprint is:

    a9:43:88:d1:ed:b0:99:b5:bb:e7:6d:55:44:dd:3e:48 admin@site2.fsmtesting.com

    The key's randomart image is:

    +--[ RSA 4096]----+

    | ....|

    | . . E. o|

  4. Enter the following command, and copy all of the output.
    cat /opt/phoenix/bin/.ssh/id_rsa.pub
    You will use the output as your SSH Public Key for Site 2 in later set up.

  5. Exit the admin user in the SSH session by entering the following command:

    exit

Step 3. Set up Disaster Recovery on Primary (Site 1)

  1. Navigate to ADMIN > License > Nodes.

  2. Click Add.

  3. On the Add Node window, in the Type drop-down list, select Secondary (DR).
    The primary (Site 1) node configuration fields appear in the left column, and the secondary (Site 2) node configuration fields appear in the right column.

  4. Under the Host Info Role Primary column, take the following steps:

    1. In the Host field, enter the host name of the Site 1 FortiSIEM.

    2. In the IP field, enter the IP of the Site 1 FortiSIEM.

    3. In the SSH Public Key field, enter/paste the SSH Public Key of the Site 1 FortiSIEM that you obtained earlier.

    4. For the SSH Private Key Path, enter the following into the field:

      /opt/phoenix/bin/.ssh/id_rsa

    5. For Replication Frequency, select a value for the Site 1 FortiSIEM.

    6. Select the EventDB Replication check box if you would also like the Event Database to be replicated. This is NOT required for Elasticsearch.

      Note 1: For Local/NFS Event DB installs, this value is used for SVN and ProfileDB synchronization.

      Note 2: For Local/NFS Event DB installs, rsync is used, and this runs continually in the background.

  5. Under the Host Info Role Secondary column, take the following steps:

    1. In the Host field, enter the host name of the Site 2 FortiSIEM.

    2. In the IP field, enter the IP address of the Site 2 FortiSIEM.

    3. In the UUID field, enter/paste the Hardware ID of the Site 2 FortiSIEM that you obtained earlier.

    4. In the SSH Public Key field, enter/paste the SSH Public Key of the Site 2 FortiSIEM that you obtained earlier.

    5. For the SSH Private Key Path, enter the following into the field:

      /opt/phoenix/bin/.ssh/id_rsa

    6. Select the EventDB Replication check box if you would also like the Event Database to be replicated. If you are running Elasticsearch, then see the Disaster Recovery for Elasticsearch Guide here.

    7. Click Export and download a file named replicate.json.
      Note: This file contains all of the Disaster Recovery settings, and can be used as a backup.

  6. Click Save.

    At this point, the Site 1 (Primary) node will begin configuration and the step and progress of the Disaster Recovery is displayed in the GUI.

    When completed, the message "Replicate Settings applied." will appear.

Service Status on Primary and Secondary

On the Primary node, all FortiSIEM ph* services will be in an "up" state.

On the Secondary node, most ph* services will be "down" except for phQueryMaster, phQueryWorker, phDataPurger, and phMonitor.

This can be seen in the following images. They illustrate the Primary Node and Secondary Node after a full CMDB sync:

On the Secondary node, all backend processes should be down on the Supervisor and Workers except for phQueryMaster, phQueryWorker, DataPurger, DBServer, and AppServer.

Viewing Replication Health

Replication progress is available by navigating to ADMIN > Health > Replication Health. For details see here.

Permitted User Activities on Secondary

When operating in DR Replication mode, there are a few things to bear in mind:

  • Both the Primary (Site 1) and Secondary (Site 2) nodes GUI are available for login.
  • The Secondary (Site 2) is only available for read-only operations. From Secondary (Site 2), expect the following:
    • Able to view CMDB, Incidents, Cases, Tasks, Resources and all settings in the ADMIN page except the License Usage Page, etc

    • Cannot run any queries on ANALYTICS and all widgets on Dashboards and all report related graphs such as the License Usage Page have no data.

    • Cannot do any Editing operations on all GUI pages.

    • All actions related to update operations do not work.

Configuring Disaster Recovery

Configuration

Ensure you have followed the Prerequisites for a Successful DR Implementation prior to this configuration.

Assume there are two sites, Site 1 needs to be set up as Primary, and Site 2 as Secondary.

Take the following steps to configure Disaster Recovery.

Step 1. Collect UUID and SSH Public Key from Primary (Site 1)

  1. For the UUID, obtain the Hardware ID value through an SSH session by running the following command on Site 1.

    /opt/phoenix/bin/phLicenseTool --show

    For example:

  2. Enter/paste the Hardware ID into the UUID field for the Site 1 FortiSIEM.

  3. Under Configuration and Profile Replication, generate the SSH Public Key and SSH Private Key Path by entering the following in your SSH session from Site 1:

    su – admin

    ssh-keygen -t rsa -b 4096

    Leave the file location as default, and press enter at the passphrase prompt.

    The output will appear similar to the following:

    Generating public/private rsa key pair.

    Enter file in which to save the key (/opt/phoenix/bin/.ssh/id_rsa):

    Created directory '/opt/phoenix/bin/.ssh'.

    Enter passphrase (empty for no passphrase):

    Enter same passphrase again:

    Your identification has been saved in /opt/phoenix/bin/.ssh/id_rsa.

    Your public key has been saved in /opt/phoenix/bin/.ssh/id_rsa.pub.

    The key fingerprint is:

    a9:43:88:d1:ed:b0:99:b5:bb:e7:6d:55:44:dd:3e:48 admin@site1.fsmtesting.com

    The key's randomart image is:

    +--[ RSA 4096]----+

    | ....|

    | . . E. o|

  4. For the SSH Public Key enter the following command, and copy all of the output.

    cat /opt/phoenix/bin/.ssh/id_rsa.pub

Step 2. Collect UUID and SSH Public Key from Secondary (Site 2)

  1. On the Site 2 FortiSIEM node, SSH as root.

  2. Run the following command to get the Hardware ID, also known as the UUID. Record this Site 2 Hardware ID, as you will need it later.

    /opt/phoenix/bin/phLicenseTool --show


  3. Generate a public key for Site 2 by running the following commands.

    su – admin

    ssh-keygen -t rsa -b 4096

    Leave the file location as default, and press enter at the passphrase prompt. Your output will appear similar to the following.

    Generating public/private rsa key pair.

    Enter file in which to save the key (/opt/phoenix/bin/.ssh/id_rsa):

    Created directory '/opt/phoenix/bin/.ssh'.

    Enter passphrase (empty for no passphrase):

    Enter same passphrase again:

    Your identification has been saved in /opt/phoenix/bin/.ssh/id_rsa.

    Your public key has been saved in /opt/phoenix/bin/.ssh/id_rsa.pub.

    The key fingerprint is:

    a9:43:88:d1:ed:b0:99:b5:bb:e7:6d:55:44:dd:3e:48 admin@site2.fsmtesting.com

    The key's randomart image is:

    +--[ RSA 4096]----+

    | ....|

    | . . E. o|

  4. Enter the following command, and copy all of the output.
    cat /opt/phoenix/bin/.ssh/id_rsa.pub
    You will use the output as your SSH Public Key for Site 2 in later set up.

  5. Exit the admin user in the SSH session by entering the following command:

    exit

Step 3. Set up Disaster Recovery on Primary (Site 1)

  1. Navigate to ADMIN > License > Nodes.

  2. Click Add.

  3. On the Add Node window, in the Type drop-down list, select Secondary (DR).
    The primary (Site 1) node configuration fields appear in the left column, and the secondary (Site 2) node configuration fields appear in the right column.

  4. Under the Host Info Role Primary column, take the following steps:

    1. In the Host field, enter the host name of the Site 1 FortiSIEM.

    2. In the IP field, enter the IP of the Site 1 FortiSIEM.

    3. In the SSH Public Key field, enter/paste the SSH Public Key of the Site 1 FortiSIEM that you obtained earlier.

    4. For the SSH Private Key Path, enter the following into the field:

      /opt/phoenix/bin/.ssh/id_rsa

    5. For Replication Frequency, select a value for the Site 1 FortiSIEM.

    6. Select the EventDB Replication check box if you would also like the Event Database to be replicated. This is NOT required for Elasticsearch.

      Note 1: For Local/NFS Event DB installs, this value is used for SVN and ProfileDB synchronization.

      Note 2: For Local/NFS Event DB installs, rsync is used, and this runs continually in the background.

  5. Under the Host Info Role Secondary column, take the following steps:

    1. In the Host field, enter the host name of the Site 2 FortiSIEM.

    2. In the IP field, enter the IP address of the Site 2 FortiSIEM.

    3. In the UUID field, enter/paste the Hardware ID of the Site 2 FortiSIEM that you obtained earlier.

    4. In the SSH Public Key field, enter/paste the SSH Public Key of the Site 2 FortiSIEM that you obtained earlier.

    5. For the SSH Private Key Path, enter the following into the field:

      /opt/phoenix/bin/.ssh/id_rsa

    6. Select the EventDB Replication check box if you would also like the Event Database to be replicated. If you are running Elasticsearch, then see the Disaster Recovery for Elasticsearch Guide here.

    7. Click Export and download a file named replicate.json.
      Note: This file contains all of the Disaster Recovery settings, and can be used as a backup.

  6. Click Save.

    At this point, the Site 1 (Primary) node will begin configuration and the step and progress of the Disaster Recovery is displayed in the GUI.

    When completed, the message "Replicate Settings applied." will appear.

Service Status on Primary and Secondary

On the Primary node, all FortiSIEM ph* services will be in an "up" state.

On the Secondary node, most ph* services will be "down" except for phQueryMaster, phQueryWorker, phDataPurger, and phMonitor.

This can be seen in the following images. They illustrate the Primary Node and Secondary Node after a full CMDB sync:

On the Secondary node, all backend processes should be down on the Supervisor and Workers except for phQueryMaster, phQueryWorker, DataPurger, DBServer, and AppServer.

Viewing Replication Health

Replication progress is available by navigating to ADMIN > Health > Replication Health. For details see here.

Permitted User Activities on Secondary

When operating in DR Replication mode, there are a few things to bear in mind:

  • Both the Primary (Site 1) and Secondary (Site 2) nodes GUI are available for login.
  • The Secondary (Site 2) is only available for read-only operations. From Secondary (Site 2), expect the following:
    • Able to view CMDB, Incidents, Cases, Tasks, Resources and all settings in the ADMIN page except the License Usage Page, etc

    • Cannot run any queries on ANALYTICS and all widgets on Dashboards and all report related graphs such as the License Usage Page have no data.

    • Cannot do any Editing operations on all GUI pages.

    • All actions related to update operations do not work.