Configuring Disaster Recovery

• Configuring Disaster Recovery - New Install

• Configuring Disaster Recovery - Existing Install

Configuring Disaster Recovery - New Install

Ensure you have followed the Prerequisites for a Successful DR Implementation and Basic Requirements (below) prior to this configuration.

This configuration assumes a completely new FortiSIEM install. To facilitate configuration, assume that there are two Sites: Site 1 and Site 2. Initially, Site 1 is Primary and Site 2 is Secondary.

Follow all the proceeding steps to configure a bidirectional Elasticsearch Cross-Cluster replication (CCR) that works with FortiSIEM Disaster Recovery (DR).

Basic Requirements

Site 1 and Site 2 must have an identical setup for its Supervisor, Workers, and Elasticsearch cluster (Master, Coordinator only and Data Nodes) if the Secondary Site needs to take the workload of the Primary Site for extended periods of time. Specifically, this means for Site 1 and Site 2:

• They must have the same number of Workers.

• The Super and Workers hardware configuration must be identical.

• They must have the same number of Master node, Coordinating nodes, Hot, Warm, and Cold Data nodes.

Step 1. Set Up Elasticsearch for Site 1 and Site 2

Set up two separate Elasticsearch Clusters, one as Site 1 and one as Site 2. Do not add the Elasticsearch cluster to FortiSIEM yet. This will be done after cross-cluster replication (CCR) is setup.

Step 2. Enable Remote Cluster Client for Both Sites

Take the following steps to set up the Elasticsearch Clusters for Site 1 and Site 2.

1. Modify the elasticsearch.yml file for each node in Site 1 with:
   

   node.remote_cluster_client: true
   

2. Restart each node in the cluster for Site 1.

3. Modify the elasticsearch.yml file for each node in Site 2 with:
   

   node.remote_cluster_client: true
   

4. Restart each node in the cluster for Site 2.

Step 3. Add X-Pack's Auto Create Index for Both Sites

X-Pack needs its indices to be created. To create these indices to action.auto_create_index list, take the following steps:

1. Run the following command against the Site 1 Coordinator node.
   

   PUT /_cluster/settings?pretty
   {
     "persistent": {
       "action.auto_create_index": "-fortisiem-event-*,fortisiem-*,.monitoring-*"
     }
   }
   

2. Run the same command against the Site 2 Coordinator node.
   

   PUT /_cluster/settings?pretty
   {
     "persistent": {
       "action.auto_create_index": "-fortisiem-event-*,fortisiem-*,.monitoring-*"
     }
   }
   

Step 4. Define Remote Clusters for Site 1

Note: Do not add the master dedicated node to seeds. This is because dedicated master nodes are never selected as gateway nodes. It is recommended that at least three nodes with low traffic, node.remote_cluster_client enabled, and transport port opened be added in the list of seed nodes, such as the coordinator node.

Take the following steps:

1. Login to Kibana.

2. Navigate to Kibana Home > Analytics section > Discover > Remote Clusters.
   
   

3. Add Site 2's nodes as the remote servers to Site 1. After adding the Site 1's nodes, click Save.
   
   

Step 5. Define Auto-Follow Patterns in Site 1

Since indices are dynamically created in Site 2, you must configure auto-follow pattern in Site 1 to enable the dynamically generated indices in Site 2 to be replicated to Site 1.

Take the following steps:

1. Login to Kibana.

2. Navigate to Kibana Home > Analytics section > Discover > Cross-Cluster Replication.

3. Define an Auto-Follow patterns for fortisiem-event-* time-series indices. Do NOT make similar definitions for other indices. The example screenshot here shows the fortisiem-event.auto_follow_pattern being defined.
   

   

   
   

   The screenshot here shows that auto-follow patterns have been created for fortisiem-event-* time-series indices.
   

   When completed, Elasticsearch on Site 1 is now ready for replication.
   

Step 6. Define Remote Clusters for Site 2

Since Site 2 will initiate the replication, the Site 1 nodes must be defined in Site 2 using Kibana.

Note: Do not add the master dedicated node to seeds. This is because dedicated master nodes are never selected as gateway nodes. It is recommended that at least three nodes with low traffic, node.remote_cluster_client enabled, and transport port opened be added in the list of seed nodes, such as the coordinator node.

Take the following steps:

1. Login to Kibana.

2. Navigate to Kibana Home > Analytics section > Discover > Remote Clusters.
   

   
   

3. Add Site 1's nodes as the remote servers to Site 2. After adding the Site 1's nodes, click Save.
   

   
   

Step 7. Define Auto-Follow Patterns in Site 2

Since indices are dynamically created in Site 1, you must configure auto-follow pattern in Site 2 to enable the dynamically generated indices in Site 1 to be replicated to Site 2.

Take the following steps:

1. Login to Kibana.

2. Navigate to Kibana Home > Analytics section > Discover > Cross-Cluster Replication.

3. Define Auto-Follow patterns for fortisiem-event-* time-series indices. Do NOT make similar definitions for other indices. The example screenshot here shows the fortisiem-event.auto_follow_pattern being defined.
   

   

   
   

   The screenshot here shows that auto-follow patterns have been created for fortisiem-event-* time-series indices.
   
   

   When completed, elasticsearch on Site 2 is now ready for replication.

Step 8. Set Up Site 1 FortiSIEM with Elasticsearch Storage

Take the following steps to set up Site 1 FortiSIEM with Elasticsearch as its online storage.

1. Login to the FortiSIEM GUI.

2. Navigate to ADMIN > Setup > Storage > Online.

3. Select the Elasticsearch radio button from the three available options (Local Disk, NFS, Elasticsearch) and configure.

4. Click Save.

Step 9. Set Up Site 2 FortiSIEM with Elasticsearch Storage

Take the following steps to set up Site 2 FortiSIEM with Elasticsearch as its online storage.

1. Login to the FortiSIEM GUI.

2. Navigate to ADMIN > Setup > Storage > Online.

3. Select the Elasticsearch radio button from the three available options (Local Disk, NFS, Elasticsearch) and configure.

4. Click Save.

Step 10. Set Up Disaster Recovery in FortiSIEM GUI

See Configuring Disaster Recovery in the latest High Availability and Disaster Recovery Procedures - EventDB Guide here.

Step 11. Verify Site 1 to Site 2 Event Replication

Take the following steps to check on Elasticsearch event replication.

1. Login to Kibana.

2. Navigate to Kibana Home > Analytics section > Discover > Cross-Cluster Replication.

3. Verify that the follower indices are created automatically.
   

Step 12. Verify ILM is Working for Follower Index in Site 2

To verify that index lifecycle management (ILM) is working on site 2, you will need to take the following steps:

1. Login to Kibana.

2. Navigate to Kibana Home > Analytics section > Discover > Index Management.

3. Under Indices, select one follower event index and under Index lifecycle management, check Lifecycle policy. It should be fsiem_ilm_policy.
   

Step 13. Verify Site 2 to Site 1 Event Replication

Take the following steps to check on Elasticsearch event replication.

1. Login to Kibana.

2. Navigate to Kibana Home > Analytics section > Discover > Cross-Cluster Replication.

3. Verify that the follower indices are created automatically.
   

   
   

Step 14. Verify ILM is Working for Follower Index in Site 1

To verify that index lifecycle management (ILM) is working on site 1, you will need to take the following steps:

1. Login to Kibana.

2. Navigate to Kibana Home > Analytics section > Discover > Index Management.

3. Under Indices, select one follower event index and under Index lifecycle management, check Lifecycle policy. It should be fsiem_ilm_policy.
   

   
   

Configuring Disaster Recovery - Existing Install

This section assumes that Elasticsearch is already running on FortiSIEM on Site 1 (Primary). Disaster Recovery needs to be set up for Site 1 and Site 2, with Site 2 to be used as Secondary.

Basic Requirements

Site 1 and Site 2 must have an identical setup for its Supervisor, Workers, and Elasticsearch cluster (Master, Coordinator only and Data Nodes) if the Secondary Site needs to take the workload of the Primary Site for extended periods of time. Specifically, this means for Site 1 and Site 2:

• They must have the same number of Workers.

• The Super and Workers hardware configuration must be identical.

• They must have the same number of Master node, Coordinating nodes, Hot, Warm, and Cold Data nodes.

Step 1. Set Up Elasticsearch for Site 2

Set up two separate Elasticsearch clusters, one as Site 1, and one as Site 2. Do not add the Elasticsearch cluster to FortiSIEM yet. This will be done after cross-cluster replication (CCR) is set up.

Step 2. Enable Remote Cluster Client for Both Sites

Take the following steps to set up the Elasticsearch Cluster for Site 1 and Site 2.

1. Modify the elasticsearch.yml file for each node in Site 1 with:
   

   node.remote_cluster_client: true
   

2. Restart each node in the cluster for Site 1.

3. Modify the elasticsearch.yml file for each node in Site 2 with:
   

   node.remote_cluster_client: true
   

4. Restart each node in the cluster for Site 2.

Step 3. Add X-Pack's Auto Create Index for Both Sites

X-Pack needs its indices to be created. To add X-Pack's Auto Create Index to action.auto_create_index list, take the following steps:

1. Run the following command against the Site 1 Coordinator node.
   

   PUT /_cluster/settings?pretty
   {
     "persistent": {
       "action.auto_create_index": "-fortisiem-event-*,fortisiem-*,.monitoring-*"
     }
   }
   

2. Run the same command against the Site 2 Coordinator node.
   

   PUT /_cluster/settings?pretty
   {
     "persistent": {
       "action.auto_create_index": "-fortisiem-event-*,fortisiem-*,.monitoring-*"
     }
   }
   

Step 4. Define Remote Clusters for Site 1

Note: Do not add the master dedicated node to seeds. This is because dedicated master nodes are never selected as gateway nodes. It is recommended that at least three nodes with low traffic, node.remote_cluster_client enabled, and transport port opened be added in the list of seed nodes, such as the coordinator node.

Take the following steps:

1. Login to Kibana.

2. Navigate to Kibana Home > Analytics section > Discover > Remote Clusters.
   

3. Add Site 2's nodes as the remote servers to Site 1. After adding Site 1's nodes, click Save.

   
   

Step 5. Define Auto-Follow Patterns in Site 1

Since indices are dynamically created in Site 2, you must configure auto-follow pattern in Site 1 to enable the dynamically generated indices in Site 2 to be replicated to Site 1.

Take the following steps:

1. Login to Kibana.

2. Navigate to Kibana Home > Analytics section > Discover > Cross-Cluster Replication.

3. Define Auto-Follow patterns for fortisiem-event-* time-series indices. Do NOT make similar definitions for other indices.

   

   The screenshot here shows that an auto-follow pattern has been created for for fortisiem-event-* time-series indices.
   

   

   When completed, Elasticsearch on Site 1 is now ready for replication.

Step 6. Define Remote Clusters for Site 2

Since Site 2 will initiate the replication, the Site 1 nodes must be defined in Site 2 using Kibana.

Note: Do not add the master dedicated node to seeds. This is because dedicated master nodes are never selected as gateway nodes. It is recommended that at least three nodes with low traffic, node.remote_cluster_client enabled, and transport port opened be added in the list of seed nodes, such as the coordinator node.

Take the following steps:

1. Login to Kibana.

2. Navigate to Kibana Home > Analytics section > Discover > Remote Clusters.
   

   
   

3. Add Site 1's nodes as the remote servers to Site 2. After adding the Site 1's nodes, click Save.
   

   
   

Step 7. Define Auto-Follow Patterns in Site 2

Since indices are dynamically created in Site 1, you must configure auto-follow pattern in Site 2 to enable the dynamically generated indices in Site 1 to be replicated to Site 2.

Take the following steps:

1. Login to Kibana.

2. Navigate to Kibana Home > Analytics section > Discover > Cross-Cluster Replication.

3. Define Auto-Follow patterns for fortisiem-event-* time-series indices. Do NOT make similar definitions for other indices. The example screenshot here shows the fortisiem-event.auto_follow_pattern being defined.
   

   

   
   

   The screenshot here shows that auto-follow patterns have been created for fortisiem-event-* time-series indices.
   

   When completed, Elasticsearch on Site 2 is now ready for replication.

Step 8. Set Up Site 2 FortiSIEM with Elasticsearch Storage

Take the following steps to set up Site 2 FortiSIEM with Elasticsearch as its online storage.

1. Login to the FortiSIEM GUI.

2. Navigate to ADMIN > Setup > Storage > Online.

3. Select the Elasticsearch radio button from the available options (Local Disk, NFS, Elasticsearch) and configure.

4. Click Save.

Step 9. Set Up Disaster Recovery in FortiSIEM GUI

See Configuring Disaster Recovery in the latest High Availability and Disaster Recovery Procedures - EventDB Guide here.

Step 10. Verify Site 1 to Site 2 Event Replication

Take the following steps to check on Elasticsearch event replication.

1. Login to Kibana.

2. Navigate to Kibana Home > Analytics section > Discover > Cross-Cluster Replication.

3. Verify that the follower indices are created automatically.
   

Step 11. Verify ILM is Working for Follower Index in Site 2

To verify that index lifecycle management (ILM) is working on site 2, you will need to take the following steps:

1. Login to Kibana.

2. Navigate to Kibana Home > Analytics section > Discover > Index Management.

3. Under Indices, select one follower event index and under Index lifecycle management, check Lifecycle policy. It should be fsiem_ilm_policy.
   

Step 12. Verify Site 2 to Site 1 Event Replication

Take the following steps to check on Elasticsearch event replication.

1. Login to Kibana.

2. Navigate to Kibana Home > Analytics section > Discover > Cross-Cluster Replication.

3. Verify that the follower indices are created automatically.
   

   
   

Step 13. Verify ILM is Working for Follower Index in Site 1

To verify that index lifecycle management (ILM) is working on site 1, you will need to take the following steps:

1. Login to Kibana.

2. Navigate to Kibana Home > Analytics section > Discover > Index Management.

3. Under Indices, select one follower event index and under Index lifecycle management, check Lifecycle policy. It should be fsiem_ilm_policy.
   

   

Step 14. (Optional) Copy Older Indices from Site 1 to Site 2

Elasticsearch only replicates events after it has been configured. Use Elasticsearch API to copy all older indices to Site 2 by taking the following steps.

1. Use Kibana to copy older fortisiem-event-* indices to Site 2.

2. Login to Kibana.

3. Navigate to Kibana Home > Stack Management > Cross-Cluster Replication > Follower indices.

4. In Add follower index, take the following steps:

   1. Provide the Leader index.

   2. Provide the Follower index.

   3. Click Create.