EMS nodes in HA with Postgres DB HA cluster using EMS PostgreSQL HA Docker, single region

EMS requires a custom PostgreSQL extension to function and Fortinet provides an EMS PostgreSQL HA repmgr custom docker image where that custom extension is already installed and ready for EMS. This image is further customized by Fortinet to include additional node reconciliation scripts to add another layer of protection and prevention of split brain.

When using this image, Fortinet is responsible for providing support for matters related to replication, failover, split-brain, and syncing between nodes, but are not responsible nor supports anything related to the database itself (PostgreSQL), docker, the host OS where the container will be running or network issues in the infrastructure that the cluster or DB nodes are part of.

The following guide gives instructions on performing a fresh install of EMS HA with Postgres HA using EMS PostgreSQL HA Docker and for upgrading a previously installed EMS HA deployment with Postgres HA using EMS PostgreSQL HA Docker to 7.4.5:

• To configure EMS HA with Postgres HA using EMS PostgreSQL HA Docker:
• To upgrade a previously installed EMS deployment with Postgres HA using EMS PostgreSQL HA Docker to 7.4.5:

To configure EMS HA with Postgres HA using EMS PostgreSQL HA Docker:

1. Prepare five Linux machines. Three act as Postgres DB nodes (two for data nodes and one for the witness node) and the other two are EMS nodes.

   Refer to the EMS management capacity guide to plan the sizing of the data nodes.

2. If using cloned virtual machines for the EMS nodes, run the following commands on the cloned machine:

   sudo rm /etc/machine-id
   sudo rm /var/lib/dbus/machine-id
   sudo systemd-machine-id-setup

3. Run sudo -i to log in to the shell with root privileges. Perform all following steps with root privileges.
4. Configure the Postgres HA cluster:
   1. Load the EMS Postgres HA Docker image in all Postgres DB nodes:
      1. Install Docker:

         apt install docker.io

      2. Download the Postgres Docker image forticlientems_postgres-ha.tar.gz file from the Fortinet Support site.
      3. Load the image:

         docker load -i forticlientems_postgres-ha.tar.gz

         

      4. List the images on Docker to verify the image has been created or loaded:

         docker image ls

   2. For each of the Postgres DB nodes (not the witness nodes), tune Postgres based on the host server specs by applying the recommended configuration as follows:

      While there are various tools you can use to find the recommended configuration, the following instructions use PGTune to generate and copy the recommended configuration.

      1. Create a folder called conf.d and configure the folder to be root owned with special permissions so it can be read by the containers:

         sudo chgrp -R root /path/to/conf.d/
         sudo chmod -R g+rwX /path/to/conf.d/

      2. Create a file in the folder called extra.conf.

      3. Go to PGTune and enter the following information:

         Field	Value
         DB version	15
         OS Type	Linux
         DB Type	Online transaction processing system
         Total Memory (RAM)	Enter the total memory for your Postgres server. In this example, it is 4 GB.
         Number of CPUs	Enter the total number of CPUs for your Postgres server. In this example, it is 4.
         Number of Connections	1092
         Data Storage	Enter the data storage type as per your device. In this example, it is SSD storage.

      4. Click Generate.

         

      5. Click Copy configuration and save the configuration to the extra.conf file you created earlier.

   3. Start the primary DB:

      1. Start the EMS PostgreSQL HA container on the primary DB node. In this example, the primary DB node is pg-1:

         docker run --restart always --detach --name pg-1 -p 5432:5432 \
           --env POSTGRESQL_PASSWORD=<pg_password> \
           --env POSTGRESQL_DAEMON_USER=<pg_username> \
           --env REPMGR_PASSWORD=<pg_password> \
           --env REPMGR_PRIMARY_HOST=pg-1 \
           --env REPMGR_PRIMARY_PORT=5432 \
           --env REPMGR_PARTNER_NODES=pg-1,pg-2,pgw-1:5432 \
           --env REPMGR_NODE_NAME=pg-1 \
           --env REPMGR_NODE_NETWORK_NAME=pg-1 \
           --env REPMGR_PORT_NUMBER=5432 \
           --env REPMGR_DEGRADED_MONITORING_TIMEOUT=-1 \
           --env REPMGR_NODE_PRIORITY=100 \
           --volume pg_1_data:/fortinet/postgresql \
           --volume <path to conf.d>:/fortinet/postgresql/conf/conf.d \
           --shm-size 1g \
           --add-host pg-1:<ip/fqdn node 1> \
           --add-host pg-2:<ip/fqdn node 2> \
           --add-host pgw-1:<ip/fqdn witness 1 dc 1> \
         ems-postgres-ha

         Replace <pg_password>, <pg_username>, and <path to conf.d> with your actual Postgres password, username, and path to conf.d file that you created earlier in the PostgreSQL configuration (such as /home/ems/conf.d). Replace <ip/fqdn xxxx> with the correct IP or FQDN to the corresponding PostgreSQL or witness node.

         The following provides details for some of the parameters:

         Parameter	Value
         REPMGR_PRIMARY_HOST	Hostname or IP address of where this node is running, which other nodes can access
         REPMGR_PARTNER_NODES	Comma-separated list of the hostnames or IP addresses of all nodes that form this cluster
         REPMGR_NODE_NAME	Node name. This can but does not necessarily need to match the hostname. This value shows when you perform step d to verify the cluster status.
         REPMGR_NODE_NETWORK_NAME	Hostname or IP address of where this node is running.

      2. Verify if the container is running:

         docker ps -a

      3. Verify that the node has joined the cluster successfully:

         docker exec -it pg-1 bash /opt/fortinet/scripts/postgresql-repmgr/entrypoint.sh repmgr -f /opt/fortinet/repmgr/conf/repmgr.conf service status

   4. From the secondary DB node, start the DB:
      1. Start the EMS PostgreSQL HA container on the secondary node. In this example, the secondary node is pg-2:

         docker run --restart always --detach --name pg-2 -p 5432:5432 \
           --env POSTGRESQL_PASSWORD=<pg_password> \
           --env POSTGRESQL_DAEMON_USER=<pg_username> \
           --env REPMGR_PASSWORD=<pg_password> \
           --env REPMGR_PRIMARY_HOST=pg-1 \
           --env REPMGR_PRIMARY_PORT=5432 \
           --env REPMGR_PARTNER_NODES=pg-1,pg-2,pgw-1:5432 \
           --env REPMGR_NODE_NAME=pg-2 \
           --env REPMGR_NODE_NETWORK_NAME=pg-2 \
           --env REPMGR_PORT_NUMBER=5432 \
           --env REPMGR_DEGRADED_MONITORING_TIMEOUT=-1 \
           --env REPMGR_NODE_PRIORITY=100 \
           --volume pg_2_data:/fortinet/postgresql \
           --volume <path to conf.d>:/fortinet/postgresql/conf/conf.d \
           --shm-size 1g \
           --add-host pg-1:<ip/fqdn node 1> \
           --add-host pg-2:<ip/fqdn node 2> \
           --add-host pgw-1:<ip/fqdn witness 1 dc 1> \
         ems-postgres-ha

         Replace <pg_password>, <pg_username>, and <path to conf.d> with your actual Postgres password, username, and path to conf.d file that you created earlier in the PostgreSQL configuration (such as /home/ems/conf.d). Replace <ip/fqdn xxxx> with the correct IP or FQDN to the corresponding PostgreSQL or witness node.

         The following provides details for some of the parameters:

         Parameter	Value
         REPMGR_PRIMARY_HOST	Hostname or IP address of where the primary node is running, which other nodes can access
         REPMGR_PARTNER_NODES	Comma-separated list of the hostnames or IP addresses of all nodes that form this cluster
         REPMGR_NODE_NAME	Node name. This can but does not necessarily need to match the hostname.
         REPMGR_NODE_NETWORK_NAME	Hostname or IP address of where this node is running

      2. Verify if the container is running:

         docker ps -a

      3. Verify that the node has joined the cluster successfully:

         docker exec -it pg-2 bash /opt/fortinet/scripts/postgresql-repmgr/entrypoint.sh repmgr -f /opt/fortinet/repmgr/conf/repmgr.conf service status

   5. Start the PostgreSQL witness node:

      The witness node does not take part in the data replication process and is a member of the cluster for the purposes of voting on node promotion. It requires as little as 2 GB RAM and 2 CPU.

      1. Start the Postgres witness node. In this example, the witness node is pgw-1:

         docker run --restart always --detach --name pgw-1 -p 5432:5432 \
           --env POSTGRESQL_PASSWORD=<pg_password> \
           --env POSTGRESQL_DAEMON_USER=<pg_username> \
           --env REPMGR_PASSWORD=<pg_password> \
           --env REPMGR_PRIMARY_HOST=pg-1 \
           --env REPMGR_PRIMARY_PORT=5432 \
           --env REPMGR_PARTNER_NODES=pg-1,pg-2,pgw-1:5432 \
           --env REPMGR_NODE_NAME=pgw-1 \
           --env REPMGR_NODE_NETWORK_NAME=pgw-1 \
           --env REPMGR_PORT_NUMBER=5432 \
           --env REPMGR_DEGRADED_MONITORING_TIMEOUT=-1 \
           --env REPMGR_NODE_TYPE=witness \
           --env REPMGR_NODE_ID_START_SEED=2000 \
           --volume pgw_1_data:/fortinet/postgresql \
           --shm-size 1g \
           --add-host pg-1:<ip/fqdn node 1> \
           --add-host pg-2:<ip/fqdn node 2> \
           --add-host pgw-1:<ip/fqdn witness 1 dc 1> \
         ems-postgres-ha

         Replace <pg_password> and <pg_username> with your actual Postgres password and username. Replace <ip/fqdn xxxx> with the correct IP or FQDN to the corresponding PostgreSQL or witness node.

         The following provides details for some of the parameters:

         Parameter	Value
         REPMGR_PRIMARY_HOST	Hostname or IP address of where the primary node is running, which other nodes can access
         REPMGR_PARTNER_NODES	Comma-separated list of the hostnames or IP addresses of all nodes that form this cluster
         REPMGR_NODE_NAME	Node name. This can but does not necessarily need to match the hostname.
         REPMGR_NODE_NETWORK_NAME	Hostname or IP address of where this node is running

      2. Verify if the container is running:

         docker ps -a

      3. Verify that the node has joined the cluster successfully:

         docker exec -it pgw-1 bash /opt/fortinet/scripts/postgresql-repmgr/entrypoint.sh repmgr -f /opt/fortinet/repmgr/conf/repmgr.conf service status

5. After the DB cluster is up and running, configure EMS HA:

   1. On both nodes, do the following:
      1. Download the forticlientems_7.4.5.2111.M.bin file from the Fortinet Support site.
      2. Change permissions and add execute permissions to the installation file:

         chmod +x forticlientems_7.4.5.2111.M.bin

   2. On the primary node, install EMS:
      1. Set umask to 022 on file /etc/login.defs if the existing umask setting is more restrictive.

      2. If you are installing EMS on Red Hat Enterprise Linux (RHEL) 9, do one of the following. EMS 7.4.5 and later versions support install on RHEL 9.:

         • If you are installing EMS on RHEL on Azure, run the following:

           sudo dnf repolist enabled

           Verify if codeready-builder-for-rhel-9-x86_64-eus-rhui-rpms is in the list. If it is in the list, it is enabled. If it is not in the list, then run the following:

           sudo subscription-manager repos --enable codeready-builder-for-rhel-9-x86_64-eus-rhui-rpms

           Run the following commands:

           sudo dnf install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-$(rpm -E %rhel)-$(uname -m)/pgdg-redhat-repo-latest.noarch.rpm
           sudo dnf config-manager --disable pgdg17 pgdg16
           sudo dnf install -y https://rpms.remirepo.net/enterprise/remi-release-$(rpm -q --qf "%{VERSION}\n" redhat-release).rpm
           sudo curl -o /etc/pki/rpm-gpg/RPM-GPG-KEY-remi2021 https://rpms.remirepo.net/RPM-GPG-KEY-remi2021
           sudo rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-remi2021
           sudo ln -sf /etc/pki/rpm-gpg/RPM-GPG-KEY-remi2021 /etc/pki/rpm-gpg/RPM-GPG-KEY-remi.el$(rpm -q --qf "%{VERSION}\n" redhat-release)

         • If you are installing EMS on RHEL on AWS, run the following command:

           sudo dnf config-manager --set-enabled codeready-builder-for-rhel-9-rhui-rpms

      3. Install EMS:

         ./forticlientems_7.4.5.2111.M.bin -- --db_host "<ip/fqdn node 1>,<ip/fqdn node 2>" --db_user postgres --db_pass postgres --skip_db_install --allowed_hosts '*' --enable_remote_https

         Run the installer to and from any directory other than /tmp. Running the installer to or from /tmp causes issues.

         Replace <ip/fqdn xxxx> with the correct IP address or FQDN of the corresponding DB node.

      4. After installation completes, check that all EMS services are running by entering the following command:

         systemctl --all --type=service | grep -E 'fcems|apache|redis|postgres'

         

         The output shows that postgresql.service status displays as exited. This is the expected status. EMS does not create this service, which only exists to pass commands to version-specific Postgres services. It displays as part of the output as the command filters for all services that contain "postgres" in the name.

   3. On the secondary node, install EMS:
      1. Set umask to 022 if the existing umask setting is more restrictive.

      2. If you are installing EMS on Red Hat Enterprise Linux (RHEL) 9, do one of the following. EMS 7.4.5 and later versions support install on RHEL 9.:

         • If you are installing EMS on RHEL on Azure, run the following:

           sudo dnf repolist enabled

           Verify if codeready-builder-for-rhel-9-x86_64-eus-rhui-rpms is in the list. If it is in the list, it is enabled. If it is not in the list, then run the following:

           sudo subscription-manager repos --enable codeready-builder-for-rhel-9-x86_64-eus-rhui-rpms

           Run the following commands:

           sudo dnf install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-$(rpm -E %rhel)-$(uname -m)/pgdg-redhat-repo-latest.noarch.rpm
           sudo dnf config-manager --disable pgdg17 pgdg16
           sudo dnf install -y https://rpms.remirepo.net/enterprise/remi-release-$(rpm -q --qf "%{VERSION}\n" redhat-release).rpm
           sudo curl -o /etc/pki/rpm-gpg/RPM-GPG-KEY-remi2021 https://rpms.remirepo.net/RPM-GPG-KEY-remi2021
           sudo rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-remi2021
           sudo ln -sf /etc/pki/rpm-gpg/RPM-GPG-KEY-remi2021 /etc/pki/rpm-gpg/RPM-GPG-KEY-remi.el$(rpm -q --qf "%{VERSION}\n" redhat-release)

         • If you are installing EMS on RHEL on AWS, run the following command:

           sudo dnf config-manager --set-enabled codeready-builder-for-rhel-9-rhui-rpms

      3. Install EMS:

         ./forticlientems_7.4.5.2111.M.bin -- --db_host "<ip/fqdn node 1>,<ip/fqdn node 2>" --db_user postgres --db_pass postgres --skip_db_install --skip_db_deploy --allowed_hosts '*' --enable_remote_https

         Replace <ip/fqdn xxxx> with the correct IP address or FQDN of the corresponding DB node.

         Run the installer to and from any directory other than /tmp. Running the installer to or from /tmp causes issues.

      4. After installation completes, check that EMS services are running by entering the following command:

         systemctl --all --type=service | grep -E 'fcems|apache|redis|postgres'

   4. After installation on both nodes completes, do the following:

      1. On the primary node, log in to EMS. License EMS as Starting FortiClient EMS and logging in describes. EMS HA requires a single license for the primary node and the secondary node(s). You only need to add the license to the primary node.
      2. Go to System Settings > EMS Settings.
      3. Under Shared Settings, confirm that Listen on IP is set to All.
      4. Enable Use FQDN.
      5. Enter the desired FQDN.

         

      6. In the Custom hostname field, enter a virtual IP address (VIP) that is configured in the FortiGate load balancer (LB) as the VIP for EMS. In this example, the VIP is 172.16.1.50.

      7. Configure the High Availability Keep Alive Internal field with a value between 5 and 30 seconds.
      8. Go to Dashboard > Status. Confirm that the System Information widget displays that EMS is running in HA mode. If running in HA mode, the widget also lists the HA primary and secondary nodes and their statuses.

6. Configure a FortiGate as an LB for EMS HA:

   1. Create a health check:
      1. Go to Policy & Objects > Health Check. Click Create New.

      2. For Type, select TCP.

      3. In the Interval field, enter 10.
      4. In the Timeout field, enter 2.
      5. In the Retry field, enter 3.

      6. In the Port field, enter 8013. Click OK.

   2. Create a virtual server:
      1. Go to Policy & Objects and create a virtual server.
      2. Configure the fields as follows:

         Field	Value
         Virtual server IP	VIP that you configured in step 4.f. In this example, the VIP is 172.16.1.50.
         Virtual server port	10443
         Load Balancing method	First Alive
         Health check	Monitor that you configured.
         Network Type	TCP

      3. Under Real Servers, select Create New.
      4. In the IPv4 address field, enter the primary EMS node IP address. In this example, it is 192.168.1.10.
      5. In the Port field, enter 10443.
      6. In the Max connections field, enter 0.
      7. For Mode, select Active.
      8. Create a real server for the secondary EMS node. Click Save.
   3. Repeat steps i-ix to create five additional virtual servers. The additional servers use ports 443, 8013, 8015, 8443, and 8871, but otherwise have identical settings to the first virtual server created. If you have enabled Chromebook management, create a virtual server for port 8443. Similarly, if you require importing an ACME certificate, create a virtual server for port 80.
   4. Create a security policy that includes the LB virtual server as a destination address:
      1. Go to Policy & Objects > Firewall Policy.
      2. Click Create New.
      3. Configure the Incoming Interface and Outgoing Interface fields. The outgoing interface connects to the primary EMS node.
      4. For Source, select all.
      5. In the Destination field, select ports 10443, 443, 8013, 8015, 8443, and 8871.
      6. For Service, select ALL.
      7. For Inspection Mode, select Proxy-based.
      8. Save the policy.
      9. If the EMS nodes are in different subnets, repeat these steps to configure a policy for the secondary EMS node. In this example, the nodes are in the same subnet, so you do not need to add a separate policy for the secondary EMS.

7. After the FortiGate LB configuration is complete, you can access EMS using the VIP configured in the FortiGate LB. If after initially installing EMS 7.4.5 you need to upgrade to a newer build, repeat steps 4.a.-c. with the new installation file.

To upgrade a previously installed EMS deployment with Postgres HA using EMS PostgreSQL HA Docker to 7.4.5:

If you are upgrading a previously installed EMS deployment with Postgres HA using EMS PostgreSQL HA (Bitnami) Docker from 7.4.3 or an earlier 7.4 version to 7.4.5, you must use the following procedure to ensure that your Postgres instances are upgraded without incurring data loss.

To upgrade your EMS nodes in a high availability (HA) cluster to 7.4.5, see Upgrading EMS in HA in the EMS Administration Guide.

1. Confirm that replication extensions are enabled on each node by running the following:

   docker exec -ti pg-X psql -U postgres -c "ALTER SYSTEM SET shared_preload_libraries TO pg_stat_statements, repmgr;"

   Replace pg-x with the actual Docker container name on each node. For example, the Docker container names may be pg-1, pg-2, and so on.

2. Stop the containers on each node:

   docker stop pg-X

   Replace pg-x with the actual Docker container name on each node. For example, the Docker container names may be pg-1, pg-2, and so on.

3. Download the new image from the Fortinet Support portal and load it on the new primary node:

   docker load -i forticlientems_7.4.5.2111.M_postgres-ha.tar.gz

4. Create a new instance (pgn-1 in this example) that points to the existing primary volume:

   docker run --restart always --detach --name pgn-1 -p 5432:5432 \
     --env POSTGRESQL_PASSWORD=<pg_password> \
     --env POSTGRESQL_DAEMON_USER=<pg_username> \
     --env REPMGR_PASSWORD=<pg_password> \
     --env REPMGR_PRIMARY_HOST=pgn-1 \
     --env REPMGR_PRIMARY_PORT=5432 \
     --env REPMGR_PARTNER_NODES=pgn-1,pgn-2:5432 \
     --env REPMGR_NODE_NAME=pgn-1 \
     --env REPMGR_NODE_NETWORK_NAME=pgn-1 \
     --env REPMGR_PORT_NUMBER=5432 \
     --env REPMGR_DEGRADED_MONITORING_TIMEOUT=-1 \
     --env REPMGR_NODE_PRIORITY=100 \
     --env REPMGR_UPGRADE_EXTENSION=yes \
     --volume pg_1_data:/fortinet/postgresql \
     --volume <path to conf.d>:/fortinet/postgresql/conf/conf.d \
     --shm-size 1g \
     --add-host pgn-1:<ip/fqdn node 1> \
     --add-host pgn-2:<ip/fqdn node 2> \
   ems-postgres-ha

   Replace <pg_password>, <pg_username>, and <path to conf.d> with your actual Postgres password, username, and path to conf.d file that you created earlier in the PostgreSQL configuration (such as /home/ems/conf.d). Replace <ip/fqdn xxxx> with the correct IP or FQDN to the corresponding PostgreSQL node.

5. Register the new primary node by updating the extensions to recognize the new node configurations:

   docker exec -it pgn-1 /opt/fortinet/scripts/postgresql-repmgr/entrypoint.sh repmgr -f /opt/fortinet/repmgr/conf/repmgr.conf primary register --force

6. Download the new image from the Fortinet Support portal and load it on the new secondary node:

   docker load -i forticlientems_7.4.5.2111.M_postgres-ha.tar.gz

7. Create a new instance (pgn-2 in this example) that points to the existing secondary volume:

   docker run --restart always --detach --name pgn-2 -p 5432:5432 \
     --env POSTGRESQL_PASSWORD=<pg_password> \
     --env POSTGRESQL_DAEMON_USER=<pg_username> \
     --env REPMGR_PASSWORD=<pg_password> \
     --env REPMGR_PRIMARY_HOST=pgn-1 \
     --env REPMGR_PRIMARY_PORT=5432 \
     --env REPMGR_PARTNER_NODES=pgn-1,pgn-2:5432 \
     --env REPMGR_NODE_NAME=pgn-2 \
     --env REPMGR_NODE_NETWORK_NAME=pgn-2 \
     --env REPMGR_PORT_NUMBER=5432 \
     --env REPMGR_DEGRADED_MONITORING_TIMEOUT=-1 \
     --env REPMGR_NODE_PRIORITY=100 \
     --env REPMGR_UPGRADE_EXTENSION=yes \
     --volume pg_1_data:/fortinet/postgresql \
     --volume <path to conf.d>:/fortinet/postgresql/conf/conf.d \
     --shm-size 1g \
     --add-host pgn-1:<ip/fqdn node 1> \
     --add-host pgn-2:<ip/fqdn node 2> \
   ems-postgres-ha

   Replace <pg_password>, <pg_username>, and <path to conf.d> with your actual Postgres password, username, and path to conf.d file that you created earlier in the PostgreSQL configuration (such as /home/ems/conf.d). Replace <ip/fqdn xxxx> with the correct IP or FQDN to the corresponding PostgreSQL node.

To validate the EMS HA configuration:

1. Go to Deployment & Installers > Manage Deployment. Create a deployment package to deploy FortiClient to endpoints. See Adding a FortiClient installer.
2. On an endpoint, download the deployment package from the download link.
3. Install FortiClient on the endpoint.
4. Ensure that FortiClient can register to the EMS server successfully using the FQDN.
5. Simulate HA by stopping FortiClient Endpoint Management Server Monitor Service on the primary EMS node. Ensure that the secondary EMS node is now the primary.
6. Ensure that FortiClient can still register to the EMS successfully using the FQDN.