EMS HA with Postgres HA using Bitnami Docker

The following provides instructions for installing EMS in a high availability (HA) configuration with the PostgreSQL (Postgres) databases also in an HA configuration using Bitnami Docker.

To configure EMS HA with Postgres HA using Bitnami Docker:

1. Prepare four Ubuntu machines. Two act as Postgres database hosts and the other two are EMS hosts. If using cloned virtual machines for the EMS hosts, run the following commands on the cloned machine:

   sudo rm /etc/machine-id
   sudo rm /var/lib/dbus/machine-id
   sudo systemd-machine-id-setup

2. Run sudo -i to log in to the shell with root privileges. Perform all following steps with root privileges.
3. Configure the Postgres HA cluster:
   1. Load the EMS Postgres Bitnami Docker image in both Postgres database host nodes:
      1. Install Docker:

         apt install docker.io

      2. Download the PostgreSQL Docker image forticlientems_postgres-ha.tar.gz file from the Fortinet Support site.
      3. Load the image:

         docker load -i forticlientems_postgres-ha.tar.gz
         

      4. List the images on Docker to verify the image has been created/loaded:

         docker image ls

   2. Create the Docker cluster and network on the primary database host:

      1. Create the Docker cluster:

         docker swarm init

      2. Create the network that the Postgres cluster will use:

         docker network create --opt encrypted --driver=overlay --attachable ems_pg_ha

      3. Start the Postgres Bitnami container on the primary host. In this example, the primary host is pg-1.

         docker run --restart always --detach --network ems_pg_ha --name pg-1 -p 5432:5432 \
         --env POSTGRESQL_POSTGRES_PASSWORD=postgres \
         --env ALLOW_EMPTY_PASSWORD=yes \
         --env POSTGRESQL_DAEMON_USER=postgres \
         --env REPMGR_PASSWORD=postgres \
         --env REPMGR_PRIMARY_HOST=pg-1 \
         --env REPMGR_PRIMARY_PORT=5432 \
         --env REPMGR_PARTNER_NODES=pg-1,pg-2:5432 \
         --env REPMGR_NODE_NAME=pg-1 \
         --env REPMGR_NODE_NETWORK_NAME=pg-1 \
         --env REPMGR_PORT_NUMBER=5432 \
         --volume pg_1_data:/bitnami/postgresql \
         -it \
         ems-postgres-ha

      4. Verify if the container is running:

         docker ps –a

      5. Find a command to add the secondary node to the Postgres cluster:

         docker swarm join-token worker

   3. From the secondary database host, join the Docker cluster:
      1. Join the Docker cluster using the command that you retrieved in step 3.b.v:

         docker swarm join --token SWMTKN-1-443i6pp588vdi1qs2esszhp2229v586eqgwu1oyofhwatnm6b8-48oyb4mf5yzkzb4lwql0i14lz 172.16.1.12:2377

      2. Start the Postgres Bitnami container on the secondary host. In this example, the secondary host is pg-2:

         docker run --restart always --detach --network ems_pg_ha --name pg-2 -p 5432:5432 \
         --env POSTGRESQL_POSTGRES_PASSWORD=postgres \
         --env ALLOW_EMPTY_PASSWORD=yes \
         --env POSTGRESQL_DAEMON_USER=postgres \
         --env REPMGR_PASSWORD=postgres \
         --env REPMGR_PRIMARY_HOST=pg-1 \
         --env REPMGR_PRIMARY_PORT=5432 \
         --env REPMGR_PARTNER_NODES=pg-1,pg-2:5432 \
         --env REPMGR_NODE_NAME=pg-2 \
         --env REPMGR_NODE_NETWORK_NAME=pg-2 \
         --env REPMGR_PORT_NUMBER=5432 \
         --volume pg_2_data:/bitnami/postgresql \
         -it \
         ems-postgres-ha

      3. Verify if the container is running:

         docker ps –a

   4. From the primary database node, verify the cluster status:

      docker exec -it pg-1 bash /opt/bitnami/scripts/postgresql-repmgr/entrypoint.sh repmgr -f /opt/bitnami/repmgr/conf/repmgr.conf service status

4. After the database cluster is up and running, configure EMS HA:

   1. On both nodes, do the following:
      1. Download the forticlientems_7.4.0.XXXX.bin file from https://support.fortinet.com.
      2. Change permissions and add execute permissions to the installation file:

         chmod +x forticlientems_7.4.0.XXXX.bin

   2. On the primary node, install EMS:
      1. Set umask to 022 on file /etc/login.defs if the existing umask setting is more restrictive.
      2. Install EMS:

         ./forticlientems_7.4.0.1745.bin -- --db_host "172.16.1.12,172.16.1.15" --db_user postgres --db_pass postgres --skip_db_install --skip_event_feature --allowed_hosts '*' --enable-remote-https

         Run the installer to/from any directory other than /tmp. Running the installer to/from /tmp causes issues.

         In the example, db_host contains IP addresses for both database nodes. Replace the IP addresses with your database server IP addresses or FQDNs.

      3. After installation completes, check that all EMS services are running by entering the following command:

         systemctl --all --type=service | grep -E 'fcems|apache|redis|postgres'

         

         The output shows that postgresql.service status displays as exited. This is the expected status. EMS does not create this service, which only exists to pass commands to version-specific Postgres services. It displays as part of the output as the command filters for all services that contain "postgres" in the name.

   3. On the secondary node, install EMS:
      1. Set umask to 022 if the existing umask setting is more restrictive.
      2. Install EMS:

         ./forticlientems_7.4.0.XXXX.bin -- --db_host "172.16.1.12,172.16.1.15" --db_user postgres --db_pass postgres --skip_db_install --skip_db_deploy --skip_event_feature --allowed_hosts '*' --enable-remote-https

         Run the installer to/from any directory other than /tmp. Running the installer to/from /tmp causes issues.

      3. After installation completes, check that EMS services are running by entering the following command. On the secondary EMS, only fcems_monitor, fcems_pgbouncer, fcems_wspgbouncer, and redis-server services should be running:

         systemctl --all --type=service | grep -E 'fcems|apache|redis|postgres'

   4. After installation on both nodes is complete, access the EMS GUI from the primary node using a browser by going to https://localhost. Complete initial configuration for EMS by doing the following:

      1. Set the password for the default administrator. See Starting FortiClient EMS and logging in.

      2. Configure the EMS FQDN and remote access. See Configuring EMS after installation.

      3. License EMS. See Licensing FortiClient EMS.

      4. Confirm that Listen on IP is set to All. See Configuring EMS settings.

   5. Go to System Settings > EMS Settings.

   6. In the Custom hostname field, enter a virtual IP address (VIP) that is configured in the FortiGate load balancer (LB) as the VIP for EMS. In this example, the VIP is 172.16.1.50.

5. Configure a FortiGate as an LB for EMS HA:

   1. Create a health check:
      1. Go to Policy & Objects > Health Check. Click Create New.

      2. For Type, select TCP.

      3. In the Interval field, enter 10.
      4. In the Timeout field, enter 2.
      5. In the Retry field, enter 3.

      6. In the Port field, enter 8013. Click OK.

   2. Create a virtual server:
      1. Go to Policy & Objects and create a virtual server.
      2. Configure the fields as follows:

         Field	Value
         Virtual server IP	VIP that you configured in step 4.f. In this example, the VIP is 172.16.1.50.
         Virtual server port	10443
         Load Balancing method	First Alive
         Health check	Monitor that you configured.
         Network Type	TCP

      3. Under Real Servers, select Create New.
      4. In the IPv4 address field, enter the primary EMS node IP address. In this example, it is 192.168.1.10.
      5. In the Port field, enter 10443.
      6. In the Max connections field, enter 0.
      7. For Mode, select Active.
      8. Create a real server for the secondary EMS node. Click Save.
   3. Repeat steps i-ix to create five additional virtual servers. The additional servers use ports 443, 8013, 8015, 8443, and 8871, but otherwise have identical settings to the first virtual server created. If you have enabled Chromebook management, create a virtual server for port 8443. Similarly, if you require importing an ACME certificate, create a virtual server for port 80.
   4. Create a security policy that includes the LB virtual server as a destination address:
      1. Go to Policy & Objects > Firewall Policy.
      2. Click Create New.
      3. Configure the Incoming Interface and Outgoing Interface fields. The outgoing interface connects to the primary EMS node.
      4. For Source, select all.
      5. In the Destination field, select ports 10443, 443, 8013, 8015, 8443, and 8871.
      6. For Service, select ALL.
      7. For Inspection Mode, select Proxy-based.
      8. Save the policy.
      9. If the EMS nodes are in different subnets, repeat these steps to configure a policy for the secondary EMS node. In this example, the nodes are in the same subnet, so you do not need to add a separate policy for the secondary EMS.

6. After the FortiGate LB configuration is complete, you can access EMS using the VIP configured in the FortiGate LB. If after initially installing EMS 7.4.0 you need to upgrade to a newer build, repeat steps 4.a.-c. with the new installation file.