EMS HA installation with native Postgres HA without Docker

This guide gives instructions on how to set up a PostgreSQL (Postgres) high availability (HA) cluster when Postgres is natively installed (not Postgres containers) and install EMS HA.

Postgres HA requires at least three servers or virtual machines (VM):

Node	Purpose
Postgres node 1	Starts as the primary node
Postgres node 2	Starts as the standby node
Postgres witness 1	Serves as a witness to prevent split-brain scenarios where both nodes may self-promote to become the primary and clash

The configuration also requires two servers or VMs for EMS nodes.

To configure Postgres node 1:

1. Install Postgres 15:

   sudo apt install -y --no-install-recommends curl ca-certificates
   sudo install -d /usr/share/postgresql-common/pgdg
   sudo curl -o /usr/share/postgresql-common/pgdg/apt.postgresql.org.asc --fail https://www.postgresql.org/media/keys/ACCC4CF8.asc
   sudo sh -c 'echo "deb [signed-by=/usr/share/postgresql-common/pgdg/apt.postgresql.org.asc] https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
   sudo apt update
   sudo apt install -y postgresql-15

2. Install Replication Manager. Postgres recommends this replication tool:

   sudo apt-get install apt-transport-https
   sudo sh -c 'echo "deb https://apt.2ndquadrant.com/ $(lsb_release -cs)-2ndquadrant main" > /etc/apt/sources.list.d/2ndquadrant.list'
   sudo apt-get install curl ca-certificates
   curl https://apt.2ndquadrant.com/site/keys/9904CD4BD6BAF0C3.asc| sudo apt-key add
   sudo apt-get update
   sudo apt-get install -y postgresql-15-repmgr

3. Install EMS custom extensions (ems_pg_extensions.tar.gz) for Postgres. You can download the extensions from the Fortinet Support site. Use the ems_pg_extensions.tar.gz file under the EMS 7.4.0 page to install extensions for EMS 7.4.1:

   sudo tar zxvf ems_pg_extensions.tar.gz -C /

4. Create the symmetric key required that the custom extension requires. Copy the value in symmetric_key.txt as you must share it with the other nodes:

   sh -c 'head -c 20 /dev/urandom | md5sum | head -c 20;' |sudo tee /var/lib/postgresql/15/symmetric_key.txt > /dev/null
   cat /var/lib/postgresql/15/symmetric_key.txt

5. Set the password for the Postgres Linux user:

   sudo passwd postgres

6. Modify postgresql.conf to add replication parameters. Change the following settings on /etc/postgresql/15/main/postgresql.conf with the indicated values:

   listen_addresses = ‘localhost, <ip of the PostgreSQL node1 server>’
   max_wal_senders = 10
   max_replication_slots = 10
   wal_level = replica
   hot_standby = on
   archive_mode = on
   archive_command = '/bin/true'
   shared_preload_libraries = 'repmgr'

7. Set up Postgres access to the replication user. Add the following to /etc/postgresql/15/main/pg_hba.conf:

   host all all 0.0.0.0/0 trust
   local replication repmgr trust
   host replication repmgr 127.0.0.1/32 trust
   host replication repmgr <node1 ip>/32 trust
   host replication repmgr <node2 ip>/32 trust
   host replication repmgr <witnesse ip>/32 trust
   local repmgr repmgr trust
   host repmgr repmgr 127.0.0.1/32 trust
   host repmgr repmgr <node1 ip>/32 trust
   host repmgr repmgr <node2 ip>/32 trust
   host repmgr repmgr <witness ip>/32 trust

   Replace <node1 ip>, <node2 ip>, and <witness ip> accordingly.

8. Create the repmgr user and database. Run the following to create the repmgr user and database on Postgres:

   sudo -u postgres createuser -s repmgr
   sudo -u postgres createdb repmgr -O repmgr

9. Restart the postgresql service:

   sudo systemctl restart postgresql@15-main.service

To configure Postgres node 2:

1. Install Postgres 15:

   sudo apt install -y --no-install-recommends curl ca-certificates
   sudo install -d /usr/share/postgresql-common/pgdg
   sudo curl -o /usr/share/postgresql-common/pgdg/apt.postgresql.org.asc --fail https://www.postgresql.org/media/keys/ACCC4CF8.asc
   sudo sh -c 'echo "deb [signed-by=/usr/share/postgresql-common/pgdg/apt.postgresql.org.asc] https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
   sudo apt update
   sudo apt install -y postgresql-15

2. Install Replication Manager. Postgres recommends this replication tool:

   sudo apt-get install apt-transport-https
   sudo sh -c 'echo "deb https://apt.2ndquadrant.com/ $(lsb_release -cs)-2ndquadrant main" > /etc/apt/sources.list.d/2ndquadrant.list'
   sudo apt-get install curl ca-certificates
   curl https://apt.2ndquadrant.com/site/keys/9904CD4BD6BAF0C3.asc| sudo apt-key add
   sudo apt-get update
   sudo apt-get install -y postgresql-15-repmgr

3. Install EMS custom extensions (ems_pg_extensions.tar.gz) for Postgres. You can download the extensions from the Fortinet Support site:

   sudo tar zxvf ems_pg_extensions.tar.gz -C /

4. Create the symmetric key required that the custom extension requires. The value must be the one copied from the symmetric key from the PostgreSQL node 1:

   sh -c 'echo "<value copied from PostgreSQL node 1>"|head -c 20 | head -c 20;'|sudo tee /var/lib/postgresql/15/symmetric_key.txt > /dev/null

5. Set the password for the Postgres Linux user:

   sudo passwd postgres

6. Modify postgresql.conf to add replication parameters. Change the following settings on /etc/postgresql/15/main/postgresql.conf with the indicated values:

   listen_addresses = ‘localhost, <ip of the PostgreSQL node2 server>
   max_wal_senders = 
   max_replication_slots = 10
   wal_level = replica
   hot_standby = on
   archive_mode = on
   archive_command = '/bin/true'
   shared_preload_libraries = 'repmgr'

7. Set up Postgres access to the replication user. Add the following to /etc/postgresql/15/main/pg_hba.conf:

   host all all 0.0.0.0/0 trust
   local replication repmgr trust
   host replication repmgr 127.0.0.1/32 trust
   host replication repmgr <node1 ip>/32 trust
   host replication repmgr <node2 ip>/32 trust
   host replication repmgr <witnesse ip>/32 trust
   local repmgr repmgr trust
   host repmgr repmgr 127.0.0.1/32 trust
   host repmgr repmgr <node1 ip>/32 trust
   host repmgr repmgr <node2 ip>/32 trust
   host repmgr repmgr <witness ip>/32 trust

   Replace <node1 ip>, <node2 ip>, and <witness ip> accordingly.

8. Restart the postgresql service:

   sudo systemctl restart postgresql@15-main.service

To set up SSH access between Postgres nodes 1 and 2:

When the CLI prompts you, enter the postgres password.

1. On Postgres node 1, run the following:

   sudo -H -u postgres bash -c “ssh-keygen -t rsa -f ~/.ssh/id_rsa -N ‘’”
   sudo -H -u postgres bash -c “ssh-copy-id -i ~/.ssh/id_rsa.pub -o StrictHostKeyChecking=no 'postgres@<node2 ip>’”

2. On Postgres node 2, run the following:

   sudo -H -u postgres bash -c “ssh-keygen -t rsa -f ~/.ssh/id_rsa -N ‘’”
   sudo -H -u postgres bash -c “ssh-copy-id -i ~/.ssh/id_rsa.pub -o StrictHostKeyChecking=no 'postgres@<node2 ip>’”
   

To register Postgres node1 to the cluster:

1. Create the /etc/repmgr.conf file on Postgres node1:

   cluster='emscluster'
   node_id=1
   node_name=postgresqlnode1
   conninfo='host=<PostgresSQL node1 ip> user=repmgr dbname=repmgr connect_timeout=2' data_directory='/var/lib/postgresql/15/main/'
   pg_bindir='/usr/bin/'
   
   log_file='/var/log/repmgr/repmgr.log'
   log_level=DEBUG
   
   failover=automatic
   promote_command='/usr/bin/repmgr standby promote -f /etc/repmgr.conf'
   follow_command='/usr/bin/repmgr standby follow -f /etc/repmgr.conf --upstream-node-id=%n'

   Replace <node1 ip> accordingly.

2. Create the log directory for repmgr:

   sudo mkdir /var/log/repmgr
   sudo chown -R postgres /var/log/repmgr

3. Register the node as the primary:

   sudo -u postgres /usr/bin/repmgr -f /etc/repmgr.conf primary register

4. Confirm registration:

   sudo -u postgres /usr/bin/repmgr -f /etc/repmgr.conf cluster show

5. Create a systemd service config for repmgr by creating the file:

   [Unit]
   Description=PostgreSQL Replication Manager Daemon
   After=network.target postgresql.service
   
   [Service]
   Type=forking
   User=postgres
   ExecStart=/usr/bin/repmgrd -f /etc/repmgr.conf --daemonize
   PIDFile=/tmp/repmgrd.pid
   ExecStop=/bin/kill -s TERM $(cat /tmp/repmgrd.pid)
   ExecReload=/bin/kill -s HUP $(cat /tmp/repmgrd.pid)
   Restart=on-failure
   LimitNOFILE=16384
   
   [Install] WantedBy=multi-user.target

6. Edit /etc/default/repmgrd to change default settings. The following provides an example:

   # default settings for repmgrd. This file is source by /bin/sh from
   # /etc/init.d/repmgrd
   
   # disable repmgrd by default so it won't get started upon installation
   # valid values: yes/no

   REPMGRD_ENABLED=yes

   # configuration file (required)
   REPMGRD_CONF="/etc/repmgr.conf"
   
   # additional options
   #REPMGRD_OPTS=""
   
   # user to run repmgrd as
   #REPMGRD_USER=postgres
   
   # repmgrd binary
   #REPMGRD_BIN=/usr/bin/repmgrd
   
   # pid file REPMGRD_PIDFILE=/tmp/repmgrd.pid

   REPMGR_ENABLED changed to yes, REPMGRD_CONF was uncommented and value changed to "/tmp/repmgrd.conf" and REPMGRD_PIDFILE was uncommented and value changed to /tmp/repmgrd.pid.

7. Kill any previous instance of repmgrd that may be running:

   ps -ef|grep "bin/repmgrd"|grep -v grep|xargs -t -i sudo kill {}

8. Enable and start the repmgrd service:

   sudo systemctl enable repmgrd.service && sudo systemctl start repmgrd

9. Create a monitor and reconciliation service to support automatic failover. Create the script /var/lib/postgresql/node_monitor.sh with ownership to the Postgres user:

   #!/bin/bash
   
   # Script to detect a new primary and rejoin the cluster as standby if necessary
   
   # Configuration variables
   REPMGR_CONF="/etc/repmgr.conf"
   PG_SERVICE="postgresql@15-main"
   CHECK_INTERVAL=10 # Check every 10 seconds
   REPMGR_CMD="/usr/bin/repmgr"
   PG_CTL="/usr/bin/pg_ctl"
   NODE_NAME="node1"
   
   # Function to log messages
   log_message() {
   	echo "$(date '+%Y-%m-%d %H:%M:%S') - $1"
   }
   
   # Main loop
   while true; do
   # Ensure PostgreSQL is running before checking the cluster status
   systemctl is-active --quiet $PG_SERVICE
   PG_STATUS=$?
   
   if [ $PG_STATUS -eq 0 ]; then
   	# Check if there is another primary in the cluster
   	OTHER_PRIMARY=$(sudo -u postgres $REPMGR_CMD -f $REPMGR_CONF cluster show 2>/dev/null | grep -v "$NODE_NAME" | grep "primary"|grep "running as primary"|sed -n 's/.*host=\([^ ]*\).*/\1/p')
   
   	# Get current node's role
   	CURRENT_ROLE=$(sudo -u postgres $REPMGR_CMD -f $REPMGR_CONF node check --role 2>/dev/null | grep -oP '(?<=\().*(?=\))'|awk '{print $NF}')
   
   	if [[ ! -z "$OTHER_PRIMARY" && "$CURRENT_ROLE" == "primary" ]]; then
   		log_message "Another primary detected in the cluster. Attempting to rejoin as standby."
   
   		# Ensure PostgreSQL is stopped before rejoining
   		systemctl stop $PG_SERVICE
   		sleep 5
   
   		# Ensure PostgreSQL is fully stopped
   		systemctl is-active --quiet $PG_SERVICE
   		if [ $? -eq 0 ]; then
   			log_message "Failed to stop PostgreSQL service. Skipping rejoin."
   		else
   			# Clone the data from the current primary and rejoin as standby
   			sudo -u postgres $REPMGR_CMD -h $OTHER_PRIMARY -U repmgr -d repmgr -f $REPMGR_CONF standby clone --force 2>/dev/null && \
   			sudo systemctl start $PG_SERVICE && \
   			sudo -u postgres $REPMGR_CMD -f $REPMGR_CONF standby register --force 2>/dev/null
   
   			if [ $? -eq 0 ]; then
   				log_message "Node rejoined as standby successfully."
   			else
   				log_message "Failed to rejoin node as standby."
   			fi
   		fi
   	elif [[ -z "$CLUSTER_STATUS" ]]; then
   		log_message "No other primary detected, or the current node is already a standby. Nothing to do."
   		fi
   	else
   		log_message "PostgreSQL service is not running. Skipping checks."
   	fi
   	# Wait for the next check
   	sleep $CHECK_INTERVAL
   done

10. Create a monitor service by creating the file /etc/systemd/system/pg_node_monitor.service:

    [Unit]
    Description=Automatic Rejoin Standby Service
    After=network.target postgresql.service
    
    [Service]
    Type=simple
    User=root
    ExecStart=/var/lib/postgresql/node_monitor.sh
    Restart=on-failure
    RestartSec=5s
    
    [Install]
    WantedBy=multi-user.target

11. Enable and start the monitor service:

    sudo systemctl enable pg_node_monitor && sudo systemctl start pg_node_monitor

To register Postgres node2 to the cluster:

1. Create the /etc/repmgr.conf file on Postgres node2:

   cluster='emscluster'
   node_id=2
   node_name=postgresqlnode2
   conninfo='host=<PostgresSQL node2 ip> user=repmgr dbname=repmgr connect_timeout=2' data_directory='/var/lib/postgresql/15/main/'
   pg_bindir='/usr/bin/'
   
   log_file='/var/log/repmgr/repmgr.log'
   log_level=DEBUG
   
   failover=automatic
   promote_command='/usr/bin/repmgr standby promote -f /etc/repmgr.conf'
   follow_command='/usr/bin/repmgr standby follow -f /etc/repmgr.conf --upstream-node-id=%n'

   Replace <node2 ip> accordingly.

2. Create the log directory for repmgr:

   sudo mkdir /var/log/repmgr
   sudo chown -R postgres /var/log/repmgr

3. Stop the Postgres service:

   sudo systemctl stop postgresql

4. Create a clone from Postgres node1:

   sudo -u postgres repmgr -h <node1 ip> -U repmgr -d repmgr -f /etc/repmgr.conf standby clone --force

   Replace <node1 ip> accordingly.

5. Restart the Postgres service:

   sudo systemctl start postgresql

6. Register the node as the standby:

   sudo -u postgres /usr/bin/repmgr -f /etc/repmgr.conf standby register

7. Confirm registration:

   sudo -u postgres /usr/bin/repmgr -f /etc/repmgr.conf cluster show

8. Create a systemd service config for repmgr by creating file /etc/systemd/system/repmgrd.service:

   [Unit]
   Description=PostgreSQL Replication Manager Daemon
   After=network.target postgresql.service
   
   [Service]
   Type=forking
   User=postgres
   ExecStart=/usr/bin/repmgrd -f /etc/repmgr.conf --daemonize
   PIDFile=/tmp/repmgrd.pid
   ExecStop=/bin/kill -s TERM $(cat /tmp/repmgrd.pid)
   ExecReload=/bin/kill -s HUP $(cat /tmp/repmgrd.pid)
   Restart=on-failure
   LimitNOFILE=16384
   
   [Install] WantedBy=multi-user.target

9. Edit /etc/default/repmgrd to change default settings. The following provides an example:

   # default settings for repmgrd. This file is source by /bin/sh from
   # /etc/init.d/repmgrd
   
   # disable repmgrd by default so it won't get started upon installation
   # valid values: yes/no

   REPMGRD_ENABLED=yes

   # configuration file (required)
   REPMGRD_CONF="/etc/repmgr.conf"
   
   # additional options
   #REPMGRD_OPTS=""
   
   # user to run repmgrd as
   #REPMGRD_USER=postgres
   
   # repmgrd binary
   #REPMGRD_BIN=/usr/bin/repmgrd
   
   # pid file REPMGRD_PIDFILE=/tmp/repmgrd.pid

   REPMGR_ENABLED changed to yes, REPMGRD_CONF was uncommented and value changed to "/tmp/repmgrd.conf" and REPMGRD_PIDFILE was uncommented and value changed to /tmp/repmgrd.pid.

10. Kill any previous instance of repmgrd that may be running:

    ps -ef|grep "bin/repmgrd"|grep -v grep|xargs -t -i sudo kill {}

11. Enable and start the repmgrd service:

    sudo systemctl enable repmgrd.service && sudo systemctl start repmgrd

12. Create a monitor and reconciliation service to support automatic failover. Create the script /var/lib/postgresql/node_monitor.sh with ownership to the Postgres user:

    #!/bin/bash
    
    # Script to detect a new primary and rejoin the cluster as standby if necessary
    
    # Configuration variables
    REPMGR_CONF="/etc/repmgr.conf"
    PG_SERVICE="postgresql@15-main"
    CHECK_INTERVAL=10 # Check every 10 seconds
    REPMGR_CMD="/usr/bin/repmgr"
    PG_CTL="/usr/bin/pg_ctl"
    NODE_NAME="node2"
    
    # Function to log messages
    log_message() {
    	echo "$(date '+%Y-%m-%d %H:%M:%S') - $1"
    }
    
    # Main loop
    while true; do
    # Ensure PostgreSQL is running before checking the cluster status
    systemctl is-active --quiet $PG_SERVICE
    PG_STATUS=$?
    
    if [ $PG_STATUS -eq 0 ]; then
    	# Check if there is another primary in the cluster
    	OTHER_PRIMARY=$(sudo -u postgres $REPMGR_CMD -f $REPMGR_CONF cluster show 2>/dev/null | grep -v "$NODE_NAME" | grep "primary"|grep "running as primary"|sed -n 's/.*host=\([^ ]*\).*/\1/p')
    
    	# Get current node's role
    	CURRENT_ROLE=$(sudo -u postgres $REPMGR_CMD -f $REPMGR_CONF node check --role 2>/dev/null | grep -oP '(?<=\().*(?=\))'|awk '{print $NF}')
    
    	if [[ ! -z "$OTHER_PRIMARY" && "$CURRENT_ROLE" == "primary" ]]; then
    		log_message "Another primary detected in the cluster. Attempting to rejoin as standby."
    
    		# Ensure PostgreSQL is stopped before rejoining
    		systemctl stop $PG_SERVICE
    		sleep 5
    
    		# Ensure PostgreSQL is fully stopped
    		systemctl is-active --quiet $PG_SERVICE
    		if [ $? -eq 0 ]; then
    			log_message "Failed to stop PostgreSQL service. Skipping rejoin."
    		else
    			# Clone the data from the current primary and rejoin as standby
    			sudo -u postgres $REPMGR_CMD -h $OTHER_PRIMARY -U repmgr -d repmgr -f $REPMGR_CONF standby clone --force 2>/dev/null && \
    			sudo systemctl start $PG_SERVICE && \
    			sudo -u postgres $REPMGR_CMD -f $REPMGR_CONF standby register --force 2>/dev/null
    
    			if [ $? -eq 0 ]; then
    				log_message "Node rejoined as standby successfully."
    			else
    				log_message "Failed to rejoin node as standby."
    			fi
    		fi
    	elif [[ -z "$CLUSTER_STATUS" ]]; then
    		log_message "No other primary detected, or the current node is already a standby. Nothing to do."
    		fi
    	else
    		log_message "PostgreSQL service is not running. Skipping checks."
    	fi
    	# Wait for the next check
    	sleep $CHECK_INTERVAL
    done

13. Create a monitor service by creating the file /etc/systemd/system/pg_node_monitor.service:

    [Unit]
    Description=Automatic Rejoin Standby Service
    After=network.target postgresql.service
    
    [Service]
    Type=simple
    User=root
    ExecStart=/var/lib/postgresql/node_monitor.sh
    Restart=on-failure
    RestartSec=5s
    
    [Install]
    WantedBy=multi-user.target

14. Enable and start the monitor service:

    sudo systemctl enable pg_node_monitor && sudo systemctl start pg_node_monitor

To register Postgres witness server:

1. Install Postgres 15:

   sudo apt install -y --no-install-recommends curl ca-certificates
   sudo install -d /usr/share/postgresql-common/pgdg
   sudo curl -o /usr/share/postgresql-common/pgdg/apt.postgresql.org.asc --fail https://www.postgresql.org/media/keys/ACCC4CF8.asc
   sudo sh -c 'echo "deb [signed-by=/usr/share/postgresql-common/pgdg/apt.postgresql.org.asc] https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
   sudo apt update
   sudo apt install -y postgresql-15

2. Install Replication Manager. Postgres recommends this replication tool:

   sudo apt-get install apt-transport-https
   sudo sh -c 'echo "deb https://apt.2ndquadrant.com/ $(lsb_release -cs)-2ndquadrant main" > /etc/apt/sources.list.d/2ndquadrant.list'
   sudo apt-get install curl ca-certificates
   curl https://apt.2ndquadrant.com/site/keys/9904CD4BD6BAF0C3.asc| sudo apt-key add
   sudo apt-get update
   sudo apt-get install -y postgresql-15-repmgr

3. Modify postgresql.conf to add replication parameters. Change the following settings on /etc/postgresql/15/main/postgresql.conf with the indicated values:

   listen_addresses = ‘localhost, <ip of witness server>’
   shared_preload_libraries = 'repmgr'

4. Set up Postgres access to the replication user. Add the following to /etc/postgresql/15/main/pg_hba.conf:

   host all all 0.0.0.0/0 trust
   local replication repmgr trust
   host replication repmgr 127.0.0.1/32 trust
   host replication repmgr <node1 ip>/32 trust
   host replication repmgr <node2 ip>/32 trust
   host replication repmgr <witnesse ip>/32 trust
   local repmgr repmgr trust
   host repmgr repmgr 127.0.0.1/32 trust
   host repmgr repmgr <node1 ip>/32 trust
   host repmgr repmgr <node2 ip>/32 trust
   host repmgr repmgr <witness ip>/32 trust

   Replace <node1 ip>, <node2 ip>, and <witness ip> accordingly.

5. Restart the postgresql service:

   sudo sytemctl restart postgresql

6. Create the repmgr user and database. Run the following to create the repmgr user and database on Postgres:

   sudo -u postgres createuser -s repmgr
   sudo -u postgres createdb repmgr -O repmgr

7. Create the log directory for repmgr:

   sudo mkdir /var/log/repmgr
   sudo chown -R postgres /var/log/repmgr

8. Create the /etc/repmgr.conf file:

   cluster='emscluster'
   node_id=3
   node_name='witness1'
   conninfo='host=<witness node ip> user=repmgr dbname=repmgr connect_timeout=2'
   data_directory='/var/lib/postgresql/15/main' # Not used but required by repmgr
   pg_bindir='/usr/bin/'
   log_file='/var/log/repmgr/repmgr.log'

9. Register as witness informing the host of the primary node:

   sudo -u postgres /usr/bin/repmgr -f /etc/repmgr.conf witness register -h <node1 ip>

10. Create a systemd service config for repmgr by creating file /etc/systemd/system/repmgrd.service:

    [Unit]
    Description=PostgreSQL Replication Manager Daemon
    After=network.target postgresql.service
    
    [Service]
    Type=forking
    User=postgres
    ExecStart=/usr/bin/repmgrd -f /etc/repmgr.conf --daemonize
    PIDFile=/tmp/repmgrd.pid
    ExecStop=/bin/kill -s TERM $(cat /tmp/repmgrd.pid)
    ExecReload=/bin/kill -s HUP $(cat /tmp/repmgrd.pid)
    Restart=on-failure LimitNOFILE=16384
    
    [Install]
    WantedBy=multi-user.target

11. Edit /etc/default/repmgrd to change default settings. It should look like this:

    # default settings for repmgrd. This file is source by /bin/sh from
    # /etc/init.d/repmgrd
    # disable repmgrd by default so it won't get started upon installation
    # valid values: yes/no
    REPMGRD_ENABLED=yes
    
    # configuration file (required)
    REPMGRD_CONF="/etc/repmgr.conf"
    
    # additional options
    #REPMGRD_OPTS=""
    
    # user to run repmgrd as
    #REPMGRD_USER=postgres
    
    # repmgrd binary
    #REPMGRD_BIN=/usr/bin/repmgrd
    
    # pid file
    REPMGRD_PIDFILE=/tmp/repmgrd.pid

    REPMGR_ENABLED changed to yes, REPMGRD_CONF was uncommented and value changed to "/tmp/repmgrd.conf" and REPMGRD_PIDFILE was uncommented and value changed to /tmp/repmgrd.pid.

12. Kill any previous instance of the repmgrd that might be running:

    ps -ef|grep "bin/repmgrd"|grep -v grep|xargs -t -i sudo kill {}

13. Enable and start the repmgrd service:

    sudo systemctl enable repmgrd.service && sudo systemctl start repmgrd

To test failover:

Stop the Postgres service on the primary node or shut down the primary node. By default, the health check happens every 10 seconds. Before promoting itself to the primary node, the secondary node checks if the primary is down at least six times. Therefore, failover takes 60 seconds to happen. This is configurable but an acceptable timeout or downtime.

To configure EMS HA:

1. After the database cluster is up and running, configure EMS HA:

   1. On both nodes, do the following:
      1. Download the forticlientems_7.4.1.XXXX.bin file from https://support.fortinet.com.
      2. Change permissions and add execute permissions to the installation file:

         chmod +x forticlientems_7.4.0.XXXX.bin

   2. On the primary node, install EMS:
      1. Set umask to 022 on file /etc/login.defs if the existing umask setting is more restrictive.
      2. Install EMS:

         ./forticlientems_7.4.0.1745.bin -- --db_host "172.16.1.12,172.16.1.15" --db_user postgres --db_pass postgres --skip_db_install --allowed_hosts '*' --enable_remote_https

         Run the installer to/from any directory other than /tmp. Running the installer to/from /tmp causes issues.

         In the example, db_host contains IP addresses for both database nodes. Replace the IP addresses with your database server IP addresses or FQDNs.

      3. After installation completes, check that all EMS services are running by entering the following command:

         systemctl --all --type=service | grep -E 'fcems|apache|redis|postgres'

         

         The output shows that postgresql.service status displays as exited. This is the expected status. EMS does not create this service, which only exists to pass commands to version-specific Postgres services. It displays as part of the output as the command filters for all services that contain "postgres" in the name.

   3. On the secondary node, install EMS:
      1. Set umask to 022 if the existing umask setting is more restrictive.
      2. Install EMS:

         ./forticlientems_7.4.1.XXXX.bin -- --db_host "172.16.1.12,172.16.1.15" --db_user postgres --db_pass postgres --skip_db_install --skip_db_deploy --allowed_hosts '*' --enable_remote_https

         Run the installer to/from any directory other than /tmp. Running the installer to/from /tmp causes issues.

      3. After installation completes, check that EMS services are running by entering the following command. On the secondary EMS, only fcems_monitor, fcems_pgbouncer, fcems_wspgbouncer, and redis-server services should be running:

         systemctl --all --type=service | grep -E 'fcems|apache|redis|postgres'

   4. After installation on both nodes is complete, access the EMS GUI from the primary node using a browser by going to https://localhost. Complete initial configuration for EMS by doing the following:

      1. Set the password for the default administrator. See Starting FortiClient EMS and logging in.

      2. Configure the EMS FQDN and remote access. See Configuring EMS after installation.

      3. License EMS. See Licensing FortiClient EMS.

      4. Confirm that Listen on IP is set to All. See Configuring EMS settings.

   5. Go to System Settings > EMS Settings.

   6. In the Custom hostname field, enter a virtual IP address (VIP) that is configured in the FortiGate load balancer (LB) as the VIP for EMS. In this example, the VIP is 172.16.1.50.

2. Configure a FortiGate as an LB for EMS HA:

   1. Create a health check:
      1. Go to Policy & Objects > Health Check. Click Create New.

      2. For Type, select TCP.

      3. In the Interval field, enter 10.
      4. In the Timeout field, enter 2.
      5. In the Retry field, enter 3.

      6. In the Port field, enter 8013. Click OK.

   2. Create a virtual server:
      1. Go to Policy & Objects and create a virtual server.
      2. Configure the fields as follows:

         Field	Value
         Virtual server IP	VIP that you configured in step 4.f. In this example, the VIP is 172.16.1.50.
         Virtual server port	10443
         Load Balancing method	First Alive
         Health check	Monitor that you configured.
         Network Type	TCP

      3. Under Real Servers, select Create New.
      4. In the IPv4 address field, enter the primary EMS node IP address. In this example, it is 192.168.1.10.
      5. In the Port field, enter 10443.
      6. In the Max connections field, enter 0.
      7. For Mode, select Active.
      8. Create a real server for the secondary EMS node. Click Save.
   3. Repeat steps i-ix to create five additional virtual servers. The additional servers use ports 443, 8013, 8015, 8443, and 8871, but otherwise have identical settings to the first virtual server created. If you have enabled Chromebook management, create a virtual server for port 8443. Similarly, if you require importing an ACME certificate, create a virtual server for port 80.
   4. Create a security policy that includes the LB virtual server as a destination address:
      1. Go to Policy & Objects > Firewall Policy.
      2. Click Create New.
      3. Configure the Incoming Interface and Outgoing Interface fields. The outgoing interface connects to the primary EMS node.
      4. For Source, select all.
      5. In the Destination field, select ports 10443, 443, 8013, 8015, 8443, and 8871.
      6. For Service, select ALL.
      7. For Inspection Mode, select Proxy-based.
      8. Save the policy.
      9. If the EMS nodes are in different subnets, repeat these steps to configure a policy for the secondary EMS node. In this example, the nodes are in the same subnet, so you do not need to add a separate policy for the secondary EMS.

3. After the FortiGate LB configuration is complete, you can access EMS using the VIP configured in the FortiGate LB. If after initially installing EMS 7.4.1 you need to upgrade to a newer build, repeat steps 4.a.-c. with the new installation file.