Deploying FortiGate-VM active-passive HA AWS between multiple zones

This guide provides sample configuration of active-passive FortiGate-VM high availability (HA) on AWS between multiple zones.

You can configure FortiGate's native HA feature without using an AWS supplementary mechanism with two FortiGate instances: one acting as the master/primary node and the other as the slave/secondary node, located in two different availability zones (AZs) within a single VPC.

This is called "unicast HA" specific to the AWS environment in comparison to an equivalent feature that physical FortiGate units provide. The FortiGates run heartbeats between dedicated ports and synchronize OS configurations. When the primary node fails, the secondary node takes over as the primary node so endpoints continue to communicate with external resources over the FortiGate.

This feature is important because it solves a critical issue of HA, which is the ability to recover in the event of a catastrophic failure. In the case that both FortiGates are located in the same Availability Zone and that AZ happens to fail, then both FortiGates would go down and HA would be useless. Thus, there is a need to support HA configuration where both FortiGates are in separate AZs.

These paired FortiGate instances act as a single logical instance and share interface IP addressing. The main benefits of this solution are:

• Fast failover of FortiOS and AWS SDN without external automation/services
• Automatic AWS SDN updates to elastic IP addresses (EIP) and route targets
• Native FortiOS configuration synchronization
• Ease of use as the cluster is treated as single logical FortiGate

The following depicts the network topology for this sample deployment:

The following lists the IP address assignments for this sample deployment for FortiGate A:

Port	AWS primary address	Subnet
port1	10.0.0.11	10.0.0.0/24 EIP
port2	10.0.1.11	10.0.1.0/24
port3	10.0.2.11	10.0.2.0/24
port4	10.0.3.11	10.0.3.0/24 EIP

The following lists the IP address assignments for this sample deployment for FortiGate B:

Port	AWS primary address	Subnet
port1	10.0.10.11	10.0.10.0/24 EIP
port2	10.0.11.11	10.0.11.0/24
port3	10.0.12.11	10.0.12.0/24
port4	10.0.13.11	10.0.13.0/24 EIP

IPsec VPN phase 1 configuration does not synchronize between primary and secondary FortiGates across AZs. Phase 2 configuration does synchronize.

To check the prerequisites:

• Ensure that two FortiGates exist in the same VPC but different AZs. The two FortiGates must also have the same FortiOS build (FGT_VM64_AWS or FGT_VM64_AWSONDEMAND) installed.
• If using FGT_VM64_AWS, ensure that both FortiGates have valid licenses.
• The following summarizes minimum sufficient IAM roles for this deployment:

  {

  "Version": "2012-10-17",

  "Statement": [

  {

  "Action": [

  "ec2:Describe*",

  "ec2:AssociateAddress",

  "ec2:AssignPrivateIpAddresses",

  "ec2:UnassignPrivateIpAddresses",

  "ec2:ReplaceRoute"

  ],

  "Resource": "*",

  "Effect": "Allow"

  }

  ]

  }

To configure FortiGate-VM HA in AWS:

1. In the AWS management console, create a VPC. The VPC in this example has been created with 10.0.0.0/16 CIDR.
2. Create eight subnets. In this example, the eight subnets are as follows:
   1. Four in AZ A:
      1. Public WAN: 10.0.0.0/24
      2. Internal network: 10.0.1.0/24
      3. Heartbeat network: 10.0.2.0/24
      4. Management network: 10.0.3.0/24
   2. Four in AZ B:
      1. Public WAN: 10.0.10.0/24
      2. Internal network: 10.0.11.0/24
      3. Heartbeat network: 10.0.12.0/24
      4. Management: 10.0.13.0/24
3. Create a single, open security group as shown:

   

4. Create an IAM role. The IAM role is necessary for HA failover. Ensure that the IAM role can read and write EC2 information to read, detach, and reattach network interfaces and edit routing tables.
5. Create three elastic IP addresses:
   1. One public WAN IP address. This will be attached to the instance NIC1's secondary IP address.
   2. One FortiGate A management IP address
   3. One FortiGate B management IP address
6. Create two FortiGate instances. You can use any instance type with at least four vCPUs, since the configuration requires four NICs:
   1. Configure FortiGate A:
      1. Attach the IAM role created earlier.
      2. Create the instance in the VPC created earlier and in the public WAN subnet, with no ephemeral public IP address.
      3. Configure an internal IP address of 10.0.0.11.

         

      4. Attach a security group.
   2. Configure FortiGate B by repeating the steps for FortiGate A above. For FortiGate B, configure the instance in the public WAN subnet in AZ B, and configure an internal IP address of 10.0.10.11.
   3. Attach three NICs to each FortiGate according to the IP assignment in the appropriate subnet:
      1. FortiGate A:
         1. port2 (AWS primary 10.0.1.11) (internal network)
         2. port3 (AWS primary 10.0.2.11) (Heartbeat network)
         3. port4 (AWS primary 10.0.3.11) (management network)
      2. FortiGate B:
         1. port2 (AWS primary 10.0.11.11) (internal network)
         2. port3 (AWS primary 10.0.12.11) (Heartbeat network)
         3. port4 (AWS primary 10.0.13.11) (management network)
7. Attach the two elastic IP addresses to the port1 primary IP addresses of FortiGate A and FortiGate B. This allows access to the FortiGates via SSH for configuration purposes. The default password for the FortiGates is their instance IDs. The following shows the elastic IP address assigned to FortiGate A:

   

   The following shows the elastic IP address assigned to FortiGate B:

   

To configure FortiGate A using the CLI:

Run the following commands in the FortiOS CLI on FortiGate A:

config sys glo

set hostname master

end

config system interface

edit port1

set mode static

set ip 10.0.0.11 255.255.255.0

set allowaccess https ping ssh fgfm

set alias external

next

edit port2

set mode static

set ip 10.0.1.11 255.255.255.0

set allowaccess https ping ssh fgfm

set alias internal

next

edit port3

set mode static

set ip 10.0.2.11 255.255.255.0

set allowaccess https ping ssh fgfm

set alias hasync

next

edit port4

set mode static

set ip 10.0.3.11 255.255.255.0

set allowaccess https ping ssh fgfm

set alias hamgmt

next

end

config router static

edit 1

set device port1

set gateway 10.0.0.1

next

edit 2

set device port2

set gateway 10.0.1.1

set dst 10.0.11.0/24

next

end

config firewall policy

edit 0

set name "outgoing"

set srcintf "port2"

set dstintf "port1"

set srcaddr "all"

set dstaddr "all"

set action accept

set schedule "always"

set service "ALL"

set logtraffic disable

set nat enable

next

end

config system ha

set group-name "test"

set mode a-p

set hbdev "port3" 50

set session-pickup enable

set ha-mgmt-status enable

config ha-mgmt-interfaces

edit 1

set interface "port4"

set gateway 10.0.3.1

next

end

set override disable

set priority 255

set unicast-hb enable

set unicast-hb-peerip 10.0.12.11

end

To configure FortiGate B using the CLI:

Run the following commands in the FortiOS CLI on FortiGate B:

config sys glo

set hostname slave

end

config system interface

edit port1

set mode static

set ip 10.0.10.11 255.255.255.0

set allowaccess https ping ssh fgfm

set alias external

next

edit port2

set mode static

set ip 10.0.11.11 255.255.255.0

set allowaccess https ping ssh fgfm

set alias internal

next

edit port3

set mode static

set ip 10.0.12.11 255.255.255.0

set allowaccess https ping ssh fgfm

set alias hasync

next

edit port4

set mode static

set ip 10.0.13.11 255.255.255.0

set allowaccess https ping ssh fgfm

set alias hamgmt

next

end

config router static

edit 1

set device port1

set gateway 10.0.10.1

next

edit 2

set device port2

set gateway 10.0.11.1

set dst 10.0.1.0/24

next

end

config firewall policy

edit 0

set name "outgoing"

set srcintf "port2"

set dstintf "port1"

set srcaddr "all"

set dstaddr "all"

set action accept

set schedule "always"

set service "ALL"

set logtraffic disable

set nat enable

next

end

config system ha

set group-name "test"

set mode a-p

set hbdev "port3" 50

set session-pickup enable

set ha-mgmt-status enable

config ha-mgmt-interfaces

edit 1

set interface "port4"

set gateway 10.0.13.1

next

end

set override disable

set priority 1

set unicast-hb enable

set unicast-hb-peerip 10.0.2.11

end

After completing configuration of FortiGate B, remove the EIP to the FortiGate B public IP address. You can connect to the FortiGates via the management ports instead.

To configure the routing tables in AWS:

You must configure three routing tables.

The following shows the public WAN routing table. Ensure to point the 0.0.0.0/0 CIDR to the Internet gateway:

The following shows the internal network routing table. Ensure to point the 0.0.0.0/0 CIDR to FortiGate A's port2 NIC.

The following shows the Heartbeat and management networks' routing table:

To configure a VDOM exception:

You must configure a VDOM exception to prevent interface synchronization between the two FortiGates.

config system vdom-exception

edit 1

set object system.interface

next

edit 2

set object router.static

next

edit 3

set object firewall.vip

next

end

To test FortiGate-VM HA:

1. Run get system ha status to check that the FortiGates are in sync:

   master # get sys ha stat

   HA Health Status: OK

   Model: FortiGate-VM64-AWSONDEMAND

   Mode: HA A-P

   Group: 0

   Debug: 0

   Cluster Uptime: 3 days 1:50:18

   Cluster state change time: 2019-01-31 18:20:47

   Master selected using:

   <2019/01/31 18:20:47> FGTAWS0006AB1961 is selected as the master because it has the largest value of override priority.

   <2019/01/31 18:20:47> FGTAWS0006AB1961 is selected as the master because it's the only member in the cluster.

   ses_pickup: enable, ses_pickup_delay=disable

   override: disable

   unicast_hb: peerip=10.0.12.11, myip=10.0.2.11, hasync_port='port3'

   Configuration Status:

   FGTAWS0006AB1961(updated 3 seconds ago): in-sync

   FGTAWS000B29804F(updated 4 seconds ago): in-sync

   System Usage stats:

   FGTAWS0006AB1961(updated 3 seconds ago):

   sessions=18, average-cpu-user/nice/system/idle=0%/0%/0%/100%, memory=10%

   FGTAWS000B29804F(updated 4 seconds ago):

   sessions=2, average-cpu-user/nice/system/idle=0%/0%/0%/100%, memory=10%

   HBDEV stats:

   FGTAWS0006AB1961(updated 3 seconds ago):

   port3: physical/00, up, rx-bytes/packets/dropped/errors=430368/1319/0/0, tx=560457/1280/0/0

   FGTAWS000B29804F(updated 4 seconds ago):

   port3: physical/00, up, rx-bytes/packets/dropped/errors=870505/2061/0/0, tx=731630/2171/0/0

   Master: master , FGTAWS0006AB1961, HA cluster index = 1

   Slave : slave , FGTAWS000B29804F, HA cluster index = 0

   number of vcluster: 1

   vcluster 1: work 10.0.2.11

   Master: FGTAWS0006AB1961, HA operating index = 0

   Slave : FGTAWS000B29804F, HA operating index = 1

2. Ensure that failover functions as configured:
   1. Turn on debug mode on FortiGate B:

      slave # di de en

      slave # di de application awsd -1

      Debug messages will be on for unlimited time.

   2. Shut down the primary FortiGate A. In the event of a successful failover, FortiGate B's CLI shows the following:

      slave # Become HA master

      send_vip_arp: vd root master 1 intf port1 ip 10.0.10.11

      send_vip_arp: vd root master 1 intf port2 ip 10.0.11.11

      awsd get instance id i-0b29804fd38976af4

      awsd get iam role WikiDemoHARole

      awsd get region us-east-1

      awsd get vpc id vpc-0ade7ea6e64befbfc

      awsd doing ha failover for vdom root

      awsd associate elastic ip for port1

      awsd associate elastic ip allocation eipalloc-06b849dbb0f76555f to 10.0.10.11 of eni eni-0ab045a4d6dce664a

      awsd associate elastic ip successfully

      awsd update route table rtb-0a7b4fec57feb1a21, replace route of dst 0.0.0.0/0 to eni-0c4c085477aaff8c5

      awsd update route successfully

   3. Verify on AWS that the public and internal networks' secondary IP addresses moved to the new primary FortiGate, and that the routing table changes to point to the secondary FortiGate's internal network ENI.