Deploying FortiGate-VM A-P HA on AWS within one zone

This guide provides sample configuration of active-passive (A-P) FortiGate-VM high availability (HA) on AWS within one zone.

You can configure FortiGate's native HA feature without using an AWS supplementary mechanism with two FortiGate instances: one acting as the primary node and the other as the secondary node, located in two availability zones (AZs) within a single VPC.

This is called "unicast HA" specific to the AWS environment in comparison to an equivalent feature that physical FortiGate units provide. The FortiGates run heartbeats between dedicated ports and synchronize OS configurations. When the primary node fails, the secondary node takes over as the primary node so endpoints continue to communicate with external resources over the FortiGate.

These paired FortiGate instances act as a single logical instance and share interface IP addressing. The main benefits of this solution are:

• Fast failover of FortiOS and AWS SDN without external automation/services
• Automatic AWS SDN updates to elastic IP addresses (EIP) and route targets
• Native FortiOS configuration synchronization
• Ease of use as the cluster is treated as single logical FortiGate

The following depicts the network topology for this sample deployment:

The following depicts a failover event for this sample deployment:

The following lists the IP address assignments for this sample deployment for FortiGate A:

Port	AWS primary address	AWS secondary address
port1	10.0.0.11	10.0.0.13
port2	10.0.1.11	10.0.1.13
port3	10.0.2.11	N/A
port4	10.0.3.11	N/A

The following lists the IP address assignments for this sample deployment for FortiGate B:

Port	AWS primary address
port1	10.0.0.12
port2	10.0.1.12
port3	10.0.2.12
port4	10.0.3.12

To check the prerequisites:

• Ensure that two FortiGates exist in the same VPC and AZ. The two FortiGates must also have the same build of FortiOS (FGT_VM64_AWS or FGT_VM64_AWSONDEMAND) installed.
• If using FGT_VM64_AWS, ensure that both FortiGates have valid licenses.

To configure FortiGate-VM HA in AWS:

1. In the AWS management console, create a VPC. The VPC in this example has been created with 10.0.0.0/16 CIDR.
2. Create four subnets. In this example, the four subnets are as follows:
   1. Public WAN: 10.0.0.0/24
   2. Internal network: 10.0.1.0/24
   3. Heartbeat network: 10.0.2.0/24
   4. Management network: 10.0.3.0/24

   If the FortiGates are to be deployed on the outpost, create the subnets all on outposts.

3. Create a single open security group as shown:

   

   

4. Create an IAM role. The IAM role is necessary for HA failover. Ensure that the IAM role can read and write EC2 information to read, detach, and reattach network interfaces and edit routing tables.
5. Create five EIPs. Setting up the environment requires five EIPs, but you are left with three IP addresses at the end:
   • One public WAN IP address. This is attached to the instance NIC1's secondary IP address.
   • One FortiGate A management IP address
   • One FortiGate B management IP address
   • Two temporary IP addresses
6. Create two FortiGate instances. You can use any instance type with at least four vCPUs, since the configuration requires four NICs:
   1. Configure FortiGate A:
      1. Attach the IAM role created earlier.
      2. Create the instance in the VPC created earlier and in the public WAN subnet, with no ephemeral public IP address.
      3. Configure an internal IP address of 10.0.0.11, and a secondary IP address of 10.0.0.13.

         

      4. Attach a security group.
   2. Configure FortiGate B by repeating the steps for FortiGate A above. For FortiGate B, configure an internal IP address of 10.0.0.12, and no internal IP address.
   3. Attach three NICs to each FortiGate according to the IP assignment in the appropriate subnet:
      • FortiGate A:
        • port2 (AWS primary 10.0.1.11/AWS secondary 10.0.1.13) (internal network)
        • port3 (AWS primary 10.0.2.11) (Heartbeat network)
        • port4 (AWS primary 10.0.3.11) (management network)
      • FortiGate B:
        • port2 (AWS primary 10.0.1.12) (internal network)
        • port3 (AWS primary 10.0.2.12) (Heartbeat network)
        • port4 (AWS primary 10.0.3.12) (management network)
7. Attach the two temporary EIPs to the port1 primary IP addresses of FortiGate A and FortiGate B. This allows access to the FortiGates via SSH for configuration purposes. The default password for the FortiGates is their instance IDs. The following shows the temporary EIP assigned to FortiGate A:

   

   The following shows the temporary EIP assigned to FortiGate B:

   

To configure FortiGate A using the CLI:

Run the following commands in the FortiOS CLI on FortiGate A:

config system global

set hostname master

end

config system interface

edit port1

set mode static

set ip 10.0.0.13 255.255.255.0

set allowaccess https ping ssh fgfm

set alias external

next

edit port2

set mode static

set ip 10.0.1.13 255.255.255.0

set allowaccess https ping ssh fgfm

set alias internal

next

edit port3

set mode static

set ip 10.0.2.11 255.255.255.0

set allowaccess https ping ssh fgfm

set alias hasync

next

edit port4

set mode static

set ip 10.0.3.11 255.255.255.0

set allowaccess https ping ssh fgfm

set alias hamgmt

next

end

config router static

edit 1

set device port1

set gateway 10.0.0.1

next

end

config system dns

set primary 8.8.8.8

end

config firewall policy

edit 0

set name "outgoing"

set srcintf "port2"

set dstintf "port1"

set srcaddr "all"

set dstaddr "all"

set action accept

set schedule "always"

set service "ALL"

set logtraffic disable

set nat enable

next

end

config system ha

set group-name "test"

set mode a-p

set hbdev "port3" 50

set session-pickup enable

set ha-mgmt-status enable

config ha-mgmt-interfaces

edit 1

set interface "port4"

set gateway 10.0.3.1

next

end

set override disable

set priority 1

set unicast-hb enable

set unicast-hb-peerip 10.0.2.12

end

To configure FortiGate B using the CLI:

Run the following commands in the FortiOS CLI on FortiGate B:

config system global

set hostname slave

end

config system interface

edit port1

set mode static

set ip 10.0.0.12 255.255.255.0

set allowaccess https ping ssh fgfm

set alias external

next

edit port2

set mode static

set ip 10.0.1.12 255.255.255.0

set allowaccess https ping ssh fgfm

set alias internal

next

edit port3

set mode static

set ip 10.0.2.12 255.255.255.0

set allowaccess https ping ssh fgfm

set alias hasync

next

edit port4

set mode static

set ip 10.0.3.12 255.255.255.0

set allowaccess https ping ssh fgfm

set alias hamgmt

next

end

config router static

edit 1

set device port1

set gateway 10.0.0.1

next

end

config system dns

set primary 8.8.8.8

end

config firewall policy

edit 0

set name "outgoing"

set srcintf "port2"

set dstintf "port1"

set srcaddr "all"

set dstaddr "all"

set action accept

set schedule "always"

set service "ALL"

set logtraffic disable

set nat enable

next

end

config system ha

set group-name "test"

set mode a-p

set hbdev "port3" 50

set session-pickup enable

set ha-mgmt-status enable

config ha-mgmt-interfaces

edit 1

set interface "port4"

set gateway 10.0.3.1

next

end

set override disable

set priority 1

set unicast-hb enable

set unicast-hb-peerip 10.0.2.11

end

After completing configuration of FortiGate B, remove the two temporary IP addresses. You can connect to the FortiGates via the management ports instead.

To configure the routing tables in AWS:

You must configure three routing tables.

The following shows the public WAN routing table:

The following shows the internal network routing table. Ensure to point the 0.0.0.0/0 CIDR to FortiGate A's port2 NIC.

The following shows the heartbeat and management networks' routing table:

To test FortiGate-VM HA:

1. Run get system ha status to check that the FortiGates are in sync:

   master # get system ha stat

   HA Health Status: OK

   Model: FortiGate-VM64-AWSONDEMAND

   Mode: HA A-P

   Group: 0

   Debug: 0

   Cluster Uptime: 0 days 0:42:46

   Cluster state change time: 2019-01-15 17:23:02

   Master selected using:

   <2019/01/15 17:23:02> FGTAWS000F19C1A0 is selected as the master because it has the largest value of uptime.

   <2019/01/15 17:09:47> FGTAWS000F19C1A0 is selected as the master because it's the only member in the cluster.

   ses_pickup: enable, ses_pickup_delay=disable

   override: disable

   unicast_hb: peerip=10.0.2.12, myip=10.0.2.11, hasync_port='port3'

   Configuration Status:

   FGTAWS000F19C1A0(updated 4 seconds ago): in-sync

   FGTAWS000ECBF4EF(updated 4 seconds ago): in-sync

   System Usage stats:

   FGTAWS000F19C1A0(updated 4 seconds ago):

   sessions=2, average-cpu-user/nice/system/idle=0%/0%/0%/100%, memory=5%

   FGTAWS000ECBF4EF(updated 4 seconds ago):

   sessions=0, average-cpu-user/nice/system/idle=0%/0%/0%/100%, memory=5%

   HBDEV stats:

   FGTAWS000F19C1A0(updated 4 seconds ago):

   port3: physical/1000full, up, rx-bytes/packets/dropped/errors=3135309/12092/0/0, tx=9539178/17438/0/0

   FGTAWS000ECBF4EF(updated 4 seconds ago):

   port3: physical/1000full, up, rx-bytes/packets/dropped/errors=9300105/17602/0/0, tx=3293016/11828/0/0

   Master: master , FGTAWS000F19C1A0, HA cluster index = 0

   Slave : slave , FGTAWS000ECBF4EF, HA cluster index = 1

   number of vcluster: 1

   vcluster 1: work 10.0.2.11

   Master: FGTAWS000F19C1A0, HA operating index = 0

   Slave : FGTAWS000ECBF4EF, HA operating index = 1

2. Ensure that failover functions as configured:
   1. Turn on debug mode on FortiGate B:

      slave # di de en

      slave # di de application awsd -1

      Debug messages will be on for unlimited time.

   2. Shut down the primary FortiGate A. In the event of a successful failover, FortiGate B's CLI shows the following:

      slave # Become HA master

      send_vip_arp: vd root master 1 intf port1 ip 10.0.0.13

      send_vip_arp: vd root master 1 intf port2 ip 10.0.1.13

      awsd get instance id i-0ecbf4ef4c14ba1bb

      awsd get iam role WikiDemoHARole

      awsd get region us-east-1

      awsd doing ha failover for vdom root

      awsd moving secondary ip for port1

      awsd moving secip 10.0.0.13 from eni-016c35b5d998a3995 to eni-0d318aeb7cdfe72fb

      awsd move secondary ip successfully

      awsd associate elastic ip allocation eipalloc-0e5ff7daabd5f46dc to 10.0.0.13 of eni eni-0d318aeb7cdfe72fb

      awsd associate elastic ip successfully

      awsd moving secondary ip for port2

      awsd moving secip 10.0.1.13 from eni-0f19c02934d82c086 to eni-004d87ffb05329b28

      awsd move secondary ip successfully

      awsd update route table rtb-0bc0aaaea8fe56192, replace route of dst 0.0.0.0/0 to eni-004d87ffb05329b28

      awsd update route successfully

   3. Verify on AWS that the public and internal networks' secondary IP addresses moved, and that the routing table changes to point to FortiGate B's internal network ENI.
3. Initiate an SSH session (or another protocol with similar long keep-alive session characteristics) to an external IP address on Ubuntu or an internal VM used for testing purposes. Retest failover and check that the session continues to function without needing to reconnect, and that the session list on the primary and failed over secondary FortiGates are synced.
4. You must configure a VDOM exception to prevent interface synchronization between the two FortiGates. Run the following commands in the FortiOS CLI:

   config system vdom-exception

   edit 1

   set object system.interface

   next

   edit 2

   set object router.static

   next

   edit 3

   set object firewall.vip

   next

   end