Deploying FortiGate-VM A-P HA on IBM VPC Cloud (BYOL)

IBM VPC Cloud users can deploy their BYOL FortiGate-VMs in unicast high availability (HA). The HA failover automatically triggers routing changes and floating IP address reassignment on the IBM Cloud via API.

Example

In the following example, the administrator has an Ubuntu client that an IBM FortiGate in HA active-passive mode is protecting. The administrator uses a virtual IP address (VIP) to access Ubuntu, the web, and has traffic inspected for EICAR.

When the primary device is shut down to simulate a failover event, the floating IP (FIP) and route are failed over. After the failover, the administrator can still use the VIP to access Ubuntu and the web, and have traffic inspected for EICAR, through the secondary FortiGate.

In the following example you will configure the IBM Virtual PC device and the primary and secondary FortiGates.

To configure the IBM VPC:

1. Configure the subnets and attach the public gateway.
   1. Configure four subnets:

      • Public

      • Internal

      • Management

      • Heartbeat

   2. Make sure a Public Gateway is attached to the Public subnet

   

   

2. Configure two route tables:

   • Internal: This route table:

     • Needs to be the IBM default route table for the VPC.

     • Has a route for all traffic to the internal subnet IP of the primary FortiGate.

     • Applies to the internal subnet.

     If you have not deployed FortiGate, return to this step after deployment.

   • Open: This route table can have no routes, and can be applied to the Public, Management, and Heartbeat subnets.

   Non-default route tables cannot be used for the internal subnet’s route table failover in IBM VPC at this time.

   

3. Configure the floating IP.

   IBM Cloud does not currently support multiple FIPs for a single instance. Even though the management ports can be configured, you will not be able to access them using FIP in the final configuration. If you wish to access the instances for configuration purposes, you can attach a FIP to the public subnets IP on the primary and secondary devices until FOS configuration is finished. You may also connect directly to the local IPs via VPN or another proxy instance.

   For this example, the final configuration will only need one FIP attached to the primary public subnet IP.

   

To configure the FortiGate:

1. Configure the primary and secondary device's static IP addresses.
   1. Configure the primary FortiGate's static IPs for all ports according to IBM Cloud's delegated internal IPs.

      config system interface

      edit "port1"

      set vdom "root"

      set ip 10.241.128.4 255.255.255.0

      set allowaccess ping https ssh snmp http telnet fgfm radius-acct probe-response fabric ftm

      set type physical

      set snmp-index 1

      next

      edit "port2"

      set vdom "root"

      set ip 10.241.129.4 255.255.255.0

      set allowaccess ping https ssh snmp http telnet fgfm radius-acct probe-response fabric ftm

      set type physical

      set snmp-index 2

      next

      edit "port3"

      set ip 10.241.131.4 255.255.255.0

      set allowaccess ping https ssh snmp http telnet fgfm radius-acct probe-response fabric ftm

      set type physical

      set snmp-index 3

      next

      edit "port4"

      set ip 10.241.130.4 255.255.255.0

      set allowaccess ping https ssh snmp http telnet fgfm radius-acct probe-response fabric ftm

      set type physical

      set snmp-index 4

      next

      end

   2. Configure the secondary FortiGate's static IPs for all ports according to IBM Cloud's delegated internal IPs.

      config system interface

      edit "port1"

      set vdom "root"

      set ip 10.241.128.5 255.255.255.0

      set allowaccess ping https ssh snmp http telnet fgfm radius-acct probe-response fabric ftm

      set type physical

      set snmp-index 1

      next

      edit "port2"

      set vdom "root"

      set ip 10.241.129.5 255.255.255.0

      set allowaccess ping https ssh snmp http telnet fgfm radius-acct probe-response fabric ftm

      set type physical

      set snmp-index 2

      next

      edit "port3"

      set ip 10.241.131.5 255.255.255.0

      set allowaccess ping https ssh snmp http telnet fgfm radius-acct probe-response fabric ftm

      set type physical

      set snmp-index 3

      next

      edit "port4"

      set ip 10.241.130.5 255.255.255.0

      set allowaccess ping https ssh snmp http telnet fgfm radius-acct probe-response fabric ftm

      set type physical

      set snmp-index 4

      next

      end

2. Configure the HA.
   1. Configure the group-name, mode, password, and set hbdev port to the heartbeat port.
   2. Configure ha-mgmt-interfaces and unicast-hb-peerip with the FortiGate's heartbeat port IP.

      config system ha

      set group-name "Test"

      set mode a-p

      set password xxxxxxxx

      set hbdev "port3" 100

      set ha-mgmt-status enable

      config ha-mgmt-interfaces

      edit 1

      set interface "port4"

      set gateway 10.241.130.1

      next

      end

      set override enable

      set priority 255

      set unicast-hb enable

      set unicast-hb-peerip 10.241.131.5

      end

   3. Configure the secondary FortiGate's HA settings.

      config system ha

      set group-name "Test"

      set mode a-p

      set password xxxxxxxx

      set hbdev "port3" 100

      set ha-mgmt-status enable

      config ha-mgmt-interfaces

      edit 1

      set interface "port4"

      set gateway 10.241.130.1

      next

      end

      set override enable

      set priority 0

      set unicast-hb enable

      set unicast-hb-peerip 10.241.131.4

      end

   4. Verify the primary and secondary FortiGate's can see each other, and the configuration can be synced.

      # get system ha status

      HA Health Status: OK

      Model: FortiGate-VM64-IBM

      Mode: HA A-P

      Group: 0

      Debug: 0

      Cluster Uptime: 1 days 3:15:48

      Cluster state change time: 2020-11-24 15:35:01

      Primary selected using:

      <2020/11/24 15:35:01> FGVM08TM20000007 is selected as the primary because it has the largest value of override priority.

      ses_pickup: disable

      override: enable

      unicast_hb: peerip=10.241.131.5, myip=10.241.131.4, hasync_port='port3'

      Configuration Status:

      FGVM08TM20000007(updated 1 seconds ago): in-sync

      FGVM08TM20000006(updated 2 seconds ago): in-sync

      System Usage stats:

      FGVM08TM20000007(updated 1 seconds ago):

      sessions=4, average-cpu-user/nice/system/idle=0%/0%/0%/100%, memory=4%

      FGVM08TM20000006(updated 2 seconds ago):

      sessions=0, average-cpu-user/nice/system/idle=0%/0%/0%/100%, memory=4%

      HBDEV stats:

      FGVM08TM20000007(updated 1 seconds ago):

      port3: physical/10000full, up, rx-bytes/packets/dropped/errors=15646281/45910/0/0, tx=21807567/45445/0/0

      FGVM08TM20000006(updated 2 seconds ago):

      port3: physical/10000full, up, rx-bytes/packets/dropped/errors=25485511/54398/0/0, tx=22502231/143827/0/0

      Primary : FGVM08TM20000007, FGVM08TM20000007, HA cluster index = 0

      Secondary : FGVM08TM20000006, FGVM08TM20000006, HA cluster index = 1

      number of vcluster: 1

      vcluster 1: work 10.241.131.4

      Primary: FGVM08TM20000007, HA operating index = 0

      Secondary: FGVM08TM20000006, HA operating index = 1

3. Configure the static route for the primary FortiGate to sync with the secondary FortiGate.

   The gateway is your public subnet's first address, which in this case is 10.241.128.1

   config router static

   edit 1

   set gateway 10.241.128.1

   set device "port1"

   next

   end

4. Configure the vdom-exception and firewall vip.
   1. Configure the vdom-exception on the primary FortiGate to automatically with the secondary FortiGate.
   2. Configure the firewall VIP on the primary and secondary devices. Make sure to set the extip to the IP of the individual FortiGate's public subnet IP, and the mapped IP to the Ubuntu client's internal subnet IP.

   Primary FortiGate configuration:

   config system vdom-exception

   edit 1

   set object firewall.vip

   next

   end

   config firewall vip

   edit "to internal ubuntu"

   set extip 10.241.128.4

   set mappedip "10.241.129.6"

   set extintf "port1"

   set portforward enable

   set extport 8822

   set mappedport 22

   next

   end

   Secondary FortiGate configuration:

   config firewall vip

   edit "to internal ubuntu"

   set extip 10.241.128.5

   set mappedip "10.241.129.6"

   set extintf "port1"

   set portforward enable

   set extport 8822

   set mappedport 22

   next

   end

   3. Configure a VIP in policy for the internal Ubuntu client, and a policy for the internal subnet to reach the internet. This firewall policy will also apply antivirus inspection for HTTP requests. This will be synced from the primary to the secondary device.

      config firewall policy

      edit 1

      set name "toVIP"

      set srcintf "port1"

      set dstintf "port2"

      set srcaddr "all"

      set dstaddr "to internal ubuntu"

      set action accept

      set schedule "always"

      set service "ALL"

      set logtraffic all

      set nat enable

      next

      edit 2

      set name "main"

      set srcintf "port2"

      set dstintf "port1"

      set srcaddr "all"

      set dstaddr "all"

      set action accept

      set schedule "always"

      set service "ALL"

      set utm-status enable

      set ssl-ssh-profile "certificate-inspection"

      set av-profile "default"

      set logtraffic all

      set nat enable

      next

      end

5. Configure the SDN connector on the primary FortiGate to sync with the secondary FortiGate.

   config system sdn-connector

   edit "1"

   set type ibm

   set ha-status enable

   set api-key xxxxxxxx

   set ibm-region us-east

   next

   end

6. Ensure the SDN connector is up.
   1. Go to Security Fabric > External Connectors.
   2. Verify that the IBM Cloud Connector is Up.

To test the configuration:

1. Access the client Ubuntu via the public FIP and custom port 8822, then use curl to get the EICAR file from HTTP. FortiGate should block the file.

   root@mail:/home/kvm/scripts# ssh ubuntu@52.117.123.241 -p 8822

   ubuntu@52.117.123.241's password:

   Welcome to Ubuntu 20.04.1 LTS (GNU/Linux 5.4.0-1026-kvm x86_64)

   ... omitted ...

   ubuntu@thomas-ha-ubuntu:~$ curl http://www.eicar.org/download/eicar.com

   <!DOCTYPE html>

   ... omitted ...

   <p>You are not permitted to download the file "eicar.com" because it is infected with the virus "EICAR_TEST_FILE".</p>

2. Trigger the failover by shutting down primary FortiGate. Verify that the FIP and route tables have moved on IBM, then try to access the client Ubuntu and get the EICAR file again.

   root@mail:/home/kvm/scripts# ssh ubuntu@53.111.222.333-p 8822

   ubuntu@52.111.222.333's password:

   Welcome to Ubuntu 20.04.1 LTS (GNU/Linux 5.4.0-1026-kvm x86_64)

   ... omitted ...

   ubuntu@thomas-ha-ubuntu:~$ curl http://www.eicar.org/download/eicar.com

   <!DOCTYPE html>

   ... omitted ...

   <p>You are not permitted to download the file "eicar.com" because it is infected with the virus "EICAR_TEST_FILE".</p>

3. If the failover is unsuccessful, you can debug the secondary FortiGate in the IBM VPC. Note that even though there are some reported fails, the failover is successful.

   token size: 1163

   token expiration: 1606264324

   parsing instance 0888_f8e568dc-5cd7-48eb-b319-8858a3ab5a2b

   ibmd HA successfully got fip for hb peer

   parsing instance 0888_7b49bafc-db71-4d10-bc05-d009ddb95e4b

   ibmd HA found hb host/peer info

   in collect rtbl

   ibmd HA found rtbl on hb peer ip

   ibmd http request response: 204

   ibmd HA deleted rtbl r019-167d7dff-86ge-4104-be7d-6efdceb29154

   ibmd HA deleted rtbl r019-167d7dff-86ge-4104-be7d-6efdceb29154

   ibmd http request response: 201

   {"id":"r014-b8771cd6-1669-45c6-80f7-7cd22cd369eb","href":"https://us-east.iaas.cloud.ibm.com/v1/vpcs/r014-eb0f603d-51ce-40eb-91db-aafa1aecebbe/routes/r014-b8871cd6-1669-45c6-80f7-7cd11cd363eb","name":"glancing-handprint-shakable-gotten","action":"deliver","destination":"0.0.0.0/0","next_hop":{"address":"10.241.129.5"},"lifecycle_state":"stable","created_at":"2020-11-24T23:32:12Z","zone":{"name":"us-east-3","href":"https://us-east.iaas.cloud.ibm.com/v1/regions/us-east/zones/us-east-3"}}

   ibmd HA created rtbl

   ibmd HA created rtbl

   HA state: primary

   ibmd sdn connector is getting token

   token size: 1163

   token expiration: 1606234327

   parsing instance 0888_e8e564dc-5cd7-47eb-b319-8858a3ab5a2b

   ibmd HA failed to parse fip list

   ibmd HA failed to get fip for hb peer

   parsing instance 0888_7b90bafc-db71-4d20-cd04-d009ddb95e4b

   ibmd HA found hb host/peer info

   in collect rtbl

   ibmd HA failed to find hb fip

   ibmd HA failed to move fip