Unicast HA on IBM VPC Cloud

IBM VPC Cloud users can deploy their BYOL FortiGate VMs in unicast HA. The HA failover will automatically trigger routing changes and floating IP reassignment on the IBM Cloud via the API.

Example

In this example, an administrator has an Ubuntu client protected by an IBM FortiGate in HA A-P mode. The administrator uses a VIP to access Ubuntu, the web, and has traffic inspected for EICAR.

When the primary device is shut down to simulate a failover event, the floating IP (FIP) and route fail over. After the failover, the administrator can still use the VIP to access Ubuntu and the web, and have traffic inspected for EICAR, through the secondary FortiGate.

In this example you will configure the IBM VPC device and the primary and secondary FortiGates.

To configure the IBM VPC:

1. Configure the subnets and attach the public gateway (see Using the IBM Cloud console to create VPC resources).
   1. Configure four subnets:

      • Public

      • Internal

      • Management

      • Heartbeat

   2. Make sure a public gateway is attached to the public subnet.

   

   

2. Configure the two route tables (see Creating a routing table).
   1. Configure the internal route table as follows:
      1. It needs to be the IBM default route table for the VPC.
      2. It has a route for all traffic to the internal subnet IP of the primary FortiGate.
      3. It applies to the internal subnet.
   2. Configure the open (non-default) route table as follows:
      1. This route table can have no routes.
      2. It applies to the public, management, and heartbeat subnets.

         Non-default route tables cannot be used for the internal subnet’s route table failover in IBM VPCs at this time.

         

3. Configure the floating IP (see Managing network interfaces).

   IBM Cloud does not currently support multiple FIPs for a single instance. Even though the management ports can be configured, you will not be able to access them using a FIP in the final configuration. If you want to access the instances for configuration purposes, you can attach a FIP to the public subnet's IP on the primary and secondary devices until the FortiOS configuration is finished. Also, you can connect directly to the local IPs through a VPN or another proxy instance.

   In this example, the final configuration only needs one FIP attached to the primary public subnet IP.

   

To configure the primary FortiGate:

1. Configure the static IP addresses:

   config system interface
       edit "port1"
           set vdom "root"
           set ip 10.241.128.4 255.255.255.0
           set allowaccess ping https ssh snmp http telnet fgfm radius-acct probe-response fabric ftm
           set type physical
           set snmp-index 1
       next
       edit "port2"
           set vdom "root"
           set ip 10.241.129.4 255.255.255.0
           set allowaccess ping https ssh snmp http telnet fgfm radius-acct probe-response fabric ftm
           set type physical
           set snmp-index 2
       next
       edit "port3"
           set ip 10.241.131.4 255.255.255.0
           set allowaccess ping https ssh snmp http telnet fgfm radius-acct probe-response fabric ftm
           set type physical
           set snmp-index 3
       next
       edit "port4"
           set ip 10.241.130.4 255.255.255.0
           set allowaccess ping https ssh snmp http telnet fgfm radius-acct probe-response fabric ftm
           set type physical
           set snmp-index 4
       next
   end

2. Configure the HA settings:

   config system ha
       set group-name "Test"
       set mode a-p
       set password **********
       set hbdev "port3" 100
       set ha-mgmt-status enable
       config ha-mgmt-interfaces
           edit 1
               set interface "port4"
               set gateway 10.241.130.1
           next
       end
       set override enable
       set priority 255
       set unicast-hb enable
       set unicast-hb-peerip 10.241.131.5
   end

3. Verify that the primary and secondary FortiGates see each other and are synchronized:

   # get system ha status
   HA Health Status: OK
   Model: FortiGate-VM64-IBM
   Mode: HA A-P
   Group: 0
   Debug: 0
   Cluster Uptime: 1 days 3:15:48
   Cluster state change time: 2020-11-24 15:35:01
   Primary selected using:
       <2020/11/24 15:35:01> FGVM08TM20000007 is selected as the primary because it has the largest value of override priority.
   ses_pickup: disable
   override: enable
   unicast_hb: peerip=10.241.131.5, myip=10.241.131.4, hasync_port='port3'
   Configuration Status:
       FGVM08TM20000007(updated 1 seconds ago): in-sync
       FGVM08TM20000006(updated 2 seconds ago): in-sync
   System Usage stats:
       FGVM08TM20000007(updated 1 seconds ago):
           sessions=4, average-cpu-user/nice/system/idle=0%/0%/0%/100%, memory=4%
       FGVM08TM20000006(updated 2 seconds ago):
           sessions=0, average-cpu-user/nice/system/idle=0%/0%/0%/100%, memory=4%
   HBDEV stats:
       FGVM08TM20000007(updated 1 seconds ago):
           port3: physical/10000full, up, rx-bytes/packets/dropped/errors=15646281/45910/0/0, tx=21807567/45445/0/0
       FGVM08TM20000006(updated 2 seconds ago):
           port3: physical/10000full, up, rx-bytes/packets/dropped/errors=25485511/54398/0/0, tx=22502231/143827/0/0
   Primary     : FGVM08TM20000007, FGVM08TM20000007, HA cluster index = 0
   Secondary   : FGVM08TM20000006, FGVM08TM20000006, HA cluster index = 1
   number of vcluster: 1
   vcluster 1: work 10.241.131.4
   Primary: FGVM08TM20000007, HA operating index = 0
   Secondary: FGVM08TM20000006, HA operating index = 1

4. Configure the static route. The gateway is the public subnet's first address:

   config router static
       edit 1
           set gateway 10.241.128.1
           set device "port1"
       next
   end

5. Configure the VDOM exception:

   config system vdom-exception
       edit 1
           set object firewall.vip
       next
   end

6. Configure the VIP:

   config firewall vip
       edit "to internal ubuntu"
           set extip 10.241.128.4
           set mappedip "10.241.129.6"
           set extintf "port1"
           set portforward enable
           set extport 8822
           set mappedport 22
       next
   end

7. Configure the firewall policies for the Ubuntu client and internal subnet:

   config firewall policy
       edit 1
           set name "toVIP"
           set srcintf "port1"
           set dstintf "port2"
           set srcaddr "all"
           set dstaddr "to internal ubuntu"
           set action accept
           set schedule "always"
           set service "ALL"
           set logtraffic all
           set nat enable
       next
       edit 2
           set name "main"
           set srcintf "port2"
           set dstintf "port1"
           set srcaddr "all"
           set dstaddr "all"
           set action accept
           set schedule "always"
           set service "ALL"
           set utm-status enable
           set ssl-ssh-profile "certificate-inspection"
           set av-profile "default"
           set logtraffic all
           set nat enable
       next
   end

8. Configure the SDN connector:

   config system sdn-connector
       edit "1"
           set type ibm
           set ha-status enable
           set api-key **********
           set ibm-region us-east
       next
   end

To configure the secondary FortiGate:

1. Configure the static IP addresses:

   config system interface
       edit "port1"
           set vdom "root"
           set ip 10.241.128.5 255.255.255.0
           set allowaccess ping https ssh snmp http telnet fgfm radius-acct probe-response fabric ftm
           set type physical
           set snmp-index 1
       next
       edit "port2"
           set vdom "root"
           set ip 10.241.129.5 255.255.255.0
           set allowaccess ping https ssh snmp http telnet fgfm radius-acct probe-response fabric ftm
           set type physical
           set snmp-index 2
       next
       edit "port3"
           set ip 10.241.131.5 255.255.255.0
           set allowaccess ping https ssh snmp http telnet fgfm radius-acct probe-response fabric ftm
           set type physical
           set snmp-index 3
       next
       edit "port4"
           set ip 10.241.130.5 255.255.255.0
           set allowaccess ping https ssh snmp http telnet fgfm radius-acct probe-response fabric ftm
           set type physical
           set snmp-index 4
       next
   end

2. Configure the HA settings:

   config system ha
       set group-name "Test"
       set mode a-p
       set password **********
       set hbdev "port3" 100
       set ha-mgmt-status enable
       config ha-mgmt-interfaces
           edit 1
               set interface "port4"
               set gateway 10.241.130.1
           next
       end
       set override enable
       set priority 0
       set unicast-hb enable
       set unicast-hb-peerip 10.241.131.4
   end

3. Configure the VIP:

   config firewall vip
       edit "to internal ubuntu"
           set extip 10.241.128.5
           set mappedip "10.241.129.6"
           set extintf "port1"
           set portforward enable
           set extport 8822
           set mappedport 22
       next
   end

To test the configuration:

1. Access the Ubuntu client via the public FIP and custom port 8822, then use cURL to get the EICAR file from HTTP. The FortiGate should block the file:

   root@mail:/home/kvm/scripts# ssh ubuntu@xx.xxx.xxx.xxx -p 8822
   ubuntu@xx.xxx.xxx.xxx's password:
   Welcome to Ubuntu 20.04.1 LTS (GNU/Linux 5.4.0-1026-kvm x86_64)
   ... omitted ...
   ubuntu@xxxxxx-ha-ubuntu:~$ curl http://www.eicar.org/download/eicar.com
   <!DOCTYPE html>
   ... omitted ...
       <p>You are not permitted to download the file "eicar.com" because it is infected with the virus "EICAR_TEST_FILE".</p>

2. Trigger the failover by shutting down the primary FortiGate. Verify that the FIP and route tables have moved, then try to access the Ubuntu client and get the EICAR file again:

   root@mail:/home/kvm/scripts# ssh ubuntu@xx.xxx.xxx.xxx -p 8822
   ubuntu@xx.xxx.xxx.xxx's password:
   Welcome to Ubuntu 20.04.1 LTS (GNU/Linux 5.4.0-1026-kvm x86_64)
   ... omitted ...
   ubuntu@xxxxxx-ha-ubuntu:~$ curl http://www.eicar.org/download/eicar.com
   <!DOCTYPE html>
   ... omitted ...
       <p>You are not permitted to download the file "eicar.com" because it is infected with the virus "EICAR_TEST_FILE".</p>

3. If the failover is unsuccessful, you can debug the secondary FortiGate in the IBM VPC. Note that even though there are some reported fails, the failover is successful:

   HA event
   HA state: primary
   ibmd sdn connector is getting token
   token size: 1163
   token expiration: 1606264324
   parsing instance 0777_e8e111aa-1aa1-11aa-a111-1111a1aa1a1a
   ibmd HA successfully got fip for hb peer
   parsing instance 0777_2b22bbbb-bb22-2b22-bb22-b222bbb22b2b
   ibmd HA found hb host/peer info
   in collect rtbl
   ibmd HA found rtbl on hb peer ip
   ibmd http request response: 204
   
   ibmd HA deleted rtbl r014-167a1aaa-12ab-1111-bb2a-2ababbb22222
   ibmd HA deleted rtbl r014-167a1aaa-12ab-1111-bb2a-2ababbb22222
   ibmd http request response: 201
   {"id":"r014-b8771aa1-1111-22aa-22bb-1aa22bb222ab","href":"https://us-east.iaas.cloud.ibm.com/v1/vpcs/r014-ab1b121a-21ba-21ab-11ab-aaba1abaabba/routes/r014-b8771aa1-1111-22aa-22bb-1aa22bb222ab","name":"glancing-handprint-shakable-gotten","action":"deliver","destination":"0.0.0.0/0","next_hop":{"address":"10.241.129.5"},"lifecycle_state":"stable","created_at":"2020-11-24T23:32:12Z","zone":{"name":"us-east-3","href":"https://us-east.iaas.cloud.ibm.com/v1/regions/us-east/zones/us-east-3"}}
   
   ibmd HA created rtbl
   ibmd HA created rtbl
   HA state: primary
   ibmd sdn connector is getting token
   token size: 1163
   token expiration: 1606264337
   parsing instance 0777_e8e111aa-1aa1-11aa-a111-1111a1aa1a1a
   ibmd HA failed to parse fip list
   ibmd HA failed to get fip for hb peer
   parsing instance 0777_2b22bbbb-bb22-2b22-bb22-b222bbb22b2b
   ibmd HA found hb host/peer info
   in collect rtbl
   ibmd HA failed to find hb fip
   ibmd HA failed to move fip