FGCP multi-version cluster upgrade

The FGCP multi-version cluster (MVC) upgrade mode allows manual control over the cluster member that is being upgraded. HA members can temporarily run in an MVC while administrators perform tests to confirm traffic can pass through the upgraded member smoothly.

The syntax of the existing upgrade mode has been updated starting in 7.4.1:

config system ha
    set upgrade-mode {simultaneous | uninterruptible | local-only | secondary-only}
end

upgrade-mode {simultaneous | uninterruptible | local-only | secondary-only}

Set the mode to upgrade a cluster. simultaneous: all HA members upgrade at the same time (set uninterruptible-upgrade disable in 7.4.0 and earlier). uninterruptible: secondary HA members are upgraded first, followed by the primary member (set uninterruptible-upgrade enable in 7.4.0 and earlier). local-only: only upgrade the local member in which the firmware is uploaded. secondary-only: only upgrade the secondary members.

The local-only and secondary-only upgrade options are advanced configurations that should only be used to temporarily put the HA cluster in MVC operation mode. While in this operation, states and sessions (such as the session table and routing table) are synchronized, but configuration changes are not synchronized between cluster members in different builds. If more than two members are in the cluster, the configurations between members in the same builds will be synchronized. The configurations for the entire cluster will be synchronized once the upgrade process has completed.

How it works

In local-only and secondary-only modes, the specific cluster member is upgraded and sessions are synchronized to it. The following tables show which members are upgraded based on the mode and where the upgrade is initiated.

local-only
Upgrade method	Outcome	Recommendation
Initiate the upload or upgrade on the primary.	The primary member is upgraded.	Not recommended.
Initiate the upload or upgrade on the secondary member.	The secondary member where the image is uploaded is upgraded.	Recommended when selecting a specific HA member to upgrade.

secondary-only
Upgrade method	Outcome	Recommendation
Initiate the upload or upgrade on the primary.	All non-primary members are upgraded.	Recommended for scenarios where there is more than one secondary HA member.
Initiate the upload or upgrade on the secondary member.	The secondary member where the image is uploaded is upgraded.	Same result as initiating an upgrade on a secondary member in local-only mode.

This can apply to any HA clusters with two or more members. Administrators can initiate an upgrade on a secondary member by using its CLI console or accessing the device's GUI from its HA management interface.

Initially, when you prepare an HA cluster in A-P mode for upgrade, traffic passes through the primary unit (Node-A) as the secondary unit (Node-B) sits on standby.

After the upgrade is completed on a secondary unit, states and sessions are synchronized. The members are now operating in MVC mode; however, traffic continues to pass through Node-A.

Administrators can manually trigger failover to make Node-B the new primary when ready. This can be done by resetting the HA uptime or changing device priorities, whichever method is desired. Traffic now passes through Node-B.

The upgraded system (Node-B) can be tested to verify that traffic can pass smoothly. If verification fails, administrators can trigger a failover to fail back to Node-A to avoid any downtime.

If verification is successful, administrators can manually trigger an upgrade on Node-A to bring the HA member up to the same version as Node-B to complete the HA upgrade procedure. This can be performed by accessing Node-A’s GUI from its HA management interface or using its CLI console.

Example 1: upgrade a single secondary member using the local-only upgrade option

In this example, three HA members are running in an FGCP A-P HA cluster.

The member FGVM02TM22027808 is acting as the primary and forwarding traffic. The member FGVM02TM22027810 is chosen for upgrade.

The cluster is originally running build 2456. The secondary unit is upgraded to build 2461. Fictitious build numbers are used in this example to demonstrate functionality of the feature.

To configure the HA cluster:

config system ha
    set group-id 260
    set group-name "kkk"
    set mode a-p
    set hbdev "port3" 0
    set session-pickup enable
    set upgrade-mode local-only
end

To perform the upgrade:

1. On the secondary member (FGVM02TM22027810), log in to the CLI console.

2. Execute a TFTP upgrade:

   FGVM02TM22027810 # execute restore image tftp /home/Images/FortiOS/v7.00/images/build2461/FGT_VM64-v7-build2461-FORTINET.out 172.16.100.71
   This operation will replace the current firmware version!
   Do you want to continue? (y/n)y
   
   Please wait...
   
   Connect to ftp server 172.16.100.71 ...
   Get image from ftp server OK.
   Verifying the signature of the firmware image.
   
   Please wait for system to restart.

3. After the upgrade is complete, verify the version running on the secondary member:

   FGVM02TM22027810 # get system status
   Version: FortiGate-VM64 v7.4.1,build2461,230828 (interim)
   …

4. On the primary unit, verify that HA is still formed between the three members:

   FGVM02TM22027808 # diagnose sys ha dump-by group
   <hatalk> vcluster_1: ha_prio=0(primary), state/chg_time/now=2(work)/1692750721/1693262149
               HA information.
   group-id=260, group-name='kkk'
   has_no_aes128_gcm_sha256_member=0
   
   gmember_nr=3
   'FGVM02TM22027808': ha_ip_idx=2, hb_packet_version=10, last_hb_jiffies=0, linkfails=0, weight/o=0/0, support_aes128_gcm_sha256=1
   'FGVM02TM22027809': ha_ip_idx=1, hb_packet_version=12, last_hb_jiffies=51142842, linkfails=3, weight/o=0/0, support_aes128_gcm_sha256=1
           hbdev_nr=1: port3(mac=000c..de, last_hb_jiffies=51142842, hb_lost=0),
   'FGVM02TM22027810': ha_ip_idx=0, hb_packet_version=4, last_hb_jiffies=51142858, linkfails=3, weight/o=0/0, support_aes128_gcm_sha256=1
           hbdev_nr=1: port3(mac=000c..1a, last_hb_jiffies=51142858, hb_lost=0),
   
   vcluster_nr=1
   vcluster-1: start_time=1692750718(2023-08-22 17:31:58), state/o/chg_time=2(work)/2(work)/1692750721(2023-08-22 17:32:01)
           pingsvr_flip_timeout/expire=3600s/0s
           mondev: port1(prio=50,is_aggr=0,status=1) port7(prio=50,is_aggr=0,status=1) port8(prio=50,is_aggr=0,status=1)
           'FGVM02TM22027808': ha_prio/o=0/0, link_failure=0, pingsvr_failure=0, flag=0x00000001, mem_failover=0, uptime/reset_cnt=510868/0
           'FGVM02TM22027809': ha_prio/o=1/1, link_failure=0, pingsvr_failure=0, flag=0x00000000, mem_failover=0, uptime/reset_cnt=510857/0
           'FGVM02TM22027810': ha_prio/o=2/2, link_failure=0, pingsvr_failure=0, flag=0x00000000, mem_failover=0, uptime/reset_cnt=0/0

5. Fail over the HA cluster so that the secondary member, FGVM02TM22027810, becomes the primary. Since override is not enabled and the HA primary is determined by uptime, you can reset the HA uptime on the units that were not upgraded:

   # diagnose sys ha reset-uptime

6. Once verification on the upgraded member is successful, repeat step 2 to perform upgrades on the remaining units.

Example 2: upgrade multiple secondary members using the secondary-only upgrade option

Using the same topology as example 1, the three HA cluster members are originally running build 2456. Both secondary units are upgraded using the secondary-only upgrade option. Fictitious build numbers are used in this example to demonstrate functionality of the feature.

To configure the HA cluster:

config system ha
    set group-id 260
    set group-name "kkk"
    set mode a-p
    set hbdev "port3" 0
    set session-pickup enable
    set upgrade-mode secondary-only
end

To perform the upgrade:

1. On the primary unit (FGVM02TM22027808), log in to the CLI console.

2. Execute a TFTP upgrade:

   FGVM02TM22027808 # execute restore image tftp /home/Images/FortiOS/v7.00/images/build2461/FGT_VM64-v7-build2461-FORTINET.out 172.16.100.71

3. After the upgrade is complete, verify the version running on the secondary members.

   1. Member 1:

      FGVM02TM22027809 # get system status
      Version: FortiGate-VM64 v7.4.1,build2461,230828 (interim)
      …

   2. Member 2:

      FGVM02TM22027810 # get system status
      Version: FortiGate-VM64 v7.4.1,build2461,230828 (interim)
      …

4. On the primary unit, verify that HA is still formed between the three members:

   FGVM02TM22027808 # diagnose sys ha dump-by group
               HA information.
   group-id=260, group-name='kkk'
   has_no_aes128_gcm_sha256_member=0
   
   gmember_nr=3
   'FGVM02TM22027808': ha_ip_idx=2, hb_packet_version=19, last_hb_jiffies=0, linkfails=0, weight/o=0/0, support_aes128_gcm_sha256=1
   'FGVM02TM22027809': ha_ip_idx=1, hb_packet_version=4, last_hb_jiffies=51358055, linkfails=3, weight/o=0/0, support_aes128_gcm_sha256=1
           hbdev_nr=1: port3(mac=000c..de, last_hb_jiffies=51358055, hb_lost=0),
   'FGVM02TM22027810': ha_ip_idx=0, hb_packet_version=5, last_hb_jiffies=51358057, linkfails=3, weight/o=0/0, support_aes128_gcm_sha256=1
           hbdev_nr=1: port3(mac=000c..1a, last_hb_jiffies=51358057, hb_lost=0),
   
   vcluster_nr=1
   vcluster-1: start_time=1692750718(2023-08-22 17:31:58), state/o/chg_time=2(work)/2(work)/1692750721(2023-08-22 17:32:01)
           pingsvr_flip_timeout/expire=3600s/0s
           mondev: port1(prio=50,is_aggr=0,status=1) port7(prio=50,is_aggr=0,status=1) port8(prio=50,is_aggr=0,status=1)
           'FGVM02TM22027808': ha_prio/o=0/0, link_failure=0, pingsvr_failure=0, flag=0x00000001, mem_failover=0, uptime/reset_cnt=512775/0
           'FGVM02TM22027809': ha_prio/o=2/2, link_failure=0, pingsvr_failure=0, flag=0x00000000, mem_failover=0, uptime/reset_cnt=0/0
           'FGVM02TM22027810': ha_prio/o=1/1, link_failure=0, pingsvr_failure=0, flag=0x00000000, mem_failover=0, uptime/reset_cnt=1/0