Check HA synchronization status

The HA synchronization status can be viewed in the GUI through either a widget on the Dashboard or on the System > HA page. It can also be confirmed through the CLI. When a cluster is out of synchronization, administrators should correct the issue as soon as possible as it affects the configuration integrity and can cause issues to occur.

When units are out of synchronization in an HA cluster, the GUI will compare the HA checksums and display the tables that caused HA to be out of synchronization. This can be visualized on the HA monitor page and in the HA status widget.

HA synchronization status in the GUI

Following HA setup, the HA Status widget can be added to the Dashboard that shows the HA synchronization statuses of the members.

A green checkmark is shown next to each member that is in synchronization.

A member that is out of synchronization is highlighted in red. Hover the cursor over the unsynchronized device to see the tables that are out of synchronization and the checksum values.

You can also go to System > HA to see the synchronization statuses of the members. A member that is out of synchronization will have a red icon next to its name. Hover the cursor over the unsynchronized device to see the tables that are out of synchronization and the checksum values.

Synchronized:

Unsynchronized:

HA synchronization status in the CLI

In the CLI, run the get system ha status command to see if the cluster is in synchronization . The synchronization status is reported under Configuration Status.

When both members are in synchronization:

# get system ha status
HA Health Status: OK
Model: FortiGate-VM64
Mode: HA A-P
Group Name: docs
Group ID: 0
Debug: 0
Cluster Uptime: 0 days 0:52:39
Cluster state change time: 2021-04-29 13:17:03
Primary selected using:
    <2021/04/29 13:17:03> FGVMEV0000000002 is selected as the primary because its uptime is larger than peer member FGVMEV7000000005.
    <2021/04/29 12:37:17> FGVMEV0000000002 is selected as the primary because it's the only member in the cluster.
ses_pickup: disable
override: disable
Configuration Status:
    FGVMEV0000000002(updated 3 seconds ago): in-sync
    FGVMEV7000000005(updated 2 seconds ago): in-sync
System Usage stats:
    FGVMEV0000000002(updated 3 seconds ago):
        sessions=9, average-cpu-user/nice/system/idle=1%/0%/0%/99%, memory=66%
    FGVMEV7000000005(updated 2 seconds ago):
        sessions=0, average-cpu-user/nice/system/idle=0%/0%/0%/100%, memory=65%
HBDEV stats:
    FGVMEV0000000002(updated 3 seconds ago):
        port2: physical/1000auto, up, rx-bytes/packets/dropped/errors=7698164/22719/0/0, tx=7815947/23756/0/0
        port4: physical/1000auto, up, rx-bytes/packets/dropped/errors=714501/1749/0/0, tx=724254/1763/0/0
    FGVMEV7000000005(updated 2 seconds ago):
        port2: physical/1000auto, up, rx-bytes/packets/dropped/errors=7819515/23764/0/0, tx=7697305/22724/0/0
        port4: physical/1000auto, up, rx-bytes/packets/dropped/errors=726500/1766/0/0, tx=714129/1751/0/0
MONDEV stats:
    FGVMEVYKXTDJN932(updated 3 seconds ago):
        port3: physical/1000auto, up, rx-bytes/packets/dropped/errors=4610/15/0/0, tx=1224/21/0/0
    FGVMEV7000000005(updated 2 seconds ago):
        port3: physical/1000auto, up, rx-bytes/packets/dropped/errors=1200/20/0/0, tx=630/10/0/0
Primary     : FGDocs-P        , FGVMEV0000000002, HA cluster index = 0
Secondary   : FGDocs-S        , FGVMEV7000000005, HA cluster index = 1
number of vcluster: 1
vcluster 1: work 169.254.0.1
Primary: FGVMEV0000000002, HA operating index = 0
Secondary: FGVMEV7000000005, HA operating index = 1 

When one of the members is out of synchronization:

# get system ha status
HA Health Status: OK
Model: FortiGate-VM64
Mode: HA A-P
Group Name: docs
Group ID: 0
Debug: 0
Cluster Uptime: 0 days 2:24:46
Cluster state change time: 2021-04-29 13:17:03
Primary selected using:
    <2021/04/29 13:17:03> FGVMEV0000000002 is selected as the primary because its uptime is larger than peer member FGVMEV7000000005.
    <2021/04/29 12:37:17> FGVMEV0000000002 is selected as the primary because it's the only member in the cluster.
ses_pickup: disable
override: disable
Configuration Status:
    FGVMEV0000000002(updated 0 seconds ago): in-sync
    FGVMEV7000000005(updated 3 seconds ago): out-of-sync
System Usage stats:
    FGVMEV0000000002(updated 0 seconds ago):
        sessions=11, average-cpu-user/nice/system/idle=1%/0%/0%/99%, memory=67%
    FGVMEV7000000005(updated 3 seconds ago):
        sessions=0, average-cpu-user/nice/system/idle=0%/0%/0%/100%, memory=65%
HBDEV stats:
    FGVMEV0000000002(updated 0 seconds ago):
        port2: physical/1000auto, up, rx-bytes/packets/dropped/errors=22257271/64684/0/0, tx=24404848/69893/0/0
        port4: physical/1000auto, up, rx-bytes/packets/dropped/errors=12026623/29407/0/0, tx=12200664/29417/0/0
    FGVMEV7000000005(updated 3 seconds ago):
        port2: physical/1000auto, up, rx-bytes/packets/dropped/errors=24401109/69877/0/0, tx=22245634/64666/0/0
        port4: physical/1000auto, up, rx-bytes/packets/dropped/errors=12195025/29401/0/0, tx=12018480/29390/0/0
MONDEV stats:
    FGVMEV0000000002(updated 0 seconds ago):
        port3: physical/1000auto, up, rx-bytes/packets/dropped/errors=4610/15/0/0, tx=1224/21/0/0
    FGVMEV7000000005(updated 3 seconds ago):
        port3: physical/1000auto, up, rx-bytes/packets/dropped/errors=1200/20/0/0, tx=630/10/0/0
Primary     : FGDocs-P        , FGVMEV0000000002, HA cluster index = 0
Secondary   : FGDocs-S        , FGVMEV7000000005, HA cluster index = 1
number of vcluster: 1
vcluster 1: work 169.254.0.1
Primary: FGVMEV0000000002, HA operating index = 0
Secondary: FGVMEV7000000005, HA operating index = 1

Filtering the synchronization status by virtual cluster ID

The HA status information can be filtered for a specific virtual cluster in the CLI. The following example HA configuration has 10 virtual clusters, and the status information is filtered for virtual cluster 8.

To view the HA synchronization status of the entire cluster:

# get system ha status
HA Health Status:
    WARNING: FG101FTK19000007 has hbdev down;
    WARNING: FG101FTK19000009 has hbdev down;
Model: FortiGate-101F
Mode: HA A-P
Group Name: FGDocs
Group ID: 5
Debug: 0
Cluster Uptime: 0 days 0:8:29
Cluster state change time: 2023-05-24 15:00:56
Primary selected using:
  virtual cluster 1:
<2023/05/24 14:53:11> vcluster-1: FG101FTK19000007 is selected as the primary because its override priority is larger than peer member FG101FTK19000009.
  virtual cluster 2:
<2023/05/24 14:58:49> vcluster-2: FG101FTK19000007 is selected as the primary because its override priority is larger than peer member FG101FTK19000009.
  virtual cluster 3:
<2023/05/24 14:58:49> vcluster-3: FG101FTK19000007 is selected as the primary because its override priority is larger than peer member FG101FTK19000009.
  ...
  virtual cluster 10:
<2023/05/24 15:00:56> vcluster-10: FG101FTK19000009 is selected as the primary because its override priority is larger than peer member FG101FTK19000007.
<2023/05/24 15:00:55> vcluster-10: FG101FTK19000007 is selected as the primary because its override priority is larger than peer member FG101FTK19000009.
ses_pickup: disable
override:
    vcluster_1 disable
    vcluster_2 disable
    vcluster_3 disable
    ...
    vcluster_10 disable
Configuration Status:
    FG101FTK19000007(updated 2 seconds ago): in-sync
    FG101FTK19000007 chksum dump: bb 79 03 7c bf 28 16 dc 14 99 23 9f 53 94 1a f0
    FG101FTK19000009(updated 1 seconds ago): out-of-sync
    FG101FTK19000009 chksum dump: 92 24 8c 43 db ed a3 f2 6f 6c cc 06 56 f5 0f 76
System Usage stats:
    FG101FTK19000007(updated 2 seconds ago):
        sessions=11, average-cpu-user/nice/system/idle=0%/0%/0%/99%, memory=33%
    FG101FTK19000009(updated 1 seconds ago):
        sessions=0, average-cpu-user/nice/system/idle=0%/0%/0%/99%, memory=32%
HBDEV stats:
    FG101FTK19000007(updated 2 seconds ago):
        ha1: physical/1000auto, up, rx-bytes/packets/dropped/errors=8490968/16615/0/0, tx=3116622/7352/0/0
        ha2: physical/00, down, rx-bytes/packets/dropped/errors=0/0/0/0, tx=0/0/0/0
    FG101FTK19000009(updated 1 seconds ago):
        ha1: physical/1000auto, up, rx-bytes/packets/dropped/errors=7359328/16036/0/0, tx=4369340/8183/0/0
        ha2: physical/00, down, rx-bytes/packets/dropped/errors=0/0/0/0, tx=0/0/0/0
Primary     : FortiGate-101F  , FG101FTK19000007, HA cluster index = 0
Secondary   : FortiGate-101F  , FG101FTK19000009, HA cluster index = 1
number of vcluster: 10
vcluster 1: work 169.254.0.1
Primary: FG101FTK19000007, HA operating index = 0
Secondary: FG101FTK19000009, HA operating index = 1
vcluster 2: work 169.254.0.1
Primary: FG101FTK19000007, HA operating index = 0
Secondary: FG101FTK19000009, HA operating index = 1
vcluster 3: work 169.254.0.1
Primary: FG101FTK19000007, HA operating index = 0
Secondary: FG101FTK19000009, HA operating index = 1
...
vcluster 10: standby 169.254.0.2
Secondary: FG101FTK19000007, HA operating index = 1
Primary: FG101FTK19000009, HA operating index = 0

To view the HA synchronization status of virtual cluster 8:

# get system ha status 8
HA Health Status:
    WARNING: FG101FTK19000007 has hbdev down;
    WARNING: FG101FTK19000009 has hbdev down;
Model: FortiGate-101F
Mode: HA A-P
Group Name: FGDocs
Group ID: 5
Debug: 0
Cluster Uptime: 0 days 0:8:48
Cluster state change time: 2023-05-24 15:00:56
Primary selected using:
  virtual cluster 8:
<2023/05/24 15:00:55> vcluster-8: FG101FTK19000007 is selected as the primary because its override priority is larger than peer member FG101FTK19000009.
ses_pickup: disable
override:
    vcluster_8 disable
Configuration Status:
    FG101FTK19000007(updated 2 seconds ago): in-sync
    FG101FTK19000007 chksum dump: bb 79 03 7c bf 28 16 dc 14 99 23 9f 53 94 1a f0
    FG101FTK19000009(updated 1 seconds ago): out-of-sync
    FG101FTK19000009 chksum dump: 92 24 8c 43 db ed a3 f2 6f 6c cc 06 56 f5 0f 76
System Usage stats:
    FG101FTK19000007(updated 2 seconds ago):
        sessions=11, average-cpu-user/nice/system/idle=0%/0%/0%/99%, memory=33%
    FG101FTK19000009(updated 1 seconds ago):
        sessions=0, average-cpu-user/nice/system/idle=0%/0%/0%/99%, memory=32%
HBDEV stats:
    FG101FTK19000007(updated 2 seconds ago):
        ha1: physical/1000auto, up, rx-bytes/packets/dropped/errors=8721683/17144/0/0, tx=3214512/7536/0/0
        ha2: physical/00, down, rx-bytes/packets/dropped/errors=0/0/0/0, tx=0/0/0/0
    FG101FTK19000009(updated 1 seconds ago):
        ha1: physical/1000auto, up, rx-bytes/packets/dropped/errors=7616471/16547/0/0, tx=4440283/8383/0/0
        ha2: physical/00, down, rx-bytes/packets/dropped/errors=0/0/0/0, tx=0/0/0/0
Primary     : FortiGate-101F  , FG101FTK19000007, HA cluster index = 0
Secondary   : FortiGate-101F  , FG101FTK19000009, HA cluster index = 1
number of vcluster: 10
vcluster 8: work 169.254.0.1
Primary: FG101FTK19000007, HA operating index = 0
Secondary: FG101FTK19000009, HA operating index = 1