Cluster virtual MAC addresses

In a cluster, the FGCP assigns virtual MAC addresses (VMACs) to each primary device interface. HA uses VMAC addresses so that if a failover occurs, the new primary device interfaces will have the same VMAC addresses and IP addresses as the failed primary device. As a result, most network equipment will identify the new primary device as the same device as the failed primary device and still be able to communicate with the cluster.

If a cluster is operating in NAT mode, the FGCP assigns a different VMAC address to each primary device interface. VLAN subinterfaces are assigned the same VMAC address as the physical interface that the VLAN subinterface is added to. Redundant or 802.3ad aggregate interfaces are assigned the VMAC address of the first interface in the redundant or aggregate list.

If a cluster is operating in transparent mode, the FGCP assigns a VMAC address to the primary device's management IP address. Since you can connect to the management IP address from any interface, all FortiGate interfaces appear to have the same VMAC address.

The MAC address of a reserved management interface does not change to a VMAC address; it keeps its original MAC address.

Subordinate device MAC addresses do not change. Use diagnose hardware deviceinfo nic <interface> on the subordinate device to display the MAC addresses of each interface.

A MAC address conflict can occur when two clusters are operating on the same network using the same group ID (see Diagnosing packet loss). It is recommended that each cluster in the same network and broadcast domain uses a unique group ID.

Failover

When the new primary device is selected after a failover, the primary device sends gratuitous ARP packets to update the devices connected to the cluster interfaces (usually layer 2 switches) with the VMAC addresses. This is sometimes called using gratuitous ARP packets (or GARP packets) to train the network. The gratuitous ARP packets sent from the primary unit are intended to make sure that the layer 2 switch forwarding databases (FDBs) are updated as quickly as possible.

Sending gratuitous ARP packets is not a requirement because connected devices will eventually learn of the new ports to forward the packets to. However, many network switches will update their FDBs more quickly after a failover if the new primary device sends gratuitous ARP packets.

Configuring ARP packet settings

The following settings can be configured.

config system ha
    set arps <integer>
    set arps-interval <integer>
    set gratuitous-arps {enable | disable}
    set link-failed-signal {enable | disable}
end

arps <integer>	Set the number of gratuitous ARPs; lower the value to reduce traffic, and increase the value to reduce failover time (1 - 60, default = 5).
arps-interval <integer>	Set the time between gratuitous ARPs; lower the value to reduce failover time, and increase the value to reduce traffic, in seconds (1 - 20, default = 8).
gratuitous-arps {enable | disable}	Enable/disable gratuitous ARPs (default = enable).
link-failed-signal {enable | disable}	Enable/disable shutting down all interfaces for one second after a failover. Use if gratuitous ARPs do not update the network (default = disable).

If you disable sending gratuitous ARP packets, it is recommended to enable the link-failed-signal setting. The linked-fail-signal alerts the connected switches of a failed link, which triggers them to react immediately to the changes.

For more information about gratuitous ARP packets see RFC 826 and RFC 3927.

Determining VMAC addresses

A VMAC address is determined using specific logic and variables. The variables are:

• Virtual cluster (vcluster) ID: What is the vcluster ID that the interface belongs to?

  The vcluster ID used in the following logic is the number configured in the FortiGate minus 1. In this example the cluster ID is 0: config vcluster edit 1

• Group ID: What is the group ID that the interface belongs to?

• Physical index: What is the value of the physical index receiving the VMAC?

  The following command can be used to locate the physical index number:

  # diagnose sys ha dump-by debug-zone 
  (…)
  <hatalk> 		mgmt 	ifindex=4	phyindex=0		mac=04.d5…
  <hatalk>		ha	ifindex=3	phyindex=1		mac=04.d5…
  <hatalk>		wan1	ifindex=17	phyindex=2		mac=04.d5…
  <hatalk>		wan2	ifindex=18	phyindex=3		mac=04.d5…

The physical indexes used in this document are examples and may not match your interface indexes.

The logic uses the vcluster ID, group ID, and physical index variables to determine VMAC addresses as summarized in the following table:

Logic	Vcluster ID	Group ID	Physical index
Logic 1	0 or 1	Greater than 255	Less than 128
Logic 2	0 or 1	Less than 256	Less than 128
Logic 3	0 or 1	Less than 256	Greater than 127
Logic 4	Greater than 1	0-7	Less than 1024

Logic 1: vcluster ID 0 or 1, group ID > 255, and physical index < 128

The start of the VMAC address is always e0:23:ff:--:--:-- with the last 24 bits defined as follows:

Preset bits	Group ID	Vcluster ID	Physical index
1 1 1 1 1 1	- - : - - - - - - - - :	-	- - - - - - -

This example uses group ID = 500 and vcluster ID = 0:

• Group ID

  • 500 – 256 = 244

  • 244₁₀ = 011110100₂

  • Group ID = 0011110100

• Vcluster ID

  • 0₁₀ = 0₂

  • Vcluster ID = 0

• Physical index

  • port7 physical index = 10

    • 10₁₀ = 0001010₂

  • port8 physical index = 12

    • 12₁₀ = 0001100₂

  • port9 physical index = 14

    • 14₁₀ = 0001110₂

Resulting in these VMAC addresses:

Interface	VMAC binary (last 24 bits)				VMAC hex (full)
	Preset bits	Group ID	Vcluster ID	Physical index
port1	1 1 1 1 1 1	0 0 1 1 1 1 0 1 0 0	0	0 0 0 1 0 1 0	e0:23:ff:fc:f4:0a
port2	1 1 1 1 1 1	0 0 1 1 1 1 0 1 0 0	0	0 0 0 1 1 0 0	e0:23:ff:fc:f4:0c
port3	1 1 1 1 1 1	0 0 1 1 1 1 0 1 0 0	0	0 0 0 1 1 1 0	e0:23:ff:fc:f4:0e

Logic 2: vcluster ID 0 & 1, group ID < 256, and physical index < 128

The start of the VMAC address is always 00:09:0f:09:--:-- with the last 16 bits defined as follows:

Group ID	Vcluster ID	Physical index
- - - - - - - - : -	-	- - - - - - -

This example uses group ID = 200, vcluster ID = 1, and interfaces with physical indexes less than 128:

• Group ID:

  • 200₁₀ = 11001000₂

  • Group ID = 11001000

• Vcluster ID:

  • 1₁₀ = 1₂

  • Vcluster ID = 1

• Interfaces:

  • port25 physical index = 100

    • 100₁₀ = 1100100₂

  • port31 physical index = 120

    • 120₁₀ = 1111000₂

  • port38 physical index = 127

    • 127₁₀ = 1111111₂

Resulting in these VMAC addresses:

Interface	VMAC binary (last 16 bits)			VMAC hex (full)
	Group ID	Vcluster ID	Physical index
port1	1 1 0 0 1 0 0 0	1	1 1 0 0 1 0 0	00:09:0f:09:c8:e4
port2	1 1 0 0 1 0 0 0	1	1 1 1 1 0 0 0	00:09:0f:09:c8:f8
port3	1 1 0 0 1 0 0 0	1	1 1 1 1 1 1 1	00:09:0f:09:c8:ff

Logic 3: vcluster ID 0 & 1, group ID < 256, and physical index > 127

The start of the VMAC address is always 70:4c:a5:--:--:-- with the last 24 bits defined as follows:

Physical index	Vcluster ID	Group ID
- - - - - - - - : - - - - - - -	- :	- - - - - - - -

This example uses group ID = 25, vcluster ID = 1, and interfaces with physical indexes above 127:

• Group ID:

  • 25₁₀ = 11001₂

  • Group ID = 00011001

• Vcluster ID:

  • 1₁₀ = 1₂

  • Vcluster ID = 1

• Interfaces:

  • port40 physical index = 230

    • 230 – 128 = 102

    • 102₁₀ = 1100110₂

  • port45 physical index = 240

    • 240 – 128 = 112

    • 112₁₀ = 1110000₂

  • port50 physical index = 250

    • 250 – 128 = 122

    • 122₁₀ = 1111010₂

Resulting in these VMAC addresses:

Interface	VMAC binary (last 24 bits)			VMAC hex (full)
	Physical index	Vcluster ID	Group ID
port1	0 0 0 0 0 0 0 0 1 1 0 0 1 1 0	1	0 0 0 1 1 0 0 1	70:4c:a5:00:cd:19
port2	0 0 0 0 0 0 0 0 1 1 1 0 0 0 0	1	0 0 0 1 1 0 0 1	70:4c:a5:00:e1:19
port3	0 0 0 0 0 0 0 0 1 1 1 1 0 1 0	1	0 0 0 1 1 0 0 1	70:4c:a5:00:f5:19

Logic 4: vcluster ID >= 2

When the vcluster ID is 2 or greater, the group ID must be between 0 and 7.

The start of the VMAC address is always e0:23:ff:--:--:-- with the last 24 bits defined as follows:

Preset bits	Physical index	Vcluster ID	Group ID
1 1 1 1 1 1	- - : - - - - - - - - :	- - - - -	- - -

This example uses group ID = 6 and vcluster ID = 9:

• Group ID

  • 6₁₀ = 110₂

  • Group ID = 110

• Vcluster ID

  • 9₁₀ = 1001₂

  • Vcluster ID = 01001

• Interfaces

  • port1 physical index = 3₁₀

    • 3₁₀ = 0000000011₂

  • port2 physical index = 6₁₀

    • 6₁₀ = 0000000110₂

  • port3 physical index = 9₁₀

    • 9₁₀ = 0000001001₂

Resulting in these VMAC addresses:

Interface	VMAC binary (last 24 bits)				VMAC hex (full)
	Preset bits	Physical index	Vcluster ID	Group ID
port1	1 1 1 1 1 1	0 0 0 0 0 0 0 0 1 1	0 1 0 0 1	1 1 0	e0:23:ff:fc:03:4e
port2	1 1 1 1 1 1	0 0 0 0 0 0 0 1 1 0	0 1 0 0 1	1 1 0	e0:23:ff:fc:06:4e
port3	1 1 1 1 1 1	0 0 0 0 0 0 1 0 0 1	0 1 0 0 1	1 1 0	e0:23:ff:fc:09:4e

Displaying VMAC addresses

Each FortiGate physical interface has two MAC addresses: the permanent and current hardware addresses. The permanent hardware address cannot be changed, as it is the actual MAC address of the interface hardware. The current hardware address can be changed, as it is the address seen by the network.

To change the current hardware address on a FortiGate not operating in HA:

config system interface
    edit <name>
        set macaddr <address>
    next
end

In an operating cluster, the current hardware address of each cluster device interface is changed to the HA virtual MAC address by the FGCP. The macaddr option is not available for a functioning cluster.

To display MAC addresses on a FortiGate operating in HA:

# diagnose hardware deviceinfo nic port1
...
Current_HWaddr 00:09:0f:09:ff:02
Permanent_HWaddr 08:5b:0e:72:3b:b2

Diagnosing packet loss

A network can experience packet loss when two FortiGate HA clusters are deployed in the same broadcast domain due to MAC address conflicts. You can resolve the MAC address conflict by changing the HA group ID (or cluster ID) configuration of the two clusters.

You can diagnose packet loss by pinging from one cluster to the other, or by pinging both of the clusters from a device within the broadcast domain.

To check for a MAC address conflict in a HA cluster:

1. On Cluster_1 and Cluster_2, check the VMAC address (Current_HWaddr) used in an interface on the primary device:

   # diagnose hardware deviceinfo nic <interface>

   If the group prefix and group hexadecimal ID are identical, there will be MAC address conflicts.

2. Change one of the clusters to use a different group ID:

   config system ha
       set group-id <integer>
   end