Cluster virtual MAC addresses

In a cluster, each primary device interface requires a virtual MAC (VMAC) address. HA uses VMAC addresses during failover. If a failover occurs, the new primary device interfaces will have the same VMAC addresses and IP addresses as the failed primary device. As a result, most network equipment will identify the new primary device as the same device as the failed primary device and continue to communicate with the cluster.

The following methods are available to assign VMAC addresses to interfaces:

• Manual assignment

• Automatic assignment based on MAC address

• Automatic assignment based on vcluster ID, group ID, and physical index

See Determining VMAC addresses for more information.

If a cluster is operating in NAT mode with an automatic VMAC assignment method, FGCP assigns a different VMAC address to each primary device interface. VLAN subinterfaces are assigned the same VMAC address as the physical interface that the VLAN subinterface is added to. Redundant or 802.3ad aggregate interfaces are assigned the VMAC address of the first interface in the redundant or aggregate list.

If a cluster is operating in transparent mode with an automatic VMAC assignment method, FGCP assigns a VMAC address to the primary device's management IP address. Since you can connect to the management IP address from any interface, all FortiGate interfaces appear to have the same VMAC address.

The MAC address of a reserved management interface does not change to a VMAC address; it keeps its original MAC address.

Subordinate device MAC addresses do not change. Use diagnose hardware deviceinfo nic <interface> on the subordinate device to display the MAC addresses of each interface.

A MAC address conflict can occur when two clusters are operating on the same network using the same group ID (see Diagnosing packet loss). It is recommended that each cluster in the same network and broadcast domain uses a unique group

Failover

When the new primary device is selected after a failover, the primary device sends gratuitous ARP packets to update the devices connected to the cluster interfaces (usually layer 2 switches) with the VMAC addresses. This is sometimes called using gratuitous ARP packets (or GARP packets) to train the network. The gratuitous ARP packets sent from the primary unit are intended to make sure that the layer 2 switch forwarding databases (FDBs) are updated as quickly as possible.

Sending gratuitous ARP packets is not a requirement because connected devices will eventually learn of the new ports to forward the packets to. However, many network switches will update their FDBs more quickly after a failover if the new primary device sends gratuitous ARP packets.

Configuring ARP packet settings

The following settings can be configured.

config system ha
    set arps <integer>
    set arps-interval <integer>
    set gratuitous-arps {enable | disable}
    set link-failed-signal {enable | disable}
end

arps <integer>	Set the number of gratuitous ARPs; lower the value to reduce traffic, and increase the value to reduce failover time (1 - 60, default = 5).
arps-interval <integer>	Set the time between gratuitous ARPs; lower the value to reduce failover time, and increase the value to reduce traffic, in seconds (1 - 20, default = 8).
gratuitous-arps {enable | disable}	Enable/disable gratuitous ARPs (default = enable).
link-failed-signal {enable | disable}	Enable/disable shutting down all interfaces for one second after a failover. Use if gratuitous ARPs do not update the network (default = disable).

If you disable sending gratuitous ARP packets, it is recommended to enable the link-failed-signal setting. The linked-fail-signal alerts the connected switches of a failed link, which triggers them to react immediately to the changes.

For more information about gratuitous ARP packets see RFC 826 and RFC 3927.

Determining VMAC addresses

The following methods are available to assign virtual MAC addresses to interfaces, and they are listed in order of highest to lowest priority:

• Manual VMAC assignment

• Automatic VMAC assignment based on hardware MAC address

• Automatic VMAC assignment based on vcluster ID, group ID, and physical index

Manual VMAC assignment

You can manually assign virtual MAC addresses to each interface, including physical, EMAC, or FortiExtender interfaces. Manual virtual MAC assignments override other virtual MAC address assignments on the interface.

config system interface
    edit <interface>
        set virtual-mac <mac_address>
    next
end

To manually assign a virtual MAC address to an interface:

config system interface
    edit "wan1"
        set ip 172.16.200.1 255.255.255.0
        set allowaccess ping https ssh snmp http telnet
        set virtual-mac 06:d5:90:04:f8:9c
        set type physical
        set snmp-index 3
        config ipv6
            set ip6-address 2000:172:16:200::1/64
            set ip6-allowaccess ping https ssh snmp http telnet
        end
    next
end

Automatic VMAC assignment based on hardware MAC address

FGCP can automatically assign virtual MAC addresses to physical interfaces based on hardware MAC address of the primary device with the locally administered bit (U/L bit) changed to 1. For example, 00:xx:xx:xx:xx:xx becomes 02:xx:xx:xx:xx:xx. You must use set auto-virtual-mac-interface to the interfaces to use this feature.

config system ha
    set auto-virtual-mac-interface <interface> [interface(s)]
end

In a 48-bit MAC address, the U/L bit refers to the second least significant bit in the first octet of the hexadecimal MAC address. When this bit is 0, it indicates that the MAC address is Universal, meaning that it is assigned by a central authority. When this bit is 1, it indicates that the MAC address is Local, meaning that it is assigned locally. For example, the first octet of 00 represented in binary is 00000000, where the U/L bit is 0. Whereas the first octet of 02 represented in binary is 00000010, where the U/L bit is set to 1.

To configure automatic virtual MAC address assignment:

config system ha
    set group-id 20
    set group-name "MMMMM"
    set mode a-p
    set hbdev "ha1" 50 "ha2" 100
    set auto-virtual-mac-interface "wan1" "port1" "port2" "ha1" "ha2" "port3" "port4" "port5" "port6" "port7" "port8" "dmz"
    set upgrade-mode simultaneous
    set override enable
    set priority 200
end

Automatic VMAC assignment based on vcluster ID, group ID, and physical index

FGCP can automatically assign virtual MAC addresses to each primary device interface based on vcluster ID, group ID, and physical index. This automatic VMAC assignment method is used when the set auto-virtual-mac-interface option is disabled.

With this method, specific logic and variables determine VMAC addresses. The variables are:

• Virtual cluster (vcluster) ID: What is the vcluster ID that the interface belongs to?

  The vcluster ID used in the following logic is the number configured in the FortiGate minus 1. In this example the cluster ID is 0: config vcluster edit 1

• Group ID: What is the group ID that the interface belongs to?

• Physical index: What is the value of the physical index receiving the VMAC?

  The following command can be used to locate the physical index number:

  # diagnose sys ha dump-by debug-zone 
  (…)
  <hatalk> 		mgmt 	ifindex=4	phyindex=0		mac=04.d5…
  <hatalk>		ha	ifindex=3	phyindex=1		mac=04.d5…
  <hatalk>		wan1	ifindex=17	phyindex=2		mac=04.d5…
  <hatalk>		wan2	ifindex=18	phyindex=3		mac=04.d5…

The physical indexes used in this document are examples and may not match your interface indexes.

The logic uses the vcluster ID, group ID, and physical index variables to determine VMAC addresses as summarized in the following table:

Logic	Vcluster ID	Group ID	Physical index
Logic 1	0 or 1	Greater than 255	Less than 128
Logic 2	0 or 1	Less than 256	Less than 128
Logic 3	0 or 1	Less than 256	Greater than 127
Logic 4	Greater than 1	0-7	Less than 1024

Logic 1: vcluster ID 0 or 1, group ID > 255, and physical index < 128

The start of the VMAC address is always e0:23:ff:--:--:-- with the last 24 bits defined as follows:

Preset bits	Group ID	Vcluster ID	Physical index
1 1 1 1 1 1	- - : - - - - - - - - :	-	- - - - - - -

This example uses group ID = 500 and vcluster ID = 0:

• Group ID

  • 500 – 256 = 244

  • 244₁₀ = 011110100₂

  • Group ID = 0011110100

• Vcluster ID

  • 0₁₀ = 0₂

  • Vcluster ID = 0

• Physical index

  • port7 physical index = 10

    • 10₁₀ = 0001010₂

  • port8 physical index = 12

    • 12₁₀ = 0001100₂

  • port9 physical index = 14

    • 14₁₀ = 0001110₂

Resulting in these VMAC addresses:

Interface	VMAC binary (last 24 bits)				VMAC hex (full)
	Preset bits	Group ID	Vcluster ID	Physical index
port1	1 1 1 1 1 1	0 0 1 1 1 1 0 1 0 0	0	0 0 0 1 0 1 0	e0:23:ff:fc:f4:0a
port2	1 1 1 1 1 1	0 0 1 1 1 1 0 1 0 0	0	0 0 0 1 1 0 0	e0:23:ff:fc:f4:0c
port3	1 1 1 1 1 1	0 0 1 1 1 1 0 1 0 0	0	0 0 0 1 1 1 0	e0:23:ff:fc:f4:0e

Logic 2: vcluster ID 0 & 1, group ID < 256, and physical index < 128

The start of the VMAC address is always 00:09:0f:09:--:-- with the last 16 bits defined as follows:

Group ID	Vcluster ID	Physical index
- - - - - - - - : -	-	- - - - - - -

This example uses group ID = 200, vcluster ID = 1, and interfaces with physical indexes less than 128:

• Group ID:

  • 200₁₀ = 11001000₂

  • Group ID = 11001000

• Vcluster ID:

  • 1₁₀ = 1₂

  • Vcluster ID = 1

• Interfaces:

  • port25 physical index = 100

    • 100₁₀ = 1100100₂

  • port31 physical index = 120

    • 120₁₀ = 1111000₂

  • port38 physical index = 127

    • 127₁₀ = 1111111₂

Resulting in these VMAC addresses:

Interface	VMAC binary (last 16 bits)			VMAC hex (full)
	Group ID	Vcluster ID	Physical index
port1	1 1 0 0 1 0 0 0	1	1 1 0 0 1 0 0	00:09:0f:09:c8:e4
port2	1 1 0 0 1 0 0 0	1	1 1 1 1 0 0 0	00:09:0f:09:c8:f8
port3	1 1 0 0 1 0 0 0	1	1 1 1 1 1 1 1	00:09:0f:09:c8:ff

Logic 3: vcluster ID 0 & 1, group ID < 256, and physical index > 127

The start of the VMAC address is always 70:4c:a5:--:--:-- with the last 24 bits defined as follows:

Physical index	Vcluster ID	Group ID
- - - - - - - - : - - - - - - -	- :	- - - - - - - -

This example uses group ID = 25, vcluster ID = 1, and interfaces with physical indexes above 127:

• Group ID:

  • 25₁₀ = 11001₂

  • Group ID = 00011001

• Vcluster ID:

  • 1₁₀ = 1₂

  • Vcluster ID = 1

• Interfaces:

  • port40 physical index = 230

    • 230 – 128 = 102

    • 102₁₀ = 1100110₂

  • port45 physical index = 240

    • 240 – 128 = 112

    • 112₁₀ = 1110000₂

  • port50 physical index = 250

    • 250 – 128 = 122

    • 122₁₀ = 1111010₂

Resulting in these VMAC addresses:

Interface	VMAC binary (last 24 bits)			VMAC hex (full)
	Physical index	Vcluster ID	Group ID
port1	0 0 0 0 0 0 0 0 1 1 0 0 1 1 0	1	0 0 0 1 1 0 0 1	70:4c:a5:00:cd:19
port2	0 0 0 0 0 0 0 0 1 1 1 0 0 0 0	1	0 0 0 1 1 0 0 1	70:4c:a5:00:e1:19
port3	0 0 0 0 0 0 0 0 1 1 1 1 0 1 0	1	0 0 0 1 1 0 0 1	70:4c:a5:00:f5:19

Logic 4: vcluster ID >= 2

When the vcluster ID is 2 or greater, the group ID must be between 0 and 7.

The start of the VMAC address is always e0:23:ff:--:--:-- with the last 24 bits defined as follows:

Preset bits	Physical index	Vcluster ID	Group ID
1 1 1 1 1 1	- - : - - - - - - - - :	- - - - -	- - -

This example uses group ID = 6 and vcluster ID = 9:

• Group ID

  • 6₁₀ = 110₂

  • Group ID = 110

• Vcluster ID

  • 9₁₀ = 1001₂

  • Vcluster ID = 01001

• Interfaces

  • port1 physical index = 3₁₀

    • 3₁₀ = 0000000011₂

  • port2 physical index = 6₁₀

    • 6₁₀ = 0000000110₂

  • port3 physical index = 9₁₀

    • 9₁₀ = 0000001001₂

Resulting in these VMAC addresses:

Interface	VMAC binary (last 24 bits)				VMAC hex (full)
	Preset bits	Physical index	Vcluster ID	Group ID
port1	1 1 1 1 1 1	0 0 0 0 0 0 0 0 1 1	0 1 0 0 1	1 1 0	e0:23:ff:fc:03:4e
port2	1 1 1 1 1 1	0 0 0 0 0 0 0 1 1 0	0 1 0 0 1	1 1 0	e0:23:ff:fc:06:4e
port3	1 1 1 1 1 1	0 0 0 0 0 0 1 0 0 1	0 1 0 0 1	1 1 0	e0:23:ff:fc:09:4e

Displaying VMAC addresses

Each FortiGate physical interface has two MAC addresses: the permanent and current hardware addresses. The permanent hardware address cannot be changed, as it is the actual MAC address of the interface hardware. The current hardware address can be changed, as it is the address seen by the network.

To change the current hardware address on a FortiGate not operating in HA:

config system interface
    edit <name>
        set macaddr <address>
    next
end

In an operating cluster, the current hardware address of each cluster device interface is changed to the HA virtual MAC address by the FGCP. The macaddr option is not available for a functioning cluster.

To display MAC addresses on a FortiGate operating in HA:

# diagnose hardware deviceinfo nic port1
...
Current_HWaddr 00:09:0f:09:ff:02
Permanent_HWaddr 08:5b:0e:72:3b:b2

Diagnosing packet loss

A network can experience packet loss when two FortiGate HA clusters are deployed in the same broadcast domain due to MAC address conflicts. You can resolve the MAC address conflict by changing the HA group ID (or cluster ID) configuration of the two clusters.

You can diagnose packet loss by pinging from one cluster to the other, or by pinging both of the clusters from a device within the broadcast domain.

To check for a MAC address conflict in a HA cluster:

1. On Cluster_1 and Cluster_2, check the VMAC address (Current_HWaddr) used in an interface on the primary device:

   # diagnose hardware deviceinfo nic <interface>

   If the group prefix and group hexadecimal ID are identical, there will be MAC address conflicts.

2. Change one of the clusters to use a different group ID:

   config system ha
       set group-id <integer>
   end