Out-of-band management with reserved management interfaces

As part of an HA configuration, you can reserve up to four management interfaces to provide direct management access to all cluster units. For each reserved management interface, you can configure a different IP address, administrative access, and other interface settings, for each cluster unit. By connecting these interfaces to your network, you can separately manage each cluster unit from different IP addresses.

• Reserved management interfaces provide direct management access to each cluster unit, and give each cluster unit a different identity on your network. This simplifies using external services, such as SNMP, to monitor separate cluster units.

• Reserved management interfaces are not assigned HA virtual MAC addresses. They retain the permanent hardware address of the physical interface, unless you manually change it using the config system interface command.

• Reserved management interfaces and their IP addresses should not be used for managing a cluster using FortiManager. To manage a FortiGate HA cluster with FortiManager, use the IP address of one of the cluster unit interfaces.

• Configuration changes to a reserved management interface are not synchronized to other cluster units. Other configuration changes are automatically synchronized to all cluster units.

You can configure an in-band management interface for a cluster unit. See In-band management for information. In-band management does not reserve the interface exclusively for HA management.

Management interface

Enable HTTPS or HTTP administrative access on the reserved management interfaces to connect to the GUI of each cluster unit. On secondary units, the GUI has the same features as the primary unit, except for unit specific information, for example:

• The System Information widget on the Status dashboard shows the secondary unit's serial number.

• In the cluster members list at System > HA, you can change the HA configuration of the unit that you are logged into. You can only change the host name and device priority of the primary and other secondary units.

• The system events logs show logs for the device that you are logged into. Use the HA device drop down to view the log messages for other cluster units, including the primary unit.

Enable SSH administrative access on the reserved management interfaces to connect to the CLI of each cluster unit. The CLI prompt includes the host of the cluster unit that you are connected to. Use the execute ha manage command to connect to other cluster unit CLIs.

Enable SNMP administrative access on a reserved management interface to use SNMP to monitor each cluster unit using the interface's IP address. Direct management of cluster members must also be enabled, see Configuration examples.

Reserved management interfaces are available in both NAT and transparent mode, and when the cluster is operating with multiple VDOMs.

FortiCloud, FortiSandbox, and other management services

By default, management services such as FortiCloud, FortiSandbox, SNMP, remote logging, and remote authentication, use a cluster interface. This means that communication from each cluster unit will come from a cluster interface of the primary unit, and not from the individual cluster unit's interface.

You can configure HA reserved management interfaces to be used for communication with management services by enabling the ha-direct option. This separates management traffic for each cluster unit, and allows each unit to be individually managed. This is especially useful when cluster units are in different physical locations.

The following management features will then use the HA reserved management interface:

• Remote logging, including syslog, FortiAnalyzer, and FortiCloud

• Remote authentication and certificate verification

• Communication with FortiSandbox

• Netflow and sflow, see Routing NetFlow data over the HA management interface for information.

• SNMP queries and traps

Syntax for HA reserved management interfaces is as follows:

config system ha
    set ha-direct enable
    set ha-mgmt-status enable
    config ha-mgmt-interfaces
        edit 1
            set interface <interface>
            set dst <destination IP>
            set gateway <IPv4 gateway>
            set gateway6 <IPv6 gateway>
        next
    end
end

The ha-direct option is a pre-requisite for allowing communication on each HA reserved management interface for various management services listed above. Once enabled, all source-ip settings will be unset from log related, netflow and sflow management services. SNMP requires ha-direct to be configured under SNMP settings only. See below for more configuration options.

Configuration examples

The configuration examples below will use the following topology:

Two FortiGate units are already operating in a cluster. On each unit, port8 is connected to the internal network through a switch and configured as an out-of-band reserved management interface.

Configuration changes to the reserved management interface are not synchronized to other cluster units.

Administrative access and default route for HA management interface

To configure the primary unit's reserved management interface, configure an IP address and management access on port8. Then, configure the necessary HA settings to enable the HA reserved management interface and its route. To configure the secondary unit's reserved management interface, access the unit's CLI through the primary unit, and configure an IP address, management access on port8, and the necessary HA settings. Configuration changes to the reserved management interface are not synchronized to other cluster units.

To configure the primary unit reserved management interface to allow HTTPS, SSH, and ICMP access:

1. From a computer on the internal network, connect to the CLI at 10.11.101.100 on port2.

2. Change the port8 IP address and management access:

   config system interface
       edit port8
           set ip 10.11.101.101/24
           set allowaccess https ping ssh
       next
   end

3. Configure the HA settings for the HA reserved management interface by defining a default route to route to the gateway 10.11.101.2:

   config system ha
       set ha-mgmt-status enable
       config ha-mgmt-interfaces
           edit 1
               set interface port8
               set gateway 10.11.101.2
           next
       end
   end

   You can now log into the primary unit's GUI by browsing to https://10.11.101.101. You can also log into the primary unit's CLI by using an SSH client to connect to 10.11.101.101.

To configure secondary unit reserved management interfaces to allow HTTPS, SSH, and ICMP access:

1. From a computer on the internal network, connect to the primary unit's CLI.

2. Connect to the secondary unit with the following command:

   execute ha manage <unit id> <username> <password>

3. Change the port8 IP address and management access:

   config system interface
       edit port8
           set ip 10.11.101.102/24
           set allowaccess https ping ssh
       next
   end

   exit

4. Configure the HA settings for the HA reserved management interface by defining a default route to route to the gateway 10.11.101.2:

   config system ha
       set ha-mgmt-status enable
       config ha-mgmt-interfaces
           edit 1
               set interface port8
               set gateway 10.11.101.2
           next
       end
   end

   You can now log into the secondary unit's GUI by browsing to https://10.11.101.102. You can also log into the secondary unit's CLI by using an SSH client to connect to 10.11.101.102.

SNMP monitoring

The SNMP server can get status information from the cluster members. To use the reserved management interfaces, you must add at least one HA direct management host to an SNMP community. If the SNMP configuration includes SNMP users with user names and passwords, HA direct management must be enabled for the users.

To configure the cluster for SNMP management using the reserved management interfaces in the CLI:

1. Allow SNMP on port8 on both primary and secondary units:

   config system interface
       edit port8
           append allowaccess snmp
       next
   end

2. Add an SNMP community with a host for the reserved management interface of each cluster member. The host includes the IP address of the SNMP server.

   config system snmp community
       edit 1
           set name "Community"
           config hosts
               edit 1
                   set ip 10.11.101.20 255.255.255.255
                   set ha-direct enable
               next
           end
       next
   end

   Enabling ha-direct in a non-HA environment will make SNMP unusable.

3. Add an SNMP user for the reserved management interface:

   config system snmp user
       edit "1"
           set notify-hosts 10.11.101.20
           set ha-direct enable
       next
   end

The SNMP configuration is synchronized to all cluster units.

To get CPU, memory, and network usage information from the SNMP manager for each cluster unit using the reserved management IP addresses:

1. Connect to the SNMP manager CLI.

2. Get resource usage information for the primary unit using the MIB fields:

   snmpget -v2c -c Community 10.11.101.101 fgHaStatsCpuUsage
   snmpget -v2c -c Community 10.11.101.101 fgHaStatsMemUsage
   snmpget -v2c -c Community 10.11.101.101 fgHaStatsNetUsage

3. Get resource usage information for the primary unit using the OIDs:

   snmpget -v2c -c Community 10.11.101.101 1.3.6.1.4.1.12356.101.13.2.1.1.3.1
   snmpget -v2c -c Community 10.11.101.101 1.3.6.1.4.1.12356.101.13.2.1.1.4.1
   snmpget -v2c -c Community 10.11.101.101 1.3.6.1.4.1.12356.101.13.2.1.1.5.1

4. Get resource usage information for the secondary unit using the MIB fields:

   snmpget -v2c -c Community 10.11.101.102 fgHaStatsCpuUsage
   snmpget -v2c -c Community 10.11.101.102 fgHaStatsMemUsage
   snmpget -v2c -c Community 10.11.101.102 fgHaStatsNetUsage

5. Get resource usage information for the primary unit using the OIDs:

   snmpget -v2c -c Community 10.11.101.102 1.3.6.1.4.1.12356.101.13.2.1.1.3.1
   snmpget -v2c -c Community 10.11.101.102 1.3.6.1.4.1.12356.101.13.2.1.1.4.1
   snmpget -v2c -c Community 10.11.101.102 1.3.6.1.4.1.12356.101.13.2.1.1.5.1

Firewall local-in policies for the reserved management interface

Enabling ha-mgmt-intf-only applies the local-in policy only to the VDOM that contains the reserved management interface. The incoming interface is set to match any interface in the VDOM.

To add local-in policies for the reserved management interface:

config firewall local-in-policy
    edit 0
        set ha-mgmt-intf-only enable
        set intf any
        set srcaddr internal-net
        set dstaddr mgmt-int
        set action accept
        set service HTTPS
        set schedule weekdays
    next
end

NTP over reserved management interfaces

When NTP is enabled in an HA cluster, the primary unit will always be the unit to contact the NTP server and synchronize system time to the secondary units over the HA heartbeat interface. However, in the event that the primary should contact the NTP server over the HA reserved management interface, then the ha-direct option should be enabled under the config system ha settings.

config system interface
    edit port5
        set ip 172.16.79.46 255.255.255.0
    next
end

config system ha
    set group-name FGT-HA
    set mode a-p
    set ha-mgmt-status enable
    config ha-mgmt-interfaces
        edit 1
            set interface port5
            set gateway 172.16.79.1
        next
    end
    set ha-direct enable
end

config system ntp
    set ntpsync enable
    set syncinterval 5
end