Out-of-band management with reserved management interfaces

As part of an HA configuration, you can reserve up to four management interfaces to provide direct management access to all cluster units. For each reserved management interface, you can configure a different IP address, administrative access, and other interface settings, for each cluster unit. By connecting these interfaces to your network, you can separately manage each cluster unit from different IP addresses.

• Reserved management interfaces provide direct management access to each cluster unit, and give each cluster unit a different identity on your network. This simplifies using external services, such as SNMP, to monitor and managed separate cluster units.

• Reserved management interfaces are not assigned HA virtual MAC addresses. They retain the permanent hardware address of the physical interface, unless you manually change it using the config system interface command.

• Reserved management interfaces and their IP addresses should not be used for managing a cluster using FortiManager. To manage a FortiGate HA cluster with FortiManager, use the IP address of one of the cluster unit interfaces.

• Configuration changes to a reserved management interface are not synchronized to other cluster units. Other configuration changes are automatically synchronized to all cluster units.

You can configure an in-band management interface for a cluster unit. See In-band management for information. In-band management does not reserve the interface exclusively for HA management.

Management interface

Enable HTTPS or HTTP administrative access on the reserved management interfaces to connect to the GUI of each cluster unit. On secondary units, the GUI has the same features as the primary unit, except for unit specific information, for example:

• The System Information widget on the Status dashboard shows the secondary units serial number.

• In the cluster members list at System > HA, you can change the HA configuration of the unit that you are logged into. You can only change the host name and device priority of the primary and other secondary units.

• The system events logs shows logs for the device that you are logged into. Use the HA device drop down to view the log messages for other cluster units, including the primary unit.

Enable SSH administrative access on the reserved management interfaces to connect to the CLI of each cluster unit. The CLI prompt includes the host of the cluster unit that you are connected to. Use the execute ha manage command to connect to other cluster unit CLIs.

Enable SNMP administrative access on a reserved management interface to use SNMP to monitor each cluster unit using the interface's IP address. Direct management of cluster members must also be enabled, see Configuring SNMP remote management of individual cluster units example.

Reserved management interfaces are available in both NAT and transparent mode, and when the cluster is operating with multiple VDOMs.

FortiCloud, FortiSandbox, and other management services

By default, management services such as FortiCloud, FortiSandbox, SNMP, remote logging, and remote authentication, use a cluster interface. This means that communication from each cluster unit will come from a cluster interface, and not from the individual cluster unit's interface.

You can configure HA reserved management interfaces to be used for communication with management services by enabling the ha-direct option. This separates management traffic for each cluster unit, and allows each unit to be individually managed. This is especially useful when cluster unit are in different physical locations.

The following management features will then use the HA reserved management interface:

• Remote logging, including syslog, FortiAnalyzer, and FortiCloud

• SNMP queries and traps

• Remote authentication and certificate verification

• Communication with FortiSandbox

• Netflow and sflow, see Routing NetFlow data over the HA management interface for information.

The HA reserved management interfaces can also be configured for only SNMP remote management, see Configuring SNMP remote management of individual cluster units example.

To configure HA reserved management interfaces for communication with management services:

config system ha
    set ha-direct enable
end

Enabling ha-direct in a non-HA environment will make SNMP unusable.

Configuring SNMP remote management of individual cluster units example

In this example, two FortiGate units are already operating in a cluster. On each unit, port8 is connected to the internal network through a switch and configured as a reserved management interface with SNMP remote management.

Configuration changes to the reserved management interface are not synchronized to other cluster units.

To configure management interface reservation in the GUI:

1. Go to System > HA and edit the primary unit.

2. Enable Management Interface Reservation.

3. Set Interface to port8. This interface must not be referenced anywhere else.

4. Set Gateway to 10.11.101.2. The gateway is not synchronized to secondary units.

   

5. Optionally, enter a Destination subnet to indicate the destinations that should use the defined gateway. By default, 0.0.0.0/0 is used.

6. Click OK.

To configure management interface reservation in the CLI:

config system ha
    set ha-mgmt-status enable
    config ha-mgmt-interfaces
        edit 1
            set interface "port8"
            set gateway 10.11.101.2
        next
    end
end

The reserved management interface default route is not synchronized to other cluster units.

GUI access

To configure the primary unit's reserved management interface, configure an IP address and management access on port8. Then, to configure the secondary unit's reserved management interface, access the unit's CLI through the primary unit, and configure an IP address and management access on port8. Configuration changes to the reserved management interface are not synchronized to other cluster units.

To configure the primary unit reserved management interface to allow GUI access in the CLI:

1. From a computer on the internal network, connect to the CLI at 10.11.101.100.

2. Change the port8 IP address and management access:

   config system interface
       edit port8
           set ip 10.11.101.101/24
           set allowaccess https ping ssh snmp
       next
   end

   You can now log into the primary unit's GUI by browsing to https://10.11.101.101. You can also log into the primary unit's CLI by using an SSH client to connect to 10.11.101.101.

To configure secondary unit reserved management interfaces to allow GUI access:

1. From a computer on the internal network, connect to the primary unit's CLI.

2. Connect to the secondary unit with the following command:

   execute ha manage <unit id> <username> <password>

3. Change the port8 IP address and management access:

   config system interface
       edit port8
           set ip 10.11.101.102/24
           set allowaccess https ping ssh snmp
       next
   end

   exit

   You can now log into the secondary unit's GUI by browsing to https://10.11.101.102. You can also log into the secondary unit's CLI by using an SSH client to connect to 10.11.101.102.

SNMP management

The SNMP server can get status information from the cluster members. To use the reserved management interfaces, you must add at least one HA direct management host to an SNMP community. If the SNMP configuration includes SNMP users with user names and passwords, HA direct management must be enabled for the users.

To configure the cluster for SNMP management using the reserved management interfaces in the CLI:

1. Add an SNMP community with a host for the reserved management interface of each cluster member. The host includes the IP address of the SNMP server.

   config system snmp community
       edit 1
           set name "Community"
           config hosts
               edit 1
                   set ip 10.11.101.20 255.255.255.255
                   set ha-direct enable
               next
           end
       next
   end

   Enabling ha-direct in a non-HA environment will make SNMP unusable.

2. Add an SNMP user for the reserved management interface

   config system snmp user
       edit "1"
           set notify-hosts 10.11.101.20
           set ha-direct enable
       next
   end

The SNMP configuration is synchronized to all cluster units.

To get CPU, memory, and network usage information from the SNMP manager for each cluster unit using the reserved management IP addresses:

1. Connect to the SNMP manager CLI.

2. Get resource usage information for the primary unit using the MIB fields:

   snmpget -v2c -c Community 10.11.101.101 fgHaStatsCpuUsage
   snmpget -v2c -c Community 10.11.101.101 fgHaStatsMemUsage
   snmpget -v2c -c Community 10.11.101.101 fgHaStatsNetUsage

3. Get resource usage information for the primary unit using the OIDs:

   snmpget -v2c -c Community 10.11.101.101 1.3.6.1.4.1.12356.101.13.2.1.1.3.1
   snmpget -v2c -c Community 10.11.101.101 1.3.6.1.4.1.12356.101.13.2.1.1.4.1
   snmpget -v2c -c Community 10.11.101.101 1.3.6.1.4.1.12356.101.13.2.1.1.5.1

4. Get resource usage information for the secondary unit using the MIB fields:

   snmpget -v2c -c Community 10.11.101.102 fgHaStatsCpuUsage
   snmpget -v2c -c Community 10.11.101.102 fgHaStatsMemUsage
   snmpget -v2c -c Community 10.11.101.102 fgHaStatsNetUsage

5. Get resource usage information for the primary unit using the OIDs:

   snmpget -v2c -c Community 10.11.101.102 1.3.6.1.4.1.12356.101.13.2.1.1.3.1
   snmpget -v2c -c Community 10.11.101.102 1.3.6.1.4.1.12356.101.13.2.1.1.4.1
   snmpget -v2c -c Community 10.11.101.102 1.3.6.1.4.1.12356.101.13.2.1.1.5.1

Firewall local-in policies for the reserved management interface

Enabling ha-mgmt-intf-only applies the local-in policy only to the VDOM that contains the reserved management interface. The incoming interface is set to match any interface in the VDOM..

To add local-in policies for the reserved management interface:

config firewall local-in-policy
    edit 0
        set ha-mgmt-intf-only enable
        set intf any
        set srcaddr internal-net
        set dstaddr mgmt-int
        set action accept
        set service HTTPS
        set schedule weekdays
    next
end

NTP over reserved management interfaces

If reserved management interfaces are configured for each cluster member, and NTP is enabled, then the primary unit will contact the NTP server using the reserved management interface. The system time is then synchronized to the secondary units over the HA heartbeat interface.

config system interface
    edit port5
        set ip 172.16.79.46 255.255.255.0
    next
end

config system ha
    set group-name FGT-HA
    set mode a-p
    set ha-mgmt-status enable
    config ha-mgmt-interfaces
        edit 1
            set interface port5
            set gateway 172.16.79.1
        next
    end
    set ha-direct enable
end

config system ntp
    set ntpsync enable
    set syncinterval 5
end