Fortinet black logo

Setting up FortiGate-VM HA for a Microsoft Hyper-V Live Migration environment

Copy Link
Copy Doc ID d8128d8d-a933-11ec-9fd1-fa163e15d75b:397930
Download PDF

Setting up FortiGate-VM HA for a Microsoft Hyper-V Live Migration environment

This guide provides sample configuration of Live Migration FortiGate-VM HA in a Microsoft Hyper-V environment. This feature enables the live migration of a running FortiGate-VM from one physical server to another with zero downtime, continuous service availability, and complete transaction integrity. It also provides transparency to users.

In VM environments that do not support broadcast communication, you can set up a unicast HA heartbeat when configuring HA. Setting up a unicast HA heartbeat consists of enabling the feature and adding a peer IP address. The peer IP address is the IP address of the HA heartbeat interface of the other FortiGate-VM in the HA cluster.

This configuration consists of the following components:

  • Two Windows Server 2019 machines, each with four network adapters installed and with Hyper-V role. This guide assumes that you have installed and set up these machines as per Microsoft documentation, and that they have joined a domain. In this example, the machines are HV2019S02 and HV2019S03 and have joined domain example.org.
  • Two FortiGate-VM64-HVs deployed on HV2019S02 with FortiOS 6.4.3
To set up FortiGate-VM HA for a Microsoft Hyper-V Live Migration environment:
  1. Set up hosts for live migration without failover clustering. See Set up hosts for live migration without Failover Clustering.
  2. Use live migration without failover clustering to move a virtual machine. See Use live migration without Failover Clustering to move a virtual machine.
  3. In Active Directory Users and Computers, configure the following for HV2019S02 and HV2019S03. See Live Migration via Constrained Delegation with Kerberos in Windows Server 2016 for details:
    1. Configure constrained delegation.
    2. On the Delegation tab, add cifs and Microsoft Virtual System Migration Service.

  4. Configure HV2019S02 settings:

    PS C:\Users\exampleuser> Add-VMMigrationNetwork 192.168.255.0/24 PS C:\Users\exampleuser> Get-VMMigrationNetwork | fl * Subnet : 192.168.255.0/24 Priority : 0 CimSession : CimSession: . ComputerName : HV2019S02 IsDeleted : False PS C:\Users\exampleuser> PS C:\Users\exampleuser> Get-VMHost | fl * ComputerName : HV2019S02 LogicalProcessorCount : 24 ResourceMeteringSaveInterval : 01:00:00 HostNumaStatus : {HV2019S02} NumaStatus : {ip-172-18-70-169} IovSupport : True IovSupportReasons : InternalNetworkAdapters : {MGMT, VSW-port3, VSW-port2, VSW-port4} ExternalNetworkAdapters : {SRIOV-X710-p1_External, MGMT_External, VSW-port1_External, SRIOV-X710-p2_External} SupportedVmVersions : {5.0, 6.2, 7.0, 7.1…} SecureBootTemplates : {MicrosoftWindows, MicrosoftUEFICertificateAuthority, OpenSourceShieldedVM} EnableEnhancedSessionMode : False FibreChannelWwnn : C003FF0000FFFF00 FibreChannelWwpnMaximum : C003FF105350FFFF FibreChannelWwpnMinimum : C003FF1053500000 MacAddressMaximum : 00155DD775FF MacAddressMinimum : 00155DD77500 NumaSpanningEnabled : True VirtualHardDiskPath : E:\vms\ VirtualMachinePath : E:\vms\ FullyQualifiedDomainName : example.org MemoryCapacity : 68185321472 Name : HV2019S02 MaximumStorageMigrations : 2 MaximumVirtualMachineMigrations : 2 UseAnyNetworkForMigration : False VirtualMachineMigrationAuthenticationType : Kerberos VirtualMachineMigrationEnabled : True VirtualMachineMigrationPerformanceOption : SMB CimSession : CimSession: . IsDeleted : False PS C:\Users\exampleuser>

  5. Configure HV2019S03 settings:

    PS C:\Users\exampleuser> Add-VMMigrationNetwork 192.168.255.0/24 PS C:\Users\exampleuser> Get-VMMigrationNetwork | fl * Subnet : 192.168.255.0/24 Priority : 0 CimSession : CimSession: . ComputerName : HV2019S03 IsDeleted : False PS C:\Users\exampleuser> PS C:\Users\exampleuser> Get-VMHost | fl * ComputerName : HV2019S03 LogicalProcessorCount : 24 ResourceMeteringSaveInterval : 01:00:00 HostNumaStatus : {HV2019S03} NumaStatus : {ip-172-18-70-170, FGT_VM64_HV_b1723_B, FGT_VM64_HV_b1723_A} IovSupport : True IovSupportReasons : InternalNetworkAdapters : {VSW-port4, VSW-port2, VSW-port3, MGMT} ExternalNetworkAdapters : {VSW-port1_External, SRIOV-X710-p2_External, MGMT_External, SRIOV-X710-p1_External} SupportedVmVersions : {5.0, 6.2, 7.0, 7.1…} SecureBootTemplates : {MicrosoftWindows, MicrosoftUEFICertificateAuthority, OpenSourceShieldedVM} EnableEnhancedSessionMode : False FibreChannelWwnn : C003FF0000FFFF00 FibreChannelWwpnMaximum : C003FF06263EFFFF FibreChannelWwpnMinimum : C003FF06263E0000 MacAddressMaximum : 00155D8B30FF MacAddressMinimum : 00155D8B3000 NumaSpanningEnabled : True VirtualHardDiskPath : E:\vms\ VirtualMachinePath : E:\vms\ FullyQualifiedDomainName : example.org MemoryCapacity : 68185321472 Name : HV2019S03 MaximumStorageMigrations : 2 MaximumVirtualMachineMigrations : 2 UseAnyNetworkForMigration : False VirtualMachineMigrationAuthenticationType : Kerberos VirtualMachineMigrationEnabled : True VirtualMachineMigrationPerformanceOption : SMB CimSession : CimSession: . IsDeleted : False PS C:\Users\exampleuser>

  6. There are several virtual switches created on each Microsoft Hyper-V server for FortiGate-VMs to connect to physical networks and those VMs on protected networks:

    Switch

    Connection

    VSW-port1

    External network to Internet.

    MGMT

    External network to management network.

    SRIOV-X710-p1

    External network to a closed network.

    SRIOV-X710-p2

    External network to protected networks.

    Ensure that each FortiGate-VM's interfaces are connected to the virtual switches per the following and that you have enabled MAC address spoofing to all interfaces:

    Port

    Switch

    port1

    VSW-port1

    port2

    MGMT

    port3

    SRIOV-X710-p1

    port4

    SRIOV-X710-p2

  7. Configure the FortiGate-VMs for high availability (HA). For details, see High availability:
    1. Configure FortiGate A:

      config router static edit 100 set gateway 172.31.250.1 set device port1 next end config system interface edit "port1" set vdom "root" set mode static set ip 172.31.250.11/24 set allowaccess ping set alias "to_Internet" next end config system interface edit "port2" set vdom "root" set mode static set ip 172.18.70.181/24 set allowaccess ping https ssh set alias "ha-mgmt" next end config system interface edit "port3" set vdom "root" set mode static set ip 192.168.30.11/24 set allowaccess ping set alias "HA-Sync" next end config system interface edit "port4" set vdom "root" set ip 192.168.40.1 255.255.255.0 set allowaccess ping ssh https set type physical set snmp-index 4 config ipv6 set ip6-address 2001:db8:c0a8:2800::1/64 set ip6-allowaccess ping ssh https set ip6-send-adv enable set ip6-manage-flag enable set ip6-other-flag enable config ip6-prefix-list edit 2001:db8:c0a8:2800::/64 set valid-life-time 600 set preferred-life-time 600 next end end next end config system ha set group-name "FGVM-HA-DEMO" set mode a-p set hbdev "port3" 100 set session-pickup enable set session-pickup-connectionless enable set ha-mgmt-status enable config ha-mgmt-interfaces edit 1 set interface "port2" set gateway 172.18.70.1 next end set override disable set ha-direct enable end

    2. Configure FortiGate B:

      config system ha set group-name "FGVM-HA-DEMO" set mode a-p set hbdev "port3" 100 set session-pickup enable set session-pickup-connectionless enable set ha-mgmt-status enable config ha-mgmt-interfaces edit 1 set interface "port2" set gateway 172.18.70.1 next end set override disable set ha-direct enable end

    3. Verify HA status:

      FGT_VM64_HV_A # get system ha status HA Health Status: OK Model: FortiGate-VM64-HV Mode: HA A-P Group: 0 Debug: 0 Cluster Uptime: 0 days 0:20:0 Cluster state change time: 2020-09-25 13:32:14 Primary selected using: <2020/09/25 13:32:14> FGVM08TM20004598 is selected as the primary because it has the largest value of uptime. <2020/09/25 13:31:37> FGVM08TM20004598 is selected as the primary because it's the only member in the cluster. <2020/09/25 13:31:28> FGVM08TM20004598 is selected as the primary because the peer member FGVM08TM20003583 has SET_AS_SECONDARY flag set. <2020/09/25 13:27:43> FGVM08TM20003583 is selected as the primary because it has the largest value of uptime. ses_pickup: enable, ses_pickup_delay=disable override: disable Configuration Status: FGVM08TM20004598(updated 3 seconds ago): in-sync FGVM08TM20003583(updated 0 seconds ago): in-sync System Usage stats: FGVM08TM20004598(updated 3 seconds ago): sessions=6, average-cpu-user/nice/system/idle=0%/0%/0%/100%, memory=16% FGVM08TM20003583(updated 0 seconds ago): sessions=1, average-cpu-user/nice/system/idle=0%/0%/0%/100%, memory=16% HBDEV stats: FGVM08TM20004598(updated 3 seconds ago): port3: physical/10000full, up, rx-bytes/packets/dropped/errors=3863085/10517/0/0, tx=4211732/11429/0/0 FGVM08TM20003583(updated 0 seconds ago): port3: physical/10000full, up, rx-bytes/packets/dropped/errors=2583600/6499/0/0, tx=2054873/6169/0/0 Primary : FGT_VM64_HV_A , FGVM08TM20004598, HA cluster index = 0 Secondary : FGT_VM64_HV_B , FGVM08TM20003583, HA cluster index = 1 number of vcluster: 1 vcluster 1: work 169.254.0.1 Primary: FGVM08TM20004598, HA operating index = 0 Secondary: FGVM08TM20003583, HA operating index = 1 FGT_VM64_HV_A #

  8. Migrate the FortiGate-VMs from HV2019S02 to HV2019S03:
    1. On HV2019S02, in Hyper-V Manager, right-click FGT_VM64_HV_A and select Move.
    2. Select Move the virtual machine, then click Next.
    3. Browse and select HV2019S03 for the destination computer, then click Next.
    4. Select Move all of the virtual machines data to a single location, then click Next.
    5. For the destination location, enter D:\vms\fgt_vm64_hv_b1764_a\, then click Next.
    6. Verify the summary, then click Finish.
    7. Repeat steps a-f to move FGT_VM64_HA_B.

      Both FortiGate-VMs move to HV2019S03 and continue running.

Setting up FortiGate-VM HA for a Microsoft Hyper-V Live Migration environment

This guide provides sample configuration of Live Migration FortiGate-VM HA in a Microsoft Hyper-V environment. This feature enables the live migration of a running FortiGate-VM from one physical server to another with zero downtime, continuous service availability, and complete transaction integrity. It also provides transparency to users.

In VM environments that do not support broadcast communication, you can set up a unicast HA heartbeat when configuring HA. Setting up a unicast HA heartbeat consists of enabling the feature and adding a peer IP address. The peer IP address is the IP address of the HA heartbeat interface of the other FortiGate-VM in the HA cluster.

This configuration consists of the following components:

  • Two Windows Server 2019 machines, each with four network adapters installed and with Hyper-V role. This guide assumes that you have installed and set up these machines as per Microsoft documentation, and that they have joined a domain. In this example, the machines are HV2019S02 and HV2019S03 and have joined domain example.org.
  • Two FortiGate-VM64-HVs deployed on HV2019S02 with FortiOS 6.4.3
To set up FortiGate-VM HA for a Microsoft Hyper-V Live Migration environment:
  1. Set up hosts for live migration without failover clustering. See Set up hosts for live migration without Failover Clustering.
  2. Use live migration without failover clustering to move a virtual machine. See Use live migration without Failover Clustering to move a virtual machine.
  3. In Active Directory Users and Computers, configure the following for HV2019S02 and HV2019S03. See Live Migration via Constrained Delegation with Kerberos in Windows Server 2016 for details:
    1. Configure constrained delegation.
    2. On the Delegation tab, add cifs and Microsoft Virtual System Migration Service.

  4. Configure HV2019S02 settings:

    PS C:\Users\exampleuser> Add-VMMigrationNetwork 192.168.255.0/24 PS C:\Users\exampleuser> Get-VMMigrationNetwork | fl * Subnet : 192.168.255.0/24 Priority : 0 CimSession : CimSession: . ComputerName : HV2019S02 IsDeleted : False PS C:\Users\exampleuser> PS C:\Users\exampleuser> Get-VMHost | fl * ComputerName : HV2019S02 LogicalProcessorCount : 24 ResourceMeteringSaveInterval : 01:00:00 HostNumaStatus : {HV2019S02} NumaStatus : {ip-172-18-70-169} IovSupport : True IovSupportReasons : InternalNetworkAdapters : {MGMT, VSW-port3, VSW-port2, VSW-port4} ExternalNetworkAdapters : {SRIOV-X710-p1_External, MGMT_External, VSW-port1_External, SRIOV-X710-p2_External} SupportedVmVersions : {5.0, 6.2, 7.0, 7.1…} SecureBootTemplates : {MicrosoftWindows, MicrosoftUEFICertificateAuthority, OpenSourceShieldedVM} EnableEnhancedSessionMode : False FibreChannelWwnn : C003FF0000FFFF00 FibreChannelWwpnMaximum : C003FF105350FFFF FibreChannelWwpnMinimum : C003FF1053500000 MacAddressMaximum : 00155DD775FF MacAddressMinimum : 00155DD77500 NumaSpanningEnabled : True VirtualHardDiskPath : E:\vms\ VirtualMachinePath : E:\vms\ FullyQualifiedDomainName : example.org MemoryCapacity : 68185321472 Name : HV2019S02 MaximumStorageMigrations : 2 MaximumVirtualMachineMigrations : 2 UseAnyNetworkForMigration : False VirtualMachineMigrationAuthenticationType : Kerberos VirtualMachineMigrationEnabled : True VirtualMachineMigrationPerformanceOption : SMB CimSession : CimSession: . IsDeleted : False PS C:\Users\exampleuser>

  5. Configure HV2019S03 settings:

    PS C:\Users\exampleuser> Add-VMMigrationNetwork 192.168.255.0/24 PS C:\Users\exampleuser> Get-VMMigrationNetwork | fl * Subnet : 192.168.255.0/24 Priority : 0 CimSession : CimSession: . ComputerName : HV2019S03 IsDeleted : False PS C:\Users\exampleuser> PS C:\Users\exampleuser> Get-VMHost | fl * ComputerName : HV2019S03 LogicalProcessorCount : 24 ResourceMeteringSaveInterval : 01:00:00 HostNumaStatus : {HV2019S03} NumaStatus : {ip-172-18-70-170, FGT_VM64_HV_b1723_B, FGT_VM64_HV_b1723_A} IovSupport : True IovSupportReasons : InternalNetworkAdapters : {VSW-port4, VSW-port2, VSW-port3, MGMT} ExternalNetworkAdapters : {VSW-port1_External, SRIOV-X710-p2_External, MGMT_External, SRIOV-X710-p1_External} SupportedVmVersions : {5.0, 6.2, 7.0, 7.1…} SecureBootTemplates : {MicrosoftWindows, MicrosoftUEFICertificateAuthority, OpenSourceShieldedVM} EnableEnhancedSessionMode : False FibreChannelWwnn : C003FF0000FFFF00 FibreChannelWwpnMaximum : C003FF06263EFFFF FibreChannelWwpnMinimum : C003FF06263E0000 MacAddressMaximum : 00155D8B30FF MacAddressMinimum : 00155D8B3000 NumaSpanningEnabled : True VirtualHardDiskPath : E:\vms\ VirtualMachinePath : E:\vms\ FullyQualifiedDomainName : example.org MemoryCapacity : 68185321472 Name : HV2019S03 MaximumStorageMigrations : 2 MaximumVirtualMachineMigrations : 2 UseAnyNetworkForMigration : False VirtualMachineMigrationAuthenticationType : Kerberos VirtualMachineMigrationEnabled : True VirtualMachineMigrationPerformanceOption : SMB CimSession : CimSession: . IsDeleted : False PS C:\Users\exampleuser>

  6. There are several virtual switches created on each Microsoft Hyper-V server for FortiGate-VMs to connect to physical networks and those VMs on protected networks:

    Switch

    Connection

    VSW-port1

    External network to Internet.

    MGMT

    External network to management network.

    SRIOV-X710-p1

    External network to a closed network.

    SRIOV-X710-p2

    External network to protected networks.

    Ensure that each FortiGate-VM's interfaces are connected to the virtual switches per the following and that you have enabled MAC address spoofing to all interfaces:

    Port

    Switch

    port1

    VSW-port1

    port2

    MGMT

    port3

    SRIOV-X710-p1

    port4

    SRIOV-X710-p2

  7. Configure the FortiGate-VMs for high availability (HA). For details, see High availability:
    1. Configure FortiGate A:

      config router static edit 100 set gateway 172.31.250.1 set device port1 next end config system interface edit "port1" set vdom "root" set mode static set ip 172.31.250.11/24 set allowaccess ping set alias "to_Internet" next end config system interface edit "port2" set vdom "root" set mode static set ip 172.18.70.181/24 set allowaccess ping https ssh set alias "ha-mgmt" next end config system interface edit "port3" set vdom "root" set mode static set ip 192.168.30.11/24 set allowaccess ping set alias "HA-Sync" next end config system interface edit "port4" set vdom "root" set ip 192.168.40.1 255.255.255.0 set allowaccess ping ssh https set type physical set snmp-index 4 config ipv6 set ip6-address 2001:db8:c0a8:2800::1/64 set ip6-allowaccess ping ssh https set ip6-send-adv enable set ip6-manage-flag enable set ip6-other-flag enable config ip6-prefix-list edit 2001:db8:c0a8:2800::/64 set valid-life-time 600 set preferred-life-time 600 next end end next end config system ha set group-name "FGVM-HA-DEMO" set mode a-p set hbdev "port3" 100 set session-pickup enable set session-pickup-connectionless enable set ha-mgmt-status enable config ha-mgmt-interfaces edit 1 set interface "port2" set gateway 172.18.70.1 next end set override disable set ha-direct enable end

    2. Configure FortiGate B:

      config system ha set group-name "FGVM-HA-DEMO" set mode a-p set hbdev "port3" 100 set session-pickup enable set session-pickup-connectionless enable set ha-mgmt-status enable config ha-mgmt-interfaces edit 1 set interface "port2" set gateway 172.18.70.1 next end set override disable set ha-direct enable end

    3. Verify HA status:

      FGT_VM64_HV_A # get system ha status HA Health Status: OK Model: FortiGate-VM64-HV Mode: HA A-P Group: 0 Debug: 0 Cluster Uptime: 0 days 0:20:0 Cluster state change time: 2020-09-25 13:32:14 Primary selected using: <2020/09/25 13:32:14> FGVM08TM20004598 is selected as the primary because it has the largest value of uptime. <2020/09/25 13:31:37> FGVM08TM20004598 is selected as the primary because it's the only member in the cluster. <2020/09/25 13:31:28> FGVM08TM20004598 is selected as the primary because the peer member FGVM08TM20003583 has SET_AS_SECONDARY flag set. <2020/09/25 13:27:43> FGVM08TM20003583 is selected as the primary because it has the largest value of uptime. ses_pickup: enable, ses_pickup_delay=disable override: disable Configuration Status: FGVM08TM20004598(updated 3 seconds ago): in-sync FGVM08TM20003583(updated 0 seconds ago): in-sync System Usage stats: FGVM08TM20004598(updated 3 seconds ago): sessions=6, average-cpu-user/nice/system/idle=0%/0%/0%/100%, memory=16% FGVM08TM20003583(updated 0 seconds ago): sessions=1, average-cpu-user/nice/system/idle=0%/0%/0%/100%, memory=16% HBDEV stats: FGVM08TM20004598(updated 3 seconds ago): port3: physical/10000full, up, rx-bytes/packets/dropped/errors=3863085/10517/0/0, tx=4211732/11429/0/0 FGVM08TM20003583(updated 0 seconds ago): port3: physical/10000full, up, rx-bytes/packets/dropped/errors=2583600/6499/0/0, tx=2054873/6169/0/0 Primary : FGT_VM64_HV_A , FGVM08TM20004598, HA cluster index = 0 Secondary : FGT_VM64_HV_B , FGVM08TM20003583, HA cluster index = 1 number of vcluster: 1 vcluster 1: work 169.254.0.1 Primary: FGVM08TM20004598, HA operating index = 0 Secondary: FGVM08TM20003583, HA operating index = 1 FGT_VM64_HV_A #

  8. Migrate the FortiGate-VMs from HV2019S02 to HV2019S03:
    1. On HV2019S02, in Hyper-V Manager, right-click FGT_VM64_HV_A and select Move.
    2. Select Move the virtual machine, then click Next.
    3. Browse and select HV2019S03 for the destination computer, then click Next.
    4. Select Move all of the virtual machines data to a single location, then click Next.
    5. For the destination location, enter D:\vms\fgt_vm64_hv_b1764_a\, then click Next.
    6. Verify the summary, then click Finish.
    7. Repeat steps a-f to move FGT_VM64_HA_B.

      Both FortiGate-VMs move to HV2019S03 and continue running.