Fortinet Document Library

Version:


Table of Contents

KVM Cookbook

Resources

Upgrade Path Tool
6.2.0
Download PDF
Copy Link

Deploying the FortiGate-VM on KVM

Deploying a FortiGate-VM on KVM with QAT support consists of the following steps:

  1. Create the FortiGate-VM on KVM.
  2. Inject SR-IOV network VFs into the FortiGate-VM.
  3. Configure interrupt affinities.
To create the FortiGate-VM on KVM:

To create a FortiGate-VM on KVM, refer to Deploying the FortiGate-VM.

To inject SR-IOV network VFs into the FortiGate-VM:

You can inject an SR-IOV network VF into a Linux KVM VM using one of the following ways:

  • Connecting an SR-IOV VF to a KVM VM by directly importing the VF as a PCI device using the PCI bus information that the host OS assigned to it when it was created
  • Using the Virtual Manager GUI
  • Adding an SR-IOV network adapter to the KVM VM as a VF network adapter connected to a macvtap on the host
  • Creating an SR-IOV VF network adapter using a KVM virtual network pool of adapters

See Configure SR-IOV Network Virtual Functions in Linux* KVM*.

In the following example, virtual network adapter pools were created for KVM04:

[root@localhost ~]# vnlist
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 default              active     yes           yes
 p5p1-pool            active     no            no
 p5p2-pool            active     no            no
 p7p1-pool            active     no            no
 p7p2-pool            active     no            no

[root@localhost ~]# vf2pf


Virtual Functions on Intel Corporation Ethernet Controller X710 for 10GbE SFP+. (p5p1):
PCI BDF         Interface
=======         =========
0000:86:02.0    p5p1_0
0000:86:02.1    p5p1_1
0000:86:02.2    p5p1_2
0000:86:02.3    p5p1_3
0000:86:02.4    p5p1_4
0000:86:02.5    p5p1_5
0000:86:02.6    p5p1_6
0000:86:02.7    p5p1_7

Virtual Functions on Intel Corporation Ethernet Controller X710 for 10GbE SFP+. (p5p2):
PCI BDF         Interface
=======         =========
0000:86:0a.0    p5p2_0
0000:86:0a.1    p5p2_1
0000:86:0a.2    p5p2_2
0000:86:0a.3    p5p2_3
0000:86:0a.4    p5p2_4
0000:86:0a.5    p5p2_5
0000:86:0a.6    p5p2_6
0000:86:0a.7    p5p2_7

Virtual Functions on Intel Corporation Ethernet Controller X710 for 10GbE SFP+. (p7p1):
PCI BDF         Interface
=======         =========
0000:88:02.0    p7p1_0
0000:88:02.1    p7p1_1
0000:88:02.2    p7p1_2
0000:88:02.3    p7p1_3
0000:88:02.4    p7p1_4
0000:88:02.5    p7p1_5
0000:88:02.6    p7p1_6
0000:88:02.7    p7p1_7

Virtual Functions on Intel Corporation Ethernet Controller X710 for 10GbE SFP+. (p7p2):
PCI BDF         Interface
=======         =========
0000:88:0a.0    p7p2_0
0000:88:0a.1    p7p2_1
0000:88:0a.2    p7p2_2
0000:88:0a.3    p7p2_3
0000:88:0a.4    p7p2_4
0000:88:0a.5    p7p2_5
0000:88:0a.6    p7p2_6
0000:88:0a.7    p7p2_7

The XML file is as follows. <cputune> locks the virtual CPUs to the same NUMA node, while <hostdev mode='subsystem' type='pci' managed='yes'> creates the QAT VFs:

[root@localhost ~]# virsh dumpxml vm04numa1
<domain type='kvm'>
  <name>vm04numa1</name>
  <uuid>fc5e1cec-8b4e-4bb8-9f89-e86f1abfffeb</uuid>
  <memory unit='KiB'>6291456</memory>
  <currentMemory unit='KiB'>6291456</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='1048576' unit='KiB'/>
    </hugepages>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='17'/>
    <vcpupin vcpu='1' cpuset='19'/>
    <vcpupin vcpu='2' cpuset='21'/>
    <vcpupin vcpu='3' cpuset='23'/>
    <emulatorpin cpuset='17,19,21,23'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='1'/>
  </numatune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.6.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='custom' match='exact' check='partial'>
    <model fallback='allow'>Skylake-Server-IBRS</model>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/vm04numa1.0984'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <interface type='direct'>
      <mac address='52:54:00:7c:07:50'/>
      <source dev='em1' mode='bridge'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:7c:07:53'/>
      <source network='p5p1-pool'/>
      <model type='i40e'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:7c:07:54'/>
      <source network='p5p2-pool'/>
      <model type='i40e'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:7c:07:55'/>
      <source network='p7p1-pool'/>
      <model type='i40e'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:7c:07:56'/>
      <source network='p7p2-pool'/>
      <model type='i40e'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0d' function='0x0'/>
    </interface>
    <serial type='tcp'>
      <source mode='bind' host='0.0.0.0' service='10004'/>
      <protocol type='telnet'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='tcp'>
      <source mode='bind' host='0.0.0.0' service='10004'/>
      <protocol type='telnet'/>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='5904' autoport='no' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>                                        
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0xb1' slot='0x01' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0e' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0xb1' slot='0x01' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0f' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0xb1' slot='0x01' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x10' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0xb1' slot='0x01' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x11' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
</domain>

The following shows the Virtual Manager GUI:

To configure interrupt affinities:

The example topology is as follows:

TestCenter----KVM port2----KVM port3===IPSEC tunnel===KVM port5-----KVM port4----TestCenter

After configuring the IPsec tunnel in the KVM, you must configure interrupt affinities and CPU masking to improve throughput for FortiGate-VM platforms. In this example, you can configure interrupt affinities as follows. You must manually set cpu-affinity for each QAT VF; otherwise, they only go through the first CPU.

config system affinity-interrupt
    edit 1
        set interrupt "i40evf-port2-TxRx-0"
        set affinity-cpumask "0x01"
    next
    edit 2
        set interrupt "i40evf-port2-TxRx-1"
        set affinity-cpumask "0x01"
    next
    edit 3
        set interrupt "i40evf-port2-TxRx-2"
        set affinity-cpumask "0x01"
    next
    edit 4
        set interrupt "i40evf-port2-TxRx-3"
        set affinity-cpumask "0x01"
    next
    edit 5
        set interrupt "i40evf-port3-TxRx-0"
        set affinity-cpumask "0x02"
    next
    edit 6
        set interrupt "i40evf-port3-TxRx-1"
        set affinity-cpumask "0x02"
    next
    edit 7
        set interrupt "i40evf-port3-TxRx-2"
        set affinity-cpumask "0x02"
    next
    edit 8
        set interrupt "i40evf-port3-TxRx-3"
        set affinity-cpumask "0x02"
    next
    edit 9
        set interrupt "i40evf-port4-TxRx-0"
        set affinity-cpumask "0x04"
    next
    edit 10
        set interrupt "i40evf-port4-TxRx-1"
        set affinity-cpumask "0x04"
    next
    edit 11
        set interrupt "i40evf-port4-TxRx-2"
        set affinity-cpumask "0x04"
    next
    edit 12
        set interrupt "i40evf-port4-TxRx-3"
        set affinity-cpumask "0x04"
    next
    edit 13
        set interrupt "i40evf-port5-TxRx-0"
        set affinity-cpumask "0x08"
    next
    edit 14
        set interrupt "i40evf-port5-TxRx-1"
        set affinity-cpumask "0x08"
    next
    edit 15
        set interrupt "i40evf-port5-TxRx-2"
        set affinity-cpumask "0x08"
    next
    edit 16
        set interrupt "i40evf-port5-TxRx-3"
        set affinity-cpumask "0x08"
    next
    edit 17
        set interrupt "qat_00:14.00"
        set affinity-cpumask "0x01"
    next
    edit 18
        set interrupt "qat_00:15.00"
        set affinity-cpumask "0x02"
    next
    edit 19
        set interrupt "qat_00:16.00"
        set affinity-cpumask "0x04"
    next
    edit 20
        set interrupt "qat_00:17.00"
        set affinity-cpumask "0x08"
    next
end

This way, all four CPUs are balanced:

FGVM04TM19001384 (global) # get system performance status

CPU states: 0% user 2% system 0% nice 63% idle 0% iowait 0% irq 35% softirq

CPU0 states: 0% user 1% system 0% nice 69% idle 0% iowait 0% irq 30% softirq

CPU1 states: 0% user 3% system 0% nice 55% idle 0% iowait 0% irq 42% softirq

CPU2 states: 0% user 1% system 0% nice 72% idle 0% iowait 0% irq 27% softirq

CPU3 states: 0% user 2% system 0% nice 59% idle 0% iowait 0% irq 39% softirq

Memory: 6131096k total, 1092248k used (17.8%), 4908480k free (80.1%), 130368k freeable (2.1%)

Average network usage: 3825681 / 3813407 kbps in 1 minute, 1392107 / 1389299 kbps in 10 minutes, 632093 / 631744 kbps in 30 minutes

Average sessions: 37 sessions in 1 minute, 31 sessions in 10 minutes, 24 sessions in 30 minutes

Average session setup rate: 0 sessions per second in last 1 minute, 0 sessions per second in last 10 minutes, 0 sessions per second in last 30 minutes

Virus caught: 0 total in 1 minute

IPS attacks blocked: 0 total in 1 minute

Uptime: 0 days, 0 hours, 7 minutes

For how CPU interrupt affinity optimizes FortiGate-VM performance, see Technical Note: Optimize FortiGate-VM performance by configuring CPU interrupt affinity.

FGVM04TM19001384 (global) # diagnose hardware sysinfo interrupts
           CPU0       CPU1       CPU2       CPU3
  0:         26          0          0          0   IO-APIC-edge      timer
  1:          9          0          0          0   IO-APIC-edge      i8042
  4:         15          0          0          0   IO-APIC-edge      serial
  8:          0          0          0          0   IO-APIC-edge      rtc
  9:          0          0          0          0   IO-APIC-fasteoi   acpi
 10:          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb3, uhci_hcd:usb4
 11:         16          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb1, uhci_hcd:usb2
 12:          3          0          0          0   IO-APIC-edge      i8042
 14:          0          0          0          0   IO-APIC-edge      ata_piix
 15:          0          0          0          0   IO-APIC-edge      ata_piix
 40:          0          0          0          0   PCI-MSI-edge      virtio1-config
 41:        629          0          0          0   PCI-MSI-edge      virtio1-requests
 42:          0          0          0          0   PCI-MSI-edge      virtio3-config
 43:        978          0          0          0   PCI-MSI-edge      virtio3-input.0
 44:          1          0          0          0   PCI-MSI-edge      virtio3-output.0
 45:     255083          0          0          0   PCI-MSI-edge      qat_00:14.00
 46:         17     537891          0          0   PCI-MSI-edge      qat_00:15.00
 47:         17          0    1244511          0   PCI-MSI-edge      qat_00:16.00
 48:         17          0          0    1224563   PCI-MSI-edge      qat_00:17.00
 49:        173          0          0          0   PCI-MSI-edge      i40evf-0000:00:0a.0:mbx
 50:     119912          0          0          0   PCI-MSI-edge      i40evf-port2-TxRx-0
 51:          1     200309          0          0   PCI-MSI-edge      i40evf-port2-TxRx-1
 52:          1          0     538905          0   PCI-MSI-edge      i40evf-port2-TxRx-2
 53:          1          0          0     532128   PCI-MSI-edge      i40evf-port2-TxRx-3
 54:        172          0          0          0   PCI-MSI-edge      i40evf-0000:00:0b.0:mbx
 55:     254849          0          0          0   PCI-MSI-edge      i40evf-port3-TxRx-0
 56:          1     443186          0          0   PCI-MSI-edge      i40evf-port3-TxRx-1
 57:          1          0     600793          0   PCI-MSI-edge      i40evf-port3-TxRx-2
 58:          1          0          0     850484   PCI-MSI-edge      i40evf-port3-TxRx-3
 59:        172          0          0          0   PCI-MSI-edge      i40evf-0000:00:0c.0:mbx
 60:      72971          0          0          0   PCI-MSI-edge      i40evf-port4-TxRx-0
 61:          1     376044          0          0   PCI-MSI-edge      i40evf-port4-TxRx-1
 62:          1          0     531843          0   PCI-MSI-edge      i40evf-port4-TxRx-2
 63:          1          0          0     539088   PCI-MSI-edge      i40evf-port4-TxRx-3
 64:        172          0          0          0   PCI-MSI-edge      i40evf-0000:00:0d.0:mbx
 65:     197132          0          0          0   PCI-MSI-edge      i40evf-port5-TxRx-0
 66:          1     421851          0          0   PCI-MSI-edge      i40evf-port5-TxRx-1
 67:          1          0     850741          0   PCI-MSI-edge      i40evf-port5-TxRx-2
 68:          1          0          0     600896   PCI-MSI-edge      i40evf-port5-TxRx-3
NMI:          0          0          0          0   Non-maskable interrupts
LOC:      41936      46038      41842      46773   Local timer interrupts
SPU:          0          0          0          0   Spurious interrupts
PMI:          0          0          0          0   Performance monitoring interrupts
IWI:          0          0          0          0   IRQ work interrupts
RES:     282399       1450        787        751   Rescheduling interrupts
CAL:         46        106         62        107   Function call interrupts
TLB:         10         11          5          8   TLB shootdowns


FGVM04TM19001384 (vdom-1) # diagnose vpn ipsec status
All ipsec crypto devices in use:
QAT:
    Encryption (encrypted/decrypted)
        null             : 0                0
        des              : 0                0
        3des             : 0                0
        aes              : 48025403         29252461
        aes-gcm          : 0                0
        aria              : 0                0
        seed             : 0                0
        chacha20poly1305  : 0                0
    Integrity (generated/validated)
        null              : 0                0
        md5             : 0                0
        sha1             : 47967645        29250506
        sha256           : 0                0
        sha384           : 0                0
        sha512           : 0                0

SOFTWARE:
    Encryption (encrypted/decrypted)
        null             : 0                0
        des              : 0                0
        3des             : 0                0
        aes              : 0                0
        aes-gcm          : 0                0
        aria             : 0                0
        seed             : 0                0
        chacha20poly1305 : 0                0
    Integrity (generated/validated)
        null             : 0                0
        md5             : 0                0
        sha1             : 0                0
        sha256           : 0                0
        sha384           : 0                0
        sha512           : 0                0

Test Results:
1360 bytes IPSEC packet loss results with QAT in KVM04: 10,654m(v6.2 build 0984)

Resources

Deploying the FortiGate-VM on KVM

Deploying a FortiGate-VM on KVM with QAT support consists of the following steps:

  1. Create the FortiGate-VM on KVM.
  2. Inject SR-IOV network VFs into the FortiGate-VM.
  3. Configure interrupt affinities.
To create the FortiGate-VM on KVM:

To create a FortiGate-VM on KVM, refer to Deploying the FortiGate-VM.

To inject SR-IOV network VFs into the FortiGate-VM:

You can inject an SR-IOV network VF into a Linux KVM VM using one of the following ways:

  • Connecting an SR-IOV VF to a KVM VM by directly importing the VF as a PCI device using the PCI bus information that the host OS assigned to it when it was created
  • Using the Virtual Manager GUI
  • Adding an SR-IOV network adapter to the KVM VM as a VF network adapter connected to a macvtap on the host
  • Creating an SR-IOV VF network adapter using a KVM virtual network pool of adapters

See Configure SR-IOV Network Virtual Functions in Linux* KVM*.

In the following example, virtual network adapter pools were created for KVM04:

[root@localhost ~]# vnlist
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 default              active     yes           yes
 p5p1-pool            active     no            no
 p5p2-pool            active     no            no
 p7p1-pool            active     no            no
 p7p2-pool            active     no            no

[root@localhost ~]# vf2pf


Virtual Functions on Intel Corporation Ethernet Controller X710 for 10GbE SFP+. (p5p1):
PCI BDF         Interface
=======         =========
0000:86:02.0    p5p1_0
0000:86:02.1    p5p1_1
0000:86:02.2    p5p1_2
0000:86:02.3    p5p1_3
0000:86:02.4    p5p1_4
0000:86:02.5    p5p1_5
0000:86:02.6    p5p1_6
0000:86:02.7    p5p1_7

Virtual Functions on Intel Corporation Ethernet Controller X710 for 10GbE SFP+. (p5p2):
PCI BDF         Interface
=======         =========
0000:86:0a.0    p5p2_0
0000:86:0a.1    p5p2_1
0000:86:0a.2    p5p2_2
0000:86:0a.3    p5p2_3
0000:86:0a.4    p5p2_4
0000:86:0a.5    p5p2_5
0000:86:0a.6    p5p2_6
0000:86:0a.7    p5p2_7

Virtual Functions on Intel Corporation Ethernet Controller X710 for 10GbE SFP+. (p7p1):
PCI BDF         Interface
=======         =========
0000:88:02.0    p7p1_0
0000:88:02.1    p7p1_1
0000:88:02.2    p7p1_2
0000:88:02.3    p7p1_3
0000:88:02.4    p7p1_4
0000:88:02.5    p7p1_5
0000:88:02.6    p7p1_6
0000:88:02.7    p7p1_7

Virtual Functions on Intel Corporation Ethernet Controller X710 for 10GbE SFP+. (p7p2):
PCI BDF         Interface
=======         =========
0000:88:0a.0    p7p2_0
0000:88:0a.1    p7p2_1
0000:88:0a.2    p7p2_2
0000:88:0a.3    p7p2_3
0000:88:0a.4    p7p2_4
0000:88:0a.5    p7p2_5
0000:88:0a.6    p7p2_6
0000:88:0a.7    p7p2_7

The XML file is as follows. <cputune> locks the virtual CPUs to the same NUMA node, while <hostdev mode='subsystem' type='pci' managed='yes'> creates the QAT VFs:

[root@localhost ~]# virsh dumpxml vm04numa1
<domain type='kvm'>
  <name>vm04numa1</name>
  <uuid>fc5e1cec-8b4e-4bb8-9f89-e86f1abfffeb</uuid>
  <memory unit='KiB'>6291456</memory>
  <currentMemory unit='KiB'>6291456</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='1048576' unit='KiB'/>
    </hugepages>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='17'/>
    <vcpupin vcpu='1' cpuset='19'/>
    <vcpupin vcpu='2' cpuset='21'/>
    <vcpupin vcpu='3' cpuset='23'/>
    <emulatorpin cpuset='17,19,21,23'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='1'/>
  </numatune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.6.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='custom' match='exact' check='partial'>
    <model fallback='allow'>Skylake-Server-IBRS</model>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/vm04numa1.0984'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <interface type='direct'>
      <mac address='52:54:00:7c:07:50'/>
      <source dev='em1' mode='bridge'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:7c:07:53'/>
      <source network='p5p1-pool'/>
      <model type='i40e'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:7c:07:54'/>
      <source network='p5p2-pool'/>
      <model type='i40e'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:7c:07:55'/>
      <source network='p7p1-pool'/>
      <model type='i40e'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:7c:07:56'/>
      <source network='p7p2-pool'/>
      <model type='i40e'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0d' function='0x0'/>
    </interface>
    <serial type='tcp'>
      <source mode='bind' host='0.0.0.0' service='10004'/>
      <protocol type='telnet'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='tcp'>
      <source mode='bind' host='0.0.0.0' service='10004'/>
      <protocol type='telnet'/>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='5904' autoport='no' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>                                        
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0xb1' slot='0x01' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0e' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0xb1' slot='0x01' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0f' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0xb1' slot='0x01' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x10' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0xb1' slot='0x01' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x11' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
</domain>

The following shows the Virtual Manager GUI:

To configure interrupt affinities:

The example topology is as follows:

TestCenter----KVM port2----KVM port3===IPSEC tunnel===KVM port5-----KVM port4----TestCenter

After configuring the IPsec tunnel in the KVM, you must configure interrupt affinities and CPU masking to improve throughput for FortiGate-VM platforms. In this example, you can configure interrupt affinities as follows. You must manually set cpu-affinity for each QAT VF; otherwise, they only go through the first CPU.

config system affinity-interrupt
    edit 1
        set interrupt "i40evf-port2-TxRx-0"
        set affinity-cpumask "0x01"
    next
    edit 2
        set interrupt "i40evf-port2-TxRx-1"
        set affinity-cpumask "0x01"
    next
    edit 3
        set interrupt "i40evf-port2-TxRx-2"
        set affinity-cpumask "0x01"
    next
    edit 4
        set interrupt "i40evf-port2-TxRx-3"
        set affinity-cpumask "0x01"
    next
    edit 5
        set interrupt "i40evf-port3-TxRx-0"
        set affinity-cpumask "0x02"
    next
    edit 6
        set interrupt "i40evf-port3-TxRx-1"
        set affinity-cpumask "0x02"
    next
    edit 7
        set interrupt "i40evf-port3-TxRx-2"
        set affinity-cpumask "0x02"
    next
    edit 8
        set interrupt "i40evf-port3-TxRx-3"
        set affinity-cpumask "0x02"
    next
    edit 9
        set interrupt "i40evf-port4-TxRx-0"
        set affinity-cpumask "0x04"
    next
    edit 10
        set interrupt "i40evf-port4-TxRx-1"
        set affinity-cpumask "0x04"
    next
    edit 11
        set interrupt "i40evf-port4-TxRx-2"
        set affinity-cpumask "0x04"
    next
    edit 12
        set interrupt "i40evf-port4-TxRx-3"
        set affinity-cpumask "0x04"
    next
    edit 13
        set interrupt "i40evf-port5-TxRx-0"
        set affinity-cpumask "0x08"
    next
    edit 14
        set interrupt "i40evf-port5-TxRx-1"
        set affinity-cpumask "0x08"
    next
    edit 15
        set interrupt "i40evf-port5-TxRx-2"
        set affinity-cpumask "0x08"
    next
    edit 16
        set interrupt "i40evf-port5-TxRx-3"
        set affinity-cpumask "0x08"
    next
    edit 17
        set interrupt "qat_00:14.00"
        set affinity-cpumask "0x01"
    next
    edit 18
        set interrupt "qat_00:15.00"
        set affinity-cpumask "0x02"
    next
    edit 19
        set interrupt "qat_00:16.00"
        set affinity-cpumask "0x04"
    next
    edit 20
        set interrupt "qat_00:17.00"
        set affinity-cpumask "0x08"
    next
end

This way, all four CPUs are balanced:

FGVM04TM19001384 (global) # get system performance status

CPU states: 0% user 2% system 0% nice 63% idle 0% iowait 0% irq 35% softirq

CPU0 states: 0% user 1% system 0% nice 69% idle 0% iowait 0% irq 30% softirq

CPU1 states: 0% user 3% system 0% nice 55% idle 0% iowait 0% irq 42% softirq

CPU2 states: 0% user 1% system 0% nice 72% idle 0% iowait 0% irq 27% softirq

CPU3 states: 0% user 2% system 0% nice 59% idle 0% iowait 0% irq 39% softirq

Memory: 6131096k total, 1092248k used (17.8%), 4908480k free (80.1%), 130368k freeable (2.1%)

Average network usage: 3825681 / 3813407 kbps in 1 minute, 1392107 / 1389299 kbps in 10 minutes, 632093 / 631744 kbps in 30 minutes

Average sessions: 37 sessions in 1 minute, 31 sessions in 10 minutes, 24 sessions in 30 minutes

Average session setup rate: 0 sessions per second in last 1 minute, 0 sessions per second in last 10 minutes, 0 sessions per second in last 30 minutes

Virus caught: 0 total in 1 minute

IPS attacks blocked: 0 total in 1 minute

Uptime: 0 days, 0 hours, 7 minutes

For how CPU interrupt affinity optimizes FortiGate-VM performance, see Technical Note: Optimize FortiGate-VM performance by configuring CPU interrupt affinity.

FGVM04TM19001384 (global) # diagnose hardware sysinfo interrupts
           CPU0       CPU1       CPU2       CPU3
  0:         26          0          0          0   IO-APIC-edge      timer
  1:          9          0          0          0   IO-APIC-edge      i8042
  4:         15          0          0          0   IO-APIC-edge      serial
  8:          0          0          0          0   IO-APIC-edge      rtc
  9:          0          0          0          0   IO-APIC-fasteoi   acpi
 10:          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb3, uhci_hcd:usb4
 11:         16          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb1, uhci_hcd:usb2
 12:          3          0          0          0   IO-APIC-edge      i8042
 14:          0          0          0          0   IO-APIC-edge      ata_piix
 15:          0          0          0          0   IO-APIC-edge      ata_piix
 40:          0          0          0          0   PCI-MSI-edge      virtio1-config
 41:        629          0          0          0   PCI-MSI-edge      virtio1-requests
 42:          0          0          0          0   PCI-MSI-edge      virtio3-config
 43:        978          0          0          0   PCI-MSI-edge      virtio3-input.0
 44:          1          0          0          0   PCI-MSI-edge      virtio3-output.0
 45:     255083          0          0          0   PCI-MSI-edge      qat_00:14.00
 46:         17     537891          0          0   PCI-MSI-edge      qat_00:15.00
 47:         17          0    1244511          0   PCI-MSI-edge      qat_00:16.00
 48:         17          0          0    1224563   PCI-MSI-edge      qat_00:17.00
 49:        173          0          0          0   PCI-MSI-edge      i40evf-0000:00:0a.0:mbx
 50:     119912          0          0          0   PCI-MSI-edge      i40evf-port2-TxRx-0
 51:          1     200309          0          0   PCI-MSI-edge      i40evf-port2-TxRx-1
 52:          1          0     538905          0   PCI-MSI-edge      i40evf-port2-TxRx-2
 53:          1          0          0     532128   PCI-MSI-edge      i40evf-port2-TxRx-3
 54:        172          0          0          0   PCI-MSI-edge      i40evf-0000:00:0b.0:mbx
 55:     254849          0          0          0   PCI-MSI-edge      i40evf-port3-TxRx-0
 56:          1     443186          0          0   PCI-MSI-edge      i40evf-port3-TxRx-1
 57:          1          0     600793          0   PCI-MSI-edge      i40evf-port3-TxRx-2
 58:          1          0          0     850484   PCI-MSI-edge      i40evf-port3-TxRx-3
 59:        172          0          0          0   PCI-MSI-edge      i40evf-0000:00:0c.0:mbx
 60:      72971          0          0          0   PCI-MSI-edge      i40evf-port4-TxRx-0
 61:          1     376044          0          0   PCI-MSI-edge      i40evf-port4-TxRx-1
 62:          1          0     531843          0   PCI-MSI-edge      i40evf-port4-TxRx-2
 63:          1          0          0     539088   PCI-MSI-edge      i40evf-port4-TxRx-3
 64:        172          0          0          0   PCI-MSI-edge      i40evf-0000:00:0d.0:mbx
 65:     197132          0          0          0   PCI-MSI-edge      i40evf-port5-TxRx-0
 66:          1     421851          0          0   PCI-MSI-edge      i40evf-port5-TxRx-1
 67:          1          0     850741          0   PCI-MSI-edge      i40evf-port5-TxRx-2
 68:          1          0          0     600896   PCI-MSI-edge      i40evf-port5-TxRx-3
NMI:          0          0          0          0   Non-maskable interrupts
LOC:      41936      46038      41842      46773   Local timer interrupts
SPU:          0          0          0          0   Spurious interrupts
PMI:          0          0          0          0   Performance monitoring interrupts
IWI:          0          0          0          0   IRQ work interrupts
RES:     282399       1450        787        751   Rescheduling interrupts
CAL:         46        106         62        107   Function call interrupts
TLB:         10         11          5          8   TLB shootdowns


FGVM04TM19001384 (vdom-1) # diagnose vpn ipsec status
All ipsec crypto devices in use:
QAT:
    Encryption (encrypted/decrypted)
        null             : 0                0
        des              : 0                0
        3des             : 0                0
        aes              : 48025403         29252461
        aes-gcm          : 0                0
        aria              : 0                0
        seed             : 0                0
        chacha20poly1305  : 0                0
    Integrity (generated/validated)
        null              : 0                0
        md5             : 0                0
        sha1             : 47967645        29250506
        sha256           : 0                0
        sha384           : 0                0
        sha512           : 0                0

SOFTWARE:
    Encryption (encrypted/decrypted)
        null             : 0                0
        des              : 0                0
        3des             : 0                0
        aes              : 0                0
        aes-gcm          : 0                0
        aria             : 0                0
        seed             : 0                0
        chacha20poly1305 : 0                0
    Integrity (generated/validated)
        null             : 0                0
        md5             : 0                0
        sha1             : 0                0
        sha256           : 0                0
        sha384           : 0                0
        sha512           : 0                0

Test Results:
1360 bytes IPSEC packet loss results with QAT in KVM04: 10,654m(v6.2 build 0984)