IPsec SA key retrieval from a KMS server using KMIP

In environments that require centralized management of cryptographic keys where no key derivations or algorithmic operations are allowed on edge devices (such as the FortiGate), they will deploy a Key Management Services (KMS) server cluster to generate and manage all cryptographic keys. Then, the Key Management Interoperability Protocol (KMIP) is used on the edge devices to locate the KMS server, create keys if they do not exist, and retrieve keys to be used for securing these edge devices.

FortiGates have a KMIP client that sends KMIP requests to locate the Key Management Services (KMS) server, creates keys if they do not exist on the KMS server, and retrieves keys from the KMS server to use as IPsec security association (SA) keys for IKEv2 only.

This feature allows the FortiGate to offload the task of generating IPsec SA keys to a KMS server, regardless of specific IPsec VPN topologies with a FortiGate, when the administrator has the requirement to centralize cryptographic keys management in a KMS server.

The FortiGate's integrated KMIP client also supports the following:

• If the KMS server is unavailable, then the FortiGate continues to use the previous keys to avoid a network blackout.

• ADVPN configurations for the hub and spoke, so that shortcuts between two spokes will use their own encryption keys retrieved from the KMS server.

• Multiple tunnels between the same tunnel endpoints using multiple VRFs.

To configure the KMIP server:

config vpn kmip-server
    edit <KMS_server_ID>
        config server-list
            edit <ID>
                set server <server_IP>
                set cert <string>
            next
        end
        set username <username_defined_on_KMS_server>
        set password <password>
    next
end

To apply the KMS server in the phase 1 interface settings:

config vpn ipsec phase1-interface
    edit <name>
        set kms <KMS_server_ID>
    next
end

IPsec tunnels will not be established if a FortiGate VPN peer does not support KMS, or has not configured kms <KMS_server_ID> in config vpn ipsec phase1-interface.

The following diagnostic commands have been added:

• get vpn ike kms-keys
• diagnose debug application kmipd -1
• execute kmip {create | destroy | get | locate | rekey} <parameter>

Example

In this example, there is a topology with an ADVPN hub FortiGate and two spoke FortiGates. There is a cluster or three KMS server VMs (172.16.200.221, 172.16.200.222, and 172.16.200.223) that operates in round-robin mode. The testuser1_Cert certificate is issued by the KMS server, and the testuser1 user is defined on the KMS server. Authentication to the KMS server by the KMIP client requires both a certificate and a password.

The Hub FortiGate acting as the responder will try to locate keys on the KMS server first. If they do not exist, the FortiGate requests to create new keys on KMS server. The responder sends the keys’ names to the Spoke1 and Spoke2 FortiGates acting as the initiators using IKE messages, and these initiators locate and retrieve keys from KMS server using the keys’ names. The keylifeseconds parameter in phase 2 defines how often the FortiGate will try to synchronize local keys to those on the KMS server.

The keys are retrieved from the KMS server and used as IPsec SA keys in IPsec tunnels. The key format used is: [IDi/r]-[IDr/i]-[phase2name]-ENC/AUTH-[keyalg]-[keylen].

First, this example focuses on the Hub FortiGate and the IPsec VPN connection between the Spoke1 and Hub FortiGate. Second, this example focuses on the spoke-to-spoke tunnel, also known as a shortcut tunnel or shortcut, which is established when traffic flows between the Spoke1 and Spoke2 FortiGates.

To configure IPsec SA key retrieval from a KMS server on the Hub FortiGate:

1. Configure the KMIP server:

   config vpn kmip-server
       edit "KMS_server"
           config server-list
               edit 1
                   set server "172.16.200.221"
                   set cert "testuser1_Cert"
               next
               edit 2
                   set server "172.16.200.222"
                   set cert "testuser1_Cert"
               next
               edit 3
                   set server "172.16.200.223"
                   set cert "testuser1_Cert"
               next
           end
           set username "testuser1"
           set password **********
       next
   end

2. Configure the IPsec VPN phase 1 settings:

   config vpn ipsec phase1-interface
       edit "hub"
           set type dynamic
           set interface "port2"
           set ike-version 2
           set authmethod signature
           set peertype any
           set net-device disable
           set proposal aes128-sha256 aes256-sha256 aes128gcm-prfsha256 aes256gcm-prfsha384 chacha20poly1305-prfsha256
           set add-route disable
           set dpd on-idle
           set auto-discovery-sender enable
           set kms "KMS_server"
           set certificate "Fortinet_Factory_Backup"
           set dpd-retryinterval 60
       next
   end

   This feature is only supported in IKEv2. The localid is required in the phase 1 settings when using the PSK authentication method.

3. Configure the IPsec VPN phase 2 settings:

   config vpn ipsec phase2-interface
       edit "hub"
           set phase1name "hub"
           set proposal aes128-sha1 aes256-sha1 aes128-sha256 aes256-sha256 aes128gcm aes256gcm chacha20poly1305
           set keylifeseconds 7200
       next
   end

To verify the IPsec configuration and tunnel between the Hub and Spoke1 FortiGates:

1. Verify the tunnel state on the Hub:

   Hub # diagnose vpn tunnel list
   list all ipsec tunnel in vd 0
   ------------------------------------------------------
   name=hub ver=2 serial=1 172.16.200.4:0->0.0.0.0:0 tun_id=10.0.0.1 tun_id6=::10.0.0.1 dst_mtu=0 dpd-link=on weight=1
   bound_if=10 lgwy=static/1 tun=intf mode=dialup/2 encap=none/552 options[0228]=npu frag-rfc  role=primary accept_traffic=1 overlay_id=0
   
   proxyid_num=0 child_num=2 refcnt=4 ilast=42965007 olast=42965007 ad=/0
   stat: rxp=980 txp=1980 rxb=125003 txb=123108
   dpd: mode=on-idle on=0 idle=60000ms retry=3 count=0 seqno=0
   natt: mode=none draft=0 interval=0 remote_port=0
   fec: egress=0 ingress=0
   run_tally=0
   ------------------------------------------------------
   name=hub_0 ver=2 serial=10 172.16.200.4:0->172.16.200.1:0 tun_id=10.10.10.2 tun_id6=::10.0.0.16 dst_mtu=1500 dpd-link=on weight=1
   bound_if=10 lgwy=static/1 tun=intf mode=dial_inst/3 encap=none/74408 options[122a8]=npu rgwy-chg frag-rfc  run_state=0 role=primary accept_traffic=1 overlay_id=0
   
   parent=hub index=0
   proxyid_num=1 child_num=0 refcnt=5 ilast=6 olast=6 ad=s/1
   stat: rxp=21 txp=39 rxb=2644 txb=2389
   dpd: mode=on-idle on=1 idle=60000ms retry=3 count=0 seqno=1
   natt: mode=none draft=0 interval=0 remote_port=0
   fec: egress=0 ingress=0
   proxyid=hub proto=0 sa=1 ref=3 serial=1 ads
     src: 0:0.0.0.0-255.255.255.255:0
     dst: 0:0.0.0.0-255.255.255.255:0
     SA:  ref=6 options=826 type=00 soft=0 mtu=1438 expire=6673/0B replaywin=2048
          seqno=15 esn=0 replaywin_lastseq=00000002 qat=0 rekey=0 hash_search_len=1
     life: type=01 bytes=0/0 timeout=7191/7200
     dec: spi=628d1814 esp=aes key=16 5dad0d8d3568eab7c3f259349dc64039
          ah=sha1 key=20 e660f491b80b2cfdcdb0d737942bea2e853dac8d
     enc: spi=471dfe2e esp=aes key=16 1de4b8e8accaa792e0934fbd9f933a6a
          ah=sha1 key=20 1fa244d3971b4d4df59b8d7b3655a1b77f8e65af
     dec:pkts/bytes=22/2696, enc:pkts/bytes=59/4949
     npu_flag=03 npu_rgwy=172.16.200.1 npu_lgwy=172.16.200.4 npu_selid=e dec_npuid=1 enc_npuid=0
   ------------------------------------------------------
   name=hub_1 ver=2 serial=f 172.16.200.4:0->172.16.200.3:0 tun_id=10.10.10.3 tun_id6=::10.0.0.15 dst_mtu=1500 dpd-link=on weight=1
   bound_if=10 lgwy=static/1 tun=intf mode=dial_inst/3 encap=none/74408 options[122a8]=npu rgwy-chg frag-rfc  run_state=0 role=primary accept_traffic=1 overlay_id=0
   
   parent=hub index=1
   proxyid_num=1 child_num=0 refcnt=5 ilast=2 olast=2 ad=s/1
   stat: rxp=21 txp=43 rxb=2615 txb=2718
   dpd: mode=on-idle on=1 idle=60000ms retry=3 count=0 seqno=1
   natt: mode=none draft=0 interval=0 remote_port=0
   fec: egress=0 ingress=0
   proxyid=hub proto=0 sa=1 ref=3 serial=1 ads
     src: 0:0.0.0.0-255.255.255.255:0
     dst: 0:0.0.0.0-255.255.255.255:0
     SA:  ref=6 options=826 type=00 soft=0 mtu=1438 expire=6665/0B replaywin=2048
          seqno=17 esn=0 replaywin_lastseq=00000002 qat=0 rekey=0 hash_search_len=1
     life: type=01 bytes=0/0 timeout=7187/7200
     dec: spi=628d1813 esp=aes key=16 5fcca9194ced21b0a586a8fd7a27cbf7
          ah=sha1 key=20 6d6d9dc77d5af89f062927c4d4695d404df1ffe3
     enc: spi=8d568113 esp=aes key=16 2006f323b760238048fcd6f7783b0a04
          ah=sha1 key=20 bd6db68ee035088f35174b2b5c58a51fbbe3f5b5
     dec:pkts/bytes=22/2686, enc:pkts/bytes=65/5566
     npu_flag=03 npu_rgwy=172.16.200.3 npu_lgwy=172.16.200.4 npu_selid=d dec_npuid=1 enc_npuid=0

2. Verify the KMS keys for the VPN tunnel between the Hub and Spoke1:

   Hub # get vpn ike  kms-keys
   
   vd: root/0
   name: hub_1
   addr: 172.16.200.4:500 -> 172.16.200.3:500
   
     phase2
     name: hub
     server: "KMS_server"
     spi: 628d1813
       enc
         keyname: "Spoke2-hub-hub-ENC-AES-16"
         key: 5fcca9194ced21b0a586a8fd7a27cbf7
       auth
         keyname: "Spoke2-hub-hub-AUTH-SHA1-20"
         key: 6d6d9dc77d5af89f062927c4d4695d404df1ffe3
     spi: 8d568113
       enc
         keyname: "hub-Spoke2-hub-ENC-AES-16"
         key: 2006f323b760238048fcd6f7783b0a04
       auth
         keyname: "hub-Spoke2-hub-AUTH-SHA1-20"
         key: bd6db68ee035088f35174b2b5c58a51fbbe3f5b5
   
   
   vd: root/0
   name: hub_0
   addr: 172.16.200.4:500 -> 172.16.200.1:500
   
     phase2
     name: hub
     server: "KMS_server"
     spi: 628d1814
       enc
         keyname: "Spoke1-hub-hub-ENC-AES-16"
         key: 5dad0d8d3568eab7c3f259349dc64039
       auth
         keyname: "Spoke1-hub-hub-AUTH-SHA1-20"
         key: e660f491b80b2cfdcdb0d737942bea2e853dac8d
     spi: 471dfe2e
       enc
         keyname: "hub-Spoke1-hub-ENC-AES-16"
         key: 1de4b8e8accaa792e0934fbd9f933a6a
       auth
         keyname: "hub-Spoke1-hub-AUTH-SHA1-20"
         key: 1fa244d3971b4d4df59b8d7b3655a1b77f8e65af

3. Verify the IKE and KMIP debug messages on both FortiGates:

   # diagnose debug application ike -1
   # diagnose debug application kmipd -1

   1. For the responder FortiGate, Hub:

      ike 0: comes 172.16.200.1:500->172.16.200.4:500,ifindex=10,vrf=0....
      ike 0: IKEv2 exchange=AUTH id=6e99ee7fd238b462/82e575f08b93f44c:00000001 len=708
      ike 0:hub:537: encrypted fragment 3 of 3 queued
      ike 0:hub:537: reassembled fragmented message
      ike 0:hub:537: responder received AUTH msg
      ike 0:hub:537: processing notify type INITIAL_CONTACT
      ike 0:hub:537: processing notify type INTERFACE_ADDR4
      ike 0:hub:537: INTERFACE-ADDR4 10.10.10.2
      ike 0:hub:537: processing notify type MESSAGE_ID_SYNC_SUPPORTED
      ike 0:hub:537: processing notify type KMS_SUPPORT
      ...
      ike 0:hub:hub: sending kmip locate request: id=4321 keyname=Spoke1-hub-hub-ENC-AES-16
      ike 0:hub:hub: sending kmip locate request: id=4322 keyname=hub-Spoke1-hub-ENC-AES-16
      ike 0:hub:hub: sending kmip locate request: id=4323 keyname=Spoke1-hub-hub-AUTH-SHA1-20
      ike 0:hub:hub: sending kmip locate request: id=4324 keyname=hub-Spoke1-hub-AUTH-SHA1-20
      ...
      ike 0:hub:hub: sending kmip create request: id=4328 keyname=hub-Spoke1-hub-AUTH-SHA1-20 keyalg=7 keylen=160
      kmip_tsk_resp_finalizer()-365: server-KMS_server, vfid-0, cur_total-4, batch_count-4
      kmip_free_tsk()-144: Freeing tsk pid=6487, job_id=4321, seq=4321
      kmip_free_tsk()-144: Freeing tsk pid=6487, job_id=4322, seq=4322
      kmip_free_tsk()-144: Freeing tsk pid=6487, job_id=4323, seq=4323
      kmip_free_tsk()-144: Freeing tsk pid=6487, job_id=4324, seq=4324
      ...
      kmipd_op_create_req_check()-35: New tsk for 'KMS_server', op-create, vfid-0, pid-6487, job_id-4325, name-'Spoke1-hub-hub-ENC-AES-16'
      kmip_new_tsk()-131: New tsk pid=6487, job_id=4325, seq=4325
      ...
      kmipd_op_create_req_check()-35: New tsk for 'KMS_server', op-create, vfid-0, pid-6487, job_id-4326, name-'hub-Spoke1-hub-ENC-AES-16'
      kmip_new_tsk()-131: New tsk pid=6487, job_id=4326, seq=4326
      ...
      kmipd_op_create_req_check()-35: New tsk for 'KMS_server', op-create, vfid-0, pid-6487, job_id-4327, name-'Spoke1-hub-hub-AUTH-SHA1-20'
      kmip_new_tsk()-131: New tsk pid=6487, job_id=4327, seq=4327
      ...
      kmipd_op_create_req_check()-35: New tsk for 'KMS_server', op-create, vfid-0, pid-6487, job_id-4328, name-'hub-Spoke1-hub-AUTH-SHA1-20'
      kmip_new_tsk()-131: New tsk pid=6487, job_id=4328, seq=4328
      ...
      kmip_send_reply()-32: Sending 28 data. Job_id-4332 ret-0
      ike KMIP response received: id=4332 ret=0
      ike 0:hub:hub processing kmip get-response
      ike 0:hub:hub recevied KMS keys 4/4
      ...
      ike 0:hub: adding new dynamic tunnel for 172.16.200.1:500
      ike 0:hub_0: tunnel created tun_id 10.10.10.2/::10.0.0.12 remote_location 0.0.0.0
      ike 0:hub_0: added new dynamic tunnel for 172.16.200.1:500
      ike 0:hub_0:539: established IKE SA 709d9a9eab5b5a48/01afbbcfa47c1459
      ike 0:hub_0:539: auto-discovery sender
      ike 0:hub_0:539: auto-discovery 1
      ike 0:hub_0:539: check peer route: if_addr4_rcvd=1, if_addr6_rcvd=0, mode_cfg=0
      ike 0:hub_0:539: update peer route 0.0.0.0 -> 10.10.10.2
      ike 0:hub_0:539: processing INITIAL-CONTACT
      ike 0:hub_0: flushing
      ike 0:hub_0: flushed
      ike 0:hub_0:539: processed INITIAL-CONTACT
      ike 0:hub_0:539: local cert, subject='hub', issuer='support'
      ike 0:hub_0:539: local CA cert, subject='support', issuer='support'
      ike 0:hub_0:539: add INTERFACE-ADDR4 10.10.10.1
      ike 0:hub_0:hub: added KMS_KEY payloads
      ike 0:hub_0:539:hub:1085: replay protection enabled
      ike 0:hub_0:539:hub:1085: set sa life soft seconds=7190.
      ike 0:hub_0:539:hub:1085: set sa life hard seconds=7200.
      ike 0:hub_0:539:hub:1085: IPsec SA selectors #src=1 #dst=1
      ike 0:hub_0:539:hub:1085: src 0 7 0:0.0.0.0-255.255.255.255:0
      ike 0:hub_0:539:hub:1085: dst 0 7 0:0.0.0.0-255.255.255.255:0
      ike 0:hub_0:539:hub:1085: add dynamic IPsec SA selectors
      ike 0:hub_0:539:hub:1085: added dynamic IPsec SA proxyids, new serial 1
      ike 0:hub_0:539:hub:1085: tunnel 2 of VDOM limit 0/0
      ike 0:hub_0:539:hub:1085: add IPsec SA: SPIs=628d180e/471dfe29
      ike 0:hub_0:539:hub:1085: IPsec SA dec spi 628d180e key 16:5DAD0D8D3568EAB7C3F259349DC64039 auth 20:E660F491B80B2CFDCDB0D737942BEA2E853DAC8D
      ike 0:hub_0:539:hub:1085: IPsec SA enc spi 471dfe29 key 16:1DE4B8E8ACCAA792E0934FBD9F933A6A auth 20:1FA244D3971B4D4DF59B8D7B3655A1B77F8E65AF
      ike 0:hub_0:539:hub:1085: added IPsec SA: SPIs=628d180e/471dfe29
      ike 0:hub_0: tunnel up event
      ike 0:hub_0:539:hub:1085: sending SNMP tunnel UP trap

   2. For the initiator FortiGate, Spoke1:

      ike 0:spoke1: schedule auto-negotiate
      ike 0:spoke1:spoke1: initiator received KMS_KEY: "Spoke1-hub-hub-ENC-AES-16" "hub-Spoke1-hub-ENC-AES-16" "Spoke1-hub-hub-AUTH-SHA1-20" "hub-Spoke1-hub-AUTH-SHA1-20"
      ...
      ike 0:spoke1:spoke1: sending kmip locate request: id=77 keyname=Spoke1-hub-hub-ENC-AES-16
      ike 0:spoke1:spoke1: sending kmip locate request: id=78 keyname=hub-Spoke1-hub-ENC-AES-16
      ike 0:spoke1:spoke1: sending kmip locate request: id=79 keyname=Spoke1-hub-hub-AUTH-SHA1-20
      ike 0:spoke1:spoke1: sending kmip locate request: id=80 keyname=hub-Spoke1-hub-AUTH-SHA1-20
      ...
      kmipd_op_locate_req_check()-48: New tsk for 'KMS_server', op-locate, vfid-0, pid-3341, job_id-78, name-'hub-Spoke1-hub-ENC-AES-16'
      kmip_new_tsk()-131: New tsk pid=3341, job_id=78, seq=78
      ...
      kmipd_op_locate_req_check()-48: New tsk for 'KMS_server', op-locate, vfid-0, pid-3341, job_id-79, name-'Spoke1-hub-hub-AUTH-SHA1-20'
      kmip_new_tsk()-131: New tsk pid=3341, job_id=79, seq=79
      ...
      kmipd_op_locate_req_check()-48: New tsk for 'KMS_server', op-locate, vfid-0, pid-3341, job_id-80, name-'hub-Spoke1-hub-AUTH-SHA1-20'
      kmip_new_tsk()-131: New tsk pid=3341, job_id=80, seq=80
      ...
      kmipd_op_locate_req_check()-48: New tsk for 'KMS_server', op-locate, vfid-0, pid-3341, job_id-77, name-'Spoke1-hub-hub-ENC-AES-16'
      kmip_new_tsk()-131: New tsk pid=3341, job_id=77, seq=77
      ...
      kmipd_op_get_req_check()-35: New tsk for 'KMS_server', op-get, vfid-0, pid-3341, job_id-81, keyid-'a98f50b20bfe4037a7c47283eef578e61b474bf3829f45beb4a6c972c31a5d63'
      kmip_new_tsk()-131: New tsk pid=3341, job_id=81, seq=81
      ...
      kmipd_op_get_req_check()-35: New tsk for 'KMS_server', op-get, vfid-0, pid-3341, job_id-82, keyid-'b4867ef7052b484faea2e7916b585bfc171e0981b843444097ee39d67fba30ea'
      kmip_new_tsk()-131: New tsk pid=3341, job_id=82, seq=82
      ...
      kmipd_op_get_req_check()-35: New tsk for 'KMS_server', op-get, vfid-0, pid-3341, job_id-83, keyid-'41d4e37c4a014811a78cd1e1053d6370edc62a5a975e46c8a8aeda3bf4d76061'
      kmip_new_tsk()-131: New tsk pid=3341, job_id=83, seq=83
      ...
      kmipd_op_get_req_check()-35: New tsk for 'KMS_server', op-get, vfid-0, pid-3341, job_id-84, keyid-'2ba130bff7174ba7a237d7ea53611121383b132cf18a4fd183890ca196296cb4'
      kmip_new_tsk()-131: New tsk pid=3341, job_id=84, seq=84
      ...
      ike 0:spoke1:spoke1 processing kmip get-response
      ike 0:spoke1:spoke1 recevied KMS keys 4/4
      ike 0:spoke1:536:spoke1:549: replay protection enabled
      ike 0:spoke1:536:spoke1:549: set sa life soft seconds=6901.
      ike 0:spoke1:536:spoke1:549: set sa life hard seconds=7200.
      ike 0:spoke1:536:spoke1:549: IPsec SA selectors #src=1 #dst=1
      ike 0:spoke1:536:spoke1:549: src 0 7 0:0.0.0.0-255.255.255.255:0
      ike 0:spoke1:536:spoke1:549: dst 0 7 0:0.0.0.0-255.255.255.255:0
      ike 0:spoke1:536:spoke1:549: add IPsec SA: SPIs=471dfe29/628d180e
      ike 0:spoke1:536:spoke1:549: IPsec SA dec spi 471dfe29 key 16:1DE4B8E8ACCAA792E0934FBD9F933A6A auth 20:1FA244D3971B4D4DF59B8D7B3655A1B77F8E65AF
      ike 0:spoke1:536:spoke1:549: IPsec SA enc spi 628d180e key 16:5DAD0D8D3568EAB7C3F259349DC64039 auth 20:E660F491B80B2CFDCDB0D737942BEA2E853DAC8D
      ike 0:spoke1:536:spoke1:549: added IPsec SA: SPIs=471dfe29/628d180e
      ike 0:spoke1:536:spoke1:549: sending SNMP tunnel UP trap

To verify the IPsec configuration and tunnel between the Spoke1 and Spoke2 FortiGates:

1. Verify the tunnel state on Spoke1:

   Spoke1 # diagnose vpn tunnel list
   list all ipsec tunnel in vd 0
   ------------------------------------------------------
   name=spoke1 ver=2 serial=1 172.16.200.1:0->172.16.200.4:0 tun_id=172.16.200.4 tun_id6=::172.16.200.4 dst_mtu=1500 dpd-link=on weight=1
   bound_if=19 lgwy=static/1 tun=intf mode=auto/1 encap=none/552 options[0228]=npu frag-rfc  run_state=0 role=primary accept_traffic=1 overlay_id=0 proxyid_num=1 child_num=1 refcnt=5 ilast=35 olast=35 ad=r/2
   stat: rxp=1 txp=11 rxb=71 txb=699
   dpd: mode=on-demand on=1 idle=20000ms retry=3 count=0 seqno=5
   natt: mode=none draft=0 interval=0 remote_port=0
   fec: egress=0 ingress=0
   proxyid=spoke1 proto=0 sa=1 ref=3 serial=2 adr
     src: 0:0.0.0.0-255.255.255.255:0
     dst: 0:0.0.0.0-255.255.255.255:0
     SA:  ref=6 options=12026 type=00 soft=0 mtu=1438 expire=6621/0B replaywin=2048
          seqno=c esn=0 replaywin_lastseq=00000002 qat=0 rekey=0 hash_search_len=1
     life: type=01 bytes=0/0 timeout=6903/7200
     dec: spi=471dfe2e esp=aes key=16 1de4b8e8accaa792e0934fbd9f933a6a
          ah=sha1 key=20 1fa244d3971b4d4df59b8d7b3655a1b77f8e65af
     enc: spi=628d1814 esp=aes key=16 5dad0d8d3568eab7c3f259349dc64039
          ah=sha1 key=20 e660f491b80b2cfdcdb0d737942bea2e853dac8d
     dec:pkts/bytes=2/142, enc:pkts/bytes=22/2131
     npu_flag=03 npu_rgwy=172.16.200.4 npu_lgwy=172.16.200.1 npu_selid=1 dec_npuid=2 enc_npuid=2
   run_tally=0
   ------------------------------------------------------
   name=spoke1_0 ver=2 serial=4 172.16.200.1:0->172.16.200.3:0 tun_id=172.16.200.3 tun_id6=::172.16.200.3 dst_mtu=1500 dpd-link=on weight=1
   bound_if=19 lgwy=static/1 tun=intf mode=dial_inst/3 encap=none/66216 options[102a8]=npu rgwy-chg frag-rfc  run_state=0 role=primary accept_traffic=1 overlay_id=0 parent=spoke1 index=0
   proxyid_num=1 child_num=0 refcnt=5 ilast=10 olast=10 ad=r/2
   stat: rxp=1 txp=5 rxb=84 txb=420
   dpd: mode=on-demand on=1 idle=20000ms retry=3 count=0 seqno=1
   natt: mode=none draft=0 interval=0 remote_port=0
   fec: egress=0 ingress=0
   proxyid=spoke1 proto=0 sa=1 ref=3 serial=1 adr
     src: 0:0.0.0.0-255.255.255.255:0
     dst: 0:0.0.0.0-255.255.255.255:0
     SA:  ref=6 options=12026 type=00 soft=0 mtu=1438 expire=6947/0B replaywin=2048
          seqno=6 esn=0 replaywin_lastseq=00000402 qat=0 rekey=0 hash_search_len=1
     life: type=01 bytes=0/0 timeout=7190/7200
     dec: spi=471dfe2f esp=aes key=16 a6d6a25cd986860bcc502d58f32e99de
          ah=sha1 key=20 07d712156eaca28439fbe944e3a8c9af4c45166a
     enc: spi=8d568114 esp=aes key=16 b01c534b11792b856c1b95c78c4cad91
          ah=sha1 key=20 fe6a82177db6911b3203d1306969e5ddec8fd039
     dec:pkts/bytes=2/168, enc:pkts/bytes=10/1180
     npu_flag=03 npu_rgwy=172.16.200.3 npu_lgwy=172.16.200.1 npu_selid=4 dec_npuid=2 enc_npuid=2

2. Verify the KMS keys for the VPN tunnel between Spoke1 and Spoke2:

   Spoke1 # get vpn ike kms-keys
   
   vd: root/0
   name: spoke1
   addr: 172.16.200.1:500 -> 172.16.200.4:500
   
     phase2
     name: spoke1
     server: "KMS_server"
     spi: 628d1814
       enc
         keyname: "Spoke1-hub-hub-ENC-AES-16"
         key: 5dad0d8d3568eab7c3f259349dc64039
       auth
         keyname: "Spoke1-hub-hub-AUTH-SHA1-20"
         key: e660f491b80b2cfdcdb0d737942bea2e853dac8d
     spi: 471dfe2e
       enc
         keyname: "hub-Spoke1-hub-ENC-AES-16"
         key: 1de4b8e8accaa792e0934fbd9f933a6a
       auth
         keyname: "hub-Spoke1-hub-AUTH-SHA1-20"
         key: 1fa244d3971b4d4df59b8d7b3655a1b77f8e65af
   
   
   vd: root/0
   name: spoke1_0
   addr: 172.16.200.1:500 -> 172.16.200.3:500
   
     phase2
     name: spoke1
     server: "KMS_server"
     spi: 8d568114
       enc
         keyname: "Spoke1-Spoke2-spoke2-ENC-AES-16"
         key: b01c534b11792b856c1b95c78c4cad91
       auth
         keyname: "Spoke1-Spoke2-spoke2-AUTH-SHA1-20"
         key: fe6a82177db6911b3203d1306969e5ddec8fd039
     spi: 471dfe2f
       enc
         keyname: "Spoke2-Spoke1-spoke2-ENC-AES-16"
         key: a6d6a25cd986860bcc502d58f32e99de
       auth
         keyname: "Spoke2-Spoke1-spoke2-AUTH-SHA1-20"
         key: 07d712156eaca28439fbe944e3a8c9af4c45166a

3. Verify the FortiGate (KMIP client) connection to the KMS server:

   Spoke1 # execute kmip locate KMS_server hub-Spoke1-hub-AUTH-SHA1-20
   Locating key 'hub-Spoke1-hub-AUTH-SHA1-20', jobid=1935521133
   Ret=0, jobid=1935521133
           Key ID: 2ba130bff7174ba7a237d7ea53611121383b132cf18a4fd183890ca196296cb4

4. Verify the IKE and KMIP debug messages on Spoke1 to confirm that when the KMS server is down during IPsec rekey, IPsec tunnel does not go down:

   Spoke1 # diagnose debug application ike -1
   Spoke1 # diagnose debug application kmipd -1
   
   ike 0:spoke1:543:580 rekey in progress for SPI 471dfe32
   ike 0:spoke1:543: sent IKE msg (CREATE_CHILD): 172.16.200.1:500->172.16.200.4:500, len=416, vrf=0, id=627aee1c2562d5e5/31d6fccbac9dae7b:00000003
   ike 0:spoke1:543: sent IKE msg (RETRANSMIT_CREATE_CHILD): 172.16.200.1:500->172.16.200.4:500, len=416, vrf=0, id=627aee1c2562d5e5/31d6fccbac9dae7b:00000003
   ike 0:spoke1:543: sent IKE msg (RETRANSMIT_CREATE_CHILD): 172.16.200.1:500->172.16.200.4:500, len=416, vrf=0, id=627aee1c2562d5e5/31d6fccbac9dae7b:00000003
   ike 0: comes 172.16.200.4:500->172.16.200.1:500,ifindex=19,vrf=0....
   ike 0: IKEv2 exchange=CREATE_CHILD_RESPONSE id=627aee1c2562d5e5/31d6fccbac9dae7b:00000003 len=192
   ike 0:spoke1:543: received create-child response
   ike 0:spoke1:543: initiator received CREATE_CHILD msg
   ike 0:spoke1:543:spoke1:580: found child SA SPI 471dfe34 state=3
   ike 0:spoke1:543: processing notify type KMS_KEYS_REUSE
   ...
   ike 0:spoke1:543:spoke1:580: IPsec SA dec spi 471dfe34 key 16:1DE4B8E8ACCAA792E0934FBD9F933A6A auth 20:1FA244D3971B4D4DF59B8D7B3655A1B77F8E65AF
   ike 0:spoke1:543:spoke1:580: IPsec SA enc spi 628d181b key 16:5DAD0D8D3568EAB7C3F259349DC64039 auth 20:E660F491B80B2CFDCDB0D737942BEA2E853DAC8D
   ike 0:spoke1:543:spoke1:580: added IPsec SA: SPIs=471dfe34/628d181b
   ike 0:spoke1:543:spoke1:580: scheduling rekeyed SPI 471dfe32 for deletion
   ike 0:spoke1:543:spoke1:580: rekey in progress, old SPI 471dfe32
   ...
   ike 0:spoke1_0:spoke1: sending kmip locate request: id=166 keyname=FGT80FTK22056585-FG200E4Q17904575-spoke2-ENC-AES-16
   ike 0:spoke1_0:spoke1: sending kmip locate request: id=167 keyname=FG200E4Q17904575-FGT80FTK22056585-spoke2-AUTH-SHA1-20
   ike 0:spoke1_0:spoke1: sending kmip locate request: id=168 keyname=FGT80FTK22056585-FG200E4Q17904575-spoke2-AUTH-SHA1-20
   ...
   __kmip_conn_connect()-489: Failed to connect KMIP server 'KMS_server', vfid-0, addr-172.16.200.221:5696
   ...
   __kmip_conn_connect()-489: Failed to connect KMIP server 'KMS_server', vfid-0, addr-172.16.200.222:5696
   ...
   __kmip_conn_connect()-489: Failed to connect KMIP server 'KMS_server', vfid-0, addr-172.16.200.223:5696
   __kmip_conn_pick_one_addr()-212: No more host to try.
   __kmip_conn_schedule_next_retry()-169: server-KMS_server, st=0, vfid-0
   ike 0:spoke1_0:spoke1: kmip req expired: id=165
   ike 0:spoke1_0:544:spoke1:581: KMS: rekey using old child_sa keys.
   ike 0:spoke1: schedule auto-negotiate
   ike 0:spoke1_0:544:spoke1:581: replay protection enabled
   ike 0:spoke1_0:544:spoke1:581: set sa life soft seconds=111.
   ike 0:spoke1_0:544:spoke1:581: set sa life hard seconds=120.
   ike 0:spoke1_0:544:spoke1:581: IPsec SA selectors #src=1 #dst=1
   ike 0:spoke1_0:544:spoke1:581: src 0 4 0:0.0.0.0/0.0.0.0:0
   ike 0:spoke1_0:544:spoke1:581: dst 0 4 0:0.0.0.0/0.0.0.0:0
   ike 0:spoke1_0:544:spoke1:581: add dynamic IPsec SA selectors
   ike 0:spoke1_0:544:spoke1:581: added dynamic IPsec SA proxyids, existing serial 1
   ike 0:spoke1_0:544:spoke1:581: add IPsec SA: SPIs=471dfe35/8d56811c