ZTNA troubleshooting scenarios

This topic describes how to troubleshoot common FortiClient endpoint IP/MAC access control issues for the following topologies:

• ZTNA access control

• IP/MAC based access control

ZTNA access control

In this topology, FortiClient endpoints use an SSL encrypted connection to the FortiProxy application gateway to access protected resources. FortiProxy works with FortiClient EMS to use a combination of IP/MAC addresses and security posture tags to control FortiClient endpoint access to resources.

This section describes how to handle the following errors:

• Invalid ZTNA certificate

• ZTNA policy mismatch

Invalid ZTNA certificate

When FortiClient attempts to access a server protected by ZTNA, an Invalid ZTNA certificate error is shown. This error often appears when the serial number for the ZTNA certificate differs between the endpoint and the FortiProxy.

1. Check the serial number for the ZTNA certificate on the endpoint and the FortiProxy:

   1. On the endpoint, check the serial number for the certificate.

   2. On the FortiProxy, check the serial number for the client certificate by running the following command:

      # diagnose endpoint record list

2. If the serial number for the ZTNA certificate differs between the endpoint and the FortiProxy, and the serial number on the FortiProxy is comprised of zeros, check the following:

   1. For FortiClient, make sure that the endpoint is running FortiClient 7.0 or later. FortiClient versions earlier than 7.0 do not support ZTNA.

   2. For FortiClient EMS, make sure that ZTNA is enabled. Check the profile on EMS and the endpoint’s summary information.

   3. For licensing, make sure that you have a ZTNA agent license entitlement. Only some license types support ZTNA.

3. If the serial numbers still do not match, deregister FortiClient from EMS, and then connect FortiClient to EMS again to trigger a new certificate signing request.

ZTNA policy mismatch

In most cases, FortiProxy denies incoming ZTNA requests because the endpoint FortiClient does not meet the tagging criteria configured in the ZTNA rule and is considered a policy mismatch.

1. On the FortiProxy, look at the ZTNA event logs and the forwarded logs.

2. Run the following commands on the ZTNA server:

   # diagnose wad debug enable category policy

   # diagnose wad debug enable level verbose

   # diagnose debug enable

   The command output contains incoming ZTNA requests and the FortiProxy process for matching the connection to a ZTNA rule.

3. Verify the zero trust tags for the endpoint:

   • On FortiClient, verify the applied tags. Click the avatar to view the zero trust tags.

   • On FortiClient EMS, verify the endpoint’s tags. Go to the endpoint list and click the endpoint.

   • On FortiProxy, verify the tags using the following commands:

     • Display ZTNA cache data for all endpoints:

       # diagnose test application fcnacd 7

     • Display ZTNA cache data for an individual endpoint:

       # diagnose wad dev query-by uid <UID> <EMS S/N> <tenant ID>

4. If the tagging information differs between FortiProxy and EMS, examine the EMS tag exchange communication between FortiProxy and EMS by looking at the cmNotify and python logs in the debug diagnostics for EMS.

   For more information about FortiClient EMS diagnostics, see Generate Diagnostic Log in the FortiClient EMS Administration Guide.

IP/MAC based access control

In the following ZTNA topology, FortiClient endpoints use VPN to access resources. FortiProxy works with FortiClient EMS to use a combination of IP/MAC addresses and security posture tags to control FortiClient endpoint access to resources.

Security posture tag information missing on the FortiProxy

If the IP address for the FortiClient endpoint is not associated with a security posture tag on the FortiProxy, a firewall policy mismatch occurs, and the FortiProxy denies network access to the FortiClient endpoint.

The following workflow summarizes how FortiProxy retrieves the IP address and tags for the FortiClient endpoint to help you better understand how to troubleshoot the situation:

1. FortiClient establishes a VPN connection to the FortiProxy.

2. FortiProxy uses the API to pass FortiClient’s UUID and VPN IP address to FortiClient EMS.

3. FortiProxy requests system information and tags from FortiClient based on the response from EMS.

Based on the workflow, start troubleshooting before the FortiClient endpoint attempts to establish a VPN connection to FortiProxy. On FortiProxy, run the following commands:

# diagnose debug application fcnacd -1

# diagnose debug console timestamp enable

# diagnose endpoint filter show-large-data yes

# diagnose debug enable

The following outputs illustrate how to examine the command output. The output can differ between environments. The outputs help illustrate how to understand the communication between FortiProxy and FortiClient EMS.

In the following output, FortiProxy’s VPN daemon sends FortiClient’s UUID and the VPN IP address to FortiClient EMS using the API. The NAC daemon makes the API call to send the details to FortiClient EMS:

2024-07-11 10:57:36 [ec_daemon_submit_sock_call:49] sent 244,244
[fcems_call_vpn_client_gateway_call:1151] VPN act connect (UID: 70A5C5FABBE64A9B98B6DDA3FE8AC794, Interface: port1, IP: 10.212.134.200, VDom: root, FortiProxy-SN: FPXVULTM24000083) added to EMS emstest(FCTEMS8823005021:00000000000000000000000000000000)
2024-07-11 10:57:36 [ec_ez_worker_base_prep_resolver:374] Outgoing interface index 0 for 1 (emstest).
2024-07-11 10:57:36 [ec_ez_worker_prep_data_url:98] request (206):
"""
{"sn_list":["FPXVULTM24000083"],"uid_list":[{"uid":"70A5C5FABBE64A9B98B6DDA3FE8AC794","ip":"10.212.134.200","is_delete":false,"vdom":"root","interface":"port1","sn":"FPXVULTM24000083"}],"is_snapshot":false}
"""
2024-07-11 10:57:36 [ec_ez_worker_prep_data_url:184] Full URL: https://10.120.1.24/api/v1/fgt/gateway_details/vpn
2024-07-11 10:57:36 [ec_ez_worker_base_prep_ssl:423] verify peer method: 4, current ssl_cb: 0x6ad220, new ssl_cb: 0x6ad220
2024-07-11 10:57:36 [ec_ems_context_submit_work:519] Call submitted successfully.
    obj-id: 7, desc: REST API to send updated regarding VPN updates., entry: api/v1/fgt/gateway_details/vpn.
2024-07-11 10:57:36 [__get_ec_fctems_certificate_info:431] ems cert ca_cn: C = US, ST = California, L = Sunnyvale, O = Fortinet, OU = Certificate Authority, CN = support, emailAddress = support@fortinet.comFCTEMS8823005021
2024-07-11 10:57:36 [__get_ec_fctems_certificate_info:432] ems cert fingerprint: 1F:17:09:BF:F3:C9:E3:3A:80:D6:30:91:36:C1:0B:76:AE:38:1B:C9:F8:B8:B2:0C:C1:28:04:A1:AF:77:85:96:7C:39:2E:52:4A:D9:C4:00:A0:7E:0C:C3:53:68:42:49:F9:3F:41:8F:47:29:1B:81:F4:26:F9:5C:61:DF:E6:EF
2024-07-11 10:57:36 [__match_server_cert_key:487] verify_peer_method: 4
2024-07-11 10:57:36 [__match_server_cert_key:505] ret=1
2024-07-11 10:57:36 [ec_ez_worker_process:368] Processing call for obj-id: 7, entry: "api/v1/fgt/gateway_details/vpn"
2024-07-11 10:57:36 [ec_ez_worker_process:387] reply:
"""
{"result": {"retval": 1, "message": "FortiGate VPN connection details updated successfully"}, "data": {"rejected_uid_list": []}}
"""
2024-07-11 10:57:36 [_update_obj_stats:336] Storing (7, emstest, 0)
2024-07-11 10:57:36 [ec_ez_worker_process:475] Call completed successfully.
    obj-id: 7, desc: "REST API to send updated regarding VPN updates.", entry: "api/v1/fgt/gateway_details/vpn".
2024-07-11 10:57:38 [__ws_writefunction_once:963] received frame: opcode:TEXT(1), fin:1, mask: 0, len: 82.
2024-07-11 10:57:38 [fcems_ws_on_text:369] WebSocket Recv(83): {"message_type":"notification","message_body":{"notify": {"tags":{"value":true}}}}
2024-07-11 10:57:38 [ec_ems_context_update_call:393] Update didn't constitute a scheduled call for obj-id 12
2024-07-11 10:57:38 [ec_ems_context_update_call:393] Update didn't constitute a scheduled call for obj-id 13
2024-07-11 10:57:38 [__ws_writefunction_once:963] received frame: opcode:TEXT(1), fin:1, mask: 0, len: 85.
2024-07-11 10:57:38 [fcems_ws_on_text:369] WebSocket Recv(86): {"message_type":"notification","message_body":{"notify": {"sysinfo":{"value":true}}}}
2024-07-11 10:57:38 [ec_ems_context_update_call:393] Update didn't constitute a scheduled call for obj-id 8
2024-07-11 10:57:38 [ec_ez_worker_base_prep_resolver:374] Outgoing interface index 0 for 1 (emstest).
2024-07-11 10:57:38 [ec_ez_worker_prep_data_url:98] request (142):
"""
sn_list[]=FPXVULTM24000083&updated_after=2024-07-11%2017%3A57%3A41%2E4804469&tag_uid_offset=9C8AF06C-8585-4572-ACDC-597E6AA1B6CD&send_mac=true
"""
2024-07-11 10:57:38 [ec_ez_worker_prep_data_url:184] Full URL: https://10.120.1.24/api/v1/report/fct/tags?sn_list[]=FPXVULTM24000083&updated_after=2024-07-11%2017%3A57%3A41%2E4804469&tag_uid_offset=9C8AF06C-8585-4572-ACDC-597E6AA1B6CD&send_mac=true
2024-07-11 10:57:38 [ec_ez_worker_base_prep_ssl:423] verify peer method: 4, current ssl_cb: 0x6ad220, new ssl_cb: 0x6ad220
2024-07-11 10:57:38 [ec_ems_context_submit_work:519] Call submitted successfully.
    obj-id: 12, desc: REST API to get updates of tag endpoints., entry: api/v1/report/fct/tags.
2024-07-11 10:57:38 [ec_ez_worker_base_prep_resolver:374] Outgoing interface index 0 for 1 (emstest).
2024-07-11 10:57:38 [ec_ez_worker_prep_data_url:98] request (120):
"""
sn_list[]=FPXVULTM24000083&updated_after=2024-07-11%2017%3A57%3A41%2E4774466&uid_offset=70A5C5FABBE64A9B98B6DDA3FE8AC794
"""
2024-07-11 10:57:38 [ec_ez_worker_prep_data_url:184] Full URL: https://10.120.1.24/api/v1/report/fct/uid_tags?sn_list[]=FPXVULTM24000083&updated_after=2024-07-11%2017%3A57%3A41%2E4774466&uid_offset=70A5C5FABBE64A9B98B6DDA3FE8AC794
2024-07-11 10:57:38 [ec_ez_worker_base_prep_ssl:423] verify peer method: 4, current ssl_cb: 0x6ad220, new ssl_cb: 0x6ad220
2024-07-11 10:57:38 [ec_ems_context_submit_work:519] Call submitted successfully.
    obj-id: 13, desc: REST API to get updates of tags associated with FCT UID., entry: api/v1/report/fct/uid_tags.
2024-07-11 10:57:38 [__get_ec_fctems_certificate_info:431] ems cert ca_cn: C = US, ST = California, L = Sunnyvale, O = Fortinet, OU = Certificate Authority, CN = support, emailAddress = support@fortinet.comFCTEMS8823005021
2024-07-11 10:57:38 [__get_ec_fctems_certificate_info:432] ems cert fingerprint: 1F:17:09:BF:F3:C9:E3:3A:80:D6:30:91:36:C1:0B:76:AE:38:1B:C9:F8:B8:B2:0C:C1:28:04:A1:AF:77:85:96:7C:39:2E:52:4A:D9:C4:00:A0:7E:0C:C3:53:68:42:49:F9:3F:41:8F:47:29:1B:81:F4:26:F9:5C:61:DF:E6:EF
2024-07-11 10:57:38 [__match_server_cert_key:487] verify_peer_method: 4
2024-07-11 10:57:38 [__match_server_cert_key:505] ret=1
2024-07-11 10:57:38 [__get_ec_fctems_certificate_info:431] ems cert ca_cn: C = US, ST = California, L = Sunnyvale, O = Fortinet, OU = Certificate Authority, CN = support, emailAddress = support@fortinet.comFCTEMS8823005021
2024-07-11 10:57:38 [__get_ec_fctems_certificate_info:432] ems cert fingerprint: 1F:17:09:BF:F3:C9:E3:3A:80:D6:30:91:36:C1:0B:76:AE:38:1B:C9:F8:B8:B2:0C:C1:28:04:A1:AF:77:85:96:7C:39:2E:52:4A:D9:C4:00:A0:7E:0C:C3:53:68:42:49:F9:3F:41:8F:47:29:1B:81:F4:26:F9:5C:61:DF:E6:EF
2024-07-11 10:57:38 [__match_server_cert_key:487] verify_peer_method: 4
2024-07-11 10:57:38 [__match_server_cert_key:505] ret=1
2024-07-11 10:57:38 [ec_ez_worker_process:368] Processing call for obj-id: 13, entry: "api/v1/report/fct/uid_tags"
2024-07-11 10:57:38 [ec_ez_worker_process:387] reply:
"""
{"result": {"retval": 1, "message": null}, "data": {"uid_offset": "70A5C5FABBE64A9B98B6DDA3FE8AC794", "updated_after": "2024-07-11 17:57:41.4774466", "is_final": true, "data": "eJyrVirNTIkvSUyPz8ksLilWsqqurQUAUUEH2g==", "is_zipped": true, "unzipped_size": 20}}
"""
2024-07-11 10:57:38 [fcems_json_unzip:289] unzipped:
"""
{"uid_tag_lists":{}}
"""
2024-07-11 10:57:38 [_update_obj_stats:336] Storing (13, emstest, 0)
2024-07-11 10:57:38 [ec_ez_worker_process:475] Call completed successfully.
    obj-id: 13, desc: "REST API to get updates of tags associated with FCT UID.", entry: "api/v1/report/fct/uid_tags".
2024-07-11 10:57:38 [ec_ez_worker_base_prep_resolver:374] Outgoing interface index 0 for 1 (emstest).
2024-07-11 10:57:38 [ec_ez_worker_prep_data_url:98] request (120):
"""
sn_list[]=FPXVULTM24000083&updated_after=2024-07-11%2017%3A57%3A41%2E4614463&uid_offset=70A5C5FABBE64A9B98B6DDA3FE8AC794
"""
2024-07-11 10:57:38 [ec_ez_worker_prep_data_url:184] Full URL: https://10.120.1.24/api/v1/report/fct/sysinfo?sn_list[]=FPXVULTM24000083&updated_after=2024-07-11%2017%3A57%3A41%2E4614463&uid_offset=70A5C5FABBE64A9B98B6DDA3FE8AC794
2024-07-11 10:57:38 [ec_ez_worker_base_prep_ssl:423] verify peer method: 4, current ssl_cb: 0x6ad220, new ssl_cb: 0x6ad220
2024-07-11 10:57:38 [ec_ems_context_submit_work:519] Call submitted successfully.
    obj-id: 8, desc: REST API to get updates about system info., entry: api/v1/report/fct/sysinfo.
2024-07-11 10:57:38 [__get_ec_fctems_certificate_info:431] ems cert ca_cn: C = US, ST = California, L = Sunnyvale, O = Fortinet, OU = Certificate Authority, CN = support, emailAddress = support@fortinet.comFCTEMS8823005021
2024-07-11 10:57:38 [__get_ec_fctems_certificate_info:432] ems cert fingerprint: 1F:17:09:BF:F3:C9:E3:3A:80:D6:30:91:36:C1:0B:76:AE:38:1B:C9:F8:B8:B2:0C:C1:28:04:A1:AF:77:85:96:7C:39:2E:52:4A:D9:C4:00:A0:7E:0C:C3:53:68:42:49:F9:3F:41:8F:47:29:1B:81:F4:26:F9:5C:61:DF:E6:EF
2024-07-11 10:57:38 [__match_server_cert_key:487] verify_peer_method: 4
2024-07-11 10:57:38 [__match_server_cert_key:505] ret=1
2024-07-11 10:57:38 [ec_ez_worker_process:368] Processing call for obj-id: 8, entry: "api/v1/report/fct/sysinfo"
2024-07-11 10:57:38 [ec_ez_worker_process:387] reply:
"""
{"result": {"retval": 1, "message": null}, "data": {"uid_offset": "70A5C5FABBE64A9B98B6DDA3FE8AC794", "updated_after": "2024-07-11 17:57:52.3553944", "is_final": true, "data": "eJyFVNtu4zYQ/RVCT7uAZJAUqYuf6mubttmkTfYCFIFAi5SXgEyqFGXDG/jfO/StxrZpAQsWzxkORzPn8DXK8YTP+HIynS4yNimnZTHN5vNJulwUk1lesmj8GvX7XpvGVkMnhVeV1xsVjSOKKUtwnhCCSD7m8KOjlPO0ZCyKo/Wuqq0xqvbamsrvO9iC46ixzuu61cr4SstoTDABVHRd1et1tVUuGv9Bi7ig/AXg7Q1KYnyClLlAeUwzwJraXwF2jNoO7U1YGlPAQjE+Gns3qLBotYGKGtH2sNR95dRa9145Ja/on4NwwngI/BuD891gjDbrKwSHmaqvhfkHEwrrDbRqOXsuMMYFYdCzglHoj+4AJ3hEKDyjNAWoG1atrqsjQ3E+IpiOSFqMSAnkRtQAY5zgOqFlkpMkLROSA/NVOLkTTp2O+nQf3hOeISZRUyJcoDxFrEZcIpolXKFSoqxBQqIUo5ygFGJkSGN7b8RxsvPF0y/PD4/J04fpbz9jdiarjTBDI2o/uNDW80kxujP16BpipWqvXB4TIOpuAOTOeNW++/09+qKsCf+zx49owROaQRlbin5AdJThH3/6Fr5Vbazbw6aClBjWtj9NMrrXtbO9bTz6rI20ux4RjB6dbVTfg8xEixZSB8HFKGPJSnv0bjXoViLIw/j7U6qTFqPPdx+y8GlDr1x1/vD1YKWFAYJ8nR26C/zgvyqHFkZ2VhvfR0EGwgtXNTBt5ToHaBgOX5UlwzUmnFEu0pzXQjSCFrjMZFbylcpUKWSY/tkBtXJnhTBC5jOWkhlb4oJOwJHzokwzvFwuspxxPmUZSctiAnsbJcIIKrGFfdr0XrStkjdEs3uD2HbmLaZ/g9g13xOg6WNPK2PBvhkvQMtGagdOb/cXz984pllfTPD45dPHX5/vKcPBC0Hxa7hOdmJfQf+UA2mFZndwQwTZbKXdwNJZ628iv7MBXSVUJHl2EwGDgzuqBTOD9V9vjmhsuMv+q5z/K+MQR6fsp2SQ/V9c/LZRT3vP8jt1LDq8HF7OGrxUCO9bfazh4Sk6HA5/ATr0vWU=", "is_zipped": true, "unzipped_size": 1444}}
"""
2024-07-11 10:57:38 [fcems_json_unzip:289] unzipped:
"""
{"70A5C5FABBE64A9B98B6DDA3FE8AC794":{"sysinfo_update_time":"2024-07-11 17:57:52.3553944","gw_connection_type":0,"forticlient_id":1010,"app_sig_ver":[28,825],"av_sig_ver":[1,0],"av_eng_ver":[7,26],"fct_ver":[7,4,0],"vul_eng_ver":[3,2],"onnet":true,"online":false,"is_registered":false,"quarantined":false,"av_running":false,"vuln_scan_running":false,"fct_sn":"FCT8000814024842","ip":"10.120.1.33","public_ip":"207.102.138.19","mac":"00-0c-29-71-39-17","hardware_sn":"VMware-56 4d f9 08 73 4c 5d 26-5e 9d 6f ad 30 71 39 0d","hostname":"DESKTOP-SNBQJ04","host_manufacturer":"VMware, Inc.","host_model":"VMware7,1","cpu":"Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz","memory":"8190","os_ver":"Microsoft Windows 10 Professional Edition, 64-bit (build 19045)","os_type":"WIN64","user_name":"guodong","group_name":"Other Endpoints","avatar_fingerprint":"05b9940c015425a375caafa28096d695be6e9ad2","client_cert_sn":"411DC431C4F082AC5FD89360FFE67455B461398A","feature_av":"installed","feature_fw":"installed","feature_vpn":"installed","feature_vs":"installed","feature_wf":"installed","fct_build_no":1658,"indirectly_connected":false,"fgt_sn":"FPXVULTM24000083","gateway_interface":"port1","vdom":"root","gateway_mac":"00-0c-29-2b-2a-76","gateway_route_list":[{"gateway_info":{"fgt_sn":"FPXVULTM24000083","interface":"port1","vdom":"root"},"route_info":[{"ip":"10.120.1.33","mac":"00-0c-29-71-39-17","route_type":"direct"}]}],"user_info":{"service":"OS"}}}
"""
2024-07-11 10:57:38 [ec_rec_add:1133] called (FTCL UID 70A5C5FABBE64A9B98B6DDA3FE8AC794).
2024-07-11 10:57:38 [ec_avatar_update:440] called (FTCL UID 70A5C5FABBE64A9B98B6DDA3FE8AC794).
2024-07-11 10:57:38 [ec_record_del_user_store:1649] deleting user store for user guodong, vfid:0
2024-07-11 10:57:38 [_update_obj_stats:336] Storing (8, emstest, 0)
2024-07-11 10:57:38 [ec_ez_worker_process:475] Call completed successfully.
    obj-id: 8, desc: "REST API to get updates about system info.", entry: "api/v1/report/fct/sysinfo".
2024-07-11 10:57:39 [ec_ez_worker_process:368] Processing call for obj-id: 12, entry: "api/v1/report/fct/tags"
2024-07-11 10:57:39 [ec_ez_worker_process:387] reply:
"""
{"result":{"retval":1,"message":"Returned FCT incremental tags information"},"data":{"tag_uid_offset":"9C8AF06C-8585-4572-ACDC-597E6AA1B6CD","updated_after":"2024-07-11 17:57:41.4804469","is_final":true,"data":"eJyqVkrOz81NzEuJL0stKs7Mz1OyMtJRKk4tykzMUbJScnMOcfUNtrAwMjYwMDUwMlTSUUpJLctMTo0vqSxIVbJSSssvKslMzS1WqgUEAAD//4xEGAI=","is_zipped":true,"unzipped_size":74}}
"""
2024-07-11 10:57:39 [fcems_json_unzip:289] unzipped:
"""
{"command_version":2,"serial":"FCTEMS8823005021","device_type":"fortiems"}
"""
2024-07-11 10:57:39 [_update_obj_stats:336] Storing (12, emstest, 0)
2024-07-11 10:57:39 [ec_ez_worker_process:475] Call completed successfully.
    obj-id: 12, desc: "REST API to get updates of tag endpoints.", entry: "api/v1/report/fct/tags".

2024-07-11 10:57:49 [__ws_writefunction_once:963] received frame: opcode:TEXT(1), fin:1, mask: 0, len: 84.
2024-07-11 10:57:49 [fcems_ws_on_text:369] WebSocket Recv(85): {"message_type":"notification","message_body":{"notify": {"avatar":{"value":true}}}}

Other useful CLI commands

Output the JSON-formatted list of FortiProxy interfaces (gateways) with IP and MAC addresses. This is the list that FortiProxy sends to EMS so that EMS can identify the endpoints that are directly connected to the firewall:

# diagnose endpoint fctems json gateway-mac-request

Makes EMS execute API calls to the EMS API endpoints on demand:

# diagnose test application fcnacd 5

Send the gateway list to EMS on demand. It could be useful to execute diagnose test application fcnacd 5 right after command during troubleshooting, as EMS will have an updated list of firewall interfaces:

# diagnose test application fcnacd 99

For more commands, see ZTNA troubleshooting and debugging commands.