ZTNA troubleshooting scenarios
This topic describes how to troubleshoot common FortiClient endpoint IP/MAC access control issues for the following topologies:
ZTNA access control
In this topology, FortiClient endpoints use an SSL encrypted connection to the FortiGate access proxy to access protected resources. FortiGate works with FortiClient EMS to use a combination of IP/MAC addresses and ZTNA tags to control FortiClient endpoint access to resources.
This section describes how to handle the following errors:
Invalid ZTNA certificate
When FortiClient attempts to access a server protected by ZTNA, an Invalid ZTNA certificate error is shown. This error often appears when the serial number for the ZTNA certificate differs between the endpoint and the FortiGate.
-
Check the serial number for the ZTNA certificate on the endpoint and the FortiGate:
-
On the endpoint, check the serial number for the certificate.
-
On the FortiGate, check the serial number for the client certificate by running the following command:
# diagnose endpoint record list
-
-
If the serial number for the ZTNA certificate differs between the endpoint and the FortiGate, and the serial number on the FortiGate is comprised of zeros, check the following:
-
For FortiClient, make sure that the endpoint is running FortiClient 7.0 or later. FortiClient versions earlier than 7.0 do not support ZTNA.
-
For FortiClient EMS, make sure that ZTNA is enabled. Check the profile on EMS and the endpoint’s summary information.
-
For licensing, make sure that you have a ZTNA agent license entitlement. Only some license types support ZTNA.
-
-
If the serial numbers still do not match, deregister FortiClient from EMS, and then connect FortiClient to EMS again to trigger a new certificate signing request.
ZTNA policy mismatch
In most cases, FortiGate denies incoming ZTNA requests because the endpoint FortiClient does not meet the tagging criteria configured in the ZTNA rule and is considered a policy mismatch.
-
On the FortiGate, look at the ZTNA event logs and the forwarded logs.
-
Run the following commands on the ZTNA server:
# diagnose wad debug enable category policy
# diagnose wad debug enable level verbose
# diagnose debug enable
The command output contains incoming ZTNA requests and the FortiGate process for matching the connection to a ZTNA rule.
-
Verify the zero trust tags for the endpoint:
-
On FortiClient, verify the applied tags. Click the avatar to view the zero trust tags.
-
On FortiClient EMS, verify the endpoint’s tags. Go to the endpoint list and click the endpoint.
-
On FortiGate, verify the tags using the following commands:
-
Display ZTNA cache data for all endpoints:
# diagnose test application fcnacd 7
-
Display ZTNA cache data for an individual endpoint:
# diagnose wad dev query-by uid <UID> <EMS S/N> <tenant ID>
-
-
-
If the tagging information differs between FortiGate and EMS, examine the EMS tag exchange communication between FortiGate and EMS by looking at the cmNotify and python logs in the debug diagnostics for EMS.
For more information about FortiClient EMS diagnostics, see Generate Diagnostic Log in the FortiClient EMS Administration Guide.
IP/MAC based access control
In the following ZTNA topology, FortiClient endpoints use VPN to access resources. FortiGate works with FortiClient EMS to use a combination of IP/MAC addresses and ZTNA tags to control FortiClient endpoint access to resources.
For more information, see ZTNA IP MAC based access control example.
ZTNA tag information missing on the FortiGate
If the IP address for the FortiClient endpoint is not associated with a ZTNA tag on the FortiGate, a firewall policy mismatch occurs, and the FortiGate denies network access to the FortiClient endpoint.
The following workflow summarizes how FortiGate retrieves the IP address and tags for the FortiClient endpoint to help you better understand how to troubleshoot the situation:
-
FortiClient establishes a VPN connection to the FortiGate.
-
FortiGate uses the API to pass FortiClient’s UUID and VPN IP address to FortiClient EMS.
-
FortiGate requests system information and tags from FortiClient based on the response from EMS.
Based on the workflow, start troubleshooting before the FortiClient endpoint attempts to establish a VPN connection to FortiGate. On FortiGate, run the following commands:
# diagnose debug application fcnacd -1
# diagnose debug console timestamp enable
# diagnose endpoint filter show-large-data yes
# diagnose debug enable
The following outputs illustrate how to examine the command output. The output can differ between environments. The outputs help illustrate how to understand the communication between FortiGate and FortiClient EMS.
In the following output, FortiGate’s VPN daemon sends FortiClient’s UUID and the VPN IP address to FortiClient EMS using the API. The NAC daemon makes the API call to send the details to FortiClient EMS:
2022-10-17 08:50:41 [fcems_call_vpn_client_gateway_call:1147] VPN act connect (UID: 3358095CFDCB414B9EDA49ADE79AF428, Interface: port1, IP: 10.212.134.200, VDom: root,
FortiGate-SN: FGVM02TM22018374) added to EMS FortiClientEMS(FCTEMS8821003330:00000000000000000000000000000000)
2022-10-17 08:50:41 [ec_ez_worker_base_prep_resolver:373] Outgoing interface index 0 for 2 (FortiClientEMS).
2022-10-17 08:50:41 [ec_ez_worker_prep_data_url:98] request (206):
"""
{"sn_list":["FGVM02TM22018374"],"uid_list":[{"uid":"3358095CFDCB414B9EDA49ADE79AF428","ip":"10.212.134.200","is_delete":false,"vdom":"root","interface":"port1","sn":"F
GVM02TM22018374"}],"is_snapshot":false}
"""
2022-10-17 08:50:41 [ec_ez_worker_prep_data_url:176] Full URL: https://172.31.200.183/api/v1/fgt/gateway_details/vpn
2022-10-17 08:50:41 [ec_ems_context_submit_work:498] Call submitted successfully.
obj-id: 7, desc: REST API to send updated regarding VPN updates., entry: api/v1/fgt/gateway_details/vpn.
2022-10-17 08:50:41 [ec_daemon_submit_sock_call:49] sent 244,244
2022-10-17 08:50:42 [_renew_resolver:219] called.
2022-10-17 08:50:42 [ec_ez_worker_process:347] Processing call for obj-id: 7, entry: "api/v1/fgt/gateway_details/vpn"
2022-10-17 08:50:42 [ec_ez_worker_process:366] reply:
"""
{"result": {"retval": 1, "message": "FortiGate VPN connection details updated successfully"}}
"""
The following example from the fcmNotify.log file on FortiClient EMS shows how FortiClient EMS interprets the information sent from FortiGate:
2022-10-26 11:59:37,817 DEBUG ems_logger 6 7 [VPN Gateway Details]: Request made with params: {'is_snapshot': False, 'sn_list': ['FG10E0TB20903081', 'FG10E0TB20903034'], 'uid_list': [{'uid': 'D997B2A7A78E4E6F832309FF97FC2215', 'vdom': 'root', 'interface': 'EXT', 'sn': 'FG10E0TB20903081', 'ip': '10.1.18.61', 'is_delete': False}]}.
2022-10-26 11:59:38,281 DEBUG ems_logger 6 7 [Sysinfo c44cc74b1185431491f71c133c097f00 Certificate user: FG10E0TB20903081]: Request with SN [FG10E0TB20903034,FG10E0TB20903081] success. Returned 1 endpoints. uid_offset: D997B2A7A78E4E6F832309FF97FC2215, updated_after: 2022-10-26 15:59:37.8237471, is_final: True
2022-10-26 11:59:38,543 DEBUG ems_logger 6 7 [UID-Tags e6ecc42c058e48b2b71cf7d65ecd432c Certificate user: FG10E0TB20903081]: Request with SN [FG10E0TB20903034,FG10E0TB20903081] success. uid_offset: D997B2A7A78E4E6F832309FF97FC2215, updated_after: 2022-10-26 15:59:37.8227461, is_final: True
FortiGate uses the information from FortiClient EMS to make a targeted API call to FortiClient EMS to retrieve both system information and tag information (with the means of uid_offset and updated_after parameters) for the endpoint. The following is the API call to retrieve the tags from FortiClient EMS:
https://172.31.200.182/api/v1/report/fct/uid_tags?sn_list[]=FGVM02TM22018374&updated_after=2022-10-17 15:59:37.8227461&uid_offset=3358095CFDCB414B9EDA49ADE79AF428
The following in an example of the API call and subsequent communication between FortiGate and FortiClient EMS to retrieve tags for the FortiClient endpoint IP address:
2022-10-17 08:50:42 [ec_ez_worker_base_prep_resolver:373] Outgoing interface index 0 for 2 (FortiClientEMS).
2022-10-17 08:50:42 [ec_ez_worker_prep_data_url:98] request (26):
"""
sn_list[]=FGVM02TM22018374
"""
2022-10-17 08:50:42 [ec_ez_worker_prep_data_url:176] Full URL: https://172.31.200.183/api/v1/report/fct/uid_tags?sn_list[]=FGVM02TM22018374
2022-10-17 08:50:42 [ec_ems_context_submit_work:498] Call submitted successfully.
obj-id: 13, desc: REST API to get updates of tags associated with FCT UID., entry: api/v1/report/fct/uid_tags.
2022-10-17 08:50:43 [ec_ez_worker_process:347] Processing call for obj-id: 12, entry: "api/v1/report/fct/tags"
2022-10-17 08:50:43 [ec_ez_worker_process:366] reply:
"""
{"result": {"retval": 1, "message": "Returned FCT incremental tags information."}, "data": {"tag_uid_offset": "F200BAC5-352C-41AD-9BC2-C6D177D391B1", "updated_after":"2022-10-17 15:52:20.4951668", "is_zipped": true, "is_final": true, "unzipped_size": 3508, "data": "eJzFl0tv4zYUhf9KoXVuwadIZsfnYBYTFEgwsygKQbGYVKgsGZKcJg3mv/c66SNAa04BF87GgERa59Mhe ... BLXtfB2IRdYii2Qx9/uI6+scMw/XrUzcMp/B3U5zwm"}}
"""
2022-10-17 08:50:43 [fcems_json_unzip:285] unzipped:
"""
{"command_version":2,"serial":"FCTEMS8821003330","device_type":"fortiems","commands":[{"command":"update","addresses":[{"uuid":"814CA385-A346-4028-91FE-06011FFBC8A1","tag_properties":{"name":"vul_enabled","type":"zero_trust"},"type":"ipblock","values":[]},{"uuid":"814CA385-A346-4028- ... -93B7-E15BB3007AEC","tag_properties":{"name":"FortiESNAC.exe","type":"zero_trust"}},{"uuid":"82DF3EC6-9D1B-4200-A3C6-366D9AFF4ED0","tag_properties":{"name":"IPSEC_Allowed","type":"zero_trust"}}]}]}
"""
Other useful CLI commands
Output the JSON-formatted list of FortiGate interfaces (gateways) with IP and MAC addresses. This is the list that FortiGate sends to EMS so that EMS can identify the endpoints that are directly connected to the firewall:
# diagnose endpoint fctems json gateway-mac-request
Makes EMS execute API calls to the EMS API endpoints on demand:
# diagnose test application fcnacd 5
Send the gateway list to EMS on demand. It could be useful to execute diagnose test application fcnacd 5 right after command during troubleshooting, as EMS will have an updated list of firewall interfaces:
# diagnose test application fcnacd 99
For more commands, see ZTNA troubleshooting and debugging commands.