ZTNA troubleshooting and debugging commands
The following debug commands can be used to troubleshoot ZTNA issues:
|
Command |
Description |
|---|---|
|
# diagnose endpoint fctems test-connectivity <EMS> |
Verify FortiProxy to FortiClient EMS connectivity. |
|
# execute fctems verify <EMS> |
Verify the FortiClient EMS’s certificate. |
|
# diagnose test application fcnacd 2 |
Dump the EMS connectivity information. |
|
# diagnose debug app fcnacd -1 # diagnose debug enable |
Run real-time FortiClient NAC daemon debugs. |
|
# diagnose endpoint record list |
Show the endpoint record list. Optionally, add filters. |
|
# diagnose endpoint lls-comm send ztna find-uid <uid> <EMS_serial_number> <EMS_tenant_id> |
Query endpoints by client UID, EMS serial number, and EMS tenant ID. |
|
# diagnose endpoint lls-comm send ztna find-ip-vdom <ip> <vdom> |
Query endpoints by the client IP-VDOM pair. |
|
# diagnose wad dev query-by uid <uid> <EMS_serial_number> <EMS_tenant_id> |
Query from WAD diagnose command by UID, EMS serial number, and EMS tenant ID. |
|
# diagnose wad dev query-by ipv4 <ip> |
Query from WAD diagnose command by IP address. |
|
# diagnose firewall dynamic list
|
List EMS security posture tags and all dynamic IP and MAC addresses. |
|
# diagnose test application fcnacd 7 # diagnose test application fcnacd 8 |
Check the FortiClient NAC daemon ZTNA and route cache. |
|
# diagnose wad worker policy list |
Display statistics associated with application gateway rules. |
|
# diagnose wad debug enable category all # diagnose wad debug enable level verbose # diagnose debug enable |
Run real-time WAD debugs. |
|
# diagnose debug reset |
Reset debugs when completed |
|
The WAD daemon handles proxy related processing. The FortiClient NAC daemon (fcnacd) handles FortiProxy to EMS connectivity. |
Troubleshooting usage and output
-
Verify the FortiProxy to EMS connectivity and EMS certificate:
# diagnose endpoint fctems test-connectivity 1 Connection test was successful. # execute fctems verify 1 EMS already verified. # diagnose test application fcnacd 2 EMS context status: FortiClient EMS number 1: name(id): emstest(1) confirmed: yes fetched-serial-number: FCTEMS8823005021 fetched-tenant-id: 00000000000000000000000000000000 user-data: verified capabilities: true verified identity: true interface-selection-method: 0 verify-peer-method: 4 Websocket status: connected, oif: 0 -
If fcnacd does not report the proper status, run real-time fcnacd debugs:
# diagnose debug app fcnacd -1 # diagnose debug enable
-
Verify the following information about an endpoint:
-
Network information
-
Registration information
-
Client certificate information
-
Device information
-
Vulnerability status
-
Relative position with the FortiProxy
# dia end record list 10.120.1.26 Record #1: IP Address = 10.120.1.26 MAC Address = 00:0c:29:8d:33:5c MAC list = VDOM = root (0) EMS serial number: FCTEMS8823005021 EMS tenant id: 00000000000000000000000000000000 Client cert SN: 696348D6485176EECFEC967D9E762D5166CF00EB Public IP address: 207.102.138.19 Quarantined: no Online status: online Registration status: registered On-net status: on-net Gateway Interface: port1 FortiClient version: 7.2.4 AVDB version: 0.0 FortiClient app signature version: 0.0 FortiClient vulnerability scan engine version: 2.40 FortiClient UID: 0F37F57235724AD19FADAABB1B34AE6F Host Name: DESKTOP-CLIUU77 OS Type: WIN64 OS Version: Microsoft Windows 10 Professional Edition, 64-bit (build 19045) Host Description: Domain: qa.domaintest.com Last Login User: userc Owner: Host Model: VMware Virtual Platform Host Manufacturer: VMware, Inc. CPU Model: Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz Memory Size: 4095 AV Feature: 0 FW Feature: 0 WF Feature: 0 AS Feature: 0 VS Feature: 1 VN Feature: 1 Last vul message received time: N/A Last vul scanned time: N/A Last vul statistic: critical=0, high=0, medium=0, low=0, info=0 Avatar fingerprint: 05b9940c015425a375caafa28096d695be6e9ad2 Avatar source username: userc Avatar source email: Avatar source: OS Phone number: Number of Routes: (1) Gateway Route #0: - IP:10.120.1.26, MAC: 00:0c:29:8d:33:5c, VPN: no - Interface:port1, VFID:0, SN: FPXVULTM24000082 online records: 1; offline records: 0; quarantined records: 0; out-of-sync records: 0 -
-
Query endpoint information from WAD by UID or IP address:
# diagnose wad dev query-by uid 0F37F57235724AD19FADAABB1B34AE6F FCTEMS8823005021 00000000000000000000000000000000 Attr of type=0, length=83, value(ascii)=0F37F57235724AD19FADAABB1B34AE6F Attr of type=4, length=0, value(ascii)= Attr of type=6, length=1, value(ascii)=true Attr of type=5, length=40, value(ascii)=696348D6485176EECFEC967D9E762D5166CF00EB Attr of type=3, length=18, value(ascii)=EMS1_ZTNA_usergrp2 Attr of type=3, length=22, value(ascii)=MAC_EMS1_ZTNA_usergrp2 Attr of type=3, length=17, value(ascii)=EMS1_ZTNA_disk-en Attr of type=3, length=21, value(ascii)=MAC_EMS1_ZTNA_disk-en Attr of type=3, length=32, value(ascii)=EMS1_ZTNA_all_registered_clients Attr of type=3, length=36, value(ascii)=MAC_EMS1_ZTNA_all_registered_clients Attr of type=3, length=23, value(ascii)=EMS1_ZTNA_anti-virus-ok Attr of type=3, length=27, value(ascii)=MAC_EMS1_ZTNA_anti-virus-ok Attr of type=3, length=18, value(ascii)=EMS1_ZTNA_usergrp3 Attr of type=3, length=22, value(ascii)=MAC_EMS1_ZTNA_usergrp3 Response termination due to no more data # diagnose wad dev query-by ipv4 10.120.1.26 Attr of type=0, length=83, value(ascii)=0F37F57235724AD19FADAABB1B34AE6F Attr of type=4, length=0, value(ascii)= Attr of type=6, length=1, value(ascii)=true Attr of type=5, length=40, value(ascii)=696348D6485176EECFEC967D9E762D5166CF00EB Attr of type=3, length=18, value(ascii)=EMS1_ZTNA_usergrp2 Attr of type=3, length=22, value(ascii)=MAC_EMS1_ZTNA_usergrp2 Attr of type=3, length=17, value(ascii)=EMS1_ZTNA_disk-en Attr of type=3, length=21, value(ascii)=MAC_EMS1_ZTNA_disk-en Attr of type=3, length=32, value(ascii)=EMS1_ZTNA_all_registered_clients Attr of type=3, length=36, value(ascii)=MAC_EMS1_ZTNA_all_registered_clients Attr of type=3, length=23, value(ascii)=EMS1_ZTNA_anti-virus-ok Attr of type=3, length=27, value(ascii)=MAC_EMS1_ZTNA_anti-virus-ok Attr of type=3, length=18, value(ascii)=EMS1_ZTNA_usergrp3 Attr of type=3, length=22, value(ascii)=MAC_EMS1_ZTNA_usergrp3 Response termination due to no more data
- List all the dynamic ZTNA IP and MAC addresses learned from EMS:
# diagnose firewall dynamic list List all dynamic addresses: IP dynamic addresses in VDOM root(vfid: 0): CMDB name: FCTEMS_ALL_FORTICLOUD_SERVERS FCTEMS_ALL_FORTICLOUD_SERVERS: ID(239) ADDR(75.237.184.208) Total IP dynamic range blocks: 0. Total IP dynamic addresses: 1. CMDB name: EMS1_ZTNA_AntiVirus_ON TAG name: AntiVirus_ON EMS1_ZTNA_AntiVirus_ON: ID(109) Total IP dynamic range blocks: 0. Total IP dynamic addresses: 0. CMDB name: EMS1_ZTNA_anti-virus-software TAG name: anti-virus-software EMS1_ZTNA_anti-virus-software: ID(213) Total IP dynamic range blocks: 0. Total IP dynamic addresses: 0. CMDB name: EMS1_ZTNA_anti-virus-ok TAG name: anti-virus-ok EMS1_ZTNA_anti-virus-ok: ID(115) ADDR(10.120.1.26) Total IP dynamic range blocks: 0. Total IP dynamic addresses: 1. … -
Check the FortiClient NAC daemon ZTNA and route cache:
# diagnose test application fcnacd 7 ZTNA Cache V2: Entry #1: - UID: 0F37F57235724AD19FADAABB1B34AE6F - EMS Fabric ID: FCTEMS8823005021:00000000000000000000000000000000 - Domain: qa.domaintest.com - User: userc - Owner: - Certificate SN: 696348D6485176EECFEC967D9E762D5166CF00EB - online: true - Routes (1): -- Route #0: IP=10.120.1.26, vfid=0 - FWAddrNames (10): -- Name (#0): EMS1_ZTNA_usergrp2 -- Name (#1): MAC_EMS1_ZTNA_usergrp2 -- Name (#2): EMS1_ZTNA_disk-en -- Name (#3): MAC_EMS1_ZTNA_disk-en -- Name (#4): EMS1_ZTNA_all_registered_clients -- Name (#5): MAC_EMS1_ZTNA_all_registered_clients -- Name (#6): EMS1_ZTNA_anti-virus-ok -- Name (#7): MAC_EMS1_ZTNA_anti-virus-ok -- Name (#8): EMS1_ZTNA_usergrp3 -- Name (#9): MAC_EMS1_ZTNA_usergrp3 lls_idx_mask = 0x00000001,
-
Troubleshoot WAD with real-time debugs to understand how the proxy handled a client request:
# diagnose wad debug enable category all # diagnose wad debug enable level verbose # diagnose debug enable [I][p:1608][s:2079431665][r:7] wad_dump_http_request :2833 hreq=0x7fbfcc8cc228 Received request from client: 10.100.1.21:49974 GET /icons/ubuntu-logo.png HTTP/1.1 Host: 10.100.1.59:7831 user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0 accept: image/avif,image/webp,*/* accept-language: en-US,en;q=0.5 accept-encoding: gzip, deflate, br dnt: 1 sec-gpc: 1 referer: https://10.100.1.59:7831/ sec-fetch-dest: image sec-fetch-mode: no-cors sec-fetch-site: same-origin te: trailers [V][p:1608][s:2079431665][r:7] wad_http_marker_uri :1270 path=/icons/ubuntu-logo.png len=22 [V][p:1608][s:2079431665][r:7] wad_http_parse_host :1649 host_len=16 [V][p:1608][s:2079431665][r:7] wad_http_parse_host :1685 len=11 [V][p:1608][s:2079431665][r:7] wad_http_parse_host :1694 len=4 [I][p:1608][s:2079431665][r:7] wad_http_str_canonicalize :2200 enc=0 path=/icons/ubuntu-logo.png len=22 changes=0 [V][p:1608][s:2079431665][r:7] wad_http_normalize_uri :2432 host_len=11 path_len=22 query_len=0 [I][p:1608][s:2079431665][r:7] wad_vs_proxy_match_gwy :4251 3:ZTNA_server01: matching gwy with vhost(_def_virtual_host_) [V][p:1608][s:2079431665][r:7] wad_vs_proxy_match_vhost :4312 3:ZTNA_server01: matching vhost by: 10.100.1.59 [V][p:1608][s:2079431665][r:7] wad_vs_matcher_map_find :709 Empty matcher! [V][p:1608][s:2079431665][r:7] wad_vs_proxy_match_vhost :4315 3:ZTNA_server01: no host matched. [I][p:1608][s:2079431665][r:7] wad_vs_proxy_match_gwy :4270 3:ZTNA_server01: matching gwy by (/icons/ubuntu-logo.png) with vhost(_def_virtual_host_). [V][p:1608][s:2079431665][r:7] wad_pattern_matcher_search :1307 pattern-match succ: '/icons/ubuntu-logo.png' [I][p:1608][s:2079431665][r:7] wad_vs_proxy_match_gwy :4288 3:ZTNA_server01: Matched gwy(1) type(https). [I][p:1608][s:2079431665][r:7] wad_http_req_update_by_vs_server :1058 3:ZTNA_server01:1: hostname after rewrite: Host: 10.120.1.78 [I][p:1608][s:2079431665][r:7] wad_http_vs_check_dst_ovrd :1225 3:ZTNA_server01:1: Found server: 10.120.1.78:443 [V][p:1608][s:2079431665][r:7] wad_http_req_exec_act :15123 request(0x7fbfcc8cc228), intercept(pass), block(0) [V][p:1608][s:2079431665][r:7] wad_http_req_exec_act :15215 dst_addr_type=3 wc_nontp=0 sec_web=1 web_cache=0 req_bypass=1 [V][p:1608][s:2079431665][r:7] wad_http_req_vs_check_policy :13895 HTTP req=0x7fbfcc8cc228 out_intf=3, vwl=0 [V][p:1608][s:2079431665][r:7] wad_http_req_check_policy :13627 start match policy vd=0(ses_ctx:ct|Pvx|Mde|Hf|C|A1|Og) (10.120.1.209:6092@6->10.120.1.78:443@3) absUrl=0 [V][p:1608][s:2079431665][r:7] wad_fw_addr_match_l7_vip :1253 matching vip:ZTNA_server01(13) type:AP-HTTPS with vip addr:ZTNA_server01(13) [I][p:1608][s:2079431665][r:7] wad_url_filter_req_alloc :698 url_req=0x7fbfc8b56978 id=0 [I][p:1608][s:2079431665][r:7] wad_url_filter_rating_request :4146 hreq=0x7fbfcc8cc228 url_req=0x7fbfc8b56978 hostname=10.100.1.59, url=/icons/ubuntu-logo.png vd=root id=0 ses_ctx.dst=10.120.1.78 req.srv=10.120.1.78 [V][p:1608][s:2079431665][r:7] wad_url_fetch_cate2 :1535 host=10.100.1.59 ip=10.120.1.78 [I][p:1608][s:2079431665][r:7] wad_url_cate_dump_req_ctx :280 (fetch-done): req/wfp=1/0 cate: cate=255 webf=255 sslexempt=255 url/ip=0/0 done: bal=0,local/user/cache/ftgd/ia_cache=1/1/1/1/0 matched[url]: block/allow/user=0/0/0 ftgd=0 sub=0 log=0 invalid=0 [I][p:1608][s:2079431665][r:7] wad_http_policy_get_cate_info :272 get category right away [I][p:1608][s:2079431665][r:7] wad_url_filter_cancel :710 type=2 data=0x7fbfcc8cc228 url_req=0x7fbfc8b56978 id=0 [I][p:1608][s:2079431665][r:7] wad_http_policy_match_one :511 fw_pol_id=1(pol_ctx:th|Ac|7|=p) pflag:H|W|U|Ac asyn_info=1 [I][p:1608][s:2079431665][r:7] wad_fw_policy_async_match :7401 pol_ctx:th|Ac|7|=d [I][p:1608][s:2079431665][r:7] wad_http_req_policy_set :11734 match policy-id=1(pol_ctx:th|Ac|7|=d) vd=0(ses_ctx:ct|Pvx|Mde|Hf|C|A1|Og) (10.100.1.21:49974@6 -> 10.120.1.78:443@3) [V][p:1608][s:2079431665][r:7] wad_https_ap_pol_info_get :11542 policy info created, req=0x7fbfcc8cc228, ses_ctx=0x7fbfc8b9c570, info=0x7fbfc8bcfab0 [I][p:1608][s:2079431665][r:7] wad_http_req_proc_policy :11236 ses_ctx:ct|Pvx|Mde|Hf|C|A1|Og conn_srv=0 fwd_srv=<nil> [I][p:1608][s:2079431665][r:7] wad_http_req_proc_policy :11370 policy result:vf_id=0:0 sec_profile=0x7fbfc89ad048 set_cookie=0 [W][p:1608][s:2079431665][r:7] wad_fw_policy_async_match :7391 no policy to match. [V][p:1608][s:2079431665][r:7] wad_setup_shaping_policy :536 did not match any shaping policy [I][p:1608][s:2079431665][r:7] wad_http_urlfilter_check :392 uri_norm=1 inval_host=0 inval_url=0 scan-hdr/body=1/0 url local=0 block=0 user-cat=0 allow=0 ftgd=0 keyword=0 wisp=0 [I][p:1608][s:2079431665][r:7] wad_http_req_proc_waf :1338 req=0x7fbfcc8cc228 ssl.deep_scan=1 proto=10 exempt=0 waf=(nil) body_len=18446744073709551615 ua=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0 skip_scan=0 [V][p:1608][s:2079431665][r:7] wad_http_req_proc_antiphish :9091 No profile [V][p:1608][s:2079431665][r:7] wad_http_parse_auth_cookie :1421 cookie_parsed=0 strip=1 pid=1608 [I][p:1608][s:2079431665][r:7] wad_vs_ssl_cert_sni_check :12995 sni=10.120.1.78 len=11 name=(null): cannot find the matched SNI in the certificate CN or SAN. [I][p:1608][s:2079431665][r:7] wad_http_srvs_ssl_tnode_get_srv :811 found server with matched sent sni:10.120.1.78 even cert not match, use it [I][p:1608][s:2079431665][r:7] wad_http_srv_attach_req :461 [0x7fbfcc8cc228] Use old server0x7fbfcc8df280: 10.120.1.78:443 [V][p:1608][s:2079431665][r:7] wad_http_req_get_svr :10077 http session 0x7fbfc8d76208 req=0x7fbfcc8cc228 connected [V][p:1608][s:2079431665][r:7] wad_http_msg_start_setup_proc :2320 msg(0x7fbfcc8cc228) proc-setup started from: req_casb. [V][p:1608][s:2079431665][r:7] wad_http_def_proc_msg_plan :2282 msg(0x7fbfcc8cc228) setting up processor(req_casb) [V][p:1608][s:2079431665][r:7] wad_http_def_proc_msg_plan :2282 msg(0x7fbfcc8cc228) setting up processor(req_scan)
Always reset the debugs after using them:
# diagnose debug reset