ZTNA troubleshooting and debugging commands
The following debug commands can be used to troubleshoot ZTNA issues:
|
Command |
Description |
|---|---|
|
# diagnose endpoint fctems test-connectivity <EMS> |
Verify FortiGate to FortiClient EMS connectivity. |
|
# execute fctems verify <EMS> |
Verify the FortiClient EMS’s certificate. |
|
# diagnose test application fcnacd 2 |
Dump the EMS connectivity information. |
|
# diagnose debug app fcnacd -1 # diagnose debug enable |
Run real-time FortiClient NAC daemon debugs. |
|
# diagnose endpoint ec-shm list <ip> <mac> <EMS_serial_number> <EMS_tenant_id> |
Show the endpoint record list. Optionally, add filters. |
|
# diagnose endpoint lls-comm send ztna find-uid <uid> <EMS_serial_number> <EMS_tenant_id> |
Query endpoints by client UID, EMS serial number, and EMS tenant ID. |
|
# diagnose endpoint lls-comm send ztna find-ip-vdom <ip> <vdom> |
Query endpoints by the client IP-VDOM pair. |
|
# diagnose wad dev query-by uid <uid> <EMS_serial_number> <EMS_tenant_id> |
Query from WAD diagnose command by UID, EMS serial number, and EMS tenant ID. |
|
# diagnose wad dev query-by ipv4 <ip> |
Query from WAD diagnose command by IP address. |
|
# diagnose firewall dynamic list |
List EMS security posture tags and all dynamic IP and MAC addresses. |
|
# diagnose test application fcnacd 7 # diagnose test application fcnacd 8 |
Check the FortiClient NAC daemon ZTNA and route cache. |
|
# diagnose wad worker policy list |
Display statistics associated with application gateway rules. |
|
# diagnose wad debug enable category all # diagnose wad debug enable level verbose # diagnose debug enable |
Run real-time WAD debugs. |
|
# diagnose debug reset |
Reset debugs when completed. |
|
# diagnose wad user list |
List the ZTNA/proxy users. |
|
# diagnose wad user clear <id> <ip> <vdom> |
Clear a single ZTNA/proxy user. |
|
# diagnose wad user clear |
Clear all ZTNA/proxy users. |
|
The WAD daemon handles proxy related processing. The FortiClient NAC daemon (fcnacd) handles FortiGate to EMS connectivity. |
Troubleshooting usage and output
-
Verify the FortiGate to EMS connectivity and EMS certificate:
# diagnose endpoint fctems test-connectivity WIN10-EMS Connection test was successful:
# execute fctems verify WIN10-EMS Server certificate already verified.
# diagnose test application fcnacd 2 EMS context status: FortiClient EMS number 1: name: WIN10-EMS confirmed: yes fetched-serial-number: FCTEMS0000109188 Websocket status: connected -
If fcnacd does not report the proper status, run real-time fcnacd debugs:
# diagnose debug app fcnacd -1 # diagnose debug enable
-
Verify the following information about an endpoint:
-
Network information
-
Registration information
-
Client certificate information
-
Device information
-
Vulnerability status
-
Relative position with the FortiGate
# diagnose endpoint ec-shm list 10.6.30.214 Record 0: IP Address = 10.6.30.214 MAC Address = 00:0c:29:ba:1e:61 MAC list = 00:0c:29:ba:1e:61;00:0c:29:ba:1e:6b; VDOM = root (0) EMS serial number: FCTEMS8821001322 EMS tenant id: 00000000000000000000000000000000 Client cert SN: 17FF6595600A1AF53B87627AB4EBEDD032593E64 Quarantined: no Online status: online Registration status: registered On-net status: on-net Gateway Interface: port2 FortiClient version: 7.0.0 AVDB version: 84.778 FortiClient app signature version: 18.43 FortiClient vulnerability scan engine version: 2.30 FortiClient UID: 5FCFA3ECDE4D478C911D9232EC9299FD Host Name: ADPC … Number of Routes: (1) Gateway Route #0: - IP:10.1.100.214, MAC: 00:0c:29:ba:1e:6b, Indirect: no - Interface:port2, VFID:0, SN: FG5H1E5819902474 online records: 1; offline records: 0; quarantined records: 0 -
-
Query the endpoint information, include security posture tags, by UID or IP address:
# diagnose endpoint lls-comm send ztna find-uid 5FCFA3ECDE4D478C911D9232EC9299FD FCTEMS8821001322 00000000000000000000000000000000 UID: 5FCFA3ECDE4D478C911D9232EC9299FD EMS Fabric ID: FCTEMS8821001322:00000000000000000000000000000000 status code:ok Domain: qa.wangd.com User: user1 Cert SN:17FF6595600A1AF53B87627AB4EBEDD032593E64 EMS SN: FCTEMS8821001322 Routes(1): - route[0]: IP=10.1.100.214, VDom=root Tags(3): - tag[0]: name=ZT_OS_WIN - tag[1]: name=all_registered_clients - tag[2]: name=Medium# diagnose endpoint lls-comm send ztna find-ip-vdom 10.1.100.214 root UID: 5FCFA3ECDE4D478C911D9232EC9299FD status code:ok Domain: qa.wangd.com User: user1 Cert SN:17FF6595600A1AF53B87627AB4EBEDD032593E64 EMS SN: FCTEMS8821001322 Routes(1): - route[0]: IP=10.1.100.214, VDom=root Tags(3): - tag[0]: name=ZT_OS_WIN - tag[1]: name=all_registered_clients - tag[2]: name=Medium -
Query endpoint information from WAD by UID or IP address:
# diagnose wad dev query-by uid 5FCFA3ECDE4D478C911D9232EC9299FD FCTEMS8821001322 00000000000000000000000000000000 Attr of type=0, length=32, value(ascii)=5FCFA3ECDE4D478C911D9232EC9299FD Attr of type=4, length=30, value(ascii)=MAC_FCTEMS8821001322_ZT_OS_WIN Attr of type=4, length=26, value(ascii)=FCTEMS8821001322_ZT_OS_WIN Attr of type=4, length=43, value(ascii)=MAC_FCTEMS8821001322_all_registered_clients Attr of type=4, length=39, value(ascii)=FCTEMS8821001322_all_registered_clients Attr of type=4, length=27, value(ascii)=MAC_FCTEMS8821001322_Medium Attr of type=4, length=23, value(ascii)=FCTEMS8821001322_Medium Attr of type=5, length=18, value(ascii)=FOSQA@qa.wangd.com Attr of type=6, length=40, value(ascii)=17FF6595600A1AF53B87627AB4EBEDD032593E64
# diagnose wad dev query-by ipv4 10.1.100.214 Attr of type=0, length=32, value(ascii)=5FCFA3ECDE4D478C911D9232EC9299FD Attr of type=4, length=30, value(ascii)=MAC_FCTEMS8821001322_ZT_OS_WIN Attr of type=4, length=26, value(ascii)=FCTEMS8821001322_ZT_OS_WIN Attr of type=4, length=43, value(ascii)=MAC_FCTEMS8821001322_all_registered_clients Attr of type=4, length=39, value(ascii)=FCTEMS8821001322_all_registered_clients Attr of type=4, length=27, value(ascii)=MAC_FCTEMS8821001322_Medium Attr of type=4, length=23, value(ascii)=FCTEMS8821001322_Medium Attr of type=5, length=18, value(ascii)=FOSQA@qa.wangd.com Attr of type=6, length=40, value(ascii)=17FF6595600A1AF53B87627AB4EBEDD032593E64
- List all the dynamic ZTNA IP and MAC addresses learned from EMS:
# diagnose firewall dynamic list List all dynamic addresses: FCTEMS0000109188_all_registered_clients: ID(51) ADDR(172.17.194.209) ADDR(192.168.40.8) … FCTEMS0000109188_Low: ID(78) ADDR(172.17.194.209) ADDR(192.168.40.8) … FCTEMS0000109188_Malicious-File-Detected: ID(190) ADDR(172.17.194.209) ADDR(192.168.40.8) … -
Check the FortiClient NAC daemon ZTNA and route cache:
# diagnose test application fcnacd 7 ZTNA Cache: -uid 5FCFA3ECDE4D478C911D9232EC9299FD: { "tags": [ "ZT_OS_WIN", "all_registered_clients", "Medium" ], "domain": "qa.wangd.com", "user_name": "user1", "client_cert_sn": "17FF6595600A1AF53B87627AB4EBEDD032593E64", "owner": "FOSQA@qa.wangd.com", "gateway_route_list": [ { "gateway_info": { "fgt_sn": "FG5H1E5819902474", "interface": "port2", "vdom": "root" }, "route_info": [ { "ip": "10.1.100.214", "mac": "00-0c-29-ba-1e-6b", "route_type": "direct" } ] } ], "ems_sn": "FCTEMS8821001322" }# diagnose test application fcnacd 8 IP-VfID Cache: IP: 10.1.100.206, vfid: 0, uid: 3DED29B54386416E9888F2DCBD2B9D21 IP: 10.1.100.214, vfid: 0, uid: 5FCFA3ECDE4D478C911D9232EC9299FD
-
Troubleshoot WAD with real-time debugs to understand how the proxy handled a client request:
# diagnose wad debug enable category all # diagnose wad debug enable level verbose # diagnose debug enable [0x7fbd7a46bb60] Received request from client: 10.10.10.20:56312 GET / HTTP/1.1 Host: 192.168.2.86:8443 Connection: keep-alive Cache-Control: max-age=0 Upgrade-Insecure-Requests: 1 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36 Edg/89.0.774.57 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9 Sec-Fetch-Site: none Sec-Fetch-Mode: navigate Sec-Fetch-User: ?1 Sec-Fetch-Dest: document Accept-Encoding: gzip, deflate, br Accept-Language: en-US,en;q=0.9 [p:29957][s:458767][r:1] wad_http_marker_uri(1269): path=/ len=1 [p:29957][s:458767][r:1] wad_http_parse_host(1641): host_len=17 [p:29957][s:458767][r:1] wad_http_parse_host(1677): len=12 [p:29957][s:458767][r:1] wad_http_parse_host(1686): len=4 [p:29957][s:458767][r:1] wad_http_str_canonicalize(2180): path=/ len=1 changes=0 [p:29957][s:458767][r:1] wad_http_str_canonicalize(2189): path=/ len=1 changes=0 [p:29957][s:458767][r:1] wad_http_normalize_uri(2232): host_len=12 path_len=1 query_len=0 [p:29957][s:458767][r:1] wad_vs_proxy_match_gwy(2244): 6:WIN2K16-P1: matching gwy with vhost(_def_virtual_host_) [p:29957][s:458767][r:1] wad_vs_proxy_match_vhost(2293): 6:WIN2K16-P1: matching vhost by: 192.168.2.86 [p:29957][s:458767][r:1] wad_vs_matcher_map_find(477): Empty matcher! [p:29957][s:458767][r:1] wad_vs_proxy_match_vhost(2296): 6:WIN2K16-P1: no host matched. [p:29957][s:458767][r:1] wad_vs_proxy_match_gwy(2263): 6:WIN2K16-P1: matching gwy by (/) with vhost(_def_virtual_host_). [p:29957][s:458767][r:1] wad_pattern_matcher_search(1210): pattern-match succ:/ [p:29957][s:458767][r:1] wad_vs_proxy_match_gwy(2271): 6:WIN2K16-P1: Matched gwy(1) type(https). [p:29957][s:458767][r:1] wad_http_vs_check_dst_ovrd(776): 6:WIN2K16-P1:1: Found server: 192.168.20.6:443 [p:29957][s:458767][r:1] wad_http_req_exec_act(9296): dst_addr_type=3 wc_nontp=0 sec_web=1 web_cache=0 req_bypass=0 [p:29957][s:458767][r:1] wad_http_req_check_policy(8117): starting policy matching(vs_pol= 1):10.10.10.20:56312->192.168.20.6:443 [p:29957][s:458767][r:1] wad_fw_addr_match_ap(1524): matching ap:WIN2K16(7) with vip addr:WIN2K16-P1(10) [p:29957][s:458767][r:1] wad_fw_addr_match_ap(1524): matching ap:WIN2K16-P1(10) with vip addr:WIN2K16-P1(10) [p:29957][s:458767][r:1] wad_http_req_policy_set(6811): match pid=29957 policy-id=2 vd=0 in_if=3, out_if=7 10.10.10.20:56312 -> 192.168.20.6:443 [p:29957][s:458767][r:1] wad_cifs_profile_init(93): CIFS Profile 0x7fbd7a5bf200 [] of type 0 created [p:29957][s:458767][r:1] wad_http_req_proc_policy(6622): web_cache(http/https=0/0, fwd_srv=<nil>. [p:29957][s:458767][r:1] wad_auth_inc_user_count(1668): increased user count, quota:128000, n_shared_user:2, vd_used: 2, vd_max: 0, vd_gurantee: 0 [p:29957][s:458767][r:1] __wad_fmem_open(563): fmem=0xaaee3e8, fmem_name='cmem 336 bucket', elm_sz=336, block_sz=73728, overhead=20, type=advanced [p:29957][s:458767][r:1] __wad_hauth_user_node_hold(2107): wad_hauth_user_node_alloc (1568): holding node 0x7fbd76d48060 mapping user_node:0x7fbd76d48060, user_ip:0x7fbd7a57b408(0), user:0x7fbd7a5cf420(0) [p:29957][s:458767][r:1] __wad_hauth_user_node_hold(2107): wad_user_node_stats_hold (483): holding node 0x7fbd76d48060 [p:29957][s:458767][r:1] __wad_hauth_user_node_hold(2107): wad_http_session_upd_user_node (4813): holding node 0x7fbd76d48060 [p:29957][s:458767][r:1] wad_http_req_proc_policy(6698): policy result:vf_id=0:0 sec_profile=0x7fbd7a5bef00 set_cookie=0 [p:29957][s:458767][r:1] wad_http_urlfilter_check(381): uri_norm=1 inval_host=0 inval_url=0 scan-hdr/body=1/0 url local=0 block=0 user-cat=0 allow=0 ftgd=0 keyword=0 wisp=0 [p:29957][s:458767][r:1] wad_http_req_proc_waf(1309): req=0x7fbd7a46bb60 ssl.deep_scan=1 proto=10 exempt=0 waf=(nil) body_len=0 ua=Chrome/89.0.4389.90 skip_scan=0 [p:29957][s:458767][r:1] wad_http_req_proc_antiphish(5376): Processing antiphish request [p:29957][s:458767][r:1] wad_http_req_proc_antiphish(5379): No profile [p:29957][s:458767][r:1] wad_http_connect_server(4696): http session 0x7fbd7a532ac8 req=0x7fbd7a46bb60 [p:29957][s:458767][r:1] wad_http_srv_still_good(4575): srv((nil)) nontp(0) dst_type(3) req: dst:192.168.20.6:443, proto:10) hcs: dst:N/A:0, proto:1)
Always reset the debugs after using them:
# diagnose debug reset