Fortinet Document Library

Version:

Version:

Version:

Version:


Table of Contents

Administration Guide

Download PDF
Copy Link

ZTNA troubleshooting and debugging

The following debug commands can be used to troubleshoot ZTNA issues:

Command

Description

# diagnose endpoint fctems test-connectivity <EMS>

Verify FortiGate to FortiClient EMS connectivity.

# execute fctems verify <EMS>

Verify the FortiClient EMS’s certificate.

# diagnose test application fcnacd 2

Dump the EMS connectivity information.

# diagnose debug app fcnacd -1

# diagnose debug enable

Run real-time FortiClient NAC daemon debugs.

# diagnose endpoint record list <ip>

 

Show the endpoint record list. Optionally, filter by the endpoint IP address.

# diagnose endpoint lls-comm send ztna find-uid <uid>

Query endpoints by client UID.

# diagnose endpoint lls-comm send ztna find-ip-vdom <ip> <vdom>

Query endpoints by the client IP-VDOM pair.

# diagnose wad dev query-by uid <uid>

Query from WAD diagnose command by UID.

# diagnose wad dev query-by ipv4 <ip>

Query from WAD diagnose command by IP address.

# diagnose firewall dynamic list

 

List EMS ZTNA tags and all dynamic IP and MAC addresses.

# diagnose test application fcnacd 7

# diagnose test application fcnacd 8

Check the FortiClient NAC daemon ZTNA and route cache.

# diagnose wad worker policy list

Display statistics associated with access proxy rules.

# diagnose wad debug enable category all

# diagnose wad debug enable level verbose

# diagnose debug enable

Run real-time WAD debugs.

# diagnose debug reset

Reset debugs when completed

Note

The WAD daemon handles proxy related processing. The FortiClient NAC daemon (fcnacd) handles FortiGate to EMS connectivity.

Troubleshooting usage and output

  1. Verify the FortiGate to EMS connectivity and EMS certificate:

    # diagnose endpoint fctems test-connectivity WIN10-EMS
    Connection test was successful:
    
    # execute fctems verify WIN10-EMS
    Server certificate already verified.
    
    # diagnose test application fcnacd 2
    EMS context status:
    FortiClient EMS number 1:
            name: WIN10-EMS confirmed: yes
            fetched-serial-number: FCTEMS0000109188
    Websocket status: connected
    
  2. If fcnacd does not report the proper status, run real-time fcnacd debugs:

    # diagnose debug app fcnacd -1
    # diagnose debug enable
    
  3. Verify the following information about an endpoint:

    • Network information

    • Registration information

    • Client certificate information

    • Device information

    • Vulnerability status

    • Relative position with the FortiGate

    # diagnose endpoint record list 10.6.30.214
    Record #1:
                    IP Address = 10.6.30.214
                    MAC Address = 00:0c:29:ba:1e:61
                    MAC list = 00:0c:29:ba:1e:61;00:0c:29:ba:1e:6b;
                    VDOM = root (0)
                    EMS serial number: FCTEMS8821001322
                    Client cert SN: 17FF6595600A1AF53B87627AB4EBEDD032593E64
                    Quarantined: no
                    Online status: online
                    Registration status: registered
                    On-net status: on-net
                    Gateway Interface: port2
                    FortiClient version: 7.0.0
                    AVDB version: 84.778
                    FortiClient app signature version: 18.43
                    FortiClient vulnerability scan engine version: 2.30
                    FortiClient UID: 5FCFA3ECDE4D478C911D9232EC9299FD
                    Host Name: ADPC
        … 
                    Number of Routes: (1)
                            Gateway Route #0:
                                    - IP:10.1.100.214, MAC: 00:0c:29:ba:1e:6b, Indirect: no
                                    - Interface:port2, VFID:0, SN: FG5H1E5819902474
    online records: 1; offline records: 0; quarantined records: 0
    
  4. Query the endpoint information, include ZTNA tags, by UID or IP address:

    # diagnose lls-comm send ztna find-uid 5FCFA3ECDE4D478C911D9232EC9299FD
    UID: 5FCFA3ECDE4D478C911D9232EC9299FD
            status code:ok
            Domain: qa.wangd.com
            User: user1
            Cert SN:17FF6595600A1AF53B87627AB4EBEDD032593E64
            EMS SN: FCTEMS8821001322
            Routes(1):
             - route[0]: IP=10.1.100.214, VDom=root
            Tags(3):
             - tag[0]: name=ZT_OS_WIN
             - tag[1]: name=all_registered_clients
             - tag[2]: name=Medium
    
    # diagnose endpoint lls-comm send ztna find-ip-vdom 10.1.100.214 root
    UID: 5FCFA3ECDE4D478C911D9232EC9299FD
            status code:ok
            Domain: qa.wangd.com
            User: user1
            Cert SN:17FF6595600A1AF53B87627AB4EBEDD032593E64
            EMS SN: FCTEMS8821001322
            Routes(1):
             - route[0]: IP=10.1.100.214, VDom=root
            Tags(3):
             - tag[0]: name=ZT_OS_WIN
             - tag[1]: name=all_registered_clients
             - tag[2]: name=Medium
    
  5. Query endpoint information from WAD by UID or IP address:

    # diagnose wad dev query-by uid 5FCFA3ECDE4D478C911D9232EC9299FD
    Attr of type=0, length=32, value(ascii)=5FCFA3ECDE4D478C911D9232EC9299FD
    Attr of type=4, length=30, value(ascii)=MAC_FCTEMS8821001322_ZT_OS_WIN
    Attr of type=4, length=26, value(ascii)=FCTEMS8821001322_ZT_OS_WIN
    Attr of type=4, length=43, value(ascii)=MAC_FCTEMS8821001322_all_registered_clients
    Attr of type=4, length=39, value(ascii)=FCTEMS8821001322_all_registered_clients
    Attr of type=4, length=27, value(ascii)=MAC_FCTEMS8821001322_Medium
    Attr of type=4, length=23, value(ascii)=FCTEMS8821001322_Medium
    Attr of type=5, length=18, value(ascii)=FOSQA@qa.wangd.com
    Attr of type=6, length=40, value(ascii)=17FF6595600A1AF53B87627AB4EBEDD032593E64
    
    # diagnose wad dev query-by ipv4 10.1.100.214
    Attr of type=0, length=32, value(ascii)=5FCFA3ECDE4D478C911D9232EC9299FD
    Attr of type=4, length=30, value(ascii)=MAC_FCTEMS8821001322_ZT_OS_WIN
    Attr of type=4, length=26, value(ascii)=FCTEMS8821001322_ZT_OS_WIN
    Attr of type=4, length=43, value(ascii)=MAC_FCTEMS8821001322_all_registered_clients
    Attr of type=4, length=39, value(ascii)=FCTEMS8821001322_all_registered_clients
    Attr of type=4, length=27, value(ascii)=MAC_FCTEMS8821001322_Medium
    Attr of type=4, length=23, value(ascii)=FCTEMS8821001322_Medium
    Attr of type=5, length=18, value(ascii)=FOSQA@qa.wangd.com
    Attr of type=6, length=40, value(ascii)=17FF6595600A1AF53B87627AB4EBEDD032593E64
    
  6. List all the dynamic ZTNA IP and MAC addresses learned from EMS:
    # diagnose firewall dynamic list
    List all dynamic addresses:
    FCTEMS0000109188_all_registered_clients: ID(51)
            ADDR(172.17.194.209)
            ADDR(192.168.40.8)
    …
    FCTEMS0000109188_Low: ID(78)
            ADDR(172.17.194.209)
            ADDR(192.168.40.8)
    …
    FCTEMS0000109188_Malicious-File-Detected: ID(190)
            ADDR(172.17.194.209)
            ADDR(192.168.40.8)
    …
    
  7. Check the FortiClient NAC daemon ZTNA and route cache:

    # diagnose test application fcnacd 7
    ZTNA Cache:
    -uid 5FCFA3ECDE4D478C911D9232EC9299FD: { "tags": [ "ZT_OS_WIN", "all_registered_clients", "Medium" ], "domain": "qa.wangd.com", "user_name": "user1", "client_cert_sn": "17FF6595600A1AF53B87627AB4EBEDD032593E64", "owner": "FOSQA@qa.wangd.com", "gateway_route_list": [ { "gateway_info": { "fgt_sn": "FG5H1E5819902474", "interface": "port2", "vdom": "root" }, "route_info": [ { "ip": "10.1.100.214", "mac": "00-0c-29-ba-1e-6b", "route_type": "direct" } ] } ], "ems_sn": "FCTEMS8821001322" }
    
    # diagnose test application fcnacd 8
    IP-VfID Cache:
    IP: 10.1.100.206, vfid: 0, uid: 3DED29B54386416E9888F2DCBD2B9D21
    IP: 10.1.100.214, vfid: 0, uid: 5FCFA3ECDE4D478C911D9232EC9299FD 
    
  8. Troubleshoot WAD with real-time debugs to understand how the proxy handled a client request:

    # diagnose wad debug enable category all
    # diagnose wad debug enable level verbose
    # diagnose debug enable
    
    [0x7fbd7a46bb60] Received request from client: 10.10.10.20:56312
    GET / HTTP/1.1 Host: 192.168.2.86:8443 Connection: keep-alive Cache-Control: max-age=0 Upgrade-Insecure-Requests: 1 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36 Edg/89.0.774.57 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9 Sec-Fetch-Site: none Sec-Fetch-Mode: navigate Sec-Fetch-User: ?1 Sec-Fetch-Dest: document Accept-Encoding: gzip, deflate, br Accept-Language: en-US,en;q=0.9 [p:29957][s:458767][r:1] wad_http_marker_uri(1269): path=/ len=1
    [p:29957][s:458767][r:1] wad_http_parse_host(1641): host_len=17
    [p:29957][s:458767][r:1] wad_http_parse_host(1677): len=12
    [p:29957][s:458767][r:1] wad_http_parse_host(1686): len=4
    [p:29957][s:458767][r:1] wad_http_str_canonicalize(2180): path=/ len=1 changes=0
    [p:29957][s:458767][r:1] wad_http_str_canonicalize(2189): path=/ len=1 changes=0
    [p:29957][s:458767][r:1] wad_http_normalize_uri(2232): host_len=12 path_len=1 query_len=0
    [p:29957][s:458767][r:1] wad_vs_proxy_match_gwy(2244): 6:WIN2K16-P1: matching gwy with vhost(_def_virtual_host_)
    [p:29957][s:458767][r:1] wad_vs_proxy_match_vhost(2293): 6:WIN2K16-P1: matching vhost by: 192.168.2.86
    [p:29957][s:458767][r:1] wad_vs_matcher_map_find(477): Empty matcher!
    [p:29957][s:458767][r:1] wad_vs_proxy_match_vhost(2296): 6:WIN2K16-P1: no host matched.
    [p:29957][s:458767][r:1] wad_vs_proxy_match_gwy(2263): 6:WIN2K16-P1: matching gwy by (/) with vhost(_def_virtual_host_).
    [p:29957][s:458767][r:1] wad_pattern_matcher_search(1210): pattern-match succ:/
    [p:29957][s:458767][r:1] wad_vs_proxy_match_gwy(2271): 6:WIN2K16-P1: Matched gwy(1) type(https).
    [p:29957][s:458767][r:1] wad_http_vs_check_dst_ovrd(776): 6:WIN2K16-P1:1: Found server: 192.168.20.6:443
    [p:29957][s:458767][r:1] wad_http_req_exec_act(9296): dst_addr_type=3 wc_nontp=0 sec_web=1 web_cache=0 req_bypass=0
    [p:29957][s:458767][r:1] wad_http_req_check_policy(8117): starting policy matching(vs_pol= 1):10.10.10.20:56312->192.168.20.6:443
    [p:29957][s:458767][r:1] wad_fw_addr_match_ap(1524): matching ap:WIN2K16(7) with vip addr:WIN2K16-P1(10)
    [p:29957][s:458767][r:1] wad_fw_addr_match_ap(1524): matching ap:WIN2K16-P1(10) with vip addr:WIN2K16-P1(10)
    [p:29957][s:458767][r:1] wad_http_req_policy_set(6811): match pid=29957 policy-id=2 vd=0 in_if=3, out_if=7 10.10.10.20:56312 -> 192.168.20.6:443
    [p:29957][s:458767][r:1] wad_cifs_profile_init(93): CIFS Profile 0x7fbd7a5bf200 [] of type 0 created
    [p:29957][s:458767][r:1] wad_http_req_proc_policy(6622): web_cache(http/https=0/0, fwd_srv=<nil>.
    [p:29957][s:458767][r:1] wad_auth_inc_user_count(1668): increased user count, quota:128000, n_shared_user:2, vd_used: 2, vd_max: 0, vd_gurantee: 0
    [p:29957][s:458767][r:1] __wad_fmem_open(563): fmem=0xaaee3e8, fmem_name='cmem 336 bucket', elm_sz=336, block_sz=73728, overhead=20, type=advanced
    [p:29957][s:458767][r:1] __wad_hauth_user_node_hold(2107): wad_hauth_user_node_alloc (1568): holding node 0x7fbd76d48060
    mapping user_node:0x7fbd76d48060, user_ip:0x7fbd7a57b408(0), user:0x7fbd7a5cf420(0)
    [p:29957][s:458767][r:1] __wad_hauth_user_node_hold(2107): wad_user_node_stats_hold (483): holding node 0x7fbd76d48060
    [p:29957][s:458767][r:1] __wad_hauth_user_node_hold(2107): wad_http_session_upd_user_node (4813): holding node 0x7fbd76d48060
    [p:29957][s:458767][r:1] wad_http_req_proc_policy(6698): policy result:vf_id=0:0 sec_profile=0x7fbd7a5bef00 set_cookie=0
    [p:29957][s:458767][r:1] wad_http_urlfilter_check(381): uri_norm=1 inval_host=0 inval_url=0 scan-hdr/body=1/0 url local=0 block=0 user-cat=0 allow=0 ftgd=0 keyword=0 wisp=0
    [p:29957][s:458767][r:1] wad_http_req_proc_waf(1309): req=0x7fbd7a46bb60 ssl.deep_scan=1 proto=10 exempt=0 waf=(nil) body_len=0 ua=Chrome/89.0.4389.90 skip_scan=0
    [p:29957][s:458767][r:1] wad_http_req_proc_antiphish(5376): Processing antiphish request
    [p:29957][s:458767][r:1] wad_http_req_proc_antiphish(5379): No profile
    [p:29957][s:458767][r:1] wad_http_connect_server(4696): http session 0x7fbd7a532ac8 req=0x7fbd7a46bb60
    [p:29957][s:458767][r:1] wad_http_srv_still_good(4575): srv((nil)) nontp(0) dst_type(3)
    req: dst:192.168.20.6:443, proto:10)
    hcs: dst:N/A:0, proto:1)
    
    Tooltip

    Always reset the debugs after using them:

    # diagnose debug reset

 

ZTNA troubleshooting and debugging

The following debug commands can be used to troubleshoot ZTNA issues:

Command

Description

# diagnose endpoint fctems test-connectivity <EMS>

Verify FortiGate to FortiClient EMS connectivity.

# execute fctems verify <EMS>

Verify the FortiClient EMS’s certificate.

# diagnose test application fcnacd 2

Dump the EMS connectivity information.

# diagnose debug app fcnacd -1

# diagnose debug enable

Run real-time FortiClient NAC daemon debugs.

# diagnose endpoint record list <ip>

 

Show the endpoint record list. Optionally, filter by the endpoint IP address.

# diagnose endpoint lls-comm send ztna find-uid <uid>

Query endpoints by client UID.

# diagnose endpoint lls-comm send ztna find-ip-vdom <ip> <vdom>

Query endpoints by the client IP-VDOM pair.

# diagnose wad dev query-by uid <uid>

Query from WAD diagnose command by UID.

# diagnose wad dev query-by ipv4 <ip>

Query from WAD diagnose command by IP address.

# diagnose firewall dynamic list

 

List EMS ZTNA tags and all dynamic IP and MAC addresses.

# diagnose test application fcnacd 7

# diagnose test application fcnacd 8

Check the FortiClient NAC daemon ZTNA and route cache.

# diagnose wad worker policy list

Display statistics associated with access proxy rules.

# diagnose wad debug enable category all

# diagnose wad debug enable level verbose

# diagnose debug enable

Run real-time WAD debugs.

# diagnose debug reset

Reset debugs when completed

Note

The WAD daemon handles proxy related processing. The FortiClient NAC daemon (fcnacd) handles FortiGate to EMS connectivity.

Troubleshooting usage and output

  1. Verify the FortiGate to EMS connectivity and EMS certificate:

    # diagnose endpoint fctems test-connectivity WIN10-EMS
    Connection test was successful:
    
    # execute fctems verify WIN10-EMS
    Server certificate already verified.
    
    # diagnose test application fcnacd 2
    EMS context status:
    FortiClient EMS number 1:
            name: WIN10-EMS confirmed: yes
            fetched-serial-number: FCTEMS0000109188
    Websocket status: connected
    
  2. If fcnacd does not report the proper status, run real-time fcnacd debugs:

    # diagnose debug app fcnacd -1
    # diagnose debug enable
    
  3. Verify the following information about an endpoint:

    • Network information

    • Registration information

    • Client certificate information

    • Device information

    • Vulnerability status

    • Relative position with the FortiGate

    # diagnose endpoint record list 10.6.30.214
    Record #1:
                    IP Address = 10.6.30.214
                    MAC Address = 00:0c:29:ba:1e:61
                    MAC list = 00:0c:29:ba:1e:61;00:0c:29:ba:1e:6b;
                    VDOM = root (0)
                    EMS serial number: FCTEMS8821001322
                    Client cert SN: 17FF6595600A1AF53B87627AB4EBEDD032593E64
                    Quarantined: no
                    Online status: online
                    Registration status: registered
                    On-net status: on-net
                    Gateway Interface: port2
                    FortiClient version: 7.0.0
                    AVDB version: 84.778
                    FortiClient app signature version: 18.43
                    FortiClient vulnerability scan engine version: 2.30
                    FortiClient UID: 5FCFA3ECDE4D478C911D9232EC9299FD
                    Host Name: ADPC
        … 
                    Number of Routes: (1)
                            Gateway Route #0:
                                    - IP:10.1.100.214, MAC: 00:0c:29:ba:1e:6b, Indirect: no
                                    - Interface:port2, VFID:0, SN: FG5H1E5819902474
    online records: 1; offline records: 0; quarantined records: 0
    
  4. Query the endpoint information, include ZTNA tags, by UID or IP address:

    # diagnose lls-comm send ztna find-uid 5FCFA3ECDE4D478C911D9232EC9299FD
    UID: 5FCFA3ECDE4D478C911D9232EC9299FD
            status code:ok
            Domain: qa.wangd.com
            User: user1
            Cert SN:17FF6595600A1AF53B87627AB4EBEDD032593E64
            EMS SN: FCTEMS8821001322
            Routes(1):
             - route[0]: IP=10.1.100.214, VDom=root
            Tags(3):
             - tag[0]: name=ZT_OS_WIN
             - tag[1]: name=all_registered_clients
             - tag[2]: name=Medium
    
    # diagnose endpoint lls-comm send ztna find-ip-vdom 10.1.100.214 root
    UID: 5FCFA3ECDE4D478C911D9232EC9299FD
            status code:ok
            Domain: qa.wangd.com
            User: user1
            Cert SN:17FF6595600A1AF53B87627AB4EBEDD032593E64
            EMS SN: FCTEMS8821001322
            Routes(1):
             - route[0]: IP=10.1.100.214, VDom=root
            Tags(3):
             - tag[0]: name=ZT_OS_WIN
             - tag[1]: name=all_registered_clients
             - tag[2]: name=Medium
    
  5. Query endpoint information from WAD by UID or IP address:

    # diagnose wad dev query-by uid 5FCFA3ECDE4D478C911D9232EC9299FD
    Attr of type=0, length=32, value(ascii)=5FCFA3ECDE4D478C911D9232EC9299FD
    Attr of type=4, length=30, value(ascii)=MAC_FCTEMS8821001322_ZT_OS_WIN
    Attr of type=4, length=26, value(ascii)=FCTEMS8821001322_ZT_OS_WIN
    Attr of type=4, length=43, value(ascii)=MAC_FCTEMS8821001322_all_registered_clients
    Attr of type=4, length=39, value(ascii)=FCTEMS8821001322_all_registered_clients
    Attr of type=4, length=27, value(ascii)=MAC_FCTEMS8821001322_Medium
    Attr of type=4, length=23, value(ascii)=FCTEMS8821001322_Medium
    Attr of type=5, length=18, value(ascii)=FOSQA@qa.wangd.com
    Attr of type=6, length=40, value(ascii)=17FF6595600A1AF53B87627AB4EBEDD032593E64
    
    # diagnose wad dev query-by ipv4 10.1.100.214
    Attr of type=0, length=32, value(ascii)=5FCFA3ECDE4D478C911D9232EC9299FD
    Attr of type=4, length=30, value(ascii)=MAC_FCTEMS8821001322_ZT_OS_WIN
    Attr of type=4, length=26, value(ascii)=FCTEMS8821001322_ZT_OS_WIN
    Attr of type=4, length=43, value(ascii)=MAC_FCTEMS8821001322_all_registered_clients
    Attr of type=4, length=39, value(ascii)=FCTEMS8821001322_all_registered_clients
    Attr of type=4, length=27, value(ascii)=MAC_FCTEMS8821001322_Medium
    Attr of type=4, length=23, value(ascii)=FCTEMS8821001322_Medium
    Attr of type=5, length=18, value(ascii)=FOSQA@qa.wangd.com
    Attr of type=6, length=40, value(ascii)=17FF6595600A1AF53B87627AB4EBEDD032593E64
    
  6. List all the dynamic ZTNA IP and MAC addresses learned from EMS:
    # diagnose firewall dynamic list
    List all dynamic addresses:
    FCTEMS0000109188_all_registered_clients: ID(51)
            ADDR(172.17.194.209)
            ADDR(192.168.40.8)
    …
    FCTEMS0000109188_Low: ID(78)
            ADDR(172.17.194.209)
            ADDR(192.168.40.8)
    …
    FCTEMS0000109188_Malicious-File-Detected: ID(190)
            ADDR(172.17.194.209)
            ADDR(192.168.40.8)
    …
    
  7. Check the Fo