Visiting Server-Policy Has Long Response Time

1. Confirm the issue:
   • Check if the issue only occurs on one policy or impact all policies on the same FortiWeb;
   • Check if the issue happens on HTTP/HTTPS only or both service;
   • Check when the issue happens, if all services on FortiWeb are not available, including the HTTPS/SSH service to the management portal and the HTTP/HTTPS access to the server-policy;
2. Confirm the response time:
   • Use curl to check the response time when visiting the back-end server from FortiWeb, or run a script on FortiWeb to visit the back-end server periodically and record the return code & response time:

     curl -o /dev/null -s -w %{time_total}\\n HTTP://<back-end server_IP>:<port>

     curl -v HTTPs://<domain/IP>/ -A "check_HTTP" -so /dev/null --resolve <domain>:<port>:<IP> -k -w %{time_namelookup}::%{time_connect}::%{time_starttransfer}::%{time_total}::%{speed_download}"\n"

     #* Added direct.ama01.com:443:3.96.215.58 to DNS cache

     * Hostname direct.ama01.com was found in DNS cache

     * Trying 3.96.215.58:443...

     * Connected to direct.ama01.com (3.96.215.58) port 443 (#0)

     … …

     0.000052::0.076644::0.353207::0.353298::0.000

   • Use other tools to test and show the response time when visiting from a client:

     E.g. install a python tool named HTTPstat (actually uses curl with parameters) to better show the test result:

     test@utmaserver01:~$ sudo pip install HTTPstat

     test@utmaserver01:~$ HTTPstat HTTP://<test_domain>/

     

   • Save the visit as an HAR file:

     All major browsers including Chrome, Edge and Firefox support saving a visit as an HAR file. The saved HAR file records the timeline of loading each visited page resource, and can be imported to an HAR viewer such as Chrome extension HTTP Archive Viewer for further analysis.
     
     View with HTTP Archive Viewer:
     

3. Check the system resources (CPU, Memory usage) when the issue happens;
4. Collect diagnose output and debug logs for further support analysis:
   • Diagnose debug flow to check traffic flow processing details;
   • Capture traffic on FortiWeb at the same time and download the pcap files;
   • Turn /proc/tproxy/debug levels and check packets process in kernels:
   • Export configuration files and download debug logs via GUI.
5. Check special configuration and take action to try:
   • If cache or compression is enabled - can disable and test again;
   • Remove web protection profile or modules included from the server-policy, and visit again;
   • Set noparse enable in server-policy policy to bypass waf functions.
     Note: Do not enable noparse on content routing, otherwise content routing will not work.