Server-policy outage

Similar to server policy intermittently inaccessible problems, traffic outage also means service access interruption, but mainly refers to sudden break off, and all services do not respond to requests. So though the troubleshooting steps are similar, there are a few special emphasis.

1. Check if all services on FortiWeb are not available, including the HTTPS/SSH service to the management portal, and the HTTP/HTTPS access to the server policies;
2. Check if reboot, crash or coredump occurred when the issue happened;
   • Check system uptime or event logs to see if power off or reboot ever occurred;
   • Check core* or coredump* files via GUI > System > Maintenance > Backup & Restore > GUI File Download/Upload or "/var/log/gui_upload".

     Please note that kernel coredump files cannot be displayed by diagnose debug crashlog show on 7.0.1 and earlier builds, while they can be shown on 7.0.2 and newer builds.

3. Check if any new operation is performed or configuration are changed before the issue happened;

   Event logs can be checked for configuration change event, while detailed CLIs are not included.

4. Check if FortiWeb has system resource shortage;
   Outage may occur when available system resources are extremely low. For example, the memory size of a FortiWeb VM is 4G or lower (not recommended), or the system is configured with too many configuration entries such as server policies or other policies/rules, or OOM (out of memory) happens.
   Please refer to similar steps in Server policy intermittently inaccessible.
5. Check if traffic reaches FortiWeb's performance bottlenecks
   Check if there is any traffic (CPS/Throughput/Attack) burst or shift when the issue happened;

   Traffic burst usually leads to high CPU usage, so you can check the Event logs, nmon records, or 3rd party network monitoring history to confirm.
   Please refer to similar steps in Server policy intermittently inaccessible.

6. Collect other debug logs or files for further investigation.

   • Execute diagnose system top and diagnose system perf several times to find the top CPU-consuming processes;

   • Collect pstack information of proxyd to check where proxyd may stuck at;

     On 6.3:

     FortiWeb # fn sh

     /#

     /# pidof proxyd

     8602

     /# pstack 8602 #replace with the actual proxyd_pid … …

     On 7.0 and later builds:

     FortiWeb # fn pidof proxyd

     28913

     FortiWeb # fn pstack 28913 #replace with the actual proxyd_pid … …

     If proxyd gets stuck for 5 seconds, watchdog files like “watchdog-proxyd-3991-1658580435.bt” will be generated and will be zipped to the debug log "console_log.tar.gz". For more information on pstack, see Retrieving system logs in backend system.

   • Check the output on console terminal;

     Some critical system messages will be printed to console but not written to system logs, so sometimes the console output is very useful for locating the problem. But keep in mind that printing a large amount of messages to console may reduce system performance.

   • Download system debug logs, including the one-click download debug log "console_log.tar.gz" and other logs that require to be manually downloaded.

     Most of the necessary system logs are included in the archived "console_log.tar.gz", while some require to be downloaded manually especially on FortiWeb old versions.

     For more information on collecting "console_log.tar.gz", see Collecting core/coredump files and logs.

     for more information on the content of these logs, see Retrieving system logs in backend system.

     The more complete logs you collect, the better it will help for further analysis.

7. Check if a high volume of logs generated or sent to FortiAnylazer or other outside log servers (may be CPU consuming)

Temporary Actions/Solution

• Check the status of proxyd with ps | grep proxyd;

• Execute exec session-cleanup to restart proxyd or other processes.
  You can also execute “kill <pidof_proxyd>” on the backeend shell or “fn kill <pidof_proxyd>” on the front-end CLI to restart proxyd. Just note that from 7.0.4, you need to enable shell-access and login into the back-end shell for this.

  Please to Run backend-shell commands for how to configure shell-access.

• Collect system and debug logs for further support analysis:

  • Most important system logs can be fetched by one-click download via GUI > System > Maintenance > Debug > Download:

    Please note that you need to enable GUI > System > Config > Feature Visibility > Debug before seeing such option:

  • Sometimes newly-added debug logs may not be included in the archive file downloaded through above method, then it’s better to check and download such logs via GUI > System > Maintenance > Backup & Restore > GUI File Download/Upload:

    Similarly, you needs to enable the GUI File Download/Upload via CLI:

    config system settings

    set enable-file-upload enable

    end