FortiNDR troubleshooting tips
For more information about the CLI commands below, please see the FortiNDR CLI Reference Guide.
Best practices:
Issue Type |
Recommendations / Possible cause |
CLI command |
Comments |
---|---|---|---|
General hang issue and disconnect issues |
Reload all services to see if the issue is still reproducible |
|
|
CPU usage consistently above 85% |
Turn off feature learning to save more resources |
|
|
GUI not responsive/DB related error (and you can afford to lose all current data) |
If you installed an interim build (other than GA) and are willing to wipe all db records |
|
Run exec reload to see if issue is still reproducible |
GUI not responsive/DB related error (and you to make a best effort to keep your data) |
If you installed an interim build (other than GA) and cannot wipe all db records |
|
Patches db at best efforts. |
General Issues |
Retrieve and record all information |
|
If you are seeing high CPU and MEM usage, please consider provisioning more resources. |
General VM issues |
Retrieve and record all information for VMs |
|
Observe for any FDS code other than 200, and if not 200, please check connections to FDN and license status. |
Recommended Debug Setup:
- A syslog server for FortiNDR events log as the GUI only has 1 days events.
- A TFTP server for PCAP capture transfer.
General Debug Logs Retrieval
Scenario |
CLI |
---|---|
Collect all crash logs from the first day FortiNDR started | diagnose debug crashlog <crash_log_date>
|
Record kernel related logs from the bootup and save it to a file | diagnose debug kernel display
|
File scanning related issues
The following troubleshooting tips are intended to diagnose the error message: File Not Accepted (Client side shows files are submitted but NDR does not have details of file).
To perform a general check:
- Check and record network conditions from the FortiNDR server to file submitting clients using the following CLI commands:
exec ping
exec traceroute
- Make sure all KDBs are updated. For example, no pending updates, no out of date db and no updating.
- Try submitting a lower throughput, (no archive file type, smaller file size) to see if it is still reproducible.
- Follow the PCAP dumping guide to dump files from port1 or port2 to make sure the traffic is there. Open dapture pcap with Wireshark to see if there are any redline/blacklines from Wireshark default filter setting which indicates bad network traffic quality. From previous troubleshooting experience, this is the most frequent cause of File Not Accepted.
Troubleshooting ICAP issues:
- After you reproduce the issue:
- Retrieve the latest ICAP server logs by running the CLI command:
diag debug icap
- Save the server logs to a file.
- Retrieve the latest ICAP server logs by running the CLI command:
- Usually you can resolve any outstanding issues by running the following CLI command:
exec reload
Troubleshooting OFTP issues:
- From OFTP clients (usually FortiGate), record all traffic forward/AntiVirus Event logs from the FortiGate side.
- Refer to PCAP capturing guide, and save corresponding PCAPs.
Troubleshooting HTTP2 issues from FortiGate v7.0 onwards:
Recommendation |
Run the following CLI command: |
---|---|
Record output and check for errors | diagnose system csf global
|
Record output and make sure status is authorized | diagnose system csf upstream
|
Collect logs | diag debug enable and diag debug application csfd 7 |
Manual Upload/API Submission/FortiSandbox Integration
For all issues:
Start with a single file upload and fetch results from the same subnet as directed from where the client resides. See Appendix A: API guide.
To verify the process is successful:
If a single file submit/fetch is working from the previous step. Run the following CLI commands:
diag debug enable
and
diagnose debug application 7
Record all output and look for any non 200 http
code or stack traces.
File Submitted but not processed
Collect all the information from the process and record it using the following CLI commands:
diag debug enable
and
diagnose debug process <process_name>
Information for support tickets
For an optimal support experience, we recommend including the following information in your support ticket:
Model Name |
What Model is your platform |
Firmware Version |
Which Firmware is your platform |
Always Reproducible |
True/False |
Reproducible Steps |
If this issue is reproducible, please include the steps to reproduce this issue. |
Actual Result vs Expected Results |
What are the expected results and actual results. |
Troubleshootingrecommendations used |
Please describe the troubleshooting recommendations you attempted and the outcome. |
Crash Log output |
Run the following CLI command For more information see, diagnose debug. |
Kernel Log output |
Record the output
from For more information see, diagnose debug. |
Application Log Output |
Specified log output from affected application. For more information see, diagnose debug. |
Database Error Log |
Record the log output
from |
FortiNDR Event Log |
If configured with FAZ/FortiSIEM, please provide logs from FAZ/FortiSIEM. Otherwise, please include a screen shot from Event tab. |
Client-Side Log |
When FortiNDR acts as a server role (ICAP, OFTP and HTTP2 inline blocking etc.), please also include logs from the client side. |
System Status |
Provide the CPU and MEM usage when the issue occurs. |
FortiNDR configuration |
How many ports are configured/used? How many applications are used, and how are they configured? If you have a backup configuration file, please include it in your support ticket. For more information, see Backup or restore the system configuration. |
FortiNDR Load |
For configured applications, how much load are applied. For example:
|
For non-hardware platform: |
|
Virtual machine provisioning |
|
Hosting Hardware specifications |
CPU Model Number, RAM Channels, DISK model number. |
Physical hosting hardware load |
A historical log of host hardware status to indicate CPU and MEM usage |