Retrieving system logs in backend system

1. dmesg

Dmesg is used to examine or control the kernel ring buffer. It includes all important kernel information such as hardware loading and call trace information. Kernel level traffic debug logs will be also included in dmesg.

One can check such logs with “# dmesg” or “#dmesg | grep xxx” directly;

For further troubleshooting, you can archive all logs under the directory /var/log/dmesg/:

tar czf /var/log/gui_upload/dmesg.tar.gz /var/log/dmesg/

2. Apache error logs

If one failed to do some GUI related operation, please collect this logs for analysis:

/var/log# ls apache_logs/

error_log

3. CMDB logs

For configuration deployment issues, please collect cmdb logs for analysis:

# ls /var/log/cmdb/cmdb.log.*

cmdb/cmdb.log.0 cmdb/cmdb.log.155 cmdb/cmdb.log.211 cmdb/cmdb.log.44

#ls /var/log/dbg_cli/

4. /var/log/debug/

Some real-time logs will be generated and stored at /var/log/debug/:

/# ls /var/log/debug/

collect_tcpdump_para.txt daemon_log_flag proxyd_dbg

coredump_log_flag dbsync_log sample

crash.log kernel.log system-startup.log

crash_log_flag kernel_log_flag tmp

crl_updated_dbg netstat_log_flag daemon.log nstd

5. /var/log/gui_upload/

1) Core, coredump and some real-time logs will be generated and stored at /var/log/gui_upload/:

/# ls /var/log/gui_upload/

core-proxyd-2141-1630609770 dlog_logd ha_event_log

core-proxyd-7794-1630610047 ints.txt debug_disk.txtirq

jeprof.out.51146.1630448785.heap perf.data kern.log

debug_out_d_cond_cpu.sh.txt debug_out_d_mem.sh.txt debug_out_d_net.sh.txt

debug_out_d_proc.sh.txt

2) Some logs named as “debug_<function name>.txt” (or with the prefix “debug_out_d_” in some intermediate builds) are generated after 6.4.1.

• Scripts in /var/log/debug/sample/ are samples to run in /var/log/outgoing;

• Scripts in /var/log/outgoing/ are scripts actually run in /var/log/outgoing;

• Currently these system information are collected:

  /# ls /var/log/debug/sample/ #script samples

  README d_cond_cpu.sh d_mem.sh d_net.sh d_proc.sh first_flag

  /# ls /var/log/outgoing/ #scripts actually run

  d_cond_cpu.sh d_mem.sh d_net.sh d_proc.sh

  /# ls -l /var/log/gui_upload/debug_out_d_* (in new builds files are debug_<function name>.txt)

  -rw-r--r-- 1 root 0 65018 Sep 28 18:03 /var/log/gui_upload/debug_out_d_cond_cpu.sh.txt

  -rw-r--r-- 1 root 0 119859 Sep 28 18:03 /var/log/gui_upload/debug_out_d_mem.sh.txt

  -rw-r--r-- 1 root 0 66371 Sep 28 18:03 /var/log/gui_upload/debug_out_d_net.sh.txt

  -rw-r--r-- 1 root 0 126484 Sep 28 18:03

  /var/log/gui_upload/debug_out_d_proc.sh.txt

• The information collected by these scripts mainly include:

  d_cond_cpu.sh: If the CPU usage more than 90% - date, top 10 daemons of CPU usage, perf top for 10 seconds

  d_mem.sh: date, free, /proc/meminfo, etc.

  d_net.sh: date, netstat -natpu, route -n

  d_proc.sh: date, top -b -n1, ps

• The running interval for these scripts can be set with CLI:

  FortiWeb # show full system global

  config system global

  set debug-monitor-interval 5 #minutes

  End

  If the script is blocked for 30 sec, the system will kill it and call it in the next debug-monitor-interval.

• If necessary, one can add scripts (shell or python) to this directory to collect system information; (NOT Recommended, because too many these manually-added tasks may impact system running & stability)

• The size of “debug_<function name>.txt” is limited to 25MB. If the size gets greater, it will be moved to an .old file. And there are only two files rotated.

3) NMON logs are generated after 6.4.0.

NMON (shorthand for Nigel's Monitor) is a system monitor tool that can collect system performance statistics including CPU, Mem, Disk, Net, etc.

• NMON log files (with a suffix .nmon) are generated automatically and stored at /var/log/debug/tmp, and will be archived and can be downloaded via the method described in below section 10.2. The maximum number of .nmon files stored is 180.

• A .nmon file is generated with a sampling interval of 5 minutes, and each time when system boots up, a new .nmon file will be generated. So generally only one .nmon file named “FortiWeb_220107_1734.nmon” (may be different on some previous builds) will be generated each day. Multiple .nmon files generated in one day indicate that system rebooted or crashed.

  

• After processed by an nmon analyzer: