Checking core files and basic coredump information

When you suspect that a system or daemon crash happened, one can use diagnose commands to confirm and check the basic information.

1. Confirm that enable-debug-log is enabled, so that FortiWeb will record crash, daemon, kernel, netstat, and core dump logs.

   FortiWeb# show full-configuration sys settings

   config system settings

   set enable-debug-log enable #enabled by default

   end

2. Check if enable-core-file is enabled or not.

   FortiWeb# show full-configuration system settings

   config system settings

   set enable-core-file enable #disabled by default on 7.0.4 and later builds

   end

   1) On 7.0.3 and previous builds including 6.3.x, this option is enabled by default. That means if daemon coredump happens, a coredump file which includes the snapshot of the current memory will be generated.

   However, generating a coredump file usually takes from several seconds to several minutes, especially on a device with large memory size. During this period, the program stops providing service.

   2) On 7.0.4 and later builds, enable-core-file is set as disable by default. FortiWeb also optimizes the coredump mechanism thus more useful information can be recorded even if without a coredump file. You need to to use “diagnose debug crashlog show” as below to collect stack information for the crashed daemon.

3. Use “diagnose debug crashlog show” to check if any coredump files are generated or collect the stack information.
   1) On 7.0.1 and previous builds including 6.3.x:

   Only daemon (proxyd, ml, etc.) coredump files can be listed, so it is better to double check via GUI to see if kernel coredump occurred.

   The files are named with a Unix timestamp, so you need to convert it into a human-readable date and time format to see if it’s a newly generated one. (Refer to the below section Notes for details)

   FWB# diagnose debug crashlog show

   core-proxyd-2141-1630609770

   core-proxyd-60152-16306095792)

   2) On 7.0.2 and later builds:

   • Both daemon and kernel coredump files will be shown by this command.

   • The files are named with a readable timestamp. Just note the timestamp for kernel core & coredump files are adjusted to fit the system time & timezone, while the daemon coredump filename is using the UTC time.

     FWB# diagnose debug crashlog show

     core-2022-11-09-16_40-7.0.2-B0090

     coredump-2022-11-09-16_40-7.0.2-B0090

     core-proxyd-10984-1668037208-UTC-2022-11-09-23_40-7.0.2-B0090

   • If enable-core-file is disabled, the function stack information will be shown instead of the coredump files. Please collect these information for developer analysis.

     FWB# diagnose debug crashlog show

     2022-11-09 15:59:20 <10359> application proxyd

     2022-11-09 15:59:20 <10359> *** signal 6 received ***

     2022-11-09 15:59:20 <10359> __poll[0x7fcb81055580 + 0x4f]

     2022-11-09 15:59:20 <10359> dbg_writedisk_unlock_func[0x8ab4b0 + 0x20f2]

     2022-11-09 15:59:20 <10359> main[0x4df5a0 + 0x154]

     2022-11-09 15:59:20 <10359> __libc_start_main[0x7fcb80f8cc20 + 0xeb]

     2022-11-09 15:59:20 <10359> _start[0x4e0f20 + 0x2a]

     On all builds, an easier way to judge if core or coredump files are new is checking the Last Modify Time on GUI. This time adapts to the current system time.
     

4. Use “diagnose debug coredumplog show” to show daemon coredump. Only daemon coredump information can be shown here.

   FortiWeb# diagnose debug coredumplog show

   ======== coredump about /var/log/gui_upload/core-proxyd-4830-1639993541 =======

   (gdb) 0 0x0000563f7b340e24 in pth_comm_add_pb_adom_entry ()

   1 0x0000563f7b48584c in session_management_get_weight ()

   0000002 0x0000563f7b4b23b2 in ip_intelligence_session_init_do_action ()

   3 0x0000563f7b4b262d in ip_intelligence_session_init ()

   4 0x0000563f7b3343ff in pth_init_modinfo ()

   5 0x0000563f7b310797 in pt_service_HTTP_init ()

   6 0x0000563f7b30959c in pt_service_init ()

   7 0x0000563f7b3837bb in pt_stream_create_service ()

   8 0x0000563f7b3842f1 in pt_stream_create ()

   9 0x0000563f7b38a3d4 in session_accept ()

   10 0x0000563f7b350cba in fd_epoll_poll ()

   11 0x0000563f7b39622d in _worker_loop ()

   0000012 0x0000563f7b3965f8 in worker_run ()

   13 0x00007fa62d314f27 in start_thread () from /fwdev2//lib/libpthread.so.0

   14 0x00007fa6269ff1df in clone () from /fwdev2//lib/libc.so.6

   (gdb)

   From above information from bug #770008 (already fixed on 6.3.18), it seems the coredump is related to client management configuration, so one workaround applied at that time was disable the block settings in client management

   Note:

   Notes: On 7.0.1 and previous builds, the format of core files are defined by:

   /# more /proc/sys/kernel/core_pattern

   /var/log/gui_upload/core-%e-%p-%t

   %e: daemon/process name

   %p: PID of the process

   %t: UNIX timestamp; one can use online tools to convert it to a human-readable date. This is useful to confirm the recent & current coredump files if there are many files. (Of course, you can also check the file created time from “Last Modified Time” via System > Maintenance > Backup & Restore > GUI File Download/Upload)

   E.g.:

   Epoch Converter - Unix Timestamp Converter

   

Actually you have another way to simply check the file generation date from GUI; just check the section below to find "Download core/coredump files".