Fortinet black logo

Administration Guide

Home FortiGate / FortiOS 7.0.15 Administration Guide
7.0.15
7.4.3
7.4.2
7.4.1
7.4.0
7.2.8
7.2.7
7.2.6
7.2.5
7.2.4
7.2.3
7.2.2
7.2.1
7.2.0
7.0.15
7.0.14
7.0.13
7.0.12
7.0.11
7.0.10
7.0.9
7.0.8
7.0.7
7.0.6
7.0.5
7.0.4
7.0.3
7.0.2
7.0.1
7.0.0
6.4.15
6.4.14
6.4.13
6.4.12
6.4.11
6.4.10
6.4.9
6.4.8
6.4.7
6.4.6
6.4.5
6.4.4
6.4.3
6.4.2
6.4.1
6.4.0

Real-time file system integrity checking

Real-time file system integrity checking

Real-time file system integrity checking has two main purposes:

  • Prevent unauthorized modification of important binaries.
  • Detect unauthorized binaries and prevent them from running.

How it works

When the FortiGate boots, the system performs a BIOS level integrity check on important internal files, the AV engine file, and the IPS engine file. These files are signed by the process described in BIOS-level signature and file integrity checking, and the BIOS verifies their signature against their certificates.

Once these files are verified to be authentic, the BIOS can boot the root filesystem and other executables and libraries. Once loaded, real-time protection begins. The important executables and binaries are protected from write access and any modifications. It also blocks the kernel from loading any modules. Any unauthorized loading of modules is blocked. If violations are found, logs are triggered.

A hash of all executable binaries and libraries is taken and stored in memory. If there is a hash mismatch when attempting to run a binary, that binary is blocked from running, and the system is rebooted. A log will be generated with ID 20234.

If there is a missing hash when attempting to run a binary, then the system is rebooted. A log will be generated with ID 20223.

The system also runs a periodic check to verify the integrity of important binaries and AV and IPS engines.

Log summary

The following logs are recorded when specific actions take place.

Log

Description

20230 - LOG_ID_SYS_SECURITY_WRITE_VIOLATION 432

The root filesystem is read only. Any modification triggers this log.

20231 - LOG_ID_SYS_SECURITY_HARDLINK_VIOLATION 432

An attacker trying to replace symlink triggers this log.

20232 - LOG_ID_SYS_SECURITY_LOAD_MODULE_VIOLATION 433

Only the kernel can load modules. Any unusual loading of modules triggers this log.

20233 - LOG_ID_SYS_SECURITY_FILE_HASH_MISSING 434

File hashes are generated for legitimate files during bootup. If a hash cannot be found, the file may be suspicious as it could be a new routine inserted by an attacker. The binary is blocked.

20234 - LOG_ID_SYS_SECURITY_FILE_HASH_MISMATCH 434

File hashes are generated for legitimate files during bootup. If a hash does not match when the file is exercised, it is an indication that it could have been modified by an attacker. The system is rebooted.

Detection examples

Example 1: system reboots due to mismatched hash

fos_ima: fos_process_appraise 110: Executable File(/bin/node) doesn't match previous hash, it has been changed
Restarting system.
…
fos_ima: fos_process_appraise 110: Executable File(/lib/libc.so.6) doesn't match previous hash, it has been changed
Restarting system.
…

Logs similar to the following are captured:

date="2023-06-16" time="12:01:44" id=7245222014288399309 bid=471609558 dvid=6533 itime=1686909705 euid=3 epid=3 dsteuid=3 dstepid=3 logver=604132092 logid="0100020234" type="event" subtype="system" level="alert" msg="Hash of executable file(/bin/init) doesn't match the previous." logdesc="Integrity check of Run/loading Excutable File failed without Integrity measure" severity="alert" eventtime=1686909705825483706 tz="+0200" devid="xxxxxxxxx" vd="root" devname="xxxxxxxxx"

date="2023-06-15" time="09:57:54" id=7244819017507013700 bid=470303007 dvid=1431 itime=1686815875 euid=3 epid=3 dsteuid=3 dstepid=3 logver=604132092 logid="0100020234" type="event" subtype="system" level="alert" msg="Hash of executable file(/lib/libc.so.6) doesn't match the previous." logdesc="Integrity check of Run/loading Excutable File failed without Integrity measure" severity="alert" eventtime=1686815874936267770 tz="+0200" devid=" xxxxxxxxx " vd="root" devname=" xxxxxxxxx”

Example 2: suspected compromise due to an observed indicator of compromise (IoC)

fos_ima: fos_process_appraise 99: Suspicous Executable File(/data2/libcrashpad.so) is missing hash
…
fos_ima: fos_process_appraise 99: Suspicous Executable File(/data2/flatkc_info) is missing hash
…

No logs are found.

Corrective action

In the previous examples where a mismatched or missing hash occurs, alert technical support straight away so that they may gather information to start a forensic analysis with our internal PSIRT team. There are two possible outcomes:

  1. The firewall is reporting a false positive, in which a bug causes a mismatched or missing hash.

    Once verified by technical support, the corrective action may to be upgrade to a newer build where the bug is fixed

  2. An actual compromise has occurred, or is occurring.

    The system could be blocking an offending binary that causes the system to malfunction, or the system could reboot to protect itself from compromise.

In either case, contact technical support for further forensic analysis. If an IoC is detected and it is determined that the persistent threat resides on the FortiGate, a reflash and reload of the firmware may be recommended.

Previous
Next

Real-time file system integrity checking

Real-time file system integrity checking has two main purposes:

  • Prevent unauthorized modification of important binaries.
  • Detect unauthorized binaries and prevent them from running.

How it works

When the FortiGate boots, the system performs a BIOS level integrity check on important internal files, the AV engine file, and the IPS engine file. These files are signed by the process described in BIOS-level signature and file integrity checking, and the BIOS verifies their signature against their certificates.

Once these files are verified to be authentic, the BIOS can boot the root filesystem and other executables and libraries. Once loaded, real-time protection begins. The important executables and binaries are protected from write access and any modifications. It also blocks the kernel from loading any modules. Any unauthorized loading of modules is blocked. If violations are found, logs are triggered.

A hash of all executable binaries and libraries is taken and stored in memory. If there is a hash mismatch when attempting to run a binary, that binary is blocked from running, and the system is rebooted. A log will be generated with ID 20234.

If there is a missing hash when attempting to run a binary, then the system is rebooted. A log will be generated with ID 20223.

The system also runs a periodic check to verify the integrity of important binaries and AV and IPS engines.

Log summary

The following logs are recorded when specific actions take place.

Log

Description

20230 - LOG_ID_SYS_SECURITY_WRITE_VIOLATION 432

The root filesystem is read only. Any modification triggers this log.

20231 - LOG_ID_SYS_SECURITY_HARDLINK_VIOLATION 432

An attacker trying to replace symlink triggers this log.

20232 - LOG_ID_SYS_SECURITY_LOAD_MODULE_VIOLATION 433

Only the kernel can load modules. Any unusual loading of modules triggers this log.

20233 - LOG_ID_SYS_SECURITY_FILE_HASH_MISSING 434

File hashes are generated for legitimate files during bootup. If a hash cannot be found, the file may be suspicious as it could be a new routine inserted by an attacker. The binary is blocked.

20234 - LOG_ID_SYS_SECURITY_FILE_HASH_MISMATCH 434

File hashes are generated for legitimate files during bootup. If a hash does not match when the file is exercised, it is an indication that it could have been modified by an attacker. The system is rebooted.

Detection examples

Example 1: system reboots due to mismatched hash

fos_ima: fos_process_appraise 110: Executable File(/bin/node) doesn't match previous hash, it has been changed
Restarting system.
…
fos_ima: fos_process_appraise 110: Executable File(/lib/libc.so.6) doesn't match previous hash, it has been changed
Restarting system.
…

Logs similar to the following are captured:

date="2023-06-16" time="12:01:44" id=7245222014288399309 bid=471609558 dvid=6533 itime=1686909705 euid=3 epid=3 dsteuid=3 dstepid=3 logver=604132092 logid="0100020234" type="event" subtype="system" level="alert" msg="Hash of executable file(/bin/init) doesn't match the previous." logdesc="Integrity check of Run/loading Excutable File failed without Integrity measure" severity="alert" eventtime=1686909705825483706 tz="+0200" devid="xxxxxxxxx" vd="root" devname="xxxxxxxxx"

date="2023-06-15" time="09:57:54" id=7244819017507013700 bid=470303007 dvid=1431 itime=1686815875 euid=3 epid=3 dsteuid=3 dstepid=3 logver=604132092 logid="0100020234" type="event" subtype="system" level="alert" msg="Hash of executable file(/lib/libc.so.6) doesn't match the previous." logdesc="Integrity check of Run/loading Excutable File failed without Integrity measure" severity="alert" eventtime=1686815874936267770 tz="+0200" devid=" xxxxxxxxx " vd="root" devname=" xxxxxxxxx”

Example 2: suspected compromise due to an observed indicator of compromise (IoC)

fos_ima: fos_process_appraise 99: Suspicous Executable File(/data2/libcrashpad.so) is missing hash
…
fos_ima: fos_process_appraise 99: Suspicous Executable File(/data2/flatkc_info) is missing hash
…

No logs are found.

Corrective action

In the previous examples where a mismatched or missing hash occurs, alert technical support straight away so that they may gather information to start a forensic analysis with our internal PSIRT team. There are two possible outcomes:

  1. The firewall is reporting a false positive, in which a bug causes a mismatched or missing hash.

    Once verified by technical support, the corrective action may to be upgrade to a newer build where the bug is fixed

  2. An actual compromise has occurred, or is occurring.

    The system could be blocking an offending binary that causes the system to malfunction, or the system could reboot to protect itself from compromise.

In either case, contact technical support for further forensic analysis. If an IoC is detected and it is determined that the persistent threat resides on the FortiGate, a reflash and reload of the firmware may be recommended.

Previous
Next