Real-time file system integrity checking
Real-time file system integrity checking has two main purposes:
- Prevent unauthorized modification of important binaries.
- Detect unauthorized binaries and prevent them from running.
How it works
A hash of all executable binary files and shared libraries are taken during image build time. The file containing these hashes, called the executable hash, is also hashed. This new hash is signed together with other important files like the FortiOS firmware, AV and IPS engine files.
When the FortiGate boots, the system performs a BIOS level integrity check on the firmware image, the AV engine file, and the IPS engine file. These files are signed by the process described in BIOS-level signature and file integrity checking, and the BIOS verifies their signature against their certificates. The hash of the executable hash will also follow the BIOS-level signature and integrity check during the boot process, to prevent any unauthorized changes from being loaded.
Once these files are verified to be authentic, the BIOS can extract and boot the root filesystem and other executables and libraries, which then go through another file integrity check.
Once the file integrity checks passes, the corresponding hashes of every executable and shared library can be extracted from the executable hash and loaded into memory. The system is started and real-time protection begins.
The important executables and binaries are protected from write access and any modifications. It also blocks the kernel from loading any modules. Any unauthorized loading of modules is blocked. If violations are found, logs are triggered.
When each executable and shared library is initialized, it will be verified against its respective hashes to ensure integrity. If there is a hash mismatch when attempting to run a binary, that binary is blocked from running, and the system is rebooted. A log will be generated with ID 20234.
If there is a missing hash when attempting to run a binary, then the system is rebooted. A log will be generated with ID 20223.
The system also runs a periodic check to verify the integrity of important binaries and AV and IPS engines.
Log summary
The following logs are recorded when specific actions take place.
|
Log |
Description |
|---|---|
|
20230 - LOG_ID_SYS_SECURITY_WRITE_VIOLATION 432 |
The root filesystem is read only. Any modification triggers this log. |
|
20231 - LOG_ID_SYS_SECURITY_HARDLINK_VIOLATION 432 |
An attacker trying to replace symlink triggers this log. |
|
20232 - LOG_ID_SYS_SECURITY_LOAD_MODULE_VIOLATION 433 |
Only the kernel can load modules. Any unusual loading of modules triggers this log. |
|
20233 - LOG_ID_SYS_SECURITY_FILE_HASH_MISSING 434 |
File hashes are generated for legitimate files during bootup. If a hash cannot be found, the file may be suspicious as it could be a new routine inserted by an attacker. The binary is blocked. |
|
20234 - LOG_ID_SYS_SECURITY_FILE_HASH_MISMATCH 434 |
File hashes are generated for legitimate files during bootup. If a hash does not match when the file is exercised, it is an indication that it could have been modified by an attacker. The system is rebooted. |
Detection examples
Example 1: system reboots due to mismatched hash
fos_ima: fos_process_appraise 110: Executable File(/bin/node) doesn't match previous hash, it has been changed Restarting system. … fos_ima: fos_process_appraise 110: Executable File(/lib/libc.so.6) doesn't match previous hash, it has been changed Restarting system. …
Logs similar to the following are captured:
date="2023-06-16" time="12:01:44" id=7245222014288399309 bid=471609558 dvid=6533 itime=1686909705 euid=3 epid=3 dsteuid=3 dstepid=3 logver=604132092 logid="0100020234" type="event" subtype="system" level="alert" msg="Hash of executable file(/bin/init) doesn't match the previous." logdesc="Integrity check of Run/loading Excutable File failed without Integrity measure" severity="alert" eventtime=1686909705825483706 tz="+0200" devid="xxxxxxxxx" vd="root" devname="xxxxxxxxx" date="2023-06-15" time="09:57:54" id=7244819017507013700 bid=470303007 dvid=1431 itime=1686815875 euid=3 epid=3 dsteuid=3 dstepid=3 logver=604132092 logid="0100020234" type="event" subtype="system" level="alert" msg="Hash of executable file(/lib/libc.so.6) doesn't match the previous." logdesc="Integrity check of Run/loading Excutable File failed without Integrity measure" severity="alert" eventtime=1686815874936267770 tz="+0200" devid=" xxxxxxxxx " vd="root" devname=" xxxxxxxxx”
Example 2: suspected compromise due to an observed indicator of compromise (IoC)
fos_ima: fos_process_appraise 99: Suspicous Executable File(/data2/libcrashpad.so) is missing hash … fos_ima: fos_process_appraise 99: Suspicous Executable File(/data2/flatkc_info) is missing hash …
No logs are found.
Corrective action
In the previous examples where a mismatched or missing hash occurs, alert technical support straight away so that they may gather information to start a forensic analysis with our internal PSIRT team. There are two possible outcomes:
-
The firewall is reporting a false positive, in which a bug causes a mismatched or missing hash.
Once verified by technical support, the corrective action may to be upgrade to a newer build where the bug is fixed
-
An actual compromise has occurred, or is occurring.
The system could be blocking an offending binary that causes the system to malfunction, or the system could reboot to protect itself from compromise.
In either case, contact technical support for further forensic analysis. If an IoC is detected and it is determined that the persistent threat resides on the FortiGate, a reflash and reload of the firmware may be recommended.
Unauthorized firmware modification attempt reporting
In the rare event that unauthorized modification is detected in the firmware, the system will immediately log and report the modification attempt to FortiGuard through a secure channel. Payloads are encrypted to ensure the security of the transferred information. Information about the attempted modification of firmware helps Fortinet Inc. proactively investigate the incident and protect future malicious attempts at compromising the system.
After reporting the modification attempt, the FortiGate real-time file system integrity checking feature continues with the required actions based on the assessed threat. This may involve reverting the change and rebooting the firewall to mitigate the threat.
Example
This example demonstrates when an attempt to alter files in the 'bin' directory was made by a threat actor.
Captured log:
1: date=2024-02-16 time=18:29:15 eventtime=1708136955710925685 tz="-0800" logid="0100020230" type="event" subtype="system" level="alert" vd="vd1" logdesc="Write Permission Violation" msg="[Write Violation: try to write readonly file](/bin/lspci)."
The FortiGate sends an encrypted report to FortiGuard with information about the affected platform and the Modification Attempt such as:
-
FortiGate serial number
-
Model number
-
FortiOS firmware
-
Type of modification attempt (such as Write violation)
-
File path (such as /bin/lspci)
-
File size
-
Time of access and modification