Flow mode stream-based scanning

In flow mode with AV engine 7.0, FortiOS automatically uses stream-based antivirus scanning for HTML and Javascript files. The AV engine determines the necessary amount of file payload to buffer and scans the partial buffer in certain instances, eliminating the need to cache the entire file, and potentially improving memory usage.

Prior to AV engine 7.0, flow AV operates in a hybrid mode where the IPS engine will attempt an in-process AV scan by default. If the built-in AV engine in the IPS process indicates a full scan is required, the file is sent to the scanunit process for a full scan. In this scenario, the whole file is cached before scanunit can begin scanning the file.

With this stream-based AV scanning enhancement, the built-in AV engine in the IPS process can attempt to scan HTML and Javascript files as it buffers the file. This provides better performance and potentially less memory usage overall compared to a full scan.

The full antivirus scanning method is retained for file types and configurations unsupported by stream-based scanning.

The following table summarizes the types of scans and when they are automatically used:

Default antivirus scan

Automatically uses stream-based scanning in flow mode for HTML and Javascript files. Automatically scans other supported files using the flow DB. Triggers a legacy scan for unsupported configurations and file types.

Full antivirus scan

Automatically used for files types unsupported by default antivirus scans. Automatically used when any of the following antivirus scanning features are enabled: Machine learning-based malware detection (set machine-learning-detection) Extreme antivirus database (set use-extreme-db) Antivirus PUP/PUA grayware checks Mobile malware database (set mobile-malware-db) External block list (set external-blocklist) EMS external feed FortiGuard outbreak prevention Automatically used when any of the following scanning features are used: Data loss prevention (DLP) File filter

Example

When the default antivirus scan is used, the AV engine uses stream-based scanning to partially buffer the file and scan it:

# diagnose debug application ipsengine 0x1000
diagnose sys scanunit debug all
diagnose debug enable
...
[flav-402] open file size: 68, ftype: 0
[flav-402] flowav config allows quickscan: yes
[flav-402] fsa_enabled=0 fsa_mode=2
[flav-402] heur=0 (0) bz2=1 fsa_ft=0 grayware=0 scantypes: 1
[flav-402] av_flow_write_0 flav_ctx=0x7f35a61a1000, buflen=68, rc = 2 
[flav-402] file is infected or suspicious. Wait until file close due to av_exempt enabled
[flav-402] ips_avscan_file_close
[flav-402] [118]: quickscan_close() flav_ctx=0x7f35a61a1000, rc = 2
[flav-402] [118]: cached length 0, flow_bytes 68
[flav-402] [118]: virus EICAR_TEST_FILE detected!
[flav-402] quickscan_destroy(), flow_writes=1, flow_bytes=68, flav_ctx=0x7f35a61a1000
[flav-402] [118]: quickscan finalized with action 1
[flav-402] ips_avscan_file_close, action=1
[flav-402] ips_avscan_file_destroy

The above debug is taken while a user attempts to download an EICAR file. Partial buffering occurred and the file is scanned inside the IPS engine.

When the default antivirus scan (stream-based scanning) cannot be used for a file, the full antivirus scan is used, and the IPS engine buffers the entire file before sending it to scanunit for scanning:

# diagnose debug application ipsengine 0x1000
# diagnose sys scanunit debug all
# diagnose debug enable
...
[flav-496] [41]: quickscan_close() flav_ctx=0x7fdfc41a1000, rc = -7
[flav-496] [41]: file requires fullscan
[flav-496] attempting switch to fullscan
[flav-496] succesfully switched to fullscan
[flav-496] got FlowAV fullscan request: query_id=41 view_id=3 file_size=12939
[flav-496] quickscan_destroy(), flow_writes=10, flow_bytes=12939, flav_ctx=0x7fdfc41a1000
su 2388 open

The Flow AV statistics monitor can be used to view whether the default or full (legacy) scan method was used:

# diagnose test app ipsmonitor 24  

pid: 23498 from 20240404-09:23:05 to 20240404-09:23:59
av_failopen: enabled
    FlowAV mmap                   : 0
    FlowAV file open              : 0
    FlowAV timeout                : 0
    FlowAV req success            : 0
    FlowAV req fail               : 0
    FlowAV req retry success      : 0
    FlowAV req retry fail         : 0
    FlowAV bypassed scan          : 0
    FlowAV buffer scan            : 0
    FlowAV file scan              : 0
    FlowAV interface file open    : 0
    FlowAV interface file close   : 0
    FlowAV interface file destroy : 0
    FlowAV ignored files          : 0
    FlowAV legacy scan            : 1 
    FlowAV default scan           : 1