Fortinet black logo

Whats New in 6.2.1

Whats New in 6.2.1

This document describes new features, bug fixes, and known issue for the FortiSIEM 6.2.1 (build) release.

New Features

Reputation Check from FortiGuard

FortiGuard IOC provides reputation checks for different Malware like IP, Domain, URL, and Hash. This release adds FortiGuard IOC to the supported list of reputation lookup engines. This can be invoked in of two ways.

  • When an incident triggers, users can manually do a lookup via FortiGuard IOC, VirusTotal, or RiskIQ. Any external website can also be looked up, but the results of FortiGuard IOC, VirusTotal or RiskIQ are parsed and the user is provided with a simplified Verdict (Safe, Unsure, or Malicious based on thresholds).

  • Users can automate this reputation check by including this in their Incident Notification Policies. When included, the reputation check will automatically be done when an incident triggers. Incident comments are updated with the results.

Every licensed FortiSIEM can do FortiGuard IOC lookup. No special configuration is required.

For setting up manual external lookup, see FortiGuard IOC Integration.

For automating via notification policy, see Adding Incident Notification Settings.

For information on doing lookup, see Lookups Via External Websites.

Bug Fixes and Minor Enhancements

The current release includes the following bug fixes and minor enhancement:

Bug ID

Severity

Module

Description

710084

Major

App Server

After a period of time, all Windows Agents may become disconnected (error PH_AUDIT_AGENT_DISABLED logs). This happens very rarely.

710715

Major

Elasticsearch

If an empty action was sent to Supervisor on port 7919, the Java Query Server would stop working.

710109

Major

Report Engine

For some baseline reports, COUNT DISTINCT calculations resulted in very large partial inline report results to be transferred from phReportWorker to the phReportMaster process. This sometimes resulted in ACE_SSL errors and caused baseline reports to contan incomplete data, as that shown here. o 2021-03-26T01:05:11.482049-04:00 w2 phReportWorker[7507]: ACE_SSL (7507|8200) error code: 336462231 - error:140E0197:SSL routines:SSL_shutdown:shutdown while in init

Note: This fix works for new installs of 6.2.1, but upgrades to 6.2.1 are still hindered. The upgrade process will be fixed in 6.3.0+ releases. See Known Issue after 6.2.1 Upgrade for workaround.

712172

Minor

Data Manager

Occasionally, the phDataPurger process would terminate because of initialization issues, resulting in Event DB or Elasticsearch storing events disk space management errors.

711564

Minor

Elasticsearch

The phDataManager process may erroneously introduce a new event field called "UnknownIPVersion" when it encounters an empty IP address field. This can occur for DNS lookup failures in Windows Event Forwarding when the parser cannot resolve the original windows server host name (see bug ID 711542). This can cause Elasticsearch insert failures.

707185

Minor

GUI

The SD-WAN Interface Dashboard failed to load.

711542

Minor

Parser

When the reporting IP is empty and the source IP is 127.0.0.1, the phParser may set the source IP to be empty, resulting in an invalid database insertion entry. This can be identified in Windows Event Forwarding cases when the phParser cannot resolve the original windows server host name to its IP.

707524

Minor

System

Upgrade failed if the DNS Server could not reverse lookup its own name/IP.

710541

Minor

System

Upgrade failed if a FortiSIEM node could not ping the CentOS repository os-pkgs-cdn.fortisiem.fortinet.com.

713893

Enhancement

System

During an upgrade, the previous upgrade related ansible log file would be overwritten. Note: Fixed by retaining the latest 5 upgrade log files.

Windows Agent 4.1.1 Bug Fixes

This release fixes the following Windows Agent issues:

  • Bug ID 702090: The Windows Agent does not generate events when a monitoring template is chosen with a large set of comma separated event IDs. The current code does not allow more than 50 event IDs or 250 characters, including the comma separator. The root cause of this is that the Windows Event pulling API itself has a limitation on the number of characters. With the current fix, the user is allowed the maximum possible allowed by the API; specifically the user can choose a list of event IDs as long as the number of characters including the comma does not exceed 1,200. The FortiSIEM GUI will enforce this restriction. If you need more than this limit, you can always create multiple monitoring templates.

  • Bug ID 710074: When Windows Event Forwarding is configured, the FortiSIEM Agent running on the forwarded server may sometimes fail to get the message in Security Events. Since the message contains key information, the event collected may not contain any meaningful information. A new API is now used to collect the events from the Windows Forwarded Events folder.

Known Issues

Shutting Down Hardware

On hardware appliances running FortiSIEM 6.6.0 or earlier, FortiSIEM execute shutdown CLI does not work correctly. Please use the Linux shutdown command instead.

Remediation Steps for CVE-2021-44228

Three FortiSIEM modules (SVNLite, phFortiInsightAI and 3rd party ThreatConnect SDK) use Apache log4j version 2.14, 2.13 and 2.8 respectively for logging purposes, and hence are vulnerable to the recently discovered Remote Code Execution vulnerability (CVE-2021-44228).

These instructions specify the steps needed to mitigate this vulnerability without upgrading Apache log4j to the latest stable version 2.16 or higher. Actions need to be taken on the Supervisor and Worker nodes only.

On Supervisor Node

  1. Logon via SSH as root.

  2. Mitigating SVNLite module:

    1. Run the script fix-svnlite-log4j2.sh (here). It will restart SVNlite module with Dlog4j2.formatMsgNoLookups=true option and print the success/failed status.

  3. Mitigating 3rd party ThreatConnect SDK module:

    1. Delete these log4j jar files under /opt/glassfish/domains/domain1/applications/phoenix/lib

      1. log4j-core-2.8.2.jar

      2. log4j-api-2.8.2.jar

      3. log4j-slf4j-impl-2.6.1.jar

  4. Mitigating phFortiInsightAI module:

    1. Delete these log4j jar files under /opt/fortiinsight-ai/lib/

      1. log4j-core-2.13.0.jar

      2. log4j-api-2.13.0.jar

  5. Restart all Java Processes by running: “killall -9 java”

On Worker Node

  1. Logon via SSH as root.

  2. Mitigating phFortiInsightAI module:

    1. Delete these log4j jar files under /opt/fortiinsight-ai/lib/

      1. log4j-core-2.13.0.jar

      2. log4j-api-2.13.0.jar

  3. Restart all Java Processes by running: “killall -9 java”

Issue After 6.2.1 Upgrade

As part of a fix for excessive SSL communication errors between phReportWorker and phReportMaster (Bug 710109) in 6.2.1, the value for count_distinct_precision in the /opt/phoenix/config/phoenix_config.txt file is reduced from 14 to 9 for the Supervisor and Worker nodes. While 6.2.1 fresh install sets the value correctly, 6.2.1 upgrade may keep the old value (14) and fail to set the new value (9). Because of this, you can still have excessive SSL communication errors between phReportWorker and phReportMaster and it may appear that bug 710109 is not fixed.

To fix this issue, follow the instructions in Modification on the Supervisor and Modification on each Worker for your Supervisor and Workers.

Modification on the Supervisor

  1. SSH into the super as root and edit the /opt/phoenix/config/phoenix_config.txt file.

    ssh root@<supervisor FQDN/IP>
    su admin
    vi /opt/phoenix/config/phoenix_config.txt
  2. Find "count_distinct_precision=".

  3. Modify the value to that shown here.

    count_distinct_precision=9 # in range 4-18

  4. Save the configuration.

  5. Stop phRerportMaster/Worker by running the following commands.

    phtools --stop phReportWorker
    phtools --stop phReportMaster
  6. Start phReportMaster/Worker by running the following commands.

    phtools --start phReportMaster
    phtools --start phReportWorker
  7. Monitor Stability by running the following command.

    phstatus

Modification on each Worker

  1. SSH into each Worker as root and edit the /opt/phoenix/config/phoenix_config.txt file by running the following commands.

    ssh root@<Worker FQDN/IP>
    su admin
    vi /opt/phoenix/config/phoenix_config.txt
  2. Find "count_distinct_precision=".

  3. Modify the value to that shown here.

    count_distinct_precision=9 # in range 4-18
  4. Save the configuration.

  5. Stop phReportWorker by running the following command.

    phtools --stop phReportWorker
  6. Start phReportWorker by running the following command.

    phtools --start phReportWorker
  7. Monitor Stability by running the following command.

    phstatus

Elasticsearch Based Deployments Terms Query Limit

In Elasticsearch based deployments, queries containing "IN Group X" are handled using Elastic Terms Query. By default, the maximum number of terms that can be used in a Terms Query is set to 65,536. If a Group contains more than 65,536 entries, the query will fail.

The workaround is to change the “max_terms_count” setting for each event index. Fortinet has tested up to 1 million entries. The query response time will be proportional to the size of the group.

Case 1. For already existing indices, issue the REST API call to update the setting

PUT fortisiem-event-*/_settings
{
  "index" : {
    "max_terms_count" : "1000000"
  }
}

Case 2. For new indices that are going to be created in the future, update fortisiem-event-template so those new indices will have a higher max_terms_count setting

  1. cd /opt/phoenix/config/elastic/7.7

  2. Add "index.max_terms_count": 1000000 (including quotations) to the “settings” section of the fortisiem-event-template.

    Example:

    ...

      "settings": {
        "index.max_terms_count": 1000000,
    

    ...

  3. Navigate to ADMIN > Storage > Online and perform Test and Deploy.

  4. Test new indices have the updated terms limit by executing the following simple REST API call.

    GET fortisiem-event-*/_settings

Whats New in 6.2.1

This document describes new features, bug fixes, and known issue for the FortiSIEM 6.2.1 (build) release.

New Features

Reputation Check from FortiGuard

FortiGuard IOC provides reputation checks for different Malware like IP, Domain, URL, and Hash. This release adds FortiGuard IOC to the supported list of reputation lookup engines. This can be invoked in of two ways.

  • When an incident triggers, users can manually do a lookup via FortiGuard IOC, VirusTotal, or RiskIQ. Any external website can also be looked up, but the results of FortiGuard IOC, VirusTotal or RiskIQ are parsed and the user is provided with a simplified Verdict (Safe, Unsure, or Malicious based on thresholds).

  • Users can automate this reputation check by including this in their Incident Notification Policies. When included, the reputation check will automatically be done when an incident triggers. Incident comments are updated with the results.

Every licensed FortiSIEM can do FortiGuard IOC lookup. No special configuration is required.

For setting up manual external lookup, see FortiGuard IOC Integration.

For automating via notification policy, see Adding Incident Notification Settings.

For information on doing lookup, see Lookups Via External Websites.

Bug Fixes and Minor Enhancements

The current release includes the following bug fixes and minor enhancement:

Bug ID

Severity

Module

Description

710084

Major

App Server

After a period of time, all Windows Agents may become disconnected (error PH_AUDIT_AGENT_DISABLED logs). This happens very rarely.

710715

Major

Elasticsearch

If an empty action was sent to Supervisor on port 7919, the Java Query Server would stop working.

710109

Major

Report Engine

For some baseline reports, COUNT DISTINCT calculations resulted in very large partial inline report results to be transferred from phReportWorker to the phReportMaster process. This sometimes resulted in ACE_SSL errors and caused baseline reports to contan incomplete data, as that shown here. o 2021-03-26T01:05:11.482049-04:00 w2 phReportWorker[7507]: ACE_SSL (7507|8200) error code: 336462231 - error:140E0197:SSL routines:SSL_shutdown:shutdown while in init

Note: This fix works for new installs of 6.2.1, but upgrades to 6.2.1 are still hindered. The upgrade process will be fixed in 6.3.0+ releases. See Known Issue after 6.2.1 Upgrade for workaround.

712172

Minor

Data Manager

Occasionally, the phDataPurger process would terminate because of initialization issues, resulting in Event DB or Elasticsearch storing events disk space management errors.

711564

Minor

Elasticsearch

The phDataManager process may erroneously introduce a new event field called "UnknownIPVersion" when it encounters an empty IP address field. This can occur for DNS lookup failures in Windows Event Forwarding when the parser cannot resolve the original windows server host name (see bug ID 711542). This can cause Elasticsearch insert failures.

707185

Minor

GUI

The SD-WAN Interface Dashboard failed to load.

711542

Minor

Parser

When the reporting IP is empty and the source IP is 127.0.0.1, the phParser may set the source IP to be empty, resulting in an invalid database insertion entry. This can be identified in Windows Event Forwarding cases when the phParser cannot resolve the original windows server host name to its IP.

707524

Minor

System

Upgrade failed if the DNS Server could not reverse lookup its own name/IP.

710541

Minor

System

Upgrade failed if a FortiSIEM node could not ping the CentOS repository os-pkgs-cdn.fortisiem.fortinet.com.

713893

Enhancement

System

During an upgrade, the previous upgrade related ansible log file would be overwritten. Note: Fixed by retaining the latest 5 upgrade log files.

Windows Agent 4.1.1 Bug Fixes

This release fixes the following Windows Agent issues:

  • Bug ID 702090: The Windows Agent does not generate events when a monitoring template is chosen with a large set of comma separated event IDs. The current code does not allow more than 50 event IDs or 250 characters, including the comma separator. The root cause of this is that the Windows Event pulling API itself has a limitation on the number of characters. With the current fix, the user is allowed the maximum possible allowed by the API; specifically the user can choose a list of event IDs as long as the number of characters including the comma does not exceed 1,200. The FortiSIEM GUI will enforce this restriction. If you need more than this limit, you can always create multiple monitoring templates.

  • Bug ID 710074: When Windows Event Forwarding is configured, the FortiSIEM Agent running on the forwarded server may sometimes fail to get the message in Security Events. Since the message contains key information, the event collected may not contain any meaningful information. A new API is now used to collect the events from the Windows Forwarded Events folder.

Known Issues

Shutting Down Hardware

On hardware appliances running FortiSIEM 6.6.0 or earlier, FortiSIEM execute shutdown CLI does not work correctly. Please use the Linux shutdown command instead.

Remediation Steps for CVE-2021-44228

Three FortiSIEM modules (SVNLite, phFortiInsightAI and 3rd party ThreatConnect SDK) use Apache log4j version 2.14, 2.13 and 2.8 respectively for logging purposes, and hence are vulnerable to the recently discovered Remote Code Execution vulnerability (CVE-2021-44228).

These instructions specify the steps needed to mitigate this vulnerability without upgrading Apache log4j to the latest stable version 2.16 or higher. Actions need to be taken on the Supervisor and Worker nodes only.

On Supervisor Node

  1. Logon via SSH as root.

  2. Mitigating SVNLite module:

    1. Run the script fix-svnlite-log4j2.sh (here). It will restart SVNlite module with Dlog4j2.formatMsgNoLookups=true option and print the success/failed status.

  3. Mitigating 3rd party ThreatConnect SDK module:

    1. Delete these log4j jar files under /opt/glassfish/domains/domain1/applications/phoenix/lib

      1. log4j-core-2.8.2.jar

      2. log4j-api-2.8.2.jar

      3. log4j-slf4j-impl-2.6.1.jar

  4. Mitigating phFortiInsightAI module:

    1. Delete these log4j jar files under /opt/fortiinsight-ai/lib/

      1. log4j-core-2.13.0.jar

      2. log4j-api-2.13.0.jar

  5. Restart all Java Processes by running: “killall -9 java”

On Worker Node

  1. Logon via SSH as root.

  2. Mitigating phFortiInsightAI module:

    1. Delete these log4j jar files under /opt/fortiinsight-ai/lib/

      1. log4j-core-2.13.0.jar

      2. log4j-api-2.13.0.jar

  3. Restart all Java Processes by running: “killall -9 java”

Issue After 6.2.1 Upgrade

As part of a fix for excessive SSL communication errors between phReportWorker and phReportMaster (Bug 710109) in 6.2.1, the value for count_distinct_precision in the /opt/phoenix/config/phoenix_config.txt file is reduced from 14 to 9 for the Supervisor and Worker nodes. While 6.2.1 fresh install sets the value correctly, 6.2.1 upgrade may keep the old value (14) and fail to set the new value (9). Because of this, you can still have excessive SSL communication errors between phReportWorker and phReportMaster and it may appear that bug 710109 is not fixed.

To fix this issue, follow the instructions in Modification on the Supervisor and Modification on each Worker for your Supervisor and Workers.

Modification on the Supervisor

  1. SSH into the super as root and edit the /opt/phoenix/config/phoenix_config.txt file.

    ssh root@<supervisor FQDN/IP>
    su admin
    vi /opt/phoenix/config/phoenix_config.txt
  2. Find "count_distinct_precision=".

  3. Modify the value to that shown here.

    count_distinct_precision=9 # in range 4-18

  4. Save the configuration.

  5. Stop phRerportMaster/Worker by running the following commands.

    phtools --stop phReportWorker
    phtools --stop phReportMaster
  6. Start phReportMaster/Worker by running the following commands.

    phtools --start phReportMaster
    phtools --start phReportWorker
  7. Monitor Stability by running the following command.

    phstatus

Modification on each Worker

  1. SSH into each Worker as root and edit the /opt/phoenix/config/phoenix_config.txt file by running the following commands.

    ssh root@<Worker FQDN/IP>
    su admin
    vi /opt/phoenix/config/phoenix_config.txt
  2. Find "count_distinct_precision=".

  3. Modify the value to that shown here.

    count_distinct_precision=9 # in range 4-18
  4. Save the configuration.

  5. Stop phReportWorker by running the following command.

    phtools --stop phReportWorker
  6. Start phReportWorker by running the following command.

    phtools --start phReportWorker
  7. Monitor Stability by running the following command.

    phstatus

Elasticsearch Based Deployments Terms Query Limit

In Elasticsearch based deployments, queries containing "IN Group X" are handled using Elastic Terms Query. By default, the maximum number of terms that can be used in a Terms Query is set to 65,536. If a Group contains more than 65,536 entries, the query will fail.

The workaround is to change the “max_terms_count” setting for each event index. Fortinet has tested up to 1 million entries. The query response time will be proportional to the size of the group.

Case 1. For already existing indices, issue the REST API call to update the setting

PUT fortisiem-event-*/_settings
{
  "index" : {
    "max_terms_count" : "1000000"
  }
}

Case 2. For new indices that are going to be created in the future, update fortisiem-event-template so those new indices will have a higher max_terms_count setting

  1. cd /opt/phoenix/config/elastic/7.7

  2. Add "index.max_terms_count": 1000000 (including quotations) to the “settings” section of the fortisiem-event-template.

    Example:

    ...

      "settings": {
        "index.max_terms_count": 1000000,
    

    ...

  3. Navigate to ADMIN > Storage > Online and perform Test and Deploy.

  4. Test new indices have the updated terms limit by executing the following simple REST API call.

    GET fortisiem-event-*/_settings