What's New in 6.7.5

• Important Notes

• Key Enhancements
• Bug Fixes and Enhancements

• Known Issues

Important Notes

1. FortiSIEM 6.7.5 and later API documentation is transitioning to https://fndn.fortinet.net/index.php?/fortiapi/2627-fortisiem/. Fortinet recommends checking this link first for the latest API updates.

2. 5.x Collector will not work with FortiSIEM 6.7.2 or later. This step is taken for improved security. Follow these steps to make the 5.x Collectors operational after upgrade.

   1. Upgrade the Supervisor to the latest version: 7.0.0 or higher.

   2. Copy phProvisionCollector.collector from the Supervisor to all 5.x Collectors.

      1. Login to Supervisor.

      2. Run the following command.
         

         scp /opt/phoenix/phscripts/bin/phProvisionCollector.collector root@<Collector_IP>:/opt/phoenix/bin/phProvisionCollector

   3. Update 5.x Collector password.

      1. SSH to the Collector.

      2. Run the following command.

         phProvisionCollector --update <Organization-user-name> <Organization-user-password> <Supervisor-IP> <Organization-name> <Collector-name>

      3. Make sure the Collector ID and password are present in the file /etc/httpd/accounts/passwds on Supervisors and Workers.

   4. Reboot the Collector.

Key Enhancements

This release includes published Rocky Linux OS updates until May 5, 2023. The list of updates can be found here - https://errata.rockylinux.org/. FortiSIEM Rocky Linux Repositories (os-pkgs-cdn.fortisiem.fortinet.com and os-pkgs-r8.fortisiem.fortinet.com) have also been updated to the latest. Therefore, FortiSIEM customers in versions 6.4.1 and above, can upgrade only their Rocky Linux versions by following the procedures described in https://docs.fortinet.com/document/fortisiem/6.7.3/fortisiem-os-update-procedure/574280/fortisiem-os-update-procedure.

Bug Fixes and Enhancements

Bug ID	Severity	Module	Description
913785	Major	App Server	CMDB PostGRESQL tables related to Supervisor/Worker/Collector health are not purged daily. This may result in CMDB PostGRESQL database to be eventually become full and slow down CMDB backups. The rate of growth depends on the number of Workers and Collectors in your system. As part of the fix, the table is trimmed during the upgrade process, and purged daily from then onwards.
903987, 899971	Major	App Server	Org user with custom role can see other org info when running CMDB reports.
913375	Major	DataPurger	For EventDB based deployments, the Event Integrity feature does not work because of a code change in an earlier 6.7 release.
912874	Minor	App Server	Quick Info > External IP lookup fails, since the Whois server whois.geektools.com has been recently shut down. FortiSIEM uses whois.geektools.com as the default Whois server and does not properly parse the error response returned by the server.
912616	Minor	App Server	The Event Query API /phoenix/rest/query/eventQuery does not return query results in a consistent format. This API was enhanced in release 6.7.0 to provide a detailed reason when a query does not have results.
907396	Minor	App Server	LDAP Discovery fails if LDAP user has email but no name.
907187	Minor	App Server	Rule exception is not being saved when creating the rule from ANALYTICS. There is no issue when creating a rule from RESOURCES.
907009	Minor	App Server	Rule exception is not being saved when a System Rule is cloned and an exception is added before saving the cloned rule.
903042	Minor	App Server	If an active incident from a previous month triggers again at the turn of the next month, a duplicate ServiceNow ticket is created, and the incident in FortiSIEM is updated with the new ServiceNow ID.
876941	Minor	App Server	Service Provider deployment - Collector registration for an org may fail under some situations, when ph_user_domain_mapping table contains per-Org users.
863741	Minor	App Server	Incident queries may take a long time to run if there are lots of associated Tickets/Cases.
825485	Minor	App Server	Watchlist APIs (rest/watchlist) do not correctly return org level entries.
913611	Minor	ClickHouse	In a ClickHouse based deployment, if you want to remove the Supervisor node from the ClickHouse cluster and the Supervisor's ClickHouse database is more than 50GB, then ClickHouse may not be able to start.
889759	Minor	ClickHouse	If you want to delete a entire ClickHouse shard, then this operation may fail with the wrong error message stating incomplete replication.
906326	Minor	System	FortiSIEM deployment on Nutanix virtual platform fails.
903788	Minor	System	modsec_audit log is recorded under /var/log/httpd when Collectors fail to send events to busy Workers. This operation may fill up root disk. Since these are not attacks, these failures should not be logged.
910251	Enhancement	ClickHouse	Memory usage of phDataManager module needs to be reduced when total memory is 32GB.
911625	Enhancement	Data	Computationally expensive rules are optimized by moving restriction conditions to the front in the Filter condition.

Known Issues

• ClickHouse Related

• Elasticsearch Related

• EventDB Related

• HDFS Related

• High Availability Related

ClickHouse Related

1. If you are running ClickHouse event database and want to do Active-Active Supervisor failover, then your Supervisor should not be the only ClickHouse Keeper node. In that case, once the Supervisor is down, the ClickHouse cluster will be down and inserts will fail. It is recommended that you have 3 ClickHouse Keeper nodes running on Workers.

2. If you are running ClickHouse, then during a Supervisor upgrade to FortiSIEM 6.7.0 or later, instead of shutting down Worker nodes, you need to stop the backend processes by running the following command from the command line.
   

   phtools --stop all
   

3. If you are running Elasticsearch or FortiSIEM EventDB and switch to ClickHouse, then you need to follow two steps to complete the database switch.

   1. Set up the disks on each node in ADMIN > Setup> Storage and ADMIN > License > Nodes.

   2. Configure ClickHouse topology in ADMIN > Settings > Database > ClickHouse Config.

4. In a ClickHouse environment, Queries will not return results if none of the query nodes within a shard are reachable from Supervisor and responsive. In other words, if at least 1 query node in every shard is healthy and responds to queries, then query results will be returned. To avoid this condition, make sure all Query Worker nodes are healthy.

Elasticsearch Related

1. In Elasticsearch based deployments, queries containing "IN Group X" are handled using Elastic Terms Query. By default, the maximum number of terms that can be used in a Terms Query is set to 65,536. If a Group contains more than 65,536 entries, the query will fail.

   The workaround is to change the “max_terms_count” setting for each event index. FortiSIEM has been tested up to 1 million entries. The query response time will be proportional to the size of the group.
   

   Case 1. For already existing indices, issue the REST API call to update the setting

   PUT fortisiem-event-*/_settings
   {
     "index" : {
       "max_terms_count" : "1000000"
     }
   }
   

   Case 2. For new indices that are going to be created in the future, update fortisiem-event-template so those new indices will have a higher max_terms_count setting

   1. cd /opt/phoenix/config/elastic/7.7

   2. Add "index.max_terms_count": 1000000 (including quotations) to the “settings” section of the fortisiem-event-template.
      

      Example:

      ...

        "settings": {
          "index.max_terms_count": 1000000,
      

      ...

   3. Navigate to ADMIN > Storage > Online and perform Test and Deploy.

   4. Test new indices have the updated terms limit by executing the following simple REST API call.

      GET fortisiem-event-*/_settings

2. FortiSIEM uses dynamic mapping for Keyword fields to save Cluster state. Elasticsearch needs to encounter some events containing these fields before it can determine their type. For this reason, queries containing group by on any of these fields will fail if Elasticsearch has not seen any event containing these fields. Workaround is to first run a non-group by query with these fields to make sure that these fields have non-null haves.

EventDB Related

Currently, Policy based retention for EventDB does not cover two event categories: (a) System events with phCustId = 0, e.g. a FortiSIEM External Integration Error, FortiSIEM process crash etc., and (b) Super/Global customer audit events with phCustId = 3, e.g. audit log generated from a Super/Global user running an adhoc query. These events are purged when disk usage reaches high watermark.

HDFS Related

If you are running real-time Archive with HDFS, and have added Workers after the real-time Archive has been configured, then you will need to perform a Test and Deploy for HDFS Archive again from the GUI. This will enable HDFSMgr to know about the newly added Workers.

High Availability Related

If you make changes to the following files on any node in the FortiSIEM Cluster, then you will have to manually copy these changes to other nodes.

1. FortiSIEM Config file (/opt/phoenix/config/phoenix_config.txt): If you change a Supervisor (respectively Worker, Collector) related change in this file, then the modified file should be copied to all Supervisors (respectively Workers, Collectors).

2. FortiSIEM Identity and Location Configuration file (/opt/phoenix/config/identity_Def.xml): This file should be identical in Supervisors and Workers. If you make a change to this file on any Supervisor or Worker, then you need to copy this file to all other Supervisors and Workers.

3. FortiSIEM Profile file (ProfileReports.xml): This file should be identical in Supervisors and Workers. If you make a change to this file on any Supervisor or Worker, then you need to copy this file to all other Supervisors and Workers.

4. SSL Certificate (/etc/httpd/conf.d/ssl.conf): This file should be identical in Supervisors and Workers. If you make a change to this file on any Supervisor or Worker, then you need to copy this file to all other Supervisors and Workers.

5. Java SSL Certificates (files cacerts.jks, keyfile and keystore.jks under /opt/glassfish/domains/domain1/config/): If you change these files on a Supervisor, then you have to copy these files to all Supervisors.

6. Log pulling External Certificates: Copy all log pulling external certificates to each Supervisor.

7. Event forwarding Certificates define in FortiSIEM Config file (/opt/phoenix/config/phoenix_config.txt): If you change on one node, you need to change on all nodes.

8. Custom cron job: If you change this file on a Supervisor, then you have to copy this file to all Supervisors.