Analytics and Archive logs
Logs in FortiAnalyzer are in one of the following phases.
- Real-time log: Log entries that have just arrived and have not been added to the SQL database. These logs are stored in Archive in an uncompressed file.
- Archive logs: When a real-time log file in Archive has been completely inserted, that file is compressed and considered to be offline.
- Analytics logs or historical logs: Indexed in the SQL database and online.
Use a data policy to control how long to retain Analytics and Archive logs.
Archive logs
When FortiAnalyzer receives a log, it is stored in a file. Logs will continue to populate this file until its limit is reached, at which time the file is "rolled" which involves compressing the file and creating a new one for further logs of that type. These files (rollled or otherwise) count against the archive retention limits and are referred to as Archived or Offline logs.
You cannot immediately view details about these logs in the FortiView > FortiView, Log View, and Incidents & Events/FortiSoC panes. You also cannot generate reports about the logs in the Reports pane.
Archive logs are stored unchanged and can be uploaded to a file server for use as backups.
- If you are using a FortiAnalyzer-VM, you may also choose to snapshot the data drive to backup your logs.
- If you are using a physical FortiAnalyzer which leverages RAID for storage, remember that RAID is not a backup solution.
Log storage in Archive is important since it is used to rebuild the database in the event of database corruption, or in some cases during upgrades.
Analytic logs
Immediately following the storage of a log in an archive, the same log is inserted into the SQL database. This function is also known as being indexed, and these logs are referred to as Analytic or Online logs.
Analytic logs are the only logs which are used for analysis in FortiAnalyzer FortiSoC, Log View (excluding Log Browse), Incidents and Events, and Reports.
Analytic logs are dissected during insertion and any subtypes are stored as their own category. For example, security profile logs such as web filtering logs are sent and stored as Traffic logs when archived, however, Analytics extracts the relevant web filtering fields and stores them in a web filtering table.
Indexed logs take up significantly more space than the same amount of logs in Archive.
Most administrators may need to store between 30 and 60 days in Analytics, however, this should be configured for the amount of time that you would typically need to explore the logs for.
If you need to run analytics for dates outside your Analytics retention, you may perform a database rebuild and load the particular date range. A database rebuild involves purging all logs from Analytics and loading logs for the days of interest from Archive. Once analysis is complete, you can then rebuild once more to load the most current logs into analytics from the archive.