Design for Log Collection Efficiency

Key considerations for efficient log collection are:

• Which logs to collect

• The log collection method

• The log ingestion architecture

Organizations new to SIEM technology may aim to collect all device logs across the entire organization, leading to many useless logs being collected. This impacts the deployment and creates:

• An unaffordable deployment with excessive license and hardware costs

• An unmanageable volume of logs to store

• An unmanageable volume of logs to analyze and process

When planning the deployment, consider the monitoring use-case(s) and the end technical and business goal. Then identify the most important log sources and log types that should be monitored to achieve this and focus on these. This will efficiently provide the visibility required by the use-case. This is discussed further in the Collecting Logs section.

The collection method is another important consideration for efficient log collection. When planning Windows server log collection, agentless WMI or OMI collection is an attractive option due to perceived cost savings. However, WMI and OMI are inefficient protocols:

• They are ‘pull’ protocols, which places additional overhead on FortiSIEM and may require additional Collector nodes

• They do not work well for high log rate servers

Using Windows agents for log collection provides a more efficient log collection solution. Agents provide many benefits:

• Efficient, high performance and secure log upload to a collector

• Flexible log collection policies

• Additional functionality, including file integrity monitoring, registry change detection, custom log file ingestion and more

Properly deploying collectors in the organization will improve log collection efficiency and system performance:

• Deploy collectors on remote sites to ingest logs locally, pre-process, compress and securely upload them to the central supervisor or worker nodes

• Deploy collectors at the main site to perform log ingestion and performance monitoring, removing load from the Supervisor/Worker cluster and increasing scalability

• Deploy collectors in the data center to perform server log collection, especially when using WMI or OMI based agentless log collection or server performance monitoring