Fortinet black logo

FortiSIEM Reference Architecture Using ClickHouse

Home FortiSIEM 7.1.3 FortiSIEM Reference Architecture Using ClickHouse
7.1.3
7.1.6
7.1.5
7.1.4
7.1.3
7.1.2
7.1.1
7.1.0
7.0.3
7.0.2
7.0.1
7.0.0
6.7.9
6.7.8
6.7.7
6.7.6
6.7.5
6.7.4
6.7.3
6.7.2
6.7.1
6.7.0

Collector Architectures

Collector Architectures

Although many log types can be ingested directly by the Worker or Supervisor nodes, most deployments benefit significantly from Collector nodes, and they should be used wherever possible. Collector nodes have the following key functions:

  1. Collectors are mandatory for some features, including:

    1. Ingesting logs from FortiSIEM agents

    2. Per-customer log separation in most MSSP scenarios

  2. Collectors also improve performance and functionality by assisting with the following:

    1. Perform local device discovery at remote sites

    2. Offload resource intensive SNMP and WMI/OMI jobs from the supervisor/worker nodes

    3. Ingest syslog and perform log pre-processing

    4. Compression and secure (TLS) upload to the central cluster

    5. Short term log caching in the event of a loss of connectivity to the central cluster

Smaller installations can make use of one or two Collector nodes to perform agent log collection, offload SNMP and WMI jobs from the supervisor, and support remote sites. Large installations benefit from Collector nodes across the organization, including:

  • Remote sites

  • Data centers and server rooms

  • MSSP multi-tenant and per-organization collectors

One or more Collectors should be deployed at any remote sites with significant log collection or performance monitoring requirements.

Collectors should be deployed in data centers or server rooms to offload high density, intensive server and core device monitoring via SNMP and WMI. They are also a critical component of the FortiSIEM agent architecture and are required for agent log ingestion.

In a service provider deployment, Collectors should also be deployed on a per-customer basis for onsite log ingestion and customer log separation. In service provider mode, FortiSIEM allows the MSSP to assign one or more dedicated collectors to each customer, and any logs received by a customer’s Collector(s) is assigned to that customer in the database. Per-organization collectors are also essential to allow the system to support overlapping IP address ranges found in most multi-tenant environments.

Service providers can also make use of multi-tenant Collectors in certain scenarios. A multi-tenant Collector supports ingestion of certain types of logs across multiple organizations, including:

  • Ingestion of logs from shared devices, such as service provider core firewalls. This is achieved using the event-organization mapping feature that assigns a value, such as VDOM label, to a specific organization.

  • Cloud API log pulling

  • Agent log ingestion, providing the organization doesn't have any dedicated per-organization collectors assigned

Load balancers can be deployed in front of Collectors in large deployments. They particularly benefit the following scenarios:

  • Environments with very high EPS core devices, such as large enterprise or service provider core firewalls and routers

  • Environments with a high density of devices, such as large server rooms and data centers

Each Collector has a maximum log ingestion rate of up to 10k EPS. Some devices may exceed this log rate in very large enterprise core and service provider environments. These devices can be supported by deploying several Collectors with a load balancer in front of them, presenting a single virtual IP to the device as a syslog target, but sharing the logs across multiple collectors on the back end, as shown below:

This architecture also provides the option for Collector resilience, if n+1 collectors are deployed behind the load balancer and the load balancer is performing host availability checking with traffic redirection in the event of a node failure, there should be minimal event loss if one Collector fails.

The Collector load-balancer architecture can also be used to simplify deployment in high density environments, such as server rooms or data centers, where the total inbound log rate exceeds that supported by a single Collector. The load balancer virtual IP can be used as a single target IP for inbound syslog in the log source configuration. When FortiSIEM agents are used, the Virtual Collector feature can be used to redirect the agents to the load balancer virtual IP rather than to an individual Collector. Note that a load balancer is optional in this scenario; devices can be manually assigned across Collectors if a load balancer is not available.

Previous
Next

Collector Architectures

Although many log types can be ingested directly by the Worker or Supervisor nodes, most deployments benefit significantly from Collector nodes, and they should be used wherever possible. Collector nodes have the following key functions:

  1. Collectors are mandatory for some features, including:

    1. Ingesting logs from FortiSIEM agents

    2. Per-customer log separation in most MSSP scenarios

  2. Collectors also improve performance and functionality by assisting with the following:

    1. Perform local device discovery at remote sites

    2. Offload resource intensive SNMP and WMI/OMI jobs from the supervisor/worker nodes

    3. Ingest syslog and perform log pre-processing

    4. Compression and secure (TLS) upload to the central cluster

    5. Short term log caching in the event of a loss of connectivity to the central cluster

Smaller installations can make use of one or two Collector nodes to perform agent log collection, offload SNMP and WMI jobs from the supervisor, and support remote sites. Large installations benefit from Collector nodes across the organization, including:

  • Remote sites

  • Data centers and server rooms

  • MSSP multi-tenant and per-organization collectors

One or more Collectors should be deployed at any remote sites with significant log collection or performance monitoring requirements.

Collectors should be deployed in data centers or server rooms to offload high density, intensive server and core device monitoring via SNMP and WMI. They are also a critical component of the FortiSIEM agent architecture and are required for agent log ingestion.

In a service provider deployment, Collectors should also be deployed on a per-customer basis for onsite log ingestion and customer log separation. In service provider mode, FortiSIEM allows the MSSP to assign one or more dedicated collectors to each customer, and any logs received by a customer’s Collector(s) is assigned to that customer in the database. Per-organization collectors are also essential to allow the system to support overlapping IP address ranges found in most multi-tenant environments.

Service providers can also make use of multi-tenant Collectors in certain scenarios. A multi-tenant Collector supports ingestion of certain types of logs across multiple organizations, including:

  • Ingestion of logs from shared devices, such as service provider core firewalls. This is achieved using the event-organization mapping feature that assigns a value, such as VDOM label, to a specific organization.

  • Cloud API log pulling

  • Agent log ingestion, providing the organization doesn't have any dedicated per-organization collectors assigned

Load balancers can be deployed in front of Collectors in large deployments. They particularly benefit the following scenarios:

  • Environments with very high EPS core devices, such as large enterprise or service provider core firewalls and routers

  • Environments with a high density of devices, such as large server rooms and data centers

Each Collector has a maximum log ingestion rate of up to 10k EPS. Some devices may exceed this log rate in very large enterprise core and service provider environments. These devices can be supported by deploying several Collectors with a load balancer in front of them, presenting a single virtual IP to the device as a syslog target, but sharing the logs across multiple collectors on the back end, as shown below:

This architecture also provides the option for Collector resilience, if n+1 collectors are deployed behind the load balancer and the load balancer is performing host availability checking with traffic redirection in the event of a node failure, there should be minimal event loss if one Collector fails.

The Collector load-balancer architecture can also be used to simplify deployment in high density environments, such as server rooms or data centers, where the total inbound log rate exceeds that supported by a single Collector. The load balancer virtual IP can be used as a single target IP for inbound syslog in the log source configuration. When FortiSIEM agents are used, the Virtual Collector feature can be used to redirect the agents to the load balancer virtual IP rather than to an individual Collector. Note that a load balancer is optional in this scenario; devices can be manually assigned across Collectors if a load balancer is not available.

Previous
Next