Fortinet black logo

FortiSIEM Reference Architecture Using ClickHouse

Home FortiSIEM 7.1.3 FortiSIEM Reference Architecture Using ClickHouse
7.1.3
7.1.6
7.1.5
7.1.4
7.1.3
7.1.2
7.1.1
7.1.0
7.0.3
7.0.2
7.0.1
7.0.0
6.7.9
6.7.8
6.7.7
6.7.6
6.7.5
6.7.4
6.7.3
6.7.2
6.7.1
6.7.0

Design for Future Scalability

Design for Future Scalability

FortiSIEM will become an essential core tool in your SOC environment, and its use will likely grow over time as the corporate infrastructure grows, and as more corporate stakeholders realize the benefits of a SIEM platform and want to utilize it. Designing the FortiSIEM deployment to accommodate this increase in use over time will help to ensure continued system performance and minimize the need to re-architect the system in future.

Key considerations for future scalability include:

  • Number of devices and anticipated EPS over the life of the deployment

  • Performance monitoring requirements over the life of the deployment

  • Log storage requirements, defined by EPS and retention requirements, both online and archive

  • Reporting and query performance

  • Topology changes

  • Changes in organizational and network structure over time

  • Disaster Recovery considerations

FortiSIEM with ClickHouse efficiently processes high EPS rates on a relatively small number of nodes/ shards. This means organizations can cost effectively deploy a solution that will scale to meet the planned maximum number of EPS through the life of the solution.

Some key design requirements to consider in the initial deployment for maximum scalability and minimum administrative overhead in-life:

  • Design and deploy the solution from the outset with the number of shards required to handle the maximum expected EPS during the life of the solution.

  • This is a more efficient approach with less architectural and administrative overhead than adding shards in-life. Refer to the ClickHouse Sizing Guide at https://docs.fortinet.com/product/fortisiem/ for the latest recommendations.

  • Consider adding additional shards if the deployment needs to support very heavy query or reporting.

  • Provide all nodes with enough resources (CPU, memory, etc.). Refer to the ClickHouse Sizing Guide at https://docs.fortinet.com/product/fortisiem/ for the latest recommendations.

  • Some hypervisor platforms provide the option to share vCPU, essentially oversubscribing resources. FortiSIEM nodes should be assigned dedicated CPU resources.

  • Provide the FortiSIEM nodes that are running the ClickHouse database with high performance storage. As the solution grows, the I/O will increase and high-performance storage becomes essential. Failure to provide adequate storage performance will impact analytic reporting performance and possibly insert performance.

  • Deploy enough storage capacity to meet your online data retention requirements for the mid-term. Engage with stakeholders in the organization and consider technical, managerial and compliance requirements when planning retention periods.

  • Form a plan for scaling event storage in the long term.

  • In a virtual appliance solution, the ClickHouse event storage disks can be expanded on the FortiSIEM nodes in each shard to increase storage capacity.

  • FortiSIEM with ClickHouse can archive old events to an NFS server or the use of hot and warm storage within the FortiSIEM ClickHouse configuration can be used.

  • Provide high performance, data center class local area network (LAN) connectivity between the Supervisor and Worker nodes.

Previous
Next

Design for Future Scalability

FortiSIEM will become an essential core tool in your SOC environment, and its use will likely grow over time as the corporate infrastructure grows, and as more corporate stakeholders realize the benefits of a SIEM platform and want to utilize it. Designing the FortiSIEM deployment to accommodate this increase in use over time will help to ensure continued system performance and minimize the need to re-architect the system in future.

Key considerations for future scalability include:

  • Number of devices and anticipated EPS over the life of the deployment

  • Performance monitoring requirements over the life of the deployment

  • Log storage requirements, defined by EPS and retention requirements, both online and archive

  • Reporting and query performance

  • Topology changes

  • Changes in organizational and network structure over time

  • Disaster Recovery considerations

FortiSIEM with ClickHouse efficiently processes high EPS rates on a relatively small number of nodes/ shards. This means organizations can cost effectively deploy a solution that will scale to meet the planned maximum number of EPS through the life of the solution.

Some key design requirements to consider in the initial deployment for maximum scalability and minimum administrative overhead in-life:

  • Design and deploy the solution from the outset with the number of shards required to handle the maximum expected EPS during the life of the solution.

  • This is a more efficient approach with less architectural and administrative overhead than adding shards in-life. Refer to the ClickHouse Sizing Guide at https://docs.fortinet.com/product/fortisiem/ for the latest recommendations.

  • Consider adding additional shards if the deployment needs to support very heavy query or reporting.

  • Provide all nodes with enough resources (CPU, memory, etc.). Refer to the ClickHouse Sizing Guide at https://docs.fortinet.com/product/fortisiem/ for the latest recommendations.

  • Some hypervisor platforms provide the option to share vCPU, essentially oversubscribing resources. FortiSIEM nodes should be assigned dedicated CPU resources.

  • Provide the FortiSIEM nodes that are running the ClickHouse database with high performance storage. As the solution grows, the I/O will increase and high-performance storage becomes essential. Failure to provide adequate storage performance will impact analytic reporting performance and possibly insert performance.

  • Deploy enough storage capacity to meet your online data retention requirements for the mid-term. Engage with stakeholders in the organization and consider technical, managerial and compliance requirements when planning retention periods.

  • Form a plan for scaling event storage in the long term.

  • In a virtual appliance solution, the ClickHouse event storage disks can be expanded on the FortiSIEM nodes in each shard to increase storage capacity.

  • FortiSIEM with ClickHouse can archive old events to an NFS server or the use of hot and warm storage within the FortiSIEM ClickHouse configuration can be used.

  • Provide high performance, data center class local area network (LAN) connectivity between the Supervisor and Worker nodes.

Previous
Next