Fortinet black logo

Switching Reference Architecture Guide

Home FortiSwitch 7.2.3 Switching Reference Architecture Guide

Network design principles

7.2.3
7.4.0
7.2.3
Copy Link
Copy Doc ID 05143578-c74d-11ed-8e6d-fa163e15d75b:527614
Download PDF

Network design principles

There has always been a trade-off when designing networks. A network cannot allocate the maximum traffic that could be generated by each connected device at peak all at the same time. Oversubscription must be considered when designing a network.

Additionally, the network design needs to consider the current and future traffic demands to avoid becoming obsolete in a short period of time. This is a critical factor to consider with the introduction of more and more wired and wireless devices connected to the networks, the newest WiFi 6E (802.11ax) spectrum that could potentially offer multigigabit access to a single network access device, and even the adoption of access ports for end devices at 2.5 GbE or 5 GbE. In the data center, server connectivity at the 25-GbE speed is very common.

Redundancy and resiliency form both the third and fourth pillars of network design. The requirements for fast convergence, assurance of critical services, and avoiding single points of failure while maintaining the service level agreements (SLAs) must be clearly considered in detail when designing any network.

Dimensioning

When you build a multi-tiered network, you need to consider the bandwidth oversubscription ratios for every layer of the switching hierarchy. The upstream bandwidth at each layer must provide enough bandwidth for the added traffic of each active device under it. Nevertheless, not all devices transmit or receive data at their nominal bandwidth, nor at the same time, so the ratios that make the total size of the uplink do not need to be the sum of the total amount of downstream links. This “oversubscription” ratio of uplinks must be closely followed to avoid network bottlenecks, which could cause poor network connectivity for downstream devices.

A ratio of 20:1 used to be a common downlink-to-uplink access ratio. Downlink-to-uplink aggregation ratios were 4:1 when using 1 GbE at the access layer with 2x10-GbE uplinks to the aggregation layer and 2x10-GbE uplinks to the core layer. This was when networks were designed with multiple Ethernet ports per desk and multiple ports in conference rooms and shared areas. In networks where more than 75 percent of the devices are now wireless and IoT devices continue to enter the enterprise, the number of wired ports in the network is getting close to one per user.

Moreover, with the advent of new hardware, cheaper fiber runs, and higher performing chipsets, common deployments of access switches now use 4x10-GbE links (sometimes even 2x40-GbE links) to a set of aggregation switches. The aggregation switches then send traffic from the aggregation layer to a core layer through up to 4x100-GbE links (towards two core switches) and then connect the core switches to the FortiGate devices for the core security services; the routing uses 100-Gbps links.

Hence, the common access-to-aggregation ratios are now more around 10:1 when you use gigabit Ethernet ports at the access layer, and the common aggregation-to-core ratios are around 2:1. Fortinet recommends these ratios when you design modern campuses for a refresh or a new building.

The calculation becomes more difficult when mixing multigigabit ports for access points and gigabit Ethernet for access devices like video surveillance cameras, IP phones, and video conferencing setups.

The most important consideration is what the ratio becomes if one of the devices fails. As you add more switches to a floor, you must keep in mind the distribution of the uplinks across switches, and the impact on oversubscription during failure. For access points that are dual attached to two different switches (see the figure in Tiered architecture), the network connectivity is kept. In a scenario where you lose one aggregation switch in the path, you are fully covered by the design described here because you have redundant links from the access layer to the aggregation layer (using 2x40-Gbps links or 4x10-Gbps links). Losing one of the core switches is also covered because the links from the aggregation layer to the core layer are 2x100 Gbps or 4x100 Gbps, still providing an acceptable 4:1 statistical ratio for the duration of the outage instead of a ratio of 2:1.

Quality of service

QoS allows you to set particular priorities for different applications, users, or data flows. FortiSwitch units support the following QoS capabilities:

  • Mapping the IEEE 802.1p and layer-3 QoS values (differentiated services and IP precedence) to an outbound QoS queue number.

  • Providing eight egress queues on each port.

  • Policing the maximum data rate of egress traffic on the interface.

  • Security is provided as part of the zero-trust network access because QoS coming from the devices is not trusted. Each traffic packet is remarked according to the switch classification. DoS potential threats are minimized by implementing this QoS control mechanism.

QoS involves the following elements:

  • Classification—This process determines the priority of a packet. This can be as simple as trusting the QoS markings in the packet header when it is received so that the packet is accepted. Alternatively, it can use such criteria as the incoming port, VLAN, or service that are defined by the network administrator.

  • Marking—This process involves setting bits in the packet header to indicate the priority of this packet.

  • Queuing—This process involves defining priority queues to ensure that packets marked as high priority take precedence over those marked as lower priority. If network congestion becomes so severe that dropping packet is necessary, the queuing process selects which packets to drop.

Fortinet recommends implementing queuing on switches because oversubscription might result in congestion and cause packet loss, especially during an outage. Queuing ensures that your higher priority traffic like voice conferencing and video conferencing is statistically less impacted than lower priority traffic, which is more immune to packet loss. If you select weighted-random-early-detection for the drop policy, you can enable explicit congestion notification (ECN) marking to indicate that congestion is occurring without just dropping packets.

FortiSwitch units can enable many different scenarios where QoS can be adjusted for rich multimedia and multicast handling in your campus network.

FortiSwitch units can parse LLDP (LLDP-MED) messages from voice devices such as FortiFone units and pass this information to a FortiGate device for device detection. FortiSwitchOS proposes an automatic FortiLink voice default VLAN for voice devices, and you can use a dynamic port policy to assign a device to an LLDP profile, QoS policy, and VLAN policy. When a detected device is matched to the dynamic port policy, the corresponding policy actions are applied on the switch port.

Fortinetʼs ability to inspect all traffic at the core, even encrypted packets, allows you to excel at application awareness and high quality of experience (QoE) delivery for your workforce.

Resiliency

When considering design recommendations, a critical point is resiliency: avoiding single points of failure and ensuring a fast convergence to minimize service downtime. The following sections show how resiliency is accomplished in each layer of the reference architecture described in Tiered architecture.

Tier-1/core layer resiliency

At this layer, resiliency is achieved by many different features. At the FortiGate level, the FortiGate Clustering Protocol (FGCP) provides failover protection because a cluster can provide FortiGate services even when one of the devices in the cluster encounters a problem (such as link failure, power loss, or memory or SSD failures). FortiGate devices can be configured in active-passive or active-active modes; refer to High Availability for further information. FortiGate clusters can be integrated into the load-balancing configuration using the FortiGate Session Life Support Protocol (FGSP) in a network where traffic is load balanced.

The following figure shows the redundancy mechanisms for the tier-1/core layer.

From the FortiSwitch perspective, resiliency is achieved by the following:

  • Building MCLAG pairs. All links are enabled and forward traffic without loops, reducing the convergence times in case a link fails. To protect the MCLAG configuration, Fortinet suggests using two links between the FortiSwitch units and enabling MCLAG split interface on the MCLAG pair.

  • Traffic load balancing. By default, FortiSwitch units load balance the traffic among all available uplinks. The default algorithm is based on the source-destination IP addresses, but it can be modified for other supported methods. Load balancing will work alongside the FGSP.

  • Configuring a full mesh between this first layer of FortiSwitch units and the FortiGate devices, regardless of the HA configuration, results in the maximum bandwidth and minimum convergence time.

  • Uplinks could be also formed by the aggregation of several links (link aggregation group [LAG]), which will increase the capacity and resiliency.

Tier-2/aggregation layer resiliency

At this layer, formed entirely by FortiSwitch units in MCLAG pairs, similar recommendations to the ones stated for the core layer are applied: enabling MCLAG pairs with link redundancy and full mesh wiring towards the core and access layers with LAGs.

The following figure shows the redundancy mechanisms of the tier-2/aggregation layer.

Tier-3/access layer resiliency

For the first pair of FortiSwitch units forming an MCLAG pair (the tier-3 MCLAG pair), the same recommendations as in the previous two layers are applied.

The following figure shows the redundancy mechanisms of the tier-3/access layer.

For the last meterʼs design:

  • If the last meterʼs design comprises a single FortiSwitch unit and single FortiAP unit, Fortinet recommends to connect them dual homed to the access MCLAG pair with single links or LAGs.

    The following figure shows the last meterʼs redundancy for a single FortiSwitch unit and single FortiAP unit.

  • If the last meterʼs design comprises a ring of FortiSwitch units, FortiLink automatically forms for ring topologies and takes care of the spanning tree configuration by default. The ring should have each end connected to a different access MCLAG FortiSwitch unit, and the inter-switch links could comprise one or multiple interfaces (LAGs). Other mechanisms can be put in place to harden the stability and resiliency of the access layer, such as loop protection and STP BPDU guard.

    The following figure shows the last meterʼs redundancy for a ring of FortiSwitch units.

Future proofing

The number of connected devices and the bandwidth consumption increases at a very fast pace. According to Nielsenʼs Law for Internet Bandwidth, Internet bandwidth is supposed to grow up to 50% year over year.

That number could be overwhelming in some situations, especially when planning the design for a network, but there are many details to be considered. Each network and its needs will vary from deployment to deployment.

When designing campus networks, there are many factors that impact the overall performance. The design must be planned to support the demands of the near future, benefit from the return of investment, and try to delay the changes needed in the near future.

  • Consider the device connectivity needs for the next 5 years, such as wired devices using native 2.5 GbE or 5 GbE and the rising popularity of wireless devices.

  • In the data center, server connectivity at the 25-GbE speed.

  • Make it easily scalable for wired and wireless access. Adding more capacity, at least at the access layer should not be difficult with this design, nor technically complicated. With FortiAP and FortiSwitch units, adding new elements to the topology is done seamlessly and easily, thanks to the single-pane-of-glass control from the FortiGate device and FortiManager unit.

  • Replacing existing infrastructure to increase performance or network capacity should not require starting from the beginning. Replacing existing FortiGate devices, FortiSwitch units, or FortiAP units is as simple as executing a few commands, and you do not have to start the configuration all over.

  • The enhanced visibility obtained by the tight integration of the Fortinet Security Fabric also helps to monitor the network trends and plan in advance for the near-future needs.

The following figure shows how you can use the FortiGate single-pane-of-glass interface to monitor network usage.

Previous
Next

Network design principles

There has always been a trade-off when designing networks. A network cannot allocate the maximum traffic that could be generated by each connected device at peak all at the same time. Oversubscription must be considered when designing a network.

Additionally, the network design needs to consider the current and future traffic demands to avoid becoming obsolete in a short period of time. This is a critical factor to consider with the introduction of more and more wired and wireless devices connected to the networks, the newest WiFi 6E (802.11ax) spectrum that could potentially offer multigigabit access to a single network access device, and even the adoption of access ports for end devices at 2.5 GbE or 5 GbE. In the data center, server connectivity at the 25-GbE speed is very common.

Redundancy and resiliency form both the third and fourth pillars of network design. The requirements for fast convergence, assurance of critical services, and avoiding single points of failure while maintaining the service level agreements (SLAs) must be clearly considered in detail when designing any network.

Dimensioning

When you build a multi-tiered network, you need to consider the bandwidth oversubscription ratios for every layer of the switching hierarchy. The upstream bandwidth at each layer must provide enough bandwidth for the added traffic of each active device under it. Nevertheless, not all devices transmit or receive data at their nominal bandwidth, nor at the same time, so the ratios that make the total size of the uplink do not need to be the sum of the total amount of downstream links. This “oversubscription” ratio of uplinks must be closely followed to avoid network bottlenecks, which could cause poor network connectivity for downstream devices.

A ratio of 20:1 used to be a common downlink-to-uplink access ratio. Downlink-to-uplink aggregation ratios were 4:1 when using 1 GbE at the access layer with 2x10-GbE uplinks to the aggregation layer and 2x10-GbE uplinks to the core layer. This was when networks were designed with multiple Ethernet ports per desk and multiple ports in conference rooms and shared areas. In networks where more than 75 percent of the devices are now wireless and IoT devices continue to enter the enterprise, the number of wired ports in the network is getting close to one per user.

Moreover, with the advent of new hardware, cheaper fiber runs, and higher performing chipsets, common deployments of access switches now use 4x10-GbE links (sometimes even 2x40-GbE links) to a set of aggregation switches. The aggregation switches then send traffic from the aggregation layer to a core layer through up to 4x100-GbE links (towards two core switches) and then connect the core switches to the FortiGate devices for the core security services; the routing uses 100-Gbps links.

Hence, the common access-to-aggregation ratios are now more around 10:1 when you use gigabit Ethernet ports at the access layer, and the common aggregation-to-core ratios are around 2:1. Fortinet recommends these ratios when you design modern campuses for a refresh or a new building.

The calculation becomes more difficult when mixing multigigabit ports for access points and gigabit Ethernet for access devices like video surveillance cameras, IP phones, and video conferencing setups.

The most important consideration is what the ratio becomes if one of the devices fails. As you add more switches to a floor, you must keep in mind the distribution of the uplinks across switches, and the impact on oversubscription during failure. For access points that are dual attached to two different switches (see the figure in Tiered architecture), the network connectivity is kept. In a scenario where you lose one aggregation switch in the path, you are fully covered by the design described here because you have redundant links from the access layer to the aggregation layer (using 2x40-Gbps links or 4x10-Gbps links). Losing one of the core switches is also covered because the links from the aggregation layer to the core layer are 2x100 Gbps or 4x100 Gbps, still providing an acceptable 4:1 statistical ratio for the duration of the outage instead of a ratio of 2:1.

Quality of service

QoS allows you to set particular priorities for different applications, users, or data flows. FortiSwitch units support the following QoS capabilities:

  • Mapping the IEEE 802.1p and layer-3 QoS values (differentiated services and IP precedence) to an outbound QoS queue number.

  • Providing eight egress queues on each port.

  • Policing the maximum data rate of egress traffic on the interface.

  • Security is provided as part of the zero-trust network access because QoS coming from the devices is not trusted. Each traffic packet is remarked according to the switch classification. DoS potential threats are minimized by implementing this QoS control mechanism.

QoS involves the following elements:

  • Classification—This process determines the priority of a packet. This can be as simple as trusting the QoS markings in the packet header when it is received so that the packet is accepted. Alternatively, it can use such criteria as the incoming port, VLAN, or service that are defined by the network administrator.

  • Marking—This process involves setting bits in the packet header to indicate the priority of this packet.

  • Queuing—This process involves defining priority queues to ensure that packets marked as high priority take precedence over those marked as lower priority. If network congestion becomes so severe that dropping packet is necessary, the queuing process selects which packets to drop.

Fortinet recommends implementing queuing on switches because oversubscription might result in congestion and cause packet loss, especially during an outage. Queuing ensures that your higher priority traffic like voice conferencing and video conferencing is statistically less impacted than lower priority traffic, which is more immune to packet loss. If you select weighted-random-early-detection for the drop policy, you can enable explicit congestion notification (ECN) marking to indicate that congestion is occurring without just dropping packets.

FortiSwitch units can enable many different scenarios where QoS can be adjusted for rich multimedia and multicast handling in your campus network.

FortiSwitch units can parse LLDP (LLDP-MED) messages from voice devices such as FortiFone units and pass this information to a FortiGate device for device detection. FortiSwitchOS proposes an automatic FortiLink voice default VLAN for voice devices, and you can use a dynamic port policy to assign a device to an LLDP profile, QoS policy, and VLAN policy. When a detected device is matched to the dynamic port policy, the corresponding policy actions are applied on the switch port.

Fortinetʼs ability to inspect all traffic at the core, even encrypted packets, allows you to excel at application awareness and high quality of experience (QoE) delivery for your workforce.

Resiliency

When considering design recommendations, a critical point is resiliency: avoiding single points of failure and ensuring a fast convergence to minimize service downtime. The following sections show how resiliency is accomplished in each layer of the reference architecture described in Tiered architecture.

Tier-1/core layer resiliency

At this layer, resiliency is achieved by many different features. At the FortiGate level, the FortiGate Clustering Protocol (FGCP) provides failover protection because a cluster can provide FortiGate services even when one of the devices in the cluster encounters a problem (such as link failure, power loss, or memory or SSD failures). FortiGate devices can be configured in active-passive or active-active modes; refer to High Availability for further information. FortiGate clusters can be integrated into the load-balancing configuration using the FortiGate Session Life Support Protocol (FGSP) in a network where traffic is load balanced.

The following figure shows the redundancy mechanisms for the tier-1/core layer.

From the FortiSwitch perspective, resiliency is achieved by the following:

  • Building MCLAG pairs. All links are enabled and forward traffic without loops, reducing the convergence times in case a link fails. To protect the MCLAG configuration, Fortinet suggests using two links between the FortiSwitch units and enabling MCLAG split interface on the MCLAG pair.

  • Traffic load balancing. By default, FortiSwitch units load balance the traffic among all available uplinks. The default algorithm is based on the source-destination IP addresses, but it can be modified for other supported methods. Load balancing will work alongside the FGSP.

  • Configuring a full mesh between this first layer of FortiSwitch units and the FortiGate devices, regardless of the HA configuration, results in the maximum bandwidth and minimum convergence time.

  • Uplinks could be also formed by the aggregation of several links (link aggregation group [LAG]), which will increase the capacity and resiliency.

Tier-2/aggregation layer resiliency

At this layer, formed entirely by FortiSwitch units in MCLAG pairs, similar recommendations to the ones stated for the core layer are applied: enabling MCLAG pairs with link redundancy and full mesh wiring towards the core and access layers with LAGs.

The following figure shows the redundancy mechanisms of the tier-2/aggregation layer.

Tier-3/access layer resiliency

For the first pair of FortiSwitch units forming an MCLAG pair (the tier-3 MCLAG pair), the same recommendations as in the previous two layers are applied.

The following figure shows the redundancy mechanisms of the tier-3/access layer.

For the last meterʼs design:

  • If the last meterʼs design comprises a single FortiSwitch unit and single FortiAP unit, Fortinet recommends to connect them dual homed to the access MCLAG pair with single links or LAGs.

    The following figure shows the last meterʼs redundancy for a single FortiSwitch unit and single FortiAP unit.

  • If the last meterʼs design comprises a ring of FortiSwitch units, FortiLink automatically forms for ring topologies and takes care of the spanning tree configuration by default. The ring should have each end connected to a different access MCLAG FortiSwitch unit, and the inter-switch links could comprise one or multiple interfaces (LAGs). Other mechanisms can be put in place to harden the stability and resiliency of the access layer, such as loop protection and STP BPDU guard.

    The following figure shows the last meterʼs redundancy for a ring of FortiSwitch units.

Future proofing

The number of connected devices and the bandwidth consumption increases at a very fast pace. According to Nielsenʼs Law for Internet Bandwidth, Internet bandwidth is supposed to grow up to 50% year over year.

That number could be overwhelming in some situations, especially when planning the design for a network, but there are many details to be considered. Each network and its needs will vary from deployment to deployment.

When designing campus networks, there are many factors that impact the overall performance. The design must be planned to support the demands of the near future, benefit from the return of investment, and try to delay the changes needed in the near future.

  • Consider the device connectivity needs for the next 5 years, such as wired devices using native 2.5 GbE or 5 GbE and the rising popularity of wireless devices.

  • In the data center, server connectivity at the 25-GbE speed.

  • Make it easily scalable for wired and wireless access. Adding more capacity, at least at the access layer should not be difficult with this design, nor technically complicated. With FortiAP and FortiSwitch units, adding new elements to the topology is done seamlessly and easily, thanks to the single-pane-of-glass control from the FortiGate device and FortiManager unit.

  • Replacing existing infrastructure to increase performance or network capacity should not require starting from the beginning. Replacing existing FortiGate devices, FortiSwitch units, or FortiAP units is as simple as executing a few commands, and you do not have to start the configuration all over.

  • The enhanced visibility obtained by the tight integration of the Fortinet Security Fabric also helps to monitor the network trends and plan in advance for the near-future needs.

The following figure shows how you can use the FortiGate single-pane-of-glass interface to monitor network usage.

Previous
Next