Using layer-3 routing within an MCLAG

Starting in FortiSwitchOS 7.0.1, you can now use the Virtual Router Redundancy Protocol to make layer-3 routing in an MCLAG function as a single router.

Note:

• Only IPv4 addresses are supported.
• 250 switch virtual interfaces (SVIs) are supported.
• Both peer switches must be configured.
• Multicast (PIM) routing, policy-based routing (PBR), IS-IS routing, and RIP are not supported.

There are four use cases:

• One-tier MCLAG
• Two-tier MCLAG
• One-tier MCLAG with a southbound switch
• One-tier MCLAG without a northbound MCLAG trunk

One-tier MCLAG

To use layer-3 routing for a one-tier MCLAG, you can use a combination of VRRP with static or dynamic routing (BGP or OSPF).

The following figure shows the scenario with VRRP and BGP.

For a one-tier MCLAG topology:

• Core1 and Core2 are FortiSwitch units that form the MCLAG.
• Traffic flows between northbound and southbound routers through the MCLAG peer group. The two routers can be FortiSwitch units, but this is not mandatory

Using VRRP and BGP

Enable VRRP on the switch virtual interfaces (SVIs) towards the northbound and southbound neighboring routers on both MCLAG peers. The VRRP IP address is used as the next hop or BGP neighbor in the northbound and southbound neighboring routers.

Always enable vrrp-virtual-mac for VRRP. Layer-3 lookup for the VRRP virtual MAC address on the VRRP backup is enabled automatically. By virtue of MCLAG and trunk hashing, ingress packets on the VRRP backup MCLAG core are routed without crossing the ICL if the appropriate route is available.

Enable external BGP (eBGP) between the northbound router and the MCLAG VRRP IP address of the northbound SVI and between the southbound router and the MCLAG VRRP IP address of the southbound SVI. Because the eBGP neighbor is the VRRP IP address, the router establishes a connection with only the VRRP master. Enable ebgp-enforce-multihop and set ebgp-multihop-ttl to 3.

Use internal BGP (iBGP) between the MCLAG cores across the ICL. The routes from the eBGP sessions are advertised to iBGP, and the VRRP backup obtains the appropriate routes and stores them in its routing table and hardware. This achieves northbound-southbound layer-3 routing in an MCLAG topology, avoiding traffic across the ICL and using active-active forwarding across the MCLAG cores.

Using VRRP and OSPF

OSPF can also be used as the routing protocol between MCLAG peers and northbound/southbound routers. In this case, OSPF is also the IGP. It requires an active VRRP IP address in each MCLAG peer.

Use OSPF between the virtual router IP address and the router connecting the MCLAG core switches over an MCLAG link.

Always enable vrrp-virtual-mac for VRRP.

Configure two VRRP sessions on each SVI and configure the VRRP priorities so that there is a VRRP master on each MCLAG core.

The layer-3 lookup for the VRRP virtual MAC address is automatically enabled on the VRRP backup. Because of MCLAG and trunk hashing, ingress packets on the VRRP backup core are routed without crossing the ICL if an appropriate route is available.

The result of this topology is northbound-southbound layer-3 routing in the MCLAG topology without traffic crossing the ICL, and active-active forwarding is used across MCLAG cores.

Using VRRP, BGP (northbound), and OSPF (southbound)

• Start with the BGP configuration to configure MCLAG for nouthbound routing.
• Start with the OSPF configuration to configure MCLAG for sorthbound routing.
  • In the OSPF configuration, include the BGP subnet used for northbound routing in both the OSPF network and OSPF interface configuration.

Two-tier MCLAG

For layer-3 routing between MCLAG tiers, the configuration is similar for the tier-2 and tier-3 MCLAG peers. You can use a combination of VRRP with static or dynamic routing (BGP or OSPF). The following figure shows the scenario with VRRP and BGP.

For a two-tier MCLAG topology:

• Core1 and Core2 are FortiSwitch units that form the tier-1 MCLAG. Core3 and Core4 are FortiSwitch units that form the tier-2 MCLAG.
• Traffic flows between northbound and southbound routers through the MCLAG peer groups. The two routers can be FortiSwitch units, but this is not mandatory.

Using VRRP and BGP

Each MCLAG tier has two VRRP sessions:

• One VRRP session is on the SVI that connects the router and the two core switches.
• One VRRP session is on the SVI subnet that is common between the pairs of MCLAG switches. For this subnet, the virtual router IP address belongs to the same subnet on both MCLAG pairs.

Each session has a different vrip value. Each session has a different virtual route identifier (VRID).

Configure eBGP for Core1, Core2, Core3, Core4, the northbound AS, and the southbound AS. You need to enable ebgp-enforce-multihop and set ebgp-multihop-ttl to 3.

Configure iBGP for Core1, Core2, Core3, and Core4.

When you configure VRRP, enable vrrp-virtual-mac.

Using VRRP and OSPF

Use OSPF between the virtual router IP address and the router connecting the MCLAG core switches over an MCLAG link and between the MCLAG tiers.

Always enable vrrp-virtual-mac for VRRP.

Configure two VRRP sessions on each SVI and configure the VRRP priorities so that there is a VRRP master on each MCLAG core.

The layer-3 lookup for the VRRP virtual MAC address is automatically enabled on the VRRP backup. Because of MCLAG and trunk hashing, ingress packets on the VRRP backup core are routed without crossing the ICL if an appropriate route is available.

The result of this topology is northbound-southbound layer-3 routing in the MCLAG topology without traffic crossing the ICL, and active-active forwarding is used across MCLAG cores.

One-tier MCLAG with a southbound switch

For this topology:

• Core1 and Core2 are FortiSwitch units that form the MCLAG.
• Traffic flows between the northbound router and the southbound hosts through the MCLAG peer group. The router can be a FortiSwitch unit, but this is not mandatory.
• The southbound switch or endpoint does not use eBGP with the MCLAG peer switches. The MCLAG SVI VRRP IP address is the default gateway for the endpoints.

One-tier MCLAG without a northbound MCLAG trunk

For this topology:

• Core1 and Core2 are FortiSwitch units that form the MCLAG.
• Traffic flows between northbound and southbound routers through the MCLAG peer group. The two routers can be FortiSwitch units, but this is not mandatory.
• The northbound router does not form an MCLAG trunk with the peer switches; instead, each link has its own layer-3 interface and MSTP instance. The northbound SVIs on the MCLAG peers do not need VRRP.
• Make certain that the two VLANs are on two different MSTP instances to avoid STP loops.

Using VRRP with static routing

Enable VRRP on the switch virtual interfaces (SVIs) towards the northbound and southbound neighboring routers on both MCLAG peers. The VRRP IP address is used as the next hop in the static routes in the northbound and southbound neighboring routers.

Configure static routes on both MCLAG peers pointing to the neighboring routers. In the case of tier-2 or tier-3 MCLAG, configure static routes on both MCLAG peers pointing to the VRRP IP address of the SVI on the adjacent MCLAG peers.

Always enable vrrp-virtual-mac for VRRP.

East-west traffic

For east-west traffic, where the eastbound router is connected to the east MCLAG and the westbound router is connected to the west MCLAG, traffic crosses the MCLAG ICL. Any routing protocol can be used between the routers and the FortiSwitch units; these routes can be redistributed to the FortiSwitch MCLAG peers using IGP (iBGP or OSPF).

Configuration example (BGP and VRRP)

Use the following steps to configure layer-3 routing in a one-tier MCLAG using BGP and VRRP:

1. Configure the trunks
2. Configure the layer-3 SVIs
3. Configure the layer-2 switch interfaces
4. Configure the layer-3 routing

Configure the trunks

To configure the northbound trunk on the northbound router:

config switch trunk

edit "nb1"

set mode lacp-active

set members "port49" "port50"

next

end

To configure the trunk for the FortiSwitch peer 1 (Core1):

config switch trunk

edit "fsw2"

set mode lacp-active

set mclag-icl enable

set members "port52" "port53"

next

edit "sb"

set mode lacp-active

set mclag enable

set members "port26"

next

edit "nb"

set mode lacp-active

set mclag enable

set members "port25"

next

end

To configure the trunk for the FortiSwitch peer 2 (Core2):

config switch trunk

edit "fsw1"

set mode lacp-active

set mclag-icl enable

set members "port52" "port53"

next

edit "sb"

set mode lacp-active

set mclag enable

set members "port15”

next

edit "nb"

set mode lacp-active

set mclag enable

set members "port10"

next

end

To configure the trunk on the southbound router:

config switch trunk

edit "sb1"

set mode lacp-active

set members "port4" "port5"

next

end

Configure the layer-3 SVIs

To configure the layer-3 SVI on the northbound router:

config system interface

edit "nb1" <<<<< System interface used to connect to the MCLAG core VRRP IP address

set ip 20.1.1.48 255.255.255.0

set vlanid 20

next

end

To configure the layer-3 SVI interfaces on FortiSwitch peer 1 (Core1):

config system interface

edit "sb" <<<<<< connected to the southbound router

set ip 100.1.1.21 255.255.255.0

set vrrp-virtual-mac enable

config vrrp

edit 1

set vrip 100.1.1.20

next

end

set vlanid 100

next

edit "nb" <<<<<< connected to the northbound router

set ip 20.1.1.11 255.255.255.0

set vrrp-virtual-mac enable

config vrrp

edit 5

set vrip 20.1.1.1

next

end

set vlanid 20

next

end

To configure the layer-3 SVI on the FortiSwitch peer 2 (Core2):

config system interface

edit "sb" <<<<<< connected to the southbound router

set ip 100.1.1.22 255.255.255.0

set vrrp-virtual-mac enable

config vrrp

edit 1

set vrip 100.1.1.20

next

end

set vlanid 100

next

edit "nb" <<<<<< connected to the northbound router

set ip 20.1.1.12 255.255.255.0

set vrrp-virtual-mac enable

config vrrp

edit 5

set vrip 20.1.1.1

next

end

set vlanid 20

next

end

To configure the layer-3 SVI on the southbound router:

config system interface

edit "sb" <<<<<< connected to MCLAG core switches VRRP IP address

set ip 100.1.1.10 255.255.255.0

set vlanid 100

next

end

Configure the layer-2 switch interfaces

To configure the layer-2 switch interfaces on the northbound router:

config switch interface

edit "internal"

set native-vlan 4094

set allowed-vlans 20,4094

next

edit "nb1"

set allowed-vlans 20

next

end

To configure the layer-2 switch interfaces for FortiSwitch peer 1 (Core1):

config switch interface

edit "internal"

set native-vlan 4094

set allowed-vlans 20,100,4094

next

edit "fsw2"

set native-vlan 4094

set allowed-vlans 1-4094

set dhcp-snooping trusted

set edge-port disabled

set igmp-snooping-flood-reports enable

set mcast-snooping-flood-traffic enable

next

edit "sb"

set allowed-vlans 100

next

edit "nb"

set allowed-vlans 20

next

end

To configure the layer-2 switch interfaces for FortiSwitch peer 2 (Core2):

config switch interface

edit "internal"

set native-vlan 4094

set allowed-vlans 20,100,4094

next

edit "fsw1"

set native-vlan 4094

set allowed-vlans 1-4094

set dhcp-snooping trusted

set edge-port disabled

set igmp-snooping-flood-reports enable

set mcast-snooping-flood-traffic enable

next

edit "sb"

set allowed-vlans 100

next

edit "nb"

set allowed-vlans 20

next

end

To configure the layer-2 switch interfaces on the southbound router:

config switch interface

edit "internal"

set native-vlan 4094

set allowed-vlans 100,4094

next

edit "sb1"

set allowed-vlans 100

next

end

Configure the layer-3 routing

To configure the routing for the northbound router:

config router bgp

set as 7

set router-id 20.1.1.48

config neighbor

edit "20.1.1.1" >>>>> eBGP to the MCLAG peer VRRP IP address

set ebgp-enforce-multihop enable

set ebgp-multihop-ttl 3

set remote-as 5

next

end

To configure the routing for the FortiSwitch peer 1 (Core1):

config router bgp

set as 5

set router-id 100.1.1.21

config neighbor

edit "20.1.1.48" >>>>> eBGP to the northbound router

set ebgp-enforce-multihop enable

set ebgp-multihop-ttl 3

set remote-as 7

next

edit "100.1.1.22" >>>>> iBGP to MCLAG peer

set remote-as 5

next

edit "100.1.1.10" >>>>> eBGP to the southbound router

set ebgp-enforce-multihop enable

set ebgp-multihop-ttl 3

set remote-as 9

next

end

To configure the routing the FortiSwitch peer 2 (Core2):

config router bgp

set as 5

set router-id 100.1.1.22

config neighbor

edit "20.1.1.48" >>>>> eBGP to the northbound router

set ebgp-enforce-multihop enable

set ebgp-multihop-ttl 3

set remote-as 7

next

edit "100.1.1.21" >>>>> iBGP to the MCLAG peer

set remote-as 5

next

edit "100.1.1.10" >>>>> eBGP to the southbound router

set ebgp-enforce-multihop enable

set ebgp-multihop-ttl 3

set remote-as 9

next

end

To configure the routing for the southbound router:

config router bgp

set as 9

set router-id 100.1.1.10

config neighbor

edit "100.1.1.20" >>>>> eBGP to the MCLAG peer VRRP IP address

set ebgp-enforce-multihop enable

set ebgp-multihop-ttl 3

set remote-as 5

next

end

Configuration example (OSPF and VRRP)

Use the following steps to configure layer-3 routing in a one-tier MCLAG using OSPF and VRRP:

1. Configure the trunks
2. Configure the layer-3 SVIs
3. Configure the layer-2 switch interfaces
4. Configure the layer-3 routing

Configure the trunks

To configure the northbound trunk on the northbound router:

config switch trunk

edit "nb1"

set mode lacp-active

set members "port49" "port50"

next

end

To configure the trunk for the FortiSwitch peer 1 (Core1):

config switch trunk

edit "fsw2"

set mode lacp-active

set mclag-icl enable

set members "port52" "port53"

next

edit "sb"

set mode lacp-active

set mclag enable

set members "port26"

next

edit "nb"

set mode lacp-active

set mclag enable

set members "port25"

next

end

To configure the trunk for the FortiSwitch peer 2 (Core2):

config switch trunk

edit "fsw1"

set mode lacp-active

set mclag-icl enable

set members "port52" "port53"

next

edit "sb"

set mode lacp-active

set mclag enable

set members "port15”

next

edit "nb"

set mode lacp-active

set mclag enable

set members "port10"

next

end

To configure the trunk on the southbound router:

config switch trunk

edit "sb1"

set mode lacp-active

set members "port4" "port5"

next

end

Configure the layer-3 SVIs

To configure the layer-3 SVI on the northbound router:

config system interface

edit "nb1" <<<<< System interface used to connect to the MCLAG core VRRP IP address

set ip 20.1.1.48 255.255.255.0

set vlanid 20

next

end

To configure the layer-3 SVI interfaces on FortiSwitch peer 1 (Core1):

config system interface

edit "sb" <<<<<< connected to the southbound external router using VRRP

set ip 100.1.1.21 255.255.255.0

set vrrp-virtual-mac enable

config vrrp

edit 1

set vrip 100.1.1.20

next

edit 3

set priority 200

set vrip 100.1.1.200

next

end

set vlanid 100

next

edit "nb" <<<<<< connected to the northbound external router using VRRP

set ip 20.1.1.11 255.255.255.0

set vrrp-virtual-mac enable

config vrrp

edit 5

set priority 200

set vrip 20.1.1.1

next

edit 8

set vrip 20.1.1.100

next

end

set vlanid 20

next

end

To configure the layer-3 SVI on the FortiSwitch peer 2 (Core2):

config system interface

edit "sb" <<<<<< connected to the southbound external router using VRRP

set ip 100.1.1.22 255.255.255.0

set vrrp-virtual-mac enable

config vrrp

edit 1

set priority 200

set vrip 100.1.1.20

next

edit 3

set vrip 100.1.1.200

next

end

set vlanid 100

next

edit "nb" <<<<<< connected to the northbound external router using VRRP

set ip 20.1.1.12 255.255.255.0

set vrrp-virtual-mac enable

config vrrp

edit 5

set vrip 20.1.1.1

next

edit 8

set priority 200

set vrip 20.1.1.100

next

end

set vlanid 20

next

end

To configure the layer-3 SVI on the southbound router:

config system interface

edit "sb" <<<<< System interface used to connect to the MCLAG core VRRP IP address

set ip 100.1.1.48 255.255.255.0

set vlanid 100

next

end

Configure the layer-2 switch interfaces

To configure the layer-2 switch interfaces on the northbound router:

config switch interface

edit "internal"

set native-vlan 4094

set allowed-vlans 20,4094

next

edit "nb1"

set allowed-vlans 20

next

end

To configure the layer-2 switch interfaces for FortiSwitch peer 1 (Core1):

config switch interface

edit "internal"

set native-vlan 4094

set allowed-vlans 20,100,4094

next

edit "fsw2"

set native-vlan 4094

set allowed-vlans 1-4094

set dhcp-snooping trusted

set edge-port disabled

set igmp-snooping-flood-reports enable

set mcast-snooping-flood-traffic enable

next

edit "sb"

set allowed-vlans 100

next

edit "nb"

set allowed-vlans 20

next

end

To configure the layer-2 switch interfaces for FortiSwitch peer 2 (Core2):

config switch interface

edit "internal"

set native-vlan 4094

set allowed-vlans 20,100,4094

next

edit "fsw1"

set native-vlan 4094

set allowed-vlans 1-4094

set dhcp-snooping trusted

set edge-port disabled

set igmp-snooping-flood-reports enable

set mcast-snooping-flood-traffic enable

next

edit "sb"

set allowed-vlans 100

next

edit "nb"

set allowed-vlans 20

next

end

To configure the layer-2 switch interfaces on the southbound router:

config switch interface

edit "internal"

set native-vlan 4094

set allowed-vlans 100,4094

next

edit "sb1"

set allowed-vlans 100

next

end

Configure the layer-3 routing

To configure the routing for the northbound router:

config router ospf

set router-id 20.1.1.48

config area

edit 0.0.0.100

next

end

config interface

edit "nb1"

next

end

config network

edit 1 <<< connected to the MCLAG core

set area 0.0.0.100

set prefix 20.1.1.0 255.255.255.0

next

end

end

To configure the routing for the FortiSwitch peer 1 (Core1):

config router ospf

set router-id 100.1.1.21

config area

edit 0.0.0.100

next

end

config interface

edit "sb" <<<<<< to the southbound router

next

edit "nb" <<<<<< to the northbound router

next

end

config network

edit 100 <<<<<< to the southbound router

set area 0.0.0.100

set prefix 100.1.1.0 255.255.255.0

next

edit 20 <<<<<< to the northbound router

set area 0.0.0.100

set prefix 20.1.1.0 255.255.255.0

next

end

end

To configure the routing the FortiSwitch peer 2 (Core2):

config router ospf

set router-id 100.1.1.22

config area

edit 0.0.0.100

next

end

config interface

edit "sb"

next

edit "nb"

next

end

config network

edit 100 <<< to the southbound router

set area 0.0.0.100

set prefix 100.1.1.0 255.255.255.0

next

edit 20 <<< to the northbound router

set area 0.0.0.100

set prefix 20.1.1.0 255.255.255.0

next

end

end

To configure the routing for the southbound router:

config router ospf

set router-id 100.1.1.48

config area

edit 0.0.0.100

next

end

config interface

edit "sb1"

next

end

config network

edit 1 <<< connected to the MCLAG core

set area 0.0.0.100

set prefix 100.1.1.0 255.255.255.0

next

end

end