Use maximize bandwidth to load balance traffic between ADVPN shortcuts
When ADVPN is configured on a FortiGate spoke along with an SD-WAN rule set to Maximize Bandwidth SLA (GUI) or load balance mode (CLI) as well as tie-break set to fib-best-match, then spoke-to-spoke traffic is load balanced between multiple ADVPN shortcuts when the shortcuts are within the configured SLA conditions.
Following is an example configuration with load-balance enabled and tie-break set to fib-best-match:
config system sdwan
config service
edit 3
set mode sla
set load-balance enable
set dst "all"
config sla
edit "ping"
set id 1
next
end
set priority-members 1 2
set tie-break fib-best-match
next
end
end
Example
In this example SD-WAN is configured between one hub and multiple spokes, and the SD-WAN configuration shows SD-WAN rule 3 with the following required settings to enable spoke-to-spoke traffic between multiple ADVPN shortcuts:
- set load-balance enable
- set tie-break fib-best-match
show system sdwan
config system sdwan
set status enable
config zone
edit "virtual-wan-link"
next
edit "zon2"
next
end
config members
edit 1
set interface "vd2-1"
set cost 10
next
edit 2
set interface "vd2-2"
set cost 20
next
end
config health-check
edit "ping"
set server "11.11.11.11"
set members 1 2
config sla
edit 1
set latency-threshold 200
set jitter-threshold 50
next
edit 2
next
end
next
edit "1"
next
end
config service
edit 1
set dst "033"
set priority-members 1
next
edit 2
set dst "133"
set priority-members 2
next
edit 3
set mode sla
set load-balance enable
set dst "all"
config sla
edit "ping"
set id 1
next
end
set priority-members 1 2
set tie-break fib-best-match
next
end
end
To trigger spoke-to-spoke communication, run an ICMP ping on PC A with IP address 22.1.1.22 behind spoke 1 that is destined for PC B with IP address 33.1.1.33 behind spoke 2. The spoke-to-spoke traffic will be used to demonstrate load balancing between shortcuts in the CLI output of this topic.
To verify the configuration:
-
Confirm the ADVPN shortcuts are within the SLA conditions:
# diagnose system sdwan health-check Health Check(ping): Seq(1 vd2-1): state(alive), packet-loss(0.000%) latency(0.029), jitter(0.002), mos(4.404), bandwidth-up(1999), bandwidth-dw(0), bandwidth-bi(1999) sla_map=0x3 Seq(1 vd2-1_0): state(alive), packet-loss(0.000%) latency(0.026), jitter(0.001), mos(4.404), bandwidth-up(2000), bandwidth-dw(0), bandwidth-bi(2000) sla_map=0x3 Seq(2 vd2-2): state(alive), packet-loss(0.000%) latency(0.055), jitter(0.064), mos(4.404), bandwidth-up(0), bandwidth-dw(0), bandwidth-bi(0) sla_map=0x3 Seq(2 vd2-2_0): state(alive), packet-loss(0.000%) latency(0.060), jitter(0.058), mos(4.404), bandwidth-up(0), bandwidth-dw(0), bandwidth-bi(0) sla_map=0x3
-
Confirm the settings for SD-WAN rule 3:
# diagnose system sdwan service 3 Service(3): Address Mode(IPV4) flags=0x4200 use-shortcut-sla use-shortcut Tie break: fib Gen(1), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(load-balance hash-mode=round-robin) Member sub interface(4): 1: seq_num(1), interface(vd2-1): 1: vd2-1_0(125) 3: seq_num(2), interface(vd2-2): 1: vd2-2_0(127) Members(4): 1: Seq_num(1 vd2-1), alive, sla(0x1), gid(2), num of pass(1), selected 2: Seq_num(1 vd2-1_0), alive, sla(0x1), gid(2), num of pass(1), selected 3: Seq_num(2 vd2-2), alive, sla(0x1), gid(2), num of pass(1), selected 4: Seq_num(2 vd2-2_0), alive, sla(0x1), gid(2), num of pass(1), selected Dst address(1): 0.0.0.0-255.255.255.255 -
Confirm firewall policing routing list:
# diagnose firewall proute list 2131230723 list route policy info(vf=vd2): id=2131230723(0x7f080003) vwl_service=3 vwl_mbr_seq=1 1 2 2 dscp_tag=0xfc 0xfc flags=0x90 load-balance hash-mode=round-robin fib-best-match tos=0x00 tos_mask=0x00 protocol=0 sport=0-65535 iif=0(any) dport=1-65535 path(4) oif=116(vd2-1) num_pass=1 oif=125(vd2-1_0) num_pass=1 oif=117(vd2-2) num_pass=1 oif=127(vd2-2_0) num_pass=1 destination(1): 0.0.0.0-255.255.255.255 source wildcard(1): 0.0.0.0/0.0.0.0 hit_count=117 last_used=2023-04-21 15:49:59
-
Confirm the routing table:
# get router info routing-table bgp Routing table for VRF=0 B* 0.0.0.0/0 [200/0] via 10.10.100.254 (recursive via vd2-1 tunnel 11.1.1.11), 01:26:14, [1/0] [200/0] via 10.10.200.254 (recursive via vd2-2 tunnel 11.1.2.11), 01:26:14, [1/0] B 1.1.1.1/32 [200/0] via 11.1.1.1 [2] (recursive via 12.1.1.1, vd2-vlan12), 01:26:14, [1/0] B 11.11.11.11/32 [200/0] via 10.10.100.254 (recursive via vd2-1 tunnel 11.1.1.11), 01:26:14, [1/0] [200/0] via 10.10.200.254 (recursive via vd2-2 tunnel 11.1.2.11), 01:26:14, [1/0] B 33.1.1.0/24 [200/0] via 10.10.100.3 [2] (recursive is directly connected, vd2-1_0), 01:19:41, [1/0] [200/0] via 10.10.200.3 [2] (recursive is directly connected, vd2-2_0), 01:19:41, [1/0] B 100.1.1.0/24 [200/0] via 10.10.100.254 (recursive via vd2-1 tunnel 11.1.1.11), 01:26:14, [1/0] [200/0] via 10.10.200.254 (recursive via vd2-2 tunnel 11.1.2.11), 01:26:14, [1/0] -
Check the packet sniffer output for the default setting.
This step demonstrates routing for the default setting of
set tie-break zone. The following packet sniffer output of ICMP pings demonstrates how spoke-to-spoke traffic (ping from 22.1.1.22 to 33.1.1.13) is load balanced between all parent tunnels and shortcuts, and is not limited to shortcuts within SLA.# diagnose sniffer packet any "host 33.1.1.13" 4 interfaces=[any] filters=[host 33.1.1.13] 14.665232 vd22-vlan22 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 14.665234 npu0_vlink1 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 14.665240 vd2-vlan22 in 22.1.1.22 -> 33.1.1.13: icmp: echo request 14.665262 vd2-1_0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 14.665274 vd3-1_0 in 22.1.1.22 -> 33.1.1.13: icmp: echo request 14.665284 vd3-vlan33 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 14.665285 npu0_vlink0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 14.665289 vd33-vlan33 in 22.1.1.22 -> 33.1.1.13: icmp: echo request 14.665299 vd33-vlan33 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 14.665300 npu0_vlink1 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 14.665306 vd3-vlan33 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply 14.665314 vd3-1_0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 14.665326 vd2-1_0 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply 14.665331 vd2-vlan22 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 14.665332 npu0_vlink0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 14.665337 vd22-vlan22 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply 24.190955 vd22-vlan22 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 24.190957 npu0_vlink1 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 24.190963 vd2-vlan22 in 22.1.1.22 -> 33.1.1.13: icmp: echo request 24.190982 vd2-2 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 24.190993 p2 in 22.1.1.22 -> 33.1.1.13: icmp: echo request 24.191002 p2 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 24.191020 vd3-2 in 22.1.1.22 -> 33.1.1.13: icmp: echo request 24.191031 vd3-vlan33 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 24.191032 npu0_vlink0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 24.191036 vd33-vlan33 in 22.1.1.22 -> 33.1.1.13: icmp: echo request 24.191046 vd33-vlan33 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 24.191047 npu0_vlink1 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 24.191053 vd3-vlan33 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply 24.191063 vd3-2 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 24.191074 p2 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply 24.191079 p2 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 24.191090 vd2-2 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply 24.191094 vd2-vlan22 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 24.191095 npu0_vlink0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 24.191100 vd22-vlan22 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply 51.064984 vd22-vlan22 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 51.064985 npu0_vlink1 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 51.064991 vd2-vlan22 in 22.1.1.22 -> 33.1.1.13: icmp: echo request 51.065011 vd2-2_0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 51.065022 vd3-2_0 in 22.1.1.22 -> 33.1.1.13: icmp: echo request 51.065031 vd3-vlan33 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 51.065032 npu0_vlink0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 51.065036 vd33-vlan33 in 22.1.1.22 -> 33.1.1.13: icmp: echo request 51.065046 vd33-vlan33 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 51.065047 npu0_vlink1 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 51.065054 vd3-vlan33 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply 51.065063 vd3-2_0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 51.065075 vd2-2_0 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply 51.065082 vd2-vlan22 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 51.065082 npu0_vlink0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 51.065087 vd22-vlan22 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply 67.257123 vd22-vlan22 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 67.257125 npu0_vlink1 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 67.257131 vd2-vlan22 in 22.1.1.22 -> 33.1.1.13: icmp: echo request 67.257150 vd2-1 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 67.257162 p1 in 22.1.1.22 -> 33.1.1.13: icmp: echo request 67.257170 p1 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 67.257189 vd3-1 in 22.1.1.22 -> 33.1.1.13: icmp: echo request 67.257199 vd3-vlan33 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 67.257200 npu0_vlink0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 67.257205 vd33-vlan33 in 22.1.1.22 -> 33.1.1.13: icmp: echo request 67.257216 vd33-vlan33 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 67.257217 npu0_vlink1 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 67.257223 vd3-vlan33 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply 67.257234 vd3-1 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 67.257245 p1 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply 67.257250 p1 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 67.257261 vd2-1 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply 67.257266 vd2-vlan22 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 67.257267 npu0_vlink0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 67.257272 vd22-vlan22 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply ^C 84 packets received by filter 0 packets dropped by kernel
-
Check the sniffer packet output after changing the setting to
set tie-break fib-best-match.The following packet sniffer output of ICMP pings demonstrates how load balancing of spoke-to-spoke is limited and only occurs between shortcuts vd2-1_0 and vd2-2_0, which are within SLA.
# diagnose sniffer packet any "host 33.1.1.13" 4 interfaces=[any] filters=[host 33.1.1.13] 2.592392 vd22-vlan22 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 2.592394 npu0_vlink1 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 2.592400 vd2-vlan22 in 22.1.1.22 -> 33.1.1.13: icmp: echo request 2.592420 vd2-1_0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 2.592432 vd3-1_0 in 22.1.1.22 -> 33.1.1.13: icmp: echo request 2.592441 vd3-vlan33 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 2.592442 npu0_vlink0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 2.592447 vd33-vlan33 in 22.1.1.22 -> 33.1.1.13: icmp: echo request 2.592484 vd33-vlan33 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 2.592485 npu0_vlink1 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 2.592491 vd3-vlan33 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply 2.592498 vd3-1_0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 2.592510 vd2-1_0 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply 2.592515 vd2-vlan22 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 2.592516 npu0_vlink0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 2.592520 vd22-vlan22 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply 8.808792 vd22-vlan22 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 8.808793 npu0_vlink1 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 8.808799 vd2-vlan22 in 22.1.1.22 -> 33.1.1.13: icmp: echo request 8.808816 vd2-2_0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 8.808827 vd3-2_0 in 22.1.1.22 -> 33.1.1.13: icmp: echo request 8.808838 vd3-vlan33 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 8.808838 npu0_vlink0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 8.808842 vd33-vlan33 in 22.1.1.22 -> 33.1.1.13: icmp: echo request 8.808852 vd33-vlan33 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 8.808853 npu0_vlink1 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 8.808858 vd3-vlan33 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply 8.808866 vd3-2_0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 8.808877 vd2-2_0 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply 8.808882 vd2-vlan22 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 8.808883 npu0_vlink0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 8.808887 vd22-vlan22 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply 18.024377 vd22-vlan22 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 18.024379 npu0_vlink1 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 18.024385 vd2-vlan22 in 22.1.1.22 -> 33.1.1.13: icmp: echo request 18.024400 vd2-1_0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 18.024411 vd3-1_0 in 22.1.1.22 -> 33.1.1.13: icmp: echo request 18.024421 vd3-vlan33 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 18.024422 npu0_vlink0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 18.024427 vd33-vlan33 in 22.1.1.22 -> 33.1.1.13: icmp: echo request 18.024436 vd33-vlan33 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 18.024437 npu0_vlink1 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 18.024443 vd3-vlan33 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply 18.024449 vd3-1_0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 18.024459 vd2-1_0 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply 18.024463 vd2-vlan22 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 18.024464 npu0_vlink0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 18.024468 vd22-vlan22 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply 24.216469 vd22-vlan22 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 24.216470 npu0_vlink1 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 24.216477 vd2-vlan22 in 22.1.1.22 -> 33.1.1.13: icmp: echo request 24.216493 vd2-2_0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 24.216506 vd3-2_0 in 22.1.1.22 -> 33.1.1.13: icmp: echo request 24.216518 vd3-vlan33 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 24.216519 npu0_vlink0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 24.216525 vd33-vlan33 in 22.1.1.22 -> 33.1.1.13: icmp: echo request 24.216535 vd33-vlan33 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 24.216536 npu0_vlink1 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 24.216542 vd3-vlan33 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply 24.216548 vd3-2_0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 24.216559 vd2-2_0 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply 24.216563 vd2-vlan22 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 24.216564 npu0_vlink0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 24.216568 vd22-vlan22 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply ^C 70 packets received by filter 0 packets dropped by kernel
-
Check SD-WAN heath.
When an ADVPN shortcut is out of SLA, traffic does not run on it. Shortcut vd2-1_0 is out of SLA.
# diagnose system sdwan health-check Health Check(ping): Seq(1 vd2-1): state(alive), packet-loss(6.000%) latency(0.026), jitter(0.001), mos(4.401), bandwidth-up(1999), bandwidth-dw(0), bandwidth-bi(1999) sla_map=0x0 Seq(1 vd2-1_0): state(alive), packet-loss(18.182%) latency(0.033), jitter(0.003), mos(4.395), bandwidth-up(2000), bandwidth-dw(0), bandwidth-bi(2000) sla_map=0x0 Seq(2 vd2-2): state(alive), packet-loss(0.000%) latency(0.024), jitter(0.001), mos(4.404), bandwidth-up(0), bandwidth-dw(0), bandwidth-bi(0) sla_map=0x3 Seq(2 vd2-2_0): state(alive), packet-loss(0.000%) latency(0.033), jitter(0.005), mos(4.404), bandwidth-up(0), bandwidth-dw(0), bandwidth-bi(0) sla_map=0x3
-
Check the sniffer packet:
No traffic runs on Shortcut vd2-1_0 because it is out of SLA.
# diagnose sniffer packet any "host 33.1.1.13" 4 interfaces=[any] filters=[host 33.1.1.13] 8.723075 vd22-vlan22 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 8.723077 npu0_vlink1 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 8.723084 vd2-vlan22 in 22.1.1.22 -> 33.1.1.13: icmp: echo request 8.723103 vd2-2_0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 8.723115 vd3-2_0 in 22.1.1.22 -> 33.1.1.13: icmp: echo request 8.723148 vd3-vlan33 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 8.723149 npu0_vlink0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 8.723154 vd33-vlan33 in 22.1.1.22 -> 33.1.1.13: icmp: echo request 8.723166 vd33-vlan33 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 8.723166 npu0_vlink1 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 8.723171 vd3-vlan33 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply 8.723179 vd3-2_0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 8.723190 vd2-2_0 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply 8.723195 vd2-vlan22 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 8.723195 npu0_vlink0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 8.723199 vd22-vlan22 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply 17.202681 vd22-vlan22 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 17.202683 npu0_vlink1 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 17.202688 vd2-vlan22 in 22.1.1.22 -> 33.1.1.13: icmp: echo request 17.202704 vd2-2_0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 17.202716 vd3-2_0 in 22.1.1.22 -> 33.1.1.13: icmp: echo request 17.202727 vd3-vlan33 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 17.202728 npu0_vlink0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request 17.202733 vd33-vlan33 in 22.1.1.22 -> 33.1.1.13: icmp: echo request 17.202742 vd33-vlan33 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 17.202743 npu0_vlink1 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 17.202749 vd3-vlan33 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply 17.202755 vd3-2_0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 17.202767 vd2-2_0 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply 17.202771 vd2-vlan22 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 17.202772 npu0_vlink0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply 17.202777 vd22-vlan22 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply