Use maximize bandwidth to load balance traffic between ADVPN shortcuts

When ADVPN is configured on a FortiGate spoke along with an SD-WAN rule set to Maximize Bandwidth SLA (GUI) or load balance mode (CLI) as well as tie-break set to fib-best-match, then spoke-to-spoke traffic is load balanced between multiple ADVPN shortcuts when the shortcuts are within the configured SLA conditions.

Following is an example configuration with set mode load-balance and set tie-break fib-best-match enabled:

config system sdwan
    config service
        edit 3
            set mode load-balance
            set dst "all"
            config sla
                edit "ping"
                    set id 1
                next
            end
            set priority-members 1 2
            set tie-break fib-best-match
        next
    end
end

Example

In this example SD-WAN is configured between one hub and multiple spokes, and the SD-WAN configuration shows SD-WAN rule 3 with the following required settings to enable spoke-to-spoke traffic between multiple ADVPN shortcuts:

• set mode load-balance
• set tie-break fib-best-match

show system sdwan
config system sdwan
    set status enable
    config zone
        edit "virtual-wan-link"
        next
        edit "zon2"
        next
    end
    config members
        edit 1
            set interface "vd2-1"
            set cost 10
        next
        edit 2
            set interface "vd2-2"
            set cost 20
        next
    end
    config health-check
        edit "ping"
            set server "11.11.11.11"
            set members 1 2
            config sla
                edit 1
                    set latency-threshold 200
                    set jitter-threshold 50
                next
                edit 2
                next
            end
        next
        edit "1"
        next
    end
    config service
        edit 1
            set dst "033"
            set priority-members 1
        next
        edit 2
            set dst "133"
            set priority-members 2
        next
        edit 3
            set mode load-balance
            set dst "all"
            config sla
                edit "ping"
                    set id 1
                next
            end
            set priority-members 1 2
            set tie-break fib-best-match
        next
    end
end

To trigger spoke-to-spoke communication, run an ICMP ping on PC A with IP address 22.1.1.22 behind spoke 1 that is destined for PC B with IP address 33.1.1.33 behind spoke 2. The spoke-to-spoke traffic will be used to demonstrate load balancing between shortcuts in the CLI output of this topic.

To verify the configuration:

1. Confirm the ADVPN shortcuts are within the SLA conditions:

   # diagnose system sdwan health-check
   Health Check(ping):
   Seq(1 vd2-1): state(alive), packet-loss(0.000%) latency(0.029), jitter(0.002), mos(4.404), bandwidth-up(1999), bandwidth-dw(0), bandwidth-bi(1999) sla_map=0x3
   Seq(1 vd2-1_0): state(alive), packet-loss(0.000%) latency(0.026), jitter(0.001), mos(4.404), bandwidth-up(2000), bandwidth-dw(0), bandwidth-bi(2000) sla_map=0x3
   Seq(2 vd2-2): state(alive), packet-loss(0.000%) latency(0.055), jitter(0.064), mos(4.404), bandwidth-up(0), bandwidth-dw(0), bandwidth-bi(0) sla_map=0x3
   Seq(2 vd2-2_0): state(alive), packet-loss(0.000%) latency(0.060), jitter(0.058), mos(4.404), bandwidth-up(0), bandwidth-dw(0), bandwidth-bi(0) sla_map=0x3
   

2. Confirm the settings for SD-WAN rule 3:

   # diagnose system sdwan service 3
   
   Service(3): Address Mode(IPV4) flags=0x4200 use-shortcut-sla use-shortcut
    Tie break: fib
     Gen(1), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(load-balance  hash-mode=round-robin)
     Member sub interface(4):
       1: seq_num(1), interface(vd2-1):
          1: vd2-1_0(125)
       3: seq_num(2), interface(vd2-2):
          1: vd2-2_0(127)
     Members(4):
       1: Seq_num(1 vd2-1), alive, sla(0x1), gid(2), num of pass(1), selected
       2: Seq_num(1 vd2-1_0), alive, sla(0x1), gid(2), num of pass(1), selected
       3: Seq_num(2 vd2-2), alive, sla(0x1), gid(2), num of pass(1), selected
       4: Seq_num(2 vd2-2_0), alive, sla(0x1), gid(2), num of pass(1), selected
     Dst address(1):
           0.0.0.0-255.255.255.255

3. Confirm firewall policing routing list:

   # diagnose firewall proute list 2131230723
   list route policy info(vf=vd2):
   
   id=2131230723(0x7f080003) vwl_service=3 vwl_mbr_seq=1 1 2 2 dscp_tag=0xfc 0xfc flags=0x90 load-balance hash-mode=round-robin  fib-best-match tos=0x00 tos_mask=0x00 protocol=0 sport=0-65535 iif=0(any) dport=1-65535 path(4) oif=116(vd2-1) num_pass=1 oif=125(vd2-1_0) num_pass=1 oif=117(vd2-2) num_pass=1 oif=127(vd2-2_0) num_pass=1
   destination(1): 0.0.0.0-255.255.255.255
   source wildcard(1): 0.0.0.0/0.0.0.0
   hit_count=117 last_used=2023-04-21 15:49:59

4. Confirm the routing table:

   # get router info routing-table bgp
   Routing table for VRF=0
   B*      0.0.0.0/0 [200/0] via 10.10.100.254 (recursive via vd2-1 tunnel 11.1.1.11), 01:26:14, [1/0]
                     [200/0] via 10.10.200.254 (recursive via vd2-2 tunnel 11.1.2.11), 01:26:14, [1/0]
   B       1.1.1.1/32 [200/0] via 11.1.1.1 [2] (recursive via 12.1.1.1, vd2-vlan12), 01:26:14, [1/0]
   B       11.11.11.11/32 [200/0] via 10.10.100.254 (recursive via vd2-1 tunnel 11.1.1.11), 01:26:14, [1/0]
                          [200/0] via 10.10.200.254 (recursive via vd2-2 tunnel 11.1.2.11), 01:26:14, [1/0]
   B       33.1.1.0/24 [200/0] via 10.10.100.3 [2] (recursive is directly connected, vd2-1_0), 01:19:41, [1/0]
                       [200/0] via 10.10.200.3 [2] (recursive is directly connected, vd2-2_0), 01:19:41, [1/0]
   B       100.1.1.0/24 [200/0] via 10.10.100.254 (recursive via vd2-1 tunnel 11.1.1.11), 01:26:14, [1/0]
                        [200/0] via 10.10.200.254 (recursive via vd2-2 tunnel 11.1.2.11), 01:26:14, [1/0]

5. Check the packet sniffer output for the default setting.

   This step demonstrates routing for the default setting of set tie-break zone. The following packet sniffer output of ICMP pings demonstrates how spoke-to-spoke traffic (ping from 22.1.1.22 to 33.1.1.13) is load balanced between all parent tunnels and shortcuts, and is not limited to shortcuts within SLA.

   # diagnose sniffer packet any "host 33.1.1.13" 4
   interfaces=[any]
   filters=[host 33.1.1.13]   
   14.665232 vd22-vlan22 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   14.665234 npu0_vlink1 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   14.665240 vd2-vlan22 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
   14.665262 vd2-1_0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   14.665274 vd3-1_0 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
   14.665284 vd3-vlan33 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   14.665285 npu0_vlink0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   14.665289 vd33-vlan33 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
   14.665299 vd33-vlan33 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   14.665300 npu0_vlink1 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   14.665306 vd3-vlan33 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   14.665314 vd3-1_0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   14.665326 vd2-1_0 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   14.665331 vd2-vlan22 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   14.665332 npu0_vlink0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   14.665337 vd22-vlan22 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   
   24.190955 vd22-vlan22 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   24.190957 npu0_vlink1 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   24.190963 vd2-vlan22 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
   24.190982 vd2-2 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   24.190993 p2 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
   24.191002 p2 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   24.191020 vd3-2 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
   24.191031 vd3-vlan33 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   24.191032 npu0_vlink0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   24.191036 vd33-vlan33 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
   24.191046 vd33-vlan33 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   24.191047 npu0_vlink1 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   24.191053 vd3-vlan33 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   24.191063 vd3-2 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   24.191074 p2 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   24.191079 p2 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   24.191090 vd2-2 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   24.191094 vd2-vlan22 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   24.191095 npu0_vlink0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   24.191100 vd22-vlan22 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   
   51.064984 vd22-vlan22 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   51.064985 npu0_vlink1 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   51.064991 vd2-vlan22 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
   51.065011 vd2-2_0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   51.065022 vd3-2_0 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
   51.065031 vd3-vlan33 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   51.065032 npu0_vlink0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   51.065036 vd33-vlan33 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
   51.065046 vd33-vlan33 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   51.065047 npu0_vlink1 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   51.065054 vd3-vlan33 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   51.065063 vd3-2_0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   51.065075 vd2-2_0 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   51.065082 vd2-vlan22 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   51.065082 npu0_vlink0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   51.065087 vd22-vlan22 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   
   67.257123 vd22-vlan22 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   67.257125 npu0_vlink1 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   67.257131 vd2-vlan22 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
   67.257150 vd2-1 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   67.257162 p1 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
   67.257170 p1 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   67.257189 vd3-1 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
   67.257199 vd3-vlan33 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   67.257200 npu0_vlink0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   67.257205 vd33-vlan33 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
   67.257216 vd33-vlan33 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   67.257217 npu0_vlink1 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   67.257223 vd3-vlan33 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   67.257234 vd3-1 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   67.257245 p1 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   67.257250 p1 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   67.257261 vd2-1 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   67.257266 vd2-vlan22 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   67.257267 npu0_vlink0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   67.257272 vd22-vlan22 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   
   ^C
   84 packets received by filter
   0 packets dropped by kernel

6. Check the sniffer packet output after changing the setting to set tie-break fib-best-match.

   The following packet sniffer output of ICMP pings demonstrates how load balancing of spoke-to-spoke is limited and only occurs between shortcuts vd2-1_0 and vd2-2_0, which are within SLA.

   # diagnose sniffer packet any "host 33.1.1.13" 4
   
   interfaces=[any]
   filters=[host 33.1.1.13]
   2.592392 vd22-vlan22 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   2.592394 npu0_vlink1 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   2.592400 vd2-vlan22 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
   2.592420 vd2-1_0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   2.592432 vd3-1_0 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
   2.592441 vd3-vlan33 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   2.592442 npu0_vlink0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   2.592447 vd33-vlan33 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
   2.592484 vd33-vlan33 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   2.592485 npu0_vlink1 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   2.592491 vd3-vlan33 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   2.592498 vd3-1_0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   2.592510 vd2-1_0 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   2.592515 vd2-vlan22 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   2.592516 npu0_vlink0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   2.592520 vd22-vlan22 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   
   8.808792 vd22-vlan22 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   8.808793 npu0_vlink1 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   8.808799 vd2-vlan22 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
   8.808816 vd2-2_0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   8.808827 vd3-2_0 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
   8.808838 vd3-vlan33 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   8.808838 npu0_vlink0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   8.808842 vd33-vlan33 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
   8.808852 vd33-vlan33 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   8.808853 npu0_vlink1 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   8.808858 vd3-vlan33 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   8.808866 vd3-2_0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   8.808877 vd2-2_0 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   8.808882 vd2-vlan22 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   8.808883 npu0_vlink0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   8.808887 vd22-vlan22 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   
   18.024377 vd22-vlan22 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   18.024379 npu0_vlink1 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   18.024385 vd2-vlan22 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
   18.024400 vd2-1_0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   18.024411 vd3-1_0 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
   18.024421 vd3-vlan33 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   18.024422 npu0_vlink0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   18.024427 vd33-vlan33 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
   18.024436 vd33-vlan33 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   18.024437 npu0_vlink1 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   18.024443 vd3-vlan33 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   18.024449 vd3-1_0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   18.024459 vd2-1_0 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   18.024463 vd2-vlan22 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   18.024464 npu0_vlink0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   18.024468 vd22-vlan22 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   
   24.216469 vd22-vlan22 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   24.216470 npu0_vlink1 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   24.216477 vd2-vlan22 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
   24.216493 vd2-2_0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   24.216506 vd3-2_0 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
   24.216518 vd3-vlan33 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   24.216519 npu0_vlink0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   24.216525 vd33-vlan33 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
   24.216535 vd33-vlan33 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   24.216536 npu0_vlink1 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   24.216542 vd3-vlan33 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   24.216548 vd3-2_0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   24.216559 vd2-2_0 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   24.216563 vd2-vlan22 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   24.216564 npu0_vlink0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   24.216568 vd22-vlan22 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   ^C
   70 packets received by filter
   0 packets dropped by kernel

7. Check SD-WAN heath.

   When an ADVPN shortcut is out of SLA, traffic does not run on it. Shortcut vd2-1_0 is out of SLA.

   # diagnose system sdwan health-check
   Health Check(ping):
   Seq(1 vd2-1): state(alive), packet-loss(6.000%) latency(0.026), jitter(0.001), mos(4.401), bandwidth-up(1999), bandwidth-dw(0), bandwidth-bi(1999) sla_map=0x0
   Seq(1 vd2-1_0): state(alive), packet-loss(18.182%) latency(0.033), jitter(0.003), mos(4.395), bandwidth-up(2000), bandwidth-dw(0), bandwidth-bi(2000) sla_map=0x0
   Seq(2 vd2-2): state(alive), packet-loss(0.000%) latency(0.024), jitter(0.001), mos(4.404), bandwidth-up(0), bandwidth-dw(0), bandwidth-bi(0) sla_map=0x3
   Seq(2 vd2-2_0): state(alive), packet-loss(0.000%) latency(0.033), jitter(0.005), mos(4.404), bandwidth-up(0), bandwidth-dw(0), bandwidth-bi(0) sla_map=0x3

8. Check the sniffer packet:

   No traffic runs on Shortcut vd2-1_0 because it is out of SLA.

   # diagnose sniffer packet any "host 33.1.1.13" 4
   interfaces=[any]
   filters=[host 33.1.1.13]
   8.723075 vd22-vlan22 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   8.723077 npu0_vlink1 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   8.723084 vd2-vlan22 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
   8.723103 vd2-2_0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   8.723115 vd3-2_0 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
   8.723148 vd3-vlan33 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   8.723149 npu0_vlink0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   8.723154 vd33-vlan33 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
   8.723166 vd33-vlan33 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   8.723166 npu0_vlink1 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   8.723171 vd3-vlan33 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   8.723179 vd3-2_0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   8.723190 vd2-2_0 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   8.723195 vd2-vlan22 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   8.723195 npu0_vlink0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   8.723199 vd22-vlan22 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   
   17.202681 vd22-vlan22 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   17.202683 npu0_vlink1 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   17.202688 vd2-vlan22 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
   17.202704 vd2-2_0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   17.202716 vd3-2_0 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
   17.202727 vd3-vlan33 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   17.202728 npu0_vlink0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
   17.202733 vd33-vlan33 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
   17.202742 vd33-vlan33 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   17.202743 npu0_vlink1 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   17.202749 vd3-vlan33 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   17.202755 vd3-2_0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   17.202767 vd2-2_0 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   17.202771 vd2-vlan22 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   17.202772 npu0_vlink0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
   17.202777 vd22-vlan22 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply