Troubleshooting OCVPN

This document includes troubleshooting steps for the following OCVPN network topologies:

• Full mesh OCVPN.
• Hub-spoke OCVPN with ADVPN shortcut.
• Hub-spoke OCVPN with inter-overlay source NAT.

For OCVPN configurations in other network topologies, see the other OCVPN topics.

Troubleshooting full mesh network topology

• Branch_1 # diagnose vpn ocvpn status

  Current State        : Registered
  Topology             : Full-Mesh
  Role                 : Spoke
  Server Status        : Up
  Registration time    : Thu Feb 28 18:42:25 2019
  Update time          : Thu Feb 28 15:57:18 2019
  Poll time            : Fri Mar  1 15:02:28 2019

• Branch_1 # diagnose vpn ocvpn show-meta

  Topology :: auto
  License  :: full
  Members  :: 3
  Max-free :: 3

• Branch_1 # diagnose vpn ocvpn show-overlays

  QA
  PM

• Branch_1 # diagnose vpn ocvpn show-members

  Member: { "SN": "FG100D3G15801621", "IPv4": "172.16.200.1", "port": "500", "slot": 1000, "overlay": [ { "id": 0, "name": "QA", "subnets": [ "10.1.100.0\/255.255.255.0" ], "ip_range": "0.0.0.0-0.0.0.0" }, { "id": 1, "name": "PM", "subnets": [ "10.2.100.0\/255.255.255.0" ], "ip_range": "0.0.0.0-0.0.0.0" } ], "Name": "FortiGate-100D", "topology_role": "spoke" } 
  Member: { "SN": "FG900D3915800083", "IPv4": "172.16.200.4", "port": "500", "slot": 1001, "overlay": [ { "id": 0, "name": "QA", "subnets": [ "172.16.101.0\/255.255.255.0" ], "ip_range": "0.0.0.0-0.0.0.0" }, { "id": 1, "name": "PM", "subnets": [ "172.16.102.0\/255.255.255.0" ], "ip_range": "0.0.0.0-0.0.0.0" } ], "Name": "Branch3", "topology_role": "spoke" } 
  Member: { "SN": "FGT51E3U16001314", "IPv4": "172.16.200.199", "port": "500", "slot": 1002, "overlay": [ { "id": 0, "name": "QA", "subnets": [ "192.168.4.0\/255.255.255.0" ], "ip_range": "0.0.0.0-0.0.0.0" }, { "id": 1, "name": "PM", "subnets": [ "192.168.5.0\/255.255.255.0" ], "ip_range": "0.0.0.0-0.0.0.0" } ], "Name": "Branch2", "topology_role": "spoke" }  

• Branch_1 # diagnose vpn tunnel list

  list all ipsec tunnel in vd 0
  ------------------------------------------------------
  name=_OCVPN2-3.1 ver=2 serial=4 172.16.200.1:0->172.16.200.199:0 tun_id=172.16.200.199 dst_mtu=1500
  bound_if=11 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/528 options[0210]=create_dev frag-rfc  accept_traffic=1
  
  proxyid_num=2 child_num=0 refcnt=13 ilast=7 olast=0 ad=/0
  stat: rxp=0 txp=7 rxb=0 txb=588
  dpd: mode=on-demand on=1 idle=20000ms retry=3 count=0 seqno=6
  natt: mode=none draft=0 interval=0 remote_port=0
  proxyid=_OCVPN2-3.1 proto=0 sa=1 ref=2 serial=8 auto-negotiate
    src: 0:10.1.100.0-10.1.100.255:0
    dst: 0:192.168.4.0-192.168.4.255:0
    SA:  ref=3 options=18627 type=00 soft=0 mtu=1438 expire=42923/0B replaywin=2048
         seqno=8 esn=0 replaywin_lastseq=00000000 itn=0 qat=0
    life: type=01 bytes=0/0 timeout=42931/43200
    dec: spi=c34bb752 esp=aes key=16 3c5ceeff3cac1eaa2702b5ccb713ab9b
         ah=sha1 key=20 5903e358b3d8938ee64f0412887a0fe741ccb105
    enc: spi=b5bd4fe1 esp=aes key=16 8ae97a8abe24dae725d614d2a6efdcb0
         ah=sha1 key=20 9ec200d9c0cef9e1b7cf76e05dbf344c70f53214
    dec:pkts/bytes=0/0, enc:pkts/bytes=7/1064
  proxyid=_OCVPN2-3.1 proto=0 sa=0 ref=2 serial=1 auto-negotiate
    src: 0:10.1.100.0/255.255.255.0:0
    dst: 0:0.0.0.0/0.0.0.0:0
  ------------------------------------------------------
  name=_OCVPN2-4.1 ver=2 serial=6 172.16.200.1:0->172.16.200.4:0 tun_id=172.16.200.4 dst_mtu=1500
  bound_if=11 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/528 options[0210]=create_dev frag-rfc  accept_traffic=1
  
  proxyid_num=2 child_num=0 refcnt=11 ilast=19 olast=19 ad=/0
  stat: rxp=0 txp=0 rxb=0 txb=0
  dpd: mode=on-demand on=1 idle=20000ms retry=3 count=0 seqno=0
  natt: mode=none draft=0 interval=0 remote_port=0
  proxyid=_OCVPN2-4.1 proto=0 sa=1 ref=2 serial=7 auto-negotiate
    src: 0:10.1.100.0-10.1.100.255:0
    dst: 0:172.16.101.0-172.16.101.255:0
    SA:  ref=3 options=18627 type=00 soft=0 mtu=1438 expire=42911/0B replaywin=2048
         seqno=1 esn=0 replaywin_lastseq=00000000 itn=0 qat=0
    life: type=01 bytes=0/0 timeout=42931/43200
    dec: spi=c34bb750 esp=aes key=16 8c9844a8bcd3fda6c7bd8a4f2ec81ef1
         ah=sha1 key=20 680c7144346f5b52126cbad9f325821b048c7192
    enc: spi=f2d1f2d4 esp=aes key=16 f9625fc8590152829eb39eecab3a3999
         ah=sha1 key=20 5df8447416da541fa54dde9fa3e5c35fbfc4723f
    dec:pkts/bytes=0/0, enc:pkts/bytes=0/0
  proxyid=_OCVPN2-4.1 proto=0 sa=0 ref=2 serial=1 auto-negotiate
    src: 0:10.1.100.0/255.255.255.0:0
    dst: 0:0.0.0.0/0.0.0.0:0
  ------------------------------------------------------
  name=_OCVPN2-3.2 ver=2 serial=3 172.16.200.1:0->172.16.200.199:0 tun_id=172.16.200.199 dst_mtu=1500
  bound_if=11 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/528 options[0210]=create_dev frag-rfc  accept_traffic=1
  
  proxyid_num=2 child_num=0 refcnt=11 ilast=6 olast=6 ad=/0
  stat: rxp=0 txp=0 rxb=0 txb=0
  dpd: mode=on-demand on=1 idle=20000ms retry=3 count=0 seqno=0
  natt: mode=none draft=0 interval=0 remote_port=0
  proxyid=_OCVPN2-3.2 proto=0 sa=1 ref=2 serial=8 auto-negotiate
    src: 0:10.2.100.0-10.2.100.255:0
    dst: 0:192.168.5.0-192.168.5.255:0
    SA:  ref=3 options=18627 type=00 soft=0 mtu=1438 expire=42923/0B replaywin=2048
         seqno=1 esn=0 replaywin_lastseq=00000000 itn=0 qat=0
    life: type=01 bytes=0/0 timeout=42930/43200
    dec: spi=c34bb753 esp=aes key=16 58ddfad9a3699f1c49f3a9f369145c28
         ah=sha1 key=20 e749c7e6a7aaff119707c792eb73cd975127873b
    enc: spi=b5bd4fe2 esp=aes key=16 8f2366e653f5f9ad6587be1ce1905764
         ah=sha1 key=20 5347bf24e51219d483c0f7b058eceab202026204
    dec:pkts/bytes=0/0, enc:pkts/bytes=0/0
  proxyid=_OCVPN2-3.2 proto=0 sa=0 ref=2 serial=1 auto-negotiate
    src: 0:10.2.100.0/255.255.255.0:0
    dst: 0:0.0.0.0/0.0.0.0:0
  ------------------------------------------------------
  name=_OCVPN2-4.2 ver=2 serial=5 172.16.200.1:0->172.16.200.4:0 tun_id=172.16.200.4 dst_mtu=1500
  bound_if=11 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/528 options[0210]=create_dev frag-rfc  accept_traffic=1
  
  proxyid_num=2 child_num=0 refcnt=11 ilast=17 olast=17 ad=/0
  stat: rxp=0 txp=0 rxb=0 txb=0
  dpd: mode=on-demand on=1 idle=20000ms retry=3 count=0 seqno=0
  natt: mode=none draft=0 interval=0 remote_port=0
  proxyid=_OCVPN2-4.2 proto=0 sa=1 ref=2 serial=7 auto-negotiate
    src: 0:10.2.100.0-10.2.100.255:0
    dst: 0:172.16.102.0-172.16.102.255:0
    SA:  ref=3 options=18627 type=00 soft=0 mtu=1438 expire=42905/0B replaywin=2048
         seqno=1 esn=0 replaywin_lastseq=00000000 itn=0 qat=0
    life: type=01 bytes=0/0 timeout=42927/43200
    dec: spi=c34bb751 esp=aes key=16 41449ee5ea43d3e1f80df05fc632cd44
         ah=sha1 key=20 3ca2aea1c8764f35ccf987cdeca7cf6eb54331fb
    enc: spi=f2d1f2d5 esp=aes key=16 9010dd57e502c6296b27a4649a45a6ba
         ah=sha1 key=20 caf86a176ce04464221543f15fc3c63fc573b8ee
    dec:pkts/bytes=0/0, enc:pkts/bytes=0/0
  proxyid=_OCVPN2-4.2 proto=0 sa=0 ref=2 serial=1 auto-negotiate
    src: 0:10.2.100.0/255.255.255.0:0
    dst: 0:0.0.0.0/0.0.0.0:0

• Branch_1 # get router info routing-table all

  Routing table for VRF=0
  Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
         O - OSPF, IA - OSPF inter area
         N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
         E1 - OSPF external type 1, E2 - OSPF external type 2
         i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
         * - candidate default
  
  S*      0.0.0.0/0 [10/0] via 172.16.200.254, port1
  C       10.1.100.0/24 is directly connected, dmz
  C       10.2.100.0/24 is directly connected, loop
  C       11.101.1.0/24 is directly connected, wan1
  C       11.102.1.0/24 is directly connected, wan2
  S       192.168.5.0/24 [20/0] is directly connected, _OCVPN2-3.2
  C       172.16.200.0/24 is directly connected, port1
  S       172.16.101.0/24 [20/0] is directly connected, _OCVPN2-4.1
  S       172.16.102.0/24 [20/0] is directly connected, _OCVPN2-4.2
  S       192.168.4.0/24 [20/0] is directly connected, _OCVPN2-3.1

Troubleshooting hub-spoke with ADVPN shortcut

• Primary-Hub # diagnose vpn ocvpn status

  Current State        : Registered
  Topology             : Dual-Hub-Spoke
  Role                 : Primary-Hub
  Server Status        : Up
  Registration time    : Sat Mar  2 11:31:54 2019
  Poll time            : Sat Mar  2 11:46:02 2019

• Spoke1 # diagnose vpn ocvpn status

  Current State        : Registered
  Topology             : Dual-Hub-Spoke
  Role                 : Spoke
  Server Status        : Up
  Registration time    : Sat Mar  2 11:41:22 2019
  Poll time            : Sat Mar  2 11:46:44 2019

• Primary-Hub # diagnose vpn ocvpn show-members

  Member: { "sn": "FG900D3915800083", "ip_v4": "172.16.200.4", "port": 500, "slot": 0, "overlay": [ { "id": 0, "name": "QA", "subnets": [ "172.16.101.0\/255.255.255.0" ], "ip_range": "0.0.0.0-0.0.0.0" }, { "id": 1, "name": "PM", "subnets": [ "172.16.102.0\/255.255.255.0" ], "ip_range": "0.0.0.0-0.0.0.0" } ], "name": "Primary-Hub", "topology_role": "primary_hub", "eap": "disable", "auto_discovery": "enable" }
  Member: { "sn": "FG100D3G15828488", "ip_v4": "172.16.200.2", "port": 500, "slot": 1, "overlay": [ { "id": 0, "name": "QA", "subnets": [ "172.16.101.0\/255.255.255.0" ], "ip_range": "0.0.0.0-0.0.0.0" }, { "id": 1, "name": "PM", "subnets": [ "172.16.102.0\/255.255.255.0" ], "ip_range": "0.0.0.0-0.0.0.0" } ], "name": "Secondary-Hub", "topology_role": "secondary_hub", "eap": "disable", "auto_discovery": "enable" }
  Member: { "sn": "FG100D3G15801621", "ip_v4": "172.16.200.1", "port": 500, "slot": 1000, "overlay": [ { "id": 0, "name": "QA", "subnets": [ "10.1.100.0\/255.255.255.0" ], "ip_range": "0.0.0.0-0.0.0.0" }, { "id": 1, "name": "PM", "subnets": [ "10.2.100.0\/255.255.255.0" ], "ip_range": "0.0.0.0-0.0.0.0" } ], "name": "Spoke1", "topology_role": "spoke" }
  Member: { "sn": "FGT51E3U16001314", "ip_v4": "172.16.200.3", "port": 500, "slot": 1001, "overlay": [ { "id": 0, "name": "QA", "subnets": [ "192.168.4.0\/255.255.255.0" ], "ip_range": "0.0.0.0-0.0.0.0" }, { "id": 1, "name": "PM", "subnets": [ "192.168.5.0\/255.255.255.0" ], "ip_range": "0.0.0.0-0.0.0.0" } ], "name": "Spoke2", "topology_role": "spoke" }

• Primary-Hub # diagnose vpn ocvpn show-meta

  Topology :: auto
  License  :: full
  Members  :: 4
  Max-free :: 3

• Primary-Hub # diagnose vpn ocvpn show-overlays

  QA
  PM

• Spoke1 # diagnose vpn tunnel list

  list all ipsec tunnel in vd 0
  ------------------------------------------------------
  name=_OCVPN2-0.0 ver=2 serial=6 172.16.200.1:0->172.16.200.4:0 tun_id=172.16.200.4 dst_mtu=1500
  bound_if=11 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/528 options[0210]=create_dev frag-rfc  accept_traffic=1
  
  proxyid_num=1 child_num=0 refcnt=11 ilast=0 olast=0 ad=r/2
  stat: rxp=1 txp=34 rxb=152 txb=2856
  dpd: mode=on-idle on=1 idle=20000ms retry=3 count=0 seqno=46
  natt: mode=none draft=0 interval=0 remote_port=0
  proxyid=_OCVPN2-0.0 proto=0 sa=1 ref=2 serial=1 auto-negotiate adr
    src: 0:10.1.100.0/255.255.255.0:0
    dst: 0:0.0.0.0/0.0.0.0:0
    SA:  ref=3 options=1a227 type=00 soft=0 mtu=1438 expire=42895/0B replaywin=2048
         seqno=1 esn=0 replaywin_lastseq=00000000 itn=0 qat=0
    life: type=01 bytes=0/0 timeout=42901/43200
    dec: spi=048477c7 esp=aes key=16 240e064c0f1c980ca31980b9e7605c9d
         ah=sha1 key=20 6ff022cbebcaff4c5de62eefb2e6180c40a3adb2
    enc: spi=dfcffa86 esp=aes key=16 862208de164a02af377756c2bcabd588
         ah=sha1 key=20 af6e54781fd42d7a2ba2119ec95d0f95629c8448
    dec:pkts/bytes=0/0, enc:pkts/bytes=0/0
  ------------------------------------------------------
  name=_OCVPN2-1.0 ver=2 serial=8 172.16.200.1:0->172.16.200.2:0 tun_id=172.16.200.2 dst_mtu=1500
  bound_if=11 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/528 options[0210]=create_dev frag-rfc  accept_traffic=0
  
  proxyid_num=1 child_num=0 refcnt=10 ilast=934 olast=934 ad=/0
  stat: rxp=0 txp=0 rxb=0 txb=0
  dpd: mode=on-idle on=0 idle=20000ms retry=3 count=0 seqno=1
  natt: mode=none draft=0 interval=0 remote_port=0
  proxyid=_OCVPN2-1.0 proto=0 sa=0 ref=2 serial=1 auto-negotiate adr
    src: 0:10.1.100.0/255.255.255.0:0
    dst: 0:0.0.0.0/0.0.0.0:0
  ------------------------------------------------------
  name=_OCVPN2-0.1 ver=2 serial=5 172.16.200.1:0->172.16.200.4:0 tun_id=172.16.200.4 dst_mtu=1500
  bound_if=11 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/528 options[0210]=create_dev frag-rfc  accept_traffic=1
  
  proxyid_num=1 child_num=0 refcnt=11 ilast=12 olast=12 ad=r/2
  stat: rxp=0 txp=0 rxb=0 txb=0
  dpd: mode=on-idle on=1 idle=20000ms retry=3 count=0 seqno=46
  natt: mode=none draft=0 interval=0 remote_port=0
  proxyid=_OCVPN2-0.1 proto=0 sa=1 ref=2 serial=1 auto-negotiate adr
    src: 0:10.2.100.0/255.255.255.0:0
    dst: 0:0.0.0.0/0.0.0.0:0
    SA:  ref=3 options=1a227 type=00 soft=0 mtu=1438 expire=42895/0B replaywin=2048
         seqno=1 esn=0 replaywin_lastseq=00000000 itn=0 qat=0
    life: type=01 bytes=0/0 timeout=42901/43200
    dec: spi=048477c8 esp=aes key=16 701ec608767f4988b76c2f662464e654
         ah=sha1 key=20 93c65d106dc610d7ee3f04487f08601a9e00ffdd
    enc: spi=dfcffa87 esp=aes key=16 02b2d04dce3d81ebab69e128d45cb7ca
         ah=sha1 key=20 4a9283847f852c83a75691fad44d07d8409a2267
    dec:pkts/bytes=0/0, enc:pkts/bytes=0/0
  ------------------------------------------------------
  name=_OCVPN2-1.1 ver=2 serial=7 172.16.200.1:0->172.16.200.2:0 tun_id=172.16.200.2 dst_mtu=1500
  bound_if=11 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/528 options[0210]=create_dev frag-rfc  accept_traffic=0
  
  proxyid_num=1 child_num=0 refcnt=10 ilast=934 olast=934 ad=/0
  stat: rxp=0 txp=0 rxb=0 txb=0
  dpd: mode=on-idle on=0 idle=20000ms retry=3 count=0 seqno=1
  natt: mode=none draft=0 interval=0 remote_port=0
  proxyid=_OCVPN2-1.1 proto=0 sa=0 ref=2 serial=1 auto-negotiate adr
    src: 0:10.2.100.0/255.255.255.0:0
    dst: 0:0.0.0.0/0.0.0.0:0

• Spoke1 # get router info routing-table all

  Routing table for VRF=0
  Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
         O - OSPF, IA - OSPF inter area
         N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
         E1 - OSPF external type 1, E2 - OSPF external type 2
         i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
         * - candidate default
  
  S*      0.0.0.0/0 [10/0] via 172.16.200.254, port1
  C       10.1.100.0/24 is directly connected, dmz
  C       10.2.100.0/24 is directly connected, loop
  C       11.101.1.0/24 is directly connected, wan1
  C       11.102.1.0/24 is directly connected, wan2
  S       172.16.102.0/24 [20/0] is directly connected, _OCVPN2-0.1
  C       172.16.200.0/24 is directly connected, port1
  S       172.16.101.0/24 [20/0] is directly connected, _OCVPN2-0.0
  S       192.168.4.0/24 [20/0] is directly connected, _OCVPN2-0.0
  S       192.168.5.0/24 [20/0] is directly connected, _OCVPN2-0.1

• Generate traffic from spoke1 to spoke2 to trigger the ADVPN shortcut and check the VPN tunnel and routing-table again on spoke1.

  branch1 # diagnose vpn tunnel list 
  list all ipsec tunnel in vd 0
  ------------------------------------------------------
  name=_OCVPN2-0.0_0 ver=2 serial=a 172.16.200.1:0->172.16.200.3:0 tun_id=172.16.200.3 dst_mtu=1500
  bound_if=11 lgwy=static/1 tun=intf/0 mode=dial_inst/3 encap=none/720 options[02d0]=create_dev no-sysctl rgwy-chg frag-rfc  accept_traffic=1
  
   parent=_OCVPN2-0.0 index=0
  proxyid_num=1 child_num=0 refcnt=14 ilast=0 olast=0 ad=r/2
  stat: rxp=7 txp=7 rxb=1064 txb=588
  dpd: mode=on-idle on=1 idle=20000ms retry=3 count=0 seqno=0
  natt: mode=none draft=0 interval=0 remote_port=0
  proxyid=_OCVPN2-0.0 proto=0 sa=1 ref=2 serial=1 auto-negotiate add-route adr
    src: 0:10.1.100.0-10.1.100.255:0
    dst: 0:192.168.4.0-192.168.4.255:0
    SA:  ref=3 options=1a227 type=00 soft=0 mtu=1438 expire=43180/0B replaywin=2048
         seqno=8 esn=0 replaywin_lastseq=00000008 itn=0 qat=0
    life: type=01 bytes=0/0 timeout=43187/43200
    dec: spi=048477c9 esp=aes key=16 27c35d53793013ef24cf887561e9f313
         ah=sha1 key=20 2c8cfd328c3b29104db0ca74a00c6063f46cafe4
    enc: spi=fb9e13fd esp=aes key=16 9d0d3bf6c84b7ddaf9d9196fe74002ed
         ah=sha1 key=20 d1f541db787dea384c6a4df16fc228abeb7ae334
    dec:pkts/bytes=7/588, enc:pkts/bytes=7/1064
  ------------------------------------------------------
  name=_OCVPN2-0.0 ver=2 serial=6 172.16.200.1:0->172.16.200.4:0 tun_id=172.16.200.4 dst_mtu=1500
  bound_if=11 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/528 options[0210]=create_dev frag-rfc  accept_traffic=1
  
  proxyid_num=1 child_num=1 refcnt=12 ilast=7 olast=7 ad=r/2
  stat: rxp=2 txp=35 rxb=304 txb=2940
  dpd: mode=on-idle on=1 idle=20000ms retry=3 count=0 seqno=65
  natt: mode=none draft=0 interval=0 remote_port=0
  proxyid=_OCVPN2-0.0 proto=0 sa=1 ref=2 serial=1 auto-negotiate adr
    src: 0:10.1.100.0/255.255.255.0:0
    dst: 0:0.0.0.0/0.0.0.0:0
    SA:  ref=3 options=1a227 type=00 soft=0 mtu=1438 expire=42500/0B replaywin=2048
         seqno=2 esn=0 replaywin_lastseq=00000002 itn=0 qat=0
    life: type=01 bytes=0/0 timeout=42901/43200
    dec: spi=048477c7 esp=aes key=16 240e064c0f1c980ca31980b9e7605c9d
         ah=sha1 key=20 6ff022cbebcaff4c5de62eefb2e6180c40a3adb2
    enc: spi=dfcffa86 esp=aes key=16 862208de164a02af377756c2bcabd588
         ah=sha1 key=20 af6e54781fd42d7a2ba2119ec95d0f95629c8448
    dec:pkts/bytes=1/84, enc:pkts/bytes=1/152
  ------------------------------------------------------
  name=_OCVPN2-1.0 ver=2 serial=8 172.16.200.1:0->172.16.200.2:0 tun_id=172.16.200.2 dst_mtu=1500
  bound_if=11 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/528 options[0210]=create_dev frag-rfc  accept_traffic=0
  
  proxyid_num=1 child_num=0 refcnt=10 ilast=1328 olast=1328 ad=/0
  stat: rxp=0 txp=0 rxb=0 txb=0
  dpd: mode=on-idle on=0 idle=20000ms retry=3 count=0 seqno=1
  natt: mode=none draft=0 interval=0 remote_port=0
  proxyid=_OCVPN2-1.0 proto=0 sa=0 ref=2 serial=1 auto-negotiate adr
    src: 0:10.1.100.0/255.255.255.0:0
    dst: 0:0.0.0.0/0.0.0.0:0
  ------------------------------------------------------
  name=_OCVPN2-0.1 ver=2 serial=5 172.16.200.1:0->172.16.200.4:0 tun_id=172.16.200.4 dst_mtu=1500
  bound_if=11 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/528 options[0210]=create_dev frag-rfc  accept_traffic=1
  
  proxyid_num=1 child_num=0 refcnt=11 ilast=5 olast=5 ad=r/2
  stat: rxp=0 txp=0 rxb=0 txb=0
  dpd: mode=on-idle on=1 idle=20000ms retry=3 count=0 seqno=66
  natt: mode=none draft=0 interval=0 remote_port=0
  proxyid=_OCVPN2-0.1 proto=0 sa=1 ref=2 serial=1 auto-negotiate adr
    src: 0:10.2.100.0/255.255.255.0:0
    dst: 0:0.0.0.0/0.0.0.0:0
    SA:  ref=3 options=1a227 type=00 soft=0 mtu=1438 expire=42500/0B replaywin=2048
         seqno=1 esn=0 replaywin_lastseq=00000000 itn=0 qat=0
    life: type=01 bytes=0/0 timeout=42901/43200
    dec: spi=048477c8 esp=aes key=16 701ec608767f4988b76c2f662464e654
         ah=sha1 key=20 93c65d106dc610d7ee3f04487f08601a9e00ffdd
    enc: spi=dfcffa87 esp=aes key=16 02b2d04dce3d81ebab69e128d45cb7ca
         ah=sha1 key=20 4a9283847f852c83a75691fad44d07d8409a2267
    dec:pkts/bytes=0/0, enc:pkts/bytes=0/0
  ------------------------------------------------------
  name=_OCVPN2-1.1 ver=2 serial=7 172.16.200.1:0->172.16.200.2:0 tun_id=172.16.200.2 dst_mtu=1500
  bound_if=11 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/528 options[0210]=create_dev frag-rfc  accept_traffic=0
  
  proxyid_num=1 child_num=0 refcnt=10 ilast=1328 olast=1328 ad=/0
  stat: rxp=0 txp=0 rxb=0 txb=0
  dpd: mode=on-idle on=0 idle=20000ms retry=3 count=0 seqno=1
  natt: mode=none draft=0 interval=0 remote_port=0
  proxyid=_OCVPN2-1.1 proto=0 sa=0 ref=2 serial=1 auto-negotiate adr
    src: 0:10.2.100.0/255.255.255.0:0
    dst: 0:0.0.0.0/0.0.0.0:0
  
  
  Routing table for VRF=0
  Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
         O - OSPF, IA - OSPF inter area
         N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
         E1 - OSPF external type 1, E2 - OSPF external type 2
         i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
         * - candidate default
  
  S*      0.0.0.0/0 [10/0] via 172.16.200.254, port1
  C       10.1.100.0/24 is directly connected, dmz
  C       10.2.100.0/24 is directly connected, loop
  C       11.101.1.0/24 is directly connected, wan1
  C       11.102.1.0/24 is directly connected, wan2
  S       172.16.102.0/24 [20/0] is directly connected, _OCVPN2-0.1
  C       172.16.200.0/24 is directly connected, port1
  S       172.16.101.0/24 [20/0] is directly connected, _OCVPN2-0.0
  S       192.168.4.0/24 [15/0] via 172.16.200.3, _OCVPN2-0.0_0
  S       192.168.5.0/24 [20/0] is directly connected, _OCVPN2-0.1

• Simulate the primary hub being unavailable where all spokes' dialup VPN tunnels will switch to the secondary hub, to check VPN tunnel status and routing-table.

  list all ipsec tunnel in vd 0
  ------------------------------------------------------
  name=_OCVPN2-0.0 ver=2 serial=6 172.16.200.1:0->172.16.200.4:0 tun_id=172.16.200.4 dst_mtu=1500
  bound_if=11 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/528 options[0210]=create_dev frag-rfc  accept_traffic=0
  
  proxyid_num=1 child_num=0 refcnt=10 ilast=25 olast=25 ad=/0
  stat: rxp=0 txp=0 rxb=0 txb=0
  dpd: mode=on-idle on=0 idle=20000ms retry=3 count=0 seqno=82
  natt: mode=none draft=0 interval=0 remote_port=0
  proxyid=_OCVPN2-0.0 proto=0 sa=0 ref=2 serial=1 auto-negotiate adr
    src: 0:10.1.100.0/255.255.255.0:0
    dst: 0:0.0.0.0/0.0.0.0:0
  ------------------------------------------------------
  name=_OCVPN2-1.0 ver=2 serial=8 172.16.200.1:0->172.16.200.2:0 tun_id=172.16.200.2 dst_mtu=1500
  bound_if=11 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/528 options[0210]=create_dev frag-rfc  accept_traffic=1
  
  proxyid_num=1 child_num=0 refcnt=11 ilast=14 olast=14 ad=r/2
  stat: rxp=0 txp=0 rxb=0 txb=0
  dpd: mode=on-idle on=1 idle=20000ms retry=3 count=0 seqno=9
  natt: mode=none draft=0 interval=0 remote_port=0
  proxyid=_OCVPN2-1.0 proto=0 sa=1 ref=2 serial=1 auto-negotiate adr
    src: 0:10.1.100.0/255.255.255.0:0
    dst: 0:0.0.0.0/0.0.0.0:0
    SA:  ref=3 options=1a227 type=00 soft=0 mtu=1438 expire=42723/0B replaywin=2048
         seqno=1 esn=0 replaywin_lastseq=00000000 itn=0 qat=0
    life: type=01 bytes=0/0 timeout=42898/43200
    dec: spi=048477cd esp=aes key=16 9bb363a32378b5897cd42890c92df811
         ah=sha1 key=20 2ed40583b9544e37867349b4adc7c013024d7e17
    enc: spi=f345fb42 esp=aes key=16 3ea31dff3310b245700a131db4565851
         ah=sha1 key=20 522862dfb232514b845e436133b148da0e67b7c4
    dec:pkts/bytes=0/0, enc:pkts/bytes=0/0
  ------------------------------------------------------
  name=_OCVPN2-0.1 ver=2 serial=5 172.16.200.1:0->172.16.200.4:0 tun_id=172.16.200.4 dst_mtu=1500
  bound_if=11 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/528 options[0210]=create_dev frag-rfc  accept_traffic=0
  
  proxyid_num=1 child_num=0 refcnt=10 ilast=19 olast=19 ad=/0
  stat: rxp=0 txp=0 rxb=0 txb=0
  dpd: mode=on-idle on=0 idle=20000ms retry=3 count=0 seqno=83
  natt: mode=none draft=0 interval=0 remote_port=0
  proxyid=_OCVPN2-0.1 proto=0 sa=0 ref=2 serial=1 auto-negotiate adr
    src: 0:10.2.100.0/255.255.255.0:0
    dst: 0:0.0.0.0/0.0.0.0:0
  ------------------------------------------------------
  name=_OCVPN2-1.1 ver=2 serial=7 172.16.200.1:0->172.16.200.2:0 tun_id=172.16.200.2 dst_mtu=1500
  bound_if=11 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/528 options[0210]=create_dev frag-rfc  accept_traffic=1
  
  proxyid_num=1 child_num=0 refcnt=11 ilast=12 olast=12 ad=r/2
  stat: rxp=0 txp=0 rxb=0 txb=0
  dpd: mode=on-idle on=1 idle=20000ms retry=3 count=0 seqno=9
  natt: mode=none draft=0 interval=0 remote_port=0
  proxyid=_OCVPN2-1.1 proto=0 sa=1 ref=2 serial=1 auto-negotiate adr
    src: 0:10.2.100.0/255.255.255.0:0
    dst: 0:0.0.0.0/0.0.0.0:0
    SA:  ref=3 options=1a227 type=00 soft=0 mtu=1438 expire=42728/0B replaywin=2048
         seqno=1 esn=0 replaywin_lastseq=00000000 itn=0 qat=0
    life: type=01 bytes=0/0 timeout=42902/43200
    dec: spi=048477cf esp=aes key=16 b6f0ca7564abcd8559b5b0ebb3fd04c1
         ah=sha1 key=20 4130d040554b39daca72adac7583b9cc83cce3c8
    enc: spi=f345fb43 esp=aes key=16 727582f20fcedff884ba693ed2164bcd
         ah=sha1 key=20 b0a625803fde701ed9d28d256079e908954b7fc8
    dec:pkts/bytes=0/0, enc:pkts/bytes=0/0
  
  
  Routing table for VRF=0
  Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
         O - OSPF, IA - OSPF inter area
         N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
         E1 - OSPF external type 1, E2 - OSPF external type 2
         i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
         * - candidate default
  
  S*      0.0.0.0/0 [10/0] via 172.16.200.254, port1
  C       10.1.100.0/24 is directly connected, dmz
  C       10.2.100.0/24 is directly connected, loop
  C       11.101.1.0/24 is directly connected, wan1
  C       11.102.1.0/24 is directly connected, wan2
  S       172.16.102.0/24 [21/0] is directly connected, _OCVPN2-1.1
  C       172.16.200.0/24 is directly connected, port1
  S       172.16.101.0/24 [21/0] is directly connected, _OCVPN2-1.0
  S       192.168.4.0/24 [21/0] is directly connected, _OCVPN2-1.0
  S       192.168.5.0/24 [21/0] is directly connected, _OCVPN2-1.1

Troubleshooting hub-spoke with inter-overlay source NAT

• Primary-Hub # diagnose vpn ocvpn status

  Current State        : Registered
  Topology             : Dual-Hub-Spoke
  Role                 : Primary-Hub
  Server Status        : Up
  Registration time    : Sat Mar  2 11:31:54 2019
  Update time          : Sat Mar  2 13:57:05 2019
  Poll time            : Sat Mar  2 14:03:31 2019

• Spoke1 # diagnose vpn ocvpn status

  Current State        : Registered
  Topology             : Dual-Hub-Spoke
  Role                 : Spoke
  Server Status        : Up
  Registration time    : Sat Mar  2 13:58:01 2019
  Poll time            : Sat Mar  2 14:04:22 2019

• Primary-Hub # diagnose vpn ocvpn show-members

  Member: { "sn": "FG900D3915800083", "ip_v4": "172.16.200.4", "port": 500, "slot": 0, "overlay": [ { "id": 0, "name": "QA", "subnets": [ "172.16.101.0\/255.255.255.0" ], "ip_range": "172.16.101.100-172.16.101.200" }, { "id": 1, "name": "PM", "subnets": [ "172.16.102.0\/255.255.255.0" ], "ip_range": "172.16.102.100-172.16.102.200" } ], "name": "Primary-Hub", "topology_role": "primary_hub", "eap": "disable", "auto_discovery": "enable" }
  Member: { "sn": "FG100D3G15828488", "ip_v4": "172.16.200.2", "port": 500, "slot": 1, "overlay": [ { "id": 0, "name": "QA", "subnets": [ "172.16.101.0\/255.255.255.0" ], "ip_range": "0.0.0.0-0.0.0.0" }, { "id": 1, "name": "PM", "subnets": [ "172.16.102.0\/255.255.255.0" ], "ip_range": "0.0.0.0-0.0.0.0" } ], "name": "Secondary-Hub", "topology_role": "secondary_hub", "eap": "disable", "auto_discovery": "enable" }
  Member: { "sn": "FGT51E3U16001314", "ip_v4": "172.16.200.3", "port": 500, "slot": 1001, "overlay": [ { "id": 0, "name": "QA", "subnets": [ "192.168.4.0\/255.255.255.0" ], "ip_range": "0.0.0.0-0.0.0.0" }, { "id": 1, "name": "PM", "subnets": [ "192.168.5.0\/255.255.255.0" ], "ip_range": "0.0.0.0-0.0.0.0" } ], "name": "Spoke2", "topology_role": "spoke" }
  Member: { "sn": "FG100D3G15801621", "ip_v4": "172.16.200.1", "port": 500, "slot": 1000, "overlay": [ { "id": 0, "name": "QA", "subnets": [ "10.1.100.0\/255.255.255.0" ], "ip_range": "0.0.0.0-0.0.0.0" }, { "id": 1, "name": "PM", "subnets": [ "10.2.100.0\/255.255.255.0" ], "ip_range": "0.0.0.0-0.0.0.0" } ], "name": "Spoke1", "topology_role": "spoke" }

• Primary-Hub # diagnose vpn ocvpn show-meta

  Topology :: auto
  License  :: full
  Members  :: 4
  Max-free :: 3

• Primary-Hub # diagnose vpn ocvpn show-overlays

  QA
  PM

• Spoke1 # diagnose vpn tunnel list

  list all ipsec tunnel in vd 0
  ------------------------------------------------------
  name=_OCVPN2-0.0 ver=2 serial=c 172.16.200.1:0->172.16.200.4:0 tun_id=172.16.200.4 dst_mtu=1500
  bound_if=11 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/528 options[0210]=create_dev frag-rfc  accept_traffic=1
  
  proxyid_num=3 child_num=0 refcnt=13 ilast=17 olast=17 ad=/0
  stat: rxp=0 txp=0 rxb=0 txb=0
  dpd: mode=on-idle on=1 idle=20000ms retry=3 count=0 seqno=29
  natt: mode=none draft=0 interval=0 remote_port=0
  proxyid=_OCVPN2-0.0 proto=0 sa=1 ref=2 serial=1 auto-negotiate
    src: 0:10.1.100.0/255.255.255.0:0
    dst: 0:0.0.0.0/0.0.0.0:0
    SA:  ref=3 options=18227 type=00 soft=0 mtu=1438 expire=42299/0B replaywin=2048
         seqno=1 esn=0 replaywin_lastseq=00000000 itn=0 qat=0
    life: type=01 bytes=0/0 timeout=42899/43200
    dec: spi=0484795d esp=aes key=16 10eeb76fadd49f00c333350d83509095
         ah=sha1 key=20 971bde5dcfca7e52fd1573cb3489e9c855f6154e
    enc: spi=dfcffaaa esp=aes key=16 d07a4dd683ee093af2dca9485aa436eb
         ah=sha1 key=20 65369be35d5ecad8cae63557318419cd6005c230
    dec:pkts/bytes=0/0, enc:pkts/bytes=0/0
  proxyid=_OCVPN2-0.0_nat proto=0 sa=1 ref=2 serial=3 auto-negotiate
    src: 0:172.16.101.101-172.16.101.101:0
    dst: 0:0.0.0.0-255.255.255.255:0
    SA:  ref=3 options=18627 type=00 soft=0 mtu=1438 expire=42303/0B replaywin=2048
         seqno=1 esn=0 replaywin_lastseq=00000000 itn=0 qat=0
    life: type=01 bytes=0/0 timeout=42898/43200
    dec: spi=04847961 esp=aes key=16 ea181036b02e8bc8711fb520b3e98a60
         ah=sha1 key=20 b3c449d96d5d3f090975087a62447f6918ce7930
    enc: spi=dfcffaac esp=aes key=16 f7ea5e42e9443698e6b8b32161ace40e
         ah=sha1 key=20 a7e36dd1ec0bdb6eff0aa66e442707427400c700
    dec:pkts/bytes=0/0, enc:pkts/bytes=0/0
  proxyid=_OCVPN2-0.0_nat proto=0 sa=0 ref=2 serial=2 auto-negotiate
    src: 0:0.0.0.0/0.0.0.0:0
    dst: 0:0.0.0.0/0.0.0.0:0
  ------------------------------------------------------
  name=_OCVPN2-1.0 ver=2 serial=e 172.16.200.1:0->172.16.200.2:0 tun_id=172.16.200.2 dst_mtu=0
  bound_if=11 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/528 options[0210]=create_dev frag-rfc  accept_traffic=1
  
  proxyid_num=2 child_num=0 refcnt=10 ilast=599 olast=599 ad=/0
  stat: rxp=0 txp=0 rxb=0 txb=0
  dpd: mode=on-idle on=0 idle=20000ms retry=3 count=0 seqno=0
  natt: mode=none draft=0 interval=0 remote_port=0
  proxyid=_OCVPN2-1.0 proto=0 sa=0 ref=2 serial=1 auto-negotiate
    src: 0:10.1.100.0/255.255.255.0:0
    dst: 0:0.0.0.0/0.0.0.0:0
  proxyid=_OCVPN2-1.0_nat proto=0 sa=0 ref=2 serial=2 auto-negotiate
    src: 0:0.0.0.0/0.0.0.0:0
    dst: 0:0.0.0.0/0.0.0.0:0
  ------------------------------------------------------
  name=_OCVPN2-0.1 ver=2 serial=b 172.16.200.1:0->172.16.200.4:0 tun_id=172.16.200.4 dst_mtu=1500
  bound_if=11 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/528 options[0210]=create_dev frag-rfc  accept_traffic=1
  
  proxyid_num=3 child_num=0 refcnt=13 ilast=17 olast=17 ad=/0
  stat: rxp=0 txp=0 rxb=0 txb=0
  dpd: mode=on-idle on=1 idle=20000ms retry=3 count=0 seqno=29
  natt: mode=none draft=0 interval=0 remote_port=0
  proxyid=_OCVPN2-0.1 proto=0 sa=1 ref=2 serial=1 auto-negotiate
    src: 0:10.2.100.0/255.255.255.0:0
    dst: 0:0.0.0.0/0.0.0.0:0
    SA:  ref=3 options=18227 type=00 soft=0 mtu=1438 expire=42297/0B replaywin=2048
         seqno=1 esn=0 replaywin_lastseq=00000000 itn=0 qat=0
    life: type=01 bytes=0/0 timeout=42897/43200
    dec: spi=0484795e esp=aes key=16 106eaa95a2be64b566e7d1ca0aa88f6a
         ah=sha1 key=20 5dddfba7070b03d5a31931d41db06ff96e7bc542
    enc: spi=dfcffaab esp=aes key=16 29c774dbd7e54464ee298c381e71a94e
         ah=sha1 key=20 c3da7372789c0a53b3752e69baaba1a42d798820
    dec:pkts/bytes=0/0, enc:pkts/bytes=0/0
  proxyid=_OCVPN2-0.1_nat proto=0 sa=1 ref=2 serial=3 auto-negotiate
    src: 0:172.16.102.101-172.16.102.101:0
    dst: 0:0.0.0.0-255.255.255.255:0
    SA:  ref=3 options=18627 type=00 soft=0 mtu=1438 expire=42307/0B replaywin=2048
         seqno=1 esn=0 replaywin_lastseq=00000000 itn=0 qat=0
    life: type=01 bytes=0/0 timeout=42902/43200
    dec: spi=04847962 esp=aes key=16 b7daa5807cfa86906592a012a9d2478f
         ah=sha1 key=20 39c8bb4c9e3f1e9e451f22c58a172ff01155055d
    enc: spi=dfcffaad esp=aes key=16 2ecc644def4cebe6b0c4b7729da43d8e
         ah=sha1 key=20 469c6f319e83bd73468f55d430566afcd6215138
    dec:pkts/bytes=0/0, enc:pkts/bytes=0/0
  proxyid=_OCVPN2-0.1_nat proto=0 sa=0 ref=2 serial=2 auto-negotiate
    src: 0:0.0.0.0/0.0.0.0:0
    dst: 0:0.0.0.0/0.0.0.0:0
  ------------------------------------------------------
  name=_OCVPN2-1.1 ver=2 serial=d 172.16.200.1:0->172.16.200.2:0 tun_id=172.16.200.2 dst_mtu=0
  bound_if=11 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/528 options[0210]=create_dev frag-rfc  accept_traffic=1
  
  proxyid_num=2 child_num=0 refcnt=10 ilast=599 olast=599 ad=/0
  stat: rxp=0 txp=0 rxb=0 txb=0
  dpd: mode=on-idle on=0 idle=20000ms retry=3 count=0 seqno=0
  natt: mode=none draft=0 interval=0 remote_port=0
  proxyid=_OCVPN2-1.1 proto=0 sa=0 ref=2 serial=1 auto-negotiate
    src: 0:10.2.100.0/255.255.255.0:0
    dst: 0:0.0.0.0/0.0.0.0:0
  proxyid=_OCVPN2-1.1_nat proto=0 sa=0 ref=2 serial=2 auto-negotiate
    src: 0:0.0.0.0/0.0.0.0:0
    dst: 0:0.0.0.0/0.0.0.0:0

• Spoke1 # get router info routing-table all

  Routing table for VRF=0
  Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
         O - OSPF, IA - OSPF inter area
         N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
         E1 - OSPF external type 1, E2 - OSPF external type 2
         i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
         * - candidate default
  
  S*      0.0.0.0/0 [10/0] via 172.16.200.254, port1
  C       10.1.100.0/24 is directly connected, dmz
  C       10.2.100.0/24 is directly connected, loop
  C       11.101.1.0/24 is directly connected, wan1
  C       11.102.1.0/24 is directly connected, wan2
  S       172.16.101.0/24 [20/0] is directly connected, _OCVPN2-0.1
  C       172.16.101.101/32 is directly connected, _OCVPN2-0.1
  C       172.16.200.0/24 is directly connected, port1
  S       172.16.102.0/24 [20/0] is directly connected, _OCVPN2-0.0
  C       172.16.102.101/32 is directly connected, _OCVPN2-0.0
  S       192.168.4.0/24 [20/0] is directly connected, _OCVPN2-0.0
  S       192.168.5.0/24 [20/0] is directly connected, _OCVPN2-0.1

• Spoke1 # show firewall policy

   ..............................
  
      edit 9
          set name "_OCVPN2-1.1_nat"
          set uuid 3f7a84b8-3d36-51e9-ee97-8f418c91e666
          set srcintf "any"
          set dstintf "_OCVPN2-1.1"
          set srcaddr "all"
          set dstaddr "_OCVPN2-1.1_remote_networks"
          set action accept
          set schedule "always"
          set service "ALL"
          set comments "Generated by OCVPN Cloud Service."
          set nat enable
      next
      edit 12
          set name "_OCVPN2-1.0_nat"
          set uuid 3fafec98-3d36-51e9-80c0-5d99325bad83
          set srcintf "any"
          set dstintf "_OCVPN2-1.0"
          set srcaddr "all"
          set dstaddr "_OCVPN2-1.0_remote_networks"
          set action accept
          set schedule "always"
          set service "ALL"
          set comments "Generated by OCVPN Cloud Service."
          set nat enable
      next
     .................................