Fortinet black logo

Administration Guide

Connectivity issues

Connectivity issues

One of your first tests when configuring a new device should be to determine whether data such as video is received, and whether commands and schedules are being sent to it. You should also test whether notification email can be sent, and user accounts can log in to the GUI and view live video feeds.

After initial setup, connectivity should not be interrupted. FortiRecorder might sometimes be able to recover if, for example, a camera receives a new IP address from the DHCP server. However this can cause disruptions to recording, and camera log messages such as:

Camera 'c1' experienced an interruption that may result in a loss of recording.

If connections fail or perform erratically, examine the following in order:

  1. Checking hardware connections
  2. Bringing up network interfaces
  3. Examining the ARP table
  4. Examining routing
  5. Discovery fails
  6. Connectivity issues
  7. Unauthorized DHCP clients or DHCP pool exhaustion
  8. Examining IP sessions

  9. Resolving IP address conflicts
  10. Examining live video streams
  11. Packet tracing
Note

Troubleshooting is in order from more low-level OSI layers of your network (physical, link/Ethernet, network/IP) to the higher-level, more application-specific protocols. If your network is not new, then lower-level networking is usually already working, so you might want to start with later sections instead. Application-layer protocols such as RTSP are more often specific to FortiRecorder, cameras, and surveillance devices.

Checking hardware connections

If no traffic arrives on a network interface even though the configuration appears to be correct, or if network performance is less than you expect, then it might be a problem with the physical hardware. Verify the following in order:

  1. If the cable or its connector are loose or damaged, or you are unsure about the cable's type or quality, change it or test with a loopback jack.
  2. Verify that the LED lights for the network ports indicate a firm electrical contact when you plug network cables into the appliance. For LED indications, see your model's QuickStart Guide.
  3. If traffic ingresses and egresses, but performance is not what you expect, verify that the MTU matches other devices on your network.
  4. If the hardware connections are firm and the appliance is powered on, but you cannot connect — even using a local console connection to the CLI rather then a network connection — then there might be a problem during boot. Verify that the network interface is up (see Bringing up network interfaces). If the network interface is already up, or if you cannot connect to verify it, contact Fortinet Technical Support.

Bringing up network interfaces

If a network interface is disabled, all connections will fail—regardless of whether the cables are physically connected.

Note

If the network interface's Status column is a red "down" arrow, its administrative status is currently down and it will not receive or emit packets, even if you otherwise configure it. To bring up the network interface, change the Administrative Status setting. The column does not indicate the detected physical link status; it is the administrative status that indicates whether you permit network interface to receive and/or transmit packets.

For example, if the cable is physically unplugged, diagnose netlink interface list port1 may indicate that the link is down, even though you have administratively enabled it.

In the GUI, go to System > Network > Interface. If the status is down (a down arrow on red circle), click Bring Up next to it in the Status column to bring up the link.

Alternatively, you can enable an interface in the CLI:

config system interface

edit port2

set status up

end

Examining the ARP table

When connectivity cannot be established or is periodically interrupted, but hardware and link status is not an issue, the first place to look is at a slightly higher layer in network connections: the address resolution protocol (ARP) table. While most devices' MAC address is bound to the hardware at the manufacturer and not easily changed, some devices have configurable or virtual MACs. In this case, you should verify that there is no conflict which could cause the IP to resolve to a different network port whenever that other device is connected to your network.

Functioning ARP is especially important in high availability (HA) topologies. If changes in which MAC address resolves to which IP address are not correctly propagated through your network, failovers may not work.

To display the ARP table in the CLI, enter:

diagnose network arp list

Examining routing

If the MAC resolves correctly, but IP connectivity fails, try using ICMP to determine if the host is reachable, or to locate the point on your network at which connectivity fails. You can do this from the FortiRecorder appliance.

IP layer connectivity fails when routes are incorrectly configured. Static routes specify where to send traffic when it leaves the FortiRecorder appliance: you can specify through which network interface a packet will leave, and the IP address of a next-hop router that is reachable from that network interface. Routers are aware of which IP addresses are reachable through various network pathways, and can forward those packets along pathways capable of reaching the packets' ultimate destinations. Your FortiRecorder itself does not need to know the full route, as long as the routers can pass along the packet.

You must configure FortiRecorder with at least one static route that points to a router, often a router that is the gateway to the Internet. You may need to configure multiple static routes if you have multiple gateway routers (for example, each of which should receive packets destined for a different subset of IP addresses), redundant routers (for example, redundant Internet, WAN, or ISP links), or other special routing cases.

However, often you will only need to configure one route: a default route.

For example, if a web server is directly attached to one physical port on the FortiRecorder, but all other destinations, such as connecting clients, are located on distant networks, such as the Internet, you might need to add only one route: a default route that indicates the gateway router through which the FortiRecorder appliance can send traffic in the direction towards the Internet.

Tooltip

If your computer is not directly attached to one of the physical ports of the FortiRecorder appliance, you might also require a static route so that your computer can connect with the GUI and CLI.

To determine which route a packet will be subject to, FortiRecorder examines each packet's destination IP address and compares it to those of the static routes. It will forward the packet along to the route with the largest prefix match, automatically egressing from the network interface on that network. (Egress port for a route cannot be manually configured.) If multiple routes match the packet, the FortiRecorder appliance will apply the route with the smallest index number. For this reason, you should give more specific routes a smaller index number than the default route.

The ping command sends a small data packet to the destination and waits for a response. The response has a timer that may expire, indicating that the destination is unreachable via ICMP. ICMP is part of Layer 3 on the OSI Networking Model. ping sends Internet Control Message Protocol (ICMP) ECHO_REQUEST packets to the destination, and listens for ECHO_RESPONSE packets in reply. Beyond basic existence of a possible route between the source and destination, ping tells you the amount of packet loss (if any), how long it takes the packet to make the round trip (latency), and the variation in that time from packet to packet (jitter).

Similarly, traceroute sends ICMP packets to test each hop along the route. It sends three packets to the destination, and then increases the time to live (TTL) setting by one, and sends another three packets to the destination. As the TTL increases, packets go one hop farther along the route until they reach the destination.

Most traceroute commands display their maximum hop count — that is, the maximum number of steps it will take before declaring the destination unreachable — before they start tracing the route. The TTL setting may result in routers or firewalls along the route timing out due to high latency. If you specify the destination using a domain name, the traceroute output can also indicate DNS problems, such as an inability to connect to a DNS server.

By default, FortiRecorder appliances will respond to ping and traceroute. However, if FortiRecorder does not respond, and there are no firewall policies that block it, ICMP type 0 (ECHO_REPSPONSE or "pong") might be effectively disabled. By default, traceroute uses UDP with destination ports numbered from 33434 to 33534. The traceroute utility usually has an option to specify use of ICMP ECHO_REQUEST (type 8) instead, as used by the Windows tracert utility. If you have a firewall and you want traceroute to work regardless of your computer's OS (Apple macOS, Linux, and Microsoft Windows), then you must allow both protocols inbound through your firewall (UDP ports 33434 - 33534 and ICMP type 8).

Some networks block ICMP packets because they can be used in a ping flood or denial of service (DoS) attack if the network does not have anti-DoS capabilities, or because ping can be used by an attacker to find potential targets on the network.

To enable ping & traceroute responses from FortiRecorder

  1. Go to System > Network > Interface.

    To access this part of the GUI, you must have Read and Write permission in your administrator's account access profile to items in the Router Configuration category.

  2. In the row for the network interface which you want to respond to ICMP type 8 (ECHO_REQUEST) for ping and UDP for traceroute, click Edit.
  3. Enable Access: PING.

    Note

    Disabling PING only prevents FortiRecorder from receiving ICMP type 8 (ECHO_REQUEST) or type 30 and traceroute-related UDP.

    It does not disable FortiRecorder CLI commands such as execute ping or execute traceroute that send such traffic.

    Since you typically use these tools only during troubleshooting, you can allow ICMP, the protocol used by these tools, on interfaces only when you need them. Otherwise, disable ICMP for improved security and performance.

  4. Click OK.

    The appliance should now respond when another device such as your management computer sends a ping or traceroute to that network interface.

To verify routes between cameras and FortiRecorder

  1. Use the execute ping command on FortiRecorder with the camera's IP address to verify that a route exists between the two.

  2. If possible, temporarily connect a computer at the camera's usual physical location, using the camera's usual IP address, so that you can use its ping command to test traffic movement along the path in both directions: from the location of the camera (temporarily, the computer) to the FortiRecorder, and the FortiRecorder to the camera.

    If the routing test succeeds, continue with step 4.

    Note

    Connectivity via ICMP only proves that a route exists. It does not prove that connectivity also exists via other protocols at other layers such as HTTP.

    If ping shows some packet loss, investigate:

    • cabling to eliminate loose connections
    • ECMP, split horizon, or network loops
    • dynamic routing such as OSPF
    • all equipment between the ICMP source and destination to minimize hops

    If the routing test fails, and ping shows total packet loss:

    • verify cabling to eliminate loose connections
    • continue to the next step
    Tooltip

    Both ping and traceroute require that network nodes respond to ICMP. If you have disabled responses to ICMP on your network, hosts may appear to be unreachable to ping and traceroute, even if connections using other protocols can succeed.

    For example, you might use ping to determine that 172.16.1.10 is reachable:

    FortiRecorder-200D# execute ping 172.16.1.10

    PING 172.16.1.10 (172.16.1.10): 56 data bytes

    64 bytes from 172.16.1.10: icmp_seq=0 ttl=64 time=2.4 ms

    64 bytes from 172.16.1.10: icmp_seq=1 ttl=64 time=1.4 ms

    64 bytes from 172.16.1.10: icmp_seq=2 ttl=64 time=1.4 ms

    64 bytes from 172.16.1.10: icmp_seq=3 ttl=64 time=0.8 ms

    64 bytes from 172.16.1.10: icmp_seq=4 ttl=64 time=1.4 ms

    --- 172.20.120.167 ping statistics ---

    5 packets transmitted, 5 packets received, 0% packet loss

    round-trip min/avg/max = 0.8/1.4/2.4 ms

    or that 192.168.1.10 is not reachable:

    FortiRecorder-200D# execute ping 192.168.1.10

    PING 192.168.1.10 (192.168.1.10): 56 data bytes

    Timeout ...

    Timeout ...

    Timeout ...

    Timeout ...

    Timeout ...

    --- 192.168.1.10 ping statistics ---

    5 packets transmitted, 0 packets received, 100% packet loss

  3. Use the tracert or traceroute command on both the computer (which is temporarily substituting for the camera) and FortiRecorder to locate the point of failure along the route, the router hop or host at which the connection fails.

    For example, if it fails at the second hop, you might see:

    FortiRecorder-200D# execute traceroute 192.168.1.10

    traceroute to 192.168.1.10 (192.168.1.10), 32 hops max, 72 byte packets

    1 192.168.1.2 2 ms 0 ms 1 ms

    2 * * *

    Each line lists the routing hop number, the IP address and FQDN (if any) of that hop, and the 3 response times from that hop. Typically a value of <1 ms indicates a local router. The asterisks ( * ) indicate no response from that hop in the network routing.

    If the route is broken when it reaches the FortiRecorder, first examine its network interfaces and routes. To display network interface addresses and subnets, enter:

    FortiRecorder-200D# show system interface

    To display all recently-used routes (the routing table cache) with their priorities, enter:

    FortiRecorder-200D# diagnose netlink rtcache list

    Tooltip

    The index number of the route in the list of static routes in the GUI is not necessarily the same as its position in the cached routing table (diagnose netlink rtcache list).

    You may need to verify that there are no misconfigured DNS records or other problems at the physical, network, and transport layer.

    If these tests succeed and a route exists, but you cannot receive video feeds or use FortiRecorder to update the camera's network settings, an application-layer problem is preventing connectivity.

  4. For application-layer problems, on the FortiRecorder, examine the:

    • camera network settings (these may have become out-of-sync if you modified them while the camera was disabled)
    • certificates (if connecting via HTTPS)

    On routers and firewalls between the host and the FortiRecorder appliance, verify that they permit HTTP, HTTPS, and RTP connectivity between them.

    Also, if the computer's DNS query cannot resolve the host name, output similar to the following appears:

    example.com: Name or service not known

    Cannot handle "host" cmdline arg `example.com' on position 1 (argc 1)

Discovery fails

Discovery of devices by the FortiRecorder uses UPnP and ONVIF. For it to work, devices usually must be on the same IP subnet as the FortiRecorder, and must not be blocked by firewalls or other network filtering. If cameras are not on the same subnet, you may still be able to facilitate discovery traffic by configuring your FortiGate or other device with multicast forwarding.

If you do not know which device is impeding discovery, you can either:

  • Temporarily attach the cameras to a closer point on the network, such as a local switch or directly to the FortiRecorder, so that discovery is not blocked.
  • Manually add the camera to the FortiRecorder unit's list of known cameras, skipping discovery.

Multiple DHCP servers

The FortiRecorder appliance has a built-in DHCP server. By default, it is disabled.

If you enable it and your network has another DHCP server (for example, your ISP's cable modem, a router, or a Windows or Linux server), verify that:

  • both are not serving requests on the same network segment (which could create a race condition)
  • both are not using the same pool of IP addresses (which could lead to IP address conflicts — see Resolving IP address conflicts)

To verify that your appliance and cameras are sending and receiving lease requests, you can perform a packet trace (see Packet tracing) and/or use the event log to look for:

  • DHCPDISCOVER (destination IP address is broadcast, not specifically to FortiRecorder)
  • DHCPOFFER
  • DHCPREQUEST
  • DHCPACK

Unauthorized DHCP clients or DHCP pool exhaustion

Usually, returning DHCP clients will receive the same IP address lease. However if computers or other devices are accidentally using IP addresses that the FortiRecorder's built-in DHCP server should be allocating to cameras, and the pool of available DHCP IP addresses becomes exhausted, cameras may be unable to get or retain an IP address.

To determine which devices are using your pool of DHCP IP addresses, compare the MAC address of each device's network adapter to the list of current DHCP clients in Monitor > DHCP > DHCP or enter this command in the CLI:

execute dhcp lease-list

Output is like the following:

To correct this situation, first configure unintentional DHCP clients so that they do not use DHCP (that is, they have a static IP address) and so their IP address is not in the range used by the DHCP pool. Second, clear the list of DHCP clients to allow legitimate DHCP clients (your cameras) to obtain a lease:

execute dhcp clear-lease

New clients that were previously unable to get an IP address will obtain an IP address for the first time. Returning clients' s IP addresses may change as the built-in DHCP server no longer has any memory of their previous lease, and may assign them a new IP address if another client has claimed that IP address first. (This may result in temporary IP address conflicts and therefore connectivity interruptions while the DHCP server assigns new leases.)

Examining IP sessions

If a route exists, but there appears to be a problem establishing or maintaining TCP or IP-layer sessions between FortiRecorder and a computer or camera on your IP network, then there are multiple possible causes, such as:

You can view a snapshot of FortiRecorder's session table according to the IP layer. Go to FortiView > Sessions > Sessions.

GUI Item

Description

Protocol

The protocol of the session according to the "protocol" ID number field (or, for IPv6, "next header") in the IP header of the packets.

  • icmp — 1 (Due to the speed of ICMP messages, this is rarely seen in the session list.)
  • tcp — 6
  • udp — 17 (Due to the speed of UDP datagrams, this is rarely seen in the session list.)

From IP

The source of the session according the source field in the IP header. If source NAT is occurring, this is not necessarily the IP address in the original packet from the client.

From Port

The source port number.

For a list of port numbers that can originate from the FortiRecorder, see Appendix A: Port numbers.

To IP

The destination according to the destination field in the IP header. If destination NAT is occurring, this is not necessarily the IP address in the original packet from the client.

To Port

The destination port number.

For a list of port numbers that can be received by the FortiRecorder, see Appendix A: Port numbers.

Expire (seconds)

The session timeout in seconds. The expiry counter is reset when packets are sent or received, indicating that the session is still active.

To refresh the session list snapshot with the most current list, click the dotted circle (Refresh) icon to the left of Records per page.

To sort the session list based upon the contents of a column, hover your mouse cursor over the column's heading then click the arrow that appears on the right side of the heading, and select either Sort Ascending or Sort Descending.

If you expect sessions that do not exist, be aware that some protocol designs (notably UDP) do not have persistent sessions. Their sessions will almost immediately expire and be removed from the session list, and therefore it may be very difficult to get a session list snapshot during the short time that the datagram is being transmitted. TCP has persistent connections, where the socket is maintained until data transmission either is confirmed to be finished or times out. Therefore TCP connections persist in the session table for a much longer time.

If you still do not see the sessions that you expect, verify that your firewall or router allows traffic to or from those IP addresses, on all expected source and destination port numbers (see Appendix A: Port numbers).

If you notice unexpected sessions with the FortiRecorder GUI or CLI, verify that you have configured all accounts' Trusted hosts setting.

Resolving IP address conflicts

If two or more devices are configured to use the same IP address on your network, this will cause a problem called an IP address conflict. Only one of those identically addressed devices can have IP-layer connectivity at a given time. The other will be ignored, effectively causing it to behave as if it were disconnected. (If multiple devices were to use the same IP address, routers and switches would not be able to determine with certainty where to deliver a packet destined for that IP address. To prevent this, routers and switches will only let one of the devices use the IP.)

Typically IP conflicts are caused when either:

  • you have accidentally configured 2 devices with the same static IP address
  • you have accidentally configured a device with a static IP address that belongs to the DHCP pool
  • 2 DHCP servers accidentally have pools in the same range of IP addresses, and are each independently assigning their clients the same IP addresses

Your cameras, of course, have no screen, and cannot display any IP address conflict error message. However, you may notice symptoms such as interrupted video streams whenever a new device connects to the network or reboots.

If you have configured your FortiRecorder's built-in DHCP server, first verify that it is not using the same DHCP pool as another DHCP server on your network. Next, you can use the CLI to determine whether MAC addresses from other devices' network adapters have stolen IP addresses that should belong to your cameras. See Unauthorized DHCP clients or DHCP pool exhaustion. If, however, you have transitioned your cameras to use static IP addresses, you must use another method.

  • Use the ARP table of either your FortiRecorder (see Examining the ARP table) or router to determine which MAC address (and therefore which computer or device's network adapter) has taken the IP address.
  • If a computer is using the same IP address as another device, such as your cameras, it may periodically complain of an IP address conflict. This computer may be the source of the conflict.

Once you have found the source of the problem, configure that computer or device to use a unique IP address that is not used by any other device on your network.

Examining live video streams

Live video feeds are sensitive to packet loss, latency, and high bandwidth usage. This can cause video that is blurry or motion that is not smooth (low video frame rate). For details on how to diagnose these problems, see Packet tracing and Video performance.

Packet tracing

When troubleshooting networks, verifying hardware connections and routing with execute ping and execute traceroute CLI commands are often enough to diagnose the problem. For more rare problems, you can use traffic capture to see more low-level information.

Packet tracing is also known as packet sniffing, packet capture, packet analysis, or traffic capture. It can record information about all of the Ethernet frames that arrive—regardless of whether the destination MAC address matches the network interface.

FortiRecorder appliances have a built-in packet trace feature. By using it, you or Fortinet Support can diagnose problems that are otherwise difficult to detect. For example, it can show:

  • malformed packets (protocol is not RFC-compliant)
  • dropped packets
  • intermittent packet loss during ping
  • wrong network address translation (NAT)
  • wrong port forwarding
  • wrong session setup
  • ARP broadcast storms or floods
  • OSI Layer 2 bridge loops
  • SSL/TLS handshake errors
  • wrong source address when a device has multiple IP addresses, on multiple networks

For example, you can make a packet trace during a continuous ping. This shows ARP MAC address resolution, the source and destination IP addresses, the protocols and port number used, and whether FortiRecorder received a reply (the destination was reachable via that route) as expected.

Note

CPU usage and disk space usage is increased during a packet trace. On busy networks, this can be very resource intensive, and cause delays in live video streams. Performance is better if you packet trace only:

  • while required, and stop when finished
  • when the network is not busy, maybe at night
  • with a local console CLI connection, not network (Telnet/SSH/HTTP/HTTPS)

If you only need a simple packet capture, configure it via GUI, where it can be given a time limit.

Caution

Before using packet capture, think about filter criteria. Too much data is more difficult to analyze. It will be easier if you :

  • limit the time range, or number of packets
  • exclude IP addresses, port numbers, and protocols that are not relevant

For example, maybe you only need to trace communications with one device, or only RTP video packets.

More complex IP address, port number, and protocol filters can be configured via CLI.

Packet tracing via GUI

  1. Go to System > Network > Traffic Capture.

  2. Click New.

  3. Configure the following settings:

    Setting Name

    Description

    Description

    Enter a unique name for the packet trace file.

    Duration

    Enter a time range for the packet trace.

    Interface

    Select which network interface you want to trace traffic with, or All to trace traffic on all network interfaces.

    IP/Host

    Optional. If you want to trace traffic only to specific destination IP addresses, enter up to 3 domain names or IP addresses.

    Alternatively, configure Exclusion instead.

    Filter

    Select either:

    • None: Capture all port numbers and protocols.This can be required by protocols that use multiple port numbers, or to see a correlated sequential order of multiple protocols. For example, it is common to see DNS over UDP before HTTPS over TCP, and RTCP before RTP.
    • Use protocol: Only capture traffic on the port number you specify. Valid range of port numbers is from 1 to 65535.

    Exclusion

    Optional. If you don't want to trace traffic with specific IP addresses or port numbers, enter the domain names or IP addresses and port numbers to exclude.

    Alternatively, configure IP/Host instead.

  4. Click Create.

    Packet tracing begins immediately.

  5. When the packet trace is complete, either refresh the page in your web browser, or return to System > Network > Traffic Capture. (If the traffic capture is complete, its Status column will show Stopped.)

  6. Click to select the packet trace file, and then click Download.

  7. On your computer, open the PCAP file in compatible software such as Wireshark.

    Compare the captured traffic with the expected behavior.

Packet tracing via CLI

Instead of reading packet capture output directly in your CLI display, you usually should save the output to a plain text file using your CLI client. Saving the output provides several advantages. Packets can arrive more rapidly than you may be able to read them in the buffer of your CLI display, and many protocols transfer data using encodings other than US-ASCII. Therefore it is usually easier to analyze the output by saving it to a file, and then opening it in a network protocol analyzer application.

For example, you could use PuTTY to save the output to a file. Methods vary. See the documentation for your CLI client.

Requirements

Note

The fgt2eth.pl script is provided as-is, without any implied warranty or technical support.

To use packet capture with PuTTY and Wireshark

  1. On your computer, start PuTTY. Connect to the FortiRecorder appliance using a local console, SSH, or Telnet connection.
  2. Type the packet capture command, but do not press Enter yet.

    For example, to capture HTTPS and HTTP traffic with the source IP address 10.0.0.1, you can type:

    diagnose sniffer packet port1 'src host 10.0.0.1 and (tcp port 443 or tcp port 80)' 3

    See also diagnose sniffer CLI syntax.

  3. In the upper left corner of the window, click the PuTTY icon to open its drop-down menu, then click Change Settings.

    A dialog appears where you can configure PuTTY to save output to a plain text file.

  4. In the Category tree on the left, go to Session > Logging.
  5. In the Session logging area, select Printable output.
  6. In Log file name, click the Browse button, then choose where to save the packet capture file, such as C:\Users\MyAccount\Downloads\packet_capture.txt. (You do not need to save it with the .log file extension.)

  7. Click Apply.
  8. Press Enter on the CLI to start packet capture.

    If you did not specify a number of packets to capture, then when you have captured enough, press Ctrl + C.

  9. Close the PuTTY window.
  10. Start Notepad++. Open the packet capture file.
  11. Delete the first and last lines, which look like this:

    =~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2023.06.16 11:58:00 =~=~=~=~=~=~=~=~=~=~=~=

    ...

    FK400D3016000000 #

    These are a PuTTY timestamp and the command prompt, which are not part of the packet capture. If you do not delete them, they could interfere with the conversion script in the next step.

  12. Open a command prompt (cmd.exe). Enter the commands to to use fgt2eth.pl to convert plain text file to a PCAP format:

    cd C:\Users\MyAccount\Downloads\

    fgt2eth.pl -in packet_capture.txt -out packet_capture.pcap

    where:

    • packet_capture.txt is the name of the packet capture file from PuTTY
    • packet_capture.pcap is the name of the converted output file
  13. On your computer, open the PCAP file in compatible software such as Wireshark.

    Compare the captured traffic with the expected behavior.

diagnose sniffer CLI syntax

Packet capture syntax on FortiRecorder appliances is similar to that of FortiGate appliances. If you omit all parameters for the command, the command captures all packets on all network interfaces until you press Ctrl + C.

diagnose sniffer packet [{any | <interface_name>}

[{none | '<filter_str>'}

[{1 | 2 | 3 | 4 | 5 | 6}

[<packets_int>

[{a | <any_str>}]]]]]

where:

  • <interface_name> is the name of a network interface, such as port1. Alternatively, type any to match all network interfaces.
  • '<filter_str>' is the filter that specifies which protocols and port numbers that you want to capture, such as 'tcp port 80'.Filters use tcpdump syntax. If you do not want a filter, but you want to configure parameters after this one, type none.
  • <packets_int> is the number of packets to capture before stopping. If you do not specify this parameter, packet capture output is printed to your CLI display until you stop it by pressing Ctrl+C.
  • {a | <any_str>} is either a (to include an absolute, full UTC timestamp in the format yyyy-mm-dd hh:mm:ss.ms; use this format if you need to correlate a FortiRecorder packet capture with another device's packet capture), or any other text (to include a timestamp that is the amount of time since he start of the packet capture, in the format ss.ms)
  • {1 | 2 | 3 | 4 | 5 | 6} is an integer indicating which details to include about network interface names, Ethernet frames, IP packet headers, and/or packet payloads.

    • 1 — Include the packet capture timestamp, and basic fields of the IP header: source IP address, destination IP address, protocol name, and destination port number. Does not display all fields of the IP protocol header.

      Example:

      interfaces=[any]
      filters=[none]
      0.747195 172.20.131.143.6970 -> 172.20.131.43.33740: udp 1448
    • 2 — All of the output from 1, plus the packet payload in both hexadecimal and ASCII.

      Example:

      interfaces=[any]
      filters=[none]
      0.927132 172.25.188.213.60407 -> 172.20.131.43.22: ack 594494863
      000000   45 00 00 28 63 d2 40 00 7c 06 02 cf ac 19 bc d5        E..(c.@.|.......
      000010   ac 14 83 2b eb f7 00 16 39 9f 83 88 23 6f 45 8f        ...+....9...#oE.
      000020   50 10 17 ff ed 72 00 00 00 00 00 00 00 00              P....r........
    • 3 — All of the output from 2, plus the link frame (Ethernet) header.

      Example:

      interfaces=[any]
      filters=[none]
      0.131709 172.20.132.187.6970 -> 172.20.132.43.38262: udp 99
      000000   00 00 00 00 00 01 00 22 f4 ce 5e 04 08 00 45 00        ......."..^...E.
      000010   00 7f 62 15 00 00 40 11 b7 49 ac 14 84 bb ac 14        ..b...@..I......
      000020   84 2b 1b 3a 95 76 00 6b c6 48 80 e0 cf f4 59 ba        .+.:.v.k.H....Y.
      000030   bd 77 12 be 36 32 41 9b 98 01 89 8c 05 21 1e 23        .w..62A......!.#
      000040   35 c4 6b 11 8f 34 10 47 88 d8 38 8c d8 41 1e 23        5.k..4.G..8..A.#
      000050   60 d0 8e c4 78 8f 11 8e fa 23 70 27 e6 80 c2 11        `...x....#p'....
      000060   c3 7a 80 f1 9b 01 fb 36 9f 50 16 3c 9d dc 07 9e        .z.....6.P.<....
      000070   a0 08 48 fc 02 cc 27 80 57 0f c1 56 68 06 e8 47        ..H...'.W..Vh..G
      000080   01 1e 23 81 08 47 00 66 07 e0 04 20 80                 ..#..G.f.....
    • 4 — All of the output from 1, plus the network interface name. This can be necessary if you are capturing packets from multiple network interfaces at once, and need to know which packet was seen by which interface.

      Example:

      interfaces=[any]
      filters=[none]
      0.942892 port1 in 172.20.131.143.6970 -> 172.20.131.43.33740: udp 1448
    • 5 — All of the output from 2, plus the network interface name.

      Example:

      interfaces=[any]
      filters=[none]
      0.394962 port1 in 172.25.188.213.60407 -> 172.20.131.43.22: ack 594500767
      000000   45 00 00 28 66 a9 40 00 7c 06 ff f7 ac 19 bc d5        E..(f.@.|.......
      000010   ac 14 83 2b eb f7 00 16 39 9f 8d 98 23 6f 5c 9f        ...+....9...#o\.
      000020   50 10 18 00 cc 51 00 00 00 00 00 00 00 00              P....Q........
    • 6 — All of the output from 3, plus the network interface name.

      Example:

      interfaces=[any]
      filters=[none]
      0.790631 port1 in 172.25.188.213.60407 -> 172.20.131.43.22: ack 594502111
      000000   00 00 00 00 00 01 90 6c ac 99 ce 60 08 00 45 00        .......l...`..E.
      000010   00 28 67 10 40 00 7c 06 ff 90 ac 19 bc d5 ac 14        .(g.@.|.........
      000020   83 2b eb f7 00 16 39 9f 8f 78 23 6f 61 df 50 10        .+....9..x#oa.P.
      000030   18 00 c5 31 00 00 00 00 00 00 00 00                    ...1........
Example: HTTPS packet capture

In this example, you capture all TCP port 443 (typically HTTPS) traffic occurring through port1, regardless of its source or destination IP address. The capture uses verbosity level 3.

A specific number of packets to capture is not specified. As a result, the packet capture continues until the administrator presses Ctrl + C. The command confirms that five packets were seen by that network interface.

(Verbose output can be very long. As a result, output shown below is truncated after only one packet.)

FK400D3016000013 # diagnose sniffer packet port1 'tcp port 443' 3

System Time:  2023-06-16 16:07:51 EDT (Uptime: 4d 6h 48m)
interfaces=[port1]
filters=[tcp port 443]
4.795077 172.25.188.213.56340 -> 172.20.131.43.443: syn 457421310
000000   00 10 f3 37 6c e3 90 6c ac 99 ce 60 08 00 45 00        ...7l..l...`..E.
000010   00 34 69 6b 40 00 7c 06 fd 29 ac 19 bc d5 ac 14        .4ik@.|..)......
000020   83 2b dc 14 01 bb 1b 43 b1 fe 00 00 00 00 80 02        .+.....C........
000030   fd 80 2e bb 00 00 02 04 05 48 01 03 03 08 01 01        .........H......
000040   04 02                                                  ..

Connectivity issues

One of your first tests when configuring a new device should be to determine whether data such as video is received, and whether commands and schedules are being sent to it. You should also test whether notification email can be sent, and user accounts can log in to the GUI and view live video feeds.

After initial setup, connectivity should not be interrupted. FortiRecorder might sometimes be able to recover if, for example, a camera receives a new IP address from the DHCP server. However this can cause disruptions to recording, and camera log messages such as:

Camera 'c1' experienced an interruption that may result in a loss of recording.

If connections fail or perform erratically, examine the following in order:

  1. Checking hardware connections
  2. Bringing up network interfaces
  3. Examining the ARP table
  4. Examining routing
  5. Discovery fails
  6. Connectivity issues
  7. Unauthorized DHCP clients or DHCP pool exhaustion
  8. Examining IP sessions

  9. Resolving IP address conflicts
  10. Examining live video streams
  11. Packet tracing
Note

Troubleshooting is in order from more low-level OSI layers of your network (physical, link/Ethernet, network/IP) to the higher-level, more application-specific protocols. If your network is not new, then lower-level networking is usually already working, so you might want to start with later sections instead. Application-layer protocols such as RTSP are more often specific to FortiRecorder, cameras, and surveillance devices.

Checking hardware connections

If no traffic arrives on a network interface even though the configuration appears to be correct, or if network performance is less than you expect, then it might be a problem with the physical hardware. Verify the following in order:

  1. If the cable or its connector are loose or damaged, or you are unsure about the cable's type or quality, change it or test with a loopback jack.
  2. Verify that the LED lights for the network ports indicate a firm electrical contact when you plug network cables into the appliance. For LED indications, see your model's QuickStart Guide.
  3. If traffic ingresses and egresses, but performance is not what you expect, verify that the MTU matches other devices on your network.
  4. If the hardware connections are firm and the appliance is powered on, but you cannot connect — even using a local console connection to the CLI rather then a network connection — then there might be a problem during boot. Verify that the network interface is up (see Bringing up network interfaces). If the network interface is already up, or if you cannot connect to verify it, contact Fortinet Technical Support.

Bringing up network interfaces

If a network interface is disabled, all connections will fail—regardless of whether the cables are physically connected.

Note

If the network interface's Status column is a red "down" arrow, its administrative status is currently down and it will not receive or emit packets, even if you otherwise configure it. To bring up the network interface, change the Administrative Status setting. The column does not indicate the detected physical link status; it is the administrative status that indicates whether you permit network interface to receive and/or transmit packets.

For example, if the cable is physically unplugged, diagnose netlink interface list port1 may indicate that the link is down, even though you have administratively enabled it.

In the GUI, go to System > Network > Interface. If the status is down (a down arrow on red circle), click Bring Up next to it in the Status column to bring up the link.

Alternatively, you can enable an interface in the CLI:

config system interface

edit port2

set status up

end

Examining the ARP table

When connectivity cannot be established or is periodically interrupted, but hardware and link status is not an issue, the first place to look is at a slightly higher layer in network connections: the address resolution protocol (ARP) table. While most devices' MAC address is bound to the hardware at the manufacturer and not easily changed, some devices have configurable or virtual MACs. In this case, you should verify that there is no conflict which could cause the IP to resolve to a different network port whenever that other device is connected to your network.

Functioning ARP is especially important in high availability (HA) topologies. If changes in which MAC address resolves to which IP address are not correctly propagated through your network, failovers may not work.

To display the ARP table in the CLI, enter:

diagnose network arp list

Examining routing

If the MAC resolves correctly, but IP connectivity fails, try using ICMP to determine if the host is reachable, or to locate the point on your network at which connectivity fails. You can do this from the FortiRecorder appliance.

IP layer connectivity fails when routes are incorrectly configured. Static routes specify where to send traffic when it leaves the FortiRecorder appliance: you can specify through which network interface a packet will leave, and the IP address of a next-hop router that is reachable from that network interface. Routers are aware of which IP addresses are reachable through various network pathways, and can forward those packets along pathways capable of reaching the packets' ultimate destinations. Your FortiRecorder itself does not need to know the full route, as long as the routers can pass along the packet.

You must configure FortiRecorder with at least one static route that points to a router, often a router that is the gateway to the Internet. You may need to configure multiple static routes if you have multiple gateway routers (for example, each of which should receive packets destined for a different subset of IP addresses), redundant routers (for example, redundant Internet, WAN, or ISP links), or other special routing cases.

However, often you will only need to configure one route: a default route.

For example, if a web server is directly attached to one physical port on the FortiRecorder, but all other destinations, such as connecting clients, are located on distant networks, such as the Internet, you might need to add only one route: a default route that indicates the gateway router through which the FortiRecorder appliance can send traffic in the direction towards the Internet.

Tooltip

If your computer is not directly attached to one of the physical ports of the FortiRecorder appliance, you might also require a static route so that your computer can connect with the GUI and CLI.

To determine which route a packet will be subject to, FortiRecorder examines each packet's destination IP address and compares it to those of the static routes. It will forward the packet along to the route with the largest prefix match, automatically egressing from the network interface on that network. (Egress port for a route cannot be manually configured.) If multiple routes match the packet, the FortiRecorder appliance will apply the route with the smallest index number. For this reason, you should give more specific routes a smaller index number than the default route.

The ping command sends a small data packet to the destination and waits for a response. The response has a timer that may expire, indicating that the destination is unreachable via ICMP. ICMP is part of Layer 3 on the OSI Networking Model. ping sends Internet Control Message Protocol (ICMP) ECHO_REQUEST packets to the destination, and listens for ECHO_RESPONSE packets in reply. Beyond basic existence of a possible route between the source and destination, ping tells you the amount of packet loss (if any), how long it takes the packet to make the round trip (latency), and the variation in that time from packet to packet (jitter).

Similarly, traceroute sends ICMP packets to test each hop along the route. It sends three packets to the destination, and then increases the time to live (TTL) setting by one, and sends another three packets to the destination. As the TTL increases, packets go one hop farther along the route until they reach the destination.

Most traceroute commands display their maximum hop count — that is, the maximum number of steps it will take before declaring the destination unreachable — before they start tracing the route. The TTL setting may result in routers or firewalls along the route timing out due to high latency. If you specify the destination using a domain name, the traceroute output can also indicate DNS problems, such as an inability to connect to a DNS server.

By default, FortiRecorder appliances will respond to ping and traceroute. However, if FortiRecorder does not respond, and there are no firewall policies that block it, ICMP type 0 (ECHO_REPSPONSE or "pong") might be effectively disabled. By default, traceroute uses UDP with destination ports numbered from 33434 to 33534. The traceroute utility usually has an option to specify use of ICMP ECHO_REQUEST (type 8) instead, as used by the Windows tracert utility. If you have a firewall and you want traceroute to work regardless of your computer's OS (Apple macOS, Linux, and Microsoft Windows), then you must allow both protocols inbound through your firewall (UDP ports 33434 - 33534 and ICMP type 8).

Some networks block ICMP packets because they can be used in a ping flood or denial of service (DoS) attack if the network does not have anti-DoS capabilities, or because ping can be used by an attacker to find potential targets on the network.

To enable ping & traceroute responses from FortiRecorder

  1. Go to System > Network > Interface.

    To access this part of the GUI, you must have Read and Write permission in your administrator's account access profile to items in the Router Configuration category.

  2. In the row for the network interface which you want to respond to ICMP type 8 (ECHO_REQUEST) for ping and UDP for traceroute, click Edit.
  3. Enable Access: PING.

    Note

    Disabling PING only prevents FortiRecorder from receiving ICMP type 8 (ECHO_REQUEST) or type 30 and traceroute-related UDP.

    It does not disable FortiRecorder CLI commands such as execute ping or execute traceroute that send such traffic.

    Since you typically use these tools only during troubleshooting, you can allow ICMP, the protocol used by these tools, on interfaces only when you need them. Otherwise, disable ICMP for improved security and performance.

  4. Click OK.

    The appliance should now respond when another device such as your management computer sends a ping or traceroute to that network interface.

To verify routes between cameras and FortiRecorder

  1. Use the execute ping command on FortiRecorder with the camera's IP address to verify that a route exists between the two.

  2. If possible, temporarily connect a computer at the camera's usual physical location, using the camera's usual IP address, so that you can use its ping command to test traffic movement along the path in both directions: from the location of the camera (temporarily, the computer) to the FortiRecorder, and the FortiRecorder to the camera.

    If the routing test succeeds, continue with step 4.

    Note

    Connectivity via ICMP only proves that a route exists. It does not prove that connectivity also exists via other protocols at other layers such as HTTP.

    If ping shows some packet loss, investigate:

    • cabling to eliminate loose connections
    • ECMP, split horizon, or network loops
    • dynamic routing such as OSPF
    • all equipment between the ICMP source and destination to minimize hops

    If the routing test fails, and ping shows total packet loss:

    • verify cabling to eliminate loose connections
    • continue to the next step
    Tooltip

    Both ping and traceroute require that network nodes respond to ICMP. If you have disabled responses to ICMP on your network, hosts may appear to be unreachable to ping and traceroute, even if connections using other protocols can succeed.

    For example, you might use ping to determine that 172.16.1.10 is reachable:

    FortiRecorder-200D# execute ping 172.16.1.10

    PING 172.16.1.10 (172.16.1.10): 56 data bytes

    64 bytes from 172.16.1.10: icmp_seq=0 ttl=64 time=2.4 ms

    64 bytes from 172.16.1.10: icmp_seq=1 ttl=64 time=1.4 ms

    64 bytes from 172.16.1.10: icmp_seq=2 ttl=64 time=1.4 ms

    64 bytes from 172.16.1.10: icmp_seq=3 ttl=64 time=0.8 ms

    64 bytes from 172.16.1.10: icmp_seq=4 ttl=64 time=1.4 ms

    --- 172.20.120.167 ping statistics ---

    5 packets transmitted, 5 packets received, 0% packet loss

    round-trip min/avg/max = 0.8/1.4/2.4 ms

    or that 192.168.1.10 is not reachable:

    FortiRecorder-200D# execute ping 192.168.1.10

    PING 192.168.1.10 (192.168.1.10): 56 data bytes

    Timeout ...

    Timeout ...

    Timeout ...

    Timeout ...

    Timeout ...

    --- 192.168.1.10 ping statistics ---

    5 packets transmitted, 0 packets received, 100% packet loss

  3. Use the tracert or traceroute command on both the computer (which is temporarily substituting for the camera) and FortiRecorder to locate the point of failure along the route, the router hop or host at which the connection fails.

    For example, if it fails at the second hop, you might see:

    FortiRecorder-200D# execute traceroute 192.168.1.10

    traceroute to 192.168.1.10 (192.168.1.10), 32 hops max, 72 byte packets

    1 192.168.1.2 2 ms 0 ms 1 ms

    2 * * *

    Each line lists the routing hop number, the IP address and FQDN (if any) of that hop, and the 3 response times from that hop. Typically a value of <1 ms indicates a local router. The asterisks ( * ) indicate no response from that hop in the network routing.

    If the route is broken when it reaches the FortiRecorder, first examine its network interfaces and routes. To display network interface addresses and subnets, enter:

    FortiRecorder-200D# show system interface

    To display all recently-used routes (the routing table cache) with their priorities, enter:

    FortiRecorder-200D# diagnose netlink rtcache list

    Tooltip

    The index number of the route in the list of static routes in the GUI is not necessarily the same as its position in the cached routing table (diagnose netlink rtcache list).

    You may need to verify that there are no misconfigured DNS records or other problems at the physical, network, and transport layer.

    If these tests succeed and a route exists, but you cannot receive video feeds or use FortiRecorder to update the camera's network settings, an application-layer problem is preventing connectivity.

  4. For application-layer problems, on the FortiRecorder, examine the:

    • camera network settings (these may have become out-of-sync if you modified them while the camera was disabled)
    • certificates (if connecting via HTTPS)

    On routers and firewalls between the host and the FortiRecorder appliance, verify that they permit HTTP, HTTPS, and RTP connectivity between them.

    Also, if the computer's DNS query cannot resolve the host name, output similar to the following appears:

    example.com: Name or service not known

    Cannot handle "host" cmdline arg `example.com' on position 1 (argc 1)

Discovery fails

Discovery of devices by the FortiRecorder uses UPnP and ONVIF. For it to work, devices usually must be on the same IP subnet as the FortiRecorder, and must not be blocked by firewalls or other network filtering. If cameras are not on the same subnet, you may still be able to facilitate discovery traffic by configuring your FortiGate or other device with multicast forwarding.

If you do not know which device is impeding discovery, you can either:

  • Temporarily attach the cameras to a closer point on the network, such as a local switch or directly to the FortiRecorder, so that discovery is not blocked.
  • Manually add the camera to the FortiRecorder unit's list of known cameras, skipping discovery.

Multiple DHCP servers

The FortiRecorder appliance has a built-in DHCP server. By default, it is disabled.

If you enable it and your network has another DHCP server (for example, your ISP's cable modem, a router, or a Windows or Linux server), verify that:

  • both are not serving requests on the same network segment (which could create a race condition)
  • both are not using the same pool of IP addresses (which could lead to IP address conflicts — see Resolving IP address conflicts)

To verify that your appliance and cameras are sending and receiving lease requests, you can perform a packet trace (see Packet tracing) and/or use the event log to look for:

  • DHCPDISCOVER (destination IP address is broadcast, not specifically to FortiRecorder)
  • DHCPOFFER
  • DHCPREQUEST
  • DHCPACK

Unauthorized DHCP clients or DHCP pool exhaustion

Usually, returning DHCP clients will receive the same IP address lease. However if computers or other devices are accidentally using IP addresses that the FortiRecorder's built-in DHCP server should be allocating to cameras, and the pool of available DHCP IP addresses becomes exhausted, cameras may be unable to get or retain an IP address.

To determine which devices are using your pool of DHCP IP addresses, compare the MAC address of each device's network adapter to the list of current DHCP clients in Monitor > DHCP > DHCP or enter this command in the CLI:

execute dhcp lease-list

Output is like the following:

To correct this situation, first configure unintentional DHCP clients so that they do not use DHCP (that is, they have a static IP address) and so their IP address is not in the range used by the DHCP pool. Second, clear the list of DHCP clients to allow legitimate DHCP clients (your cameras) to obtain a lease:

execute dhcp clear-lease

New clients that were previously unable to get an IP address will obtain an IP address for the first time. Returning clients' s IP addresses may change as the built-in DHCP server no longer has any memory of their previous lease, and may assign them a new IP address if another client has claimed that IP address first. (This may result in temporary IP address conflicts and therefore connectivity interruptions while the DHCP server assigns new leases.)

Examining IP sessions

If a route exists, but there appears to be a problem establishing or maintaining TCP or IP-layer sessions between FortiRecorder and a computer or camera on your IP network, then there are multiple possible causes, such as:

You can view a snapshot of FortiRecorder's session table according to the IP layer. Go to FortiView > Sessions > Sessions.

GUI Item

Description

Protocol

The protocol of the session according to the "protocol" ID number field (or, for IPv6, "next header") in the IP header of the packets.

  • icmp — 1 (Due to the speed of ICMP messages, this is rarely seen in the session list.)
  • tcp — 6
  • udp — 17 (Due to the speed of UDP datagrams, this is rarely seen in the session list.)

From IP

The source of the session according the source field in the IP header. If source NAT is occurring, this is not necessarily the IP address in the original packet from the client.

From Port

The source port number.

For a list of port numbers that can originate from the FortiRecorder, see Appendix A: Port numbers.

To IP

The destination according to the destination field in the IP header. If destination NAT is occurring, this is not necessarily the IP address in the original packet from the client.

To Port

The destination port number.

For a list of port numbers that can be received by the FortiRecorder, see Appendix A: Port numbers.

Expire (seconds)

The session timeout in seconds. The expiry counter is reset when packets are sent or received, indicating that the session is still active.

To refresh the session list snapshot with the most current list, click the dotted circle (Refresh) icon to the left of Records per page.

To sort the session list based upon the contents of a column, hover your mouse cursor over the column's heading then click the arrow that appears on the right side of the heading, and select either Sort Ascending or Sort Descending.

If you expect sessions that do not exist, be aware that some protocol designs (notably UDP) do not have persistent sessions. Their sessions will almost immediately expire and be removed from the session list, and therefore it may be very difficult to get a session list snapshot during the short time that the datagram is being transmitted. TCP has persistent connections, where the socket is maintained until data transmission either is confirmed to be finished or times out. Therefore TCP connections persist in the session table for a much longer time.

If you still do not see the sessions that you expect, verify that your firewall or router allows traffic to or from those IP addresses, on all expected source and destination port numbers (see Appendix A: Port numbers).

If you notice unexpected sessions with the FortiRecorder GUI or CLI, verify that you have configured all accounts' Trusted hosts setting.

Resolving IP address conflicts

If two or more devices are configured to use the same IP address on your network, this will cause a problem called an IP address conflict. Only one of those identically addressed devices can have IP-layer connectivity at a given time. The other will be ignored, effectively causing it to behave as if it were disconnected. (If multiple devices were to use the same IP address, routers and switches would not be able to determine with certainty where to deliver a packet destined for that IP address. To prevent this, routers and switches will only let one of the devices use the IP.)

Typically IP conflicts are caused when either:

  • you have accidentally configured 2 devices with the same static IP address
  • you have accidentally configured a device with a static IP address that belongs to the DHCP pool
  • 2 DHCP servers accidentally have pools in the same range of IP addresses, and are each independently assigning their clients the same IP addresses

Your cameras, of course, have no screen, and cannot display any IP address conflict error message. However, you may notice symptoms such as interrupted video streams whenever a new device connects to the network or reboots.

If you have configured your FortiRecorder's built-in DHCP server, first verify that it is not using the same DHCP pool as another DHCP server on your network. Next, you can use the CLI to determine whether MAC addresses from other devices' network adapters have stolen IP addresses that should belong to your cameras. See Unauthorized DHCP clients or DHCP pool exhaustion. If, however, you have transitioned your cameras to use static IP addresses, you must use another method.

  • Use the ARP table of either your FortiRecorder (see Examining the ARP table) or router to determine which MAC address (and therefore which computer or device's network adapter) has taken the IP address.
  • If a computer is using the same IP address as another device, such as your cameras, it may periodically complain of an IP address conflict. This computer may be the source of the conflict.

Once you have found the source of the problem, configure that computer or device to use a unique IP address that is not used by any other device on your network.

Examining live video streams

Live video feeds are sensitive to packet loss, latency, and high bandwidth usage. This can cause video that is blurry or motion that is not smooth (low video frame rate). For details on how to diagnose these problems, see Packet tracing and Video performance.

Packet tracing

When troubleshooting networks, verifying hardware connections and routing with execute ping and execute traceroute CLI commands are often enough to diagnose the problem. For more rare problems, you can use traffic capture to see more low-level information.

Packet tracing is also known as packet sniffing, packet capture, packet analysis, or traffic capture. It can record information about all of the Ethernet frames that arrive—regardless of whether the destination MAC address matches the network interface.

FortiRecorder appliances have a built-in packet trace feature. By using it, you or Fortinet Support can diagnose problems that are otherwise difficult to detect. For example, it can show:

  • malformed packets (protocol is not RFC-compliant)
  • dropped packets
  • intermittent packet loss during ping
  • wrong network address translation (NAT)
  • wrong port forwarding
  • wrong session setup
  • ARP broadcast storms or floods
  • OSI Layer 2 bridge loops
  • SSL/TLS handshake errors
  • wrong source address when a device has multiple IP addresses, on multiple networks

For example, you can make a packet trace during a continuous ping. This shows ARP MAC address resolution, the source and destination IP addresses, the protocols and port number used, and whether FortiRecorder received a reply (the destination was reachable via that route) as expected.

Note

CPU usage and disk space usage is increased during a packet trace. On busy networks, this can be very resource intensive, and cause delays in live video streams. Performance is better if you packet trace only:

  • while required, and stop when finished
  • when the network is not busy, maybe at night
  • with a local console CLI connection, not network (Telnet/SSH/HTTP/HTTPS)

If you only need a simple packet capture, configure it via GUI, where it can be given a time limit.

Caution

Before using packet capture, think about filter criteria. Too much data is more difficult to analyze. It will be easier if you :

  • limit the time range, or number of packets
  • exclude IP addresses, port numbers, and protocols that are not relevant

For example, maybe you only need to trace communications with one device, or only RTP video packets.

More complex IP address, port number, and protocol filters can be configured via CLI.

Packet tracing via GUI

  1. Go to System > Network > Traffic Capture.

  2. Click New.

  3. Configure the following settings:

    Setting Name

    Description

    Description

    Enter a unique name for the packet trace file.

    Duration

    Enter a time range for the packet trace.

    Interface

    Select which network interface you want to trace traffic with, or All to trace traffic on all network interfaces.

    IP/Host

    Optional. If you want to trace traffic only to specific destination IP addresses, enter up to 3 domain names or IP addresses.

    Alternatively, configure Exclusion instead.

    Filter

    Select either:

    • None: Capture all port numbers and protocols.This can be required by protocols that use multiple port numbers, or to see a correlated sequential order of multiple protocols. For example, it is common to see DNS over UDP before HTTPS over TCP, and RTCP before RTP.
    • Use protocol: Only capture traffic on the port number you specify. Valid range of port numbers is from 1 to 65535.

    Exclusion

    Optional. If you don't want to trace traffic with specific IP addresses or port numbers, enter the domain names or IP addresses and port numbers to exclude.

    Alternatively, configure IP/Host instead.

  4. Click Create.

    Packet tracing begins immediately.

  5. When the packet trace is complete, either refresh the page in your web browser, or return to System > Network > Traffic Capture. (If the traffic capture is complete, its Status column will show Stopped.)

  6. Click to select the packet trace file, and then click Download.

  7. On your computer, open the PCAP file in compatible software such as Wireshark.

    Compare the captured traffic with the expected behavior.

Packet tracing via CLI

Instead of reading packet capture output directly in your CLI display, you usually should save the output to a plain text file using your CLI client. Saving the output provides several advantages. Packets can arrive more rapidly than you may be able to read them in the buffer of your CLI display, and many protocols transfer data using encodings other than US-ASCII. Therefore it is usually easier to analyze the output by saving it to a file, and then opening it in a network protocol analyzer application.

For example, you could use PuTTY to save the output to a file. Methods vary. See the documentation for your CLI client.

Requirements

Note

The fgt2eth.pl script is provided as-is, without any implied warranty or technical support.

To use packet capture with PuTTY and Wireshark

  1. On your computer, start PuTTY. Connect to the FortiRecorder appliance using a local console, SSH, or Telnet connection.
  2. Type the packet capture command, but do not press Enter yet.

    For example, to capture HTTPS and HTTP traffic with the source IP address 10.0.0.1, you can type:

    diagnose sniffer packet port1 'src host 10.0.0.1 and (tcp port 443 or tcp port 80)' 3

    See also diagnose sniffer CLI syntax.

  3. In the upper left corner of the window, click the PuTTY icon to open its drop-down menu, then click Change Settings.

    A dialog appears where you can configure PuTTY to save output to a plain text file.

  4. In the Category tree on the left, go to Session > Logging.
  5. In the Session logging area, select Printable output.
  6. In Log file name, click the Browse button, then choose where to save the packet capture file, such as C:\Users\MyAccount\Downloads\packet_capture.txt. (You do not need to save it with the .log file extension.)

  7. Click Apply.
  8. Press Enter on the CLI to start packet capture.

    If you did not specify a number of packets to capture, then when you have captured enough, press Ctrl + C.

  9. Close the PuTTY window.
  10. Start Notepad++. Open the packet capture file.
  11. Delete the first and last lines, which look like this:

    =~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2023.06.16 11:58:00 =~=~=~=~=~=~=~=~=~=~=~=

    ...

    FK400D3016000000 #

    These are a PuTTY timestamp and the command prompt, which are not part of the packet capture. If you do not delete them, they could interfere with the conversion script in the next step.

  12. Open a command prompt (cmd.exe). Enter the commands to to use fgt2eth.pl to convert plain text file to a PCAP format:

    cd C:\Users\MyAccount\Downloads\

    fgt2eth.pl -in packet_capture.txt -out packet_capture.pcap

    where:

    • packet_capture.txt is the name of the packet capture file from PuTTY
    • packet_capture.pcap is the name of the converted output file
  13. On your computer, open the PCAP file in compatible software such as Wireshark.

    Compare the captured traffic with the expected behavior.

diagnose sniffer CLI syntax

Packet capture syntax on FortiRecorder appliances is similar to that of FortiGate appliances. If you omit all parameters for the command, the command captures all packets on all network interfaces until you press Ctrl + C.

diagnose sniffer packet [{any | <interface_name>}

[{none | '<filter_str>'}

[{1 | 2 | 3 | 4 | 5 | 6}

[<packets_int>

[{a | <any_str>}]]]]]

where:

  • <interface_name> is the name of a network interface, such as port1. Alternatively, type any to match all network interfaces.
  • '<filter_str>' is the filter that specifies which protocols and port numbers that you want to capture, such as 'tcp port 80'.Filters use tcpdump syntax. If you do not want a filter, but you want to configure parameters after this one, type none.
  • <packets_int> is the number of packets to capture before stopping. If you do not specify this parameter, packet capture output is printed to your CLI display until you stop it by pressing Ctrl+C.
  • {a | <any_str>} is either a (to include an absolute, full UTC timestamp in the format yyyy-mm-dd hh:mm:ss.ms; use this format if you need to correlate a FortiRecorder packet capture with another device's packet capture), or any other text (to include a timestamp that is the amount of time since he start of the packet capture, in the format ss.ms)
  • {1 | 2 | 3 | 4 | 5 | 6} is an integer indicating which details to include about network interface names, Ethernet frames, IP packet headers, and/or packet payloads.

    • 1 — Include the packet capture timestamp, and basic fields of the IP header: source IP address, destination IP address, protocol name, and destination port number. Does not display all fields of the IP protocol header.

      Example:

      interfaces=[any]
      filters=[none]
      0.747195 172.20.131.143.6970 -> 172.20.131.43.33740: udp 1448
    • 2 — All of the output from 1, plus the packet payload in both hexadecimal and ASCII.

      Example:

      interfaces=[any]
      filters=[none]
      0.927132 172.25.188.213.60407 -> 172.20.131.43.22: ack 594494863
      000000   45 00 00 28 63 d2 40 00 7c 06 02 cf ac 19 bc d5        E..(c.@.|.......
      000010   ac 14 83 2b eb f7 00 16 39 9f 83 88 23 6f 45 8f        ...+....9...#oE.
      000020   50 10 17 ff ed 72 00 00 00 00 00 00 00 00              P....r........
    • 3 — All of the output from 2, plus the link frame (Ethernet) header.

      Example:

      interfaces=[any]
      filters=[none]
      0.131709 172.20.132.187.6970 -> 172.20.132.43.38262: udp 99
      000000   00 00 00 00 00 01 00 22 f4 ce 5e 04 08 00 45 00        ......."..^...E.
      000010   00 7f 62 15 00 00 40 11 b7 49 ac 14 84 bb ac 14        ..b...@..I......
      000020   84 2b 1b 3a 95 76 00 6b c6 48 80 e0 cf f4 59 ba        .+.:.v.k.H....Y.
      000030   bd 77 12 be 36 32 41 9b 98 01 89 8c 05 21 1e 23        .w..62A......!.#
      000040   35 c4 6b 11 8f 34 10 47 88 d8 38 8c d8 41 1e 23        5.k..4.G..8..A.#
      000050   60 d0 8e c4 78 8f 11 8e fa 23 70 27 e6 80 c2 11        `...x....#p'....
      000060   c3 7a 80 f1 9b 01 fb 36 9f 50 16 3c 9d dc 07 9e        .z.....6.P.<....
      000070   a0 08 48 fc 02 cc 27 80 57 0f c1 56 68 06 e8 47        ..H...'.W..Vh..G
      000080   01 1e 23 81 08 47 00 66 07 e0 04 20 80                 ..#..G.f.....
    • 4 — All of the output from 1, plus the network interface name. This can be necessary if you are capturing packets from multiple network interfaces at once, and need to know which packet was seen by which interface.

      Example:

      interfaces=[any]
      filters=[none]
      0.942892 port1 in 172.20.131.143.6970 -> 172.20.131.43.33740: udp 1448
    • 5 — All of the output from 2, plus the network interface name.

      Example:

      interfaces=[any]
      filters=[none]
      0.394962 port1 in 172.25.188.213.60407 -> 172.20.131.43.22: ack 594500767
      000000   45 00 00 28 66 a9 40 00 7c 06 ff f7 ac 19 bc d5        E..(f.@.|.......
      000010   ac 14 83 2b eb f7 00 16 39 9f 8d 98 23 6f 5c 9f        ...+....9...#o\.
      000020   50 10 18 00 cc 51 00 00 00 00 00 00 00 00              P....Q........
    • 6 — All of the output from 3, plus the network interface name.

      Example:

      interfaces=[any]
      filters=[none]
      0.790631 port1 in 172.25.188.213.60407 -> 172.20.131.43.22: ack 594502111
      000000   00 00 00 00 00 01 90 6c ac 99 ce 60 08 00 45 00        .......l...`..E.
      000010   00 28 67 10 40 00 7c 06 ff 90 ac 19 bc d5 ac 14        .(g.@.|.........
      000020   83 2b eb f7 00 16 39 9f 8f 78 23 6f 61 df 50 10        .+....9..x#oa.P.
      000030   18 00 c5 31 00 00 00 00 00 00 00 00                    ...1........
Example: HTTPS packet capture

In this example, you capture all TCP port 443 (typically HTTPS) traffic occurring through port1, regardless of its source or destination IP address. The capture uses verbosity level 3.

A specific number of packets to capture is not specified. As a result, the packet capture continues until the administrator presses Ctrl + C. The command confirms that five packets were seen by that network interface.

(Verbose output can be very long. As a result, output shown below is truncated after only one packet.)

FK400D3016000013 # diagnose sniffer packet port1 'tcp port 443' 3

System Time:  2023-06-16 16:07:51 EDT (Uptime: 4d 6h 48m)
interfaces=[port1]
filters=[tcp port 443]
4.795077 172.25.188.213.56340 -> 172.20.131.43.443: syn 457421310
000000   00 10 f3 37 6c e3 90 6c ac 99 ce 60 08 00 45 00        ...7l..l...`..E.
000010   00 34 69 6b 40 00 7c 06 fd 29 ac 19 bc d5 ac 14        .4ik@.|..)......
000020   83 2b dc 14 01 bb 1b 43 b1 fe 00 00 00 00 80 02        .+.....C........
000030   fd 80 2e bb 00 00 02 04 05 48 01 03 03 08 01 01        .........H......
000040   04 02                                                  ..