FortiOS diagnostics
FortiOS has a collection of diagnostic commands that you can use to troubleshoot and monitor the performance of your network. The get
and diagnose
CLI commands are the two main groups of diagnostic commands. Both commands display information about system resources, connections, and settings that allow you to locate problems or monitor system performance.
This topic includes diagnostics commands to help with:
- Date and time
- Resource usage
- Proxy operation
- Hardware NIC
- Traffic trace
- Session table
- Firewall session setup rate
- Finding object dependencies
- Flow trace
- Packet sniffing and packet capture
- NPU-based interfaces
- Debug command
- The execute tac report command
- Other commands
Date and time
The system date and time are important for FortiGuard services, logging events, and sending alerts. The wrong time makes the log entries confusing and difficult to use.
Use Network Time Protocol (NTP) to set the date and time, if possible. This is an automatic method that doesn't require manual intervention. However, you must ensure that the port is allowed through the firewalls on your network. FortiToken synchronization requires NTP in many situations.
How to set the date and time - GUI
- Go to the System Information widget on the Dashboard. The date and time are displayed next to System Time.
- To adjust the date and time settings, go to System > Settings.
- In System Time, you can set the time zone, date and time, and select NTP usage.
How to check the date and time - CLI
You can check the date and time using the CLI commands execute date
and execute time
.
config system global
set timezone <integer>
end
config system ntp
set type custom
config ntpserver
edit 1
set server “ntp1.fortinet.net”
next
edit 2
set server “ntp2.fortinet.net”
next
end
set ntpsync enable
set syncinterval 60
end
Use the set timezone ?
command to display a list of timezones and the integers that represent them.
Resource usage
Each program that runs on a computer has one or more processes associated with it. For example, if you open a Telnet program, it has an associated Telnet process. The same is true in FortiOS. All processes share the system resources in FortiOS, including memory and CPU.
Use the get system performance status
command to show the FortiOS performance status.
Sample output:
FGT# get system performance status
CPU states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU0 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU1 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU2 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU3 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
Memory: 4050332k total, 527148k used (13%), 3381312k free (83%), 141872k freeable (3%)
Average network usage: 41 / 28 kbps in 1 minute, 54 / 44 kbps in 10 minutes, 42 / 34 kbps in 30 minutes
Average sessions: 33 sessions in 1 minute, 48 sessions in 10 minutes, 38 sessions in 30 minutes
Average session setup rate: 0 sessions per second in last 1 minute, 0 sessions per second in last 10 minutes, 0 sessions per second in last 30 minutes
Virus caught: 0 total in 1 minute
IPS attacks blocked: 0 total in 1 minute
Uptime: 0 days, 22 hours, 59 minutes
Monitor the CPU and memory usage of internal processes, using the following command:
get system performance top <delay> <max_lines>
The data that the command lists includes the name of the daemon, the process ID, whether the process is sleeping or running, the CPU use percentage, and the memory use percentage.
Sample output:
get system performance top 10 100
Run Time: 0 days, 23 hours and 4 minutes
0U, 0N, 0S, 100I, 0WA, 0HI, 0SI, 0ST; 3955T, 3298F
httpsd 212 S 0.4 0.6
forticron 169 S 0.4 0.4
newcli 4054 R 0.4 0.2
reportd 174 S 0.0 1.4
pyfcgid 325 S 0.0 0.8
cmdbsvr 141 S 0.0 0.7
miglogd 160 S 0.0 0.6
httpsd 211 S 0.0 0.6
src-vis 180 S 0.0 0.6
pyfcgid 327 S 0.0 0.6
pyfcgid 328 S 0.0 0.6
pyfcgid 329 S 0.0 0.6
httpsd 162 S 0.0 0.5
cw_acd 189 S 0.0 0.5
httpsd 3998 S 0.0 0.5
httpsd 4050 S 0.0 0.5
updated 176 S 0.0 0.5
httpsd 4052 S 0.0 0.4
miglogd 203 S 0.0 0.4
miglogd 204 S 0.0 0.4
Proxy operation
Monitor proxy operations, using the following command:
diagnose test application <application> <option>
o display a list of available <application>
values, enter:
diagnose test application ?
The <option>
value depends on the application value that you use in the command. To display a list of available <option>
values, enter:
diagnose test application <application> ?
For example, if the application is http
, the CLI command that displays the <option>
values is:
diagnose test application http ?
Hardware NIC
To monitor hardware network operations, use the following command:
diagnose hardware deviceinfo nic <interface>
The information that this command shows is important because errors at the interface indicate data link or physical layer issues which may impact the performance of the FortiGate.
The following example shows a sample output when you set <interface>
to lan
:
System_Device_Name lan
Current_HWaddr 00:09:0f:68:35:60
Permanent_HWaddr 00:09:0f:68:35:60
State up
Link up
Speed 100
Duplex full
[……]
Rx_Packets=5685708
Tx_Packets=4107073
Rx_Bytes=617908014
Tx_Bytes=1269751248
Rx_Errors=0
Tx_Errors=0
Rx_Dropped=0
Tx_Dropped=0
[……]
The diagnose hardware deviceinfo nic
command displays a list of error names and values that are related to hardware. The following table describes possible hardware errors:
Field |
Description |
---|---|
Rx_Errors = rx error count |
Bad frame was marked as error by PHY |
Rx_CRC_Errors + Rx_Length_Errors - Rx_Align_Errors |
This error is only valid in 10/100M mode |
Rx_Dropped or Rx_No_Buffer_Count |
Running out of buffer space |
Rx_Missed_Errors |
Equals Rx_FIFO_Errors + CEXTERR (Carrier Extension Error Count); only valid in 1000M mode, which is marked by PHY |
Tx_Errors = Tx_Aborted_Errors |
ECOL (Excessive Collisions Count); only valid in half-duplex mode |
Tx_Window_Errors |
Late Collisions (LATECOL) Count Late collisions are collisions that occur after 64-byte time into the transmission of the packet while working in 10 to 100 Mb/s data rate and 512-byte time into the transmission of the packet while working in the 1,000 Mb/s data rate. This register only increments if transmits are enabled and the device is in half-duplex mode. |
Rx_Dropped |
See Rx_Errors |
Tx_Dropped |
Not defined |
Collisions |
Total number of collisions experienced by the transmitter; valid in half-duplex mode |
Rx_Length_Errors |
Transmission length error |
Rx_Over_Errors |
Not defined |
Rx_CRC_Errors |
Frame CRC error |
Rx_Frame_Errors |
Same as Rx_Align_Errors This error is only valid in 10/100M mode. |
Rx_FIFO_Errors |
Same as Rx_Missed_Errors - a missed packet count |
Tx_Aborted_Errors |
See Tx_Errors |
Tx_Carrier_Errors |
The PHY should assert the internal carrier sense signal during every transmission. Failure to do so may indicate that the link has failed or the PHY has an incorrect link configuration. This register only increments if transmits are enabled. This register isn't valid in internal SerDes 1 mode (TBI mode for the 82544GC/EI) and is valid only when the Ethernet controller is operating at full duplex. |
Tx_FIFO_Errors |
Not defined |
Tx_Heartbeat_Errors |
Not defined |
Tx_Window_Errors |
See LATECOL |
Tx_Single_Collision_Frames |
Counts the number of times that a successfully transmitted packet encountered a single collision The value increments only if transmits are enabled and the Ethernet controller is in half-duplex mode. |
Tx_Multiple_Collision_Frames |
A Multiple Collision Count which indicates the number of times that a transmit encountered more than one collision, but less than 16. The value increments only if transmits are enabled and the Ethernet controller is in half-duplex mode. |
Tx_Deferred |
Counts defer events A deferred event occurs when the transmitter can't immediately send a packet due to the medium being busy because another device is transmitting, the IPG timer hasn't expired, half-duplex deferral events are occurring, XOFF frames are being received, or the link isn't up. This register only increments if transmits are enabled. This counter doesn't increment for streaming transmits that are deferred due to TX IPG. |
Rx_Frame_Too_Longs |
The Rx frame is oversized |
Rx_Frame_Too_Shorts |
The Rx frame is too short |
Rx_Align_Errors |
This error is only valid in 10/100M mode |
Symbol Error Count |
Counts the number of symbol errors between reads - SYMERRS. The count increases for every bad symbol that's received, whether or not a packet is currently being received and whether or not the link is up. This register increments only in internal SerDes mode. |
Traffic trace
Traffic tracing allows you to follow a specific packet stream. This is useful to confirm that packets are taking the route you expect them to take on your network.
View the characteristics of a traffic session though specific security policies using:
diagnose sys session
Trace per-packet operations for flow tracing using:
diagnose debug flow
Trace per-Ethernet frame using:
diagnose sniffer packet
Trace a route from a FortiGate to a destination IP address using:
# execute traceroute www.fortinet.com
traceroute to www.fortinet.com (66.171.121.34), 32 hops max, 84 byte packets
1 172.20.120.2 0.637 ms 0.653 ms 0.279 ms
2 209.87.254.221 <static-209-87-254-221.storm.ca> 2.448 ms 2.519 ms 2.458 ms
3 209.87.239.129 <core-2-g0-2.storm.ca> 2.917 ms 2.828 ms 9.324 ms
4 209.87.239.199 <core-3-bdi1739.storm.ca> 13.248 ms 12.401 ms 13.009 ms
5 216.66.41.113 <v502.core1.tor1.he.net> 17.181 ms 12.422 ms 12.268 ms
6 184.105.80.9 <100ge1-2.core1.nyc4.he.net> 21.355 ms 21.518 ms 21.597 ms
7 198.32.118.41 <ny-paix-gni.twgate.net> 83.297 ms 84.416 ms 83.782 ms
8 203.160.228.217 <217-228-160-203.TWGATE-IP.twgate.net> 82.579 ms 82.187 ms 82.066 ms
9 203.160.228.229 <229-228-160-203.TWGATE-IP.twgate.net> 82.055 ms 82.455 ms 81.808 ms
10 203.78.181.2 82.262 ms 81.572 ms 82.015 ms
11 203.78.186.70 83.283 ms 83.243 ms 83.293 ms
12 66.171.127.177 84.030 ms 84.229 ms 83.550 ms
13 66.171.121.34 <www.fortinet.com> 84.023 ms 83.903 ms 84.032 ms
14 66.171.121.34 <www.fortinet.com> 83.874 ms 84.084 ms 83.810 ms
Session table
A session is a communication channel between two devices or applications across the network. Sessions allow FortiOS to inspect and act on a sequential group of packets in a session all together instead of inspecting each packet individually. Each of these sessions has an entry in the session table that includes important information about the session.
Use as a tool
Session tables are useful troubleshooting tools because they allow you to verify open connections. For example, if you have a web browser open to browse the Fortinet website, you would expect a session entry from your computer, on port 80, to the IP address for the Fortinet website. Another troubleshooting method is if there are too many sessions for FortiOS to process, you can examine the session table for evidence why this is happening.
You can view the FortiGate session table from either the FortiGate GUI or CLI. The most useful troubleshooting data comes from the CLI. The session table in the GUI also provides useful summary information, particularly the current policy number that the session is using.
GUI session information
You can view session information by going to the FortiView page. Read more about FortiView consoles in the Handbook's FortiView chapter.
How to find which security policy a specific connection is using
Every program and device on your network must have a communication channel, or session, open to pass information. The FortiGate manages these sessions with features such as traffic shaping, antivirus scanning, and blocking known bad web sites. Each session has an entry in the session table.
You may want to find information for a specific session for troubleshooting. For example, if a secure web browser session isn't working properly, you can check the session table to ensure the session is still active and going to the proper address. The session table can also tell you the security policy number it matches, so you can check what's happening in that policy.
-
Know your connection information.
You need to be able to identify the session you want. For this, you need the source IP address (usually your computer), the destination IP address (if you have it), and the port number which is determined by the program that you're using. Some commons ports are:
- Port 80 (HTTP for web browsing)
- Port 80 (HTTP for web browsing)
- Port 80 (HTTP for web browsing)
- Port 80 (HTTP for web browsing)
-
Find your session and policy ID.
Go to FortiView > All Sessions. Find your session by finding your source IP address, destination IP address (if you have it), and port number. The policy ID is listed after the destination information. If the list of sessions is very long, you can filter the list to make it easier to find your session.
-
When there are many sessions, use a filter to help you find your session.
If there are multiple pages of sessions, it's difficult to find a single session. You can use a filter to block out sessions that you don’t want. Click the search icon on the column heading to select the filter. Select Source IP and enter your source IP address. Now, only sessions that originate from your IP address are displayed in the session table. If the list is still too long, you can do the same for the Source port. That makes it easy to find your session and the security policy ID.
CLI session information
The session table output that the diagnose sys session list
command generates is very large. You can use filters to display only the session data that you're interested in.
An entry is placed in the session table for each traffic session passing through a security policy. The following command lists the information for a session in the table:
diagnose sys session list
The filter option displays specific information, for example:
diagnose sys session filter <option>
The values for <option>
include the following:
|
Clear session filter |
|
Destination interface |
|
Destination port |
|
Destination IP address |
|
Duration of the session |
|
Expire |
|
Inverse filter |
|
NAT'd source port |
|
NAT'd source ip address |
|
Policy ID |
|
Protocol number |
|
Protocol state |
|
Session state1 |
|
Session state2 |
|
Source interface |
|
Source port |
|
Source IP address |
|
Index of virtual domain, |
Even though UDP is a sessionless protocol, the FortiGate keeps track of the following states:
- When UDP reply doesn't have a value of 0
- When UDP reply has a value of 1
The following table displays firewall session states from the session table:
State |
Description |
---|---|
log |
Session is being logged |
local |
Session is originated from or destined for local stack |
ext |
Session is created by a firewall session helper |
may_dirty |
Session is created by a policy For example, the session for |
ndr |
Session will be checked by IPS signature |
nds |
Session will be checked by IPS anomaly |
br |
Session is being bridged (TP) mode |
Firewall session setup rate
The number of sessions that can be established in a set period of time is useful information. A session is an end-to-end TCP/IP connection for communication with a limited lifespan. If you record the setup rate during normal operation, when you experience problems you can compare the baseline setup rate to the rate that occurs when you're troubleshooting. This can be a useful step to help you define your problem.
A reduced firewall session setup rate can be the result of a number of things, such as a lack of system resources on the FortiGate or reaching the limit of your session count for your VDOM.
To view your session setup rate method 1 - CLI
FGT# get system performance status
CPU states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU0 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU1 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU2 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU3 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
Memory: 4050332k total, 530512k used (13%), 3376844k free (83%), 142976k freeable (3%)
Average network usage: 131 / 90 kbps in 1 minute, 26 / 15 kbps in 10 minutes, 49 / 42 kbps in 30 minutes
Average sessions: 80 sessions in 1 minute, 30 sessions in 10 minutes, 42 sessions in 30 minutes
Average session setup rate: 3 sessions per second in last 1 minute, 0 sessions per second in last 10 minutes, 0 sessions per second in last 30 minutes
Virus caught: 0 total in 1 minute
IPS attacks blocked: 0 total in 1 minute
Uptime: 1 days, 2 hours, 45 minutes
The information you're looking for is the average sessions section, in the above output. This example shows that there were 80 sessions in 1 minute, or an average of 3 sessions per second. The values for 10 minutes and 30 minutes allow you to take a longer average for a more reliable value if your FortiGate is working at maximum capacity. The smallest FortiGate can have 1,000 sessions established per second across the unit.
Remember that session setup rate is a global command. If you have multiple VDOMs configured with many sessions in each one, the session setup rate per VDOM will be slower than if there are no VDOMs configured.
Finding object dependencies
An administrator may not be permitted to delete a configuration object if there are other configuration objects that depend on it. This command identifies other objects which depend on, or make reference to, the configuration object in question. If an error is displayed that an object is in use and can't be deleted, this command can help identify the source of the problem.
Additionally, if you have a virtual interface with objects that depend on it, you need to find and remove those dependencies before you delete the interface.
CLI method
When you run multiple VDOMs, you use this command in the global configuration only and it searches for the named object in both the most recently used global and VDOM configurations:
diagnose sys cmdb refcnt show <path.object.mkey> <tablename>
For example, to verify which objects a security policy with an ID of 1 refers to, enter the following command:
diagnose sys cmdb refcnt show firewall.policy.policyid 1
To check what is referred to by interface port1
, enter the following command:
diagnose sys cmdb refcnt show system.interface.name port1
To show all dependencies for an interface, enter the following command:
diagnose sys cmdb refcnt show system.interface.name <interface name>
Sample output:
entry used by table firewall.address:name '10.98.23.23_host’
entry used by table firewall.address:name 'NAS'
entry used by table firewall.address:name 'all'
entry used by table firewall.address:name 'fortinet.com'
entry used by table firewall.vip:name 'TORRENT_10.0.0.70:6883'
entry used by table firewall.policy:policyid '21'
entry used by table firewall.policy:policyid '14'
entry used by table firewall.policy:policyid '19'
In this example, the interface has dependent objects, including four address objects, one VIP, and three security policies.
GUI method
In the GUI, you can easily check and remove the object dependencies for an interface.
To remove interface object dependencies - GUI
- Go to Network > Interfaces. The Ref. column displays the number of objects that refer to this interface.
- Select the number in the Ref. column for the interface. A window listing the dependencies appears.
- Use these detailed entries to locate and remove object references to this interface. The trash can icon changes from gray when you've removed all object dependencies.
- Remove the interface by selecting the check box for the interface, and select Delete.
Flow trace
To trace the flow of packets through the FortiGate, use the following command:
diagnose debug flow trace start
Follow packet flow by setting a flow filter, using this command:
diagnose debug flow {filter | filter6} <option>
If your network uses IPv4, enter filter
. If your network uses IPv6, enter filter6
.
One of the following variables replaces <option>
:
Variable |
Description |
---|---|
|
IPv4 or IPv6 address |
|
clear filter |
|
destination IPv4 or IPv6 address |
|
destination port |
|
inverse IPv4 or IPv6 filter |
|
port |
|
protocol number |
|
source address |
|
source port |
|
index of virtual domain; |
|
|
To start flow monitoring with a specific number of packets - CLI:
diagnose debug flow trace start <N>
To stop flow tracing at any time using - CLI:
diagnose debug flow trace stop
The following example shows the flow trace for a device with an IP address of 203.160.224.97
:
diagnose debug enable
diagnose debug flow filter addr 203.160.224.97
diagnose debug flow show function-name enable
diagnose debug flow trace start 100
Flow trace output example - HTTP
To observe the debug flow trace, connect to the web site at the following address:
https://www.fortinet.com
Comment: SYN packet received:
id=20085 trace_id=209 func=resolve_ip_tuple_fast
line=2700 msg="vd-root received a packet(proto=6,
192.168.3.221:1487->203.160.224.97:80) from port5."
SYN sent and a new session is allocated:
id=20085 trace_id=209 func=resolve_ip_tuple line=2799
msg="allocate a new session-00000e90"
Lookup for next-hop gateway address:
id=20085 trace_id=209 func=vf_ip4_route_input line=1543
msg="find a route: gw-192.168.11.254 via port6"
Source NAT, lookup next available port:
id=20085 trace_id=209 func=get_new_addr line=1219
msg="find SNAT: IP-192.168.11.59, port-31925"
direction“
Matched security policy. Check to see which policy this session matches:
id=20085 trace_id=209 func=fw_forward_handler line=317
msg="Allowed by Policy-3: SNAT"
Apply source NAT:
id=20085 trace_id=209 func=__ip_session_run_tuple
line=1502 msg="SNAT 192.168.3.221->192.168.11.59:31925"
SYN ACK received:
id=20085 trace_id=210 func=resolve_ip_tuple_fast line=2700
msg="vd-root received a packet(proto=6, 203.160.224.97:80-
>192.168.11.59:31925) from port6."
Found existing session ID. Identified as the reply direction:
id=20085 trace_id=210 func=resolve_ip_tuple_fast line=2727
msg="Find an existing session, id-00000e90, reply direction"
Apply destination NAT to inverse source NAT action:
id=20085 trace_id=210 func=__ip_session_run_tuple
line=1516 msg="DNAT 192.168.11.59:31925-
>192.168.3.221:1487"
Lookup for next-hop gateway address for reply traffic:
id=20085 trace_id=210 func=vf_ip4_route_input line=1543
msg="find a route: gw-192.168.3.221 via port5"
ACK received:
id=20085 trace_id=211 func=resolve_ip_tuple_fast line=2700
msg="vd-root received a packet(proto=6,
192.168.3.221:1487->203.160.224.97:80) from port5."
Match existing session in the original direction:
id=20085 trace_id=211 func=resolve_ip_tuple_fast line=2727
msg="Find an existing session, id-00000e90, original
direction"
Apply source NAT:
id=20085 trace_id=211 func=__ip_session_run_tuple
line=1502 msg="SNAT 192.168.3.221->192.168.11.59:31925"
Receive data from client:
id=20085 trace_id=212 func=resolve_ip_tuple_fast
line=2700 msg="vd-root received a packet(proto=6,
192.168.3.221:1487->203.160.224.97:80) from port5."
Match existing session in the original direction:
id=20085 trace_id=212 func=resolve_ip_tuple_fast
line=2727 msg="Find an existing session, id-00000e90,
original direction"
Apply source NAT:
id=20085 trace_id=212 func=__ip_session_run_tuple
line=1502 msg="SNAT 192.168.3.221->192.168.11.59:31925"
Receive data from server:
id=20085 trace_id=213 func=resolve_ip_tuple_fast
line=2700 msg="vd-root received a packet(proto=6,
203.160.224.97:80->192.168.11.59:31925) from port6."
Match existing session in reply direction:
id=20085 trace_id=213 func=resolve_ip_tuple_fast
line=2727 msg="Find an existing session, id-00000e90,
reply direction"
Apply destination NAT to inverse source NAT action:
id=20085 trace_id=213 func=__ip_session_run_tuple
line=1516 msg="DNAT 192.168.11.59:31925-
>192.168.3.221:1487"
Flow trace output example - IPsec (policy-based)
id=20085 trace_id=1 msg="vd-root received a packet(proto=1, 10.72.55.240:1->10.71.55.10:8) from internal."
id=20085 trace_id=1 msg="allocate a new session-00001cd3"
id=20085 trace_id=1 msg="find a route: gw-66.236.56.230 via wan1"
id=20085 trace_id=1 msg="Allowed by Policy-2: encrypt"
id=20085 trace_id=1 msg="enter IPsec tunnel-RemotePhase1"
id=20085 trace_id=1 msg="encrypted, and send to 15.215.225.22 with source 66.236.56.226"
id=20085 trace_id=1 msg="send to 66.236.56.230 via intf-wan1“
id=20085 trace_id=2 msg="vd-root received a packet (proto=1, 10.72.55.240:1-1071.55.10:8) from internal."
id=20085 trace_id=2 msg="Find an existing session, id-00001cd3, original direction"
id=20085 trace_id=2 msg="enter IPsec ="encrypted, and send to 15.215.225.22 with source 66.236.56.226“ tunnel-RemotePhase1"
id=20085 trace_id=2 msgid=20085 trace_id=2 msg="send to 66.236.56.230 via intf-wan1"
Packet sniffing and packet capture
When you troubleshoot networks, it helps to look inside the header of the packets. This helps to determine if the packets, route, and destination are all what you expect. Packet capture can also be called a network tap, packet sniffing, or logic analyzing. FortiOS devices can sniff packets using CLI commands or capture packets using the GUI.
Packet sniffing using CLI commands is well-suited for spot checking traffic, but if you have complex filters to enter it can be a lot of work to enter them each time. You can also save the sniffing output. However, you must log to a file and then analyze the file later.
Packet capture in the GUI makes it easy for you to set up multiple filters at once and run only one or two as you need to. You can also use controls to start and stop capturing when you want to. You download packet capture output to your local computer as a *.pcap file. You must use a third party application, such as Wireshark, to read *.pcap files. This method is useful to send information to Fortinet support to help resolve an issue.
The following table presents a comparison between the two methods:
Features |
Packet sniffing |
Packet capture |
---|---|---|
Command location |
CLI |
GUI |
Third party software required |
puTTY to log plaintext output |
Wireshark, or similar application, to read *.pcap files |
Read output in plain text file |
yes |
no |
Read output as *.pcap file using Wireshark, or similar application |
no |
yes |
Easily configure single quick and simple filter |
yes |
no |
Record packet interface |
yes |
no |
Configure complex sniffer filters on multiple interface |
no |
yes |
sniff IPv6 |
hard |
easy |
sniff non-IP packets |
no |
yes |
Filter packets by protocol and/or port |
easy |
easy |
Filter packets by source and/or destination address |
easy |
easy |
Packet sniffing
If you're running a constant traffic application, such as ping, packet sniffing can tell you if the traffic is reaching the destination, what the port of entry is on the FortiGate unit, if the ARP resolution is correct, and if the traffic is being sent back to the source as expected.
Sniffing packets can also tell you if the FortiGate is silently dropping packets for reasons such as Reverse Path Forwarding (RPF). RPF, also called anti-spoofing, prevents an IP packet from being forwarded if its source IP doesn't belong to a locally attached subnet (local interface) or isn't part of the routing between the FortiGate and another source (static route, RIP, OSPF, BGP). Note that you can disable RPF by turning on asymmetric routing in the CLI (config system settings, set asymroute enable
), but this disables stateful inspection on the FortiGate and causes many features to be turned off.
|
If you configure virtual IP addresses on your FortiGate, it will use those addresses instead of the physical IP addresses. You'll notice this when you're sniffing packets because all the traffic uses the virtual IP addresses. This is due to the ARP update that's sent out when the VIP address is configured. |
Before you start sniffing packets on the CLI, you should prepare to capture the output to a file. A large amount of data may scroll by and you won't be able to see it without first saving it to a file. One method is to use a terminal program like puTTY to connect to the FortiGate CLI. Once the packet sniffing count is reached, you can end the session and analyze the output in the file.
The general form of the internal FortiOS packet sniffer command is:
diagnose sniffer packet <interface_name> <‘filter’> <verbose> <count> <tsformat>
To stop the sniffer, type CTRL+C
.
|
The name of the interface to sniff, such as “port1” or “internal”. This can also be “any” to sniff all interfaces. |
|
What to look for in the information the sniffer reads. “none” indicates no filtering, and all packets are displayed as the other arguments indicate. |
|
The level of verbosity as one of: |
|
The number of packets the sniffer reads before stopping. If you don't put a number here, the sniffer will run until you stop it with <CTRL+C>. |
|
The format of timestamp.
|
For a simple sniffing example, enter the CLI command diagnose sniffer packet port1 none 1 3
. This displays the next three packets on the port1 interface using no filtering, and verbose level 1. At this verbosity level, you can see the source IP and port, the destination IP and port, action (such as ack), and sequence numbers.
In the output below, port 443 indicates these are HTTPS packets and that 172.20.120.17 is both sending and receiving traffic.
Head_Office_620b # diagnose sniffer packet port1 none 1 3
interfaces=[port1]
filters=[none]
0.545306 172.20.120.17.52989 -> 172.20.120.141.443: psh 3177924955 ack 1854307757
0.545963 172.20.120.141.443 -> 172.20.120.17.52989: psh 1854307757 ack 3177925808
0.562409 172.20.120.17.52988 -> 172.20.120.141.443: psh 4225311614 ack 3314279933
For a more advanced example of packet sniffing, the following commands will report packets on any interface that are traveling between a computer with the host name of “PC1” and a computer with the host name of “PC2”. With verbosity 4 and above, the sniffer trace displays the interface names where traffic enters or leaves the FortiGate unit. Remember to stop the sniffer, type CTRL+C
.
FGT# diagnose sniffer packet any "host <PC1> or host <PC2>" 4
or
FGT# diagnose sniffer packet any "(host <PC1> or host <PC2>) and icmp" 4
The following CLI command for a sniffer includes the ARP protocol in the filter which may be useful to troubleshoot a failure in the ARP resolution (for example, PC2 may be down and not responding to the FortiGate ARP requests).
FGT# diagnose sniffer packet any "host <PC1> or host <PC2> or arp" 4
Packet capture
Packet capture tells you what is happening on the network at a low level. This can be very useful for troubleshooting problems, such as:
- Finding missing traffic
- Seeing if sessions are setting up properly
- Locating ARP problems such as broadcast storm sources and causes
- Confirming which address a computer is using on the network, if they have multiple addresses or are on multiple networks
- Confirming routing is working as you expect
- Connecting wireless clients
- Missing PING packets
- A particular type of packet is having problems, such as UDP, which is commonly used for streaming video
If you're running a constant traffic application such as ping, packet capture can tell you if the traffic is reaching the destination, how the port enters and exits the FortiGate, if the ARP resolution is correct, and if the traffic is returning to the source as expected. You can also use packet switching to verify that NAT or other configuration is translating addresses or routing traffic the way that you want it to.
Before you start capturing packets, you need to have a good idea of what you're looking for. Capture is used to confirm or deny your ideas about what's happening on the network. If you try capture without a plan to narrow your search, you can end up with too much data to effectively analyze. On the other hand, you need to capture enough packets to really understand all of the patterns and behavior that you're examining.
To use packet capture, the FortiGate must have a disk. You can enable the capture-packet
in the firewall policy, using the following CLI commands:
config firewall policy
edit <id>
set capture-packet enable
end
To configure packet capture filters, go to Network > Packet Capture.
When you add a packet capture filter, enter the following information and select OK.
Interface |
Select the interface to sniff from the drop-down menu. You must select one interface. You can't change the interface without deleting the filter and creating a new one, unlike the other fields. |
Max Packets to Save |
Enter the number of packets to capture before the filter stops. This number can't be zero. You can halt the capturing before this number is reached. |
Enable Filters |
Select this option to specify filter fields |
Host(s) |
Enter the IP address of one or more hosts. Separate multiple hosts with commas. To enter a range, use a dash without spaces, for example 172.16.1.5-172.16.1.15, or enter a subnet. |
Port(s) |
Enter one or more ports to capture on the selected interface. Separate multiple ports with commas. To enter a range, use a dash without spaces, for example 88-90 |
VLAN(s) |
Enter one or more VLANs (if any). Separate multiple VLANs with commas. |
Protocol |
Enter one or more protocols. Separate multiple protocols with commas. To enter a range, use a dash without spaces, for example 1-6, 17, 21-25. |
Include IPv6 Packets |
Select this option if you're troubleshooting IPv6 networking, or if your network uses IPv6. Otherwise, leave it disabled. |
Include Non-IP Packets |
The protocols in the list are all IP based except for ICMP (ping). To capture non-IP based packets, select this feature. Examples of non-IP packets include IPsec, IGMP, ARP, and ICMP. |
If you select a filter, you have the option to start and stop packet capture in the edit window, or download the captured packets. You can also see the filter status and the number of packets captured.
You can select the filter and start capturing packets. When the filter is running, the number of captured packets increases until it reaches the Max Packet Count or you stop it. When the filter is running, you can't download the output file.
When the packet capture is complete, you can download the *.pcap file. You must use a third party application, such as Wireshark, to read *,pcap files. This tool provides you with extensive analytics and the full contents of the packets that were captured.
To start, stop, or resume packet capture, use the symbols on the screen. These symbols are the same as those used for audio or video playback. Hover over the symbol to reveal explanatory text. Similarly, to download the *.pcap file, use the download symbol on the screen.
NPU-based interfaces
Many Fortinet products contain network processors, such as NP1, NP2, NP4, and NP6, which means that offloading requirements vary depending on the model.
When you use the NPU-based interfaces, you can see only the initial session setup will be seen using the diagnose debug flow
command. If the session is correctly programmed into the ASIC (fastpath), the diagnose debug flow
command won't detect the packets that arrive at the CPU. If the NPU functionality is disabled, the CPU detects all the packets. However, you should only this for troubleshooting purposes.
First, obtain the NP4 or NP6 ID and the port numbers, using the following command:
diagnose npu {np4|npu6}list
Sample output:
ID Model Slot Interface
0 On-board port1 fabric1 fabric3 fabric5
1 On-board fabric2 port2 base2 fabric4
Run the following commands:
diagnose npu {np4|npu6}fastpath disable <dev_id>
(where dev_id
is the NP4 or NP6 number)
Then, run this command:
diagnose npu {np4|npu6}fastpath-sniffer enable port1
Sample output:
NP4 Fast Path Sniffer on port1 enabled
This causes traffic on port1 of the network processor to be sent to the CPU. this means that you can take a standard sniffer trace and use other diagnose commands, if it's a standard CPU-driven port.
These commands only apply to the newer NP4 and NP6 interfaces.
Debug command
Debug output provides continuous, real-time event information and continues until you stop it or reboot the unit. Debug output can affect system performance and is continually generated even though output might not be displayed in the CLI console.
Debug information that's displayed in the console scrolls in the console display and may prevent you from entering CLI commands, such as, the command to disable the debug display. To turn off debug output as the display scrolls by, press the ↑ key to recall the recent diagnose debug command, press backspace, and type “0”, and Enter.
To enable debug output display, use the following command:
diagnose debug enable
Once you enable debug output display, specify the debug information that you require using the following command:
diagnose debug <option> <level>
Debug command options include the following:
|
Enable debug output |
|
Disable debug output |
|
Show active debug level settings |
|
Reset all debug level to default |
|
Report for tech support |
|
Crash log info |
|
Configure error log info |
|
SQL log database error info |
|
application |
|
kernel |
|
remote-extender |
|
Debug CLI |
|
Trace CLI |
|
Display rating info |
|
Authentication daemon |
|
FSSO active directory poll module |
|
Trace packet flow in kernel |
|
urlfilter |
|
Admin user |
You can set the debug level at the end of the command. For example, typical values are 2 and 3, like in the following commands:
diagnose debug application DHCPS 2
diagnose debug application spamfilter 2
Fortinet support will advise you about which debugging level you should use.
You can enable timestamps to the debug output, using the following command:
diagnose debug console timestamp enable
When you finish examining the debug output, disable it, using the following command:
diagnose debug disable
Debug output example
This example shows the IKE negotiation for a secure logging connection from a FortiGate to a FortiAnalyzer.
diagnose debug reset
diagnose vpn ike log-filter src-addr4 192.168.11.2
diagnose debug enable
Sample output:
FGh_FtiLog1: IPsec SA connect 0 192.168.11.2->192.168.10.201:500, nat_mode=0 rekey=0 phase2=FGh_FtiLog1
FGh_FtiLog1: using existing connection, dpd_fail=0
FGh_FtiLog1: found phase2 FGh_FtiLog1
FGh_FtiLog1: IPsec SA connect 0 192.168.11.2 -> 192.168.10.201:500 negotiating
FGh_FtiLog1: overriding selector 225.30.5.8 with 192.168.11.2
FGh_FtiLog1: initiator quick-mode set pfs=1536...
FGh_FtiLog1: try to negotiate with 1800 life seconds.
FGh_FtiLog1: initiate an SA with selectors: 192.168.11.2/0.0.0.0->192.168.10.201, ports=0/0, protocol=0/0
Send IKE Packet(quick_outI1):192.168.11.2:500(if0) -> 192.168.10.201:500, len=348
Initiator: sent 192.168.10.201 quick mode message #1 (OK)
FGh_FtiLog1: set retransmit: st=168, timeout=6.
In this example:
|
Source and destination gateway IP address |
|
Found existing Phase 1 |
|
Create new Phase 2 tunnel |
The execute tac report command
exec tac report
is an execute command that runs an exhaustive series of diagnostic commands. It runs commands that are only needed if you're using certain features, such as HA, VPN tunnels, or a modem. The report takes a few minutes to finish because of the amount of output that's generated. If you're logging CLI output to a file, you can run this command to familiarize yourself with the diagnostic commands.
When you contact Fortinet Support, you may be asked to use the output from this CLI command to provide information about your FortiGate and its current state.
Other commands
ARP table
To view the ARP cache, use the following command:
get system arp
To view the ARP cache in the system, use the following command:
diagnose ip arp list
Sample output:
index=14 ifname=internal 224.0.0.5 01:00:5e:00:00:05 state=00000040 use=72203 confirm=78203 update=72203 ref=1
index=13 ifname=dmz 192.168.3.100 state=00000020 use=1843 confirm=650179 update=644179 ref=2 ? VIP
index=13 ifname=dmz 192.168.3.109 02:09:0f:78:69:ff state=00000004 use=71743 confirm=75743 update=75743 ref=1
index=14 ifname=internal 192.168.11.56 00:1c:23:10:f8:20 state=00000004 use=10532 confirm=10532 update=12658 ref=4
To remove the ARP cache, use the following command:
execute clear system arp table
To remove a single ARP entry, use the following command:
diagnose ip arp delete <interface name> <IP address>
To add static ARP entries, use the following command:
config system arp-table
Time and date settings
Check time and date settings for log message timestamp synchronization (the Fortinet support group may request this) and for certificates that have a time requirement to check for validity. Use the following commands:
execute time
current time is: 12:40:48
last ntp sync:Thu Mar 16 12:00:21 2006
execute date
current date is: 2006-03-16
To force synchronization with an NTP server, use the following command:
config system ntp
set ntpsync {enable|disable}
end
If all devices have the same time, it helps to correlate log entries from different devices.
IP address
There may be times when you want to verify that the IP addresses assigned to the FortiGate interfaces are what you expect them to be. This is easily accomplished from the CLI, using the following command:
diagnose ip address list
The output from this command lists the IP address and mask (if available), the index
of the interface (a type of ID number), and the devname
(the interface name). While physical interface names are set, virtual interface names can vary. A good way to use this command is to list all of the virtual interface names. Listing all the virtual interface names is a good use of this command. For vsys_ha
and vsys_fgfm
, the IP addresses are the local host, which are virtual interfaces that are used internally.
# diagnose ip address list
IP=10.31.101.100->10.31.101.100/255.255.255.0 index=3 devname=internal
IP=172.20.120.122->172.20.120.122/255.255.255.0 index=5 devname=wan1
IP=127.0.0.1->127.0.0.1/255.0.0.0 index=8 devname=root
IP=127.0.0.1->127.0.0.1/255.0.0.0 index=11 devname=vsys_ha
IP=127.0.0.1->127.0.0.1/255.0.0.0 index=13 devname=vsys_fgfm