Debugging traffic flow at user level with diagnose commands
The most commonly used diagnose debug flow commands are combined as below:
Reset enabled diagnose settings, turn on debug log output with timestamp
diagnose debug reset
diagnose debug timestamp enable
Add filters and start the flow trace
diagnose debug flow filter flow-detail 7 #Enables messages from each packet processing module and packet flow traces
diagnose debug flow filter HTTP-detail 7 #HTTP parser details
diagnose debug flow filter module-detail status on #Turn on details from WAF modules processing the flow
diagnose debug flow filter module-detail module <module> #Specify all or specific module(s)
diagnose debug flow filter server-IP 192.168.12.12 #The VIP in RP mode or the real server IP in TP/TI mode
diagnose debug flow filter client-IP 192.168.12.1 #The client IP
diagnose debug flow filter pserver-ip <C.C.C.C> #The real server IP for RP mode only; supported from 6.3.21 and 7.0.3
diagnose debug flow trace start
diagnose debug enable
Stop output
diagnose debug flow trace stop
Diagnose debug disable
Please note the following:
-
Client-IP & server-IP are supported on all 6.3.x and 7.0.x builds; pserver-ip is supported on 6.3.21 and later, and 7.0.x builds.
-
The relationship of IP filters (client-IP, server-IP and pserver-IP) for diagnose debug flow are different on different FortiWeb builds. Please check the following description.
-
Logical relationship between IP filters on 6.3.20, 7.0.1 and earlier builds:
-
Only client-IP and server-IP are supported on these builds;
-
The logic relationship between the client-IP and server-IP are AND, that is to say, only logs for traffic flows matching both filters will be printed out;
Example 1: When only one IP filter, either client-IP or server-IP, is specified, diagnose logs for the traffic flow matching the IP filter will be printed out.
Example 2: When all two IP filters are set, diagnose logs for the traffic flow matching the both IP filters will be printed out.
-
A known limitation is that when TLS 1.3 is deployed on the back-end side (between FortiWeb and the real back-end servers) and any IP flow filter is specified, the SSL pre-master secrets for the back-end side will not be printed out. You need to remove all IP filters to retrieve the TLS 1.3 secrets.
Please refer to Decrypting SSL packets to analyze traffic issues to analyze traffic issues for more details.
-
-
Logical relationship between IP filters on 6.3.21, 7.0.3 and later 7.0.x builds:
-
Three IP filters (client-IP, server-IP and pserver-IP) are supported.
-
If only the front-end IP filters (client-IP or/and server-IP) are configured, the logic relationship between the two front-end filters is AND, but please note now log print only includes front-end TLS key but not back-end TLS key.
-
If both the front-end filters (client-IP or/and server-IP) and the back-end filter pserver-IP is specified, the relationship between the front-end filters and the back-end filter is OR, that is to say the flows either matching the front-end or back-end IP filters will be printed out, and the debug print will include TLS key of both front-end and back-end.
For example, with the following filters specified:
diagnose debug flow filter client-IP <A.A.A.A>
diagnose debug flow filter server-IP <B.B.B.B>
diagnose debug flow filter pserver-IP <C.C.C.C>
These traffic flows will be printed in diagnose logs:
From A.A.A.A to B.B.B.B, and distributed to pserver C.C.C.C
From A.A.A.A to B.B.B.B, and distributed to pserver D.D.D.D #the client side flow from A.A.A.A to B.B.B.B will be printed, while the server side flow from FortiWeb to D.D.D.D will NOT be printed
From E.E.E.E to F.F.F.F, and distributed to pserver C.C.C.C #the server side flow from FortiWeb to C.C.C.C will be printed, while the client side flow from E.E.E.E to F.F.F.F will NOT be printed
These traffic flows will NOT be printed in diagnose logs:
From A.A.A.A to F.F.F.F, and distributed to pserver D.D.D.D
-
-
Diagnose debug flow usually results in a large amount of prints and impacts the performance. So if the traffic is heavy or the system resources has been highly occupied, you should enable diagnose debug flow with caution.
Some basic recommendations:
-
Enable diagnose in the SSH terminal instead of the Serial Console.
-
Under normal circumstances, enabling the filter client-IP only is recommended when debugging issues in the production environment.
Avoid just specifying the server-IP or pserver-IP, because there might be excessive output on SSH or Console terminals.
-
Just set a low log priority level, and don’t enable unnecessary filters.
For example, if you intend to retrieve SSL pre-master secrets to decrypt SSL traffic, just set
diagnose debug flow filter flow-detail 4and do not enable module-detail. -
Don’t forget to execute
diagnose debug disableordiagnose debug resetafter debug is done.
-
Debugging traffic flow at kernel level
Change the debug levels in the back-end settings, then kernel level debug logs will be recorded in dmesg. This method is useful to track traffic flow processing in the system kernel.
/proc/tproxy/debug # for transparent mode.
-
echo "FFFF F" > proc/tproxy/debug: output logs to dmesg with a detailed level
-
echo "XXXX F" > proc/tproxy/debug: don’t forget to turn off debug logs
Use the same way to turn on debug logs for reverse-proxy and wccp mode.
Some details:
/var/log# more /proc/tproxy/debug
Debug modules : HOOK4 HOOK6 HASH POLICY
HOOK4 : for netfilter hook IPv4
HOOK6 : for netfilter hook IPv6
HASH : for tproxy hash
POLICY : for policy management
FFFF : for all above
XXXX : cleanup all above
PASS : for bypass this module in kernel path
LOIP : for enable / disable local IP filter in hook4
PIP : <PIP [1,0] ip> for only enbale this IP upto proxyd
Debug levels : 1 2 4 8
1 : for error message
2 : for data packet info
4 : for data following info
8 : for function entry/exit info
Current debug info : FFFF 15, mbypass = 0, sysmode : 2, localip : 0, proxyd-ip : 0.0.0.0
ex : echo "HOOK4 F" > debug > debug
ex : echo "PIP 1 10.200.2.1" > debug
Example:
[BEGIN] 9/13/2021 23:35:55
/# dmesg
[553897.203831] (tproxy) (/Chroot_Build/34/SVN_REPO_CHILD/FortiWEB/kernel/modules/tproxy/tproxy_policy.c:433) get vserver(240.0.0.29), vport(9781), dir(1)
[553897.203834] (tproxy) ====> get vserver(240.0.0.29), vport(9781), mark(1835264/1835264), incoming (vzone_p3p4_vlan) tcp info : src:(192.168.11.1:48310), dst:(192.168.11.2:80)
[553897.203836] (tproxy) (465) incoming (vzone_p3p4_vlan) tcp info : src:(192.168.11.1:48310), dst:(192.168.11.2:80) -ipid(63355) iptlen(60) seq(2348868809) ack_seq(0) syn(1) ack(0) fin(0) rst(0) psh(0)
[553897.203838] (tproxy) [fortiweb-tproxy] redirecting: proto 6 192.168.11.2:80 -> 240.0.0.29:9781, ipid(63355) iplen(60) mark: 1c0100
[553897.203855] (tproxy)
[553897.203855]
[553897.203855] ====> out to client : src:(192.168.11.2:80), dst:(192.168.11.1:48310)- seq(1319007036) ack_seq(2348868810) syn(1) ack(1) fin(0) rst(0) psh(0)
[553897.203856] (tproxy) [POST_ROUTING]: TO CLIENT OK, 192.168.11.2:80->192.168.11.1:48310, todevname:port3vlan101, flag 4000
/proc/rptproxy/debug #for reverse-proxy mode
/var/log# more /proc/rptproxy/debug
Debug modules : HOOK4 HOOK6 HASH POLICY
HOOK4 : for netfilter hook IPv4
HOOK6 : for netfilter hook IPv6
POLICY : for policy management
FFFF : for all above
XXXX : cleanup all above
PASS : for bypass this module in kernel path
LOIP : for enable / disable local IP filter in hook4
PIP : <PIP [1,0] ip> for only enbale this ip upto proxyd
Debug levels : 1 2 4 8
...
Current debug info : 0, mbypass = 0, sysmode : 2, localip : 0, proxyd-ip : 0.0.0.0
/proc/wproxy/debug #for wccp mode
/var/log# more /proc/wproxy/debug
Debug modules : HOOK4 HOOK6 POLICY
HOOK4 : for netfilter hook IPv4
HOOK6 : for netfilter hook IPv4
POLICY : for policy management
FFFF : for all above
XXXX : cleanup all above
PASS : for bypass this module in kernel path
Debug levels : 1 2 4 8
...
Current debug info : 0, mbypass = 0, sysmode : 1
How to capture network packets in FortiWeb
Capturing network packets is a useful and direct method when troubleshooting network issues, including TCP connection establishment issues, SSL handshake issues or analyzing HTTP issues.
Usually it’s better to enable diagnose debug flow and capture packets at the same time, then analyze them together.