Debugging traffic flow at user level with diagnose commands
The most commonly used diagnose debug flow commands are combined as below:
Reset enabled diagnose settings, turn on debug log output with timestamp
diagnose debug reset
diagnose debug enable
diagnose debug timestamp enable
Add filters and start the flow trace
diagnose debug flow filter flow-detail 7 #Enables messages from each packet processing module and packet flow traces
diagnose debug flow filter http-detail 7 #HTTP parser details
diagnose debug flow filter module-detail status on #Turn on details from modules processing the flow
diagnose debug flow filter server-ip 192.168.12.12 #The VIP in RP mode or the real server IP in TP/TI mode
diagnose debug flow filter client-ip 192.168.12.1 #The client IP
diagnose debug flow trace start
To stop output
diagnose debug flow trace stop
Diagnose debug disable
Debugging traffic flow at kernel level
Change the debug levels in the backend settings, then kernel level debug logs will be recorded in dmesg. This method is useful to track traffic flow processing in the system kernel.
1) /proc/tproxy/debug # for transparent mode.
-
echo "FFFF F" > proc/tproxy/debug: output logs to dmesg with a detailed level
-
echo "XXXX F" > proc/tproxy/debug: don’t forget to turn off debug logs
Use the same way to turn on debug logs for reverse-proxy and wccp mode.
Some details:
/var/log# more /proc/tproxy/debug
Debug modules : HOOK4 HOOK6 HASH POLICY
HOOK4 : for netfilter hook ipv4
HOOK6 : for netfilter hook ipv6
HASH : for tproxy hash
POLICY : for policy management
FFFF : for all above
XXXX : cleanup all above
PASS : for bypass this module in kernel path
LOIP : for enable / disable local ip filter in hook4
PIP : <PIP [1,0] ip> for only enbale this ip upto proxyd
Debug levels : 1 2 4 8
1 : for error message
2 : for data packet info
4 : for data following info
8 : for function entry/exit info
Current debug info : FFFF 15, mbypass = 0, sysmode : 2, localip : 0, proxyd-ip : 0.0.0.0
ex : echo "HOOK4 F" > debug > debug
ex : echo "PIP 1 10.200.2.1" > debug
Example:
[BEGIN] 9/13/2021 23:35:55
/# dmesg
[553897.203831] (tproxy) (/Chroot_Build/34/SVN_REPO_CHILD/FortiWEB/kernel/modules/tproxy/tproxy_policy.c:433) get vserver(240.0.0.29), vport(9781), dir(1)
[553897.203834] (tproxy) ====> get vserver(240.0.0.29), vport(9781), mark(1835264/1835264), incoming (vzone_p3p4_vlan) tcp info : src:(192.168.11.1:48310), dst:(192.168.11.2:80)
[553897.203836] (tproxy) (465) incoming (vzone_p3p4_vlan) tcp info : src:(192.168.11.1:48310), dst:(192.168.11.2:80) -ipid(63355) iptlen(60) seq(2348868809) ack_seq(0) syn(1) ack(0) fin(0) rst(0) psh(0)
[553897.203838] (tproxy) [fortiweb-tproxy] redirecting: proto 6 192.168.11.2:80 -> 240.0.0.29:9781, ipid(63355) iplen(60) mark: 1c0100
[553897.203855] (tproxy)
[553897.203855]
[553897.203855] ====> out to client : src:(192.168.11.2:80), dst:(192.168.11.1:48310)- seq(1319007036) ack_seq(2348868810) syn(1) ack(1) fin(0) rst(0) psh(0)
[553897.203856] (tproxy) [POST_ROUTING]: TO CLIENT OK, 192.168.11.2:80->192.168.11.1:48310, todevname:port3vlan101, flag 4000
2) /proc/rptproxy/debug #for reverse-proxy mode
/var/log# more /proc/rptproxy/debug
Debug modules : HOOK4 HOOK6 HASH POLICY
HOOK4 : for netfilter hook ipv4
HOOK6 : for netfilter hook ipv6
POLICY : for policy management
FFFF : for all above
XXXX : cleanup all above
PASS : for bypass this module in kernel path
LOIP : for enable / disable local ip filter in hook4
PIP : <PIP [1,0] ip> for only enbale this ip upto proxyd
Debug levels : 1 2 4 8
...
Current debug info : 0, mbypass = 0, sysmode : 2, localip : 0, proxyd-ip : 0.0.0.0
3) /proc/wproxy/debug #for wccp mode
/var/log# more /proc/wproxy/debug
Debug modules : HOOK4 HOOK6 POLICY
HOOK4 : for netfilter hook ipv4
HOOK6 : for netfilter hook ipv4
POLICY : for policy management
FFFF : for all above
XXXX : cleanup all above
PASS : for bypass this module in kernel path
Debug levels : 1 2 4 8
...
Current debug info : 0, mbypass = 0, sysmode : 1
How to capture network packets in FortiWeb
Capturing network packets is a useful and direct method when troubleshooting network issues, including TCP connection establishment issues, SSL handshake issues or analyzing HTTP issues.
Usually it’s better to enable diagnose debug flow
and capture packets at the same time, then analyze them together.