Explicit proxy logging enhancements

This information is also available in the FortiOS 7.4 Administration Guide: Explicit proxy logging

Explicit proxy traffic logging has been enhanced to improve troubleshooting the HTTP proxy status for each HTTP transaction with the following:

• Support monitoring HTTP header requests and responses in the UTM web filter log. This requires an SSL deep inspection profile to be configured in the corresponding firewall policy.

• Support logging the explicit web proxy forward server name using set log-forward-server, which is disabled by default.

  config web-proxy global
      set log-forward-server {enable | disable}
  end

• Support logging TCP connection failures in the traffic log when a client initiates a TCP connection to a remote host through the FortiGate, and the remote host is unreachable.

Basic configuration

The following FortiGate configuration is used in the three explicit proxy traffic logging use cases in this topic.

To configure the FortiGate:

1. Configure the web proxy profile:

   config web-proxy profile
       edit "header"
           config headers
               edit 1
                   set name "test_request_header"
                   set action monitor-request
               next
               edit 2
                   set name "ETag"
                   set action monitor-response
               next
           end
       next
   end

2. Enable forward server name logging in traffic:

   config web-proxy global
       set proxy-fqdn "100D.qa"
       set log-forward-server enable
   end
   

3. Configure the web filter banned word table to block any HTTP response containing the text, works:

   config webfilter content
       edit 1
           set name "default"
           config entries
               edit "works"
                   set status enable
                   set action block
               next
           end
       next
   end
   

4. Configure the web filter profile:

   config webfilter profile
       edit "header"
           set feature-set proxy
           config web
               set bword-table 1
           end
           config ftgd-wf
               unset options
           end
           set log-all-url enable
           set extended-log enable
           set web-extended-all-action-log enable
       next
   end

5. Configure the web proxy forwarding server:

   config web-proxy forward-server
       edit "fgt-b"
           set ip 172.16.200.20
       next
   end

6. Configure the firewall policy:

   config firewall policy
       edit 1
           set srcintf "port10"
           set dstintf "port9"
           set action accept
           set srcaddr "all"
           set dstaddr "all"
           set schedule "always"
           set service "ALL"
           set utm-status enable
           set inspection-mode proxy
           set webproxy-profile "header"
           set webproxy-forward-server "fgt-b"
           set ssl-ssh-profile "deep-inspection"
           set webfilter-profile "header"
           set logtraffic all
           set nat enable
       next
   end

A firewall policy is used in this basic configuration example and the specific examples that follow. This feature also works for the explicit web proxy or transparent web proxy with proxy policies, and the configurations are similar: Example 1: apply the web-proxy profile and webfilter profile to the proxy policy. Example 2: apply the webproxy-forward-server.

Example 1: monitoring HTTP header requests

In this example, the user wants to monitor some HTTP headers in HTTP messages forwarded through a FortiGate proxy (either transparent or explicit proxy with a firewall policy in proxy mode or a proxy policy). When the monitored headers are detected, they will be logged in the UTM web filter log.

In the web proxy profile configuration, the following HTTP headers are monitored:

• test_request_header: this is a user-customized HTTP header.
• ETag: this is a HTTP header returned by the web server's 200 OK response.

Based on the web filter profile configuration, the monitored headers in the web proxy profile will only be logged when the HTTP response received by the FortiGate triggers a block action by the banned word table. The log-all-url, extended-log, and web-extended-all-action-log settings in the web filter profile must be enabled.

The following settings are required in the firewall policy:

• set inspection-mode proxy
• set webproxy-profile "header"
• set ssl-ssh-profile "deep-inspection"
• set webfilter-profile "header"
• set logtraffic all

To verify the configuration:

1. Send a HTTP request from the client:

   curl -kv https://172.16.200.33 -H "test_request_header: aaaaa"

   This command sends a HTTP request with the header test_request_header: aaaaa through the FortiGate. Since the response from the web server contains the word works, the response will be blocked by the web filter profile (header). During this process, two logs will be generated.

2. On the FortiGate, check the traffic logs:

   # execute log filter category 3
   1: date=2023-04-19 time=19:01:19 eventtime=1681956079146481995 tz="-0700" logid="0314012288" type="utm" subtype="webfilter" eventtype="content" level="warning" vd="vdom1" policyid=1 poluuid="4d8dc396-46e3-51ea-7f3f-ee328a5bd07b" policytype="policy" sessionid=40980 srcip=10.1.100.13 srcport=54512 srccountry="Reserved" srcintf="port10" srcintfrole="undefined" srcuuid="6ce0b8ca-30ae-51ea-a388-ceacbb4fb045" dstip=172.16.200.33 dstport=443 dstcountry="Reserved" dstintf="port9" dstintfrole="undefined" dstuuid="6ce0b8ca-30ae-51ea-a388-ceacbb4fb045" proto=6 httpmethod="GET" service="HTTPS" hostname="172.16.200.33" agent="curl/7.61.1" profile="header" reqtype="direct" url="https://172.16.200.33/" sentbyte=0 rcvdbyte=0 direction="incoming" action="blocked" banword="works" msg="URL was blocked because it contained banned word(s)." rawdata="[REQ] test_request_header:=aaaaa||[RESP] Content-Type=text/html|ETag=\"34-5b23b9d3b67f4\""
   
   2: date=2023-04-19 time=19:01:19 eventtime=1681956079144896978 tz="-0700" logid="0319013317" type="utm" subtype="webfilter" eventtype="urlmonitor" level="notice" vd="vdom1" policyid=1 poluuid="4d8dc396-46e3-51ea-7f3f-ee328a5bd07b" policytype="policy" sessionid=40980 srcip=10.1.100.13 srcport=54512 srccountry="Reserved" srcintf="port10" srcintfrole="undefined" srcuuid="6ce0b8ca-30ae-51ea-a388-ceacbb4fb045" dstip=172.16.200.33 dstport=443 dstcountry="Reserved" dstintf="port9" dstintfrole="undefined" dstuuid="6ce0b8ca-30ae-51ea-a388-ceacbb4fb045" proto=6 httpmethod="GET" service="HTTPS" hostname="172.16.200.33" agent="curl/7.61.1" profile="header" action="passthrough" reqtype="direct" url="https://172.16.200.33/" sentbyte=724 rcvdbyte=2769 direction="outgoing" msg="URL has been visited" ratemethod="ip" cat=255 rawdata="[REQ] test_request_header:=aaaaa"

   Log 1 is for the blocked HTTP response that contains both monitored headers, test_request_header and ETag, and their values, aaaaa and 34-5b23b9d3b67f4, respectively. Log 2 is for the HTTP request passing through the FortiGate proxy that contains test_request_header and its aaaaa value in the rawdata field.

Example 2: logging the explicit web proxy forward server name

In this example, the user wants to see the name of the web proxy forward server in the traffic log when the traffic is forwarded by a web proxy forward server.

In the global web proxy settings, log-forward-server must be enabled.

The following settings are required in the firewall policy:

• set inspection-mode proxy
• set webproxy-forward-server "fgt-b"
• set logtraffic all

When a HTTP request is sent through the FortiGate proxy, the request will be forwarded by the FortiGate to the upstream proxy (fgt-b), and the forward server's name will be logged in the traffic log.

To verify the configuration:

1. Send a HTTP request from the client:

   curl -kv https://www.google.com

2. On the FortiGate, check the traffic logs:

   # execute log filter category 3
   1: date=2023-04-19 time=19:51:33 eventtime=1681959093510003961 tz="-0700" logid="0000000013" type="traffic" subtype="forward" level="notice" vd="vdom1" srcip=10.1.100.13 srcport=49762 srcintf="port10" srcintfrole="undefined" dstip=142.250.217.100 dstport=443 dstintf="port9" dstintfrole="undefined" srccountry="Reserved" dstcountry="United States" sessionid=43292 proto=6 action="client-rst" policyid=1 policytype="policy" poluuid="4d8dc396-46e3-51ea-7f3f-ee328a5bd07b" service="HTTPS" trandisp="snat" transip=172.16.200.1 transport=49762 duration=120 sentbyte=0 rcvdbyte=37729 sentpkt=0 rcvdpkt=33 appcat="unscanned" wanin=3779 wanout=682 lanin=879 lanout=36005 fwdsrv="fgt-b" utmaction="block" countssl=1 utmref=65506-14

Example 3: logging TCP connection failures

In this example, a client initiates a TCP connection to a remote network node through the FortiGate. The connection fails because the IP address or port of the remote node is unreachable. A Connection Failed message appears in the logs. In the firewall policy configuration, the inspection-mode can be set to either proxy or flow mode.

Based on the basic FortiGate configuration used in examples 1 and 2, the forward server may need to be removed from the firewall policy if the forward server's TCP IP port is actually reachable. If the forward server proxy tries to set up back-to-back TCP connections with the downstream FortiGate and the remote server as in the case of deep-inspection, then when the client tries to connect to a remote node (even if the IP address or port is unreachable), the downstream FortiGate is able to establish a TCP connection with the upstream forward server, so there will be no Connection Failed message in the downstream FortiGate's log.

Currently, the Connection Failed message in the downstream FortiGate's log is visible for the case when there is an unreachable TCP port only when explicit web proxy with a proxy policy is configured. Therefore, the following example that makes use of a firewall policy demonstrates this log message is only supported for the unreachable IP address case.

To verify the configuration:

1. Send a HTTP request from the client to an unreachable IP:

   curl -kv https://172.16.200.34

2. On the FortiGate, check the traffic logs:

   # execute log filter category 3
   1: date=2023-04-19 time=20:25:55 eventtime=1681961155100007061 tz="-0700" logid="0000000013" type="traffic" subtype="forward" level="notice" vd="vdom1" srcip=10.1.100.13 srcport=52452 srcintf="port10" srcintfrole="undefined" dstip=172.16.200.34 dstport=443 dstintf="port9" dstintfrole="undefined" srccountry="Reserved" dstcountry="Reserved" sessionid=44903 proto=6 action="server-rst" policyid=1 policytype="policy" poluuid="4d8dc396-46e3-51ea-7f3f-ee328a5bd07b" service="HTTPS" trandisp="snat" transip=172.16.200.1 transport=52452 duration=20 sentbyte=180 rcvdbyte=164 sentpkt=3 rcvdpkt=3 appcat="unscanned" wanin=0 wanout=0 lanin=0 lanout=0 crscore=5 craction=262144 crlevel="low" msg="Connection Failed"