Explicit proxy logging enhancements

This information is also available in the FortiProxy 7.6 Administration Guide: Explicit proxy logging

Explicit proxy traffic logging has been enhanced to improve troubleshooting the HTTP proxy status for each HTTP transaction with the following:

• Support monitoring HTTP header requests and responses in the http transaction log. This requires an SSL deep inspection profile to be configured and extented-log enabled in the corresponding firewall policy.

• Support logging the explicit web proxy forward server name using set log-forward-server, which is disabled by default.

  config web-proxy global
      set log-forward-server {enable | disable}
  end

• Support logging TCP connection failures in the traffic log when a client initiates a TCP connection to a remote host through the FortiProxy, and the remote host is unreachable.

Basic configuration

The following FortiProxy configuration is used in the three explicit proxy traffic logging use cases in this topic.

To configure the FortiProxy:

1. Configure the web proxy profile:

   config web-proxy profile
       edit "header"
           config headers
               edit 1
                   set name "test_request_header"
                   set action monitor-request
               next
               edit 2
                   set name "ETag"
                   set action monitor-response
               next
           end
       next
   end

2. Enable forward server name logging in traffic:

   config web-proxy global
       set proxy-fqdn "100D.qa"
       set log-forward-server enable
   end
   

3. Configure the firewall policy:

   config firewall policy
    edit 1
   	set srcintf "port10"
   	set dstintf "port9"
   	set action accept
   	set srcaddr "all"
   	set dstaddr "all"
   	set schedule "always"
   	set service "ALL"
   	set utm-status enable
   	set webproxy-profile "header"
   	set webproxy-forward-server "227"
   	set ssl-ssh-profile "deep-inspection"
   	set logtraffic all
   	set log-http-transaction enable
   	set extended-log enable
    next
   end

A firewall policy is used in this basic configuration example and the specific examples that follow. This feature also works for the explicit web proxy or transparent web proxy with proxy policies, and the configurations are similar: Example 1: apply the web-proxy profile to the proxy policy. Example 2: apply the webproxy-forward-server.

Example 1: monitoring HTTP header requests

In this example, the user wants to monitor some HTTP headers in HTTP messages forwarded through a FortiProxy proxy (either transparent or explicit proxy. When the monitored headers are detected, they will be logged in the http transaction log.

In the web proxy profile configuration, the following HTTP headers are monitored:

• test_request_header: this is a user-customized HTTP header.
• ETag: this is a HTTP header returned by the web server's 200 OK response.

The monitored headers in the web proxy profile will be logged when the HTTP response received by the FortiProxy. The extended-log setting must be enabled.

The following settings are required in the firewall policy:

• set webproxy-profile "header"
• set ssl-ssh-profile "deep-inspection"
• set extended-log enable

To verify the configuration:

1. Send a HTTP request from the client:

   curl -kv https://172.16.200.33 -H "test_request_header: aaaaa"

   This command sends a HTTP request with the header test_request_header: aaaaa through the FortiProxy.

2. On the FortiProxy, check the http-transaction logs:

   2: date=2024-09-26 time=16:44:02 eventtime=1727394241416669139 tz="-0700" logid="0010000099" type="traffic" subtype="http-transaction" level="notice" vd="root" srcip=10.45.1.41 dstip=172.18.20.226 clientip=10.45.1.41 scheme="https" srcport=36552 dstport=443 hostname="172.18.20.226" url="https://172.18.20.226/files/card.docx" prefetch=0 policyid=3 sessionid=1853239712 transid=16777218 reqlength=134 resplength=1736 rcvdbyte=3881 sentbyte=753 resptype="normal" httpmethod="GET" agent="curl/7.76.1" statuscode="200" rawdata="Time=-1726977ms|[REQ] Host=172.18.20.226|test_request_header=aaaaa|| |[RESP] Content-Type=application/vnd.openxmlformats-officedocument.wordprocessingml.document|ETag=\\\"595-5fca03fb77091\\\"" reqtime=1727394241 resptime=1727394241 respfinishtime=1727394241 duration=416 appcat="unscanned" utmaction="allow" countweb=1 utmref=65533-104484

   This log is for the HTTP reqest and response that contain both monitored headers, test_request_header and ETag, and their values, aaaaa and 595-5fca03fb77091, respectively.

Example 2: logging the explicit web proxy forward server name

In this example, the user wants to see the name of the web proxy forward server in the traffic log when the traffic is forwarded by a web proxy forward server.

In the global web proxy settings, log-forward-server must be enabled.

The following settings are required in the firewall policy:

• set webproxy-forward-server
• set logtraffic all

When a HTTP request is sent through the FortiProxy proxy, the request will be forwarded by the FortiProxy to the upstream proxy, and the forward server's name will be logged in the traffic log.

To verify the configuration:

1. Send a HTTP request from the client:

   curl -kv https://www.google.com

2. On the FortiProxy, check the traffic logs:

   # execute log filter category 31: date=2024-09-25 time=15:16:55 eventtime=1727302615474066335 tz="-0700" logid="0000000010" type="traffic" subtype="forward" level="notice" vd="root" srcip=10.40.1.1 srcport=52636 srcintf="port1" srcintfrole="undefined" dstcountry="United States" srccountry="Reserved" dstip=34.107.221.82 dstport=80 dstintf="port1" dstintfrole="undefined" sessionid=1533148235 service="HTTP" fwdsrv="227" proxyapptype="web-proxy" proto=6 action="accept" policyid=1 policytype="proxy-policy" poluuid="1a61a9c0-79e6-51ef-a968-b5799a9ec837" trandisp="snat+dnat" tranip=10.40.1.227 tranport=8080 transip=0.0.0.0 transport=0 clientip=10.40.1.1 duration=184 wanin=1192 rcvdbyte=1192 wanout=1240 lanin=1336 sentbyte=1336 lanout=1192 appcat="unscanned" utmaction="allow" countweb=4 utmref=65534-33278

Example 3: logging TCP connection failures

In this example, a client initiates a TCP connection to a remote network node through the FortiProxy. The connection fails because the IP address or port of the remote node is unreachable. A Connection Failed message appears in the logs.

Based on the basic FortiProxy configuration used in examples 1 and 2, the forward server may need to be removed from the policy if the forward server's TCP IP port is actually reachable. If the forward server proxy tries to set up back-to-back TCP connections with the downstream FortiProxy and the remote server as in the case of deep-inspection, then when the client tries to connect to a remote node (even if the IP address or port is unreachable), the downstream FortiProxy is able to establish a TCP connection with the upstream forward server, so there will be no Connection Failed message in the downstream FortiProxy's log.

Currently, the Connection Failed message in the downstream FortiProxy's log is visible for the case when there is an unreachable TCP port only when explicit web proxy with a proxy policy is configured. Therefore, the following example that makes use of a firewall policy demonstrates this log message is only supported for the unreachable IP address case.

To verify the configuration:

1. Send a HTTP request from the client to an unreachable IP:

   curl -kv https://172.16.200.34

2. On the FortiProxy, check the traffic logs:

   # execute log filter category 3
   5: date=2024-09-26 time=17:42:54 eventtime=1727397774498700894 tz="-0700" logid="0000000010" type="traffic" subtype="forward" level="notice" vd="root" srcip=10.40.1.226 srcport=45020 srcintf="port1" srcintfrole="undefined" dstcountry="Reserved" srccountry="Reserved" dstip=172.18.20.23 dstport=80 dstintf="port1" dstintfrole="undefined" sessionid=1853239900 service="HTTP" proxyapptype="web-proxy" proto=6 action="timeout" policyid=1 policytype="proxy-policy" poluuid="1a61a9c0-79e6-51ef-a968-b5799a9ec837" trandisp="snat" transip=10.40.1.229 transport=50970 clientip=10.40.1.226 duration=3 wanin=0 rcvdbyte=0 wanout=0 lanin=125 sentbyte=125 lanout=30331 appcat="unscanned" utmaction="allow" msg="Connection Failed" crscore=5 craction=262144 crlevel="low"