Application Delivery - URL Rewriting

Why does URL rewriting not work?

If FortiWeb is not rewriting URLs as expected, complete the following troubleshooting steps:

1. Ensure the value of Action Type is correct.

   Request Action rewrites HTTP requests from clients, and Response Action rewrites responses to clients from the web server.

2. Ensure that you have added items to the URL Rewriting Condition Table.
3. If one of your conditions uses a regular expression, ensure that the expression is valid.

   • Click the >> (double arrow) button beside the Regular Expression field to test the value.

   • For an online guide for regular expressions, go to:

     HTTP://www.regular-expressions.info/reference.html

   • For an online library of regular expressions, go to:

     HTTP://regexlib.com

   • If the page is compressed, ensure that you have configured a decompression policy.

4. Check if the webpage size is larger than the Maximum Body Cache Size.

   URL body rewriting does not work when the page is larger than the cache buffer size. The default size is 64KB.

   Go to System > Config > Advanced and adjust the value of Maximum Body Cache Size。

   To adjust the buffer using the CLI, use a command like the following example:

   config global

   config sys advanced

   set max-cache-size 1024

   end

   end

5. For a Response rewrite rule and the action is “Rewrite HTTP Body”, ensure there is a “Content-Type” header in the response from the backend server, and the Content-Type (also called Internet or MIME file types) must be supported by FortiWeb.

   FortiWeb supports the following Content-Type values only:

   • text/html

   • text/plain

   • text/javascript

   • application/xml

   • text/xml

   • application/javascript

   • application/soap+xml

   • application/x-javascript

   • application/json

   • application/rss+xml

   “Content-Type” is not a must for other types of rewrite rules including Request rewrite rules and Rewrite HTTP Header rules.

6. Specifically, if the option Content Type Filter is enabled in the match condition, only the types selected in Content Type Set will be matched and rewritten. Webpages with other unselected types will match the rewrite rule.
7. Enable diagnose logs for further analysis:

FWB # diagnose debug application url-rewrite 7

FWB # diagnose debug enable

Diagnose logs will show HTTP request & response details, url-rewrite rule & policy matching conditions (match or not), etc.

Example: url-rewrite-policy “redirect_policy_01” contains two rules.

• “redirect_rule_01” is a request redirect action that aims to remove the port 8443

• “url-rewrite-rule-ResponseAction-RewriteBody” is a response rewrite body action that targets to replace “It works!” with “Hey, It works now!!”

For request direction, all conditions are matched so redirect 301 is responded to the client.

[url rewrite][INFO](./waf_module/url_rewrite.c:2543): CLIENT -> SERVER.

[url rewrite][INFO](./waf_module/url_rewrite.c:2483): Request host: [portal.testdomain.com:8443].

[url rewrite][INFO](./waf_module/url_rewrite.c:2487): Request url: [/index.html].

[url rewrite][INFO](./waf_module/url_rewrite.c:1619): url rewrite policy name: [redirect_policy_01].

[url rewrite][INFO](./waf_module/url_rewrite.c:515): url rewrite rule name: [redirect_rule_01] ,check rule conds.

[url rewrite][INFO](./waf_module/url_rewrite.c:523): the matching host :portal.testdomain.com:8443

[url rewrite][INFO](./waf_module/url_rewrite.c:528): the matching url :/index.html

[url rewrite][INFO](./waf_module/url_rewrite.c:651): all conditons matched!

[url rewrite][INFO](./waf_module/url_rewrite.c:1658): matched...

[url rewrite][INFO](./waf_module/url_rewrite.c:1660): the pcre capture $0 is :

… …

[url rewrite][INFO](./waf_module/url_rewrite.c:1572): the action is :8

[url rewrite][INFO](./waf_module/url_rewrite.c:1342): make redirect response.

[url rewrite][INFO](./waf_module/url_rewrite.c:1351): the new location is : HTTP://portal.testdomain.com

[url rewrite][INFO](./waf_module/url_rewrite.c:2565): The response custom redirect 301.

[url rewrite][INFO](./waf_module/url_rewrite.c:2543): CLIENT -> SERVER.

[url rewrite][INFO](./waf_module/url_rewrite.c:2483): Request host: [portal.testdomain.com].

[url rewrite][INFO](./waf_module/url_rewrite.c:2487): Request url: [/].

[url rewrite][INFO](./waf_module/url_rewrite.c:1619): url rewrite policy name: [redirect_policy_01].

[url rewrite][INFO](./waf_module/url_rewrite.c:515): url rewrite rule name: [redirect_rule_01] ,check rule conds.

[url rewrite][INFO](./waf_module/url_rewrite.c:523): the matching host :portal.testdomain.com

[url rewrite][INFO](./waf_module/url_rewrite.c:643): not matched,and no invert,not matched.

For response direction, all condition is also matched so body-rewrite is also performed.

[url rewrite][INFO](./waf_module/url_rewrite.c:2607): SERVER -> CLIENT.

[url rewrite][INFO](./waf_module/url_rewrite.c:1920): response rewrite check.

[url rewrite][INFO](./waf_module/url_rewrite.c:1924): url rewrite policy name: [redirect_policy_01].

[url rewrite][INFO](./waf_module/url_rewrite.c:1763): HTTP body cache (3477) finish.

[url rewrite][DEG](./waf_module/url_rewrite.c:1814): response raw body: [HTTP/1.1 200 OK

Date: Thu, 26 May 2022 21:03:08 GMT

Server: Apache/2.4.41 (Ubuntu)

Last-Modified: Thu, 07 Oct 2021 17:55:36 GMT

ETag: "2aa6-5cdc6f84d8056-gzip"

Accept-Ranges: bytes

Vary: Accept-Encoding

Content-Encoding: gzip

Content-Length: 3138

Keep-Alive: timeout=5, max=100

Connection: Keep-Alive

Content-Type: text/html

… …

[url rewrite][INFO](./waf_module/url_rewrite.c:846): _body_rewrite_check_rule_conds...

[url rewrite][INFO](./waf_module/url_rewrite.c:912): content_type is 1

[url rewrite][INFO](./waf_module/url_rewrite.c:913): content_type_set is 65535

[url rewrite][INFO](./waf_module/url_rewrite.c:962): match ovector[0]; 385 ovector[1]: 433

[url rewrite][INFO](./waf_module/url_rewrite.c:1006): all body-rewrite conditons matched!

How will multiple rules in one rewrite policy be matched?

If multiple rules are configured in one URL rewrite policy, then these rules will be matched in order. That is to say, when the traffic matches the first rule and is processed, the following rules will be skipped and not take effect any more.

This is also one of the reasons that a rewrite rule does not take effect.

You can move a rewrite rule to adjust the order of entries via CLI or GUI as below:

How will multiple match-conditions in one rewrite rule be matched?

The relationship between multiple match-conditions are AND. So only if all conditions are matched, the request or response will be rewritten.

How will FortiWeb handle duplicate headers that are matched by rewrite rules?

From HTTP RFC7230, multiple headers with the same name (e.g. Set-Cookie, www-authenticate) are acceptable and may be received by FortiWeb.

FortiWeb will handle such situations as below:

• If a Field Name configured in HTTP Header Removal matches multiple headers, all these headers will be removed;

• For Replacement URL, Referer and Location, in theory only the first header will be replaced. However, in practice duplicate these header fields can be hardly duplicated appearing in the same HTTP packet.

Why sometimes URL rewriting rules cause Loop in browser visiting?

It’s a typical issue that sometimes after rewriting rules are added, you may observe loop failures when visiting a server-policy on browsers. However, these issues are usually caused by configuration mistakes.

Below is an example of such misconfiguration failures:

The request action is Redirect 301. The match condition object is “HTTP Host” with portal.testdomain.com, and the replacement Location is configured as “/test.html”.

The user intended to redirect the visit to the default webpage of portal.testdomain.com to portal.testdomain.com/test.html, but with this configuration, the browser will visit “HTTP://portal.testdomain.com/test.html” after receiving the 301 response, because the Location header is just “/test.html” rather than a full URI. However this new request will match the rewrite rule again and trigger another 301, thus causing an unexpected loop failure.

To resolve this issue, you can add an extra condition rule as below, then the visits to “HTTP://portal.testdomain.com/index.html” will be successfully redirected to “HTTP://portal.testdomain.com/test.html”, and no loop occurs again.

The tip here is that Location needs to be a full URI, otherwise the browser will reuse the original Host with the relative URI specified by Location.

For example, if you want to redirect a URL to HTTPs://www.google.com, then you need to configure the Location as “HTTPs://www.google.com”, not just “www.google.com”, otherwise the browser will visit “HTTP://portal.testdomain.com/www.google.com” after it received 301 redirect.