Fortinet black logo

Handbook

Static URL filter

6.0.0
Copy Link
Copy Doc ID 4afb0436-a998-11e9-81a4-00505692583a:164551
Download PDF

Static URL filter

You can allow or block access to specific URLs by adding them to the Static URL Filter list. The filter allows you to block, allow, or monitor URLs by using patterns containing text, regular expressions, or wildcard characters. The FortiGate unit allows or blocks web pages matching any specified URLs or patterns and displays a replacement message instead.

note icon

URL blocking does not block access to other services that users can access with a web browser. For example, URL blocking does not block access to ftp:// ftp.example.com. Instead, use firewall policies to deny ftp connections.

When adding a URL to the URL filter list, follow these rules:

  • Type a top-level URL or IP address to control access to all pages on a web site. For example, www.example.com or 192.168.144.155 controls access to all pages at this web site.
  • Enter a top-level URL followed by the path and file name to control access to a single page on a web site. For example, www.example.com/news.html or 192.168.144.155/news.html controls access to the news page on this web site.
  • To control access to all pages with a URL that ends with example.com, add example.com to the filter list. For example, adding example.com controls access to www.example.com, mail.example.com, www.finance.example.com, and so on.
  • Control access to all URLs that match patterns using text and regular expressions (or wildcard characters). For example, example.* matches example.com, example.org, example.net and so on.

caution icon

URLs with an action set to exempt are not scanned for viruses. If users on the network download files through the FortiGate unit from a trusted web site, add the URL of this web site to the URL filter list with an action to pass it so the FortiGate unit does not virus scan files downloaded from this URL.

URL formats

How URL formats are detected when using HTTPS

Filter HTTPS traffic by entering a top level domain name, for example, www.example.com if:

  • your unit does not support SSL content scanning and inspection
  • you have selected the URL filtering option in web content profile for HTTPS content filtering mode under Protocol Recognition.

HTTPS URL filtering of encrypted sessions works by extracting the CN from the server certificate during the SSL negotiation. Since the CN only contains the domain name of the site being accessed, web filtering of encrypted HTTPS sessions can only filter by domain names.

If your unit supports SSL content scanning and inspection and if you have selected Deep Scan, you can filter HTTPS traffic in the same way as HTTP traffic.

How URL formats are detected when using HTTP

URLs with an action set to Exempt are not scanned for viruses. If users on the network download files through the unit from trusted web site, add the URL of this web site to the URL filter list with an action set to exempt so the unit does not virus scan files downloaded from this URL.

  • Type a top-level URL or IP address to control access to all pages on a web site. For example, www.example.com or 192.168.144.155 controls access to all pages at this web site.
  • Enter a top-level URL followed by the path and filename to control access to a single page on a web site. For example, www.example.com/news.html or 192.168.144.155/news.html controls the news page on this web site.
  • To control access to all pages with a URL that ends with example.com, add example.com to the filter list. For example, adding example.com controls access to www.example.com, mail.example.com, www.finance.example.com, and so on.
  • Control access to all URLs that match patterns created using text and regular expressions (or wildcard characters). For example, example.* matches example.com, example.org, example.net and so on.
  • Fortinet URL filtering supports standard regular expressions.

note icon

If virtual domains are enabled on the unit, web filtering features are configured globally. To access these features, select Global Configuration on the main menu.

URL filter actions

You can select one of four actions for how traffic will be treated as it attempts to reach a site in the list.

Block

Attempts to access any URLs matching the URL pattern are denied. The user will be presented with a replacement message.

Allow

Any attempt to access a URL that matches a URL pattern with an allow action is permitted. The traffic is passed to the remaining antivirus proxy operations, including FortiGuard Web Filter, web content filter, web script filters, and antivirus scanning.

Allow is the default action. If a URL does not appear in the URL list, it is permitted.

Monitor

Traffic to, and reply traffic from, sites matching a URL pattern with Monitor action applied will be allowed through in the same way as the Allow action. The difference with the Monitor action is that a log message will be generated each time a matching traffic session is established. The requests will also be subject to all other Security Profiles inspections that would normally be applied to the traffic.

Exempt

Exempt allows trusted traffic to bypass the antivirus and DLP proxy operations by default, but it functions slightly differently. In general, if you’re not certain that you need to use the Exempt action, use Monitor.

note icon

Using the static URL filter to exempt scanning also prevents SSL inspection.

HTTP 1.1 connections are persistent unless declared otherwise. This means the connections will remain in place until closed or the connection times out. When a client loads a web page, the client opens a connection to the web server. If the client follows a link to another page on the same site before the connection times out, the same connection is used to request and receive the page data.

When you add a URL pattern to a URL filter list and apply the Exempt action, traffic sent to and replies traffic from sites matching the URL pattern will bypass all antivirus proxy operations. The connection itself inherits the exemption. This means that all subsequent reuse of the existing connection will also bypass all antivirus proxy operations. When the connection times out, the exemption is cancelled.

For example, consider a URL filter list that includes example.com/files configured with the Exempt action. A user opens a web browser and downloads a file from the URL example.com/sample.zip. This URL does not match the URL pattern so it is scanned for viruses. The user then downloads example.com/files/beautiful.exe and since this URL does match the pattern, the connection itself inherits the exempt action. The user then downloads example.com/virus.zip. Although this URL does not match the exempt URL pattern, a previously visited URL did, and since the connection inherited the exempt action and was re-used to download a file, the file is not scanned.

If the user next goes to an entirely different server, like example.org/photos, the connection to the current server cannot be reused. A new connection to example.org is established. This connection is not exempt. Unless the user goes back to example.com before the connection to that server times out, the server will close the connection. If the user returns after the connection is closed, a new connection to example.com is created and it is not exempt until the user visits a URL that matches the URL pattern.

Web servers typically have short time-out periods. A browser will download multiple components of a web page as quickly as possible by opening multiple connections. A web page that includes three photos will load more quickly if the browser opens four connections to the server and downloads the page and the three photos at the same time. A short time-out period on connections will close the connections faster, allowing the server to avoid unnecessarily allocating resources for a long period. The HTTP session time-out is set by the server and will vary with the server software, version, and configuration.

Using the Exempt action can have unintended consequences in certain circumstances. You have a web site at example.com and since you control the site, you trust the contents and configure example.com as exempt. But example.com is hosted on a shared server with a dozen other different sites, each with a unique domain name. Because of the shared hosting, they also share the same IP address. If you visit example.com, your connection your site becomes exempt from any antivirus proxy operations. Visits to any of the 12 other sites on the same server will reuse the same connection and the data you receive is exempt from being scanned.

Use of the Exempt action is not suitable for configuration in which connections through the FortiGate unit use an external proxy. For example, you use proxy.example.net for all outgoing web access. Also, as in the first example, URL filter list that includes a URL pattern of example.com/files configured with the Exempt action. Users are protected by the antivirus protection of the FortiGate unit until a user visits a URL that matches the of example.com/files URL pattern. The pattern is configured with the Exempt action so the connection to the server inherits the exemption. With a proxy however, the connection is from the user to the proxy. Therefore, the user is entirely unprotected until the connection times out, no matter what site he visits.

Ensure you are aware of the network topology involving any URLs to which you apply the Exempt action.

Status

The Web Site Filter has the option to either enable or disable individual web sites in the list. This allows for the temporary removal of the actions against a site so that it can be later reengaged without having to rewrite the configuration.

Configuring a URL filter

Consult the Maximum Values Table for up-to-date information on the number of URL filter entries allowed for your FortiGate.

note icon

You can only set a Static URL Filter with proxy-based inspection mode and flow-based inspection mode in profile-based NGFW mode.

For this example, the URL www.example*.com will be used. You configure the list by adding one or more URLs to it.

To add a URL to a URL filter
  1. Go to Security Profiles > Web Filter.
  2. Create a new web filter or select a one to edit.
  3. Expand Static URL Filter, enable URL Filter, and select Create.
  4. Enter the URL, without the “http”, for example: www.example*.com.
  5. Select a Type: Simple , Reg. Expression, or Wildcard. In this example, select Wildcard.
  6. Select the Action to take against matching URLs: Exempt, Block, Allow, or Monitor.
  7. Confirm that Status is enabled.
  8. Select OK.

'Simple' Filter type

If you select the Simple filter type for a URL filter, the syntax is performing an exact match. Note, however, that the domain and path are separate entities in HTTP despite the fact that a user types them as a single entity and, in the case of 'simple', the rules for each part (domain and path) are different.

The 'domain' part

For the domain part, the goal of the 'simple' format is to make it easy to block a domain and all its subdomains, such that the admin only has to type “address.xy" to block "address.xy", "www.address.xy", "talk.address.xy", etc. but not block “youraddress.xy" or "www.youraddress.xy" which are different domains from "address.xy”.

Also, the actual domain does not include http:// or https:// so this should not be entered or the URL filter will try to match a domain starting with http. For this reason, when you enter http:// in the URL filter via the GUI, it is automatically removed.

caution icon

A trailing ‘/‘ with the domain is not needed. The GUI URL filter will automatically trim this, but when using the API to provide the per-user BWL it will not! Please take this into account. Better not to use it as it might give unexpected results.

The 'path' part

For the path part, an exact match takes place. For example:

www.address.xy/news

blocks anything that starts with that exact path. So this matches:

www.address.xy/newsieswww.address.xy/newsforyou
www.address.xy/news/coetc.

Also:

www.address.xy/new

likewise blocks the same as above but includes:

/newt/newpetc.

which is a much broader filter, matching:

www.address.xy/newstand/cowww.address.xy/news/coetc.

In other words, the more you specify of the path, the more strictly it will match.

caution icon

Here as well a trailing ‘/‘ with the URL path is not needed, the GUI URL filter will automatically trim this, but when using the API to provide the per-user BWL it will not. Please take this into account. Better not to use it as it might give unexpected results.

Referrer URL

A new variable has been added to the Static URL Filter: referrer-host. If a referrer is specified, the hostname in the referrer field of the HTTP require will be compared for any entry that contains the matching URL. If the referrer matches, then the specified action will be performed by proxy.

Configuring in the GUI

The configuration can be done in the GUI but only if advance web filtering features have been enabled by entering the following commands in the CLI:

config system global

set gui-webfilter-advanced enable

end

After this command is used, a new column will be created in Security Profiles > Web Filter to set the referrer.

Configuring in the CLI

When specifying the URL filter, it needs to be identified by its ID. The URLs are listed under each entry.

To find the ID number:

config webfilter urlfilter

edit ?

A list of the current URL filters will be listed with their ID numbers in the left column.

The syntax in the CLI for configuring an entry is:

config webfilter urlfilter

edit <ID>

config entries

edit 1

set url <url>

set referrer-host <url>

set type {simple | regex | wildcard}

set action {block | allow | monitor | exempt}

set status {enable | disable}

end

end

end

Static URL filter

You can allow or block access to specific URLs by adding them to the Static URL Filter list. The filter allows you to block, allow, or monitor URLs by using patterns containing text, regular expressions, or wildcard characters. The FortiGate unit allows or blocks web pages matching any specified URLs or patterns and displays a replacement message instead.

note icon

URL blocking does not block access to other services that users can access with a web browser. For example, URL blocking does not block access to ftp:// ftp.example.com. Instead, use firewall policies to deny ftp connections.

When adding a URL to the URL filter list, follow these rules:

  • Type a top-level URL or IP address to control access to all pages on a web site. For example, www.example.com or 192.168.144.155 controls access to all pages at this web site.
  • Enter a top-level URL followed by the path and file name to control access to a single page on a web site. For example, www.example.com/news.html or 192.168.144.155/news.html controls access to the news page on this web site.
  • To control access to all pages with a URL that ends with example.com, add example.com to the filter list. For example, adding example.com controls access to www.example.com, mail.example.com, www.finance.example.com, and so on.
  • Control access to all URLs that match patterns using text and regular expressions (or wildcard characters). For example, example.* matches example.com, example.org, example.net and so on.

caution icon

URLs with an action set to exempt are not scanned for viruses. If users on the network download files through the FortiGate unit from a trusted web site, add the URL of this web site to the URL filter list with an action to pass it so the FortiGate unit does not virus scan files downloaded from this URL.

URL formats

How URL formats are detected when using HTTPS

Filter HTTPS traffic by entering a top level domain name, for example, www.example.com if:

  • your unit does not support SSL content scanning and inspection
  • you have selected the URL filtering option in web content profile for HTTPS content filtering mode under Protocol Recognition.

HTTPS URL filtering of encrypted sessions works by extracting the CN from the server certificate during the SSL negotiation. Since the CN only contains the domain name of the site being accessed, web filtering of encrypted HTTPS sessions can only filter by domain names.

If your unit supports SSL content scanning and inspection and if you have selected Deep Scan, you can filter HTTPS traffic in the same way as HTTP traffic.

How URL formats are detected when using HTTP

URLs with an action set to Exempt are not scanned for viruses. If users on the network download files through the unit from trusted web site, add the URL of this web site to the URL filter list with an action set to exempt so the unit does not virus scan files downloaded from this URL.

  • Type a top-level URL or IP address to control access to all pages on a web site. For example, www.example.com or 192.168.144.155 controls access to all pages at this web site.
  • Enter a top-level URL followed by the path and filename to control access to a single page on a web site. For example, www.example.com/news.html or 192.168.144.155/news.html controls the news page on this web site.
  • To control access to all pages with a URL that ends with example.com, add example.com to the filter list. For example, adding example.com controls access to www.example.com, mail.example.com, www.finance.example.com, and so on.
  • Control access to all URLs that match patterns created using text and regular expressions (or wildcard characters). For example, example.* matches example.com, example.org, example.net and so on.
  • Fortinet URL filtering supports standard regular expressions.

note icon

If virtual domains are enabled on the unit, web filtering features are configured globally. To access these features, select Global Configuration on the main menu.

URL filter actions

You can select one of four actions for how traffic will be treated as it attempts to reach a site in the list.

Block

Attempts to access any URLs matching the URL pattern are denied. The user will be presented with a replacement message.

Allow

Any attempt to access a URL that matches a URL pattern with an allow action is permitted. The traffic is passed to the remaining antivirus proxy operations, including FortiGuard Web Filter, web content filter, web script filters, and antivirus scanning.

Allow is the default action. If a URL does not appear in the URL list, it is permitted.

Monitor

Traffic to, and reply traffic from, sites matching a URL pattern with Monitor action applied will be allowed through in the same way as the Allow action. The difference with the Monitor action is that a log message will be generated each time a matching traffic session is established. The requests will also be subject to all other Security Profiles inspections that would normally be applied to the traffic.

Exempt

Exempt allows trusted traffic to bypass the antivirus and DLP proxy operations by default, but it functions slightly differently. In general, if you’re not certain that you need to use the Exempt action, use Monitor.

note icon

Using the static URL filter to exempt scanning also prevents SSL inspection.

HTTP 1.1 connections are persistent unless declared otherwise. This means the connections will remain in place until closed or the connection times out. When a client loads a web page, the client opens a connection to the web server. If the client follows a link to another page on the same site before the connection times out, the same connection is used to request and receive the page data.

When you add a URL pattern to a URL filter list and apply the Exempt action, traffic sent to and replies traffic from sites matching the URL pattern will bypass all antivirus proxy operations. The connection itself inherits the exemption. This means that all subsequent reuse of the existing connection will also bypass all antivirus proxy operations. When the connection times out, the exemption is cancelled.

For example, consider a URL filter list that includes example.com/files configured with the Exempt action. A user opens a web browser and downloads a file from the URL example.com/sample.zip. This URL does not match the URL pattern so it is scanned for viruses. The user then downloads example.com/files/beautiful.exe and since this URL does match the pattern, the connection itself inherits the exempt action. The user then downloads example.com/virus.zip. Although this URL does not match the exempt URL pattern, a previously visited URL did, and since the connection inherited the exempt action and was re-used to download a file, the file is not scanned.

If the user next goes to an entirely different server, like example.org/photos, the connection to the current server cannot be reused. A new connection to example.org is established. This connection is not exempt. Unless the user goes back to example.com before the connection to that server times out, the server will close the connection. If the user returns after the connection is closed, a new connection to example.com is created and it is not exempt until the user visits a URL that matches the URL pattern.

Web servers typically have short time-out periods. A browser will download multiple components of a web page as quickly as possible by opening multiple connections. A web page that includes three photos will load more quickly if the browser opens four connections to the server and downloads the page and the three photos at the same time. A short time-out period on connections will close the connections faster, allowing the server to avoid unnecessarily allocating resources for a long period. The HTTP session time-out is set by the server and will vary with the server software, version, and configuration.

Using the Exempt action can have unintended consequences in certain circumstances. You have a web site at example.com and since you control the site, you trust the contents and configure example.com as exempt. But example.com is hosted on a shared server with a dozen other different sites, each with a unique domain name. Because of the shared hosting, they also share the same IP address. If you visit example.com, your connection your site becomes exempt from any antivirus proxy operations. Visits to any of the 12 other sites on the same server will reuse the same connection and the data you receive is exempt from being scanned.

Use of the Exempt action is not suitable for configuration in which connections through the FortiGate unit use an external proxy. For example, you use proxy.example.net for all outgoing web access. Also, as in the first example, URL filter list that includes a URL pattern of example.com/files configured with the Exempt action. Users are protected by the antivirus protection of the FortiGate unit until a user visits a URL that matches the of example.com/files URL pattern. The pattern is configured with the Exempt action so the connection to the server inherits the exemption. With a proxy however, the connection is from the user to the proxy. Therefore, the user is entirely unprotected until the connection times out, no matter what site he visits.

Ensure you are aware of the network topology involving any URLs to which you apply the Exempt action.

Status

The Web Site Filter has the option to either enable or disable individual web sites in the list. This allows for the temporary removal of the actions against a site so that it can be later reengaged without having to rewrite the configuration.

Configuring a URL filter

Consult the Maximum Values Table for up-to-date information on the number of URL filter entries allowed for your FortiGate.

note icon

You can only set a Static URL Filter with proxy-based inspection mode and flow-based inspection mode in profile-based NGFW mode.

For this example, the URL www.example*.com will be used. You configure the list by adding one or more URLs to it.

To add a URL to a URL filter
  1. Go to Security Profiles > Web Filter.
  2. Create a new web filter or select a one to edit.
  3. Expand Static URL Filter, enable URL Filter, and select Create.
  4. Enter the URL, without the “http”, for example: www.example*.com.
  5. Select a Type: Simple , Reg. Expression, or Wildcard. In this example, select Wildcard.
  6. Select the Action to take against matching URLs: Exempt, Block, Allow, or Monitor.
  7. Confirm that Status is enabled.
  8. Select OK.

'Simple' Filter type

If you select the Simple filter type for a URL filter, the syntax is performing an exact match. Note, however, that the domain and path are separate entities in HTTP despite the fact that a user types them as a single entity and, in the case of 'simple', the rules for each part (domain and path) are different.

The 'domain' part

For the domain part, the goal of the 'simple' format is to make it easy to block a domain and all its subdomains, such that the admin only has to type “address.xy" to block "address.xy", "www.address.xy", "talk.address.xy", etc. but not block “youraddress.xy" or "www.youraddress.xy" which are different domains from "address.xy”.

Also, the actual domain does not include http:// or https:// so this should not be entered or the URL filter will try to match a domain starting with http. For this reason, when you enter http:// in the URL filter via the GUI, it is automatically removed.

caution icon

A trailing ‘/‘ with the domain is not needed. The GUI URL filter will automatically trim this, but when using the API to provide the per-user BWL it will not! Please take this into account. Better not to use it as it might give unexpected results.

The 'path' part

For the path part, an exact match takes place. For example:

www.address.xy/news

blocks anything that starts with that exact path. So this matches:

www.address.xy/newsieswww.address.xy/newsforyou
www.address.xy/news/coetc.

Also:

www.address.xy/new

likewise blocks the same as above but includes:

/newt/newpetc.

which is a much broader filter, matching:

www.address.xy/newstand/cowww.address.xy/news/coetc.

In other words, the more you specify of the path, the more strictly it will match.

caution icon

Here as well a trailing ‘/‘ with the URL path is not needed, the GUI URL filter will automatically trim this, but when using the API to provide the per-user BWL it will not. Please take this into account. Better not to use it as it might give unexpected results.

Referrer URL

A new variable has been added to the Static URL Filter: referrer-host. If a referrer is specified, the hostname in the referrer field of the HTTP require will be compared for any entry that contains the matching URL. If the referrer matches, then the specified action will be performed by proxy.

Configuring in the GUI

The configuration can be done in the GUI but only if advance web filtering features have been enabled by entering the following commands in the CLI:

config system global

set gui-webfilter-advanced enable

end

After this command is used, a new column will be created in Security Profiles > Web Filter to set the referrer.

Configuring in the CLI

When specifying the URL filter, it needs to be identified by its ID. The URLs are listed under each entry.

To find the ID number:

config webfilter urlfilter

edit ?

A list of the current URL filters will be listed with their ID numbers in the left column.

The syntax in the CLI for configuring an entry is:

config webfilter urlfilter

edit <ID>

config entries

edit 1

set url <url>

set referrer-host <url>

set type {simple | regex | wildcard}

set action {block | allow | monitor | exempt}

set status {enable | disable}

end

end

end