Fortinet black logo

Handbook

Configuring a Bot Detection policy

Configuring a Bot Detection policy

Bot detection policies use signatures and source behavior tracking to detect client traffic likely to be generated by robots instead of genuine clients. Some bots, such as search engine crawlers, are "good bots" that perform search indexing tasks that can result in more legitimate users being directed to your site. You enable a whitelist to permit those. "Bad bots" are known to send traffic that has an negative impact on site availability and integrity, such as DDoS attacks or content scrapping. You want to block these.

To get started, you can use predefined whitelists (known good bots) and blacklists (known bad bots). You can also specify a rate limit threshold of HTTP requests/second for sources not matched to either whitelist or blacklist. The rate limit threshold can be useful in detecting "unknown bots".

In the event of false positives, you can use the user-specified whitelist table to fine-tune detection.

Before you begin:

  • You must configure the connection to FortiGuard so the system can receive periodic WAF Signature Database updates, including "good bot" and "bad bot" signatures and lists. See Configuring FortiGuard service settings.
  • You must have Read-Write permission for Security settings.

After you have configured Bot Detection policies, you can select them in WAF profiles.

To configure a Bot Detection policy:
  1. Go to Web Application Firewall > Common Attacks Detection.
  2. Click the Bot Detection tab.
  3. Click Create New to display the configuration editor.
  4. Complete the configuration as described in Bot Detection configuration.
  5. Save the configuration.

Bot Detection configuration

Settings Guidelines

Name

Configuration name. Valid characters are A-Z, a-z, 0-9, _, and -. No spaces.

After you initially save the configuration, you cannot edit the name.

Status Enable/disable Bot detection.
Search Engine Bypass Enable/disable the predefined search engine spider whitelist. The list is included in WAF signature updates from FortiGuard.

Search Engine List

Set list of search engines. Default value is all search engines.

Bad Robot Status Enable/disable the predefined bad robot blacklist. The list is included in WAF signature updates from FortiGuard.

HTTP Request Rate

Specify a threshold (HTTP requests/second/source) to trigger the action. Bots send HTTP request traffic at extraordinarily high rates. The source is tracked by source IP address and User-Agent.

The default is 0 (off). The valid range is 0-100,000,000 requests per second.

Action

  • Alert—Allow the traffic and log the event.
  • Deny—Drop the traffic, send a 403 Forbidden to the client, and log the event.

The default is alert.

Severity

  • High—Log as high severity events.
  • Medium—Log as a medium severity events.
  • Low—Log as low severity events.

The default is low.

Block Period

The default is 3600 seconds. The valid range is 1-3600.

The maximum size of the block IP address table is 100,000 entries. If the table is full, the earliest entry will be deleted.

Whitelist

IPv4/Netmask Matching subnet (CIDR format).
URL Pattern Matching string. Regular expressions are supported.
URL Parameter Name Matching string. Regular expressions are supported.
Cookie Name Matching string. Regular expressions are supported.
User Agent Matching string. Regular expressions are supported.

Configuring a Bot Detection policy

Bot detection policies use signatures and source behavior tracking to detect client traffic likely to be generated by robots instead of genuine clients. Some bots, such as search engine crawlers, are "good bots" that perform search indexing tasks that can result in more legitimate users being directed to your site. You enable a whitelist to permit those. "Bad bots" are known to send traffic that has an negative impact on site availability and integrity, such as DDoS attacks or content scrapping. You want to block these.

To get started, you can use predefined whitelists (known good bots) and blacklists (known bad bots). You can also specify a rate limit threshold of HTTP requests/second for sources not matched to either whitelist or blacklist. The rate limit threshold can be useful in detecting "unknown bots".

In the event of false positives, you can use the user-specified whitelist table to fine-tune detection.

Before you begin:

  • You must configure the connection to FortiGuard so the system can receive periodic WAF Signature Database updates, including "good bot" and "bad bot" signatures and lists. See Configuring FortiGuard service settings.
  • You must have Read-Write permission for Security settings.

After you have configured Bot Detection policies, you can select them in WAF profiles.

To configure a Bot Detection policy:
  1. Go to Web Application Firewall > Common Attacks Detection.
  2. Click the Bot Detection tab.
  3. Click Create New to display the configuration editor.
  4. Complete the configuration as described in Bot Detection configuration.
  5. Save the configuration.

Bot Detection configuration

Settings Guidelines

Name

Configuration name. Valid characters are A-Z, a-z, 0-9, _, and -. No spaces.

After you initially save the configuration, you cannot edit the name.

Status Enable/disable Bot detection.
Search Engine Bypass Enable/disable the predefined search engine spider whitelist. The list is included in WAF signature updates from FortiGuard.

Search Engine List

Set list of search engines. Default value is all search engines.

Bad Robot Status Enable/disable the predefined bad robot blacklist. The list is included in WAF signature updates from FortiGuard.

HTTP Request Rate

Specify a threshold (HTTP requests/second/source) to trigger the action. Bots send HTTP request traffic at extraordinarily high rates. The source is tracked by source IP address and User-Agent.

The default is 0 (off). The valid range is 0-100,000,000 requests per second.

Action

  • Alert—Allow the traffic and log the event.
  • Deny—Drop the traffic, send a 403 Forbidden to the client, and log the event.

The default is alert.

Severity

  • High—Log as high severity events.
  • Medium—Log as a medium severity events.
  • Low—Log as low severity events.

The default is low.

Block Period

The default is 3600 seconds. The valid range is 1-3600.

The maximum size of the block IP address table is 100,000 entries. If the table is full, the earliest entry will be deleted.

Whitelist

IPv4/Netmask Matching subnet (CIDR format).
URL Pattern Matching string. Regular expressions are supported.
URL Parameter Name Matching string. Regular expressions are supported.
Cookie Name Matching string. Regular expressions are supported.
User Agent Matching string. Regular expressions are supported.