Configuring a Bot Detection policy

Bot detection policies use signatures and source behavior tracking to detect client traffic likely to be generated by robots instead of genuine clients. Some bots, such as search engine crawlers, are "good bots" that perform search indexing tasks that can result in more legitimate users being directed to your site. You enable a whitelist to permit those. "Bad bots" are known to send traffic that has an negative impact on site availability and integrity, such as DDoS attacks or content scrapping. You want to block these.

To get started, you can use predefined whitelists (known good bots) and blacklists (known bad bots). You can also specify a rate limit threshold of HTTP requests/second for sources not matched to either whitelist or blacklist. The rate limit threshold can be useful in detecting "unknown bots".

In the event of false positives, you can use the user-specified whitelist table to fine-tune detection.

Before you begin:

• You must configure the connection to FortiGuard so the system can receive periodic WAF Signature Database updates, including "good bot" and "bad bot" signatures and lists. See Configuring FortiGuard service settings.
• You must have Read-Write permission for Security settings.

After you have configured Bot Detection policies, you can select them in WAF profiles.

To configure a Bot Detection policy:

1. Go to Web Application Firewall > Common Attacks Detection.
2. Click the Bot Detection tab.
3. Click Create New to display the configuration editor.
4. Complete the configuration as described in Bot Detection configuration.
5. Save the configuration.

Bot Detection configuration

Settings	Guidelines
Name	Configuration name. Valid characters are A-Z, a-z, 0-9, _, and -. No spaces. After you initially save the configuration, you cannot edit the name.
Status	Enable/disable Bot detection.
Search Engine Bypass	Enable/disable the predefined search engine spider whitelist. The list is included in WAF signature updates from FortiGuard.
Search Engine List	Set list of search engines. Default value is all search engines.
Bad Robot Status	Enable/disable the predefined bad robot blacklist. The list is included in WAF signature updates from FortiGuard.
HTTP Request Rate	Specify a threshold (HTTP requests/second/source) to trigger the action. Bots send HTTP request traffic at extraordinarily high rates. The source is tracked by source IP address and User-Agent. The default is 0 (off). The valid range is 0-100,000,000 requests per second.
Action	Alert—Allow the traffic and log the event. Deny—Drop the traffic, send a 403 Forbidden to the client, and log the event. The default is alert.
Severity	High—Log as high severity events. Medium—Log as a medium severity events. Low—Log as low severity events. The default is low.
Block Period	The default is 3600 seconds. The valid range is 1-3600. The maximum size of the block IP address table is 100,000 entries. If the table is full, the earliest entry will be deleted.
Whitelist
IPv4/Netmask	Matching subnet (CIDR format).
URL Pattern	Matching string. Regular expressions are supported.
URL Parameter Name	Matching string. Regular expressions are supported.
Cookie Name	Matching string. Regular expressions are supported.
User Agent	Matching string. Regular expressions are supported.