Configuring a Bot Detection policy

Bot detection policies use signatures and source behavior tracking to detect client traffic likely to be generated by robots instead of genuine clients. Some bots, such as search engine crawlers, are "good bots" that perform search indexing tasks that can result in more legitimate users being directed to your site. You enable a allowlist to permit those. "Bad bots" are known to send traffic that has an negative impact on site availability and integrity, such as DDoS attacks or content scrapping. You want to block these.

To get started, you can use predefined allowlists (known good bots) and blocklists (known bad bots). You can also specify a rate limit threshold of HTTP requests/second for sources not matched to either allowlist or blocklist. The rate limit threshold can be useful in detecting "unknown bots".

In the event of false positives, you can use the user-specified allowlist table to fine-tune detection.

Before you begin:

• You must configure the connection to FortiGuard so the system can receive periodic WAF Signature Database updates, including "good bot" and "bad bot" signatures and lists. See Configuring FortiGuard service settings.
• You must have Read-Write permission for Security settings.

After you have configured Bot Detection policies, you can select them in WAF profiles.

To configure a Bot Detection policy:

1. Go to Web Application Firewall > Access Protection.
2. Click the Bot Detection tab.
3. Click Create New to display the configuration editor.
4. Complete the configuration as described in Bot Detection configuration.
5. Save the configuration.

Bot Detection configuration

Settings	Guidelines
Name	Configuration name. Valid characters are A-Z, a-z, 0-9, _, and -. No spaces. After you initially save the configuration, you cannot edit the name.
Status	Enable/disable Bot detection.
Search Engine Bypass	Enable/disable the predefined search engine spider allowlist. The list is included in WAF signature updates from FortiGuard.
Search Engine List	Set list of search engines. Default value is all search engines.
Bad Robot Status	Enable/disable the predefined bad robot blocklist. The list is included in WAF signature updates from FortiGuard.
HTTP Request Rate	Specify a threshold (HTTP requests/second/source) to trigger the action. Bots send HTTP request traffic at extraordinarily high rates. The source is tracked by source IP address and User-Agent. The default is 0 (off). The valid range is 0-100,000,000 requests per second.
Action	Select the action profile that you want to apply. See Configuring WAF Action objects. The default is alert.
Severity	High—Log as high severity events. Medium—Log as a medium severity events. Low—Log as low severity events. The default is low.
Whitelist
IPv4/Netmask	Matching subnet (CIDR format).
URL Pattern	Matching string. Regular expressions are supported.
URL Parameter Name	Matching string. Regular expressions are supported.
Cookie Name	Matching string. Regular expressions are supported.
User Agent	Matching string. Regular expressions are supported.