Configuring known bots
Known Bots protects your websites, mobile applications, and APIs from malicious bots such as DoS, Spam, and Crawler, etc, and known good bots such as known search engines without affecting the flow of critical traffic.
This feature identifies and manages a wide range of attacks from automated tools no matter where these applications or APIs are deployed.
Two predefined known bots rules are available here. You can also configure new known bots rules and apply the rules in a bot mitigation policy, see Configuring bot mitigation policy.
When enabled, the known bots items will skip the subsequent scans after Known Bots (See the scan sequence of Known Bots in Sequence of scans) . This feature reduces false positives and improves performance.
To configure a known bots rule
- Go to Bot Mitigation > Known Bots.
- Click Create New.
- Configure these settings.
Name
Type a name that can be referenced by other parts of the configuration.
Status
Click to enable or disable the bot check for this rule.
In each row, select the action that FortiWeb takes when it detects a violation of the rule.
Alert—Accept the request and generate an alert email and/or log message.
-
Alert & Deny—Block the request (or reset the connection) and generate an alert email and/or log message.
You can customize the web page that FortiWeb returns to the client with the HTTP status code. For details, see Customizing error and authentication pages (replacement messages).
Deny (no log)—Block the request (or reset the connection).
-
Period Block—Block subsequent requests from the client for a number of seconds. Also configure Period Block.
You can customize the web page that FortiWeb returns to the client with the HTTP status code. For details, see Customizing error and authentication pages (replacement messages).
Note: If FortiWeb is deployed behind a NAT load balancer, when using this option, you must also define an X-header that indicates the original client’s IP. Failure to do so may cause FortiWeb to block all connections when it detects a violation of this type. For details, see Defining your proxies, clients, & X-headers.
- Redirect—Redirect the request to the URL that you specify in the protection profile and generate an alert email and/or log message. Also configure Redirect URL and Redirect URL With Reason.
-
Send HTTP Response—Block and reply to the client with an HTTP error message and generate an alert email and/or log message.
You can customize the attack block page and HTTP error code that FortiWeb returns to the client. For details, see Customizing error and authentication pages (replacement messages).
-
Bypass—Accept the request.
Note: Logging and/or alert email will occur only if enabled and configured. For details, see Logging and Alert email.
In each row, type the number of seconds that you want to block subsequent requests from the client after the FortiWeb appliance detects that the client has violated the rule.
This setting is available only if the Action is set to Period Block. The valid range is from 1 to 3,600 seconds (1 hour). See also Monitoring currently blocked IPs.
Severity
When rule violations are recorded in the attack log, each log message contains a Severity Level (
severity_level) field. In each row, select which severity level the FortiWeb appliance will use when it logs a violation of the rule:- Informative
- Low
- Medium
- High
Threat Weight
Set the weight for the threat by dragging the bar.
Trigger Action
In each row, select which trigger, if any, that the FortiWeb appliance will use when it logs and/or sends an alert email about a violation of each rule. For details, see Viewing log messages.
Bot List
Click
to select the bots not to be scanned. If you want to add an exception, select the items in the Enabled List, then move it to the Disabled List.You can also add exceptions from the attack logs.
Malicious Bots
Configure to analyze the
User-Agent:HTTP header and block known content scrapers, spiders looking for vulnerabilities, and other typically unwanted automated clients.Link checkers, retrievals of entire websites for a user’s offline use, and other automated uses of the web (sometimes called robots, spiders, web crawlers, or automated user agents) often access websites at a more rapid rate than human users. However, it would be unusual for them to request the same URL within that time frame.
Usually, web crawlers request many different URLs in rapid sequence. For example, while indexing a website, a search engine’s web crawler may rapidly request the website’s most popular URLs. If the URLs are web pages, it may also follow the hyperlinks by requesting all URLs mentioned in those pages. In this way, the behavior of web crawlers differs from a typical brute force login attack, which focuses repeatedly on one URL.
Some robots, however, are not well-behaved. You can request that robots not index and/or follow links, and disallow their access to specific URLs (see http://www.robotstxt.org/). However, misbehaving robots frequently ignore the request, and there is no single standard way to rate-limit robots.
To verify that bad robot detection is being applied, attempt to download a web page using widget (http://www.gnu.org/software/wget), which is sometimes used for content scraping.
Known Good Bots
Configure to exempt popular search engines’ spiders from DoS sensors, brute force login sensors, HTTP protocol constraints, combination rate & access control (called “advanced protection” and “custom policies” in the web UI), and blocking by geographic location (Geo IP).
This option improves access for search engines. Rapid access rates, unusual HTTP usage, and other characteristics that may be suspicious for web browsers are often normal with search engines. If you block them, your websites’ rankings and visibility may be affected.
By default, this option allows all popular predefined search engines. Known search engine indexer source IPs are updated via FortiGuard Security Service. To specify which search engines are exempted, click
and select the search engines, then click OK. See also Blacklisting known bots. - Click OK.
- To apply the known bots rule, select it in Configuring bot mitigation policy.