FAQ

Why can't I see the bot_client.js be injected into the response page for Biometric Based Detection?

Please check from two aspects:

1) Double check if the request matches the rule or not.

2) Be aware that if the client is considered as not a bot, its good client status will be kept for 30 minutes. So FortiWeb won’t do biometric based checking to this client within 30 minutes.

How are Known Bots > Known Search Engines being matched?

On 7.0.1 and previous builds, Known Search Engine is only determined by matching IP address of HTTP requests with predefined IP ranges. In this way, it’s prone to trigger false negatives and false positives.

On 7.0.2, Known Search Engine is determined by a mixed of conditions:

• The first 24-bits of source IP or XFF IP checking.

  E.g. if the IP range is 104.250.147.218 to 104.250.147.219, then the condition will be matched if the source IP in the request is within 104.250.147.1 and 104.250.147.255.

• The header field “User Agent” checking.

• All IP checking and User Agent checking can have mix matching known engines.

E.g, if a request IP belongs to Google bots and the User Agent belongs to Bing, then this request will still be matched.

The predefined Known_engine data including User Agents and IP Ranges are defined in /data/etc/known_engines.xml, which will be updated as a part of the signature update with FDS versions.

Taking an item from /data/etc/known_engines.xml for example, a request sent by curl as below will match the Known Bots Engines:

curl -ikv -H "X-Forwarded-For: 104.250.147.100" -A "Gigabot/" https://www.example.com/test.html

Notes: In this test, an X-Forwarded-For rule with Use X-Header to Identify Original Client’s IP and Block Using Original Client’s IP is linked to the web protection profile for the server policy.

<item id="0006" name="Gigablast">

<ip_range id="1">

<ip_start>104.250.147.218</ip_start>

<ip_end>104.250.147.218</ip_end>

</ip_range>

...

...

<advanced id="none">

<user_agents>

<!-- cases

Gigabot/2.0att

Gigabot/2.0 (gigablast.com)

Gigabot/2.0/gigablast.com/spider.html

Gigabot/2.0; http://www.gigablast.com/spider.html

Gigabot/3.0 (http://www.gigablast.com/spider.html)

GigabotSiteSearch/2.0 (sitesearch.gigablast.com)

-->

<pattern value="Gigabot/" />

<pattern value="GigabotSiteSearch/" />

</user_agents>

</advanced>

</item>

Use the following troubleshooting methods when a request is not matched by the Known Search Engines:

1. Enable diagnose log to check the processing details:

   diagnose debug application known-bots 7

   diagnose debug enable

   E.g. a sample output of matching case:

   [Known Bots][DEBUG](bot_management_module_process-880): inside bot management process.

   [Known Bots][DEBUG](check_ip_in_ke-271): inside check ip in ke.

   [Known Bots][DEBUG](check_ip_in_ke-316): found IP match: id and name : 0006 Gigablast

   [Known Bots][DEBUG](check_ip_in_ke-346): found UA match: unit->name : Gigablast

   [Known Bots][INFO](check_ip_in_ke-360): match benign bots (1, 0006) and make action.

   [Known Bots][INFO](bot_management_make_action-80): inside make bm make action

   [Known Bots][INFO](http_statistic-147): Direction is client to server and first packet, do hit-count statistic.

   [Known Bots][INFO](http_statistic-154): Direction is client to server and first packet, do known engines, bad bots and regular statistic.

2. Check if the source IP (or XFF IP) or the User Agent in HTTP header is included in /data/etc/known_engines.xml.