Blacklisting & whitelisting clients

You can block requests from clients based upon their source IP address directly, their current reputation known to FortiGuard, or which country or region the IP address is associated with.

Conversely, you can also exempt clients from scans typically included by the policy.

Blacklisting source IPs with poor reputation

Manually identifying and blocking all known attackers in the world would be an impossible task. To block:

• botnets
• spammers
• phishers
• malicious spiders/crawlers
• virus-infected clients
• clients using anonymizing proxies
• DDoS participants

you can configure FortiWeb to use the FortiGuard IP Reputation. IP reputation leverages many techniques for accurate, early, and frequently updated identification of compromised and malicious clients so you can block attackers before they target your servers. Data about dangerous clients derives from many sources around the globe, including:

• FortiGuard service statistics
• honeypots
• botnet forensic analysis
• anonymizing proxies
• 3rd party sources in the security community

From these sources, Fortinet compiles a reputation for each public IP address. Clients will have poor reputations if they have been participating in attacks, willingly or otherwise. Because blacklisting innocent clients is equally undesirable, Fortinet also restores the reputations of clients that improve their behavior. This is crucial when an infected computer is cleaned, or in DHCP or PPPoE pools where an innocent client receives an IP address that was previously leased by an attacker.

Because IP reputation data is based on evidence of hostility rather than a client’s current physical location on the globe, if your goal is to block attackers rather than restrict delivery, this feature may be preferable. The IP Reputation feature can block or log clients based on X-header-derived client source IPs. For details, see Defining your proxies, clients, & X-headers.

IP reputation knowledge is regularly updated if you have subscribed and connected your FortiWeb to the FortiGuard IP Reputation service. Due to this, new options appear periodically. You can monitor the FortiGuard website feed (http://fortiguard.com/rss/fg.xml) for security advisories which may correlate with new IP reputation-related options. For details, see Connecting to FortiGuard services.

Because geographical IP policies are evaluated before many other techniques, defining these IP addresses can be used to improve performance. For details, see Sequence of scans.

To configure an IP reputation policy

1. If you need to exempt some clients’ public IP addresses due to possible false positives, configure IP reputation exemptions first. Go to IP Protection > IP Reputation and select the Exceptions tab to create a new exception.
2. Go to IP Protection > IP Reputation and select the IP Reputation Policy tab.
   To access this part of the web UI, your administrator’s account access profile must have Read and Write permission to items in the Web Protection Configuration category. For details, see Permissions.
3. In the Status column, enable the following categories of disreputable clients that you want to block and/or log:

Botnet	Malware that may perform many malicious tasks, such as downloading and executing additional malware, receiving commands from a control server and relaying specific information and telemetry back to the control server, updating or deleting itself, stealing login and password information, logging keystrokes, participating in a Distributed Denial of Service (DDoS) attack, or locking and encrypting the contents of your computer and demanding payment for its safe return.
Anonymous proxy	A tool that attempts to make a user's activity untraceable. It acts as an intermediary between users and the Internet so that users can access the Internet anonymously. Users often be trying to bypass geography restrictions or otherwise hide activity that they don't want traced to them.
Phishing	A social engineering technique that is used to obtain sensitive and confidential information by masquerading as communications from a trusted entity such as a well known institution, company, or website. The malware is typically not in the communication itself, but in the links within the communication.
Spam	A messaging technique in which a large volume of unsolicited messages are sent to a large number of recipients. The content of spam may be harmless, but often contain malware, too.
Tor	A type of anonymous proxy that is available as software to facilitate anonymous web browsing on the Internet. Tor directs user web traffic through an overlay network to hide information about users. Users aim to keep communication on the Internet anonymous. Tor may allow users to circumvent security measures such as geography restrictions or otherwise hide activity that they don't want traced to them.
Others	This includes threats to which the FortiGuard IP Reputation service assigns a poor reputation, including virus-infected clients and malicious spiders/crawlers.

APTs often mask their source IP using anonymizing proxies. While casual attackers will move on to easier potential targets if their initial attempts fail, APTs are motivated to persist until they achieve a successful breach. Early warning can be critical. Therefore even if some innocent anonymous clients use your web servers and you do not want to block them, you still may want to log proxied anonymous requests. Filtering your other attack logs by these anonymous IPs can help you to locate and focus on dangerous requests from these IPs, whether you want to use them to configure a defense, for law enforcement, or for forensic analysis.

4. For the categories that you enabled, configure these settings:

Action	Select the action that FortiWeb takes when it detects the category: Alert—Accept the request and generate an alert email and/or log message. Alert & Deny—Block the request (or reset the connection) and generate an alert email and/or log message. You can customize the web page that FortiWeb returns to the client with the HTTP status code. For details, see Customizing error and authentication pages (replacement messages). Deny (no log)—Block the request (or reset the connection). Period Block—Block subsequent requests from the client for a number of seconds. Also configure Block Period. You can customize the web page that FortiWeb returns to the client with the HTTP status code. For details, see Customizing error and authentication pages (replacement messages). Note: If FortiWeb is deployed behind a NAT load balancer, when using this option, you must also define an X-header that indicates the original client’s IP. For details, see Defining your proxies, clients, & X-headers. Failure to do so may cause FortiWeb to block all connections when it detects a violation of this type. Redirect—Redirect the request to the URL that you specify in the protection profile and generate an alert email and/or log message. Also configure Redirect URL and Redirect URL With Reason. Send 403 Forbidden—Reply with an HTTP 403 Access Forbidden error message and generate an alert and/or log message. The default action is Alert.
Block Period	Enter the number of seconds that you want to block subsequent requests from the client after the FortiWeb appliance detects the category. This setting is available only if the Action is set to Period Block. The valid range is from 1 to 3,600 (1 hour). The default value is 1. For details, see Monitoring currently blocked IPs.
Severity	When categories are recorded in the attack log, each log message contains a Severity Level (severity_level) field. In each row, select which severity level the FortiWeb appliance will use when it logs a violation of the rule: Informative Low Medium High The default value is High.
Trigger Action	Select which trigger, if any, that FortiWeb will carry out when it logs and/or sends an alert email about the detection of a category. For details, see Viewing log messages.

5. Click Apply.
6. To apply your IP reputation policy, enable IP Reputation in a protection profile that is used by a policy. For details, see Configuring a protection profile for inline topologies or Configuring a protection profile for an out-of-band topology or asynchronous mode of operation.

   Attack log messages contain Anonymous Proxy : IP Reputation Violation or Botnet : IP Reputation Violation when this feature detects a possible attack.

See also

• Predefined suspicious request URLs
• Recognizing data types
• Connecting to FortiGuard services
• How often does Fortinet provide FortiGuard updates for FortiWeb?

Blacklisting & whitelisting countries & regions

While many websites are truly global in nature, others are specific to a region. Government web applications that provide services only to its residents are one example.

In such cases, when requests appear to originate from other parts of the world, it may not be worth the security risk to accept them.

• DDoS botnets and mercenary hackers might be the predominant traffic source.
• Anonymizing VPN services or Tor may have been used to mask the true source IP of an attacker that is actually within your own country.

Blacklisting clients individually in this case would be time-consuming and difficult to maintain due to PPPoE or other dynamic allocations of public IP addresses, and IP blocks that are re-used by innocent clients.

FortiWeb allows you to block traffic from many IP addresses that are currently known to belong to networks in other regions. It uses a MaxMind GeoLite (https://www.maxmind.com) database of mappings between geographical regions and all public IP addresses that are known to originate from them.

You can also specify exceptions to the blacklist, which allows you to, block a country or region but allow a geographic location within that country or region. If you enable Allow Known Search Engines, blacklisting will also bypass client source IP addresses if they are using a known search engine.

Because network mappings may change as networks grow and shrink, if you use this feature, be sure to periodically update the geography-to-IP mapping database. To download the file, go to the Fortinet Customer Service & Support website:

https://support.fortinet.com

Because geographical IP policies are evaluated before many other techniques, defining these IP addresses can be used to improve performance. For details, see Sequence of scans.

To configure blocking by geography

1. Verify that client source IP addresses are visible to FortiWeb in either the X-headers or as the SRC field at the IP layer. For details, see Defining your web servers & load balancers.
   If FortiWeb is behind an external load balancer that applies SNAT, for example, you may need to configure it to append its and the client’s IP address to X-Forwarded-For: in the HTTP header so that FortiWeb can apply this feature. Otherwise, all traffic may appear to come from the same client, with a private network IP: the external load balancer.
2. If you want to use a trigger to create a log message and/or alert email when a geographically blacklisted client attempts to connect to your web servers, configure the trigger first. For details, see Viewing log messages.
3. If you need to exempt some clients’ public IP addresses, configure Geo IP reputation exemptions first:
   

• Go to IP Protection > Geo IP.
• To access this part of the web UI, your administrator’s account access profile must have Read and Write permission to items in the Web Protection Configuration category. For details, see Permissions.
• Specify a name for the exception item, and then click OK.
• Click Create New to add IPv4/IPv6 addresses (for example, 192.168.0.1 or 2001::1) or IPv4/IPv6 ranges (for example, 192.168.0.1-192.168.0.256 or 2001::1-2001::100) to the exception item, as required.

Go to IP Protection > Geo IP.

Click Create New.

Configure these settings:

Name	Type a name that can be referenced by other parts of the configuration. The maximum length is 63 characters.
Severity	When rule violations are recorded in the attack log, each log message contains a Severity Level (severity_level) field. Select which severity level the FortiWeb appliance will use when a blacklisted IP address attempts to connect to your web servers: Informative Low Medium High
Trigger Action	Select which trigger, if any, that the FortiWeb appliance uses when it logs and/or sends an alert email about a blacklisted IP address’s attempt to connect to your web servers. For details, see Viewing log messages.
Exception	If required, select the exceptions configuration you created in If you need to exempt some clients’ public IP addresses, configure Geo IP reputation exemptions first:.

Click OK.

Click Create New.

From the Country list on the left, select one or more geographical regions that you want to block, then click the right arrow to move them to the Selected Country list on the right.
In addition to countries, the Country list also includes distinct territories within a country, such as Puerto Rico and United States Minor Outlying Islands, and regions that are not associated with any country, such as Antarctica.

Click OK.
The web UI returns to the initial dialog. The countries that you are blocking will appear as individual entries.

Click OK.

To apply your geographical blocking rule, select it in a protection profile that a server policy is using. For details, see Configuring a protection profile for inline topologies or Configuring a protection profile for an out-of-band topology or asynchronous mode of operation.

See also

• Blacklisting & whitelisting clients using a source IP or source IP range
• Connecting to FortiGuard services
• How often does Fortinet provide FortiGuard updates for FortiWeb?

Blacklisting & whitelisting clients using a source IP or source IP range

You can define which source IP addresses are trusted clients, undetermined, or distrusted.

• Trusted IPs—Almost always allowed to access to your protected web servers. Trusted IPs are exempt from many (but not all) of the restrictions that would otherwise be applied by a server policy. For a list of skipped scans, see Sequence of scans.
• Blacklisted IPs—Blocked and prevented from accessing your protected web servers. Requests from blacklisted IP addresses receive a warning message as the HTTP response. The warning message page includes ID: 70007, which is the ID of all attack log messages about requests from blacklisted IPs.

If a source IP address is neither explicitly blacklisted nor trusted by an IP list policy, the client can access your web servers, unless it is blocked by any of your other configured, subsequent web protection scan techniques. For details, see Sequence of scans.

Because trusted and blacklisted IP policies are evaluated before many other techniques, defining these IP addresses can be used to improve performance. For details, see Sequence of scans.

Because many businesses, universities, and even now home networks use NAT, a packet’s source IP address may not necessarily match that of the client. Keep in mind that if you black list or white list an individual source IP, it may therefore inadvertently affect other clients that share the same IP.

To configure policies for individual source IPs

1. If you want to use a trigger to create a log message and/or alert email when a blacklisted client attempts to connect to your web servers, configure the trigger first. See Viewing log messages.
2. Go to IP Protection > IP List.
   To access this part of the web UI, your administrator’s account access profile must have Read and Write permission to items in the Web Protection Configuration category. For details, see Permissions.
3. Click Create New.
4. In Name, type a unique name that can be referenced by other parts of the configuration. The maximum length is 63 characters.
5. Click OK.
6. Click Create New to add an entry to the set.
7. Configure these settings:

Type	Select either: Trust IP—The source IP address is trusted and allowed to access your web servers, unless it fails a previous scan. For details, see Sequence of scans. Block IP—The source IP address that is distrusted, and is permanently blocked (blacklisted) from accessing your web servers, even if it would normally pass all other scans. Note: If multiple clients share the same source IP address, such as when a group of clients is behind a firewall or router performing network address translation (NAT), blacklisting the source IP address could block innocent clients that share the same source IP address with an offending client.
IPv4/IPv6 / IP Range	Type the client’s source IP address. You can enter either a single IP address or a range of addresses (e.g., 172.22.14.1-172.22.14.256 or 10:200::10:1-10:200:10:100).
Severity	When rule violations are recorded in the attack log, each log message contains a Severity Level (severity_level) field. Select which severity level the FortiWeb appliance will use when a blacklisted IP address attempts to connect to your web servers: Informative Low Medium High
Trigger Policy	Select which trigger, if any, that the FortiWeb appliance will use when it logs and/or sends an alert email about a blacklisted IP address’s attempt to connect to your web servers. For details, see Viewing log messages.

8. Click OK.
9. Repeat the previous steps for each individual IP list member that you want to add to the IP list.
10. To apply the IP list, select it in an inline or Offline Protection profile. For details, see Configuring a protection profile for inline topologies or Configuring a protection profile for an out-of-band topology or asynchronous mode of operation.
    Attack log messages contain Blacklisted IP blocked when this feature detects a blacklisted source IP address.

See also

• Blacklisting & whitelisting countries & regions
• Sequence of scans
• Monitoring currently blocked IPs

Blacklisting content scrapers, search engines, web crawlers, & other robots

You can use FortiWeb features to control access by Internet robots such as:

• search engine indexers
• automated tools such as link checkers, web crawlers, and spiders

FortiWeb keeps up-to-date the predefined signatures for malicious robots and source IPs if you have subscribed to FortiGuard Security Service.

To block typically unwanted automated tools, use Bad Robot.

To control which search engine crawlers are allowed to access your sites, go to Server Objects > Global > Known Search Engines; also configure Allow Known Search Engines.

See also

• Sequence of scans