Fortinet black logo

Handbook

Starting a web crawler test

Copy Link
Copy Doc ID 0fc7455d-2de0-11eb-96b9-00505692583a:686881
Download PDF

Starting a web crawler test

The web crawler test runs a web crawler simulation to query URLs through the DUT. This is done to test the DUT's web access security policies. FortiTester only stores the URL responses.

To start a web crawler test:
  1. Go to Cases > Security Testing > Web Protection > Web Crawler to display the test case summary page.
  2. Click + Create New to display the Select case options dialog box.
  3. In the popup dialog, for the Network Config option, select the network template you have created in Cases > Security Testing > Objects > Networks. Then the network related options will automatically be filled. See Using network configuration templates for how to create a network template.
  4. Select a Certificate Group if applicable.
  5. Select Protocol type of the simulated traffic.
  6. Click OK to continue.
  7. Configure the test case options described below.
  8. Click Start to run the test case.

FortiTester saves the configuration automatically so you can run the test again later. You can also click Save to save the test case without running it.

Tip: You can also copy an existing case, and change its settings to create a new case. In the case list, click Clone to clone the configuration. Only the case name is different from the original case.

HTTP Web Crawler Test Case configuration

Settings Guidelines
Basic Information
Name Specify the case name, or just use the default. The name appears in the list of test cases.
Network Settings
Client Ports
The graphic depicts the test ports for client-side connections. The client ports simulate the behavior of clients.

You must select at least one client port . After you select a port for client, a (check mark) is displayed on the port icon, and a tab for the port is added below the graphic. Use the tabs to toggle the Capture Packets and Subnet settings controls for each port.
Capture Packet
Capture Packet Optional. Set packet capture options if you want to capture the traffic of this port. You can capture all packets or specify a number. You can set packet capture filters for host IP/port and protocol.

Note: The system allocates temporary disk space for packet captures. The limit is 6,000,000 packets. The packets are saved to a temporary file that you can download from the running test case page. The filename indicates whether it is client or server communication and the interface port number. For example, client_port1.pcap. When a subsequent test case with packet capture enabled uses the same interface port as a previous one, the previous file is overwritten.
Subnet
Subnet IP Address or Range Specify a single IP address with standard format (for example, 10.1.2.1) or an address range like 10.1.2.1-10.1.2.99.
Netmask Specify a netmask between 1 and 31.
Gateway NAT mode only. Specify the gateway IP address.
Client (Profile)
URL Group Select the URL group. Click on Manager Group to add or delete URLs.

Starting a web crawler test

The web crawler test runs a web crawler simulation to query URLs through the DUT. This is done to test the DUT's web access security policies. FortiTester only stores the URL responses.

To start a web crawler test:
  1. Go to Cases > Security Testing > Web Protection > Web Crawler to display the test case summary page.
  2. Click + Create New to display the Select case options dialog box.
  3. In the popup dialog, for the Network Config option, select the network template you have created in Cases > Security Testing > Objects > Networks. Then the network related options will automatically be filled. See Using network configuration templates for how to create a network template.
  4. Select a Certificate Group if applicable.
  5. Select Protocol type of the simulated traffic.
  6. Click OK to continue.
  7. Configure the test case options described below.
  8. Click Start to run the test case.

FortiTester saves the configuration automatically so you can run the test again later. You can also click Save to save the test case without running it.

Tip: You can also copy an existing case, and change its settings to create a new case. In the case list, click Clone to clone the configuration. Only the case name is different from the original case.

HTTP Web Crawler Test Case configuration

Settings Guidelines
Basic Information
Name Specify the case name, or just use the default. The name appears in the list of test cases.
Network Settings
Client Ports
The graphic depicts the test ports for client-side connections. The client ports simulate the behavior of clients.

You must select at least one client port . After you select a port for client, a (check mark) is displayed on the port icon, and a tab for the port is added below the graphic. Use the tabs to toggle the Capture Packets and Subnet settings controls for each port.
Capture Packet
Capture Packet Optional. Set packet capture options if you want to capture the traffic of this port. You can capture all packets or specify a number. You can set packet capture filters for host IP/port and protocol.

Note: The system allocates temporary disk space for packet captures. The limit is 6,000,000 packets. The packets are saved to a temporary file that you can download from the running test case page. The filename indicates whether it is client or server communication and the interface port number. For example, client_port1.pcap. When a subsequent test case with packet capture enabled uses the same interface port as a previous one, the previous file is overwritten.
Subnet
Subnet IP Address or Range Specify a single IP address with standard format (for example, 10.1.2.1) or an address range like 10.1.2.1-10.1.2.99.
Netmask Specify a netmask between 1 and 31.
Gateway NAT mode only. Specify the gateway IP address.
Client (Profile)
URL Group Select the URL group. Click on Manager Group to add or delete URLs.