Configuring machine-learning templates

This section discusses how to configure machine-learning templates. Templates are required when the application uses dynamic URLs and unusual parameters. This is not very common, and templates are not required in most cases. Creating a machine-learning template has two steps:

1. URL Replacer Rule
2. URL Replacer Policy

Configure a URL replacer rule

URL replacer rules enable the machine-learning module to adapt to dynamic URLs and unusual parameters.

When web applications have dynamic URLs or unusual parameter styles, you must adapt the URL Replacer Rule to recognize them.

By default, machine learning assumes that your web applications use the most common URL structure:

As seen above, most commonly used URLs share the following characteristics:

• All parameters follow a question mark (?). They do not follow a hash (#) or any other separator character.
• If there are multiple name-value pairs, each pair is separated by an ampersand &. They are not separated by a semi-colon (;) or any other separator character.
• All paths before the question mark (?) are static—they do not change based upon input, blending the path with parameters (sometimes called a dynamic URL).

For example, the page at

/app/main

always has that same path. After you log in, the page’s URL does not become

/app/marco/main

or

/app#deepa

For another example, the URL does not dynamically reflect the inventory, such as:

/app/sprockets/widget1024894

Some web applications, however, embed parameters within the path structure of a URL, or use unusual or non-uniform parameter separator characters. If you do not configure URL replacers to handle such variations, it can cause the system to gather machine learning data incorrectly, which can lead to the following consequences:

• Machine-learning reports do not contain the correct URL structure.
• URL- or parameter-learning is endless.
• Parameter data is incomplete, despite the fact that the FortiWeb appliance has seen traffic containing the parameter.

For example, with Microsoft Outlook Web App (OWA), the user’s login name could be embedded within the path structure of the URL, such as:

/owa/tom/index.html

/owa/mary/index.html

instead of suffixed as a parameter, such as:

/owa/index.html?username=tom

/owa/index.html?username=mary

Machine learning will continue to create new URLs as new users are added to OWA. It will also expend extra resources learning about URLs and parameters that are actually the same. Additionally, machine learning may not be able to fully learn the application structure because each user may not request the same URLs.

To address this issue, you must create a URL Replacer Rule that recognizes the user name within the OWA URL as if it were a standard, suffixed parameter value so that machine learning can function properly.

To create a URL Replacer Rule:

1. Click Machine Learning > Machine Learning Templates.
2. Click the URL Replacer Rule tab.
3. Click Create New.
4. Configure the parameters as described in the table below.
5. Click OK when done.

Parameters	Function
Name	Specify a unique name that can be referenced by other parts of the configuration. Note: The name can be up to 63 characters long with no space or special character.
Type	Select either of the following: Predefined—Use one of the predefined URL replacers which can be selected from the Application Type below. Custom-Defined—Define your own URL replacer by configuring the URL Path, New URL, Param Change, and New Param fields below.
Application Type	If you have selected Predefined in the Type field above, then you must click the down arrow and select either of the following from the list menu: JSP—Use the URL replacer designed for Java server pages (JSP) web applications, where parameters are often separated by semi-colon (;). OWA 2003— Use the URL replacer designed for default URLs in Microsoft Outlook Web App (OWA), where user name and directory parameters are often embedded within the URL, as illustrated below: (^/public/)(.*) (^/exchange/)([^/]+)/*(([^/]+)/(.*))* Note: These two application types are predefined URL interpreter plug-ins used by popular web applications.
Custom-Defined	If you have selected Custom-Defined in the Type field above, then you must populate the following fields:
URL Path	Enter a regular expression, such as (^/[^/]+)/(.*), matching all and only the URLs to which the URL replacer should apply. The URL path can be up to 256 characters long. The pattern does not require a backslash (/). However, it must at least match URLs that begin with a backslash as they appear in the HTTP header, such as /index.html. Do not include the domain name, such as www.example.com. To test the regular expression against a sample text, click the >> (Test) icon. This opens the Regular Expression Validator dialog where you can fine-tune the expression. Note: If this URL replacer is to be used sequentially in a set of URL replacers, instead of being mutually exclusive, this regular expression must match the URL produced by the preceding interpreter rather than the original URL from the request.
New URL	Enter either a literal URL, such as /index.html, or a regular expression with a back-reference (such as $1) defining how the URL will be interpreted. The new URL cab be up to 256 characters long. Note: Back-references can only refer to capture groups (parts of the expression surrounded with parentheses) within the same URL replacer, and must not refer to capture groups in other URL replacers.
Param Change	Enter either the parameter’s literal value, such as user1, or a back-reference (such as $0) defining how the value will be interpreted.
New Param	Type either the parameter’s literal name, such as username, or a back-reference (such as $2) defining how the parameter’s name will be interpreted in the auto-learning report. You can use up to 256 characters. Note: Back-references can only refer to capture groups (parts of the expression surrounded with parentheses) within the same URL replacer. They must not refer to capture groups in other URL replacers.

Configuring a URL replacer policy

In order to use URL Replacer Rules with a machine-learning policy, you must group URL replacer rules into sets, which form URL replacer policies.

The sets can be mutually exclusive, where a set contains expressions for all possible URL structures, but only one of the URL replacer rules will match a given request’s URL.

They also can be sequential, where a set contains expressions to interpret multiple parameters in a single given URL; each interpreter’s URL input is the URL output of the preceding interpreter, and they each parse the URL until all parameters have been extracted; the sequential order of URL replacer rules is determined by the URL replacer rule’s priority in the set.

To configure a URL replacer policy:

1. Click Machine Learning > Machine Learning Templates.
2. Click the URL Replacer Policy tab.
3. Click Create New .
4. In Name, type a name that can be referenced by other parts of the configuration. Note: The name can be up to 63 characters long, with no space or special characters.
5. Click OK.
6. Click Create New, and select the URL replacer rule to be grouped in the URL replacer policy.
7. Click OK.

Note: You can select URL replacer policy in one or more machine-leaning profiles, using the following steps:

1. Click Machine Learning > Anomaly detection.
2. Double-click an anomaly detection pofile to open the profile.
3. Scroll down to the Action Settings section.
4. Click the URL Replacer Policy down arrow to select a URL replacer policy.
5. Repeat Steps 1 through 4 to select other URL replacer policies in the same or another machine-learning profiles.
6. Click OK when done.