Fortinet Document Library

Version:

Version:

Version:

Version:

Version:

Version:

Version:

Version:

Version:

Version:


Table of Contents

Administration Guide

Viewing domain data

The system provides three dimensions to view the domain data:

  • Overview
    A high level summary of data collected for the domain, including Top 10 URLs by Hit, Violations triggered by anomalies, HMM learning process, Event Dashboard.
  • Tree View
    Display the entire URL directory of the domain in a tree view. You can click the URL path to view its violation statistics.
  • Parameter View
    Display statistics related with parameters, such as HMM learning stages, boxplots, distribution of anomalies. You can also rebuild parameters or set the strictness level for anomalies.

To view the collected domain data:

  1. Click Machine Learning > Anomaly Detection.
  2. Double-click a server policy that contains the desired anomaly detection profile.
  3. Scroll down to the bottom of the Edit Anomaly Detection Configuration page.
  4. In the Action column, click (View Domain).

Overview

The Overview tab provides a summary of data collected for the domain through the use of the anomaly detection profile. It reports information about the entire domain, including the domain overview, Top 10 URLs by Hit, HMM Learning Progress, Violations Triggered by Anomalies, and Events Dashboard.

Domain overview

The top of the Overview page provides a high-level summary of the data that the machine-learning module has learned about the domain.

Parameters Description
Access Frequency

Indicates how frequent this application is being accessed.

Start Time

The date and time when the machine-learning module started to learn about the domain.

URL Number

The total number of URLs that the machine-learning module has learned.

Action (Alert/Block)

The total number of the alerts, including both Alert action and Alert & Deny action, that has been issued since the start time up to the present moment, as well as the percentage of each in the total number of requests.

Service(HTTP/HTTPS)

The total amount of the HTTP and the HTTPS traffic from the start time up to now.

Page Charset

The charset of URLs in the domain, such as UTF-8.

Top 10 URLs by Hit

The Top 10 URLs by Hit chart displays the top 10 URLs for page hits counts.

HMM Learning Progress

This chart displays the statistics of HMM learning states of all parameters in the domain.

Parameters Description
Collecting

Indicates that the learning progress of parameters is in the sample collecting stage.

Building

Indicates that, after successfully collected the samples, the anomaly detection module has begun to build all the needed mathematical models for the parameters. This is the mathematical models-building stage.

Testing

Indicates that, after successfully built the mathematical models, the models are being tested. All models are required to be tested against a certain number of samples until they have proved to be stable.

Running

Indicates that the mathematical models of the parameters are stable, and the anomaly detection model is running. Requests triggering an anomaly will move into the second anomaly detection layer to check whether they are actual threats.

Discarded

Indicates that FortiWeb has determined that it cannot build a mathematical model for these parameters, and therefore will not use anomaly detection to protect them.

Violations Triggered by Anomalies

This chart displays the total number of the potential anomalies and definite anomalies found by the anomaly detection pofile.

Event Dashboard

This chart displays the anomaly detection events, such as sample collection, model running, building and testing, along with the time periods when these events take place.

Tree View

The Tree View displays the entire URL directory of the domain in a tree view. You can choose either one of the URLs to view its violation statistics.

Web site directory

The left panel of the Tree View page shows the directory structure of the website. The / (backslash) indicates the root of the site. You can click a URL in the directory tree, then the violation statistics of this URL will be displayed on the right side of the Tree View page. You can also click a directory, then click Rebuild Directory to rebuild anomaly detection models for all the URLs under the selected directory.

URL-specific data

This part of the Tree View page shows the statistics of a specific URL.

Parameters Description
Access Frequency

The frequency at which this URL was accessed in last 24 hours. The frequency is divided into 7 levels, as defined below:

  • Level1 ( over 500 requests )
  • Level2 ( over 1000 requests )
  • Level3 ( over 1500 requests )
  • Level4 ( over 2000 requests )
  • Level5 ( over 2500 requests )
  • Level6 ( over 3000 requests )
  • Level7 ( over 3500 requests )
Model Initialization Date

The date and time when the mathematical model of this URL was initialized. It shows when FortiWeb began to learn about the data of this URL.

Action (Alert/Block)

The actions taken for this URL for all requests in last 24 hours, including the number of requests alerted and blocked.

Anomaly

The anomalies detected by the machine learning model.

Violation Trend

This chart shows the trend of violations in last 24 hours, including the number of violations alerted and blocked.

Triggered Violations Based on Anomaly Type

This chart shows the number of violations triggered by anomaly type in the last 24 hours.

Rebuild URL and Import buttons

The Tree View page also provides two control buttons: Rebuild URL and Import.

  • Rebuild URL—Click this button to clear the preceding mathematical model for the parameters in this URL, and then begin collecting new samples and build the models again. The samples collected for the previous model will be discarded.
  • Relearn URL—Click this button to clear the preceding mathematical model for the parameters in this URL, and then begin collecting more samples to build the model. The samples collected for the previous model will be not discarded. They will be reused to build the new model.
  • Import— Click this button to import an existing mathematical model of a specific parameter. For information on exporting data of a parameter, see Actions you can take on any parameter.
Parameters

Parameters tab shows the HMM learning states of all the parameters attached to the URL. For example, if the URL is http://www.demo.com/1.php?user_name=jack, then user_name is the parameter. An URL can contain multiple parameters. Click the (View HMM Details) icon to view details on this parameter.

Allow Method

You can set the HTTP request methods that are allowed to access the URL.

There are two ways to set the allow method: By Machine Learning, Customized.

Method Description
By Machine Learning

If you choose By Machine Learning, the system will automatically set the HTTP request methods in the Allow Method Settings based on the result of machine learning.

The system collects samples of HTTP requests for this URL. The system refers to the Trust or Black IP list configured in the Anomaly Detection profile to decide whether to collect samples from a certain client.

If the content type of the request is HTML or Text, the system collects 1024 samples for this URL. For other content types, the system collects 256 samples.

You can set the sample collection time period using the following command.

config waf machine-learning-policy

edit <policy-id>

set method-learning-time

next

end

The system will not stop collecting samples unless the expected number of samples are collected and the collection has lasted for the specified time period.

If an HTTP request method is used by more than 1% requests of the overall requests, the anomaly detection model will allow this method in the Allow Method Settings.

Click the Rebuild Method button to rebuild the methods if you think the methods learned by machine leaning model are not reasonable.

Customized

This approach allows you to customize the allow methods.

To set a custom allowed method:

  1. Click the Customized tab.
  2. Select any method(s) of interest.
  3. Click Apply.

To switch back to the default allowed method (machine learning):

  1. Click the By Machine Leaning tab.
  2. Click Apply.

Parameter View

Parameter View displays anomaly detection statistics for all the parameters. Click the parameter name in the left-side nagivation bar to see details for this paramter.

Parameter Name: The name of the parameter.

HMM Learning Stage: The stage which the HMM learning process is in. It can be one of the following:

  • Collecting—The system is collecting data samples.
  • Building—Sample collection is completed, and is building the mathematical models. Note: This phase last only a few seconds.
  • Testing—In this phase, the system collects 500 samples for this argument, and tests them against the mathematical model. If 5% of the samples for this argument are recognized as anomalies, this mathematical model is considered invalid. The system will discard the learning results and rebuild the mathematical model.
  • Running—The system enters this stage after the testing has completed successfully. FortiWeb will use this mathematical model to evaluate all new samples for this argument. If the samples are anomalies, the system will employ the second anomaly detection layer to verify whether the anomaly is an attack and take the corresponding action.
  • Discarded—FortiWeb has determined that it cannot build a mathematical model for these parameters, and therefore will not use anomaly detection to protect them.

Collected Samples: The number of samples collected during the sample collection period.

Please note that the diagrams introduced below are available only when the status is in testing or running stage.

 

Probability Boxplots

Applications change frequently as new URLs are added and existing parameters provide new functions. This means the mathematical model of the same parameter might be different than what FortiWeb originally observed during the collection phase. In this case, FortiWeb needs to re-learn the parameter and then updates the mathematical model for it.

First of all, FortiWeb needs to determine that the functions of the parameter have changed. To do that, it uses boxplots to depict numerical data and the probability distribution of a certain number of parameter values.

Every time the system observes 500 valid parameter values, it generates one boxplot to display the probability distribution of these values. During sample collection period, the system generates 2 or 4 boxplots (sample boxplots). After anomaly detection model is built, the system will keep on generating new boxplots to display the probability distribution of the new inputs. The following is an example of the boxplot diagram. The new boxplot is shown in blue, whereas the sample boxplots are brown. The system displays at most five new boxplots. With new inputs coming in and new boxplot generated, the system will remove the oldest one at the left to spare a place for the new boxplot.

In the boxplot diagram, the median rectangular area in the boxplot where most of the data is located is called the notch area, whereas the entire area containing all the data from the maximum value to the minimum value is called the entire data distribution area. Depending on the Application Change Sensitivity you set in the anomaly detection profile, when the system observes different extent of overlapping area between the new boxplot and sample boxplots, it determines that the functions of the parameter have changed and then updates mathematical model for this parameter (i.e., re-collect samples and build model).

  • Low—The system triggers model update only when the entire data distribution area of the new boxplot doesn't have any overlapping part with that of the sample boxplots.
  • Medium—The system triggers model update if the notch area of the new boxplot doesn't have any overlapping part with the entire data distribution areas of the sample boxplots.
  • High—The system triggers model update as long as the notch area of the new boxplot doesn't have any overlapping part with that of the sample boxplots.

The number of boxplots do not overlap configuration in anomaly detection profile is also a key factor to consider. For example, if you set 2 in this option, the system triggers model update when 2 new boxplots don't overlap with the sample boxplots.

Distribution of Anomalies triggered by HMM

This diagram displays the potential or definite anomalies in red and the normal requests collected during sample collection phase in blue. The system judges whether a request is normal or not based on its probability and the length of the parameter value.

Anomaly Strictness Level Details

The system uses the following formula to calculate whether a sample is an anomaly:

The probability of the anomaly > μ + the strictness level * σ

If the probability of the sample is larger than the value of "μ + the strictness level * σ", this sample will be identified as anomaly.

μ and σ are calculated based on the probabilities of all the samples collected during the sample collection period, where μ is the average value of all the parameters' probabilities, σ is the standard deviation. They are fixed values. So, the value of "μ + the strictness level * σ" varies with the strictness level you set. As shown in the following diagram, the dotted red line (that is, the value of "μ + the strictness level * σ") stays at the position where the strictness level is set to 3, as in μ + 3σ. If the strictness level is set to a smaller value, then the dotted red line will move closer to the center, which may cause some samples to be detected as anomaly. In a word, the smaller the value of the strictness level is, the more strict the anomaly detection model will be.

Manage anomaly-detecting settings

You can use the following options to experiment on the strictness levels.

Inherit global settings: Select this option if you want this parameter to inherit the strictness level you have set for the domains in the anomaly detection policy.

Custom settings: Select this option if you want a different strictness level for this parameter. Specify different values and observe the movement of dotted red line in the Anomaly Strictness Level Details diagram. Choose an appropriate value to get the most optimistic detection accuracy, meanwhile the normal samples are not be falsely detected as anomalies.

Test Sample : Click Test Sample, then enter a parameter value to verify whether it will be detected as an anomaly at the current strictness level.

Actions you can take on any parameter

There is a configuration button which, when clicked, will open a drop-down menu with the following options.

Menu option Description
Rebuild Parameter Clear the preceding mathematical model for the parameter, and then begin collecting new samples and build the models again. The samples collected for the previous model will be discarded.

Relearn Parameter

Clear the preceding mathematical model for the parameter, and then begin collecting more samples to build the model. The samples collected for the previous model will be not discarded. They will be reused to build the new model.

Discard Discards this parameter and does not re-build it. This will disable the learning for this parameter and bypass anomaly detection all together for this parameter.
Export Export the mathematical model for this parameter to a file. You can import the model to arbitrary URL. See Import under Rebuild URL and Import buttons
Noisy Samples

The abnormal samples detected during the sample collection period. They are excluded from the samples used to build the anomaly detection model.


Anomaly Samples

The samples which have been recognized as anomalies. The list may change as new strictness settings are applied.

Additional Samples

These are the samples manually added from the attack logs. For more information, see Add additional sample from attack logs.

Events

The anomaly detection events, such as sample collection, model running, building and testing, along with the time periods when these events take place. These events are also displayed in the anomaly detection Events dashboard in Overview tab.

Viewing domain data

The system provides three dimensions to view the domain data:

  • Overview
    A high level summary of data collected for the domain, including Top 10 URLs by Hit, Violations triggered by anomalies, HMM learning process, Event Dashboard.
  • Tree View
    Display the entire URL directory of the domain in a tree view. You can click the URL path to view its violation statistics.
  • Parameter View
    Display statistics related with parameters, such as HMM learning stages, boxplots, distribution of anomalies. You can also rebuild parameters or set the strictness level for anomalies.

To view the collected domain data:

  1. Click Machine Learning > Anomaly Detection.
  2. Double-click a server policy that contains the desired anomaly detection profile.
  3. Scroll down to the bottom of the Edit Anomaly Detection Configuration page.
  4. In the Action column, click (View Domain).

Overview

The Overview tab provides a summary of data collected for the domain through the use of the anomaly detection profile. It reports information about the entire domain, including the domain overview, Top 10 URLs by Hit, HMM Learning Progress, Violations Triggered by Anomalies, and Events Dashboard.

Domain overview

The top of the Overview page provides a high-level summary of the data that the machine-learning module has learned about the domain.

Parameters Description
Access Frequency

Indicates how frequent this application is being accessed.

Start Time

The date and time when the machine-learning module started to learn about the domain.

URL Number

The total number of URLs that the machine-learning module has learned.

Action (Alert/Block)

The total number of the alerts, including both Alert action and Alert & Deny action, that has been issued since the start time up to the present moment, as well as the percentage of each in the total number of requests.

Service(HTTP/HTTPS)

The total amount of the HTTP and the HTTPS traffic from the start time up to now.

Page Charset

The charset of URLs in the domain, such as UTF-8.

Top 10 URLs by Hit

The Top 10 URLs by Hit chart displays the top 10 URLs for page hits counts.

HMM Learning Progress

This chart displays the statistics of HMM learning states of all parameters in the domain.

Parameters Description
Collecting

Indicates that the learning progress of parameters is in the sample collecting stage.

Building

Indicates that, after successfully collected the samples, the anomaly detection module has begun to build all the needed mathematical models for the parameters. This is the mathematical models-building stage.

Testing

Indicates that, after successfully built the mathematical models, the models are being tested. All models are required to be tested against a certain number of samples until they have proved to be stable.

Running

Indicates that the mathematical models of the parameters are stable, and the anomaly detection model is running. Requests triggering an anomaly will move into the second anomaly detection layer to check whether they are actual threats.

Discarded

Indicates that FortiWeb has determined that it cannot build a mathematical model for these parameters, and therefore will not use anomaly detection to protect them.

Violations Triggered by Anomalies

This chart displays the total number of the potential anomalies and definite anomalies found by the anomaly detection pofile.

Event Dashboard

This chart displays the anomaly detection events, such as sample collection, model running, building and testing, along with the time periods when these events take place.

Tree View

The Tree View displays the entire URL directory of the domain in a tree view. You can choose either one of the URLs to view its violation statistics.

Web site directory

The left panel of the Tree View page shows the directory structure of the website. The / (backslash) indicates the root of the site. You can click a URL in the directory tree, then the violation statistics of this URL will be displayed on the right side of the Tree View page. You can also click a directory, then click Rebuild Directory to rebuild anomaly detection models for all the URLs under the selected directory.

URL-specific data

This part of the Tree View page shows the statistics of a specific URL.

Parameters Description
Access Frequency

The frequency at which this URL was accessed in last 24 hours. The frequency is divided into 7 levels, as defined below:

  • Level1 ( over 500 requests )
  • Level2 ( over 1000 requests )
  • Level3 ( over 1500 requests )
  • Level4 ( over 2000 requests )
  • Level5 ( over 2500 requests )
  • Level6 ( over 3000 requests )
  • Level7 ( over 3500 requests )
Model Initialization Date

The date and time when the mathematical model of this URL was initialized. It shows when FortiWeb began to learn about the data of this URL.

Action (Alert/Block)

The actions taken for this URL for all requests in last 24 hours, including the number of requests alerted and blocked.

Anomaly

The anomalies detected by the machine learning model.

Violation Trend

This chart shows the trend of violations in last 24 hours, including the number of violations alerted and blocked.

Triggered Violations Based on Anomaly Type

This chart shows the number of violations triggered by anomaly type in the last 24 hours.

Rebuild URL and Import buttons

The Tree View page also provides two control buttons: Rebuild URL and Import.

  • Rebuild URL—Click this button to clear the preceding mathematical model for the parameters in this URL, and then begin collecting new samples and build the models again. The samples collected for the previous model will be discarded.
  • Relearn URL—Click this button to clear the preceding mathematical model for the parameters in this URL, and then begin collecting more samples to build the model. The samples collected for the previous model will be not discarded. They will be reused to build the new model.
  • Import— Click this button to import an existing mathematical model of a specific parameter. For information on exporting data of a parameter, see Actions you can take on any parameter.
Parameters

Parameters tab shows the HMM learning states of all the parameters attached to the URL. For example, if the URL is http://www.demo.com/1.php?user_name=jack, then user_name is the parameter. An URL can contain multiple parameters. Click the (View HMM Details) icon to view details on this parameter.

Allow Method

You can set the HTTP request methods that are allowed to access the URL.

There are two ways to set the allow method: By Machine Learning, Customized.

Method Description
By Machine Learning

If you choose By Machine Learning, the system will automatically set the HTTP request methods in the Allow Method Settings based on the result of machine learning.

The system collects samples of HTTP requests for this URL. The system refers to the Trust or Black IP list configured in the Anomaly Detection profile to decide whether to collect samples from a certain client.

If the content type of the request is HTML or Text, the system collects 1024 samples for this URL. For other content types, the system collects 256 samples.

You can set the sample collection time period using the following command.

config waf machine-learning-policy

edit <policy-id>

set method-learning-time

next

end

The system will not stop collecting samples unless the expected number of samples are collected and the collection has lasted for the specified time period.

If an HTTP request method is used by more than 1% requests of the overall requests, the anomaly detection model will allow this method in the Allow Method Settings.

Click the Rebuild Method button to rebuild the methods if you think the methods learned by machine leaning model are not reasonable.

Customized

This approach allows you to customize the allow methods.

To set a custom allowed method:

  1. Click the Customized tab.
  2. Select any method(s) of interest.
  3. Click Apply.

To switch back to the default allowed method (machine learning):

  1. Click the By Machine Leaning tab.
  2. Click Apply.

Parameter View

Parameter View displays anomaly detection statistics for all the parameters. Click the parameter name in the left-side nagivation bar to see details for this paramter.

Parameter Name: The name of the parameter.

HMM Learning Stage: The stage which the HMM learning process is in. It can be one of the following:

  • Collecting—The system is collecting data samples.
  • Building—Sample collection is completed, and is building the mathematical models. Note: This phase last only a few seconds.
  • Testing—In this phase, the system collects 500 samples for this argument, and tests them against the mathematical model. If 5% of the samples for this argument are recognized as anomalies, this mathematical model is considered invalid. The system will discard the learning results and rebuild the mathematical model.
  • Running—The system enters this stage after the testing has completed successfully. FortiWeb will use this mathematical model to evaluate all new samples for this argument. If the samples are anomalies, the system will employ the second anomaly detection layer to verify whether the anomaly is an attack and take the corresponding action.
  • Discarded—FortiWeb has determined that it cannot build a mathematical model for these parameters, and therefore will not use anomaly detection to protect them.

Collected Samples: The number of samples collected during the sample collection period.

Please note that the diagrams introduced below are available only when the status is in testing or running stage.

 

Probability Boxplots

Applications change frequently as new URLs are added and existing parameters provide new functions. This means the mathematical model of the same parameter might be different than what FortiWeb originally observed during the collection phase. In this case, FortiWeb needs to re-learn the parameter and then updates the mathematical model for it.

First of all, FortiWeb needs to determine that the functions of the parameter have changed. To do that, it uses boxplots to depict numerical data and the probability distribution of a certain number of parameter values.

Every time the system observes 500 valid parameter values, it generates one boxplot to display the probability distribution of these values. During sample collection period, the system generates 2 or 4 boxplots (sample boxplots). After anomaly detection model is built, the system will keep on generating new boxplots to display the probability distribution of the new inputs. The following is an example of the boxplot diagram. The new boxplot is shown in blue, whereas the sample boxplots are brown. The system displays at most five new boxplots. With new inputs coming in and new boxplot generated, the system will remove the oldest one at the left to spare a place for the new boxplot.

In the boxplot diagram, the median rectangular area in the boxplot where most of the data is located is called the notch area, whereas the entire area containing all the data from the maximum value to the minimum value is called the entire data distribution area. Depending on the Application Change Sensitivity you set in the anomaly detection profile, when the system observes different extent of overlapping area between the new boxplot and sample boxplots, it determines that the functions of the parameter have changed and then updates mathematical model for this parameter (i.e., re-collect samples and build model).

  • Low—The system triggers model update only when the entire data distribution area of the new boxplot doesn't have any overlapping part with that of the sample boxplots.
  • Medium—The system triggers model update if the notch area of the new boxplot doesn't have any overlapping part with the entire data distribution areas of the sample boxplots.
  • High—The system triggers model update as long as the notch area of the new boxplot doesn't have any overlapping part with that of the sample boxplots.

The number of boxplots do not overlap configuration in anomaly detection profile is also a key factor to consider. For example, if you set 2 in this option, the system triggers model update when 2 new boxplots don't overlap with the sample boxplots.

Distribution of Anomalies triggered by HMM

This diagram displays the potential or definite anomalies in red and the normal requests collected during sample collection phase in blue. The system judges whether a request is normal or not based on its probability and the length of the parameter value.

Anomaly Strictness Level Details

The system uses the following formula to calculate whether a sample is an anomaly:

The probability of the anomaly > μ + the strictness level * σ

If the probability of the sample is larger than the value of "μ + the strictness level * σ", this sample will be identified as anomaly.

μ and σ are calculated based on the probabilities of all the samples collected during the sample collection period, where μ is the average value of all the parameters' probabilities, σ is the standard deviation. They are fixed values. So, the value of "μ + the strictness level * σ" varies with the strictness level you set. As shown in the following diagram, the dotted red line (that is, the value of "μ + the strictness level * σ") stays at the position where the strictness level is set to 3, as in μ + 3σ. If the strictness level is set to a smaller value, then the dotted red line will move closer to the center, which may cause some samples to be detected as anomaly. In a word, the smaller the value of the strictness level is, the more strict the anomaly detection model will be.

Manage anomaly-detecting settings

You can use the following options to experiment on the strictness levels.

Inherit global settings: Select this option if you want this parameter to inherit the strictness level you have set for the domains in the anomaly detection policy.

Custom settings: Select this option if you want a different strictness level for this parameter. Specify different values and observe the movement of dotted red line in the Anomaly Strictness Level Details diagram. Choose an appropriate value to get the most optimistic detection accuracy, meanwhile the normal samples are not be falsely detected as anomalies.

Test Sample : Click Test Sample, then enter a parameter value to verify whether it will be detected as an anomaly at the current strictness level.

Actions you can take on any parameter

There is a configuration button which, when clicked, will open a drop-down menu with the following options.

Menu option Description
Rebuild Parameter Clear the preceding mathematical model for the parameter, and then begin collecting new samples and build the models again. The samples collected for the previous model will be discarded.

Relearn Parameter

Clear the preceding mathematical model for the parameter, and then begin collecting more samples to build the model. The samples collected for the previous model will be not discarded. They will be reused to build the new model.

Discard Discards this parameter and does not re-build it. This will disable the learning for this parameter and bypass anomaly detection all together for this parameter.
Export Export the mathematical model for this parameter to a file. You can import the model to arbitrary URL. See Import under Rebuild URL and Import buttons
Noisy Samples

The abnormal samples detected during the sample collection period. They are excluded from the samples used to build the anomaly detection model.


Anomaly Samples

The samples which have been recognized as anomalies. The list may change as new strictness settings are applied.

Additional Samples

These are the samples manually added from the attack logs. For more information, see Add additional sample from attack logs.

Events

The anomaly detection events, such as sample collection, model running, building and testing, along with the time periods when these events take place. These events are also displayed in the anomaly detection Events dashboard in Overview tab.