Appendix B - Preload cache content and web crawler

You can configure FortiProxy to pre-load cache content based on manually defined URL patterns with scheduled crawling function. This feature is useful for schools and hotels where popular content, such as video, can be predicted ahead of schedule, downloaded outside of peak hours, and viewed by customers using the cache.

The following execute preload CLI commands list and describe configurable preload caching and web crawler options.

execute preload list

Use this command to show currently active URLs and their run schedules:

execute preload list

For example:

URL's scheduled for preload:
http://google.com
    Depth: 0, runs every 1 minutes, next run at Dec 23 16:49
http://google.ca
    Depth: 5, runs every 2 minutes, next run at Dec 23 16:52
https://news.cnn.com
    Depth: 1, runs every 5 minutes, next run at Dec 23 18:47

execute preload show-log

Use this command to display all the completed operations and their status.

execute preload url

Use this command to schedule a crawl, preload, refresh, or pin request for a given URL:

execute preload url <url> <depth> <at_time> <repeat_after> <repetitions> <user-agent> <password>

• <url>: URL to preload.

• <depth>: Depth of preload.

• <at_time>: In the format of HHH:MM. HHH is hours from present (between 0-672), and MM is minutes from present (between 0-59). The default is set to 0:00.

• <repeat_after>: HHH:MM. Set HHH between 0-168 and MM between 0-59. The default is set to 168:59 (max).

• <repetitions>: End after this many repetitions (between 1-365). The default is set to 1.

• <user-agent>: Specify client type (free text) to identify as a user agent. The default is set to "Wget/1.17 (linux-gnu)".

• <user>: Specify user name.

• <password>: Password for the user (asked for in a separate prompt).

execute preload url-delete

Use this command to delete a scheduled crawl, preload, refresh, or pin request for a given URL:

execute preload url-delete <url>

Use the following command, for example, to delete all operations for http://www.fortinet.com:

execute preload url-delete http://www.fortinet.com/

To view a list of pending crawls, see execute preload list.

Examples

The following command would fetch http://www.fortinet.com and do the following:

• preload cache immediately:

  execute preload url http://www.fortinet.com/

• crawl it to depth two immediately:

  execute preload url http://www.fortinet.com/ 2

• crawl it to depth two after ten minutes:

  execute preload url http://www.fortinet.com/ 2 00:10

• crawl it to depth two after ten minutes and after 24 hours 30 times (that is,fetch the URL in ten minutes and every day for 30 days):

  execute preload url http://www.fortinet.com/ 2 00:10 24:00 30

• crawl with the user agent “Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:21.0) Gecko/20100101 Firefox/21.0”:

  execute preload url http://www.fortinet.com/ 0 00:00 00:01 1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:21.0) Gecko/20100101 Firefox/21.0"