Using the AusCERT malicious URL feed with an API key

In this example, a list of malicious URLs is imported from AUSCERT, an Australian not for profit organization. See AUSCERT for more information.

The FortiGuard threat feed is used to import the malicious URL feed by appending the API key to the user-agent. See HTTP header for more information. The newly created threat feed is set to block in the web filter profile, and the web filter profile is applied to a firewall policy. Any traffic that passes through the FortiGate and matches the URLs in the threat feed list will be dropped, and a replacement message will be shown.

To configure the FortiGuard category threat feed in the GUI:

1. Go Security Fabric > External Connectors and click Create New.

2. In the Threat Feeds section, select FortiGuard Category.

3. Configure the following settings:

   Status	Enabled
   Name	AusCERT_Feed
   Update method	External Feed
   URL of external resource	https://www.auscert.org.au/api/v1/malurl/combo-7-txt/

4. Click OK.

5. In the CLI, enter the following:

   config system external-resource
       edit "AusCERT_Feed"
           set user-agent "Firefox\r\nAPI-Key:SECRETAPIKEY"
       next
   end

6. In the GUI, edit the connector and configure the remaining settings as needed, then click OK.

7. Edit the connector again, and click View Entries in the right pane to view the URL list.

To configure the FortiGuard category threat feed in the CLI:

config system external-resource
    edit "AusCERT_Feed"
        set category 194
        set resource "https://www.auscert.org.au/api/v1/malurl/combo-7-txt/"
        set user-agent "Firefox\r\nAPI-Key:SECRETAPIKEY"
    next
end

When configuring a FortiGuard category threat feed in the GUI, the category is set automatically. When configuring a the threat feed in the CLI, the category must be set manually. The category must be unique and in the range of 192 - 221.

To improve the security of the connection, it is recommended to enable server certificate validation (server-identity-check) in either basic or full mode. By default, it is set to none.

To apply the FortiGuard category threat feed to a web filter profile:

1. Go to Security Profiles > Web Filter and create a new web filter profile, or edit an existing one.
2. Enable FortiGuard category based filter.

3. In the Remote Categories group, set the action for the AusCERT_Feed category to Block.

4. Configure the remaining settings as needed, then click OK.

To apply the web filter profile in a firewall policy:

1. Go to Policy & Objects > Firewall Policy and create a new policy, or edit an existing one.

2. Configure the policy fields as required.

3. Under Security Profiles, enable Web Filter and select the profile used in the previous procedure.

4. Enable Log Allowed Traffic.

5. Click OK.

URLs that match the FortiGuard category threat feed list are rated as the category matching the corresponding FortiGuard category threat feed, overriding their original domain rating.

To verify that FortiGate is blocking URLs from the AusCERT feed list:

1. Visit one of the URLs from the AusCERT_Feed list.

   A replacement message should be shown.

   

2. Go to Log & Report > Security Events and select Web Filter.

3. View the log details in the GUI, or download the log file:

   1: date=2023-04-11 time=14:18:02 eventtime=1681247882561766251 tz="-0700" logid="0316013056" type="utm" subtype="webfilter" eventtype="ftgd_blk" level="warning" vd="root" policyid=1 poluuid="26540ed0-ae54-51ed-80eb-89af8af4d53f" policytype="policy" sessionid=3275 srcip=172.20.120.13 srcport=64151 srccountry="Reserved" srcintf="port2" srcintfrole="undefined" srcuuid="3342cb44-9140-51ed-5dbe-8e0787bedeec" dstip=114.142.162.65 dstport=80 dstcountry="Australia" dstintf="port3" dstintfrole="wan" dstuuid="3342cb44-9140-51ed-5dbe-8e0787bedeec" proto=6 httpmethod="GET" service="HTTP" hostname="pcmach.co.nz" agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" profile="default" action="blocked" reqtype="direct" url="http://pcmach.co.nz/" sentbyte=427 rcvdbyte=0 direction="outgoing" msg="URL belongs to a denied category in policy" ratemethod="domain" cat=194 catdesc="AusCERT_Feed"

Troubleshooting a threat feed

In this example, the user entered the URL of external resource without the trailing slash. The following commands can be used to troubleshoot connectivity issues between a FortiGate and external resource:

diagnose debug app dnsproxy -1
diagnose debug app forticron -1
diagnose debug enable

This output shows that the DNS resolution is successful, indicating that the FortiGate has connectivity to the external server:

#diagnose debug app dnsproxy -1
[worker 0] dns_local_lookup()-2476: vfid=0, real_vfid=0, qname=www.auscert.org.au, qtype=1, qclass=1, offset=36, map#=4 max_sz=512
[worker 0] dns_lookup_aa_zone()-608: vfid=0, fqdn=www.auscert.org.au
[worker 0] dns_send_request()-1398
[worker 0] dns_send_resol_request()-1234: orig id: 0xa002 local id: 0xa002 domain=www.auscert.org.au
[worker 0] dns_find_best_server()-595: found server: 96.45.46.46 
…
id:0xa002 domain=www.auscert.org.au active

This output shows that the requested resource was missing a trailing slash:

#diagnose debug app forticron -1
fcron_timer_func()-23: Timer ext_upd fired
6745-before-init: fd=-1 name='ext-root.AusCERT_Feed' http_1=0 loc=0 state=send.body info=0-DNS fail chunk=0 content-0=0 etag=0 csum=0 done=0 closed=0
    sync-0(len=0 note=0 err=0) buf-1(sz=8192 data=179 free=8013 pos=0 end=179 max=10485760)
6745-init-as: fd=-1 name='ext-root.AusCERT_Feed' http_1=0 loc=0 state=send.body info=0-None chunk=0 content-0=0 etag=0 csum=0 done=0 closed=0
    sync-0(len=0 note=0 err=0) buf-1(sz=8192 data=0 free=8192 pos=0 end=0 max=10485760)
http_request_make()-2066: HTTP request: https

GET /api/v1/malurl/combo-7-txt HTTP/1.1
Host: www.auscert.org.au
User-Agent: Firefox
API-Key: <obfuscated>
Accept: */*
Connection: close
http_request_make()-2101: fcron_get_addr(www.auscert.org.au)
__update_ext()-187: Updating EXT 'AusCERT_Feed' with HTTP
fcron_update_ext_func()-611: update ver: 0
fcron_timer_func()-32: Timer ext_upd done
fcron_epoll_before_handle()-297: BEFORE READ fd 11 handle event 0x01 read 0xc55a40 epoll events 0x01
dns_parse_resp()-102: DNS www.auscert.org.au -> 54.253.78.74
dns_parse_resp()-102: DNS www.auscert.org.au -> 13.54.251.23
…
HTTP/1.1 301 Moved Permanently
…
Location: /api/v1/malurl/combo-7-txt/

After adding a trailing slash to the external resource URL, the connection is now working:

#diagnose debug app forticron -1
fcron_timer_func()-23: Timer ext_upd fired
2832-before-init: fd=-1 name='ext-root.AusCERT_Feed' http_1=0 loc=0 state=send.header info=0-None chunk=0 content-0=0 etag=0 csum=0 done=0 closed=0
    sync-0(len=0 note=0 err=0) buf-0(sz=0 data=0 free=0 pos=0 end=0 max=10485760)
2832-init-as: fd=-1 name='ext-root.AusCERT_Feed' http_1=0 loc=0 state=send.header info=0-None chunk=0 content-0=0 etag=0 csum=0 done=0 closed=0
    sync-0(len=0 note=0 err=0) buf-1(sz=8192 data=0 free=8192 pos=0 end=0 max=10485760)
http_request_make()-2066: HTTP request: https

GET /api/v1/malurl/combo-7-txt/ HTTP/1.1
Host: www.auscert.org.au
User-Agent: Firefox
API-Key: <obfuscated>
Accept: */*
Connection: close
…
HTTP/1.1 200 OK

These troubleshooting commands can be used to resolve a variety of issues. they are not limited to this specific use case.