Fortinet black logo

New Features

Improve FortiAnalyzer log caching

Copy Link
Copy Doc ID 77966226-6996-11ec-bdf2-fa163e15d75b:942202
Download PDF

Improve FortiAnalyzer log caching

Reliable logging to FortiAnalyzer is improved to prevent lost logs when the connection between FortiOS and FortiAnalyzer is disrupted. When reliable mode is enabled:

  1. Logs are cached in a FortiOS memory queue.
  2. FortiOS sends logs to FortiAnalyzer, and FortiAnalyzer uses seq_no to track received logs.
  3. After FortiOS sends logs to FortiAnalyzer, logs are moved to a confirm queue in FortiOS.
  4. FortiOS periodically queries FortiAnalyzer for the latest seq_no of the last log received, and clears logs from the confirm queue up to the seq_no.
  5. If the connection between FortiOS and FortiAnalyzer is disrupted, FortiOS resends the logs in the confirm queue to FortiAnalyzer when the connection is reestablished.
Note

FortiAnalyzer 7.2.0 and later is required.

To enable reliable mode:
config log fortianalyzer setting
    set reliable enable
end
To view the memory and confirm queues:
  1. Verify that log synchronization is enabled for FortiAnalyzer:

    # diagnose test application fgtlogd 1 vdom-admin=0 mgmt=root fortilog: faz: global , enabled server=172.16.200.251, realtime=1, ssl=1, state=connected server_log_status=Log is allowed., src=, mgmt_name=FGh_Log_root_172.16.200.251, reliable=1, sni_prefix_type=none, required_entitlement=none, region=ca-west-1,, logsync_enabled:1, logsync_conn_id:65535, seq_no:790 ...

  2. When a network disruption disconnects FortiOS from FortiAnalyzer and FortiOS continues to generate logs, the logs are cached in the memory queue.

    • View the number of logs in the cache and queue:

      # diagnose test application fgtlogd 41
      
      cache maximum: 189516595(180MB) objects: 40 used: 27051(0MB) allocated: 29568(0MB)
      
      VDOM:root
      Memory queue for: global-faz
          queue:
              num:9 size:6976(0MB) total size:26068(0MB) max:189516595(180MB) logs:28
      Confirm queue for: global-faz
          queue:
              num:29 size:19092(0MB) total size:27051(0MB) max:189516595(180MB) logs:7
      # diagnose test application fgtlogd 30
      VDOM:root
      Memory queue for: global-faz
              queue:
                      num:9 size:6976(0MB) total size:26068(0MB) max:189516595(180MB)
                              type:3, cat=1, log_count=1, seq_no=0, data len=359 size:435
                              type:3, cat=1, log_count=1, seq_no=0, data len=307 size:383
                              ......
                              type:3, cat=0, log_count=4, seq_no=0, data len=1347 size:1423
                              type:3, cat=4, log_count=1, seq_no=0, data len=653 size:729
                      'total log count':28,  'total data len':6292
      
      Confirm queue for: global-faz
              queue:
                      num:29 size:19092(0MB) total size:26068(0MB) max:189516595(180MB)
                              type:3, cat=1, log_count=1, seq_no=1, data len=290 size:366
                              type:3, cat=1, log_count=1, seq_no=2, data len=233 size:309
                              ......
                              type:3, cat=0, log_count=1, seq_no=28, data len=524 size:600
                              type:3, cat=1, log_count=1, seq_no=29, data len=307 size:383
                      'total log count':76,  'total data len':16888

      There are nine OFTP items cached to the memory queue, and 29 OFTP items to send from FortiOS to FortiAnalyzer that are waiting for confirmation from FortiAnalyzer.

    • Go to Log & Report > Log Settings to view the queue in the GUI:

  3. Re-establish the connection between FortiOS and FortiAnalyzer and confirm that the queue has cleared by checking the seq_no, which indicates the latest confirmation log from FortiAnalyzer:

    # diagnose test application fgtlogd 30 VDOM:root Memory queue for: global-faz queue: num:0 size:0(0MB) total size:0(0MB) max:189516595(180MB) 'total log count':0, 'total data len':0 Confirm queue for: global-faz queue: num:0 size:0(0MB) total size:0(0MB) max:189516595(180MB) 'total log count':0, 'total data len':0

    The queue has been cleared, meaning that FortiOS received confirmation from FortiAnalyzer and cleared the confirm queue.

    # diagnose test application fgtlogd 1
    vdom-admin=0
    mgmt=root
    
    fortilog:
    faz: global , enabled 
            server=172.16.200.251, realtime=1, ssl=1, state=connected
            server_log_status=Log is allowed.,
            src=, mgmt_name=FGh_Log_root_172.16.200.251, reliable=1, sni_prefix_type=none,
            required_entitlement=none, region=ca-west-1,
            logsync_enabled:1, logsync_conn_id:65535, seq_no:67
                status: ver=6, used_disk=0, total_disk=0, global=0, vfid=0 conn_verified=Y
                SNs: last sn update:38 seconds ago.
                    Sn list:
                   (FAZ-VMTM21000000,age=38s)
                queue: qlen=0.

    OFTP items with a seq_no lower than 67 have been sent to FortiAnalyzer and were confirmed.

Improve FortiAnalyzer log caching

Reliable logging to FortiAnalyzer is improved to prevent lost logs when the connection between FortiOS and FortiAnalyzer is disrupted. When reliable mode is enabled:

  1. Logs are cached in a FortiOS memory queue.
  2. FortiOS sends logs to FortiAnalyzer, and FortiAnalyzer uses seq_no to track received logs.
  3. After FortiOS sends logs to FortiAnalyzer, logs are moved to a confirm queue in FortiOS.
  4. FortiOS periodically queries FortiAnalyzer for the latest seq_no of the last log received, and clears logs from the confirm queue up to the seq_no.
  5. If the connection between FortiOS and FortiAnalyzer is disrupted, FortiOS resends the logs in the confirm queue to FortiAnalyzer when the connection is reestablished.
Note

FortiAnalyzer 7.2.0 and later is required.

To enable reliable mode:
config log fortianalyzer setting
    set reliable enable
end
To view the memory and confirm queues:
  1. Verify that log synchronization is enabled for FortiAnalyzer:

    # diagnose test application fgtlogd 1 vdom-admin=0 mgmt=root fortilog: faz: global , enabled server=172.16.200.251, realtime=1, ssl=1, state=connected server_log_status=Log is allowed., src=, mgmt_name=FGh_Log_root_172.16.200.251, reliable=1, sni_prefix_type=none, required_entitlement=none, region=ca-west-1,, logsync_enabled:1, logsync_conn_id:65535, seq_no:790 ...

  2. When a network disruption disconnects FortiOS from FortiAnalyzer and FortiOS continues to generate logs, the logs are cached in the memory queue.

    • View the number of logs in the cache and queue:

      # diagnose test application fgtlogd 41
      
      cache maximum: 189516595(180MB) objects: 40 used: 27051(0MB) allocated: 29568(0MB)
      
      VDOM:root
      Memory queue for: global-faz
          queue:
              num:9 size:6976(0MB) total size:26068(0MB) max:189516595(180MB) logs:28
      Confirm queue for: global-faz
          queue:
              num:29 size:19092(0MB) total size:27051(0MB) max:189516595(180MB) logs:7
      # diagnose test application fgtlogd 30
      VDOM:root
      Memory queue for: global-faz
              queue:
                      num:9 size:6976(0MB) total size:26068(0MB) max:189516595(180MB)
                              type:3, cat=1, log_count=1, seq_no=0, data len=359 size:435
                              type:3, cat=1, log_count=1, seq_no=0, data len=307 size:383
                              ......
                              type:3, cat=0, log_count=4, seq_no=0, data len=1347 size:1423
                              type:3, cat=4, log_count=1, seq_no=0, data len=653 size:729
                      'total log count':28,  'total data len':6292
      
      Confirm queue for: global-faz
              queue:
                      num:29 size:19092(0MB) total size:26068(0MB) max:189516595(180MB)
                              type:3, cat=1, log_count=1, seq_no=1, data len=290 size:366
                              type:3, cat=1, log_count=1, seq_no=2, data len=233 size:309
                              ......
                              type:3, cat=0, log_count=1, seq_no=28, data len=524 size:600
                              type:3, cat=1, log_count=1, seq_no=29, data len=307 size:383
                      'total log count':76,  'total data len':16888

      There are nine OFTP items cached to the memory queue, and 29 OFTP items to send from FortiOS to FortiAnalyzer that are waiting for confirmation from FortiAnalyzer.

    • Go to Log & Report > Log Settings to view the queue in the GUI:

  3. Re-establish the connection between FortiOS and FortiAnalyzer and confirm that the queue has cleared by checking the seq_no, which indicates the latest confirmation log from FortiAnalyzer:

    # diagnose test application fgtlogd 30 VDOM:root Memory queue for: global-faz queue: num:0 size:0(0MB) total size:0(0MB) max:189516595(180MB) 'total log count':0, 'total data len':0 Confirm queue for: global-faz queue: num:0 size:0(0MB) total size:0(0MB) max:189516595(180MB) 'total log count':0, 'total data len':0

    The queue has been cleared, meaning that FortiOS received confirmation from FortiAnalyzer and cleared the confirm queue.

    # diagnose test application fgtlogd 1
    vdom-admin=0
    mgmt=root
    
    fortilog:
    faz: global , enabled 
            server=172.16.200.251, realtime=1, ssl=1, state=connected
            server_log_status=Log is allowed.,
            src=, mgmt_name=FGh_Log_root_172.16.200.251, reliable=1, sni_prefix_type=none,
            required_entitlement=none, region=ca-west-1,
            logsync_enabled:1, logsync_conn_id:65535, seq_no:67
                status: ver=6, used_disk=0, total_disk=0, global=0, vfid=0 conn_verified=Y
                SNs: last sn update:38 seconds ago.
                    Sn list:
                   (FAZ-VMTM21000000,age=38s)
                queue: qlen=0.

    OFTP items with a seq_no lower than 67 have been sent to FortiAnalyzer and were confirmed.