Log buffer on FortiGates with an SSD disk

FortiGates with an SSD disk have a configurable log buffer. When the connection to FortiAnalyzer is unreachable, the FortiGate is able to buffer logs on disk if the memory log buffer is full. The logs queued on the disk buffer can be sent successfully once the connection to FortiAnalyzer is restored.

The number of logs queued on the disk buffer is visible in the Log & Report > Log Settings page:

The queued logs are buffered to the memory first and then disk. Main miglogd handles the disk buffering job, while miglogd-children handles the memory buffering. Disk buffer statistics only appear under Main miglogd, and memory buffer statistics only appears under miglogd-children. If the total buffer is full, new logs will overwrite the old logs.

To configure the log buffer:

1. Allocate disk space (MB) to temporarily store logs to FortiAnalyzer:

   config system global
       set faz-disk-buffer-size 200
   end

2. Check the Main miglogd and miglogd-children statistics. The 200 MB disk buffer has been set, and there are currently no logs buffered in memory or on disk when FortiAnalyzer is reachable:

   # diagnose test application miglogd 41 0
   cache maximum: 106100940(101MB) objects: 0 used: 0(0MB) allocated: 0(0MB)
   VDOM:root
   Queue for: global-faz
    
       memory queue:
           num:0 size:0(0MB) max:101906636(97MB) logs:0
    
       disk max queue size:200MB total:0MB
           totol items:0
           disk queue agents:
               devid:-1-10-0-1
               buffer path:/var/log/qbuf/10.0/1
               saved size:0MB cached size:0
               save roll:0 restore roll:0
               restore id:0 space:0MB
   
   # diagnose test application miglogd 41 1
   cache maximum: 106100940(101MB) objects: 0 used: 0(0MB) allocated: 0(0MB)
   VDOM:root
   Queue for: global-faz
    
       memory queue:
           num:0 size:0(0MB) max:101906636(97MB) logs:0
    
       disk queue client:
           devid:-1-10-0-1 status:buffering
           Total in cache:0 size:0(0MB) max:4MB logs:0

3. Disable the connection between the FortiGate and FortiAnalyzer. For example, delete the FortiGate from the FortiAnalyzer authorized device list.

   Assuming a massive number of logs (~ 300000) are recorded during this downtime, the logs will be queued in the memory buffer first. If the memory buffer is full, then the remaining logs will be queued on the disk buffer.

4. Check the Main miglogd and miglogd-children statistics again. All 97 MB of the memory buffer is occupied, and 76 of the 200 MB has been taken from the disk buffer:

   # diagnose test application miglogd 41 0
   cache maximum: 106100940(101MB) objects: 0 used: 0(0MB) allocated: 0(0MB)
   VDOM:root
   Queue for: global-faz
   
           memory queue:
                   num:0 size:0(0MB) max:101906636(97MB) logs:0
   
           disk max queue size:200MB total:76MB
                   totol items:128917
                   disk queue agents:
                           devid:-1-10-0-1
                           buffer path:/var/log/qbuf/10.0/1
                           saved size:76MB cached size:3324984
                           save roll:19 restore roll:0
                           restore id:0 space:0MB
   
   # diagnose test application miglogd 41 1
   cache maximum: 106100940(101MB) objects: 165721 used: 101908358(97MB) allocated: 106449280(101MB)
   VDOM:root
   Queue for: global-faz
   
           memory queue:
                   num:165718 size:101906500(97MB) max:101906636(97MB) logs:165718
   
           disk queue client:
                   devid:-1-10-0-1 status:restoring
                   restore id:1267 space:0MB
                   Total in cache:3 size:1858(0MB) max:4MB logs:3

   The overall miglogd statistics shows the total cached logs is the sum of the logs buffered in memory and on disk:

   # diagnose test application miglogd 6
   mem=0, disk=11, alert=0, alarm=0, sys=0, faz=300053, faz-cloud=0, webt=0, fds=0
   interface-missed=44
   Queues in all miglogds: cur:165718  total-so-far:165718
   global log dev statistics:
   faz 0: sent=0, failed=0, cached=300053, dropped=0 , relayed=0
   Num of REST URLs: 0

5. Enable the connection between FortiAnalyzer and the FortiGate.
6. After a while, check the miglogd statistics to confirm that all buffered logs are being sent to FortiAnalyzer successfully:

   # diagnose test application miglogd 6
   mem=0, disk=11, alert=0, alarm=0, sys=0, faz=300058, faz-cloud=0, webt=0, fds=0
   interface-missed=44
   Queues in all miglogds: cur:4294832957  total-so-far:165726
   global log dev statistics:
   faz 0: sent=300058, failed=0, cached=0, dropped=0 , relayed=0
   Num of REST URLs: 15
   
   # diagnose test application miglogd 41 0
   cache maximum: 106100940(101MB) objects: 0 used: 0(0MB) allocated: 0(0MB)
   VDOM:root
   Queue for: global-faz
   
           memory queue:
                   num:0 size:0(0MB) max:101906636(97MB) logs:0
   
           disk max queue size:200MB total:0MB
                   totol items:0
                   disk queue agents:
                           devid:-1-10-0-1
                           buffer path:/var/log/qbuf/10.0/1
                           saved size:0MB cached size:0
                           save roll:20 restore roll:20
                           restore id:1267 space:0MB
                           
   # diagnose test application miglogd 41 1
   cache maximum: 106100940(101MB) objects: 0 used: 0(0MB) allocated: 0(0MB)
   VDOM:root
   Queue for: global-faz
   
           memory queue:
                   num:0 size:0(0MB) max:101906636(97MB) logs:0
   
           disk queue client:
                   devid:-1-10-0-1 status:buffering
                   Total in cache:0 size:0(0MB) max:4MB logs:0