Checking memory usage
-
Use “diagnose debug memory” to check memory usage:
This command will collect memory information via several different kinds of backend commands.
FortiWeb# diagnose debug memory
Tue Oct 26 17:42:56 UTC 2021
17:42:56 up 5 days, 19:45, load average: 2.09, 1.78, 1.82
init 1 shared 1528kB anonymous 112kB
cmdbsvr 191 shared 17132kB anonymous 33688kB
syslogd 873 shared 256kB anonymous 44kB
klogd 874 shared 256kB anonymous 48kB
hamain 875 shared 10632kB anonymous 6972kB
hasync 876 shared 9328kB anonymous 6832kB
...
...
-
After 6.4 release, the system will generate a regular monitoring file with a backend command “/bin/FortiWeb_get_memory_usage”, which includes the same output of “diagnose debug memory”.
The regular output is recorded in /var/log/gui_upload/debug_memory.txt and can be downloaded via System > Maintenance > Backup&Restore. You can download it manually or use the one-click button to archive and download it.
You can set the interval to record debug memory logs:
config system global
set debug-monitor-interval 5 #default 5 minutes and the range is from 1 to 65535
end
/# more /var/log/gui_upload/debug_memory.txt
Fri May 28 04:30:13 UTC 2021
04:30:13 up 0 min, load average: 1.07, 0.26, 0.09
init 1 shared 1376kB anonymous 104kB
cmdbsvr 2293 shared 14044kB anonymous 29032kB
syslogd 3457 shared 280kB anonymous 48kB
...
...
-
Check current memory usage in backend shell:
Please note that FortiWeb changes the way to login to the backend shell Refer to Part VI: Run backend-shell commands.
free
This command gives you a table of the total, used, free, shared, buffer/cache, and available RAM. It also shows the total amount of swap space configured, and how much is used and available. The default unit is KB.
free= total – used – buff/cache
/# free
total used free shared buffers cached
Mem: 130987692 4990340 125997352 16540 23576 129092
-/+ buffers/cache: 4837672 126150020
Swap: 0 0 0
/proc/meminfo
This is a virtual file that reports the amount of available and used memory. It contains real-time information about the system’s memory usage as well as the buffers and shared memory used by the kernel
/# cat /proc/meminfo
MemTotal: 28635360 kB
MemFree: 25998836 kB
MemAvailable: 26127368 kB
Buffers: 201340 kB
Cached: 192220 kB
SwapCached: 0 kB
Active: 1730772 kB
Inactive: 164688 kB
Active(anon): 1501972 kB
Inactive(anon): 38064 kB
Active(file): 228800 kB
Inactive(file): 126624 kB
Unevictable: 1164 kB
Mlocked: 1164 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 104 kB
Writeback: 0 kB
AnonPages: 1503120 kB
Mapped: 89856 kB
Shmem: 38164 kB
KReclaimable: 22528 kB
Slab: 94268 kB
SReclaimable: 22528 kB
SUnreclaim: 71740 kB
KernelStack: 5536 kB
PageTables: 12048 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 14317680 kB
Committed_AS: 5405984 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 64024 kB
VmallocChunk: 0 kB
Percpu: 1984 kB
DirectMap4k: 63424 kB
DirectMap2M: 3082240 kB
DirectMap1G: 28311552 kB
/# top
Some common usage: Press Shift + P to sort the five columns of data by CPU usage (the default) or Shift + M to sort by memory usage; Press “1” (number one) to check status of all logical processors.
/# top
Mem: 4919392K used, 126068300K free, 16348K shrd, 45984K buff, 134312K cached
CPU: 0.1% usr 0.0% sys 0.0% nic 99.8% idle 0.0% io 0.0% irq 0.0% sirq
Load average: 1.33 1.40 1.38 2/954 28663
PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND
6262 1 root S 9582m 7.4 55 0.0 /bin/proxyd
6276 1 root S 224m 0.1 39 0.0 /bin/confd_ha
6264 1 root S 6539m 5.1 24 0.0 /bin/bot_daemon
6273 1 root S 2498m 1.9 60 0.0 /bin/garbage -o standalone
6316 6238 root S 2098m 1.6 7 0.0 /bin/mysqld --defaults-file=/data/etc
6251 1 root S 803m 0.6 12 0.0 /bin/monitord
6269 1 root S 411m 0.3 17 0.0 /bin/sandboxd
6271 1 root S 400m 0.3 41 0.0 /bin/shibd -F -f -p /var/run/shibd.pi
6287 1 root S 256m 0.2 59 0.0 /bin/statusd
6257 1 root S 245m 0.1 63 0.0 /bin/wvsd
6244 1 root S 219m 0.1 36 0.0 /bin/logd
6272 1 root S 202m 0.1 26 0.0 /bin/fortiviewd
/# ps -l
We usually focus on the RSS and VSZ values in ps output, which can be used to check the memory utilization for a specific process.
RSS stands for Resident Set Size and shows how much RAM is utilized at present. It shows the entire stack of physically allocated memory so it is more important.
VSZ is short for Virtual Memory Size. It’s the total amount of memory a process may hypothetically access. When a process is started, VSZ memory becomes RSS memory.
If the RSS value increases continuously and does not decrease even if the traffic drops down, it might be a hint of memory leak and require further investigation.
An example as below:
At the beginning without any traffic, the VSZ is 10.3g VSZ, and RSS is 612m.
/# free
total used free shared buff/cache available
Mem: 130983668 6092968 122149324 78124 2741376 123905312
Swap: 0 0 0
/#
/# ps -l | grep “/proxyd” | grep -v grep
S 0 23653 1 10.3g 612m 0:0 11:31 00:00:33 /bin/proxyd
After traffic is pushed, both RSS and VSZ increase obviously.
/# free
total used free shared buff/cache available
Mem: 130983668 121216100 567704 78128 9199864 8769384
Swap: 0 0 0
/#
/# ps -l | grep “/proxyd” | grep -v grep
S 0 23653 1 127g 88.9g 0:0 11:31 02:25:58 /bin/proxyd
After traffic is stopped and concurrent connections are released, the RSS value may decrease obviously, but the VSZ often does not decrease.
And, since some memory is still used, generally the RSS memory may not decrease to the initial value. As shown below, RSS is decreased from the peak value 88.9g to 1982m, but not the initial value 612m. It is normal and does not indicate a memory leak.
/# free
total used free shared buff/cache available
Mem: 130983668 9952592 119414496 77964 1616580 120133272
Swap: 0 0 0
/#
/# ps -l | grep “/proxyd” | grep -v grep
S 0 23653 1 127g 1982m 0:0 11:31 02:39:38 /bin/proxyd