How to implement multi-processing for large-scale FortiAP management

You can configure multiple processors for wireless daemons to scale large numbers of FortiAP per FortiGate Controller. For FortiGate managed APs, it splits the total number of FortiAPs into smaller groups where each daemon manages a group. The processes won't be as overloaded, and if one child daemon has an issue, it only affects that group of FortiAPs instead of all the FortiAPs managed by the FortiGate.

The number of processors you can assign varies by FortiGate model and is based on the number of FortiAPs it is allowed to manage. The maximum value you can specify in varies according to the wireless-controller.wtp in table size from different platforms.

wireless-controller.wtp	Maximum acd-process-count
8192	32
4096	16
512-1024	8
128-256	4
16-64	2

You can configure the following processors:

• cw_acd

• wpad_ac

Configuring multiple cw_acd processes

The acd-process-count option allows you to specify the number of cw_acd processes to manage FortiAPs.

To configure multiple cw_acd processes:

In this example, there are about 1300 FortiAPs managed by a FortiGate with 16 cw_acd processes to handle all the FortiAPs.

1. Set the acd-process-count to 0 in wireless-controller global:

   config wireless-controller global
     set acd-process-count 16
   end
   

2. Verify the number of FortiAPs managed per cw_acd:

   # diagnose wireless wlac -c mpmt
   acd main  process pid     : 321
   acd child process count   : 16
         idx=01 pid= 321 sl=N/A                  sm=/tmp/cwAcSock_mpmt_mngr sh=                                  
         idx=02 pid= 376 sl=/tmp/cwCwAcSocket_data sm=/tmp/cwAcSock_mpmt_data sh=                                  
       * idx=03 pid= 377 sl=/tmp/cwCwAcSocket    sm=/tmp/cwAcSock_mpmt    sh=                                   ws_cnt=1305 1283(RUN)   86(cfg) 1189(oper)
         idx=04 pid= 401 sl=/tmp/cwCwAcSocket_1  sm=/tmp/cwAcSock_mpmt_1  sh=/tmp/hasync_to_cw_acd_unix_sock_1  ws_cnt=80     77(RUN)    4(cfg)   70(oper)
         idx=05 pid= 402 sl=/tmp/cwCwAcSocket_2  sm=/tmp/cwAcSock_mpmt_2  sh=/tmp/hasync_to_cw_acd_unix_sock_2  ws_cnt=78     77(RUN)    5(cfg)   72(oper)
         idx=06 pid= 403 sl=/tmp/cwCwAcSocket_3  sm=/tmp/cwAcSock_mpmt_3  sh=/tmp/hasync_to_cw_acd_unix_sock_3  ws_cnt=91     89(RUN)    6(cfg)   83(oper)
         idx=07 pid= 404 sl=/tmp/cwCwAcSocket_4  sm=/tmp/cwAcSock_mpmt_4  sh=/tmp/hasync_to_cw_acd_unix_sock_4  ws_cnt=93     92(RUN)    6(cfg)   84(oper)
         idx=08 pid= 405 sl=/tmp/cwCwAcSocket_5  sm=/tmp/cwAcSock_mpmt_5  sh=/tmp/hasync_to_cw_acd_unix_sock_5  ws_cnt=92     91(RUN)    7(cfg)   84(oper)
         idx=09 pid= 406 sl=/tmp/cwCwAcSocket_6  sm=/tmp/cwAcSock_mpmt_6  sh=/tmp/hasync_to_cw_acd_unix_sock_6  ws_cnt=92     91(RUN)   10(cfg)   81(oper)
         idx=10 pid= 407 sl=/tmp/cwCwAcSocket_7  sm=/tmp/cwAcSock_mpmt_7  sh=/tmp/hasync_to_cw_acd_unix_sock_7  ws_cnt=78     77(RUN)    4(cfg)   73(oper)
         idx=11 pid= 408 sl=/tmp/cwCwAcSocket_8  sm=/tmp/cwAcSock_mpmt_8  sh=/tmp/hasync_to_cw_acd_unix_sock_8  ws_cnt=76     74(RUN)    5(cfg)   69(oper)
         idx=12 pid= 409 sl=/tmp/cwCwAcSocket_9  sm=/tmp/cwAcSock_mpmt_9  sh=/tmp/hasync_to_cw_acd_unix_sock_9  ws_cnt=82     79(RUN)    9(cfg)   70(oper)
         idx=13 pid= 410 sl=/tmp/cwCwAcSocket_10 sm=/tmp/cwAcSock_mpmt_10 sh=/tmp/hasync_to_cw_acd_unix_sock_10 ws_cnt=76     74(RUN)    4(cfg)   70(oper)
         idx=14 pid= 411 sl=/tmp/cwCwAcSocket_11 sm=/tmp/cwAcSock_mpmt_11 sh=/tmp/hasync_to_cw_acd_unix_sock_11 ws_cnt=80     77(RUN)    6(cfg)   70(oper)
         idx=15 pid= 412 sl=/tmp/cwCwAcSocket_12 sm=/tmp/cwAcSock_mpmt_12 sh=/tmp/hasync_to_cw_acd_unix_sock_12 ws_cnt=78     78(RUN)    5(cfg)   72(oper)
         idx=16 pid= 413 sl=/tmp/cwCwAcSocket_13 sm=/tmp/cwAcSock_mpmt_13 sh=/tmp/hasync_to_cw_acd_unix_sock_13 ws_cnt=76     76(RUN)    5(cfg)   71(oper)
         idx=17 pid= 414 sl=/tmp/cwCwAcSocket_14 sm=/tmp/cwAcSock_mpmt_14 sh=/tmp/hasync_to_cw_acd_unix_sock_14 ws_cnt=78     78(RUN)    5(cfg)   73(oper)
         idx=18 pid= 415 sl=/tmp/cwCwAcSocket_15 sm=/tmp/cwAcSock_mpmt_15 sh=/tmp/hasync_to_cw_acd_unix_sock_15 ws_cnt=76     75(RUN)    1(cfg)   74(oper)
         idx=19 pid= 416 sl=/tmp/cwCwAcSocket_16 sm=/tmp/cwAcSock_mpmt_16 sh=/tmp/hasync_to_cw_acd_unix_sock_16 ws_cnt=79     78(RUN)    4(cfg)   73(oper)
   Curr Time: 683

   Each cw_acd process handles a small number of FortiAPs, about 90.

3. Verify the CPU used by cw_acd:

   # diagnose system top 5 30
   Run Time:  0 days, 0 hours and 11 minutes
   5U, 0N, 4S, 91I, 0WA, 0HI, 0SI, 0ST; 16063T, 8236F
               csfd      340      R      87.5     1.3    8
             cw_acd      377      S      12.9     6.5    6
             flpold      336      S       1.9     0.0    1
             cu_acd      325      S       1.4     0.1    0
             cw_acd      402      S       0.9     0.9    6
             cw_acd      401      S       0.9     0.9    2
             cw_acd      412      S       0.4     1.2    8
             cw_acd      404      S       0.4     1.0    10
             cw_acd      405      S       0.4     1.0    4
             cw_acd      403      S       0.4     1.0    2
             cw_acd      409      S       0.4     0.9    4
             cw_acd      408      S       0.4     0.9    6
             cw_acd      414      S       0.4     0.9    2
             cw_acd      413      S       0.4     0.9    8
               node      275      S       0.4     0.3    4
            miglogd      295      S       0.4     0.3    10
                cid      345      S       0.4     0.2    6
            miglogd      391      S       0.4     0.2    6
            miglogd      389      S       0.4     0.2    8
          forticron      282      S       0.4     0.1    6
             flcfgd      326      S       0.4     0.1    9
         fortilinkd      324      S       0.4     0.0    0
             cw_acd      376      S       0.0     2.8    3
             cw_acd      406      S       0.0     1.0    6
             cw_acd      411      S       0.0     0.9    10
             cw_acd      416      S       0.0     0.9    8
             cw_acd      407      S       0.0     0.9    2
             cw_acd      415      S       0.0     0.9    0
             cw_acd      410      S       0.0     0.8    4
            cmdbsvr      237      S       0.0     0.7    0
   
   # get system performance status 
   CPU states: 5% user 3% system 0% nice 92% idle 0% iowait 0% irq 0% softirq
   CPU0 states: 6% user 4% system 0% nice 90% idle 0% iowait 0% irq 0% softirq
   CPU1 states: 0% user 5% system 0% nice 95% idle 0% iowait 0% irq 0% softirq
   CPU2 states: 2% user 2% system 0% nice 96% idle 0% iowait 0% irq 0% softirq
   CPU3 states: 0% user 2% system 0% nice 98% idle 0% iowait 0% irq 0% softirq
   CPU4 states: 1% user 6% system 0% nice 93% idle 0% iowait 0% irq 0% softirq
   CPU5 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
   CPU6 states: 37% user 2% system 0% nice 61% idle 0% iowait 0% irq 0% softirq
   CPU7 states: 1% user 0% system 0% nice 99% idle 0% iowait 0% irq 0% softirq
   CPU8 states: 9% user 13% system 0% nice 78% idle 0% iowait 0% irq 0% softirq
   CPU9 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
   CPU10 states: 1% user 2% system 0% nice 97% idle 0% iowait 0% irq 0% softirq
   CPU11 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
   Memory: 16448692k total, 7867592k used (47.8%), 8208572k free (49.9%), 372528k freeable (2.3%)
   Average network usage: 1710 / 942 kbps in 1 minute, 18999 / 19647 kbps in 10 minutes, 15826 / 16285 kbps in 30 minutes
   Maximal network usage: 2804 / 1473 kbps in 1 minute, 27949 / 27754 kbps in 10 minutes, 31749 / 32829 kbps in 30 minutes
   Average sessions: 2864 sessions in 1 minute, 2262 sessions in 10 minutes, 1995 sessions in 30 minutes
   Maximal sessions: 2941 sessions in 1 minute, 2945 sessions in 10 minutes, 2945 sessions in 30 minutes
   Average session setup rate: 1 sessions per second in last 1 minute, 5 sessions per second in last 10 minutes, 7 sessions per second in last 30 minutes
   Maximal session setup rate: 20 sessions per second in last 1 minute, 214 sessions per second in last 10 minutes, 278 sessions per second in last 30 minutes
   Average NPU sessions: 48 sessions in last 1 minute, 45 sessions in last 10 minutes, 40 sessions in last 30 minutes
   Maximal NPU sessions: 52 sessions in last 1 minute, 59 sessions in last 10 minutes, 94 sessions in last 30 minutes
   Average nTurbo sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last 30 minutes
   Maximal nTurbo sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last 30 minutes
   Virus caught: 0 total in 1 minute
   IPS attacks blocked: 0 total in 1 minute
   Uptime: 0 days,  0 hours,  12 minutes

   Each cw_acd uses about 1% of the CPU.

Configuring multiple wpad_ac processes

The wpad-process-count allows you to configure multiple wpad_ac processes to handle WPA authentication requests. You can set the wpad-process-count to a non-zero value such as 4, so the FortiGate will have four child wpad daemons where each process can handle a small group of SSIDs.

To configure multiple wpad processes:

This example uses a FGT-101F that has a maximum wpad-process-count of 4.

1. Set the wpad-process-count under wireless-controller global:

   config wireless-controller global
     set wpad-process-count 4
   end

   Note that both wpad_ac and cw_acd processes are restarted when wpad-process-count is configured.

2. Verify the number of child wpad daemons created:

   # diagnose wpa wpad mp
       main process pid:      2221
       child process num:     4
               [1]:          2223
               [2]:          2225
               [3]:          2226
               [4]:          2227

3. Verify that VAPs with security modes of WPA-PSK, WPA-Enterprise, or radius-mac-auth are enabled and can be added to different wpad child daemons:

   # diagnose wpa wpad vap
   --------------------------   wpad[1]  -------------------------
   VAP number:    2
   VAP 0-10.10.24.20:35276-0-0 e0:22:ff:b2:19:30 state IDLE   
       AC socket: /tmp/cwCwAcSocket_1
       Radius MAC Auth:0
       wpa version: WPA2
       preauth: 1
       ssid: FOS_101f.br1
       key_mgmt: WPA-PSK WPA-FT-PSK
       rsn_pairwise: CCMP
       rsn_group: CCMP
   VAP 0-10.10.24.20:35276-1-0 e0:22:ff:b2:19:38 state IDLE   
       AC socket: /tmp/cwCwAcSocket_1
       Radius MAC Auth:0
       wpa version: WPA2
       preauth: 1
       ssid: FOS_101f.br.ent
       key_mgmt: WPA-EAP WPA-FT-EAP
       rsn_pairwise: CCMP
       rsn_group: CCMP
       auth: radius, server: wifi-radius
       Radius Auth NAS-IP: 0.0.0.0
       Radius Auth NAS-ID-TYPE: legacy
       Radius Auth NAS-ID: 10.10.24.20/35276-br2
   VAP number: 2              Radius VAP number: 1
   --------------------------   wpad[2]  -------------------------
   There is no any WPA enabled VAP!
   --------------------------   wpad[3]  -------------------------
   VAP number:    3
   VAP 0-10.6.30.254:25246-1-0 04:d5:90:b5:d7:e7 state IDLE   
       AC socket: /tmp/cwCwAcSocket_3
       Radius MAC Auth:0
       wpa version: WPA2
       preauth: 1
       ssid: FOS_101f.ssid1
       key_mgmt: WPA-PSK
       rsn_pairwise: CCMP
       rsn_group: CCMP
   VAP 0-10.6.30.254:5246-0-0 00:0c:e6:de:6f:31 state IDLE   
       AC socket: /tmp/cwCwAcSocket_3
       Radius MAC Auth:0
       wpa version: WPA2
       preauth: 1
       ssid: FOS_101f.br1
       key_mgmt: WPA-PSK WPA-FT-PSK
       rsn_pairwise: CCMP
       rsn_group: CCMP
   VAP 0-10.6.30.254:5246-1-0 00:0c:e6:de:6f:41 state IDLE   
       AC socket: /tmp/cwCwAcSocket_3
       Radius MAC Auth:0
       wpa version: WPA2
       preauth: 1
       ssid: 101f.ssid.ent
       key_mgmt: WPA-EAP
       rsn_pairwise: CCMP
       rsn_group: CCMP
       auth: radius, server: wifi-radius
       Radius Auth NAS-IP: 0.0.0.0
       Radius Auth NAS-ID-TYPE: legacy
       Radius Auth NAS-ID: 10.5.30.252/5246-101f.ssid.ent
   VAP number: 3              Radius VAP number: 1
   --------------------------   wpad[4]  -------------------------
   There is no any WPA enabled VAP!

4. Connect clients to the SSIDs and verify that each wpad child daemon can handle the authentication separately.

   # diagnose wpa wpad sta
   --------------------------   wpad[1]  -------------------------
   VAP number:    2
       STA=48:ee:0c:23:43:d1,   state: PTKINITDONE
   --------------------------   wpad[2]  -------------------------
   There is no any WPA enabled VAP!
   --------------------------   wpad[3]  -------------------------
   VAP number:    3
       STA=f8:e4:e3:d8:5e:af,   state: PTKINITDONE
   --------------------------   wpad[4]  -------------------------
   There is no any WPA enabled VAP!