Fortinet black logo

New Features

Implement multi-processing for wireless daemon for large-scale FortiAP management 7.2.4

Implement multi-processing for wireless daemon for large-scale FortiAP management 7.2.4

Note

This information is also available in the FortiWiFi and FortiAP 7.2 Configuration Guide:

This release adds the ability to configure multiple processors for the wireless daemon (cw_acd) by leveraging multi-core CPU to scale large numbers of FortiAP per FortiGate Controller.

The new acd-process-count option allows users to specify the number of cw_acd processes to manage FortiAPs. For FortiGate managed APs, it splits the total number of FortiAPs into smaller groups where each cw_acd process manages a group. The cw_acd process won't be as overloaded, and if one cw_acd has an issue, it only affects that group of FortiAPs instead of all the FortiAPs managed by the FortiGate.

The maximum value you can specify in acd-process-count varies according to the wireless-controller.wtp in table size from different platforms.

wireless-controller.wtp

Maximum acd-process-count

8192

32

4096

16

512-1024

8

128-256

4

16-64

2

To configure multiple cw_acd processes:

In this example, there are about 1300 FortiAPs managed by a FortiGate with 16 cw_acd processes to handle all the FortiAPs.

  1. Set the acd-process-count to 0 in wireless-controller global:

    config wireless-controller global
      set acd-process-count 16
    end
    
  2. Verify the number of FortiAPs managed per cw_acd:

    # diagnose wireless wlac -c mpmt
    acd main  process pid     : 321
    acd child process count   : 16
          idx=01 pid= 321 sl=N/A                  sm=/tmp/cwAcSock_mpmt_mngr sh=                                  
          idx=02 pid= 376 sl=/tmp/cwCwAcSocket_data sm=/tmp/cwAcSock_mpmt_data sh=                                  
        * idx=03 pid= 377 sl=/tmp/cwCwAcSocket    sm=/tmp/cwAcSock_mpmt    sh=                                   ws_cnt=1305 1283(RUN)   86(cfg) 1189(oper)
          idx=04 pid= 401 sl=/tmp/cwCwAcSocket_1  sm=/tmp/cwAcSock_mpmt_1  sh=/tmp/hasync_to_cw_acd_unix_sock_1  ws_cnt=80     77(RUN)    4(cfg)   70(oper)
          idx=05 pid= 402 sl=/tmp/cwCwAcSocket_2  sm=/tmp/cwAcSock_mpmt_2  sh=/tmp/hasync_to_cw_acd_unix_sock_2  ws_cnt=78     77(RUN)    5(cfg)   72(oper)
          idx=06 pid= 403 sl=/tmp/cwCwAcSocket_3  sm=/tmp/cwAcSock_mpmt_3  sh=/tmp/hasync_to_cw_acd_unix_sock_3  ws_cnt=91     89(RUN)    6(cfg)   83(oper)
          idx=07 pid= 404 sl=/tmp/cwCwAcSocket_4  sm=/tmp/cwAcSock_mpmt_4  sh=/tmp/hasync_to_cw_acd_unix_sock_4  ws_cnt=93     92(RUN)    6(cfg)   84(oper)
          idx=08 pid= 405 sl=/tmp/cwCwAcSocket_5  sm=/tmp/cwAcSock_mpmt_5  sh=/tmp/hasync_to_cw_acd_unix_sock_5  ws_cnt=92     91(RUN)    7(cfg)   84(oper)
          idx=09 pid= 406 sl=/tmp/cwCwAcSocket_6  sm=/tmp/cwAcSock_mpmt_6  sh=/tmp/hasync_to_cw_acd_unix_sock_6  ws_cnt=92     91(RUN)   10(cfg)   81(oper)
          idx=10 pid= 407 sl=/tmp/cwCwAcSocket_7  sm=/tmp/cwAcSock_mpmt_7  sh=/tmp/hasync_to_cw_acd_unix_sock_7  ws_cnt=78     77(RUN)    4(cfg)   73(oper)
          idx=11 pid= 408 sl=/tmp/cwCwAcSocket_8  sm=/tmp/cwAcSock_mpmt_8  sh=/tmp/hasync_to_cw_acd_unix_sock_8  ws_cnt=76     74(RUN)    5(cfg)   69(oper)
          idx=12 pid= 409 sl=/tmp/cwCwAcSocket_9  sm=/tmp/cwAcSock_mpmt_9  sh=/tmp/hasync_to_cw_acd_unix_sock_9  ws_cnt=82     79(RUN)    9(cfg)   70(oper)
          idx=13 pid= 410 sl=/tmp/cwCwAcSocket_10 sm=/tmp/cwAcSock_mpmt_10 sh=/tmp/hasync_to_cw_acd_unix_sock_10 ws_cnt=76     74(RUN)    4(cfg)   70(oper)
          idx=14 pid= 411 sl=/tmp/cwCwAcSocket_11 sm=/tmp/cwAcSock_mpmt_11 sh=/tmp/hasync_to_cw_acd_unix_sock_11 ws_cnt=80     77(RUN)    6(cfg)   70(oper)
          idx=15 pid= 412 sl=/tmp/cwCwAcSocket_12 sm=/tmp/cwAcSock_mpmt_12 sh=/tmp/hasync_to_cw_acd_unix_sock_12 ws_cnt=78     78(RUN)    5(cfg)   72(oper)
          idx=16 pid= 413 sl=/tmp/cwCwAcSocket_13 sm=/tmp/cwAcSock_mpmt_13 sh=/tmp/hasync_to_cw_acd_unix_sock_13 ws_cnt=76     76(RUN)    5(cfg)   71(oper)
          idx=17 pid= 414 sl=/tmp/cwCwAcSocket_14 sm=/tmp/cwAcSock_mpmt_14 sh=/tmp/hasync_to_cw_acd_unix_sock_14 ws_cnt=78     78(RUN)    5(cfg)   73(oper)
          idx=18 pid= 415 sl=/tmp/cwCwAcSocket_15 sm=/tmp/cwAcSock_mpmt_15 sh=/tmp/hasync_to_cw_acd_unix_sock_15 ws_cnt=76     75(RUN)    1(cfg)   74(oper)
          idx=19 pid= 416 sl=/tmp/cwCwAcSocket_16 sm=/tmp/cwAcSock_mpmt_16 sh=/tmp/hasync_to_cw_acd_unix_sock_16 ws_cnt=79     78(RUN)    4(cfg)   73(oper)
    Curr Time: 683

    Each cw_acd process handles a small number of FortiAPs, about 90.

  3. Verify the CPU used by cw_acd:

    # diagnose system top 5 30
    Run Time:  0 days, 0 hours and 11 minutes
    5U, 0N, 4S, 91I, 0WA, 0HI, 0SI, 0ST; 16063T, 8236F
                csfd      340      R      87.5     1.3    8
              cw_acd      377      S      12.9     6.5    6
              flpold      336      S       1.9     0.0    1
              cu_acd      325      S       1.4     0.1    0
              cw_acd      402      S       0.9     0.9    6
              cw_acd      401      S       0.9     0.9    2
              cw_acd      412      S       0.4     1.2    8
              cw_acd      404      S       0.4     1.0    10
              cw_acd      405      S       0.4     1.0    4
              cw_acd      403      S       0.4     1.0    2
              cw_acd      409      S       0.4     0.9    4
              cw_acd      408      S       0.4     0.9    6
              cw_acd      414      S       0.4     0.9    2
              cw_acd      413      S       0.4     0.9    8
                node      275      S       0.4     0.3    4
             miglogd      295      S       0.4     0.3    10
                 cid      345      S       0.4     0.2    6
             miglogd      391      S       0.4     0.2    6
             miglogd      389      S       0.4     0.2    8
           forticron      282      S       0.4     0.1    6
              flcfgd      326      S       0.4     0.1    9
          fortilinkd      324      S       0.4     0.0    0
              cw_acd      376      S       0.0     2.8    3
              cw_acd      406      S       0.0     1.0    6
              cw_acd      411      S       0.0     0.9    10
              cw_acd      416      S       0.0     0.9    8
              cw_acd      407      S       0.0     0.9    2
              cw_acd      415      S       0.0     0.9    0
              cw_acd      410      S       0.0     0.8    4
             cmdbsvr      237      S       0.0     0.7    0
    
    # get system performance status 
    CPU states: 5% user 3% system 0% nice 92% idle 0% iowait 0% irq 0% softirq
    CPU0 states: 6% user 4% system 0% nice 90% idle 0% iowait 0% irq 0% softirq
    CPU1 states: 0% user 5% system 0% nice 95% idle 0% iowait 0% irq 0% softirq
    CPU2 states: 2% user 2% system 0% nice 96% idle 0% iowait 0% irq 0% softirq
    CPU3 states: 0% user 2% system 0% nice 98% idle 0% iowait 0% irq 0% softirq
    CPU4 states: 1% user 6% system 0% nice 93% idle 0% iowait 0% irq 0% softirq
    CPU5 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
    CPU6 states: 37% user 2% system 0% nice 61% idle 0% iowait 0% irq 0% softirq
    CPU7 states: 1% user 0% system 0% nice 99% idle 0% iowait 0% irq 0% softirq
    CPU8 states: 9% user 13% system 0% nice 78% idle 0% iowait 0% irq 0% softirq
    CPU9 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
    CPU10 states: 1% user 2% system 0% nice 97% idle 0% iowait 0% irq 0% softirq
    CPU11 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
    Memory: 16448692k total, 7867592k used (47.8%), 8208572k free (49.9%), 372528k freeable (2.3%)
    Average network usage: 1710 / 942 kbps in 1 minute, 18999 / 19647 kbps in 10 minutes, 15826 / 16285 kbps in 30 minutes
    Maximal network usage: 2804 / 1473 kbps in 1 minute, 27949 / 27754 kbps in 10 minutes, 31749 / 32829 kbps in 30 minutes
    Average sessions: 2864 sessions in 1 minute, 2262 sessions in 10 minutes, 1995 sessions in 30 minutes
    Maximal sessions: 2941 sessions in 1 minute, 2945 sessions in 10 minutes, 2945 sessions in 30 minutes
    Average session setup rate: 1 sessions per second in last 1 minute, 5 sessions per second in last 10 minutes, 7 sessions per second in last 30 minutes
    Maximal session setup rate: 20 sessions per second in last 1 minute, 214 sessions per second in last 10 minutes, 278 sessions per second in last 30 minutes
    Average NPU sessions: 48 sessions in last 1 minute, 45 sessions in last 10 minutes, 40 sessions in last 30 minutes
    Maximal NPU sessions: 52 sessions in last 1 minute, 59 sessions in last 10 minutes, 94 sessions in last 30 minutes
    Average nTurbo sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last 30 minutes
    Maximal nTurbo sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last 30 minutes
    Virus caught: 0 total in 1 minute
    IPS attacks blocked: 0 total in 1 minute
    Uptime: 0 days,  0 hours,  12 minutes

    Each cw_acd uses about 1% of the CPU.

Implement multi-processing for wireless daemon for large-scale FortiAP management 7.2.4

Note

This information is also available in the FortiWiFi and FortiAP 7.2 Configuration Guide:

This release adds the ability to configure multiple processors for the wireless daemon (cw_acd) by leveraging multi-core CPU to scale large numbers of FortiAP per FortiGate Controller.

The new acd-process-count option allows users to specify the number of cw_acd processes to manage FortiAPs. For FortiGate managed APs, it splits the total number of FortiAPs into smaller groups where each cw_acd process manages a group. The cw_acd process won't be as overloaded, and if one cw_acd has an issue, it only affects that group of FortiAPs instead of all the FortiAPs managed by the FortiGate.

The maximum value you can specify in acd-process-count varies according to the wireless-controller.wtp in table size from different platforms.

wireless-controller.wtp

Maximum acd-process-count

8192

32

4096

16

512-1024

8

128-256

4

16-64

2

To configure multiple cw_acd processes:

In this example, there are about 1300 FortiAPs managed by a FortiGate with 16 cw_acd processes to handle all the FortiAPs.

  1. Set the acd-process-count to 0 in wireless-controller global:

    config wireless-controller global
      set acd-process-count 16
    end
    
  2. Verify the number of FortiAPs managed per cw_acd:

    # diagnose wireless wlac -c mpmt
    acd main  process pid     : 321
    acd child process count   : 16
          idx=01 pid= 321 sl=N/A                  sm=/tmp/cwAcSock_mpmt_mngr sh=                                  
          idx=02 pid= 376 sl=/tmp/cwCwAcSocket_data sm=/tmp/cwAcSock_mpmt_data sh=                                  
        * idx=03 pid= 377 sl=/tmp/cwCwAcSocket    sm=/tmp/cwAcSock_mpmt    sh=                                   ws_cnt=1305 1283(RUN)   86(cfg) 1189(oper)
          idx=04 pid= 401 sl=/tmp/cwCwAcSocket_1  sm=/tmp/cwAcSock_mpmt_1  sh=/tmp/hasync_to_cw_acd_unix_sock_1  ws_cnt=80     77(RUN)    4(cfg)   70(oper)
          idx=05 pid= 402 sl=/tmp/cwCwAcSocket_2  sm=/tmp/cwAcSock_mpmt_2  sh=/tmp/hasync_to_cw_acd_unix_sock_2  ws_cnt=78     77(RUN)    5(cfg)   72(oper)
          idx=06 pid= 403 sl=/tmp/cwCwAcSocket_3  sm=/tmp/cwAcSock_mpmt_3  sh=/tmp/hasync_to_cw_acd_unix_sock_3  ws_cnt=91     89(RUN)    6(cfg)   83(oper)
          idx=07 pid= 404 sl=/tmp/cwCwAcSocket_4  sm=/tmp/cwAcSock_mpmt_4  sh=/tmp/hasync_to_cw_acd_unix_sock_4  ws_cnt=93     92(RUN)    6(cfg)   84(oper)
          idx=08 pid= 405 sl=/tmp/cwCwAcSocket_5  sm=/tmp/cwAcSock_mpmt_5  sh=/tmp/hasync_to_cw_acd_unix_sock_5  ws_cnt=92     91(RUN)    7(cfg)   84(oper)
          idx=09 pid= 406 sl=/tmp/cwCwAcSocket_6  sm=/tmp/cwAcSock_mpmt_6  sh=/tmp/hasync_to_cw_acd_unix_sock_6  ws_cnt=92     91(RUN)   10(cfg)   81(oper)
          idx=10 pid= 407 sl=/tmp/cwCwAcSocket_7  sm=/tmp/cwAcSock_mpmt_7  sh=/tmp/hasync_to_cw_acd_unix_sock_7  ws_cnt=78     77(RUN)    4(cfg)   73(oper)
          idx=11 pid= 408 sl=/tmp/cwCwAcSocket_8  sm=/tmp/cwAcSock_mpmt_8  sh=/tmp/hasync_to_cw_acd_unix_sock_8  ws_cnt=76     74(RUN)    5(cfg)   69(oper)
          idx=12 pid= 409 sl=/tmp/cwCwAcSocket_9  sm=/tmp/cwAcSock_mpmt_9  sh=/tmp/hasync_to_cw_acd_unix_sock_9  ws_cnt=82     79(RUN)    9(cfg)   70(oper)
          idx=13 pid= 410 sl=/tmp/cwCwAcSocket_10 sm=/tmp/cwAcSock_mpmt_10 sh=/tmp/hasync_to_cw_acd_unix_sock_10 ws_cnt=76     74(RUN)    4(cfg)   70(oper)
          idx=14 pid= 411 sl=/tmp/cwCwAcSocket_11 sm=/tmp/cwAcSock_mpmt_11 sh=/tmp/hasync_to_cw_acd_unix_sock_11 ws_cnt=80     77(RUN)    6(cfg)   70(oper)
          idx=15 pid= 412 sl=/tmp/cwCwAcSocket_12 sm=/tmp/cwAcSock_mpmt_12 sh=/tmp/hasync_to_cw_acd_unix_sock_12 ws_cnt=78     78(RUN)    5(cfg)   72(oper)
          idx=16 pid= 413 sl=/tmp/cwCwAcSocket_13 sm=/tmp/cwAcSock_mpmt_13 sh=/tmp/hasync_to_cw_acd_unix_sock_13 ws_cnt=76     76(RUN)    5(cfg)   71(oper)
          idx=17 pid= 414 sl=/tmp/cwCwAcSocket_14 sm=/tmp/cwAcSock_mpmt_14 sh=/tmp/hasync_to_cw_acd_unix_sock_14 ws_cnt=78     78(RUN)    5(cfg)   73(oper)
          idx=18 pid= 415 sl=/tmp/cwCwAcSocket_15 sm=/tmp/cwAcSock_mpmt_15 sh=/tmp/hasync_to_cw_acd_unix_sock_15 ws_cnt=76     75(RUN)    1(cfg)   74(oper)
          idx=19 pid= 416 sl=/tmp/cwCwAcSocket_16 sm=/tmp/cwAcSock_mpmt_16 sh=/tmp/hasync_to_cw_acd_unix_sock_16 ws_cnt=79     78(RUN)    4(cfg)   73(oper)
    Curr Time: 683

    Each cw_acd process handles a small number of FortiAPs, about 90.

  3. Verify the CPU used by cw_acd:

    # diagnose system top 5 30
    Run Time:  0 days, 0 hours and 11 minutes
    5U, 0N, 4S, 91I, 0WA, 0HI, 0SI, 0ST; 16063T, 8236F
                csfd      340      R      87.5     1.3    8
              cw_acd      377      S      12.9     6.5    6
              flpold      336      S       1.9     0.0    1
              cu_acd      325      S       1.4     0.1    0
              cw_acd      402      S       0.9     0.9    6
              cw_acd      401      S       0.9     0.9    2
              cw_acd      412      S       0.4     1.2    8
              cw_acd      404      S       0.4     1.0    10
              cw_acd      405      S       0.4     1.0    4
              cw_acd      403      S       0.4     1.0    2
              cw_acd      409      S       0.4     0.9    4
              cw_acd      408      S       0.4     0.9    6
              cw_acd      414      S       0.4     0.9    2
              cw_acd      413      S       0.4     0.9    8
                node      275      S       0.4     0.3    4
             miglogd      295      S       0.4     0.3    10
                 cid      345      S       0.4     0.2    6
             miglogd      391      S       0.4     0.2    6
             miglogd      389      S       0.4     0.2    8
           forticron      282      S       0.4     0.1    6
              flcfgd      326      S       0.4     0.1    9
          fortilinkd      324      S       0.4     0.0    0
              cw_acd      376      S       0.0     2.8    3
              cw_acd      406      S       0.0     1.0    6
              cw_acd      411      S       0.0     0.9    10
              cw_acd      416      S       0.0     0.9    8
              cw_acd      407      S       0.0     0.9    2
              cw_acd      415      S       0.0     0.9    0
              cw_acd      410      S       0.0     0.8    4
             cmdbsvr      237      S       0.0     0.7    0
    
    # get system performance status 
    CPU states: 5% user 3% system 0% nice 92% idle 0% iowait 0% irq 0% softirq
    CPU0 states: 6% user 4% system 0% nice 90% idle 0% iowait 0% irq 0% softirq
    CPU1 states: 0% user 5% system 0% nice 95% idle 0% iowait 0% irq 0% softirq
    CPU2 states: 2% user 2% system 0% nice 96% idle 0% iowait 0% irq 0% softirq
    CPU3 states: 0% user 2% system 0% nice 98% idle 0% iowait 0% irq 0% softirq
    CPU4 states: 1% user 6% system 0% nice 93% idle 0% iowait 0% irq 0% softirq
    CPU5 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
    CPU6 states: 37% user 2% system 0% nice 61% idle 0% iowait 0% irq 0% softirq
    CPU7 states: 1% user 0% system 0% nice 99% idle 0% iowait 0% irq 0% softirq
    CPU8 states: 9% user 13% system 0% nice 78% idle 0% iowait 0% irq 0% softirq
    CPU9 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
    CPU10 states: 1% user 2% system 0% nice 97% idle 0% iowait 0% irq 0% softirq
    CPU11 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
    Memory: 16448692k total, 7867592k used (47.8%), 8208572k free (49.9%), 372528k freeable (2.3%)
    Average network usage: 1710 / 942 kbps in 1 minute, 18999 / 19647 kbps in 10 minutes, 15826 / 16285 kbps in 30 minutes
    Maximal network usage: 2804 / 1473 kbps in 1 minute, 27949 / 27754 kbps in 10 minutes, 31749 / 32829 kbps in 30 minutes
    Average sessions: 2864 sessions in 1 minute, 2262 sessions in 10 minutes, 1995 sessions in 30 minutes
    Maximal sessions: 2941 sessions in 1 minute, 2945 sessions in 10 minutes, 2945 sessions in 30 minutes
    Average session setup rate: 1 sessions per second in last 1 minute, 5 sessions per second in last 10 minutes, 7 sessions per second in last 30 minutes
    Maximal session setup rate: 20 sessions per second in last 1 minute, 214 sessions per second in last 10 minutes, 278 sessions per second in last 30 minutes
    Average NPU sessions: 48 sessions in last 1 minute, 45 sessions in last 10 minutes, 40 sessions in last 30 minutes
    Maximal NPU sessions: 52 sessions in last 1 minute, 59 sessions in last 10 minutes, 94 sessions in last 30 minutes
    Average nTurbo sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last 30 minutes
    Maximal nTurbo sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last 30 minutes
    Virus caught: 0 total in 1 minute
    IPS attacks blocked: 0 total in 1 minute
    Uptime: 0 days,  0 hours,  12 minutes

    Each cw_acd uses about 1% of the CPU.