Troubleshooting

This section includes basic troubleshooting techniques for a secure Fortinet wireless LAN including:

• strategies for troubleshooting Fortinet wireless devices
• how to avoid common misconfigurations
• solutions to connectivity issues
• capturing and analyzing wireless traffic
• wireless debug commands

The goal of this document is to provide you with practical knowledge that you can use to troubleshoot the FortiOS wireless controller and FortiAP devices. This includes how to use tools and apply CLI commands for maintenance and troubleshooting of your wireless network infrastructure, analyze problems per OSI layer, explore diagnostics for commissioning issues regarding at-client and access point connectivity problems, and understand the packet sniffer technique as a strong troubleshooting tool.

This section includes the following topics:

• FortiAP shell command through CAPWAP control tunnel
• Signal strength issues
• Throughput issues
• Throughput issues
• Connection issues
• General problems
• Packet sniffer
• Useful debugging commands

FortiAP shell command through CAPWAP control tunnel

Very often, the FortiAP in the field is behind a NAT device, and access to the FortiAP through Telnet or SSH is not available. As a troubleshooting enhancement, this feature allows an AP shell command up to 127-bytes sent to the FAP, and FAP will run this command, and return the results to the controller using the CAPWAP tunnel.

The maximum output from a command is limited to 4M, and the default output size is set to 32K.

The FortiAP will only report running results to the controller after the command is finished. If a new command is sent to the AP before the previous command is finished, the previous command will be canceled.

Enter the following:

diag w-c wlac wtpcmd wtp_ip wtp_port cmd [cmd-to-ap] cmd: run,show,showhex,clr,r&h,r&sh

• cmd-to-ap: any shell commands, but AP will not report results until the command is finished on the AP
• run: controller sends the ap-cmd to the FAP to run
• show: show current results reported by the AP in text
• showhex: show current results reported by the AP in hex
• clr: clear reported results
• r&s: run/show
• r&sh: run/showhex

Signal strength issues

Poor signal strength is possibly the most common customer complaint. Below you will learn where to begin identifying and troubleshooting poor signal strength, and learn what information you can obtain from the customer to help resolve signal strength issues.

Asymmetric power issue

Asymmetric power issues are a typical problem. Wireless is two-way communication; high power access points (APs) can usually transmit a long distance, however, the client's ability to transmit is usually not equal to that of the AP and, as such, cannot return transmission if the distance is too far.

Measuring signal strength in both directions

To solve an asymmetric power issue, measure the signal strength in both directions. APs usually have enough power to transmit long distances, but sometimes battery-powered clients have a reply signal that has less power, and therefore the AP cannot detect their signal.

It is recommended that you match the transmission power of the AP to the least powerful wireless client—around 10 decibels per milliwatt (dBm) for iPhones and 14dBm for most laptops.

Even if the signal is strong enough, other devices may be emitting radiation as well, causing interference. To identify the difference, read the client Rx strength from the FortiGate GUI (under Monitor > WiFi Client Monitor) or CLI.

The Signal Strength/Noise value provides the received signal strength indicator (RSSI) of the wireless client. For example, A value of -85dBm to -95dBm is equal to about 10dB levels; this is not a desirable signal strength. In the following screenshot, one of the clients is at 18dB, which is getting close to the perimeter of its range.

The Signal Strength/Noise value received from the FortiAP by clients, and vice versa, should be within the range of -20 dBm to -65 dBm.

You can also confirm the transmission (Tx) power of the controller on the AP profile (wtp-profile) and the FortiAP (iwconfig), and check the power management (auto-Tx) options.

Controller configured transmitting power - CLI:

config wireless-controller wtp-profile

config <radio>

show

(the following output is limited to power levels)

auto-power-level : enable

auto-power-high : 17

auto-power-low : 10

Actual FortiAP transmitting power - CLI:

iwconfig wlan00

Result:

wlan00 IEEE 802.11ng ESSID:"signal-check"

Mode:Master Frequency:2.412 GHz Access Point:<MAC add>

Bit Rate:130 Mb/s Tx-Power=28 dBm

Using FortiPlanner PRO with a site survey

The most thorough method to solve signal strength issues is to perform a site survey. To this end, Fortinet offers the FortiPlanner, downloadable at http://www.fortinet.com/resource_center/product_downloads.html.

Sample depiction of a site survey using FortiPlanner

The site survey provides you with optimal placement for your APs based on the variables in your environment. You must provide the site survey detailed information including a floor plan (to scale), structural materials, and more. It will allow you to place the APs on the map and adjust the radio bands and power levels while providing you with visual wireless coverage.

Below is a list of mechanisms for gathering further information on the client for Rx strength. The goal is to see how well the client is receiving the signal from the AP. You can also verify FortiAP signal strength on the client using WiFi client utilities, or third party utilities such as InSSIDer or MetaGeek Chanalyzer. You can get similar tools from the app stores on Android and iOS devices.

• Professional Site Survey software (Ekahau, Airmagnet survey Pro, FortiPlanner)
• InSSIDer
• On Windows: “netsh wlan show networks mode=bssid” (look for the BSSID, it's in % not in dBm!)
• On MacOS: Use the “airport” command:

“/System/Library/PrivateFrameworks/Apple80211.framework/Versions/A/Resources/airport” airport –s | grep <the_bssid> (live scan each time)

• On Droid: WiFiFoFum

Frequency interference

If the wireless signal seems to be strong but then periodically drops, this may be a symptom of frequency interference. Frequency interference is when another device also emits radio frequency using the same channel, co-channel, or adjacent channel, thereby overpowering or corrputing your signal. This is a common problem on a 2.4GHz network.

There are two types of interference: coherent and non-coherent.

• Coherent interference: a result of another device using the same channel as your AP, or poor planning of a wireless infrastructure (perhaps the other nearby APs are using the same channel or the signal strength is too high).
• Non-coherent interference: a result of other radio signals such as bluetooth, microwave, cordless phone, or (as in medical environments) x-ray machines.

Most common and simple solution for frequency interference is to change your operation channel. Typically, the channel can be set from 1 to 11 for the broadcast frequency, although you should always use channels 1, 6, and 11 on the 2.4GHz band.

Another solution, if it's appropriate for your location, is to use the 5GHz band instead.

MetaGeek Chanalyzer

You can perform a site survey using spectrum analysis at various points in your environment looking for signal versus interference/noise. MetaGeek Chanalyzer is an example of a third party utility which shows a noise threshold.

Note that a signal of -95dBm or less will be ignored by Fortinet wireless adapters.

Throughput issues

Sometimes communication issues can be caused by low performance.

Testing the link

You can identify delays or lost packets by sending ping packets from your wireless client. If there is more than 10ms of delay, there may be a problem with your wireless deployment, such as:

• a weak transmit signal from the client (the host does not reach the AP)
• the AP utilization is too high (your AP could be saturated with connected clients)
• interference (third party signal could degrade your AP or client's ability to detect signals between them)
• weak transmit power from the AP (the AP does not reach the host) -- not common in a properly deployed network, unless the client is too far away

Keep in mind that water will also cause a reduction in radio signal strength for those making use out of outdoor APs or wireless on a boat.

Performance testing

If the FortiAP gives bad throughput to the client, the link may drop. The throughput or performance can be measured on your smartphone with third party applications tool such as iPerf and jPerf.

Measuring file transfer speed

Another way to get a sense of your throughput issues is to measure the speed of a file transfer on your network. Create a test file at a specific size and measure the speed at which Windows measures the transfer. The command below will create a 50MB file.

• fsutil file createnew test.txt 52428800

The following image shows a network transfer speed of just over 24Mbps. The theoretical speed of 802.11g is 54Mbps, which is what this client is using. A wireless client is never likely to see the theoretical speed.

TKIP limitation

If you find that throughput is a problem, avoid WPA security encrypted with Temporal Key Integrity Protocol (TKIP) as it supports communications only at 54Mbps. Use WPA-2 AES instead.

Speeds are very much based on what the client computer can handle as well. The maximum client connection rate of 130Mbps is for 2.4GHz on a 2x2, or 300Mbps for 5Ghz on a 2x2 (using shortguard and channel bonding enabled).

If you want to get more than 54Mbps with 802.11n, do not use legacy TKIP, use CCMP instead. This is standard for legacy compatibility.

Preventing IP fragmentation in CAPWAP

TKIP is not the only possible source of decreased throughput. When a wireless client sends jumbo frames using a CAPWAP tunnel, it can result in data loss, jitter, and decreased throughput.

Using the following commands you can customize the uplink rates and downlink rates in the CAPWAP tunnel to prevent fragmentation and avoid data loss.

config wireless-controller wtp

edit new-wtp

set override-ip-fragment enable

set ip-fragment-preventing [tcp-mss-adjust | icmp-unreachable]

set tun-mtu-uplink [0 | 576 | 1500]

set tun-mtu-downlink [0 | 576 | 1500]

end

end

The default value is 0, however the recommended value will depend on the type of traffic. For example, IPsec in tunnel mode has 52 bytes of overhead, so you might use 1400 or less for uplink and downlink.

Slowness in the DTLS response

It's important to know all the elements involved in the CAPWAP association:

• Request
• Response
• DTLS
• Join
• Configuration

All of these are bidirectional. So if the DTLS response is slow, this might be the result of a configuration error. This issue can also be caused by a certificate during discovery response. You can read more about this in RFC 5416.

Connection issues

If the client has a connectivity issue that is not due to signal strength, the solution varies by the symptom.

Client connection issues

1. If client is unable to connect to FortiAP:
   • Make sure the client’s security and authentication settings match with FortiAP and check the certificates as well.
   • Try upgrading the Wi-Fi adapter driver and FortiGate/FortiAP firmware.
   • If other clients can connect, it could be interoperability; run debug commands and sniffer packets.
   • Look for rogue suppression by sniffing the wireless traffic and looking for the disconnect in the output (using the AP or wireless packet sniffer).
   • Try changing the IEEE protocol from 802.11n to 802.11bg or 802.11a only.
2. If the client drops and reconnects:
   • The client might be de-authenticating periodically. Check the sleep mode on the client.
   • The issue could be related to power-saver settings. The client may need to udpate drivers.
   • The issue could also be caused by flapping between APs. Check the roaming sensitivity settings on the client or the preferred wireless network settings on the client—if another WiFi network is available, the client may connect to it if it is a preferred network. Also, check the DHCP configuration as it may be an IP conflict.
3. If the client drops and never connects:
   • It could have roamed to another SSID, so check the standby and sleep modes.
   • You may need to bring the interface up and down.
4. If the client connects, but no IP address is acquired by the client:
   • Check the DHCP configuration and the network.
   • It could be a broadcast issue, so check the WEP encryption key and set a static IP address and VLANs.

Debug

You should also enable client debug on the controller for problematic clients to see the stage at which the client fails to connect. Try to connect from the problematic client and run the following debug command, which allows you to see the four-way handshake of the client association:

diagnose wireless-controller wlac sta_filter <client MAC address> 2

Example of a successful client connection:

The following is a sample debug output for the above command, with successful association/DHCP phases and PSK key exchange (identified in color):

FG600B3909600253 #

91155.197 <ih> IEEE 802.11 mgmt::assoc_req <== 30:46:9a:f9:fa:34 vap signal-check rId 0 wId 0 00:09:0f:f3:20:45

91155.197 <ih> IEEE 802.11 mgmt::assoc_resp ==> 30:46:9a:f9:fa:34 vap signal-check rId 0 wId 0 00:09:0f:f3:20:45 resp 0

91155.197 <cc> STA_CFG_REQ(15) sta 30:46:9a:f9:fa:34 add ==> ws (0-192.168.35.1:5246) rId 0 wId 0

91155.197 <dc> STA add 30:46:9a:f9:fa:34 vap signal-check ws (0-192.168.35.1:5246) rId 0 wId 0 bssid 00:09:0f:f3:20:45 NON-AUTH

91155.197 <cc> STA add 30:46:9a:f9:fa:34 vap signal-check ws (0-192.168.35.1:5246) rId 0 wId 0 00:09:0f:f3:20:45 sec WPA2 AUTO auth 0

91155.199 <cc> STA_CFG_RESP(15) 30:46:9a:f9:fa:34 <== ws (0-192.168.35.1:5246) rc 0 (Success)

91155.199 <eh> send 1/4 msg of 4-Way Handshake

91155.199 <eh>send IEEE 802.1X ver=1 type=3 (EAPOL_KEY) data len=95 replay cnt 1

91155.199 <eh> IEEE 802.1X (EAPOL 99B) ==> 30:46:9a:f9:fa:34 ws (0-192.168.35.1:5246) rId 0 wId 0 00:09:0f:f3:20:45

91155.217 <eh> IEEE 802.1X (EAPOL 121B) <== 30:46:9a:f9:fa:34 ws (0-192.168.35.1:5246) rId 0 wId 0 00:09:0f:f3:20:45

91155.217 <eh> recv IEEE 802.1X ver=1 type=3 (EAPOL_KEY) data len=117

91155.217 <eh> recv EAPOL-Key 2/4 Pairwise replay cnt 1

91155.218 <eh> send 3/4 msg of 4-Way Handshake

91155.218 <eh> send IEEE 802.1X ver=1 type=3 (EAPOL_KEY) data len=175 replay cnt 2

91155.218 <eh> IEEE 802.1X (EAPOL 179B) ==> 30:46:9a:f9:fa:34 ws (0-192.168.35.1:5246) rId 0 wId 0 00:09:0f:f3:20:45

91155.223 <eh> IEEE 802.1X (EAPOL 99B) <== 30:46:9a:f9:fa:34 ws (0-192.168.35.1:5246) rId 0 wId 0 00:09:0f:f3:20:45

91155.223 <eh> recv IEEE 802.1X ver=1 type=3 (EAPOL_KEY) data len=95

91155.223 <eh> recv EAPOL-Key 4/4 Pairwise replay cnt 2

91155.223 <dc> STA chg 30:46:9a:f9:fa:34 vap signal-check ws (0-192.168.35.1:5246) rId 0 wId 0 bssid 00:09:0f:f3:20:45 AUTH

91155.224 <cc> STA chg 30:46:9a:f9:fa:34 vap signal-check ws (0-192.168.35.1:5246) rId 0 wId 0 00:09:0f:f3:20:45 sec WPA2 AUTO auth 1

91155.224 <cc> STA_CFG_REQ(16) sta 30:46:9a:f9:fa:34 add key (len=16) ==> ws (0-192.168.35.1:5246) rId 0 wId 0

91155.226 <cc> STA_CFG_RESP(16) 30:46:9a:f9:fa:34 <== ws (0-192.168.35.1:5246) rc 0 (Success)

91155.226 <eh> ***pairwise key handshake completed*** (RSN)

91155.257 <dc> DHCP Request server 0.0.0.0 <== host ADMINFO-FD4I2HK mac 30:46:9a:f9:fa:34 ip 172.16.1.16

91155.258 <dc> DHCP Ack server 172.16.1.1 ==> host mac 30:46:9a:f9:fa:34 ip 172.16.1.16 mask 255.255.255.0 gw 172.16.1.1

where:

• orange represents the association phase,
• blue represents the PSK exchange,
• and green represents the DHCP phase.

It is important to note the messages for a correct association phase, four-way handshake, and DHCP phase.

Checking WiFi password

Admins can view plain text passwords (captive-portal-radius-secret and passphrase) under config wireless-controller vap.

Note that security must be set as a WPA-personal setting.

FortiAP connection issues

Clients are not the only device that can fail to connect, of course. A communication problem could arise from the FortiAP.

Some examples include:

• The FortiAP is not connecting to the wireless controller.
• One FortiAP intermittently disconnects and re-connects.
• All FortiAPs intermittently disconnect and re-connect.
• Unable to Telnet to FortiAP from controller/administrator workstation.

In the above cases:

• Check networking on the distribution system for all related FortiAPs.
• Check the authorization status of managed APs from the wireless controller.
• Restart the cw_acd process (Note: All APs will drop if you do this, and you may be troubleshooting just one AP).
• Check the controller crash log for any wireless controller daemon crash using the following command:

diagnose debug crashlog read

Debug

For a quick assessment of the association communication between the controller and the FortiAP, run the following sniffer command to see if you can verify that the AP is communicating to the controller by identifying the CAPWAP communication:

diagnose sniff packet <interface_name> “port 5246” 4

If you do not see this communication, then you can investigate the network or the settings on the AP to see why it is not reaching the controller.

The following command allows you to collect verbose output from the sniff that can be converted to a PCAP and viewed in Wireshark.

diagnose sniff packet <interface_name> “port 5246” 6 o l

The image below shows the beginning of the AP's association to the controller. You can see the discovery Request and Response at the top.

Throughout debugging it is recommended to:

• Enable Telnet login to the FortiAP device so that you can log in and issue local debugging commands:

config wireless-controller wtp

edit "<FortiAP_serial_number>"

set override-allowaccess {disable|enable}

set allowaccess {telnet | http | https | ssh}

end

• Try to connect to the wireless controller from the problematic FortiAP to verify routes exist.

• Enable wtp (FortiAP) debugging on the wireless controller for problematic FortiAPs to determine the point at which the FortiAP fails to connect:

diag wireless-controller wlac wtp_filter FP112B3X13000193 0-192.168.6.8:5246 2

(replace the serial number and IP address of the FortiAP)

di de console timestamp en

di de application cw_acd 0x7ff

di de en

Example of a successful AP and controller association:

The previous debug command provides similar output to the sample debug message below for a successful association between the FortiAP and the wireless controller. This includes the elements of the CAPWAP protocol; the Request, Response, DTLS, Join, and Configuration (identified in color). All of these are bi-directional, so if the DTLS response is slow, it may be an example of a configuration error.

56704.575 <msg> DISCOVERY_REQ (12) <== ws (0-192.168.35.1:5246)

56704.575 <msg> DISCOVERY_RESP (12) ==> ws (0-192.168.35.1:5246)

56707.575 <msg> DISCOVERY_REQ (13) <== ws (0-192.168.35.1:5246)

56707.575 <msg> DISCOVERY_RESP (13) ==> ws (0-192.168.35.1:5246)

56709.577 <aev> - CWAE_INIT_COMPLETE ws (0-192.168.35.1:5246)

56709.577 <aev> - CWAE_LISTENER_THREAD_READY ws (0-192.168.35.1:5246)

56709.577 <fsm> old CWAS_START(0) ev CWAE_INIT_COMPLETE(0) new CWAS_IDLE(1)

56709.577 <fsm> old CWAS_IDLE(1) ev CWAE_LISTENER_THREAD_READY(1) new CWAS_DTLS_SETUP(4)

56709.623 <aev> - CWAE_DTLS_PEER_ID_RECV ws (0-192.168.35.1:5246)

56709.623 <aev> - CWAE_DTLS_AUTH_PASS ws (0-192.168.35.1:5246)

56709.623 <aev> - CWAE_DTLS_ESTABLISHED ws (0-192.168.35.1:5246)

56709.623 <fsm> old CWAS_DTLS_SETUP(4) ev CWAE_DTLS_PEER_ID_RECV(7) new CWAS_DTLS_AUTHORIZE(2)

56709.623 <fsm> old CWAS_DTLS_AUTHORIZE(2) ev CWAE_DTLS_AUTH_PASS(3) new CWAS_DTLS_CONN(5)

56709.623 <fsm> old CWAS_DTLS_CONN(5) ev CWAE_DTLS_ESTABLISHED(8) new CWAS_JOIN(7)

56709.625 <msg> JOIN_REQ (14) <== ws (0-192.168.35.1:5246)

56709.625 <aev> - CWAE_JOIN_REQ_RECV ws (0-192.168.35.1:5246)

56709.626 <fsm> old CWAS_JOIN(7) ev CWAE_JOIN_REQ_RECV(12) new CWAS_JOIN(7)

56709.629 <msg> CFG_STATUS (15) <== ws (0-192.168.35.1:5246)

56709.629 <aev> - CWAE_CFG_STATUS_REQ ws (0-192.168.35.1:5246)

56709.629 <fsm> old CWAS_JOIN(7) ev CWAE_CFG_STATUS_REQ(13) new CWAS_CONFIG(8)

56710.178 <msg> CHG_STATE_EVENT_REQ (16) <== ws (0-192.168.35.1:5246)

56710.178 <aev> - CWAE_CHG_STATE_EVENT_REQ_RECV ws (0-192.168.35.1:5246)

56710.178 <fsm> old CWAS_CONFIG(8) ev CWAE_CHG_STATE_EVENT_REQ_RECV(23) new CWAS_DATA_CHAN_SETUP(10)

56710.220 <aev> - CWAE_DATA_CHAN_CONNECTED ws (0-192.168.35.1:5246)

56710.220 <msg> DATA_CHAN_KEEP_ALIVE <== ws (0-192.168.35.1:5246)

56710.220 <aev> - CWAE_DATA_CHAN_KEEP_ALIVE_RECV ws (0-192.168.35.1:5246)

56710.220 <msg> DATA_CHAN_KEEP_ALIVE ==> ws (0-192.168.35.1:5246)

56710.220 <fsm> old CWAS_DATA_CHAN_SETUP(10) ev CWAE_DATA_CHAN_CONNECTED(32) new CWAS_DATA_CHECK(11)

56710.220 <aev> - CWAE_DATA_CHAN_VERIFIED ws (0-192.168.35.1:5246)

56710.220 <fsm> old CWAS_DATA_CHECK(11) ev CWAE_DATA_CHAN_KEEP_ALIVE_RECV(35) new CWAS_DATA_CHECK(11)

56710.220 <fsm> old CWAS_DATA_CHECK(11) ev CWAE_DATA_CHAN_VERIFIED(36) new CWAS_RUN(12)

56710.228 <msg> WTP_EVENT_REQ (17) <== ws (0-192.168.35.1:5246)

56710.228 <aev> - CWAE_WTP_EVENT_REQ_RECV ws (0-192.168.35.1:5246)

56710.228 <fsm> old CWAS_RUN(12) ev CWAE_WTP_EVENT_REQ_RECV(42) new CWAS_RUN(12)

56710.230 <msg> CFG_UPDATE_RESP (1) <== ws (0-192.168.35.1:5246) rc 0 (Success)

56710.230 <aev> - CWAE_CFG_UPDATE_RESP_RECV ws (0-192.168.35.1:5246)

56710.230 <msg> WTP_EVENT_REQ (18) <== ws (0-192.168.35.1:5246)

56710.230 <aev> - CWAE_WTP_EVENT_REQ_RECV ws (0-192.168.35.1:5246)

56710.230 <fsm> old CWAS_RUN(12) ev CWAE_CFG_UPDATE_RESP_RECV(37) new CWAS_RUN(12)

56710.230 <fsm> old CWAS_RUN(12) ev CWAE_WTP_EVENT_REQ_RECV(42) new CWAS_RUN(12)

56710.231 <msg> WTP_EVENT_REQ (19) <== ws (0-192.168.35.1:5246)

56710.231 <aev> - CWAE_WTP_EVENT_REQ_RECV ws (0-192.168.35.1:5246)

56710.231 <fsm> old CWAS_RUN(12) ev CWAE_WTP_EVENT_REQ_RECV(42) new CWAS_RUN(12)

56710.232 <msg> CFG_UPDATE_RESP (2) <== ws (0-192.168.35.1:5246) rc 0 (Success)

56710.232 <aev> - CWAE_CFG_UPDATE_RESP_RECV ws (0-192.168.35.1:5246)

56710.232 <fsm> old CWAS_RUN(12) ev CWAE_CFG_UPDATE_RESP_RECV(37) new CWAS_RUN(12)

56710.233 <msg> WTP_EVENT_REQ (20) <== ws (0-192.168.35.1:5246)

56710.233 <aev> - CWAE_WTP_EVENT_REQ_RECV ws (0-192.168.35.1:5246)

56710.233 <fsm> old CWAS_RUN(12) ev CWAE_WTP_EVENT_REQ_RECV(42) new CWAS_RUN(12)

56712.253 < . > AC (2) -> WTP (0-192.168.35.1:5246) State: CWAS_RUN (12) accept 3 live 3 dbg 00000000 pkts 12493 0

56715.253 < . > AC (2) -> WTP (0-192.168.35.1:5246) State: CWAS_RUN (12) accept 3 live 6 dbg 00000000 pkts 12493 0

56718.253 < . > AC (2) -> WTP (0-192.168.35.1:5246) State: CWAS_RUN (12) accept 3 live 9 dbg 00000000 pkts 12493 0

56719.253 <aev> - CWAE_AC_ECHO_INTV_TMR_EXPIRE ws (0-192.168.35.1:5246)

56719.253 <fsm> old CWAS_RUN(12) ev CWAE_AC_ECHO_INTV_TMR_EXPIRE(39) new CWAS_RUN(12)

56719.576 <msg> ECHO_REQ (21) <== ws (0-192.168.35.1:5246)

56719.576 <aev> - CWAE_ECHO_REQ_RECV ws (0-192.168.35.1:5246)

56719.577 <fsm> old CWAS_RUN(12) ev CWAE_ECHO_REQ_RECV(27) new CWAS_RUN(12)

where:

• orange represents the Discovery phase,
• blue indicates that the control channels have been established using DTLS,
• green represents the access point Discovery and Join phase,
• purple represents the Clear Text channel,
• and pink indicates that the FortiAP successfully connected to the wireless controller.

General problems

Not all WiFi problems are related to signal strength, interference, or misconfiguration. The following OSI model identifies some of the more common issues per layer.

Best practices for troubleshooting vary depending on the affected layer (see below).

Common sources of wireless issues

Best practices for Layer 1

Common physical layer issues include:

• Weak received signal,
• WiFi capability: 802.11b, 1x1, 2x2,
• Co-channel WiFi interference,
• Side band WiFi interference,
• Non 802.11 noise (microwave ovens...).

To avoid physical layer issues:

• Determine RST (Receiver Sensitivity Threshold) for your device, or use -70dBm as a rule of thumb.
• Match AP TX output power to the client TX output power.

  Note: iPhone TX power is only 10dBm.

• Use DFS (Dynamic Frequency Selection) for high performance data 20/40 MHz.
• Use 5GHz UNII-1 & 3 (Non-DFS) bands with static channel assignment for latency-sensitive applications.
• Do not use 40MHz channels in 2.4 GHz band (channel bonding is not allowed in FortiOS).

Best practices for Layer 2

Common data link (MAC) layer issues include:

• Too many clients on a single channel (CSMA/CA) backoff,
• Too many high-priority traffic clients (WMM),
• Incorrect password or encryption settings,
• Too many beacons (in dense installs).

To avoid data link layer issues:

• Only use CCMP/AES (WPA2) encryption (not TKIP).
• In high density deployments, turn off SSID broadcast or turn down SSID rates. Review and possibly reduce the beacon interval.
• Determine the best cell size for applications:
  • For few users and low bandwidth latency sensitive applications, use high transmit power to create larger cells.
  • For high performance/high capacity installations, use lower transmit power to create smaller cells (set FortiPlanner at 10dBm TX power), but bear in mind that this will require more roaming.

Cells and co-channel interference

In high density deployments, multiple APs are used, and each one services an area called a cell. However, these cells can cause interference with each other. This is a common problem. The radio signal from one AP interferes with, or cancels out, the radio signal from another AP.

In the following diagram, note the interference zone created by one radio, causing interference on its neighbouring APs.

The interference zone can be twice the radius of the signal, and the signal at its edge can be -67dBm.

Reducing co-channel interference

For best results, use a 'honeycomb' pattern as a deployment strategy. The idea is to stagger repeated channels furthest from each other to avoid interference.

Best practices for Layer 3 and above

For TCP/IP layers and above, a common source of latency, or slowness in the wireless traffic, is too many broadcasts or multicasts. These types of issues can result from non-business and/or unwanted traffic.

To resolve issues at the TCP/IP layer and above:

• Identify business-critical applications.
• Use Application Control, Web Filtering, Traffic Shaping, and QoS to prioritize applications.
  • Identify unwanted traffic, high-bandwidth web-related traffic, and use Security Profiles.
  • Use the traffic shaper on a policy to rate-limit this traffic.

These configurations are performed directly on the FortiGate.

Packet sniffer

Capturing the traffic between the controller and the FortiAP can help you identify most FortiAP and client connection issues.

This section describes the following recommended packet sniffing techniques:

• CAPWAP packet sniffer
• Wireless traffic packet sniffer

CAPWAP packet sniffer

The first recommended technique consists of sniffing the CAPWAP traffic.

• Enable plain control on the controller and on the FortiAP to capture clear control traffic on UDP port 5246.
  • On the controller:

  diagnose wireless-controller wlac plain-ctl <FortiAP_serial_number> 1

  Result:

  WTP 0-FortiAP2223X11000107 Plain Control: enabled

  • On the FortiAP:

  cw_diag plain-ctl 1

  Result:

  Current Plain Control: enabled

Note that some issues are related to the keep-alive for control and data channel.

Data traffic on UDP port 5247 is not encrypted. The data itself is encrypted by the wireless security mechanism.

Data traffic is helpful to troubleshoot most of the issues related to station association, EAP authentication, WPA key exchange, roaming, and FortiAP configuration.

You can also set up a host or server to which you can forward the CAPWAP traffic:

1. Configure the host/server to which CAPWAP traffic is forwarded:

   diagnose wireless-controller wlac sniff-cfg <Host_IP_address> 88888

   Result:

   Current Sniff Server: 192.168.25.41, 23352

2. Choose which traffic to capture, the interface to which the FortiAP is connected, and the FortiAP’s serial number:

   diagnose wireless-controller wlac sniff <interface_name> <FortiAP_serial_number> 2

   Result:

   WTP 0-FortiAP2223X11000107 Sniff: intf port2 enabled (control and data message)

   In the above syntax, the '2' captures the control and data message—'1' would capture only the control message, and '0' would disable it.

3. Run Wireshark on the host/server to capture CAPWAP traffic from the controller.
4. Decode the traffic as IP to check inner CAPWAP traffic.

Example CAPWAP packet capture

The following image shows an example of a CAPWAP packet capture, where you can see: the Layer 2 header; the sniffed traffic encapsulated into Internet Protocol for transport; CAPWAP encapsulated into UDP for sniffer purpose and encapsulated into IP; CAPWAP control traffic on UDP port 5246; and CAPWAP payload.

Wireless traffic packet sniffer

The second recommended technique consists of sniffing the wireless traffic directly 'on the air' using your FortiAP.

Wireless traffic packet capture

Packet captures are useful for troubleshooting all wireless client related issues because you can verify data rate and 802.11 parameters, such as radio capabilities, and determine issues with wireless signal strength, interference, or congestion on the network.

A radio can only capture one frequency at a time; one of the radios is set to sniffer mode depending on the traffic or channel required. You must use two FortiAPs to capture both frequencies at the same time.

• Set a radio on the FortiAP to monitor mode.

  iwconfig wlan10

  Result:

  wlan10 IEEE 802.11na ESSID:""

  Mode:Monitor Frequency:5.18 GHz Access Point: Not-Associated

• The capture file is stored under the temp directory as wl_sniff.pcap

  /tmp/wl_sniff.cap

• Remember that the capture file is only stored temporarily. If you want to save it, upload it to a TFTP server before rebooting or changing the radio settings.
• The command cp wl_sniff.cap newname.pcap allows you to rename the file.
• Rather than TFTP the file, you can also log in to the AP and retrive the file via the web interface. Move the file using the command: mv name /usr/www

  You can verify the file was moved using the command cd/usr/www and then browsing to: <fortiAP_IP>/filename

Syntax

The following syntax demonstrates how to set the radio to sniffer mode (configurable from the CLI only). Sniffer mode provides options to filter for specific traffic to capture. Notice that you can determine the buffer size, which channel to sniff, the AP's MAC address, and select if you want to sniff the beacons, probes, controls, and data channels.

configure wireless-controller wtp-profile

edit <profile_name>

configure <radio>

set mode sniffer

set ap-sniffer-bufsize 32

set ap-sniffer-chan 1

set ap-sniffer-addr 00:00:00:00:00:00

set ap-sniffer-mgmt-beacon enable

set ap-sniffer-mgmt-probe enable

set ap-sniffer-mgmt-other enable

set ap-sniffer-ctl enable

set ap-sniffer-data enable

end

end

Once you've performed the previous CLI configuration, you'll be able to see the packet sniffer mode selected in the GUI dashboard under WiFi & Switch Controller > FortiAP Profiles and WiFi & Switch Controller > Managed FortiAPs. Bear in mind that if you change the mode from the GUI, you'll have to return to the CLI to re-enable the Sniffer mode.

To disable the sniffer profile in the CLI, use the following commands:

config wireless-controller wtp-profile

edit <profile_name>

config <radio>

set ap-sniffer-mgmt-beacon disable

set ap-sniffer-mgmt-probe disable

set ap-sniffer-mgmt-other disable

set ap-sniffer-ctl disable

set ap-sniffer-data disable

end

end

If you change the radio mode before sending the file wl_sniff.cap to an external TFTP, the file is deleted and you lose your packet capture.

Example AP packet capture

The following image shows an example of the AP packet capture. Note the capture header showing channel 36; the beacon frame; the source, destination, and BSSID of the beacon frame; and the SSID of the beacon frame.

Useful debugging commands

For a comprehensive list of useful debug options you can use the following help commands on the controller:

diagnose wireless-controller wlac help

(this command lists the options available that pertain to the wireless controller)

diagnose wireless-controller wlwtp help

(this command lists the options available that pertain to the AP)

Sample outputs

Syntax

diagnose wireless-controller wlac -c vap

(this command lists the information about the virtual access point, including its MAC address, the BSSID, its SSID, the interface name, and the IP address of the APs that are broadcasting it)

Result:

bssid ssid intf vfid:ip-port rId wId

00:09:0f:d6:cb:12 Office Office ws (0-192.168.3.33:5246) 0 0

00:09:0f:e6:6b:12 Office Office ws (0-192.168.1.61:5246) 0 0

06:0e:8e:27:dc:48 Office Office ws (0-192.168.3.36:5246) 0 0

0a:09:0f:d6:cb:12 public publicAP ws (0-192.168.3.33:5246) 0 1

Syntax

diagnose wireless-controller wlac -c darrp

(this command lists the information pertaining to the radio resource provisioning statistics, including the AP serial number, the number of channels set to choose from, and the operation channel. Note that the 5GHz band is not available on these APs listed)

Result:

wtp_id rId base_mac index nr_chan vfid 5G oper_chan age

FAP22A3U10600400 0 00:09:0f:d6:cb:12 0 3 0 No 1 87588

FW80CM3910601176 0 06:0e:8e:27:dc:48 1 3 0 No 6 822

Support for extension information for wtp, vap, and station

You can enable or disable extension information at wtp-profile, and use the diagnose option below to print out the detail of extension information.

Syntax

config wireless-controller wtp-profile

edit test

set lldp [enable | disable]

set ext-info [enable | disable] --> Enable/disable station/VAP/radio extension information.

end

end

diagnose wireless-controller wlac -d [wtp | vap | sta]

where:

• wlac -d wtp [SN|name] [reset] --> list or reset wtp info(data)
• wlac -d vap [bssid] [reset] --> list or reset vap info(data)
• wlac -d sta [mac] [reset] --> list or reset sta info(data)