Diagnosing SSL/TLS handshake failures

If the client is attempting to make an HTTPS connection, but the attempt fails after the TCP connection has been initiated, during negotiation, the problem may be with SSL/TLS.

1. Check the errors displayed on SSL/TLS client/browser.

   A SSL/TLS client or browser usually displays the SSL error code it encountered. Once can check and try to resolve them based on the specific error message.

   Common symptoms may include error messages such as:

   • ssl_error_no_cypher_overlap (Mozilla Firefox 9.0.1)

   • Error 113 (net::ERROR_SSL_VERSION_OR_CIPHER_MISMATCH): Unknown error (Google Chrome 16.0.912.75 m)

     You can search on Internet to find solutions for those common error messages and check if any problem is caused by client sides:

     How to Fix SSL Error on Firefox Browser? - A Complete List (comparecheapssl.com)

     However, we can often check SSL error codes on FortiWeb attack logs as below.

2. Check detailed SSL errors in attack logs.

   SSL errors will be displayed in attack logs once “Ignore SSL Erros” is disabled by either method as below:

   • Disable Ignore SSL Errors in Log&Report > Log Config > Other Log Settings

     

   • Check detailed SSL errors in attack logs through:

     conf log attack-log

     set no-ssl-error disable

     end

     For instance, a log is like below:

     "SSL Error(394) - dh key too small".

     This error means the length of dh pukkey in the ssl "server key exchange" is short, that is to say, it’s too weak and insecure, and the higher version will consider closing it.

     Please check the error code/message listed:

     SSL/TLS error messages (fortinet.com)

     https://mantis.fortinet.com/file_download.php?file_id=666300&type=bug

3. If SSL error is related to protocol or cipher suite, you can use OpenSSL to confirm which protocol & ciphers are supported:

   • Check whether the backend server or FortiWeb supports strong (HIGH) encryption:

     openssl s_client -connect example.com:443 -cipher HIGH

   • Check whether the backend server or FortiWeb supports old versions such as SSL 1.1:

     openssl s_client -tls1.1 -connect example.com:443

     If you have checked the errors but are not sure about the cause, please collect diagnose logs and also capture packets at the same time, then send to developers for further investigation:

4. Diagnose debug flow can output error during SSL handshake:

   diagnose debug reset

   diagnose debug enable

   diagnose debug timestamp enable

   diagnose debug flow filter flow-detail 7

   diagnose debug flow filter server-ip 192.168.12.12 #The VIP in RP mode or the real server IP in TP/TI mode

   diagnose debug flow filter client-ip 192.168.12.1

   diagnose debug flow trace start

   diagnose debug flow trace stop

   FortiWeb # <04:05:24>[work 0][flow] policy SP_01 create service:0x7fae5d14ce28

   <04:05:24>[work 0][flow] ssn 1120 policy SP_01 strm 0 dir 0 subclient 0 create HTTP substream:0x7fae5d195328

   <04:05:24>[work 0][flow] ssn 1120 policy SP_01 strm 0 dir 0 subclient 0 create stream:0x7fae5e05d908

   <04:05:24>[work 0][flow] ssn 1120 policy SP_01 strm 0 dir 0 subclient 0 session accept(104.40.29.86:46226->10.0.0.108:443), fd:27, clissl 0x7fae8568bf88, session count 1 session:0x7fae5e036a98

   <04:05:24>[work 0][flow] ssn 1120 policy SP_01 strm 0 dir 0 subclient 0 client 27 [ST-ssl-handshake], conn st 0x00000004

   <04:05:24>[conn lib]ssl handshake failed

   <04:05:24>[work 0][flow] ssn 1120 policy SP_01 strm 0 dir 0 subclient 0 ssl handshake failed for client 27

   <04:05:24>[work 0][flow] ssn 1120 policy SP_01 strm 0 dir 0 subclient 0 client 27 conn st 0x00000004, conn set err, err msg:err_ssl_handshake

   For real-time debugging, besides logging the diagnose outputs, it’s better to also capture application traffic packets at the same time like below.

5. Capture packets and check the handshake.

   Usually one can create two filter tasks in Network > Packet Capture to capture packets from a specific client and to a specific backend server in server-pool simultaneously.

   

   After the pcap files are downloaded, one can open them with Wireshark to check the TCP and SSL negotiation details. You can check statistics conversations, follow a TCP/TLS stream, or add filters such as “ip.addr==172.30.213.28 && tcp.port==23222 && ip.addr==10.159.37.1 && tcp.port==8002” to narrow down traffic flow to a specific stream.

6. If you find that SSL negotiation fails only when traffic load is heavy, you may also consider if the system reaches a certain performance bottleneck, such as TCP ports used-up, SSL performance limitation, etc.
   Please refer to Server policy intermittently inaccessible for troubleshooting methods.