Solutions by issue type

Recommended solutions vary by the type of issue.:

Login issues
Connectivity issues
Resource issues

Login issues

If an administrator is entering his or her correct account name and password, but cannot log in from some or all computers, examine that account’s trusted host definitions. It should include all locations where that person is allowed to log in, such as your office, but should not be too broad.

Connectivity issues

One of your first tests when configuring a new policy should be to determine whether allowed traffic is flowing to your servers. Investigate the following connectivity issues if traffic does not reach the destination servers:

Is there a FortiADC policy for the destination servers? By default, FortiADC allows traffic to reach a backend server. However, the virtual servers must also be configured before traffic can pass through.
If your network utilizes secure connections (HTTPS) and there is no traffic flow, is there a problem with your certificate?

Checking hardware connections

If there is no traffic flowing from the FortiADC appliance, you want to rule out hardware problems.

To check hardware connections:

Ensure the network cables are properly plugged in to the interfaces on the FortiADC appliance.
Ensure there are connection lights for the network cables on the appliance.
Change the cable if the cable or its connector are damaged or you are unsure about the cable’s type or quality.
Connect the FortiADC appliance to different hardware to see if that makes a difference.
In the web UI, go to System > Networking > Interface and ensure the link status is up for the interface. If the status is down (down arrow on red circle), edit the configuration to change its status to Up.

You can also enable an interface in CLI, for example:

config system interface

edit port2

set status up

end

If any of these checks solve the problem, it was a hardware connection issue. You should still perform some basic software tests to ensure complete connectivity.

If the hardware connections are correct and the appliance is powered on but you cannot connect using the CLI or web UI, you may be experiencing bootup problems. See Restoring firmware (“clean install”).

Checking routing

The ping and traceroute utilities are useful for investigating issues with network connectivity and routing.

Since you typically use these tools to troubleshoot, you can allow ICMP, the protocol used by these tools, in firewall policies and on interfaces only when you need them. Otherwise, disable ICMP for improved security and performance.

By default, FortiADC appliances do not respond to ping and traceroute. However, if the appliance does not respond, and there are no firewall policies that block it, ICMP type 0 (ECHO_REPSPONSE) might be effectively disabled.

To enable ping and traceroute responses:

Go to Networking > Interface.
Select the row for the network interface and click the edit icon.
Under Allow Access, enable ping.
Save the update.

The appliance should now respond when another device such as your management computer sends a ping or traceroute to that network interface.

Note: Disabling ping only prevents the system from receiving ICMP type 8 (ECHO_REQUEST) and traceroute-related UDP. It does not disable CLI commands such as execute ping or execute traceroute that send such traffic.

To verify routes between clients and your servers:

Attempt to connect through the FortiADC appliance, from a client to a backend server, via HTTP and/or HTTPS.

If the connectivity test fails, continue to the next step.

Use the ping command on both the client and the server to verify that a route exists between the two. Test traffic movement in both directions: from the client to the server, and the server to the client. Servers do not need to be able to initiate a connection, but must be able to send reply traffic along a return path.

If the routing test succeeds, continue with Step 4.

If the routing test fails, continue to the next step.

Use the tracert or traceroute command on both the client and the server (depending on their operating systems) to locate the point of failure along the route.

If the route is broken when it reaches the FortiADC appliance, first examine its network interfaces and routes. To display network interface addresses and subnets, enter the CLI command:

show system interface

To display all recently-used routes with their priorities, enter the CLI command:

diagnose netlink route list

You may need to verify that the physical cabling is reliable and not loose or broken, that there are no IP address or MAC address conflicts or blacklisting, misconfigured DNS records, and otherwise rule out problems at the physical, network, and transport layer.

If these tests succeed, a route exists, but you cannot connect using HTTP or HTTPS, an application-layer problem is preventing connectivity.

For application-layer problems, on the FortiADC, examine the:

virtual server policy and all components it references
certificates (if connecting via HTTPS)
server service/daemon

On routers and firewalls between the host and the FortiADC appliance, verify that they permit HTTP and/or HTTPS connectivity between them.

Testing for connectivity with ping

The ping command sends a small data packet to the destination and waits for a response. The response has a timer that may expire, indicating that the destination is unreachable via ICMP.

ICMP is part of Layer 3 on the OSI Networking Model. ping sends Internet Control Message Protocol (ICMP) ECHO_REQUEST (“ping”) packets to the destination, and listens for ECHO_RESPONSE (“pong”) packets in reply.

Some networks block ICMP packets because they can be used in a ping flood or denial of service (DoS) attack if the network does not have anti-DoS capabilities, or because ping can be used by an attacker to find potential targets on the network.

Beyond basic existence of a possible route between the source and destination, ping tells you the amount of packet loss (if any), how long it takes the packet to make the round trip (latency), and the variation in that time from packet to packet (jitter).

If ping shows some packet loss, investigate:

cabling to eliminate loose connections
ECMP, split horizon, or network loops
all equipment between the ICMP source and destination to minimize hops

If ping shows total packet loss, investigate:

cabling to eliminate incorrect connections
all firewalls, routers, and other devices between the two locations to verify correct IP addresses, routes, MAC lists, and policy configurations

If ping finds an outage between two points, use traceroute to locate exactly where the problem is.

To use ping:

Log into the CLI via either SSH, Telnet, or the CLI Console widget of the web UI.

If you want to adjust the behavior of execute ping, first use the execute ping‑options command.
Enter the command:

execute ping <destination_ipv4>

where <destination_ipv4> is the IP address of the device that you want to verify that the appliance can connect to, such as 192.168.1.1.

If the appliance can reach the host via ICMP, output similar to the following appears:

PING 192.168.1.1 (192.168.1.1): 56 data bytes

64 bytes from 192.168.1.1: icmp_seq=0 ttl=253 time=6.5 ms

64 bytes from 192.168.1.1: icmp_seq=1 ttl=253 time=7.4 ms

64 bytes from 192.168.1.1: icmp_seq=2 ttl=253 time=6.0 ms

64 bytes from 192.168.1.1: icmp_seq=3 ttl=253 time=5.5 ms

64 bytes from 192.168.1.1: icmp_seq=4 ttl=253 time=7.3 ms

--- 192.168.1.1 ping statistics ---

5 packets transmitted, 5 packets received, 0% packet loss

round-trip min/avg/max = 5.5/6.5/7.4 ms

If the appliance cannot reach the host via ICMP, output similar to the following appears:

PING 10.0.0.1 (10.0.0.1): 56 data bytes

Timeout ...

--- 10.0.0.1 ping statistics ---

5 packets transmitted, 0 packets received, 100% packet loss

“100% packet loss” and “Timeout” indicates that the host is not reachable.

To verify that routing is bidirectionally symmetric, you should also ping the appliance.

Testing routes and latency with traceroute

The traceroute utility sends ICMP packets to test each hop along the route. It sends three packets to the destination, and then increases the time to live (TTL) setting by one, and sends another three packets to the destination. As the TTL increases, packets go one hop farther along the route until they reach the destination.

Most traceroute commands display their maximum hop count—that is, the maximum number of steps it will take before declaring the destination unreachable—before they start tracing the route. The TTL setting may result in routers or firewalls along the route timing out due to high latency.

Where ping only tells you if the signal reached its destination and returned successfully, traceroute shows each step of its journey to its destination and how long each step takes. If you specify the destination using a domain name, the traceroute output can also indicate DNS problems, such as an inability to connect to a DNS server.

By default, the traceroute utility uses UDP with destination ports numbered from 33434 to 33534. The traceroute utility usually has an option to specify use of ICMP ECHO_REQUEST (type 8) instead, as used by the Windows tracert utility. If you have a firewall and you want traceroute to work from both machines (Unix-like systems and Windows) you will need to allow both protocols inbound through your firewall (UDP ports 33434 - 33534 and ICMP type 8).

To use traceroute:

Log into the CLI via either SSH, Telnet, or the CLI Console widget of the web UI.
Enter the command:

execute traceroute {<destination_ipv4> | <destination_fqdn>}

where {<destination_ipv4> | <destination_fqdn>} is a choice of either the device’s IP address or its fully qualified domain name (FQDN).

For example, you might enter:

execute traceroute www.example.com

If the appliance has a complete route to the destination, output similar to the following appears:

traceroute to www.fortinet.com (66.171.121.34), 32 hops max, 84 byte packets

1 172.16.1.2 0 ms 0 ms 0 ms

2 209.87.254.221 <static-209-87-254-221.storm.ca> 2 ms 2 ms 2 ms

3 209.87.239.129 <core-2-g0-1-1104.storm.ca> 2 ms 1 ms 2 ms

4 67.69.228.161 2 ms 2 ms 3 ms

5 64.230.164.17 <core2-ottawa23_POS13-1-0.net.bell.ca> 3 ms 3 ms 2 ms

6 64.230.132.234 <core2-ottawatc_POS5-0-0.net.bell.ca> 20 ms 20 ms 20 ms

7 64.230.132.58 <core4-toronto21_POS0-12-4-0.net.bell.ca> 24 ms 21 ms 24 ms

8 64.230.138.154 <bx4-toronto63_so-2-0-0-0.net.bell.ca> 8 ms 9 ms 8 ms

9 64.230.185.145 <bx2-ashburn_so2-0-0.net.bell.ca> 23 ms 23 ms 23 ms

10 12.89.71.9 23 ms 22 ms 22 ms

11 12.122.134.238 <cr2.wswdc.ip.att.net> 100 ms 12.123.10.130 <cr2.wswdc.ip.att.net> 101 ms 102 ms

12 12.122.18.21 <cr1.cgcil.ip.att.net> 101 ms 100 ms 99 ms

13 12.122.4.121 <cr1.sffca.ip.att.net> 100 ms 98 ms 100 ms

14 12.122.1.118 <cr81.sj2ca.ip.att.net> 98 ms 98 ms 100 ms

15 12.122.110.105 <gar2.sj2ca.ip.att.net> 96 ms 96 ms 96 ms

16 12.116.52.42 94 ms 94 ms 94 ms

17 203.78.181.10 88 ms 87 ms 87 ms

18 203.78.181.130 90 ms 89 ms 90 ms

19 66.171.121.34 <fortinet.com> 91 ms 89 ms 91 ms

20 66.171.121.34 <fortinet.com> 91 ms 91 ms 89 ms

Each line lists the routing hop number, the IP address and FQDN (if any) of that hop, and the 3 response times from that hop. Typically a value of <1ms indicates a local router.

If the appliance does not have a complete route to the destination, output similar to the following appears:

traceroute to 10.0.0.1 (10.0.0.1), 32 hops max, 84 byte packets

1 172.16.1.2 0 ms 0 ms 0 ms

2 172.16.1.10 0 ms 0 ms 0 ms

3 * * *

4 * * *

The asterisks ( * ) indicate no response from that hop in the network routing.

Examining the routing table

When a route does not exist, or when hops have high latency, examine the routing table. The routing table is where the FortiADC appliance caches recently used routes.

If a route is cached in the routing table, it saves time and resources that would otherwise be required for a route lookup. If the routing table is full and a new route must be added, the oldest, least-used route is deleted to make room.

To check the routing table in the CLI, enter:

diagnose netlink route list

Examining server daemons

If a route exists, but you cannot connect to the web UI using HTTP or HTTPS, an application-layer problem is preventing connectivity.

Verify that you have enabled HTTPS and/or HTTP on the network interface. Also examine routers and firewalls between the host and the FortiADC appliance to verify that they permit HTTP and/or HTTPS connectivity between them. Finally, you can also use the CLI command to verify that the daemons for the web UI and CLI, such as sshd, cli, nginx, and php-fpm are running and not overburdened:

diagnose system top delay 10

Checking port assignments

If you are attempting to connect to FortiADC on a given network port, and the connection is expected to occur on a different port number, the attempt will fail. For a list of ports used by FortiADC, see Appendix B: Port Numbers.

Performing a packet trace

When troubleshooting malformed packet or protocol errors, it helps to look inside the protocol headers of packets to determine if they are traveling along the route you expect, and with the flags and other options you expect.

If you configure virtual servers on your FortiADC appliance, packets’ destination IP addresses will be those IP addresses, not the physical IP addresses (i.e., the IP address of port1, etc.). An ARP update is sent out when a virtual IP address is configured.

If the packet trace shows that packets are arriving at your FortiADC appliance’s interfaces but no HTTP/HTTPS packets egress, check that:

Physical links are firmly connected, with no loose wires
Network interfaces are brought up
Link aggregation peers, if any, are up
VLAN IDs, if any, match
Virtual servers exist, and are enabled
Matching policies exist, and are enabled
If using HTTPS, valid server/CA certificates exist
IP-layer and HTTP-layer routes, if necessary, match
Servers are responsive, if server health checks are configured and enabled

Checking the SSL/TLS handshake & encryption

If the client is attempting to make an HTTPS connection, but the attempt fails after the connection has been initiated, during negotiation, the problem may be with SSL/TLS. Symptoms may include error messages such as:

ssl_error_no_cypher_overlap
(Mozilla Firefox 9.0.1)
Error 113 (net::ERROR_SSL_VERSION_OR_CIPHER_MISMATCH): Unknown error.
(Google Chrome 16.0.912.75 m)

The handshake is between the client and FortiADC. If the connection cannot be established, verify that the browser supports one of the key exchanges, encryption algorithms, and authentication (hashes) offered by FortiADC.

If you are not sure which cipher suites are currently supported, you can use SSL tools such as OpenSSL to discover support. For example, you could use this client-side command to know whether the server or FortiADC supports strong (HIGH) encryption:

openssl s_client -connect example.com:443 -cipher HIGH

or supports deprecated or old versions such as SSL 2.0:

openssl s_client -ssl2 -connect example.com:443

Resource issues

This section includes troubleshooting questions related to sluggish or stalled performance.

Monitoring traffic load

Heavy traffic loads can cause sustained high CPU or RAM usage. If this is unusual, no action is required. However, sustained heavy traffic load might indicate that you need a more powerful FortiADC model.

In the web UI, you can view traffic load two ways:

Monitor current HTTP traffic on the dashboard. Go to System > Dashboard > Virtual Server and examine the throughput graphs.
Examine traffic history in the traffic log. Go to Logs & Report > Log Browsing > Traffic Log.

DoS attacks

A prolonged denial of service (DoS) can bring your servers down if your FortiADC appliance and your network devices are not configured to prevent it. To prevent DoS attacks, enable the DoS and connection limit features. Also, configure protections on your FortiGate and other network devices. DoS attacks can use a variety of mechanisms. For in-depth protection against a wide variety of DoS attacks, you can use a specialized appliance such as FortiDDoS.

In the web UI, you can watch for attacks in two ways:

Monitor current traffic on the dashboard. Go to System > Dashboard and examine the system-wide throughput.
Examine attack history in the traffic log. Go to Logs & Report > Log Browsing > Security Log.