Server policy intermittently inaccessible
-
Check if networking connection is stable:
Ping continuously from a remote client to see if any failures or long response time;
Ping the back-end server from FortiWeb to see if any failures or long response time;
Visit the back-end server continuously from a remote client to see if any failures or long response time;
Visit the back-end server from FortiWeb to see if any failures or long response time when accessing the server-policy from remote client fails.
-
Check if back-end servers’ status in server-pool are stable:
If server health check is ON, check Event logs to confirm the health check down/up events;
If server health check is OFF, check the logs on the back-end server (Apache/Nginx logs or other monitor system) if possible;
Visit the back-end server continuously from FortiWeb to see if any failures or long response time from time to time or when the connectivity issue occurs.
You can use curl on FortiWeb back-end shell to visit the back-end server, and check the response time.
Samples:
curl -o /dev/null -s -w %{time_total}\\n http://<back-end server_IP>:<port>
curl -v https://<domain/IP>/ -A "check_http" -so /dev/null --resolve <domain>:<port>:<IP> -k -w %{time_namelookup}::%{time_connect}::%{time_starttransfer}::%{time_total}::%{speed_download}"\n"
You can run a script on FortiWeb back-end shell (upload the script via System > Maintenance > Backup&Restore > GUI File Download/Upload > Upload and chmod to add the execute permission) to visit the back-end server periodically and record the return code&response time. However, it’s not recommended when traffic is heavy.
-
Check if FortiWeb system has resource shortage;
Check FortiWeb event logs or other external monitor tools (if available) to confirm if there is any traffic burst when the issue occurs;
Check FortiWeb event logs to confirm if there is any high CPU or Memory usage when the issue occurs; (Log&Report > Event > Filter > Action > check-resource)
Find logs like:
CPU usage too high,CPU usage is 64, process cmdbsvr
Check if the connection/session/throughput status are overloaded when the issue occurs:
When connection per second or Throughput reaches the performance bottleneck, you may find that the system CPU usage is near to 100%, thus consider changing to a higher-end platform. But sometimes if the connection number of TIME_WAIT becomes very large, it means too many sockets may be occupied and new TCP connections can hardly be established, thus causing performance decrease or new request failures. In this case, you may consider adding
diagnose policy total-conn-psec list
diagnose policy total-session list
diagnose policy total-traffic http list
Or
/# netstat -antp | grep ESTABLISH | wc -l
19094
/# netstat -antp | grep TIME_WAIT | wc -l
38688
/# netstat -nat | awk '{print $6}' | sort | uniq -c | sort -r
56338 TIME_WAIT
33940 ESTABLISHED
427 SYN_SENT
251 LISTEN
221 FIN_WAIT1
196 FIN_WAIT2
5 SYN_RECV
1 established)
1 Foreign
- Add secondary IPs to the interface connected to the back-end server:
Secondary IPs are necessary for both below methods.
FortiWeb # sho sys interface port1
config system interface
edit "port3"
set type physical
set ip 10.13.4.254/24
set allowaccess ping ssh snmp http https FortiWeb-manager
config secondaryip
edit 1
set ip 10.13.4.253/24
next
edit 2
set ip 10.13.4.252/24
next
end
end
- Method 1: Enable
ip-src-balance or ip6-src-balance
to allow FortiWeb to connect to back-end servers using multiple IPv4 addresses configured as above.This is a global option that affects all server policies. FortiWeb uses round-robin algorithm between all primary&secondary IPs to distribute connections to back-end servers:
config system network-option
set ip-src-balance enable
set ip6-src-balance enable
End
Method 2: Enable
client-real-ip
and add available secondary IPs configured above to IP ranges, then traffic matching the specific policy will connect to back-end servers using these secondary IPs added to IP/IP Range:To ensure FortiWeb receives the server's response, configure FortiWeb as the back-end server’s gateway.
This option is available only for Reverse Proxy mode.
FortiWeb # show server-policy policy
config server-policy policy
edit "Test_Policy"
...
set client-real-ip enable
set real-ip-addr 10.13.4.253
next
end
- Check if there are system crash/coredump when the issue occurs.
- Check related logs and captured files for further analysis.
Solution
To alleviate or solve such issue, you can increase the maximum number of connection by adding IP addresses used to connect to the back-end servers: