Troubleshoot Site-Publish Issues

Compared with User/HTTP authentication, Site-Publish provides more flexible and advanced features such as single sign-on (SSO) and combination access control and authentication such as Two-factor authentication.

The sections below will introduce troubleshooting methods according to Site-Publish deployment scenarios:

Common troubleshooting methods
Typical authentication failures
Two-factor authentication issues
SAML issues
Kerberos Issues

Common troubleshooting steps for Site-Publish issues

For all issues, it's better to double check the necessary configuration steps for Site-Publish:
- Remote servers are created in User > Remote Server;
- Remote servers are added to Application Delivery > Site Publish > Authentication Server Pool;
- Site Publish Rule is created in Application Delivery > Site Publish > Site Publish Rule;
  - Published Site, Path, Client Authentication Method, and Authentication Server Pool are configured correctly;
  - Delegation servers and parameters are correctly configured;
    Please Note that some fields such as URL Path KCD SPN are case sensitive. You must input the exact upper or lower case strings.
- The Site Publish Rule is added into a Site Publish Policy;
- The Site Publish Policy is selected in a Web Protection Profile;
- The Web Protection Profile is selected by the server-policy to protect the target website.
Check the connectivity & availability of remote servers for authentication server pool:
You can check the connectivity and service availability via below steps:
- Ensure the IP address and service ports configuration on FortiWeb comply with which are provided by the remote servers;
- Use ping to confirm no connectivity issue between FortiWeb and the remote server;
- Use the Test button ("Test LDAP" or "Test Radius") in User > Remote Server > LDAP/Radius Server to test if the remote server can be connected successfully;
- Use browsers or other test clients rather than FortiWeb to visit the backend server to confirm if the backend server is reachable;
- Use a different remote server to determine if the authentication just fails with a specific type of remote server;
- Capture packets on FortiWeb and the remote server to determine if the authentication queries are sent out by FortiWeb, if responses are correctly received by FortiWeb or delayed, and if the queries are received by the remote server, etc.
Enable Event log for site-publish rule and check the login failure logs on FortiWeb.
To generate event logs, go to Application Delivery > Site Publish > Site Publish Rule > Edit a Rule > Alert Type, select Failed Only or All, then you’ll be able to see event logs when an authentication failure occurs. Such event logs are usually simple, but can help us to confirm the issue.
E.g.
v012xxxxdate=2022-05-05 time=15:19:19 log_id=11002003 msg_id=000006998393 device_id=FVVM08TM21000613 vd="root" timezone="(GMT-7:00)Mountain Time(US&Canada)" timezone_dayst="GMTb+7" type=event subtype="system" pri=alert trigger_policy="N/A" user=daemon ui=daemon action=login status=failure msg="User user01 [Site Publish] login failed on portal.testdomain.com from 172.30.212.181"
Check logs on the remote servers.
FortiWeb supports using remote servers including LDAP, Radius, KDC, SAML servers to authenticate clients, and also support
If authentication queries are sent out from FortiWeb and received by remote servers, while eventually fail to be authenticated, logs with detailed process or failure reasons can usually be generated by these servers. Checking such logs often helps to find the cause of failures.
Particularly, if FortiAuthenticator is used as the remote servers, you can check two types of FortiAuthenticator logs:
- Event logs: FortiAuthenticator > Logging > Log Access > Logs
- Debug logs: visit HTTPs://<FortiAuthenticator_IP>/debug/.
  Please refer to an 2FA auth failure caused by invalid token as below:
Check site-publish diagnose logs:
It’s simple to enable site-publish related diagnose logs, which can provide very detailed information for the packet processing flow:
# diagnose debug application site-publish 7
# diagnose debug timestamp enable
# diagnose debug enable
Besides, if you’re not sure if the issue is related to other FortiWeb features, or need logs of the complete user access session, please also enable diagnose flow logs for further investigation.
# diagnose debug flow filter flow-detail 7 #Enables messages from each packet processing module and packet flow traces
# diagnose debug flow filter HTTP-detail 7 #HTTP parser details
# diagnose debug flow filter module-detail status on #Turn on details from modules processing the flow
# diagnose debug flow filter server-ip 192.168.12.12 #The VIP in RP mode or the real server IP in TP/TI mode
# diagnose debug flow filter client-ip 192.168.12.1 #The client IP
# diagnose debug flow trace start
Some site-publish diagnose failure logs are as below:
Remote server is not reachable:
[SP: MAIN][WARN](./waf_module/site_publish.c: 6736): LDAP server [10.65.1.97, 636, 1] is down by health check, then stop and auth failed
[SP: MAIN][DBG](./waf_module/site_publish.c: 6776): fail to auth [401]: username = user01, password = [X]
Incorrect username or password:
[SP: MAIN][INFO](./waf_module/site_publish.c: 6736): got active IP [10.65.1.96] from health check
[SP: MAIN][DBG](./waf_module/site_publish.c: 6776): fail to auth [401]: username = user01, password = [X]
[SP: MAIN][DBG](./waf_module/site_publish.c: 1135): elog : username: [user01]
Incorrect service principal name when the Authentication Delegation is Kerberos Constrained Delegation:
[SP: MAIN][DBG](./waf_module/site_publish.c: 10500): kerberos constrained delegation
[SP: MAIN][DBG](./waf_module/site_publish.c: 7981): spn rule is single_server
[SP: MAIN][ERR](./waf_module/site_publish.c: 5290): fail to AS of KCD [host/test1.sitepublish.fortiweb@SITEPUBLISH.FORTIWEB]
[SP: MAIN][ERR](./waf_module/site_publish.c: 10518): fail to check AS of KCD, bypas
Capture packets on FortiWeb and the remote server to analyze the authentication traffic flow.
Analyzing packet interaction between FortiWeb and the remote server are usually the ultimate method to troubleshoot authentication failures, especially when logs on either FortiWeb or remote servers are insufficient.
You can get the following information from captured packets:
- If the authentication queries and requests are sent out by FortiWeb and received by the remote server;
- If responses (accept or challenge) are sent back by the remote server and received by FortiWeb;
- If there is any delay when FortiWeb sending out a request, or the remote server sending back the response;
  Clients, FortiWeb and remote servers usually have their own timeout settings for the authentication session. As long as either of these timeout periods elapses before the response is received, it may lead to an authentication failure.
  This problem is very common. Latency in the Internet, special or misconfigured topology often result in such issues.
- If the traffic interaction complies with the application or protocol requirement and definition;
  This method requires in-depth understanding of authentication protocols and state machine interaction such as Radius, LDAP/LDAPS, SAML, etc. A simple way to narrow down the issue is comparing the packet flow between a successful authentication and an unsuccessful access.
  For some uncommon servers and user-defined servers, this way is useful to find the protocol compatibility problem.
Some issues are related to browser behaviors. They might be issues that can be resolved by updating to the latest version. You can also change a browser and try again.

If the browser does not prompt authentication window or form

When the authentication form is not prompted by the browser when visiting the target URL or Path, you can check the following:

Check if there is any missing configuration.
- Check if the correct site publish rule is included in site publish policy, and the policy is included in the web protection profile used in the server-policy;
- Check if the Published Site & Path are correctly configured;
  
  For regular expression, use the built-in Regular Expression Validator to confirm the published site domain can be matched; for Path or URL, confirm it’s case sensitive.
- Particularly, check if any remote server is included in the authentication Server Pool selected by the site publish rule.

This is a common issue of configuration missing that often occurs in customers’ sites. Just remember to add Remote Servers to a server pool used by a site publish rule.

Check if the Path/URL matches URL Access rules.

In 6.3 and later builds, URL Access Rule is processed before Site Publish, so if the certain URL/Path matches a URL Access Rule with Action “Pass”, the site-publish rule will be skipped,

To resolve this issue, you can remove the conflicted URL Access Rule or configure the Action of the URL Access Rule as “Continue”, then FortiWeb will continue processing the request and site publish rules will be matched.

Sometimes if you suspect other WAF features cause the issue, you can check FortiWeb Admin Guide > Key concepts > Sequence of scans to see if any other features processed prior to Site-Publish are configured. You can remove the feature to try again.

If authentication fails

Authentication failures have different causes:

If the username, password, or token (2FA method) is wrong, the browser usually has kinds of behaviors such as keeping a pop-up sign-in window, prompting “Invalid credentials” or “Login Failed” message.

Check if remote server members in the Authentication Server Pool are reachable from FortiWeb.

If the remote server IP is not reachable, service port is unreachable (or incorrectly configured), or has other configuration mistakes such as Radius server secret, one can make a quick judgment from the error messages or browser behavior.

The error messages vary according to different client authentication methods or remote servers. For example, the browser may keep popping up the Sign in window (HTTP Basic Authentication method), or the Authentication Form will prompt a warning message like “Failed to connect LDAP server” or “RADIUS response timeout”, etc.

IP unreachable or Invalid secret

Incorrect port

IP or port unreachable

Service not available

You can check the connectivity and service availability issues with steps in above section: "Common troubleshooting steps for Site-Publish issues: Check the connectivity & availability of remote servers for authentication server pool".

Check if the backend server configured in Authentication Delegation behaves as expected.
- Double confirm that the corresponding servers such as KDC server for authentication delegation is correctly configured;
- Check if access to the backend server directly can be successful, rather than pass through FortiWeb.
- Change and test with a different Authentication Delegation type;
- For Form Based Delegation:
  - If needed, clone the predefined templates, and edit the settings as your desire
- For Kerberos delegation:
  - Please refer to the following section “Kerberos issues” for more details.

Some special requirement or notes on configuration:
- For Two-factor site-publish rules, “Client Authentication Method” needs to be “HTML Form Authentication”.
  
  Two-factor authentication requires configuring the “Client Authentication Method” as “HTML Form Authentication”.
  
  When choosing “HTTP Basic Authentication”, the browser will keep on prompting the Sign in window, because this browser-specific method cannot display a second authentication form that allows users to enter a token code.
- When Authentication Delegation is "HTTP Basic" in Site Publish Rule, “Basic Authentication” should be enabled in the backend IIS while Forms Authentication should be disabled to avoid conflict. This is a restriction from the IIS side.

Increase the auth-timeout when remote servers’ response is slow

In the real environment, you may find the LDAP/Radius/SAML/NTLM/OAuth servers are slow to answer authentication queries by analyzing diagnose logs or captured packets. You can adjust the authentication timeout setting to prevent the query from failing.

configure system global

set auth-timeout <milliseconds_int>

end

<milliseconds_int> is the number of milliseconds that FortiWeb will wait for the remote authentication server to respond to its query. The valid range is 1–60,000 and the default value is 2000.

Besides Authentication server pool members for Site-Publish, this setting also affects remote authentication queries for administrator accounts.

Two-factor authentication issues

Steps to troubleshoot two-factor authentication issues:

1. Check the Radius server configuration on FortiWeb; you can remove the 2FA configuration on the Radius server and use “Test Radius” button to confirm;

2. Remove 2FA authentication configuration on the Radius server, check if authentication can be successful if with only Radius;

3. Check FortiWeb configuration to ensure that “Client Authentication Method” is configured as “HTML Form Authentication” in the site-publish rule;

4. Check logs on Radius server to see if any clear failure logs:

If the Radius server is FortiAuthenticator, please refer to the above section to check detailed logs: 4.2 > Common troubleshooting > Check logs on the remote servers.

5. Capture packets on the front-end and back-end side. Analyze the traffic flow to see if any delays, response loss or abnormal packets.

Check if there is another request sent by the same client before the authentication process is done. An extra request will interrupt 2FA process and result in cookie reset;
A common example of such an extra request is favicon.ico. If it’s the case, you can try to add a URL Access rule to deny (Action is Deny) this request;
If there are other requests such as the ones generated by JS script in the web code, you may try to add a URL Access rule to bypass (Action is Pass) this request

6. For further analysis, please also enable diagnose logs for site-publish simultaneously.

Refer to above section "Common troubleshooting steps for Site-Publish issues: Collect diagnose logs".

SAML issues

1. Most SAML issues are configuration issues.

You’d better double verify the configuration on both IDP side and SP/FortiWeb side:

FortiWeb > User > Remote Server > SAML Server:

Entity ID: the unique identity of SP; the host is the domain name of vserver. The prefix must be HTTPs.
IdP Metadata: upload a valid IdP metadata file, which is exported from the IdP;

For any changes on the IdP, please export the metadata file and upload to FortiWeb again.

IdP Entity ID: double confirm this ID displayed after the IdP metadata file uploaded is identical to that shown on the IdP
After SAML Server is configured, click Generate Service Provider Metadata to export a metadata file, and import the file to IdP.

If you change any item of SAML server, you must regenerate Service Provider Metadata file and reconfigure IDP. Particularly, please make sure the “Location” in the metadata file matches the “Published Site” (Domain) configured in the Site-publish rule.

E.g.

<md:SingleLogoutService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" Location="HTTPs://portal.testdomain.com/new_saml_server/saml.sso/SLO/POST"/>

On IdP Side (AD FS, FortiAuthenticator, etc.):

SP Metadata: import the one generated on FortiWeb;

For any changes on SAML Server, it’s better to update this file again.

Make sure the user information (UPN or Email) is mapped to EPPN (urn:oid:1.3.6.1.4.1.5923.1.1.1.6), because FortiWeb uses the value of the EPPN attribute to identify users uniquely.

FortiWeb > Application Delivery > Site Publish > Site Publish Rule:

The correct SAML Server is selected;
The Published Site should be consistent with the host of SAML Server’s Entity ID.

2. Other checking points:

All IdP and SP configuration are case sensitive;
Make sure the clocks of all the related servers (DC, FortiWeb, IDP,etc.) are synchronized;
For cross domain environments, if the AD Domain trusts each other, you can share one ADFS instance. But if not, you would need one ADFS instance for each AD.
If IDP is ADFS, the global logout url is HTTPs://<ADFS_Service_FQDN>/adfs/ls/?wa=wsignout1.0

3. Enable and check diagnose logs:

# diagnose debug application site-publish 7

# diagnose debug timestamp enable

# diagnose debug enable

4. Collect SAML related logs with below steps for dev team analysis:

Edit /data/etc/saml/shibboleth/*.logger, switch the default level on the top to DEBUG.

/data/etc/saml/shibboleth#cat shibd.logger

log4j.rootCategory=DEBUG, shibd_log #The default level is WARN

/data/etc/saml/shibboleth#cat native.logger

log4j.rootCategory=DEBUG, native_log #The default level is WARN

Note:Don’t forget to restore to the default level WARN to avoid performance issues.

Restart proxyd & shibd

Reproduce the issue

Collect the logs under /var/log/shibboleth/. You may clear them before you test it

You can copy these logs to /var/og/gui_upload and download them from GUI.

~# ls -l /var/log/shibboleth/

-rw-r--r-- 1 root 0 9744 May 13 14:44 native.log

-rw-r--r-- 1 root 0 30712 May 13 14:44 shibd.log

Kerberos issues

1. Check Kerberos related configuration

The most common issues caused by Kerberos authentication failures are also configuration mistakes. When issues occur, you need to check the configuration on FortiWeb and the backend KDC server.

This section will not focus on configuration details for different KDC servers, but only introduce some general considerations or mistakes in both FortiWeb & KDC settings.

FortiWeb > User > Remote Server > KDC Server:

Delegated Realm: It should be all capitalized. It’s the domain of the domain controller (DC) that the KDC belongs to. Typically the UPN (User Principal Name) used for login has the format username@delegated_realm.
Shortname: An alias of the realm you specified. The shortname can include the domain name of the realm that is not fully qualified. With a shortname being configured, the format of UPN can be username@shortname

A shortname is used in a scenario when the complete Kerberos realm (e.g. TEST.FortiWebDEMO.COM) is different from what a client gets from their username (e.g. the username FortiWeb gets from the IDP is an email address like (xxx@FortiWebDEMO.COM). If the customer can set the UPN as the username returned to FortiWeb, shortname is not needed; otherwise, FortiWeb would have to set a shortname to make Kerberos work.

FortiWeb > Application Delivery > Site Publish > Site Publish Rule:

There are two kinds of Authentication Delegation:
- Kerberos: also called the Regular or Basic Kerberos Delegation; available only when Client Authentication Method is HTML Form Authentication or HTML Basic Authentication. You just need a username&password for delegation.
- Kerberos Constrained Authentication: available when Client Authentication Method is Client Certificate, SAML or NTLM. You just need UPN (User Principle Name); the delegator will help you get access tickets.
Delegated HTTP Service Principal Name (SPN): Make sure the Service Principal Name is configured with exactly the same string and upper/lower case with that configured in AD; and, all realm such as the domain name after @ should be upper case

The format is like:

<protocol >/<exchange_server_hostname>/<realm>

protocol: HTTP
exchange server hostname: USER-LHLGG566P0 (case-insensitive), you may also use the full name USER-U3LOJFPLH1.FortiWebdemo.com
realm: FortiWebDEMO.COM; should be capital

E.g. HTTP/USER-U3LOJFPLH1.FortiWebdemo.com@FortiWebDEMO.COM

Default Domain Prefix Support: For Regular Kerberos delegation only. The domain controller usually requires users to log in with the username format domain\username such as EXAMPLE\user1. Alternatively, enable this option and enter EXAMPLE for Default Domain Prefix, the user enters user1 for the username value and FortiWeb will automatically add EXAMPLE\ to the HTTP Authorization: header before it forwards it to the web application
Keytab File: For Kerberos Constrained Delegation only. Select the keytab file configuration for the AD user that FortiWeb uses to obtain Kerberos service tickets for clients.

For instructions on how to generate the keytab file, see FortiWeb Admin Guide > Application Delivery > Site Publishing > Creating an Active Directory (AD) user for FortiWeb - Keytab File.

Service Principal Name for Keytab File: For Regular Kerberos delegation only. It's the SPN that you used to generate the keytab specified by Keytab File. Don’t forget the realm suffix.

Particular requirement for Client Certification Authentication:

Double check that Client Certificate Verification is correctly configured and bound to the server-policy.
For Username Location in Certificate in Site Publish rule, here’s an example.

The username we need is cert1@fwbdev.com, then you may specify the location you want, Subject or Subject Alternative Name (SAN). However, the most exact one is UPN (aka Other Name > Principal Name) in SAN, so you’d better keep the default.

Some Tips on KDC servers:

Create an HTTP-delegator which is a domain account to do authentication delegation. Please refer to FortiWeb Admin Guide > Application Delivery > Site Publishing > Creating an Active Directory section for details.
Make sure the account & its password never expire.
Don’t use $setspn -A …. Instead, use $setspn -S … to create SPN for the account

2. Other checking points:

Make sure the clocks of all the related servers (DC, FortiWeb, IDP,etc.) are synchronized, otherwise Kerberos tickets will be invalid.

3. Collect information for further investigation:

Diagnose logs when the issue happens:

# diagnose debug application site-publish 7

# diagnose debug timestamp enable

# diagnose debug enable

4. Test with FortiWeb backend tool krb_test and collect the output:

Here is the usage below:

Note:

-s: the same with Delegated HTTP Service Principal Name

-h: IP or domain name of the backend web server (aka pserver); if you use -H, the value of Host header in request will be overwritten.

-t: Use SSL tunnel for the backend web server. -c and -e are used for client certificate authentication

for basic Kerberos:

-u: UPN for login, and the format must be username@DOMAIN.COM

-p: it’s password

for KCD:

-u: ditto

-n: delegator’s UPN, it’s the same with Service Principal Name for Keytab File

-k: file path of your keytab file in FortiWeb (you should upload it first)

An example of the successful result: (response returns 200 OK)

If the test fails, that means there are some problems in Kerberos authentication or backend communication. Please note the debug information on the screen.

If the test succeeds, that means configuration or login input may be incorrect. Need to check them and keep the parameters consistent with those in krb_test.