SPNEGO trust association interceptor (TAI) troubleshooting tips (deprecated)
Presented here is a list of troubleshooting tips useful in diagnosing Simple and Protected GSS-API Negotiation (SPNEGO) TAI problems and exceptions.
Deprecated feature:
In WAS v6.1, a trust association interceptor (TAI) that uses the SPNEGO to securely negotiate and authenticate HTTP requests for secured resources was introduced. In WebSphere Application Server 7.0, this function is now deprecated. SPNEGO web authentication has taken its place to provide dynamic reload of the SPNEGO filters and to enable fallback to the application login method. depfeat
This topic references one or more of the application server log files. As a recommended alternative, we can configure the server to use the High Performance Extensible Logging (HPEL) log and trace infrastructure instead of using SystemOut.log , SystemErr.log, trace.log, and activity.log files on distributed and IBM i systems. We can also use HPEL in conjunction with the native z/OS logging facilities. If we are using HPEL, we can access all of the log and trace information using the LogViewer command-line tool from the server profile bin directory. See the information about using HPEL to troubleshoot applications for more information on using HPEL.
The IBM Java Generic Security Service (JGSS) and IBM Simple and Protected GSS-API Negotiation (SPNEGO) providers use a JVM custom property to control trace information. The SPNEGO TAI uses the JRas facility to allow an administrator to trace only specific classes. The following important trace specifications or JVM custom properties should be used to debug the TAI by using tracing.
| Trace | Use |
|---|---|
| com.ibm.security.jgss.debug | Set this JVM Custom Property to all to trace through JGSS code. Messages appear in the trace.log file, and SystemOut.log. |
| com.ibm.security.krb5.Krb5Debug | Set this JVM Custom Property to all to trace through the Kerberos5-specific JGSS code. Messages appear in the trace.log file, and SystemOut.log. |
| com.ibm.ws.security.spnego.* | Set this trace on using the console > troubleshooting > Logging and Tracing > server1 > Change Log Detail Levels > com.ibm.ws.security.spnego.*. Messages appear in the trace.log file. |
Problem: WebSphere Application Server and the Active Directory (AD) Domain Controller's time are not synchronized within 5 minutes.
| Symptom | User Action |
|---|---|
[2/24/06 13:12:46:093 CST] 00000060 Context 2 com.ibm.ws .security.spnego.Context begin GSSContext accepted [2/24/06 13:12:46:093 CST] 00000060 Context E com.ibm.ws .security.spnego.Context begin CWSPN0011E: An invalid SPNEGO token has been encountered while authenticating a HttpServletRequest: 0000: 60820160 06062b06 01050502 a1820154 `..` ..+. .... ...T 0010: 30820150 a0030a01 01a10b06 092a8648 0..P .... .... .*.H 0020: 82f71201 0202a282 013a0482 01366082 .... .... .:.. .6`. 0030: 01320609 2a864886 f7120102 0203007e .2.. *.H. .... ...~ 0040: 82012130 82011da0 03020105 a1030201 ..!0 .... .... .... 0050: 1ea41118 0f323030 36303232 34313931 .... .200 6022 4191 0060: 3234365a a5050203 016b48a6 03020125 246Z .... .kH. ...% 0070: a9161b14 57535345 432e4155 5354494e .... WSSE C.AU STIN 0080: 2e49424d 2e434f4d aa2d302b a0030201 .IBM .COM .-0+ .... 0090: 00a12430 221b0448 5454501b 1a773230 ..$0 "..H TTP. .w20 00a0: 30337365 63646576 2e617573 74696e2e 03se cdev .aus tin. 00b0: 69626d2e 636f6dab 81aa1b81 a76f7267 ibm. com. .... .org 00c0: 2e696574 662e6a67 73732e47 53534578 .iet f.jg ss.G SSEx 00d0: 63657074 696f6e2c 206d616a 6f722063 cept ion, maj or c 00e0: 6f64653a 2031302c 206d696e 6f722063 ode: 10, min or c 00f0: 6f64653a 2033370a 096d616a 6f722073 ode: 37. .maj or s 0100: 7472696e 673a2044 65666563 74697665 trin g: D efec tive 0110: 20746f6b 656e0a09 6d696e6f 72207374 tok en.. mino r st 0120: 72696e67 3a20436c 69656e74 2074696d ring : Cl ient tim 0130: 65204672 69646179 2c204665 62727561 e Fr iday , Fe brua 0140: 72792032 342c2032 30303620 61742031 ry 2 4, 2 006 at 1 0150: 3a31323a 34352050 4d20746f 6f20736b :12: 45 P M to o sk 0160: 65776564 ewed | The preferred way to resolve this issue is to synchronize the WAS system time to be within 5 minutes of the AD server's time.
A best practice is to use a time server to keep all systems synchronized.
We can also add or adjust the clockskew parameter in the Kerberos configuration file.
The default for theclockskew parameter is 300 seconds (or 5 minutes). |
Problem: No factory available to create a name for mechanism 1.3.6.1.5.5.2.
| Problem | Symptom | User Action |
|---|---|---|
| Get an exception: No factory available to create a name for mechanism 1.3.6.1.5.5.2. There is no factory available to process the creation of a name for the specific mechanism. |
[4/8/05 22:51:24:542 EDT] 5003e481 SystemOut
O [JGSS_DBG_PROV] Provider IBMJGSSProvider version 1.01 does not support mech 1.3.6.1.5.5.2
[4/8/05 22:51:24:582 EDT] 5003e481 ServerCredent > com.ibm.ws.security.spnego
.ServerCredential initialize ENTRY
SPNEGO014: Kerberos initialization Failure: org.ietf.jgss.GSSException, major code: 2, minor code: 0
major string: Unsupported mechanism minor string: No factory available to create name for mechanism 1.3.6.1.5.5.2
at com.ibm.security.jgss.i18n.I18NException.throwGSSException
(I18NException.java:30)
at com.ibm.security.jgss.GSSManagerImpl.a(GSSManagerImpl.java:36)
at com.ibm.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl
.java:217)
at com.ibm.security.jgss.GSSCredentialImpl.<init>(GSSCredentialImpl
.java:264)
| Check the java.security file to ensure
that it contains the IBMSPNEGO security provider and that the provider is defined correctly. The java.security file should contain a line similar to: security.provider.6=com.ibm.security .jgss.mech.spnego.IBMSPNEGO |
Problem: Getting an exception as the JGSS library is trying to process the SPNEGO token.
| Symptom | Description | User Action |
|---|---|---|
The following error is displayed as the JGSS library is trying
to process the SPNEGO token.
Error authenticating request. Reporting to client Major code = 11, Minor code = 31 org.ietf.jgss.GSSException, major code: 11, minor code: 31 major string: General failure, unspecified at GSSAPI level minor string: Kerberos error while decoding and verifying token: com.ibm.security.krb5.internal.KrbException, status code: 31 message: Integrity check on decrypted field failed | This exception is the result of encoding the ticket using one key and attempting to decode it using a different key. There are numbers of possible reasons for this condition:
| If the problem is with the Kerberos keytab file, then regenerate
the keytab file. If the problem is with multiple SPN definitions, then remove the extra or conflicting SPN, confirm that the SPN is no longer registered with the Active Directory, and then add the SPN.
The Active Directory may need to be searched for other entries with SPNs defined that clash with the SPN.
To confirm that the SPN is not registered, the command:
Should return with the following response:
|
Problem: Single sign-on is not occurring.
| Symptom | Description | User Action |
|---|---|---|
When tracing is enabled, the following message appears:
[2/27/06 14:28:04:191 CST] 00000059 SpnegoHandler < com.ibm.ws.security.spnego .SpnegoHandler handleRequest: Received a non-SPNEGO Authorization Header RETURN | The client is returning an NT LAN manager (NTLM) response to the authorized challenge, not a SPNEGO token. This condition can occur due to any of the following reasons:
|
If the SPN is defined incorrectly as HTTP/server.realm.com@REALM.COM with the addition of @REALM.COM, then delete the user, redefine the user, and redefine the SPN. If the problem is with the Kerberos keytab file, then regenerate the keytab file. If the problem is with either category of multiple SPN definitions, then remove the extra or conflicting SPN, confirm that the SPN is no longer registered with the Active Directory, and then add the SPN. We can search the Active Directory for other SPN entries that are causing multiple SPN definitions. The following commands are useful to determine multiple SPN definitions:
|
Problem: Credential Delegation is not working.
| Symptom | Description | User Action |
|---|---|---|
An invalid option is detected. When tracing is enabled, the following message is displayed:
com.ibm.security.krb5.KrbException, status code: 101 message: Invalid option in ticket request | The Kerberos configuration file is not properly configured. | Ensure that the renewable, or proxiable are set to true. |
Problem: Unable to get SSO working using RC4-HMAC encryption.
| Symptom | Description | User Action |
|---|---|---|
Examine the following message in the trace that you receive when trace is turned on:
com.ibm.security.krb5.internal.crypto. KrbCryptoException, status code: 0 message: Checksum error; received checksum does not match computed checksum | RC4-HMAC encryption is not supported by a Microsoft Windows version before the 2003 Kerberos key distribution center (KDC). To confirm this condition, examine
the trace and identify where the exception is thrown. The content of the incoming ticket should be visible in the trace. Although the incoming ticket is encrypted, the SPN for the service is readable.
If a Microsoft Windows version before the 2003 KDC is used and the system is configured to use RC4-HMAC, the string representing the ticket for userid@REALM (instead
of the expected HTTP/hostname.realm@REALM) is displayed.
For example, this is the beginning of a ticket received from a Microsoft Windows version prior to 2003 KDC:
0000: 01 00 6e 82 04 7f 30 82 04 7b a0 03 02 01 05 a1 ..n...0......... 0010: 03 02 01 0e a2 07 03 05 00 20 00 00 00 a3 82 03 ................ 0020: a5 61 82 03 a1 30 82 03 9d a0 03 02 01 05 a1 0a .a...0.......... 0030: 1b 08 45 50 46 44 2e 4e 45 54 a2 18 30 16 a0 03 ...REALM.COM.0.. 0040: 02 01 01 a1 0f 30 0d 1b 0b 65 70 66 64 77 61 73 .....0...userid 0050: 75 6e 69 74 a3 82 03 6e 30 82 03 6a a0 03 02 01 .a.f...n0..j....The realm is REALM.COM. The service name is userid. A correctly formed ticket for the same SPN is: 0000: 01 00 6e 82 04 56 30 82 04 52 a0 03 02 01 05 a1 ..n..V0..R...... 0010: 03 02 01 0e a2 07 03 05 00 20 00 00 00 a3 82 03 ................ 0020: 82 61 82 03 7e 30 82 03 7a a0 03 02 01 05 a1 0a .a...0..z....... 0030: 1b 08 45 50 46 44 2e 4e 45 54 a2 2a 30 28 a0 03 ..REALM.COM.0... 0040: 02 01 02 a1 21 30 1f 1b 04 48 54 54 50 1b 17 75 .....0...HTTP..u 0050: 73 31 30 6b 65 70 66 77 61 73 73 30 31 2e 65 70 serid.realm.com. 0060: 66 64 2e 6e 65 74 a3 82 03 39 30 82 03 35 a0 03 ...n.....90..5.. | To correct the problem, either use the Single data encryption standard (DES) or use a Microsoft Windows 2003 Server for a KDC. Remember to regenerate the SPN, and the Kerberos keytab file. |
Problem: User receives the following message when accessing a protected URL through the SPNEGO SSO.
| Symptom | Description | User Action |
|---|---|---|
Examine the following message:
Bad Request Your browser sent a request that this server could not understand. Size of request header field exceeds server limit. Authorization: Negotiate YII | This message is generated by the Apache/IBM HTTP Server. This server is indicating that the authorization header returned by the user's browser is too large. The long string that follows the word Negotiate (in the previous error message) is the SPNEGO token. This SPNEGO token is a wrapper of the Microsoft Windows Kerberos token. Microsoft Windows includes the user's PAC information in the Kerberos token. The more security groups that the user belongs to, the more PAC information is inserted in the Kerberos token, and the larger the SPNEGO becomes. IBM HTTP Server 2.0 (also Apache 2.0 and IBM HTTP Server 6.0) limit the size of any acceptable HTTP header to be 8K. In Microsoft Windows domains having many groups, and with user membership in many groups, the size of the user's SPNEGO token may exceed the 8K limit. | If possible, reduce the number of security groups the user is a member of. IBM HTTP Server 2.0.47 fix pack PK01070 allows for HTTP header sizes up to and beyond
the Microsoft limit of 12K. WAS v6.0
users can obtain this fix in fix pack 6.0.0.2.
Non-Apache-based web servers may require differing solutions. |
Problem: Even with JGSS tracing disabled, some KRB_DBG_KDC messages appear in the SystemOut.log.
| Symptom | Description | User Action |
|---|---|---|
| Examine theSystemOut.log and note that some KRB_DBG_KDC messages appear there even with JGSS tracing disabled. | While most of the JGSS tracing is controlled by the com.ibm.security.jgss.debug property, a small set of messages are controlled by the com.ibm.security.krb5.Krb5Debug property. The com.ibm.security.krb5.Krb5Debug property has a default value to put some messages to the SystemOut.log. | .To remove all KRB_DBG_KDC messages from the SystemOut.log, set the JVM property as follows:
|
Problem: An application contains a custom HTTP 401 error page, the SPNEGO TAI-generated HTTP response page is not displayed in a browser.
| Symptom | Description | User Action |
|---|---|---|
| When an application contains a custom HTTP 401 error page, the SPNEGO trust association interceptor (TAI)-generated HTTP response page is not displayed in a browser. | When an application contains a custom HTTP 401 error page, the SPNEGO TAI-generated HTTP response page is not displayed in a browser. The custom HTTP 401 error page is displayed instead. | We can customize the HTTP 401 page to include information concerning how to configure the browser to use SPNEGO. For more information, see Configure the client browser to use SPNEGO TAI (deprecated) and SPNEGO TAI custom properties configuration (deprecated). |
Problem: HTTP Post parameters are lost during interaction with the SPNEGO TAI, when stepping down to userid/password login.
| Symptom | Description | User Action |
|---|---|---|
| HTTP Post parameters are lost during interaction with the SPNEGO TAI, when stepping down to userid/password login.
"Stepping down to userid/password login" means that the Internet Explorer tries to respond initially with a SPNEGO token. If this response is unsuccessful, then the Internet Explorer tries to respond with a NTLM token that is obtained through a userid/password challenge. | The Internet Explorer maintains state during a user's request. If a request was given the response of an "HTTP 401 Authenticate Negotiate", and the browser
responds with a NTLM token obtained through a userid/password challenge, the browser resubmits the request. If this second request is given a response of an HTML page containing a redirection to the same URL but with new arguments (via Javascript), then the browser does not resubmit the POST parameters.
To avoid this problem, it is critical to NOT perform the automatic redirection. If the user clicks a link, the problem does not occur. |
The browser responds to the Authenticate/Negotiate challenge with an NTLM token, not an SPNEGO token. The SPNEGO TAI sees the NTLM, and returns a HTTP 403 response, along with theHTML page. When the browser runs the JavaScript redirTimerfunction, any POST or GET parameters that were present from the original request are lost. By using the SPN<id>.NTLMTokenReceivedPage property, an appropriate message page can be returned to the user. The default message returned (in the absence of a user-defined property) is: "<html><head><title>An NTLM Token was Received. </title></head>" + "<body>Your browser configuration is correct, but we have not logged into a supported Windows Domain." + "<p>Please login to the application using the normal login page.</html>"; Use the SPN<id>.NTLMTokenReceivedPage property, we can customize the exact response. It is critical that the returned HTML not perform a redirection. When the SPNEGO TAI has been configured to use the shipped default HTTPHeaderFilter class as the SPN<id>.filterClass, then the SPN <id>.filter can be used to allow the second request to flow directly to the normal WAS security mechanism. In this way, the user experiences the normal authentication mechanism. An example of such a configuration follows showing the required SPNEGO TAI properties necessary and the HTML file content. ****** SPNEGO TAI Property Name ******
****** HTML File Content ******
com.ibm.ws.security.spnego.SPN1.hostName server.wasteched30.torolab.ibm.com
com.ibm.ws.security.spnego.SPN1.filterClass com.ibm.ws.security.spnego.HTTPHeaderFilter
com.ibm.ws.security.spnego.SPN1.filter request-url!=noSPNEGO
com.ibm.ws.security.spnego.SPN1.NTLMTokenReceivedPage File:///C:/temp/NTLM.html
Observe that the filter property instructs the SPNEGO TAI to NOT intercept any HTTP request containing the string "noSPNEGO". Here is an example of a generating a helpful response. <html>
<head>
<title>NTLM Authentication Received </title>
<script language="javascript">
var purl=""+document.location;
if (purl.indexOf("noSPNEGO")<0) {
if(purl.indexOf('?')>=0) purl+="&noSPNEGO";
else purl+="?noSPNEGO";
}
</script>
</head>
<body>
<p>An NTLM token was retrieved in response to the SPNEGO challenge. It is likely that you are not logged into a Windows domain.<br>
Click on the following link to get the requested website.
<script language="javascript">
document.write("<a href='"+purl+"'>");
document.write("Open the same page using the normal authentication mechanism.");
document.write("</a><br>");
</script>
You will not automatically be redirected.
</body>
</html>
|
Problem: The trust association interceptor (TAI) does not call the initialize(Properties) method
Tracing might show that TAI is loaded, but that the initialize(Properties) method is not called. Only the getVersion() method appears to be called during startup.
WebSphere's TAI processing only calls initialize(Properties) when there are custom properties defined for the TAI.
To fix this issue, define an unused TAI custom property, such as com.ibm.issw.spnegoTAI.NumberOfServers=0.
Problem: The trust association interceptor (TAI) is not loading properly
Tracing might show that TAI is not loading and the following exception text is received:
SPNEGO014: Kerberos initialization Failure: org.ietf.jgss.GSSException, major code: 13, minor code: 0 major string: Invalid credentials minor string: SubjectKeyFinder: no JAAS Subject at com.ibm.security.jgss.i18n.I18NException.throwGSSException(I18NException.java:12)
A possible cause for this is that the JVM custom property javax.security.auth.useSubjectCredsOnly is not set to a value of false.
To fix this issue, define a JVM custom property on each JVM that is enabled for the TAI, javax.security.auth.useSubjectCredsOnly=false.
Problem: JACL scripts default characters for adding trust association interceptor (TAI) parameters can cause issues
JACL scripts for adding TAI parameters accept positional parameters. To accept the defaults, a "." is specified. On some WebSphere platforms, if specified a "." it can cause the property to be added with a value of ".".
Always (regardless of platform), confirm that the properties added are as expected using the console. If they are not, manually correct them.
Related concepts
Related tasks