Kerberos and WebLogic Server - decisions, decisions

I've been away from the blog for a while - I took some vacation time and I've been working on a customer that I can't really talk about. Have you missed me?

No? I didn't think so.

I got a call yesterday asking me to help a customer trying to get SPNEGO/Kerberos working between Windows desktops and WebLogic server. Yes, Kerberos again. After talking through what they're trying to accomplish today I found that they may have started out on the wrong foot to begin with.

Here's (more or less) what they told me they're trying to accomplish:

We have an app deployed in WebLogic and we want users that are in the domain to be able to access that app without being prompted for credentials.

That's it.

There are three ways that I can think of right away to address that requirement:

1. Use WebLogic's SPNEGO Identity Asserter to have WebLogic verify the Kerberos ticket
2. Use IIS to do the Kerberos bits and then use a simple HTTP header Identity Asserter to get the identity into WebLogic
3. Use Oracle Access Manager

Each of the solutions have their own plusses and minuses. The point of this post is to try to get them written down to help you (or someone else) make a sensible decision about which one to pick.

Before I get into it let me just say that Kerberos/SPNEGO is one of those technologies that when it works it's awesome, and when it doesn't it's incredibly frustrating to figure out what's gone wrong. Add to the mix the fact that so much of what's happening is automatic and so few people actually understand what's really happening under the covers/on the wire and you have a recipe for pain, frustration, annoyance and lots of hair being pulled out.

With that said...

WLS + SPNEGO Identity Asserter

WebLogic includes an the necessary bits that allow WebLogic to verify a Kerberos ticket if you want. I've blogged about it before and there are official docs as well as oodles of threads on Oracle forums where people are struggling to get it working. It definitely works and it has gotten easier over the years, but it's still harder than flipping a switch and being done.

The requirements basically amount to getting a Kerberos service principal registered, getting the necessary secret key onto the box running WebLogic and then getting WebLogic and IE to agree to do SPNEGO. Each of those steps is manual and it's easy to make a mistake.

Plusses: baked into WebLogic so there are no dependencies on any other software, Kerberos ticket is verified inside WebLogic itself and is thus about as secure as possible.

Minuses: can be difficult to setup & troubleshoot

Use IIS to do the hard stuff

One of the great things about Microsoft embracing Kerberos is that some very difficult, very complicated things have been reduced to a checkbox. To enable Kerberos in IIS you add the web server to the domain, then you go into the IIS console, edit the Security properties of a directory or of the server and check one box - "Integrated Windows Authentication". Windows and IIS do all the hard stuff for you including setting up the Kerberos principal.

Yes, I'm glossing over a few things, but not many.

Once you've got IE and IIS doing the Kerberos/SPNEGO dance two things remain: (1) getting IIS to proxy requests over to WebLogic and (2) getting WebLogic to trust IIS' say so on the user's identity. The former is address by the IIS WebLogic Plug-in. The latter is a simple Identity Asserter like the one I discussed back in January (I really need to put that code up somewhere on the Internets!).

Plusses: One checkbox to enable Kerberos in IIS

Minuses: You have to deploy Windows & IIS in front of every WebLogic server. The security issues I mention in my older post

Use OAM

OAM provides two relevant things - SSO across web sites and access control to those web sites. The former is all we're interested in here; when a user accesses the web server OAM will detect that they haven't authenticated yet and kick them into a process to do that. The latter allows you to decide who gets access to what parts of the protected web sites; it's not something we care about in this case but it can be very useful.

The upside to using OAM in this case is that you only need one or two IIS web servers somewhere in your environment to get Kerberos working. Each of the OAM WebGates plugged into your protected web servers will redirect users to one of those machines to figure out who the user is and then back to the protected web server. This works regardless of which flavor of web server you use - OHS, Apache, IIS, or Sun One. I've seen customers do this sort of thing with as little as one IIS server, but for redundancy you want a pair of IIS boxes somewhere in your environment and probably a second pair in a disaster recovery environment as well.

Plusses: Few Windows/IIS servers needed.

Minuses: Requires additional software (OAM WebGate) to be installed

In this particular customer's case they'd tried the first option already and had run into some issues. Because of those issues and because their requirements were still in flux I suggested that they try the second option (IIS doing Kerberos along with a username Identity Asserter). Your mileage may vary and you should probably ask a knowledgable person for advice before picking one of these options, but that's (part of) what we're here for.

I'm interested in what you think. Let me know in the comments!

Testing your WebLogic/Kerberos setup

I spent several hours today beating my head against a wall and thought I'd post this in case anyone else runs into the same problem.

I have a virtual environment with two machines, one is a KDC and one is just a WebLogic server. I setup WebLogic to do Kerberos, following the directions in the the documentation and could not figure out what I was doing wrong.

Whenever I accessed WebLogic I kept seeing the header "Authorization: Negotiate TlRMTVNTUAAB..." with only a short string as the token. If you base64 decode that string you see that it starts with NTLMSSP. In other words most definitely NOT a Kerberos token.

It looks like IE6 and IE 8 on Windows 2003 figures out that it's talking to the local machine (i.e. localhost) even if you use the fully qualified domain name!

So if you're trying to get Kerberos working make sure you use a web browser on a different machine. Or just use Firefox instead.

Kerberos and WebLogic Server on Windows step-by-step

The month of Kerberos continues

I got a frantic call late last week asking for help getting WebLogic and Kerberos working. WebLogic would be deployed on Windows but, unlike in my previous post, this customer wanted IE to talk directly to WebLogic with no IIS server in between.

Easy enough, right? It is, but there are some nuances.

There are official docs available from download.oracle.com, but some people find them confusing. These are a bit simplified and are intended for this configuration only.

In this environment the desktop and web server machines are both in the same Windows domain and thus also in the same Kerberos Domain. It's important to note that in my case the machine is part of the domain, but it doesn't actually have to be in the domain to get configuration working.

Prerequisites:

• You are using WebLogic with JDK 1.6 (either JRockit or the Sun JDK)
• You have the Active Directory domain created
• You have downloaded and installed WebLogic and may have created a domain
• You are frustrated because the docs are confusing and despite working on this for hours you still can't get Kerberos authentication working

Setup the Kerberos KDC bits...

Before you do anything else we need to make sure that you're starting in a "good" state.

Delete any keytab files you've already created. Delete any cached keys (del "%USERPROFILE%\krb5cc*"). Clear out any config files you've created. And log out and back into the account in which you're running Internet Explorer. Also make keep my previous post in mind while you're testing.

If you have installed IIS on the machine uninstall it. IIS registers the kerberos service principal HTTP/machine and HTTP/machine.domain.com and if you leave IIS installed you'll never manage to get Kerberos on WebLogic working correctly.

Install the setspn utility. Info on the utility is available on MS TechNet, and it is installed as part of the Windows Server 2003 Support Tools from the Windows product CD.

Run the command "ldifde -f c:\export.txt" on your Windows server. This will export the entire contents of your Active Directory to a flat file so we can search it. What we need to do is make sure that no user or machine has already registered the Kerberos Service Principal we need for WebLogic. If you don't know what that means don't worry about it, just search the export.txt file for HTTP/machine (where machine is the name of the machine). If you find it in the file you'll need to use the setspn utility to remove the mapping. Here's what my export contained when things were broken:

dn: CN=WEBSERVER,CN=Computers,DC=kerbtest,DC=com
servicePrincipalName: HOST/WEBSERVER
servicePrincipalName: HOST/webserver.kerbtest.com
servicePrincipalName: HTTP/webserver
servicePrincipalName: HTTP/webserver.kerbtest.com

Use the setspn utility to remove the extraneous mappings. In my example above I had a mapping from the machine named "WEBSERVER" to the two principals HTTP/webserver and HTTP/webserver.kerbtest.com. I needed to remove those mappings before continuing on to the next step. You have to use the setspn utility with the -D flag to remove the mappings like so:

C:\>setspn -d HTTP/webserver webserver
Unregistering ServicePrincipalNames for CN=WEBSERVER,CN=Computers,DC=kerbtest,DC=com
HTTP/webserver
Updated object
C:\>setspn -d HTTP/webserver.kerbtest.com webserver
Unregistering ServicePrincipalNames for CN=WEBSERVER,CN=Computers,DC=kerbtest,DC=com
HTTP/webserver.kerbtest.com
Updated object

Finally we need to create a new user in the domain so that WebLogic can get the Kerberos secret associated with the Kerberos SPN. The official docs and basically everything else you'll find on the Internet says to make the username the same as the machine name; I disagree strongly. If you give the machine and user the same name you'll just confuse yourself later trying to figure out which is which plus the setspn tool assumes you're talking about the machine and there's no way to tell it to operate on the user instead. So when you create the username make it generic, for example "wlsuser", "webuser" or "webserveruser".

Once you've created that user use the setspn utility to associate the HTTP/machine and HTTP/machine.domain.com principals with the user you just created. Yes, you are going to associate the SPN with the user and NOT with the machine. I'll explain why later, but this is really important! When you do this you want HTTP to be in all caps and the machine name to be all lower case. The capitalization is important because the format has to match what Internet Explorer uses when it talks to the KDC. Here's what my output looks like:

C:\>setspn -a HTTP/webserver webuser
Registering ServicePrincipalNames for CN=web user,CN=Users,DC=kerbtest,DC=com
HTTP/webserver
Updated object
C:\>setspn -a HTTP/webserver.kerbtest.com webuser
Registering ServicePrincipalNames for CN=web user,CN=Users,DC=kerbtest,DC=com
HTTP/webserver.kerbtest.com
Updated object

You can use ldifde to export the contents of AD again just to be sure you've got it right.

Configure the WebLogic bits...

Install WebLogic as normal.
Create a new WebLogic domain using the Config Wizard.
Make a backup of config/config.xml. Don't ask, just do it!
Open the WebLogic console (http://localhost:7001/console by default)
In the left Domain Structure navigation menu click on Security Realms, then click on myrealm.

Click the Providers tab and you should see the list of Authentication Providers.

Click the New button to create a new authentication provider and select Negotiate from the drop down.

Go into the Identity Asserter's configuration and click on the Provider Specific tab. We don't want the NegotiateIdentityAsserter to run when you deploy an application with the authentication method set to Forms or Basic, so uncheck the "Form Based Negotiation Enabled" box.

Note that once you hit save and restart WebLogic you'll probably be locked out of the console until we do the next bunch of steps. If you wind up in trouble just back out the above changes by swapping your backup of config.xml in, restarting WebLogic and your web browser.

Copy a test app that requires Certificate authentication into your autodeploy directory. Then restart your server. If you don't have such app you'll have to create one.

When you access the app you should get a 401 error page from WebLogic. If you do an HTTP trace you should see an HTTP transaction that looks something like this

HTTP/1.1 401 Unauthorized
Date: Thu, 04 Feb 2010 21:44:10 GMT
Content-Length: 1518
Content-Type: text/html
WWW-Authenticate: Negotiate

the key to look for is WWW-Authenticate: Negotiate. If you see that the Negotiate Identity Asserter is running and you can go ahead and finish up the Kerberos setup.

Back to more Kerberos stuff...

We need to create the kerberos ini and keytab files:

Create a c:\windows\krb5.ini file that looks like the following. Update the domain and other information to reflect your environment:

[libdefaults]
default_realm = KERBTEST.COM
ticket_lifetime = 600

[realms]
KERBTEST.COM = {
kdc = 10.99.2.181
admin_server = testmachine
default_domain = KERBTEST.COM
}

[domain_realm]
.kerbtest.com = KERBTEST.COM

[appdefaults]
autologin = true
forward = true
forwardable = true
encrypt = true

In the official docs and all over the Internet you will find similar files with lines that start with default_tkt_enctypes and default_tgs_enctypes. As long as you are using JDK 1.6 you should be able to leave them out. The "kdc" setting is the IP address of your KDC and the "admin_server" setting is the host name of your KDC.

Run the command "c:\Oracle\Middleware\wlserver_10.3\common\bin\commEnv.cmd" to set your path and other environment variables so that you can run the Kerberos-related tools like ktab and kinit. Then cd into your domain's directory. In my case that's c:\Oracle\Middleware\user_projects\domains\base_domain.

The Kerberos related classes included with the JDK require a config file to run properly. Basically this file tells the GSS layer which classes are used to do the actual work and provides configuration information to those classes. Again, don't worry about what any of this means, just do the same thing I do and you'll be fine. Use notepad to create a krb5login.conf file with these contents:

com.sun.security.jgss.initiate {
 com.sun.security.auth.module.Krb5LoginModule required
 principal="webuser@KERBTEST.COM"
 useKeyTab=true
 keyTab=keytab
 storeKey=true
 debug=true;
};

com.sun.security.jgss.accept {
 com.sun.security.auth.module.Krb5LoginModule required
 principal="webuser@KERBTEST.COM"
 useKeyTab=true
 keyTab=keytab
 storeKey=true
 debug=true;
};

com.sun.security.jgss.krb5.accept {
 com.sun.security.auth.module.Krb5LoginModule required
 principal="webuser@KERBTEST.COM"
 useKeyTab=true
 keyTab=keytab
 storeKey=true
 debug=true;
};

We need to create the keytab file I specified in the krb5login.conf file above. To create the file you use the ktab tool, and to verify its contents you use the kinit command line tool. Both of these tools come with the JDK so you want to make sure that you're using the same JDK as WebLogic is going to use. The instructions you find in the docs talk about using the ktab and kinit command line utilities directly, and if you want you can use them. I on the other hand like lots and lots of debugging information, so I prefer to invoke the same classes by running java.exe and passing a few extra command line options. Do whichever you feel more comfortable with and as long as kinit works everything is fine.

First create the keytab with kinit:

C:\Oracle\Middleware\user_projects\domains\base_domain>java.exe -Dsun.security.krb5.debug=true sun.security.krb5.internal.tools.Ktab -k keytab -a webuser@KERBTEST.COM
Password for webuser@KERBTEST.COM:abcd1234
Config name: C:\WINDOWS\krb5.ini
Using builtin default etypes for default_tkt_enctypes
default etypes for default_tkt_enctypes: 3 1 23 16 17.
>>> KeyTabEntry: key tab entry size is 46
>>> KeyTabEntry: key tab entry size is 46
>>> KeyTabEntry: key tab entry size is 54
>>> KeyTabEntry: key tab entry size is 62
>>> KeyTabEntry: key tab entry size is 54
Done!
Service key for webuser@KERBTEST.COM is saved in keytab

If you don't see ktab create a keytab file, or if the file winds up being just a few bytes then don't proceed any further since it won't work; instead go back and check the first bunch of steps - exporting the AD contents, checking the user you created, etc.

Next run kinit to insure that the keytab file is OK and that your configuration files are all setup correctly.

C:\Oracle\Middleware\user_projects\domains\base_domain>java.exe -Dsun.security.krb5.debug=true sun.security.krb5.internal.tools.Kinit -k -t keytab webuser@KERBTEST.COM
>>>KinitOptions cache name is C:\Documents and Settings\administrator.KERBTEST\krb5cc_administrator
Principal is webuser@KERBTEST.COM
>>> Kinit using keytab
>>> Kinit keytab file name: keytab
>>> KeyTabInputStream, readName(): KERBTEST.COM
>>> KeyTabInputStream, readName(): webuser
>>> KeyTab: load() entry length: 46; type: 3
>>> KeyTabInputStream, readName(): KERBTEST.COM
>>> KeyTabInputStream, readName(): webuser
>>> KeyTab: load() entry length: 46; type: 1
>>> KeyTabInputStream, readName(): KERBTEST.COM
>>> KeyTabInputStream, readName(): webuser
>>> KeyTab: load() entry length: 54; type: 23
>>> KeyTabInputStream, readName(): KERBTEST.COM
>>> KeyTabInputStream, readName(): webuser
>>> KeyTab: load() entry length: 62; type: 16
>>> KeyTabInputStream, readName(): KERBTEST.COM
>>> KeyTabInputStream, readName(): webuser
>>> KeyTab: load() entry length: 54; type: 17
Added key: 17version: 1
Added key: 16version: 1
Added key: 23version: 1
Added key: 1version: 1
Added key: 3version: 1
Ordering keys wrt default_tkt_enctypes list
Config name: C:\WINDOWS\krb5.ini
Using builtin default etypes for default_tkt_enctypes
default etypes for default_tkt_enctypes: 3 1 23 16 17.
>>> Kinit realm name is KERBTEST.COM
>>> Creating KrbAsReq
>>> KrbKdcReq local addresses for webserver are:

      webserver/10.99.2.133
IPv4 address
Using builtin default etypes for default_tkt_enctypes
default etypes for default_tkt_enctypes: 3 1 23 16 17.
>>> KrbAsReq calling createMessage
>>> KrbAsReq in createMessage
>>> Kinit: sending as_req to realm KERBTEST.COM
>>> KrbKdcReq send: kdc=10.99.2.181 UDP:88, timeout=30000, number of retries =3,
#bytes=169
>>> KDCCommunication: kdc=10.99.2.181 UDP:88, timeout=30000,Attempt =1, #bytes=1
69
>>> KrbKdcReq send: #bytes read=210
>>> KrbKdcReq send: #bytes read=210
>>> reading response from kdc
>>> KDCRep: init() encoding tag is 126 req type is 11
>>>KRBError:
       sTime is Sun Feb 07 14:40:09 EST 2010 1265571609000
       suSec is 135477
       error code is 25
       error Message is Additional pre-authentication required
       realm is KERBTEST.COM
       sname is krbtgt/KERBTEST.COM
       eData provided.
       msgType is 30
>>>Pre-Authentication Data:
       PA-DATA type = 11
       PA-ETYPE-INFO etype = 23
>>>Pre-Authentication Data:
       PA-DATA type = 2
       PA-ENC-TIMESTAMP
>>>Pre-Authentication Data:
       PA-DATA type = 15
Kinit: PREAUTH FAILED/REQ, re-send AS-REQ
>>>KrbAsReq salt is KERBTEST.COMwebuser
Pre-Authenticaton: find key for etype = 23
AS-REQ: Add PA_ENC_TIMESTAMP now
>>> EType: sun.security.krb5.internal.crypto.ArcFourHmacEType
>>> KrbAsReq calling createMessage
>>> KrbAsReq in createMessage
>>> Kinit: sending as_req to realm KERBTEST.COM
>>> KrbKdcReq send: kdc=10.99.2.181 UDP:88, timeout=30000, number of retries =3,
#bytes=240
>>> KDCCommunication: kdc=10.99.2.181 UDP:88, timeout=30000,Attempt =1, #bytes=2
40
>>> KrbKdcReq send: #bytes read=1243
>>> KrbKdcReq send: #bytes read=1243
>>> reading response from kdc
>>> EType: sun.security.krb5.internal.crypto.ArcFourHmacEType
>>> KrbAsRep cons in KrbAsReq.getReply webuser
New ticket is stored in cache file C:\Documents and Settings\administrator.KERBTEST\krb5cc_administrator

Notice that line "New ticket is stored". If you get that line it means that the Kerberos configuration is correct and that the Java code was able to acquire the secret key.

Last configuration step...

Edit the bin\startWebLogic.cmd to add the line set JAVA_OPTIONS below in the place I've shown it:

echo starting weblogic with Java version:

%JAVA_HOME%\bin\java %JAVA_VM% -version

set JAVA_OPTIONS=%JAVA_OPTIONS% -Dsun.security.krb5.debug=true -Djavax.security.auth.useSubjectCredsOnly=false -Djava.security.auth.login.config=krb5Login.conf -Djava.security.krb5.realm=KERBTEST.COM -Djava.security.krb5.kdc=testmachine.kerbtest.com

Restart WebLogic, access the site with IE and revel in your success.

Problems? Issues?

Hit me up in the comments!