Thursday, February 14, 2013

Part 2: How to Configure OAM11g WNA for Multiple AD Forests

This post is part of a larger series on Oracle Access Manager 11g called Oracle Access Manager Academy.  An index to the entire series with links to each of the separate posts is available.

This is the second post of a three part series.  In "Part 1: Under the Covers of OAM11g WNA integration with Multiple AD Forests", I covered the flow of how WNA works and what was going on behind the scenes.  This article will cover the technical details on how to implement WNA in a way that will support multiple Active Directory Forests that either have no transient trust between them, or even all trusted; in either case this will work for you.

Before we get into the details on how to setup WNA for multi Active Directory domains I just want to point out that I will use a straw man of three Active Directory KDC servers so you can understand any additional steps needed to support more than one KDC.  However, this would also work for as few as one domain, or more than three domains.  All that is needed is to simply extrapolate the steps to fit your requirements; I will be sure to comment where necessary.


Patch It Up!


Out-of-the-box the standard OAM Kerberos plugin can be used to accomplish basic WNA integration against a single domain, which works perfectly without any need to install a patch.  However this solution is about authenticating against multiple forests or domains, and for that a custom Kerberos authentication module is required.  Don’t worry, no coding necessary, this is all out-of-the-box using the OAM Console.  From my experience the custom Kerberos authentication module did not even work until OAM11gR1 was patched up to BP05, but if you are using OAM11gR2, that should work unpatched; more on that later.  You may also want to know, that using this custom Kerberos authentication module is a good thing because the standard LDAP, Kerberos, and x509 authentication modules will soon be depreciated.

So first things first, this solution requires that OAM11gR1 (11.1.1.5.0) be patched with bundle patch BP05 (patch 14760839 released on 1/15/2013).   If this is OAM11gR2 (11.1.2.0.1) there are no WNA fixes in any of the bundle patches up to the most recent BP02, but I recommend as a best practice to patch up to the latest regardless because it usually includes bug fixes, maybe some new features, and potential critical fixes for other reasons.  Just do it.  At the time this blog was written, the most current patch for OAM11gR2 is bundle patch BP02 (patch 14760806 released on 1/15/2013).

TIP:  Please review the readme.html that is included in the bundle patch zip file.  As of BP04, there are now two independent steps to the patching process, 1) run “opatch apply” to install the initial bundle patch, and 2) a post patch step where a script domainAutomation.sh is run to make changes to the domain and the oam-config.xml file.

OAM11g has historically been one of the tricky components to patch.  I have found that after running “opatch apply”, it is a good idea to start up the WebLogic Admin server and the OAM Managed Server being patched to make sure they come up correctly, if so, then bring them back down and verify both are completely down.  You can use netstat –na | grep <port; e.g. 14100, 7001> to determine if each are completely down.  Then go ahead with the post step to run the domainAutomation.sh script.  For any other questions on patching I recommend this great article “Patch Management of an Oracle Identity Management Deployment” by Brian Eidelman of the A-Team.

For your information, BP05 includes some current WNA bug fixes and some additional WNA related fixes that were inherited from BP04.  Fixes include inconsistent behavior with WNA, a fix that caused failed authentications when a duplicate samAccountName was found, and some updates to the oam-config.xml that provide a new KTA (KerberosTokenAuthenticator) parameter, which we will take advantage of; so patch it up!


Create the Service Principal Account


A service principal has to be created in each KDC server that will be used by the OAM Windows Native Authentication.  This is basically a User account, and does not need any special permission or belong to any group, and the User name can be different across individual KDC servers.  However, even though this new account is created like a regular User, it should be considered a service account.  The account is not meant for a User to login to a domain, it is only meant as a service principle that is mapped and encrypted to a keytab, which OAM will use for WNA against each respective KDC server.

TIP: As a security measure, I would advice to use a strong password and to locate the account in a container that is for service accounts only.  I would also use any additional measures to protect this account from being used in any rogue way.  It should also be noted if the password expires WNA will fail to work.

Creating the Master Keytab File


A keytab is a file that contains an unencrypted list of service principals and their keys from which the OAM Kerberos module will use to validate the User's Kerberos service ticket without the User needing to provide any password.  The first keytab we generate will be the seed keytab file, followed by appending additional keytab files, and the final keytab file for this solution will contain all the keytabs from each KDC server that will be used by WNA; I call it the master keytab file.  This single file is one of the keys to making WNA work against multiple untrusted Active Directory domains.

Step 1 – Generating the First Keytab File


To create a keytab file the ktpass tool is required.  This is a tool found in the Windows Support install that comes with Windows Server; later versions of Windows server like Windows 2008 R2 may already include this tool.  For more information on ktpass you can go to the Microsoft Technet site.   As a tip, if using Windows 2003 Enterprise server, the version of ktpass needs to be installed from the Support Tools that comes in SP1 or higher.  The base ktpass does not support the correct Kerberos RC4-HMAC protocol.  If this means you, Microsoft provides this as a free download from the Microsoft Download Center KB892777.

If you execute “ktpass /?” at the Windows command-line, in the list of options you should find crypto.  The newer version of ktpass will display RC4-HMAC-NT, this is the version of crypto that is required to make the proper keytab.  However, if using Windows 2008 R2 Server, AES-256 may be required; read further on comments about this and special syntax required.

[- /]       crypto : DES-CBC-CRC : for compatibility
[- /]       crypto : DES-CBC-MD5 : for compatibliity
[- /]       crypto : RC4-HMAC-NT : default 128-bit encryption

So the following command will need to be run on each KDC server, but the first one will need to be run as follows.  As a Windows Domain Administrator at a command-line use the following command; you will need to modify the values per your configuration and environment.  Below are two options based on which version of Windows Server you are using; differences are highlighted in red; credit goes to Michael Storrs for catching me on missing this subtle yet important difference.  Note the syntax in red relates to a KB article "OAM 11g WNA Login Fails When Using Windows 7/2008 Clients in an Active Directory 2008 R2 Server domain [ID 1509275.1]".  Another reason to use the Windows 2008 R2 syntax is to support clients that use Windows Vista, Windows 2007, and Windows 2008, because these clients do not use RC4-HMAC-NT encryption, instead they could be configured to use AES-256 encryption type.

Windows 2003 Server Syntax -

ktpass -princ HTTP/oam.server.com@FOREST1.SPRITE.COM /
-mapuser oamkrb5 /
-pass Oracle123 /
-out forest1.krb5.keytab

Windows 2008 R2 Server Syntax -

ktpass -princ HTTP/oam.server.com@FOREST1.SPRITE.COM /
-mapuser oamkrb5 /
-pass Oracle123 /
-ptype KRB5_NT_PRINCIPAL /
-crypto ALL /
-out forest1.krb5.keytab
Parameter Description
-princ HTTP/<oam_server_hostname>@<kdc_server_hostname>
-mapuser samAccountName of the service principal account
-pass Password for the security principal account
-ptype KRB5_NT_PRINCIPAL * This is the general principal type.
-crypto ALL * Tells ktpass to output all crypto types.
-out The keytab output file name
* Syntax for Windows 2008 R2 Server

This command will generate a keytab file forest1.krb5.keytab.  The file name is arbitrary; it could be anything you want.  However in order to keep things straight, I recommend prefixing the file name with the KDC server it was generated on so that there is no question of which KDC server the keytab was last generated.  This is more important for later steps.

After you run the command you should see some screen output similar to the following.  You will see more output if using the Windows 2008 R2 syntax.

Targeting domain controller: orcl.forest1.sprite.com
Using legacy password setting method
Successfully mapped HTTP/oam.server.com to oamkrb5.
WARNING: pType and account type do not match. This might cause problems.
Key created.
Output keytab to forest1.krb5.keytab:
Keytab version: 0x502
keysize 75 HTTP/oam.server.com@FOREST1.SPRITE.COM ptype 0 (KRB5_NT_UNKNOWN) vno 3 etype 0x17 (
RC4-HMAC) keylength 16 (0x8b2318524d2e3e2e31885afc21024cf5)

TIP:  In the bold highlighted output above, you can disregard the pType warning; this is a benign error.  Also make sure the crypto version shows an output of RC4-HMAC to show we are using the correct Kerberos crypto for Windows 2003, or AES-256 for Windows 2008.

To validate the keytab was correctly mapped to the service account, open Active Directory Users and Computers > navigate to the container the service account was created > right click on the service principal User account > select Properties > select the Account tab > you should now see in the User logon name field the value HTTP/oam.server.com has been added.  This shows that when ktpass was run it mapped this account to the keytab file properly when it was generated.


Step 2 – Appending Additional Keytabs to Create the Final Master Keytab File


Now we need to generate the remaining keytabs from each KDC server and continue to append each existing keytab to a new keytab file.  For example each additional KDC server that needs to be authenticated against, you will need to run the ktpass tool again, but with an additional parameter “-in” along with the last file <kdc_server_host>.krb5.keytab created.   This is an iterative process that will ultimately create a single file that will contain all the keytabs; the final file will be named the master.krb5.keytab file.

TIP:  In the previous step I pointed out it is a good idea to prefix the keytab file name with the KDC server name; e.g. forest1.krb5.keytab, forest2.krb5.keytab, etc.  The methodology behind this approach is to help know which KDC server the keytab was run from last so that it is less likely a mistake is made as you build up all the keytabs into a single master copy.  So it would be a good idea to list all the KDC servers in some the order and follow that list to run the ktpass tool in order to build the keytab files.

As you repeat this process on each KDC server, you will need to build a new keytab file in a systematic way.  For example use the following command on the next KDC server, Forest2, but first copy the forest1.krb5.keytab file to this server to use as the input file from the previous step the first time ktpass was run.  What happens is this command will generate a keytab for Forest2, but also append the keytab from file forest1.krb5.keytab to create a new file with two keytabs.  Note if you are using Windows 2008 R2 include the additional syntax highlighted in red.

ktpass -princ HTTP/oam.server.com@FOREST2.PIXIE.COM /
-mapuser oamkrb5 /
-pass Oracle123 /
-ptype KRB5_NT_PRINCIPAL /
-crypto ALL /
-in forest1.krb5.keytab /
-out forest2.krb5.keytab
Parameter Description
-in The keytab file that contains existing keytabs

Now on the next KDC server, Forest3, copy the forest2.krb5.keytab file to the forest3 server and run the same following command, and so on. You get the point. The new keytab file is a combination of all the previous keytabs generated. The final KDC server you run the command on will be your master file.  Note if you are using Windows 2008 R2 include the additional syntax highlighted in red.

ktpass -princ HTTP/oam.server.com@FOREST3.FAY.COM /
-mapuser oamkrb5 /
-pass Oracle123 /
-ptype KRB5_NT_PRINCIPAL /
-crypto ALL /
-in forest2.krb5.keytab /
-out forest3.krb5.keytab
For the final step in this process I would copy the last keytab file created to master.krb5.keytab as follows.
copy forest3.krb5.keytab master.krb5.keytab

Now we have all our keytabs from each KDC in a single master file.  Congrats! Combining the keytabs into a single file is critical, but also a bonus because there is only one file to protect, to configured, and to manage.

Now copy the master.krb5.keytab file to each of the OAM Servers.  You can place it anywhere as long as it is accessibly by the OAM server.  I recommend the file ownership be the same as the user account that can start and stop the OAM server.

Step 3 – Validating the Master Keytab 


Before you assume the master keytab file is ready, it is important to validate all the keytabs in order to verify it will work successfully before implementing it as part of the Kerberos Authentication Module. The first step is to use a Linux tool “klist”, which allows you to output the content of the master keytab. Run the following command on a Linux terminal session to see the contents of the master keytab file.  If you are not using Linux, search for the same tool that supports your platform.  I would run this same command against each master.krb5.keytab file on every OAM Server you copied it to.
klist -k -t -K -e /u01/oracle/wna/master.krb5.keytab
The output should look similar to what is shown below.   Notice it lists out each keytab line-by-line. See the HMAC/md5, this is the proper crypto.  Note that if you used the Windows 2008 R2 syntax it would output several other crypto versions of the keytabs.

Keytab name: FILE:/u01/oracle/wna/master.krb5.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   3 12/31/69 18:00:00 HTTP/iam.acme.com@FOREST1.SPRITE.COM (ArcFour with HMAC/md5)  (0x8b2318524d2e3e2e31885afc21024cf5)
   5 12/31/69 18:00:00 HTTP/iam.acme.com @FOREST2.PIXIE.COM (ArcFour with HMAC/md5)  (0x8b2318524d2e3e2e31885afc21024cf5)
   3 12/31/69 18:00:00 HTTP/iam.acme.com @FOREST3.FAY.COM (ArcFour with HMAC/md5)  (0x8b2338524d2e3a2e31885afc21024cf5)

Finally each keytab needs to be validated by using the command tool kinit.  This basically forces each keytab to try and authenticate against the respective KDC server.  Since the master.krb5.keytab file will be on each OAM Server, the OAM Server must be able to reach each KDC server across the network otherwise the authentication will fail.  To validate each keytab, run the following command for each keytab using the same master.krb5.keytab file as follows.  I would do this same thing on each of the OAM Servers the master.krb5.keytab file was copied to.

kinit -V HTTP/iam.acme.com@FOREST1.SPRITE.COM /
-k -t /u01/oracle/wna/master.krb5.keytab

Authenticated to Kerberos v5

kinit -V HTTP/iam.acme.com@FOREST2.PIXIE.COM /
-k -t /u01/oracle/wna/master.krb5.keytab

Authenticated to Kerberos v5

kinit -V HTTP/iam.acme.com@FOREST3.FAY.COM /
-k -t /u01/oracle/wna/master.krb5.keytab

Authenticated to Kerberos v5
Notice that each time the above commands is run a unique service principle specific to a KDC server is used?  Each time kinit is run you should get an output that proves the Kerberos authentication was successful against the KDC.   If you get any type of error you will need to either verify that the OAM Server can reach the KDC host, make sure there is no firewall causing problems, network issues, and worse case run through the previous steps again to recreate the master.krb5.keytab file.


Realm check, the /etc/krb5.conf


The /etc/krb5.conf file contains the Kerberos configuration information including a mapping to all the KDCs and admin servers for the Kerberos realms that the OAM Server will iterate through to validate the Kerberos ticket.   The following is an example of how our straw man KDC mapping would be configured.  If additional KDCs are required, simply copy the respective sections from the [realms] and [domain_realm] as needed to add more.  Note you will need to be root to make any changes to the krb5.conf file.  I have also highlighted in bold some lines that should also be present in the krb5.conf file.  Once you create this file it can be copied to all the other OAM servers.  NOTE:  Make sure that all the letters below that are in ALL CAPS remain ALL CAPS; e.g. FOREST1.SPRITE.COM should not be forest1.sprite.com.

[libdefaults]
 default_realm = FOREST1.SPRITE.COM
 ticket_lifetime = 600
 dns_lookup_realm = false
 dns_lookup_kdc = false
 forwardable = yes
 udp_preference_limit=1
 default_tkt_enctypes = RC4-HMAC
 default_tgs_enctypes = RC4-HMAC

[realms]
 FOREST1.SPRITE.COM = {
  kdc = forest1.sprite.com
  admin_server = forest1.sprite.com  
  default_domain = FOREST1.SPRITE.COM
 }
 FOREST2.PIXIE.COM = {
  kdc = forest2.pixie.com
  admin_server = forest2.pixie.com  
  default_domain = FOREST2.PIXIE.COM
 }
 FOREST3.FAY.COM = {
  kdc = forest3.fay.com
  admin_server = forest3.fay.com  
  default_domain = FOREST3.FAY.COM
 }

[domain_realm]
 .forest1.sprite.com = FOREST1.SPRITE.COM
 forest1.sprite.com = FOREST1.SPRITE.COM
 .forest2.pixie.com = FOREST2.PIXIE.COM
 forest2.pixie.com = FOREST2.PIXIE.COM
.forest3.fay.com = FOREST3.FAY.COM
 forest3.fay.com = FOREST3.FAY.COM
This completes creating the master keytab and krb5.conf configuration for all the Kerberos details needed for WNA; the hard part is over, and now for the fun part.

Creating the Custom Kerberos Authentication Module


The authentication module does a number of things to make this all work.  It has plugins that retrieve the Kerberos ticket, it will then iterate through the keytabs to find a match that will validate the user’s service ticket, and finally based on the domain the user logged into it will provide the correct searchbase to find the user in the correct forest, so that it can identify the correct person even if there are duplicate samAccountNames.  Now let’s create the custom Kerberos authentication module.   Not one line of code is needed, this is all out-of-the-box, so don’t sweat it.


Step 1 – Create the Kerberos Authentication Module

Logged into the OAM Console as an OAM Administrator, and navigate to System Configuration > Access Manager Settings > expand Authentication Modules > select Custom Authentication module > click on the create button that has the plus sign.  Then enter a name for the module, say “wnaMultiDomainAuthnModule”, and enter a description, say “WNA Multi-domain Authentication Module”.


Step 2 – Create the Steps

It is important to read this and pay close attention in this section.

WARNING:  During the creation of each step DO NOT click the Apply or Save button, otherwise this will cause problems when configuring the steps in the Steps Orchestration tab.  If you accidently click Apply or Save, I recommend deleting the authentication module and start from the beginning.

Select the Steps tab, then create a step by clicking the plus sign, enter the values, and click OK.


Repeat this process for each step in the exact order referenced in the table below.

Step Name Description Plugin Name
stepKTI KTI KerberosTokenIdentifier
stepKTA KTA KerberosTokenAuthenticator
stepUIF UIF UserIdentificationPlugin
stepUI UI UserIdentificationPlugin
stepUA UA UserAuthenticationPlugin

When completed you should have five steps that will look like the following screenshot.



Step 3 – Configure the Steps Orchestration


This section is where you order the steps created previously.
WARNING:  Do not click Apply until everything in this section is completed.
Select the Steps Orchestration tab > then select “stepKTI” as the Initial Step > and then configure the orchestration of the remaining steps per the table below.

Step Name On Success On Failure On Error
stepKTI stepKTA stepUI failure
stepKTA stepUIF failure failure
stepUIF success failure failure
stepUI stepUA failure failure
stepUA success failure failure

The following is a screenshot of what the Steps Orchestration panel should look like when finished.



After all the steps have been configured properly you can finally click the Apply button, and you should get a confirmation message that says, “Authentication Module created successfully”.  Now go on to Step 4 to configure the parameter values found in the Steps tab.


Step 4 – Fill in the Parameter Values for the Steps


We are nearly there!  Now click on the Steps tab, and for each Step Name found in the table below, enter the values for each parameter.  Make sure to click Save and then Apply after entering the values for each step to make sure the data has been saved.   Note that our straw man only has three KDC servers, but if there are more or less, simply add or subtract the values as needed in the KEY_DOMAIN_DNS2DN_MAP parameter.  For example the value should be KDC server hostname in all CAPS, then colon, and then top namespace of the KDC server.  Use a semi-colon as a delimiter to separate each KDC server, so add as many as required per your environment.  Note that you should replace the <IdentityStoreName> value with the actual name of your identity store found in the OAM Console.

Step Name Parameter Value
stepKTA KEY_KRB_CONFIG_FILE /etc/krb5.conf
KEY_PRINCIPAL HTTP/oam.server.com@forest1.sprite.com
KEY_DOMAIN_DNS2DN_MAP FOREST1.SPRITE.COM:dc=forest1,dc=sprite,dc=com; FOREST2.PIXIE.COM:dc=forest2,dc=pixie,dc=com; FOREST3.FAY.COM:dc=forest3,dc=fay,dc=com
stepUIF KEY_IDENTITY_STORE_REF <IdentityStoreName>
KEY_LDAP_FILTER (uid={KEY_USERNAME})
KEY_SEARCH_BASE_URL {KEY_USERDOMAIN}
stepUI KEY_IDENTITY_STORE_REF <IdentityStoreName>
stepUA KEY_IDENTITY_STORE_REF <IdentityStoreName>

Also don’t be concerned that the KEY_PRINCIPAL value is only the first service principal for the first domain, because OAM will iterate through the keytabs found in the master.krb5.keytab file and find the correct authentication match for the User based on what KDC they logged into.
TIP:  The KEY_SEARCH_BASE_URL uses a special parameter {KEY_USERDOMAIN}.  This parameter basically tells the plugin to grab the domain the User logged in from, then uses that name space to set the searchbase when searching in OVD.  For example if the user is rey.strong@forest1.sprite.com, the plugin would grab the domain and construct (dc=forest1,dc=sprite,dc=com) as the searchbase.  This helps deal with duplicate samAccountNames across forests.  

Another helpful parameter is the  KEY_LDAP_FILTER.  It is a parameter that uses a value to tell the plugin which attribute to search for the user's samAccountName. So for example in the table above I used the value (uid={KEY_USERNAME}) to tell OAM which attribute to search with against the identity store.  I used the attribute uid because my OVD adapter is using the OAM/AD Adapter with Mapper, which maps standard inetOrgPerson attributes to Microsoft implementation of User object classes.  In this case uid would translate to sameAccountName; this is also illustrated in Part 1 of section "Searching for the Authenticated User in the Identity Store".  So basically change the attribute to what your OVD adapter is mapping to the attribute samAccountName.  

In Part 3, I will elaborate on things like using an Enterprise Directory Server instead of OVD, or including options to search for the UPN in your Identity Store if that is what you wish.  As a sneak preview I will give you a syntax value for the parameter KEY_LDAP_FILTER now; (userprincipalname={KEY_USERNAME}@{KEY_USERREALM}), more on that in Part 3.

Step 5 – Apply the new Authentication Module to an Authentication Scheme


Now that we have our required Kerberos authentication module, it needs to be applied to an authentication scheme, which can then be used in various Application Domains.

So to create an Authentication Scheme login to the OAM console as an OAM Administrator > navigate to Policy Configuration > Shared Components > Authentication Schemes > and select the KerberosScheme.  Click the duplicate button to make a copy of this scheme and fill in the values per the following table below.  Feel free to name the scheme to what ever you would like.

Parameter Value
Name WNA_AuthnScheme
Description WNA Kerberos Scheme
Authentication Level 2
Default <leave unchecked>
Challenge Method WNA
Challenge Redirect URL /oam/server/
Authentication Module wnaMultiDomainAuthnModule
Challenge Parameters <leave blank>



Now navigate to any of the Protected Resource Policies that require WNA and apply the new authentication scheme.


Configuring Integrated Windows Authentication


Before you can test WNA, IWA needs to be configured in each user's browser otherwise the default NTLM will kick in, which is where you get a pop-up login prompt.  However, in some cases the browser may not be supported over a reverse proxy, so this solution is best used on Intranets.  This section contains the instructions on configuring IWA for four of the most common Internet browsers.

Internet Explorer 2+ (Windows Only)

The following steps may vary slightly for each version of IE.
  1. Open Internet Explorer.
  2. Select Tools > Internet Options.
  3. Select the Local intranet zone and click Sites > Advanced.
  4. Enter all the site hostnames that are protected including the OAM hosts where the URL goes to “http://oam.server.com:14100/oam/server” used for authentication processing.
  5. Click OK > OK.
  6. Select the Advanced tab and make sure under Security > Enable Integrated Windows Authentication option is selected.
  7. Click OK to save the changes and close the Internet Options dialog.
TIP: In step 4, if there is a common top domain for multiple hosts, for example my.sprite.com, help.sprite.com, etc. you can use an asterisks and enter “http://*.sprite.com” to cover all combinations that match that top domain.

FireFox 3+ (Windows or OSX)

The following steps may vary slightly for each version of FireFox.
  1. Open FireFox.
  2. In the address bar enter “about:config” and press enter.
  3. Click the button “I’ll be careful, I promise!”.
  4. Using the table below type in the Preference Name to find the parameter to configure and change the value according to the Value column in the table.  NOTE:  Make sure for any http(s) values you enter the all the site hostnames that are protected along with the OAM host  where the URL goes to "http://oam.server.com:14100/oam/server" used for authentication processing.
  5. Restart FireFox.
Preference Name Type Value
network.automatic-ntlm-auth.trusted-uris string http://,https://
network.negotiate-auth.allow-proxies boolean true
network.negotiate-auth.delegation-uris string http://,https://
network.negotiate-auth.gsslib string (default:empty) - Specifies alternate GSSAPI shared library
network.negotiate-auth.trusted-uris string http://,https://
network.negotiate-auth.using-native-gsslib boolean true

TIP:  If there is a common top domain, for example my.sprite.com, help.sprite.com, etc. you can just enter “.sprite.com” to cover them all.  For multiple hosts use a comma to delimit the list of hosts.

Chrome 8+ (Windows or OSX)

Follow the steps in the section above for IE, and Google Chrome will automatically use the Internet Options.  No special configuration is needed beyond the computer being a domain member.

Safari 4+ (OSX Only)

Safari is enabled for IWA automatically on OSX.  No special configuration is needed beyond the OSX computer being a domain member.


Troubleshooting

Adding Kerberos Debugging to the OAM Server

The following steps will provide Kerberos debugging output from the OAM Server logs.  This is useful to review the Kerberos output and make sure there are no issues with OAM trying to map the User Principal Name to the correct user in the Domain that user is a member of.

  1. Login as the owner of the OAM11g install.
  2. Go to the <FMW_HOME>/user_projects/domains/<DOMAIN_HOME>/bin directory.
  3. Make a backup of the setDomainEnv.sh file.
  4. Using a text editor open the setDomainEnv.sh file.
  5. Search for the first line with EXTRA_JAVA_PROPERTIES
  6. Add the following two lines (Note the first line should not wrap in the file).
    EXTRA_JAVA_PROPERTIES="-Dsun.security.krb5.debug=true -Dsun.security.spnego.debug=true ${EXTRA_JAVA_PROPERTIES}"
    export EXTRA_JAVA_PROPERTIES
  7. Save the file and restart both the Admin Server and OAM Server.
  8. You should now be able to tail the OAM Server logs and see Kerberos debug output.  The following is an example.
Entered Krb5Context.acceptSecContext with state=STATE_NEW
>>> EType: sun.security.krb5.internal.crypto.ArcFourHmacEType
Using builtin default etypes for permitted_enctypes
default etypes for permitted_enctypes: 3 1 23 16 17.
>>> EType: sun.security.krb5.internal.crypto.ArcFourHmacEType
object 0: 1359604847000/156
object 1: 1359604776000/152
object 2: 1359604739000/151
object 0: 1359604847000/156
object 1: 1359604776000/152
object 2: 1359604739000/151
replay cache found.
>>> KrbApReq: authenticate succeed.
Krb5Context setting peerSeqNumber to: 227772863
>>> EType: sun.security.krb5.internal.crypto.ArcFourHmacEType
Krb5Context setting mySeqNumber to: 349851240
SPNEGO Negotiated Mechanism = 1.2.840.113554.1.2.2 Kerberos V5
SpNegoContext.acceptSecContext: mechanism wanted = 1.2.840.113554.1.2.2
SpNegoContext.acceptSecContext: negotiated result = ACCEPT_COMPLETE
SpNegoContext.acceptSecContext: sending token of type = SPNEGO NegTokenTarg
SpNegoToken NegTokenTarg: sending additional token for MS Interop
SpNegoContext.acceptSecContext: sending token = a1 81 eb 30 81 e8 a0 03 0a 01 00 a1 0b 06 09 2a 86 48 86 f7 12 01 02 02 a2 69 04 67 60 65 06 09 2a 86 48 86 f7 12 01 02 02 02 00 6f 56 30 54 a0 03 02 01 05 a1 03 02 01 0f a2 48 30 46 a0 03 02 01 17 a2 3f 04 3d e9 4b 77 88 6c ac f7 6a c6 8e 52 e0 16 66 51 fa 7b 59 da 15 ff e0 a7 ce b7 39 0f 57 3b 80 31 15 fa ed 92 b2 0c 03 2c dd 3a 54 42 52 40 ba b2 bd df b7 f5 90 af 35 aa 6b 1e ac b9 4d 04 a3 69 04 67 60 65 06 09 2a 86 48 86 f7 12 01 02 02 02 00 6f 56 30 54 a0 03 02 01 05 a1 03 02 01 0f a2 48 30 46 a0 03 02 01 17

Turning on OAM Server debug mode to TRACE

It is very helpful to turn the OAM server debug logs to TRACE in order to get more verbose information.  This could also be done in Enterprise Manager, but if you would rather do this from a command line, these are the steps to follow.  Some things in this mode which will provide real help are showing things like the Principal that logged in, the User Domain which shows the namespace to search, the authentication scheme used and some details about it, and other things.

  1. Make sure the Weblogic Admin Server is running.
  2. Login as the administrator that starts and stops the OAM Managed server.
  3. Change to the directory <FMW_HOME>/common/bin
  4. Run the command.
    ./wlst.sh
  5. Connect to the Admin Server by running the following command.
    connect('weblogic','<weblogic_password>','t3://localhost:7001');
  6. Run this command to list the logging modules
    listLoggers(pattern='oracle.oam.*',target='oam_server1');
  7. Run the following command to set the OAM server in trace mode.
    setLogLevel(logger='oracle.oam',level='TRACE:32',persist='0',target='oam_server1');
  8. To revert back to normal run the following command.
    setLogLevel(logger=’oracle.oam’,level=’NOTIFICATION:1’, persist=’1’, target=’oam_server1’);

Verifying LDAP searches in OVD

It is helpful to tail the diagnostic logs in OVD to determine if the correct search is being made.  This should be displayed by default without needing to make any changes.  Simply do the following.
  1. Login as the administrator that starts and stops OVD.
  2. Change to the directory <FMW_HOME>/ovd1/diagnostics/logs/OVD/ovd1
  3. Either use a text editor or tail to view the diagnostic.log log file.
You should be able to see an example search as shown below.
[2013-02-06T15:38:59.192-06:00] [octetstring] [NOTIFICATION] [] [com.octetstring.vde.chain.plugins.DumpTransactions.DumpTransactions] [tid: 27] [ecid: 4aaf7073ad441920:-49d5bd7a:13cb14e92f1:-8000-0000000000000232,0:2] !SEARCH Operation: (Transaction#Adapter_Forest4.Dump Before.30)[[
BindDN: cn=orcladmin
Base:   dc=FOREST4,dc=MELANDER,dc=US
Scope:  2
Filter: (&(uid=tim.melander)(objectclass=inetorgperson))
TypesOnly:      FALSE
Attrs:  [uid, mail, cn, description, orclguid, objectclass, displayname, uid]!
]]

Testing the Solution


Testing the solution is pretty straightforward.
  1. Login to a desktop that is a member of one of the Active Directory domains.
  2. Open an Internet browser and browse to an OAM protected application.
  3. If working properly, you should not be challenged to login; it should immediately go in.

Summary


I hope the steps I provided are useful to anyone wanting to implement WNA.  It is without a doubt a complicated integration, but my goal was to help make this a little less painful.  Next in the final "Part 3: OAM11g WNA Identity Store Considerations and Configurations", I will cover OVD and special configurations that will make sure the domain name space will work with the Kerberos Authentication Module you created.  

14 comments:

  1. Hi Tim,
    At the great article with detailed instructions. I have been trying to the exact same thing since last week and I haven't got it working yet. I, from many resources including your previous posts, have managed to configure and setup debug and test it.
    My set up is using 11g R2 IAM and I setup OVD to integrate with MSAD and OID using user adapters . I am getting a SP NEGO token, but when OAM tries to extract user and authenticate, it throws this exception.

    "[2013-02-14T17:52:59.526-06:00] [WLS_OAM1] [TRACE] [] [oracle.oam.engine.authn] [tid: [ACTIVE].ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: ] [ecid: 004pRVox0u68Xr0pnw4EyZ0000Rn000004,0:2] [SRC_CLASS: oracle.security.am.engine.authn.internal.executor.KerberosModuleExecutor] [APP: oam_server#11.1.2.0.0] [SRC_METHOD: execute] [URI: /oam/CredCollectServlet/WNA] Username null
    [2013-02-14T17:52:59.526-06:00] [WLS_OAM1] [TRACE] [] [oracle.oam.engine.authn] [tid: [ACTIVE].ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: ] [ecid: 004pRVox0u68Xr0pnw4EyZ0000Rn000004,0:2] [SRC_CLASS: oracle.security.am.engine.authn.internal.executor.Utils] [APP: oam_server#11.1.2.0.0] [SRC_METHOD: getVMVendor] [URI: /oam/CredCollectServlet/WNA] VM vendor is Oracle: java.vendor=Oracle Corporation
    [2013-02-14T17:52:59.527-06:00] [WLS_OAM1] [TRACE] [] [oracle.oam.engine.authn] [tid: [ACTIVE].ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: ] [ecid: 004pRVox0u68Xr0pnw4EyZ0000Rn000004,0:2] [SRC_CLASS: oracle.security.am.engine.authn.internal.executor.SpnegoAuthenticator] [APP: oam_server#11.1.2.0.0] [SRC_METHOD: authenticate] [URI: /oam/CredCollectServlet/WNA] keytab file /u01/app/oracle/admin/idm_domain/mserver/idm_domain/oam.keytab
    [2013-02-14T17:52:59.527-06:00] [WLS_OAM1] [TRACE] [] [oracle.oam.engine.authn] [tid: [ACTIVE].ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: ] [ecid: 004pRVox0u68Xr0pnw4EyZ0000Rn000004,0:2] [SRC_CLASS: oracle.security.am.engine.authn.internal.executor.Utils] [APP: oam_server#11.1.2.0.0] [SRC_METHOD: getVMVendor] [URI: /oam/CredCollectServlet/WNA] VM vendor is Oracle: java.vendor=Oracle Corporation
    [2013-02-14T17:52:59.530-06:00] [WLS_OAM1] [TRACE] [] [oracle.oam.engine.authn] [tid: [ACTIVE].ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: ] [ecid: 004pRVox0u68Xr0pnw4EyZ0000Rn000004,0:2] [SRC_CLASS: oracle.security.am.engine.authn.internal.executor.SpnegoAuthenticator] [APP: oam_server#11.1.2.0.0] [SRC_METHOD: authenticate] [URI: /oam/CredCollectServlet/WNA] Authentication Failed.[[
    javax.security.auth.login.LoginException: Unable to obtain password from user
    "
    Any pointers as why the username is coming as NULL?. I would expect something...Appreciate your help.
    I have tried both custom Kerberos scheme ( using StepKI ...etc) and aslo simple kerberos plugin setup.

    thanks for your suggestion.
    regards
    -raja muthu

    ReplyDelete
    Replies
    1. Thanks Raja. Just a note, the custom Kerberos authentication module in the stepUIF UserIdentificationPlugin uses a parameter KEY_LDAP_FILTER, which in this article is set to (uid={KEY_USERNAME}). In Part 3 I will mention more about how the adapters I used in OVD took advantage of the OAM/AD Mapper adapter. It really does not matter which adapter you use, but it is important to know what attribute to use that maps to the sameAccountName. In my case the adapter mapped inetOrgPerson type object class attributes to Active Directory attributes. For example uid mapped to samAccountName. If your adapter does not do that, then just use the correct attribute that maps to the samAccountName. It is also possible to use userPrincipalName, but how I set things up I simply mapped to the samAccountName. Like I said I will cover more in Part 3 on the OVD with this WNA solution. This is most likely why the username is null because the value used in the plugin may not map to the correct attribute OVD is expecting. See if that helps at all.

      Delete
    2. Raja,

      I just ran into this error yesterday, this may be the issue you are encountering.

      Does your setup match the following? Oracle Access Manager (OAM) 11g has been configured for Windows Native Authentication (WNA) with an Active Directory 2008 R2 domain. Windows 7 or Windows 2008 clients are used to access the OAM WNA-protected site.

      If so the reason you are getting that error, is because OAM is unable to decrypt the Kerberos ticket sent by the Windows client for OAM WNA login because it is encrypted using aes256-cts-hmac-sha1-96 type.

      If so you will need to do the following:

      1. Download and install the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files for the version of the JDK that is configured in the OAM WebLogic Server domain.

      2. Generate a new keytab file on the Active Directory 2008 server containing keys for all AD-supported encryption types, using flag /crypto -all.

      For example:

      ktpass -princ HTTP/sso.oracle.com@ORACLE.COM -mapuser oamuser1 -pass -ptype KRB5_NT_PRINCIPAL -crypto ALL -out keytab.service

      3. Restart the OAM managed server and test OAM WNA login.

      If you have access to Oracle support, check Doc ID 1509275.1 and Bug 15863291 for more details.

      Delete
  2. Hi Tim and Michael, thanks. I have my WNA working now. Tim, I do have the AD User Adapter setup and have mapped uid=samAccountName. Since, I have a single AD domain I used the default Kerberos plugin to test and it works. I will also configure custom kerberos plug-in with steps and test it.

    Michael, actually my setup is what you describe only that I was using Windows XP client to test, which users arcfour-hmac encryption. But your pointer really helped. When I asked the AD admin he mentioned that he ran the command on the 2008 AD server without the -ptye and the -crpto ALL options before. Once we re-run the command in Windows AD 2008 R2 as you have mentioned and copied the keytab file and bounced the OAM server, it started working.

    I will test this with Windows 7 client and install JCE, if oacle has already not fixed this in our 11g R2. thanks a bunch.

    On the same note, I have 2 adapters configured in my OVD. One User Adapter for OID and one AD User Adapter ( with the use plugins configured as in EDG guide for OAM mapping all attributes needed.) My situation is the DIT is same for both backend store AD and OID which is dc=wft,dc=root,dc=loc

    The adapter has 50 50 priority setup, so it defaults to the first one in the config file and use the OID and the WNA fails. if I set the AD adapter priority to low ( say 40) means use AD adapter, WNA works.
    So my question, is how do I ensure that the active directory users are going to AD adapter and OID users like weblogic_idm,oamadmin etc.. go to OID user adapter when the search base (DIT) are the same. Any pointers are helpful.

    thanks and regards
    -raja

    ReplyDelete
    Replies
    1. Raja,

      Please review Part 3. If you truly only have a single AD domain, then create an Identity Store in OAM to point to your AD domain. OVD is only needed in cases where you are trying to aggregate Active DIrectory forests, unless users are provisioned into a single Enterprise Directory Server.

      Once you have the new Identity Store, use it for the custom Kerberos authentication module. Then instead of using "uid={KEY_USERNAME})" for the KEY_LDAP_FILTER, use "samaccountname={KEY_USERNAME})", and then leave the KEY_SEARCH_BASE_URL field empty. If you leave KEY_SEARCH_BASE_URL empty the authentication module will just pick up the search base used in the Identity Store.

      I hope this makes sense. I believe it should solve your problem. However, if you must go through OVD, then I would think about designing the DIT in a way so that there is a rootDSE top namespace, and AD and OID would have subbranches. I realized this may impact other applications, but may solve other problems in the long run.

      -Tim Melander

      Delete
  3. Hey Tim - Isn't there a third way where you create multiple steps (stepUI, stepUI2, stepUI3 etc..) for each of the domains and then create multiple AD identity stores in OAM console so it loops from one to the next. Basically what I am saying is, I think there is a third way (using OVD, using single enterprise store - are two options you discussed). The third option is where you create multiple AD identity stores in OAM console and then create multiple of these three steps - stepUI, stepUIF, stepUA (stepKTA and stepKTI will appear only once). Isn't that a viable third option ?

    ReplyDelete
    Replies
    1. Hi Vishal,

      I have actually tried what you are point out where I created multiple stepUIF1, StepUIF2, stepUIF(n), but I did not have any luck making it consistently work if at all. I also mention this in the blog and point out I could not get that to work properly. I am not sure why because it seems logical to your point. However that said, if you have actually got this to work please post what you have done and share how it worked out.

      -Tim Melander

      Delete
    2. Sorry for the late response. I was building my env and wanted to test if everything worked successfully or not.
      I can attest that the third option I discussed is a viable option. It works !
      It is quite complicated to explain in this post but I can explain it to you on the phone.

      Delete
  4. Tim,
    I am answering my own question here.

    My question earlier was
    "On the same note, I have 2 adapters configured in my OVD. One User Adapter for OID and one AD User Adapter ( with the use plugins configured as in EDG guide for OAM mapping all attributes needed.) My situation is the DIT is same for both backend store AD and OID which is dc=wft,dc=root,dc=loc

    The adapter has 50 50 priority setup, so it defaults to the first one in the config file and use the OID and the WNA fails. if I set the AD adapter priority to low ( say 40) means use AD adapter, WNA works.
    So my question, is how do I ensure that the active directory users are going to AD adapter and OID users like weblogic_idm,oamadmin etc.. go to OID user adapter when the search base (DIT) are the same. Any pointers are helpful."

    Answer: I have got this working now. Thanks to the Coordinator plugin that is going to be part of the OVD in the next release 11gR1 PS6 ( 11.1.1.7) no ETA on the release. However, the coordinator.jar file is provided in the Oracle Support site. Please see the article ID : 1483806.1
    I highly recommend this approach, as it allows us to have a much cleaner design via OVD presenting one search base for clients even when we have mulitple Active Directory Domains/Forest with different name space. Please read the article more information.
    It feels good, when you design a solution and the product supports it and exciting to work with the solution that is less than a weeks old. :)

    regards
    -raja muthu

    ReplyDelete
  5. Hi Tim,

    We are also working on WNA for multi-forest environment with AD 2008.
    We followed your steps, for us crypto all didn't work, but without specifying and crypto details the scenario worked for users of default realm.
    That means I have two ADs, forest1 and forest2.
    forest1 is given as default realm in krb5.conf.
    krb5.conf is in same format as suggested by you and also we have merged keytabs for two forests.

    Users from forest1 are able to get SSO but for users of forest2 SSO do not work

    We get following error for forest2 -

    in oam_server1-diagnostic.log:
    Authentication Failed.[[
    javax.security.auth.login.LoginException
    at oracle.security.am.plugin.authn.SPNEGOLoginModule.login(SPNEGOLoginModule.java:148)


    in tmp/oam_14032013.out
    >>> KrbAsRep cons in KrbAsReq.getReply HTTP/tlntgcp12.gitest.telenor.com
    Found key for HTTP/tlntgcp12.gitest.telenor.com@GCADUAT.GITEST.TELENOR.COM(23)
    Entered SpNegoContext.acceptSecContext with state=STATE_NEW
    SpNegoContext.acceptSecContext: receiving token = a0 82 06 ae 30 82 06 aa a0 30 30 2e 06 09 2a 86 48 82 f7 12 01 02 02 06 09 2a 86 48 86 f7 12 01 02 .............
    SpNegoToken NegTokenInit: reading Mechanism Oid = 1.2.840.48018.1.2.2
    SpNegoToken NegTokenInit: reading Mechanism Oid = 1.2.840.113554.1.2.2
    SpNegoToken NegTokenInit: reading Mechanism Oid = 1.3.6.1.4.1.311.2.2.30
    SpNegoToken NegTokenInit: reading Mechanism Oid = 1.3.6.1.4.1.311.2.2.10
    SpNegoToken NegTokenInit: reading Mech Token
    SpNegoToken NegTokenInit : no MIC token included
    SpNegoContext.acceptSecContext: received token of type = SPNEGO NegTokenInit
    SpNegoContext: negotiated mechanism = 1.2.840.113554.1.2.2
    Entered Krb5Context.acceptSecContext with state=STATE_NEW


    Regards,
    Deepika

    ReplyDelete
    Replies
    1. Hi Deepika,

      Are both AD domains 2008? Please also compare the crypto type each domain is using and see if there are any differences. Something else you could try is to remark out the default realm. In my lab I have the [libdefaults] section of the /etc/krb5.conf as follows. Notice I remarked out the default_realm. The idea is to force OAM to iterate through the realms listed below. Please let me know if any of this helps.

      [libdefaults]
      # default_realm = FOREST1.MELANDER.US
      dns_lookup_realm = false
      dns_lookup_kdc = false
      ticket_lifetime = 600
      forwardable = yes
      default_tkt_enctypes = RC4-HMAC
      default_tgs_enctypes = RC4-HMAC

      - Tim Melander

      Delete
  6. Hi Tim,
    Can I achieve WNA in this scenario where OAM is my access management solution? The configured Identity store has all users, where UID is set to Emp Number (11111). In one AD, the SamAccountName is set to Emp Number (11111) whereas in the other AD domain, the SamAccountName is set to combination of user first name, last name (jdoe). But in both the AD, the employee number is present in the attribute Emp No. In my current setup I am able to establish WNA with the AD, where SamAccountName is same as my Identity Store UID, but how can I establish the WNA with the second AD, where the SamAccountName in the AD is not matching the search parameters defined in my WNA authentication scheme.
    Thanks&Regards
    Debi
    demohanty@deloitte.com

    ReplyDelete
  7. Hi Debi,

    So in say AD1, a samaccountname value could be 11111@ad1 where in AD2 the a sameaccountname value could be jdoe@ad2. If OVD is used, and separate Adapters are configured for each Active Directory, then as long as the DIT is design in such that there is a top namespace of say dc=example,dc=com and you would have dc=ad1 and dc=ad2, OAM should be able to search across the DIT.

    For example assuming that a use from AD1 would login and have a UPN of 11111@ad1, OAM will find the uid=11111 in AD1, where user in AD2 would similarly be found as jdoe. Am I missing something? The critical piece here is that OVD is used. There may be another option in the Kerberos authentication module using steps.

    Tim

    ReplyDelete

Note: Only a member of this blog may post a comment.