Oracle Unified Directory 11gR2PS3 Very Large Static Groups

Introduction

This post is about OUD and extremely large static groups where membership numbers exceed hundreds of thousands or even millions; yes I said millions.  I have been using Directory Services for over 15 years and the response I typically have for a customer that wants to use very large static groups is don't do it.  Then I steer them into dynamic groups or even suggest leveraging attributes from user entries.  In fact OUD has a great feature unique to itself called Virtual Static Groups that is kind of a hybrid between dynamic and static group, which has proved successful for past customers wanting very large groups yet get great performance.  That said, in this post I am going to break all the rules and say you can have static groups with even millions of members because of the new static group performance improvements that has come with OUD11gR2 PS3 (11.1.2.3.0).

Working with Oracle Unified Directory 11gR2 Transformation Framework

Introduction

If you have been using Oracle’s Identity Management software for at least the last few years you will probably be familiar or at least heard of OVD (Oracle Virtual Directory), which was originally acquired back in 2005 from a company called OctetString. OVD provides a vast number of great virtual features used to aggregate multiple backend data stores and present LDAP consumers a single unified Directory Server.  Beginning with OUD version 11.1.2.1.0, there have been a number of virtualization features added similar to what is provided in OVD.  This trend has continued through OUD 11.1.2.3.0 where features such as joining multiple backends was added.

The OUD Transformation Framework can do various things as presented in the latest documentation “Understanding the Transformation Framework”, but in order to help illustrate how this feature can really add value I recently worked with a customer where leveraging a Transformation Rule helped solved a problem. Because the existing documentation is either confusing or lacking, I decided to write this article to help learn more about the Transformation Framework and how to make it work. An important note I want to alert you is at the time this article was published in order to use the OUD virtualization features you are required to have what is called a “Oracle Directory Service Plus” license http://www.oracle.com/us/products/middleware/identity-management/oracle-directory-services/overview/index.html. If you have any questions about that please refer to your local Oracle Sales Representative.

Improve Oracle Unified Directory 11gR2 Search Performance with Index Entry Limit

Introduction

I am always looking for great tips that give big values; this one is no exception. This article is to help you understand how to tweak the index called “Index Entry Limit” to reap some dramatic ldapsearch performance improvements. I explain what this index is about, some of my own test results, how to determine the correct value, and finally how to make the index change to your OUD instance. This will be a tip you will definitely want to add to your OUD Ninja black bag.

Part 2: Advanced Apache JMeter Stress Testing OAM and LDAP

In “Part 1: How To Load Test OAM11g using Apache JMeter” I talked about an example plan that could be used to load test OAM11g, which included some common configuration elements, some samplers for login, authorization, logout, and some listeners that provided result analysis.   In Part 2, I wanted to expand on an option to make JMeter send random logins and I will explain why, and then cover how to leverage JMeter to load test an LDAP server like OUD, OID, ODSE, or OVD.

OAM LDAP connections through firewalls

In a previous post, we discussed some of the complications that can occur when a firewall is placed between WebGates and OAM Servers in a typical deployment. This post follows on from that discussion, to explore an analogous topic- firewalls between the OAM Server and the LDAP Identity Store. This post is part of a larger series on Oracle Access Manager 11g called Oracle Access Manager Academy. An index to the entire series with links to each of the separate posts is available.

Where has your LDAP connection pool gone?

You have deployed Oracle BPM and decided to run some load tests against it. You're concerned, among other things, about the behavior of your backend LDAP server under peak times, whether it's going to be able to handle the load or not. You check the security providers settings in Weblogic Server and see you have an LDAP Authenticator (or some specialization, like OVD Authenticator, for instance) with an ldap pool size set to 50 connections. But your test reveals that many more than 50 connections gets open in your LDAP server. You change that number, redo your tests, but the behavior doesn't change at all. Then you get worried about the LDAP connection handling performed by the Weblogic authenticator.

This post is to tell you, as far this scenario goes, there's nothing wrong with it. In fact, for this specific scenario, you're looking at the wrong place for your connection pool settings.

Oracle BPM, as well as SOA, WebCenter, UCM (Universal Content Management), IPM (Imaging Processing Management) and others use something called User/Role API to communicate with LDAP stores. Whenever these components need to assert a user identity or query user/group/attribute information, they go through User/Role API. It is an abstraction built on top of JNDI to ease the access to Directory servers. You can connect to any LDAP server supported by WLS authenticators and use the very same set of APIs to retrieve and write data without having to resort on JNDI, regardless of the LDAP server you connect to.

The importance of "orclguid" in Oracle Virtual Directory

This post will discuss the steps to configure the orclguid within Oracle Virtual Directory (OVD).  It is especially important when integrating OVD with Oracle Access Manager (OAM) and Weblogic Server (WLS).  I see many customers omitting this configuration which leads to errors in OAM.  


Background

All Lightweight Directory Access Protocol (LDAP) repositories contain a global unique identifier (guid) for every entry.  OVD is no different; it also has a guid object called the orclguid.  When configuring OVD with LDAP repositories it is important to map the LDAP's guid object with Oracle's guid object.  This however, is not configured by default.  In order to do this you will need to configure the VirtualAttribute plug-in for your adapter.

For more information please take a look at the OVD plug-in section in the documentation.

Forgetting this step may cause errors with respect to a) Authentication Failures and b) Identity propagation. For example, in Active Directory (AD) the guid object is called objectGuid, if this is not mapped to an orclGuid you will have issues when trying to propagate the users identity.


High-Level Steps

Once you have determined what the guid object is for your back-end LDAP repository, you will need to use the VirtualAttribute plug-in to map the two attributes. In our example above, the mapping will take the form: 

orclguid=%objectguid%

where objectguid is the guid object for AD.

If you have multiple LDAP back-ends, then you will need to configure the VirtualAttribute plug-in for each one.  Below are some screen shots that shows where you need to configure the plug-in.

Select the plug-in tab within the Adapter configuration...

Create a new Virtual Attribute plug-in with the parameters shown...

The OVD Provider in WLS, the 'orclguid' is already set as the GUID Attribute.   If not, make sure that the value of 'orclguid' is listed as below...

That's it!  The 'orclguid' will now be passed to OAM/WLS for each object with OVD.

Creating a Custom OVD Plugin

1. Introduction


In a recent engagement, I worked with a customer that had a business requirement where they needed to create and expose to their application two computed LDAP attributes, based on the value of an existing attribute.

For instance, let’s say the original attribute is "myCorpID" and its value could be something like "23451588-IT Specialist".

The requirement is to have two new defined attributes: "jobCode" and "jobTitle", that would have values "jobCode=23451588" and "jobTitle=IT Specialist".

To achieve this we would use OVD, which was already in place to virtualize different LDAP directoriess and databases.

OVD does not come with functionality out of the box to implement this requirement but does allow you to write custom plugins, where you can extend the capabilities of OVD to include requirements such as the one my customer had.

This post shows how to develop a custom OVD plug-in.

This plug-in reads a LDAP Object’s attribute value, splits the value in two, using a delimiter, and creates two new attributes with values assigned from the split operation.

Though this is a very simple requirement, and in fact my customer requirements were more complex than that, it can be used as a starting point to develop different use-case needs.

This plug-in is designed to work with a LDAP Adapter, and will run it’s logic for each LDAP "get" operation, before handing the result back to OVD.

The plug-in is flexible enough to work with any LDAP "objectClass" that has an attribute of String data type with a pattern delimiter that can be split in two.

Achieve Faster WebLogic Authentications with Faster Group Membership Lookups

In my last post  I wrote about the complicated and timely process of determining all of a user’s group memberships when an LDAP namespace includes nested and dynamic group memberships. I wrote about how you can simplify and speed up getting a user’s group memberships through the use of a dynamic “member of” attribute and specifically the orclMemberOf attribute in OID.

Today I’d like to extend this discussion to WebLogic server authentications.

Fast Group Membership Lookups in OID with the orclMemberOf Attribute

If you utilize nested and dynamic groups (and especially nested dynamic groups), then it can take a lot of effort and time to calculate all of a user’s group memberships in an LDAP directory.

First you have to search for the user and find the user’s DN. Then you have to search all your groups to figure out which groups your user is directly a member of. Then for each of those groups you have to search all your groups again to see which of those groups your user is a member of.

You have continue to search your groups with the results of each subsequent search until you reach the maximum desired level of nested memberships that you want to pursue or all the searches come back empty. All the while you have to keep yourself out of infinite loops created by repeating memberships such as when two groups are members of each other.

Many LDAP directories simplify things through a virtual “member of” attribute which is a virtual multi valued attribute containing all of the groups a user is a member of through both direct and indirect means.

It may have escaped your notice, but OID joined the party fairly recently (in 11.1.1.4 I believe) and now supports such an attribute. The attribute’s name is orclMemberOf. You can read all about the attribute here; but suffice it to say it is a dynamic multi valued attribute containing the groups to which a member belongs.

The membership includes both direct membership and indirect membership from nested groups. It also includes membership from dynamic groups and dynamic nested groups based on labeleduri.

The attribute value is computed during a search and is not stored. This means you will not see orclMemberOf populated in an LDAP data browser including ODSM. Further, the value is not returned by default in searches. You have to explicitly request it. Lastly, orclMemberOf cannot be used in a search filter.

One nice little additional feature thrown in is that the aliases of memberof and ismemberof are supported for compatibility with code written for compatibility with Active Directory and Oracle Directory Server Enterprise Edition (DSEE) / SunOne / IPlanet.

Below is a sample search with results for a specific user where I request and receive the value(s) of orclMemberOf.  You will also notice that nested memberships are returned multiple times, once for each group that the user belongs to that is a member of another given group.  So, watch out for that.

In a future post, I'll discuss how you can use the orclMemberOf attribute to greatly speed up authentication into WebLogic and Fusion Middleware Products such as SOA Suite and WebCenter which utilize WebLogic's security framework.

[oracle@oam1 bin]$ ./ldapsearch -h oam1.example.com -p 3060 -D cn=orcladmin -w Oracle1_g -b "cn=Users,dc=example,dc=com" -L -s sub -v "uid=tim.doyle" memberOf


ldap_open( oam1.example.com, 3060 )

filter pattern: uid=tim.doyle

returning: memberOf

filter is: (uid=tim.doyle)

dn: uid=tim.doyle,cn=users,dc=example,dc=com

memberof: cn=administrators,cn=groups,dc=example,dc=com

memberof: cn=groupofgroups,cn=groups,dc=example,dc=com

memberof: cn=nyusers,cn=groups,dc=example,dc=com

memberof: cn=groupofgroups,cn=groups,dc=example,dc=com

memberof: cn=nestgrp1,cn=groups,dc=example,dc=com

memberof: cn=groupofgroups,cn=groups,dc=example,dc=com

memberof: cn=oaamcsrmanagergroup,cn=groups,dc=example,dc=com

memberof: cn=groupofgroups,cn=groups,dc=example,dc=com

memberof: cn=oaamenvadmingroup,cn=groups,dc=example,dc=com

memberof: cn=groupofgroups,cn=groups,dc=example,dc=com

memberof: cn=oaamruleadministratorgroup,cn=groups,dc=example,dc=com

memberof: cn=groupofgroups,cn=groups,dc=example,dc=com

memberof: cn=product support group,cn=groups,dc=example,dc=com

memberof: cn=groupofgroups,cn=groups,dc=example,dc=com

1 matches

LibOVD: when and how

LibOVD, introduced in FMW 11.1.1.4, is a java library providing virtualization capabilities over LDAP authentication providers in Oracle Fusion Middleware. It is delivered as part of OPSS (Oracle Platform Security Services), who is available as part of the portability layer (also known as JRF – Java Required Files). In other words, if you are a JDeveloper, WebCenter, SOA or IAM customer, you already have libOVD.

LibOVD provides limited virtualization capabilities when compared to its big brother OVD (Oracle Virtual Directory), who is a full-blown server implementing the LDAP protocol with far more advanced virtualization features, including OOTB support for LDAP and database backend servers, advanced configuration for adapters, out-of-box plug-ins as well as a plug-in programming model allowing for almost limitless possibilities in transforming data and connecting to external data sources.

1. When

LibOVD is primarily designed to work as an embedded component for FMW components who need to look up users and groups across distinct identity providers. If you had a chance to look at this post, you already know the User/Role API can take into account only one authentication provider.

Tuning WebLogic LDAP Authentication Providers

I’ve been involved in a fair amount of activity over the last month involving customers who want or need to tune their WebLogic LDAP authentication providers for a production environment.

Too often I have seen customers simply lacking in awareness to the fact that the authentication providers can and should be tuned from their default settings. The fact is that the default values in the LDAP authentication providers are better sized to development environments (and in some cases the development environments of 5 years ago) then they are to today’s production environments. So, the first step is awareness that the authentication providers include setting related to cache performance, connection management, and handling of group lookups that can and should be tuned in order to maximize the performance of your applications.

So, in this post I’d like to go through the authentication provider settings that affect performance, discuss what each setting does, and discuss what some guidelines and how these differ from the defaults.

First, let’s briefly discuss how to find the settings you’ll want to tune. Login to the WLS admin console, on the left hand side under domain structure click security realms and then “myrealm”. From there, click on the providers tab and select the LDAP authentication provider that you want to tune. Once you are in the authentication provider configuration screen you’ll want to look in the “Provider Specific” and “Performance” tabs to modify the settings we are about to discuss. We will also discuss one setting that is located under the “Performance” tab of “myrealm” (or whatever you have named your active security realm) itself.

If all goes well you should see a screen that looks something like this after clicking the provider specific configuration tab:

In discussing the tuning of LDAP authentication providers, I like to divide the settings into 3 categories: LDAP connection settings, cache settings, and group lookup settings.  If you’d like to follow along with what the documentation says you can do so here: http://download.oracle.com/docs/cd/E21764_01/web.1111/e13707/atn.htm#i1216261

Connection related settings:
Connection Timeout Limit – The maximum time in seconds to wait for the connection to the LDAP server to be established. The default is set to 0 which means that there is no maximum time limit. Note that this setting only comes into play when the authenticator is trying to open up a new connection in its pool of connections to the directory.

Connection Retry Limit – The number of times the server will attempt to connect to the LDAP server if the initial attempt fails. The default is 1. Again this setting applies only to situations where the authenticator is trying to open up new connections.

Now you may ask what happens if the connection timeout limit is reached and all retry attempts fail. The answer is that the authenticator will simply give up on the current request that it is trying to open a connection to handle and return a failure. Subsequent requests will be serviced from the available pool of connections until a new one must be opened again at which time the process will repeat itself.

I could be wrong but I see no good reason why one would want to wait forever. What you want to avoid is a cycle of death situation where degradation in LDAP performance is handled poorly and leads to things backing up more than they have to in the authentication provider. The specific values that you should go with for these settings are fairly environment specific in that they depend on your directory and network infrastructure but I think that 120 seconds for the Connection Timeout Limit and 5 for the Connection Retry Limit are good starting points.

Cache related settings:
Cache TTL and Cache Size

There are two cache settings on the “provider specific” tab of most LDAP authentication providers. These are Cache TTL and Cache Size. These setting refer to a “user related” cache in the authentication providers that cache the DN lookup that translates login names to full LDAP distinguished names and possibly caches some common attribute values following the lookup. I must stress that the authentication providers do not cache username and passwords. Real username/password authentication always results in a call to the directory. With that out of the way:

Cache TTL – is the time-to-live of entries in the cache in seconds. The default is 60 which seems low to me. I would consider upping this to 5 minutes and going from there.

Cache Size – is the size of the cache in kilobytes. The default is an absurdly low 32 KB. The per-entry size of this cache is low but I don’t think upping this to 2-4 MB would hurt.

The exception to the above recommendation would be a situation where you really don’t expect an individual user from hitting the authentication provider twice in a fixed period of time. In this case I would still up the Cache size some but might leave the TTL along at 60 seconds.

Principal Validator Cache

The Principal Validator Cache is actually a setting associated with the entire realm rather than the authentication provider and is configured in the “performance” tab of the realm itself. There are two settings associated with the cache: Enable WebLogic Principal Validator Cache and Max Weblogic Principals in Cache. It is enabled by default with the max number of principals defaulting to 500.

This cache is mentioned in the documentation in vague terms as something that can improve performance and indeed it is a fairly mysterious construct. What this cache does is cache signed Abstract Principals which are used in RMI calls when a Principal Validation Provider is being used.

The long and short of it is that this cache won’t have too much affect for most people and even in situations where it will be heavily hit, it is common for the validations to be associated with a limited number of service accounts. So, for the post part you can just leave these settings as is. However, don’t be afraid to bump up the number of cached principals, the default setting is very low considering the hardware you are likely to be running WLS on in production.

Group lookup related settings:

One of the most important, if not the most important piece of tuning you can do to a WLS LDAP authenticator is to change the Group Membership Searching from unlimited to limited and set the Max Membership Search Level to an appropriate value. Not only will this improve your performance, it will prevent you from encountering a loop that prevents users from logging in when two groups are members of each other. I blogged extensively about these two settings in a previous post entitled Weblogic, LDAP Authenticators, and Groups.
Rounding out the group lookup related settings are a group of settings that can be found in the performance tab of LDAP authentication providers.

These settings all deal with the group membership hierarchy cache. This cache stores the results of recursive membership group lookups or put another way it stores what groups are members of other groups.

The settings for this cache include a check box to enable the cache which is called Enable Group Membership Lookup Hierarchy Caching, a setting that controls the size of the cache called Max Group Hierarchies in Cache, and a setting that controls the time-to-live (in seconds) of cache entries called Group Hierarchy Cache TTL.

I recommend that you enable this cache if you utilize nested groups. I recommend that you set the Max Group Hierarchies in Cache to a value larger than the total number of groups in your directory. Finally, I recommend that you set the Group Hierarchy Cache TTL to a safe appropriate number. The default is 60 seconds which will improve performance, catch changes to the hierarchy fairly quickly, but still result in a fair amount of recursive group lookups. If you up this value to 5 minutes (300 seconds) which should still be safe for most people who aren’t doing funky things with dynamic groups then you should be able to improve performance a little more with no downside.

Conclusion

Tuning of WLS LDAP Authenticators is an overlooked component of successful WLS production deployments. Taking just a little time to change the LDAP authenticator performance related configuration settings from default values to values which are appropriate for your production environment can result in a much faster and more stable system.

Extending the OID 11g schema via ldapmodify

I recently said the following to someone in an IMversation:

ldapsearch and ldapmodify are about 10 times better than a stupid GUI because you can script everything.
it's like the difference between knowing SQL and having to use TOAD or Access (*shudder*) to add rows to a db table

I think if I had a personal motto it would be something like "if I can't script it then I'm not interested." (well that or "oh look, shiny!")

I recently had to extend my LDAP schema of an OID 11g directory and I didn't have access to the ODSM GUI (for reasons that aren't important). So I had to extend the schema via ldapsearch/ldapmodify (i.e. the hard way) and thought I'd write down my steps to save you the trouble of figuring it out for yourself.

OAM 11g Connecting to an LDAP ID store over SSL (LDAPS)

Connecting to an LDAP ID store in OAM 11g over SSL (LDAPS) is a common scenario that many customers may need to implement. Unfortunately the documentation on this subject is scant and can be misleading. So as part of the OAM 11g Academy series, I'd like to discuss this commom scenario. To view the first post on the OAM 11g policy model, as well as the index to the entire OAM 11g Academy series, click here: http://fusionsecurity.blogspot.com/2011/02/oracle-access-manager-11g-academy.html.

The documentation to manage data sources can be found in Chapter 3. The section titled "Managing User Identity Store and OAM Administration Registration" describes how to register a new identity store. Specifically, Table 3-2 describes all the possible elements required to register. Looking at the 'LDAP URL' element we have the following:

The URL for the LDAP host, including the port number.

For example, the default embedded LDAP host might be:
ldap://localhost:7001

You can also specify ldaps://, which supports SSL_NO_AUTH.

That's it! Good luck :)

So what does it all mean and what do I do if the LDAPS connection fails?

SSL_NO_AUTH basically means a self signed certificate, no authentication required. 1-way and 2-way SSL modes are not supported at this time.

Once you setup the identity store using LDAPS you should always test the connection via the 'Test Conenction' button located at the top as shown here:

If there are any issues with the connection you will see an error like the one below:

You may also find an exception in the oam-diagnostic logs as follows:

####<Oct 29, 2010 1:50:52 PM CDT>

<Error> <oracle.oam.user.identity.provider> <host> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <user> <> <279b690d689ef829:-7e1652d1:12bcfcdf8f4:-8000-0000000000000c13> <1288378252365> <OAMSSA-20043> <Error validating LDAP URL and credentials : javax.naming.CommunicationException: <host>:636 [Root exception is javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target].>

This specific exception states that it cannot find the certificate in order to initiate the SSL handshake.

You may also find an error in the LDAP logs. These logs are from eDirectory for example:

12:47:42 41D15940 LDAP: New TLS connection 0xedd09c0 from IP:40557, monitor = 0x4b780940, index = 32
12:47:42 4B780940 LDAP: Monitor 0x4b780940 initiating TLS handshake on connection 0xedd09c0
12:47:42 4AF78940 LDAP: (IP:40557)(0x0000:0x00) DoTLSHandshake on connection 0xedd09c0
12:47:42 4AF78940 LDAP: (IP:40557)(0x0000:0x00) TLS accept failure 1 on connection 0xedd09c0, setting err = -5875. Error stack:
error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
12:47:42 4AF78940 LDAP: (IP:40557)(0x0000:0x00) TLS handshake failed on connection 0xedd09c0, err = -5875
12:47:42 4AF78940 LDAP: Server closing connection 0xedd09c0, socket error = -5875
12:47:42 4AF78940 LDAP: Connection 0xedd09c0 closed

Fortunately the solution is pretty simple and has two parts. First, OAM now uses the Java Key Store (JKS) format to store the certificates keys. OAM uses the JKS file 'cacerts' located at "$JAVA_HOME/lib/security/cacerts". This is where the certificate needs to be imported into. Secondly, make sure that the right certificate is imported via the keytool. For more information on how to use the keytool, you can go here. You will need to import the server certificate along with the CA Root cerficate into the keystore. Many customers make the mistake of importing only the CA Root certificate.

So to summarize, the two issues to be aware of are:

1) The difference from 10g, is that 11g now uses the default 'cacerts' location based on the $JAVA_HOME directory.

2) Make sure that both the CA Root certificarte and the server certificate (child) are imported into the keystore.

WebLogic, LDAP Authenticators, and Groups

The number one cause of mystery login problems that I see with customers using LDAP authenticators (be it OID, Sun, or AD) relates to a problem with the search done to determine what groups the user is a member of.

As part of the authentication process, LDAP authenticators do a search to determine what groups the user is a member of which in turn get used in determining the group memberships and roles for the JAAS subject and principals. If there is a problem with this search, then WebLogic will fail the entire authentication even if the user authentication check (username + password) against the directory was successful. Notice that I said problem with the search and not search failure. If the user simply doesn’t exist in any groups then the authentication will succeed. The issue is with the authenticator not even being able to successfully execute the search.

The group search failure can be cause by a couple different things and can be a very nasty situation to recognize and sort out.

Search Configuration Failures
The more straight forward cause of a group search failure is that the search itself. This can be cause by having a bad “Group Base DN” or “All Groups Filter” in the authenticator configuration. If for example you mistyped part of the base DN or the object class in the filter, then the search will fail to execute. What you’ll see in the logs is a message saying authentication has succeeded then one or more error messages about the group search, and then another message saying authentication has failed.

Problems with Nested Groups
The more insidious cause of group search failures is with nested groups. By default the authenticator will search not only what groups a user belongs to, but will go on to search what groups those groups containing the user belong to and then what groups those groups belong to and on and on…

If there are a ton of groups with lots of nesting, the authenticators can time out just in processing nested groups. However, the bigger or more common issue is that authenticators can get in infinite loops processing two groups that contain each other or certain dynamic groups.

The best way to prevent this is to limit the levels of group membership nesting that the authenticator will follow to build up the JAAS subject and principals. You do this by changing “Group Membership Searching” from unlimited to limited and changing “Max Group Membership Search Level” to the desired level of nesting you want to pursue. Leaving it at 0 will mean that the authenticator will not process any nested groups.

My recommendation is that as a best practice you should almost always switch to a limited search even if you want the authenticator to heavily process nested groups. Setting the level of nesting to something high like 5 will almost always give you all the memberships/roles you need but will limit your exposure to infinite loop situations caused by problems in the directory.