Oracle Unified Directory 11gR2PS3 Very Large Static Groups

Introduction

This post is about OUD and extremely large static groups where membership numbers exceed hundreds of thousands or even millions; yes I said millions.  I have been using Directory Services for over 15 years and the response I typically have for a customer that wants to use very large static groups is don't do it.  Then I steer them into dynamic groups or even suggest leveraging attributes from user entries.  In fact OUD has a great feature unique to itself called Virtual Static Groups that is kind of a hybrid between dynamic and static group, which has proved successful for past customers wanting very large groups yet get great performance.  That said, in this post I am going to break all the rules and say you can have static groups with even millions of members because of the new static group performance improvements that has come with OUD11gR2 PS3 (11.1.2.3.0).

OUD PS3 Static Group Improvements

Oracle has worked hard at fixing problems seen with large static groups in OUD11gPS2 or older.  As of version OUD 11gR2PS3 (11.1.2.3.0), improvements related to static groups include group cache improvements, a redesign of where members of static groups are stored, how the attributes for groups are referenced, improvements in entry cache specific to groups with more than 100,000 members, and improvements in importing large groups.  Some new tuning properties have also been added that help with larger static groups which include member-lookthrough-limit, return-attribute-value-limit, and import-big-entries-memory-percent.  All these changes have had a significant impact on performance improvements.

I personally have loaded an OUD instance with groups as large as 2 million members, and did LDAP searches, adds, modifies, and deletes at around 20 operations a second with staggering great results.  For example I have got searches with the slowest response at 87ms with an average of 1ms, and even modifications with a maximum time of 212ms and an average of 1ms.  Oracle themselves have tested OUD PS3 with groups of up to 50 million members!  This certainly breaks all the rules I have ever learned about large static groups in Directory Services, so this is quite exciting.

However, before you jump into loading up OUD with these massive groups I want to pass on some lessons learned from the considerable time I spent with OUD11gPS3 and large static groups.  Even though normal tasks you would see a Directory do like search, add, or remove a member from a group work and perform fine, there are things you will want to know about importing and exporting large groups.  Read on to learn more about the tips on importing and exporting large groups into OUD.

BULK IMPORTING VERY LARGE GROUPS

There is a hard limit when importing large groups in excess of 100,000 unique members per group when using ldapmodify. This is also documented in the latest OUD 11.1.2.3.0 documentation here https://docs.oracle.com/cd/E52734_01/oud/OUDAG/tuning_performance.htm#BBADGEFF. If you try to use ldapmodify to import a group greater than 100,000 members you will get the following error.

Connection reset

The good news is there is a way to bulk import groups with very large members using the import-ldif command utility.  One nice thing about using import-ldif is it not only imports large groups, but also updates the indexes as needed toward the end of the import so there is nothing to do once the import completes except to restart the OUD instance.  And the speed at which it bulk imports is extremely fast.  For example I bulk imported a group with 2 million members in under 2 minutes.  A big reason is that when OUD is offline and import-ldif is used, it has access to directly insert the entries directly into the database.  Below are the simple steps you will need import a group.  Note that the LDIF you import should look similar to the example below and you can have more than one group in the LDIF if needed.

Option #1 using import-ldif command:

1.  cd <ORACLE_INSTANCE>/oud1/OUD/bin

2.  ./stop-ds

3.  ./import-ldif –b dc=example,dc=com –a –i “*” –l /home/oracle/import_group.ldif

4.  ./start-ds

Where:

-b - Base DN of a branch to include in the LDIF import
-a - Append to an existing database rather than overwriting it
-i - Attribute to include in the LDIF import
-l - Path to the LDIF file to be imported
optional -
-r - Replace existing entries when appending to the database

Example LDIF:

dn: cn=Rewards,cn=Groups,dc=example,dc=com 
objectclass: top 
objectclass: groupofuniquenames 
cn: Rewards 
uniquemember: uid=josh.l.davis,cn=Users,dc=example,dc=com 
uniquemember: uid=drew.w.walton,cn=Users,dc=example,dc=com 
uniquemember: uid=arnault.o.gunter,cn=Users,dc=example,dc=com

Out of Memory Error using ldapmodify

Another possible error when using ldapmodify while trying to import a group that has greater than 100,000 members is the following.  For example increasing the member-lookthrough-limit value from the default to 100,000 to something much larger may seem like it would help you import beyond the default using ldapmodify, but I have found that I get the following error.

Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded

    at org.opends.server.util.StaticUtils.toLowerCase(StaticUtils.java:2825)

    at org.opends.server.util.LDIFReader.readAttribute(LDIFReader.java:1202)

    at org.opends.server.util.LDIFReader.parseAddChangeRecordEntry(LDIFReader.java:1986)

    at org.opends.server.util.LDIFReader.readChangeRecord(LDIFReader.java:750)

    at org.opends.server.tools.LDAPModify.readAndExecute(LDAPModify.java:344)

    at org.opends.server.tools.LDAPModify.mainModifyNoLogger(LDAPModify.java:1423)

    at org.opends.server.tools.LDAPModify.mainModify(LDAPModify.java:757)

    at org.opends.server.tools.LDAPModify.main(LDAPModify.java:707)

This is related to bug 19602418.  It is possible you could also increase the OUD JVM heap size setting to overcome this.  In one example though I had to bring OUD offline, I was able to import a group that had 2 million members in under 2 minutes.  Considering you can schedule something like that during an off peak time, that is a very fast way of adding the group without worrying about something happening during an import using ldapmodify.  Honestly I would stick with using the import-ldif utility since it is fast and also updates the indexes, which are very important.

Merging Large Bulk Members with an Existing Group

One last group import use case I want to cover is adding over 100,000 members to an existing group in bulk.  For example a company acquiring or merging with another company could possibly have a case where they need to add hundreds of thousands of members to an existing group. Using import-ldif there is an option "-a" which means append, but this is a bit misleading.  The append is actually adding a group with its members to the existing database and does not mean appending attributes.  So for example if you have a group with 1 million existing members, then use import-ldif with the option -a to import another new 250,000 members to an existing group because of some acquisition, the existing group will be overwritten.  What you will have is the same group, but with only the 250,000 members you just imported.  The existing 1 million members will be wiped out.  There are two ways to solve this problem.

1. Export the exiting group using some of the options I point out in the next section, merge the new members into a new LDIF, and re-importing the group using import-ldif with the option "-a" as mentioned above.  Just to reiterate, this is only if you are trying to bulk import new members that exceed 100,000 members.

2. Break the new members into 100,000 chunks and use ldapmodify.  For example you have 250,000 new members you want to add, you can split the members into 100,000 chunks to make 3 LDIF files.  Then use ldapmodify to import the new members, which will then merge with the existing group.

Either of the two options will work fine and you will have to decide what option makes sense for you.

BULK EXPORTING VERY LARGE GROUPS

There may be cases where you or some application may want to return all the members from a group. When it comes to groups that could potentially have millions this is a lot of data for any Directory to deal with, but with OUD PS3 this is possible with the correct tuning. By default if you try to return a group using ldapsearch that has more than 100,000 members the following error happens.

Cannot decode the provided ASN.1 sequence as an LDAP message because the sequence was null
Result Code: 2 (Protocol Error)

The good news is there are a couple options to returning very large groups using either ldapsearch or export-ldif only after making some tuning property changes.

Option #1 tune OUD using the dsconfig command:

If you want to use ldapsearch to return groups that have large members in excess of 100,000, you will need to use the dsconfig command under <OUD_INSTANCE>/OUD/bin and increase the return-attribute-value-limit and member-lookthrough-limit property. However, first you may want to see what the current value is by running the following command.

./dsconfig get-global-configuration-prop \
-h oud1.melander.us \
-p 4444 \
-D "cn=Directory Manager" \
-j passwd.txt \
--advanced \
--property returned-attribute-value-limit \
--property member-lookthrough-limit

Output:

If your properties below are the default values the output should look like the following.

Property : Value(s)
-------------------------------:---------
member-lookthrough-limit       : 100000
returned-attribute-value-limit : 100000

Where:

member-lookthrough-limit – Specifies the maximum number of members that OUD should look through in processing an operation on a static group. Setting the value to 0 (zero) says there is no limit enforced.

returned-attribute-value-limit – Specifies the maximum number of values for an attribute that OUD will return per entry while processing a search. Setting the value to 0 (zero) says there is no limit enforced.

Use the dsconfig command to adjust the returned-attribute-value-limit. For example if you have a group with 2 million users and want to return all the members using ldapsearch, you need to change the returned-attribute-value-limit value to 2000000 or greater depending on how fast the group grows. Setting the value to 0 (zero) will tell OUD there are no limits.  The following is the command to adjust the value as needed and this can be done while OUD is live and it does not need to be restarted.

./dsconfig set-global-configuration-prop \
-h oud1.melander.us \
-p 4444 \
-D "cn=Directory Manager" \
-j passwd.txt \
--advanced \
--set returned-attribute-value-limit:2000000 \
-n

Option #2 using the export-ldif command:

Alternatively if you will not be using ldapsearch to return large groups, then using the export-ldif command found under the <ORACLE_INSTANCE>/OUD/oud1 will export any size group.  The following is the command that will do the job, but if you want to see more options simply run ./export-ldif -? for all the options.

1.  cd <ORACLE_INSTANCE>/oud1/OUD/bin

2.  ./stop-ds

3.  ./import-ldif –b "cn=Rewards,cn=Groups,dc=example,dc=com" –a –i “*” –l /home/oracle/import_group.ldif

4.  ./start-ds

Which export option is fastest?

I have run some tests between using both ldapsearch and export-ldif, and ldapsearch seems to be the fastest by a large margin. However, this is only true if you increase the default value of the return-attribute.value-limit too a minimum larger than the group membership number that is being returned from the large static group or set it to 0 (zero). If you don't increase the return-attribute.value-limit value enough it will produce an error as mentioned earlier.  The following table shows some examples of results I got trying to return a group with 2 million members.

Command	OUD Status	Results
./ldapsearch -h localhost -p 1389 -D "cn=Directory Manager" -w ******  -b "cn=Bronze,ou=Groups,dc=oracle,dc=com" objectclass=* >export_group.ldif	ONLINE	real 0m18.563s user 0m5.820s sys 0m10.080s
./export-ldif -b "cn=Bronze,ou=Groups,dc=oracle,dc=com" -n userRoot -l /scratch/oracle/ldif/groups/members.ldif	ONLINE	real 10m29.260s user 11m56.572s sys 1m49.673s
./export-ldif -b "cn=Bronze,ou=Groups,dc=oracle,dc=com" -n userRoot -l /scratch/oracle/ldif/groups/members.ldif	OFFLINE	real 17m7.991s user 18m25.231s sys 0m38.981s

Summary

Upgrading to OUD PS3 has a lot of positives besides just major improvements with static groups. That said if you are a customer who do use static groups, and the groups are very large or you expect to have groups that will grow to such large numbers, then there is no question OUD PS3 is the version for you.