Showing posts with label troubleshooting. Show all posts
Showing posts with label troubleshooting. Show all posts

Monday, September 17, 2012

Creating your first OAM 11g R2 domain

So you downloaded the Identity Management R2 release bits, spun up your little test environment and created a WebLogic domain. But the first time you sign in you get the error message "The policy store is not available; please see the log file for more details." in a pop up.

like this:

The logs aren't particularly helpful:
####<Sep 13, 2012 6:19:42 PM EDT> <Error> <oracle.oam.engine.policy> <iamr2.oracleateam.com> <AdminServer> <[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <e3b75e49ebb52881:-4d179e40:139c1939ab6:-8000-00000000000005a3> <1347574782661> <BEA-000000> <The policy store is not available; please see the log file for more details.
oracle.security.am.common.policy.admin.store.PolicyStoreException: OAMSSA-06252: The policy store is not available; please see the log file for more details.
        at oracle.security.am.common.policy.util.OESUtils.checkAndThrowException(OESUtils.java:630)
        at oracle.security.am.common.policy.util.ResourceTypeHelper.setupHostIdentifierResourceType(ResourceTypeHelper.java:438)
        at oracle.security.am.common.policy.admin.provider.oes.DefaultApplicationDomain.createHostIdentifierPolicy(DefaultApplicationDomain.java:118)
        at oracle.security.am.common.policy.admin.provider.oes.DefaultApplicationDomain.<init>(DefaultApplicationDomain.java:93)
        at oracle.security.am.common.policy.admin.provider.oes.DefaultApplicationDomain.getGlobalDefault(DefaultApplicationDomain.java:461)
        at oracle.security.am.common.policy.admin.provider.oes.ApplicationManager.setupGlobalDefaultAppDomain(ApplicationManager.java:112)
        at oracle.security.am.common.policy.admin.provider.oes.ApplicationManager.<init>(ApplicationManager.java:61)
        at oracle.security.am.common.policy.admin.provider.oes.ApplicationManager.getApplicationManager(ApplicationManager.java:125)
        at oracle.security.am.common.policy.util.OESSetupHelper.loadOAMApplicationManager(OESSetupHelper.java:340)
        at oracle.security.am.common.policy.util.OESSetupHelper.loadOAMApplicationPolicies(OESSetupHelper.java:166)
        at oracle.security.am.common.policy.util.OESSetupHelper.loadApplicationPolicies(OESSetupHelper.java:154)
        at oracle.security.am.common.policy.admin.provider.oes.proxy.OESAdminProxy.init(OESAdminProxy.java:84)
        at oracle.security.am.common.policy.admin.provider.oes.OESPolicyAdminProvider.init(OESPolicyAdminProvider.java:130)
        at oracle.security.am.common.policy.admin.PolicyAdminFactory.getProvider(PolicyAdminFactory.java:241)
        at oracle.security.am.common.policy.admin.PolicyAdminFactory.init(PolicyAdminFactory.java:166)
        at oracle.security.am.common.policy.admin.PolicyAdminFactory.getPolicyAdmin(PolicyAdminFactory.java:334)
...
And in the -diagnostic log:
[2012-09-13T18:19:42.364-04:00] [AdminServer] [NOTIFICATION] [] [oracle.adfdt.model.mds.MDSApplicationService] [tid: [ACTIVE].ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: weblogic] [ecid: e3b75e49ebb52881:-4d179e40:139c1939ab6:-8000-00000000000005a3,0] [APP: oam_admin#11.1.2.0.0] [[
oracle.mds.exception.ReadOnlyStoreException: MDS-01273: The operation on the resource /oracle/oam/ui/adfm/DataBindings.cpx failed because source metadata store mapped to the namespace / DEFAULT is read only.
        at oracle.mds.core.MDSSession.checkAndSetWriteStoreInUse(MDSSession.java:2495)
        at oracle.mds.core.MDSSession.checkAndSetWriteStoreInUse(MDSSession.java:2548)
        at oracle.mds.core.MDSSession.getMutableMO(MDSSession.java:3493)
        at oracle.mds.core.MDSSession.getMutableMO(MDSSession.java:1660)
        at oracle.mds.core.MDSSession.getMutableMO(MDSSession.java:1546)
        at oracle.adfdt.model.mds.MDSApplicationService.findApplication(MDSApplicationService.java:57)
        at oracle.adfdt.model.mds.MDSModelDesignTimeContext.initServices(MDSModelDesignTimeContext.java:232)
        at oracle.adfdt.model.mds.MDSModelDesignTimeContext.<init>(MDSModelDesignTimeContext.java:82)
        at oracle.adfdt.mds.MDSDesignTimeContext.<init>(MDSDesignTimeContext.java:66)
        at oracle.adfinternal.view.page.editor.Page.getDesignTimeBindingContainer(Page.java:596)
        at oracle.adfinternal.view.page.editor.contextual.event.ContextualModelManager.getBindingContainerForView(ContextualModelManager.java:209)
        at oracle.adfinternal.view.page.editor.contextual.event.ContextualModelManager.getCurrentContextualResolver(ContextualModelManager.java:131)
        at oracle.adfinternal.view.page.editor.bean.ContextualWiringBean.getResolver(ContextualWiringBean.java:625)
        at oracle.adfinternal.view.page.editor.bean.ContextualWiringBean.clearSelection(ContextualWiringBean.java:594)
        at oracle.adfinternal.view.page.editor.bean.ContextualWiringBean.handlePageNavigation(ContextualWiringBean.java:130)
        at oracle.adfinternal.view.page.editor.contextual.event.EventHandler.processNavigation(EventHandler.java:92)
...


What did you do wrong?!

Friday, March 16, 2012

OIM 11g LDAPSync Deployment Guidelines

OIM 11g can be configured to maintain its user and role population synchronized with an LDAP directory using the LDAPSync feature.  This functionality is based on asynchronous processing through orchestration events from OIM to LDAP and on scheduled tasks for synchronization from LDAP to OIM.  This approach could mean that at some point in time, some of the entries on both repositories may be out-of-sync.  Specially when executing long running Trusted Reconciliation scheduled jobs.  The entry differences can be caused by processing errors or time lapse between user creation in OIM and user creation in LDAP.  This post details some guidelines to minimize and troubleshoot possible errors for OIM LdapSync.

Consider upgrading to OIM 11.1.1.5.2 (OIM 11g Bundle Patch 02).  BP02 provides a number of fixes that improve stability specially during bulk processing (large trusted reconciliations).

Tuning and Tips

Tune the environment to allow for better ldapsync performance when executing large trusted reconciliation jobs in the order of +30K.  These are some tuning tips, some of them straight from the documentation, others from existing deployments.


OIM Tuning

1. JVM:  PORT_MEM_ARGS="-Xms768m -Xmx2048m" in setSOADomainEnv.sh
2. MDBs: In WLS Admin Console -> Environment->Worker Managers->MaxThreadsConstraints-1  set count to 100
3. Disable reloading of Adapters:  Using weblogicExportMetadata.sh and weblogicImportMetadata.sh edit the MDS file /db/oim-config.xml.  
   Replace:
     <ADPClassLoaderConfig adapterReloadingEnabled="true" loadingStyle="ParentFirst" reloadInterval="15"     reloadingEnabled="true">
   With:
    <ADPClassLoaderConfig adapterReloadingEnabled="false" loadingStyle="ParentFirst" reloadInterval="15" reloadingEnabled="false">
   And
    <storeConfig reloadingEnabled="true" reloadingInterval="20"/>
   With:
   <storeConfig reloadingEnabled="false" reloadingInterval="20"/>

4. Database:  Monitor Perfomance with AWR reports and collect complete schema statistics using:
                  DBMS_STATS.GATHER_SCHEMA_STATS(OWNNAME=> schema_owner,
                                                                                              ESTIMATE_PERCENT=>DBMS_STATS.AUTO_SAMPLE_SIZE,
                                                                                              DEGREE=>8,
                                                                                              OPTIONS=>'GATHER AUTO',
                                                                                              NO_INVALIDATE=>FALSE);

OID Tuning
  1. Consider deploying multiple OID instances for Failover and Load Balancing. Front end the instances with an LB
  2. Number of OID LDAP Server Processes = #cpus. EM -> Administration -> Server Properties -> Performance ( for each oid instance)
  3. Number of DB Connections per Server Process = 10 (orclmaxcc). EM -> Administration ->server properties (for each oid instance)
  4. Check Skip Referral for Search. EM -> Administration -> Shared Properties (Only if not using referrals in OID).

OVD Tuning (assuming no libOVD is used)
  1. Consider deploying multiple OVD instances for Failover and Load Balancing. Front end the instances with an LB and use the OID LB virtual host as the LDAP server host.
  2. The parameter Operations Timeout for the Adapters to 30000 if needed (using ODSM)
  3. Consider increasing maxpoolsize  for the Adapters to 30-40 if needed (using ODSM)

LDAPSync Monitoring

During the time when the LdapSync Orchestration is running check the following tables and columns in the OIM Schema to verify processing:

Obtain the latest reconciliation job key (RJ_KEY) with the query:

        select max(RJ_KEY) from recon_events;

Table RECON_BATCHES:  using RJ_KEY and RB_NOTE verify that the orchestration events are being created.  The Column RB_NOTE shows the orchestration Process ID and the operation.  It could also show errors that occur.

Table ORCH_PROCESS:  Holds the generated orch processes.  ID being the Orchestration Process ID.  If Status shows Compensated it means that an event failed.  The detail can be seen in the table ORCH_EVENTS.

Table ORCH_EVENTS:  Linked to the orchestration process with the column PROCESSID.  The RESULT column has the error details in case of failure.

The Out-of-the-Box reconciliation job " Retry Failed Orchestrations " can be used to retry compensated orchestration processes.   Specify a date range ("ddMMyyyy") for multiple ones or "Orchestration ID" for single ones.  OIM BP02 also includes fixes for this task.

Note:  In the case when reconciling new users that come as disabled, an existing bug may create the orchestration disable event before the orchestration create event.  So, these disable events would fail in LDAP since the user doesn't exist yet.  As a workaround, these events can be retried with the above Recon Job (specify OPERATION=DISABLE and a date range) after all users are created in LDAP.

Monday, June 21, 2010

Troubleshooting OES 10gR3 cp4 Installation of WLS SM

I wanted to share 3 troubleshooting tips that I used in installing and configuring OES 10gR3 cp4 for the OES OWSM update post

SCM Won't Start


In order for the SCM to run, it needs to have to separate sockets - one for "public" and one for "private" requests. You can see the SCM configuration in ales32-scm/apps/scm-asi/SAR-INF/config.xml. By default the installer creates one listener for the hostname on port 7013 and another on 127.0.0.1 on port 7013. If in your environment the hostname maps to 127.0.0.1, then these are the same socket, and when the SCM tries to start the second listener, you'll get a BIND Exception. The way to fix this is to create a second loopback adapter, with a fixed IP address, and change the 127.0.0.1 to that IP address. This will fix the SCM start-up problem.

WLS Server with WLS SM won't start - non JRF Domain


After running the config tool, and all of the policies have been created, the WLS server won't start with an error like "User weblogic is not authorized to boot the server". This means that, most likely, the policies have not been distributed to the WLS Server, so you need to force a policy distribution. The surefire way to do this is as follows:

  • In the ales32-ssm/wls-ssm/instance/instancename/work/runtime remove the state.chk
  • Remove all of the files in ales32-ssm/wls-ssm/instance/instancename/work/runtime/policyA
  • Remove all of the files in ales32-ssm/wls-ssm/instance/instancename/work/runtime/policyB
  • At this point in ales32-ssm/wls-ssm/instance/instancename/work/runtime all you should have two empty directories, policyA and policyB
  • Log into the asi console, navigate to "Deployment", "Deployment Status" and click on the trash can next to the instance of SM. This won't hurt anything, just force the SM when it starts to re-register with the admin and get fresh policy.


If you did it right, when you restart the WLS domain, you'll get a warning about a "missing state.chk" which means that you're getting fresh policy, and the server should boot normally.

WLS Server with WLS SM won't start - JRF Domain


The most common example of a JRF domain is a SOA Suite Domain. A JRF domain is using OPSS and OPSS requires at least one LDAP authentication provider. The config tool for the WLS SM creates a WLS realm that only has the OES RDBMS authenticator. In this case, even if you get passed the previous issue, you'll hit an error like:

<Jun 21, 2010 4:35:06 PM EDT> <Error> <Security> <BEA-090892> <The dynamic loading of the OPSS java security policy provider class oracle.security.jps.internal.policystore.JavaPolicyProvider failed due to problem inside OPSS java security policy provider. Exception was thrown when loading or setting the JPSS policy provider. Enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: oracle.security.jps.JpsException: [PolicyUtil] Exception while getting default policy Provider>
<Jun 21, 2010 4:35:06 PM EDT> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: The dynamic loading of the OPSS java security policy provider class oracle.security.jps.internal.policystore.JavaPolicyProvider failed due to problem inside OPSS java security policy provider. Exception was thrown when loading or setting the JPSS policy provider. Enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: oracle.security.jps.JpsException: [PolicyUtil] Exception while getting default policy Provider
weblogic.security.SecurityInitializationException: The dynamic loading of the OPSS java security policy provider class oracle.security.jps.internal.policystore.JavaPolicyProvider failed due to problem inside OPSS java security policy provider. Exception was thrown when loading or setting the JPSS policy provider. Enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: oracle.security.jps.JpsException: [PolicyUtil] Exception while getting default policy Provider
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadOPSSPolicy(CommonSecurityServiceManagerDelegateImpl.java:1394)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1018)
at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:875)
at weblogic.security.SecurityService.start(SecurityService.java:141)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
Truncated. see log file for complete stacktrace

Caused By: oracle.security.jps.JpsRuntimeException: oracle.security.jps.JpsException: [PolicyUtil] Exception while getting default policy Provider
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:256)
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:248)
at oracle.security.jps.internal.policystore.JavaPolicyProvider.<init>(JavaPolicyProvider.java:130)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
Truncated. see log file for complete stacktrace

Caused By: oracle.security.jps.JpsException: [PolicyUtil] Exception while getting default policy Provider
at oracle.security.jps.internal.policystore.PolicyUtil.getDefaultPolicyStore(PolicyUtil.java:675)
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:254)
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:248)
at oracle.security.jps.internal.policystore.JavaPolicyProvider.<init>(JavaPolicyProvider.java:130)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
Truncated. see log file for complete stacktrace

Caused By: java.security.PrivilegedActionException: oracle.security.jps.JpsException: [PolicyUtil] Unable to obtain default JPS Context!
at java.security.AccessController.doPrivileged(Native Method)
at oracle.security.jps.internal.policystore.PolicyUtil.getDefaultPolicyStore(PolicyUtil.java:622)
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:254)
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:248)
at oracle.security.jps.internal.policystore.JavaPolicyProvider.<init>(JavaPolicyProvider.java:130)
Truncated. see log file for complete stacktrace

Caused By: oracle.security.jps.JpsException: [PolicyUtil] Unable to obtain default JPS Context!
at oracle.security.jps.internal.policystore.PolicyUtil$1.run(PolicyUtil.java:637)
at oracle.security.jps.internal.policystore.PolicyUtil$1.run(PolicyUtil.java:622)
at java.security.AccessController.doPrivileged(Native Method)
at oracle.security.jps.internal.policystore.PolicyUtil.getDefaultPolicyStore(PolicyUtil.java:622)
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:254)
Truncated. see log file for complete stacktrace

Caused By: oracle.security.jps.service.idstore.IdentityStoreException: Failed to create identity store service instance idstore.ldap.provider:idstore.ldap. Reason: No Default or LDAP Authenticator configured on WLS.
at oracle.security.jps.internal.idstore.ldap.LdapIdentityStoreProvider.getIdStoreConfig(LdapIdentityStoreProvider.java:227)
at oracle.security.jps.internal.idstore.ldap.LdapIdentityStoreProvider.getInstance(LdapIdentityStoreProvider.java:108)
at oracle.security.jps.internal.idstore.ldap.LdapIdentityStoreProvider.getInstance(LdapIdentityStoreProvider.java:59)
at oracle.security.jps.internal.core.runtime.ContextFactoryImpl.findServiceInstance(ContextFactoryImpl.java:139)
at oracle.security.jps.internal.core.runtime.ContextFactoryImpl.getContext(ContextFactoryImpl.java:170)
Truncated. see log file for complete stacktrace

Caused By: oracle.security.jps.JpsRuntimeException: No Default or LDAP Authenticator configured on WLS

So, you need an LDAP authentication provider. The quick and dirty way to fix this is by editing the config.xml. You can simply cut and paste the DefaultAuthenticator from the other realm (myrealm) in the file and add it before the OES RDBMS authenticator

<sec:authentication-provider xsi:type="wls:default-authenticatorType">
<wls:use-retrieved-user-name-as-principal>true</wls:use-retrieved-user-name-as-principal>
</sec:authentication-provider>
<sec:authentication-provider xmlns:ext="http://www.bea.com/ns/weblogic/90/security/extension" xsi:type="ext:database-authenticatorType">
<n1:name xmlns:n1="http://www.bea.com/ns/weblogic/90/security">DatabaseAuthenticator</n1:name>
<ext:jdbc-driver-class-name>oracle.jdbc.driver.OracleDriver</ext:jdbc-driver-class-name>
<ext:jdbc-connection-url>jdbc:oracle:thin:@localhost:1521:xe</ext:jdbc-connection-url>
<ext:database-user-login>oes10gR3cp4</ext:database-user-login>
<ext:identity-scope>RootOrg!defaultOrg!defaultUsers</ext:identity-scope>
<ext:database-user-password-encrypted>{AES}uUvyzqwh98bA/POG9/jhP6ITfnptEYE0RbkaZMofsC4=</ext:database-user-password-encrypted>
</sec:authentication-provider>


This should get the server started. Now, what I like to do is to have the Default Authenticator 1st, and have it set to sufficient. Keep the RDBMS authenticator, and also have it as sufficient. By doing this, if you want to use tools like Eclipse to deploy applications to a WLS domain protected by OES, you won't run into the issue of Eclipse de-serializing the OES principals. Trust me, this is the simplest set-up.

Summary


First all, Chris helped me with some of these pointers, but he's in Europe helping a customer, so I didn't want to wait. Full credit to Chris for wading through many of these issues first. I hope that the pointers help you get OES 10gR3 CP4 WLS SM running smoothly. If you have some pointers of your own, please post them here.

Tuesday, February 23, 2010

Oracle database 11g default password policies

I recently booted a VM that I hadn't used in some time and found this error appeared when I fired up sqlplus
ERROR:
ORA-28002: the password will expire within 7 days
From what I've been able to gather Oracle database 11g sets up a password expiration policy by default. This isn't usually a problem for me since I usually create a VM, work on it for a week or six before moving on to another project. But this VM is special - it's my combined IAM stack demo VM from OpenWorld and I need it every once in a while.

If you find yourself in this position here's how you disable the password policies and get things working again.
First turn off the password policies:
ALTER PROFILE DEFAULT LIMIT
FAILED_LOGIN_ATTEMPTS UNLIMITED
PASSWORD_LIFE_TIME UNLIMITED
PASSWORD_GRACE_TIME UNLIMITED;
Then check the status of your accounts:
set line 120
select username, account_status, expiry_date, lock_date from dba_users;
Then reenable the ods user and any other users by resetting their password:
alter user ods identified by abcd1234;
Hope this helps!

Remember: Don't do this in production!