Don’t Be that Guy – Part 2: Avoiding Outages Due to Full Disks and Partitions

A while back, I wrote about the fact that many customers experience severe outages with their Fusion Middleware products when they let the digital certificates associated with the SSL connections in their deployments expire.

To be fair, certificates are often “out of sight and out of mind” and indeed many system administrators don’t have much experience managing certificates.  However, the same cannot be said about disk space.  We all deal with managing disk space on multiple systems including our desktop clients, home PCs, and even phones. 

Today as a public service announcement I’d like to discuss the dangers of not paying attention to whether or not you have adequate disk space on your dev, test, and production machines running your middleware software. 

I’ll be honest, I see a surprising number of customers experience everything from long delays in their dev and QA cycles to real production outages because of instability caused by running out of disk space.  So, size your machines with adequate disk space, monitor your disk usage, and be aware of your logger and auditing configurations in your Fusion Middleware Products.

Most Fusion Middleware / IAM products including OAM and OIM log to the standard JAVA/WLS logs .out and .log; as well as to the Oracle diagnostic log -diagnostics.log.  The standard logs can be configured in the WLS console while the diagnostic log can be configured by editing the logging.xml file, through WLST, or in EM.
Most customers that use our auditing capabilities log directly to a database.  However, the default storage is “bus-stop files” which do reside on the local file system and obviously take up space.
 
Speaking of databases, I see a fair amount of similar pain being caused by databases running up against various size limits like tablespace or data file limits.  So, make sure you are also actively managing data size limits on the DB.

Custom transformation provider for OIM GTC connector

GTC based connector is one of the most used approaches for reconciling data into OIM, specially through the use of flat files. A common issue is that some customers do not allow direct communication between OIM and the HR system (for different reasons like outsourced HR system, security constraints and others), hence a flat file is made available to OIM so that it reconcile users.

Very often, there is a need to manipulate the data to be reconciled in OIM through the GTC connector. When that is true, most of customers end up creating event handlers to manipulate reconciled data. The problem with this approach is that in OIM 11g, only 'post process' event handlers can be used to manipulate reconciliation data (and the data can only be manipulated after reconciled into OIM), and this can make some manipulations really tricky and/or cumbersome.

OIM 11g Event Handler example

This post shows an example of a post process event handler in OIM. The example is simple and it shows how the user profile can be updated from the event handler based on the information that is provided by OIM to the event handler.

Use case description: a UDF is created in the user profile and it will hold the user's 'Director'. To simplify the use case, the 'Director' will be the 'manager's manager'. In other words, the UDF will be populated with the information from two levels up in the management chain, the value to be used is the director's login.

The first step is to create the UDF that will hold the data. An authorization policy is also needed, otherwise it will not be possible to update the UDF using the APIs. All the steps below must be done in OIM logged as system administrator.

Creating the UDF:

WebLogic crypto operations and Sparc crypto acceleration

I've been doing a LOT of writing recently. Unfortunately, or perhaps fortunately depending on how you feel about my writing, it's all been by email to folks inside Oracle and directly with customers. One thing I wrote that is probably interesting for readers of this blog was an answer to a customer's questions about the crypto acceleration capabilities of the Sparc processor. This is a slight rewrite of that doc...

The crypto acceleration of Sparc and Solaris when it's used for SSL acceleration is right in the sweet spot of confluence between networking, crypto, performance and architecture. In other words just about my favorite place!

Before we get into the technical bits we should probably have a common picture in our heads about what sort of architecture we're talking about. When you deploy web apps you usually have user traffic hit a load balancer which then routes the traffic to your servers. The simplest diagram of that sort of environment is the three boxes diagram - like so:

There are a bunch of different styles of VIP / Load Balancer and there are various ways to configure them. The most common ways (in no particular order) are:
1) Terminate SSL connections at VIP/load balancer
2) Terminate TCP connection at VIP/load balancer, but pass traffic through unchanged
3) Transparent load balancing via a NAT style networking device.

I've seen various names for these technologies, so I'm not going to try to assign names to them for the purposes of this post.

In the case of #1 above the connection between the VIP/LB and the back end server is usually simple, unsecured HTTP. You might use this model if the load balancer has SSL acceleration technologies (either via a hardware card or some software optimization). In the case of #2 and #3 all of the SSL work is handled by the backend server.

Solaris on Sparc offers a few ways to really boost the performance of WebLogic Server. Read on to see what they are.

Tell what patches are installed in your Oracle IAM 11g deployments using OPatch

People often want to know how they can tell what versions (including patches) of software they are running in their different environments. Fortunately, there is a simple standard way to get this information for all software that uses the Oracle Universal Installer. The method I speak of uses the OPatch utility that is standard for all Oracle products that use the universal installer. This includes most, if not all of Fusion Middleware 11g and all of the 11g Identity Management packages.


OPatch is the utility used to patch Oracle products that utilize the Oracle Universal Installer. It is included in most product/package installs under that package’s ORACLE_HOME/OPatch.

Now, keep in mind that when you execute OPatch it will operate on the ORACLE_HOME that is set in the environment. So, if you are operating in an environment where you have multiple products/packages installed under one Fusion Middleware Home (FMW_HOME) you want to make sure that you have an ORACLE_HOME environment variable pointed to the product/package you are interested in patching or analyzing.

To get information on the version and patch levels of your Oracle product/packages run ‘opatch lsinventory’. Again, make sure that ORACLE_HOME is set to the product/package that you want the information for. If it is not set, then OPatch picks some default ORACLE_HOME from your FMW_HOME in a manor I have yet to figure out.

You can also use OPatch to tell you all the ORACLE_HOMEs that exist under your FMW_HOME by running ‘opatch lsinventory -all’.

The documentation for OPatch can be found in the Oracle Universal Installer and OPatch User Guide

Fusion Middleware and Certificate Expirations

I wanted to take a moment to blog about one of the most common, yet most easily preventable causes of middleware system outages. The cause is the expiration of digital certificates used in middleware infrastructure.

Certificates are used throughout many Oracle Fusion Middleware products for purposes of authentication and encryption. This includes:

• Server certificates for SSL in OHS and WLS
• Certificates used for intra and inter product transport communication in WLS, OAM, OAAM, OID, OVD, OIM, OSB etc…
• Certificates used for signing and authentication in OWSM, WLS, OSB, OIF, and Oracle STS.

Despite the fact that digital certificates are such an integral part of middleware infrastructure, you might be surprised how many customers are not well organized about their PKI and in particular about when the different certificates in their infrastructure expire.

It is such a little, simple thing, but all it takes is one expired certificate in one middleware component to bring an application down.

I encourage you today to “not be that customer”, stay organized and keep track of when your certificates expire so that you can issue updated certificates before that happens.

A Note about OAM and the certificates used in Simple Mode

OAM has three different security “modes” for the communication done between the webgate (web server plug-in) and the OAM server component which processes authentication and authorization. These 3 modes are:

• Open: Unencrypted communication
• Simple: Communication over SSL using certificates generated by OAM using an Oracle CA certificate that comes with OAM. Simple mode is popular with customers who want to secure webgate to OAM server communication but don’t feel the need to use their own PKI infrastructure.
• Cert: Communication over SSL using certificates issued by a third party CA.

The mode most relevant to this discussion on expiring certificates is simple mode where the certificates are generated automatically and may be out of sight and mind.

In OAM 10g the certificates are only issued with a validity period of one year by default. If you want to extend the validity period for the certificates generated in simple mode (and I recommend that you do), then follow the instructions in appendix F, section 2.9 of the OAM Identity Admin Guide: http://download.oracle.com/docs/cd/E12530_01/oam.1014/b32419/trblsht.htm#BABDDBJF

In OAM 11g, the validity period of the certificates used in simple mode is 10 years, making early expiration of the certificates less of a problem.