Part 1: How To Load Test OAM11g using Apache JMeter

Exciting, it is Go Live day, the system goes online, everything seems ok for a while, and then Kerplunk!  Thousands of things could have happened and everyone scrambles to figure it out.  What went wrong?   My first question is, “Was a proper load test completed?”  Yes, Load Test.  Functionally the software may have worked, but under a heavy load issues can start bubbling up.  If you have been there like I have many times you know what I am talking about.  Situations like this could be avoided? 

This is where this article comes in.  I have learned the hard way that load testing can provide a lot of great feedback to determine issues whether it be the software, the design, the deployment, or the architecture.  The bottom line is if you do not have fancy software like HP Load Runner, the poor mans version, Apache JMeter, can come to the rescue.  JMeter is FREE, well documented (As much as any other software at least), and there are tons of blogs and articles on JMeter since it has been around for quite a while.  JMeter can also do much more beyond HTTP type of load testing.  I would love to cover even more about JMeter, but unfortunately I have to focus this article to a basic load test that works with OAM11g.   Minimally, I hope this article at least gets you started and even excited to learn more about this great tool.

About Apache JMeter

Apache JMeter is a free open source desktop application written in pure Java.  JMeter is used as a load testing tool to measure performance.  In addition to applying a load to a system, JMeter provides Listeners that monitor the load details so that the feedback from the listeners can be used to tune a system; e.g. OHS web servers, OAM Servers, LDAP servers, etc.

Typically HP Load Runner is the industrial strength load testing tool used by larger enterprise clients, but often I see things like the team that uses it either does not have the time nor the resources, or maybe the project did not have such a tool to complete any load testing.   To get started, first download the latest version of Apache JMeter and get it up and running.

TIP: Note that I have seen bugs with SZXParseException errors in version 2.10, so if you come across these errors download 2.9 to see if that corrects the problem.

Learning by Example

I have written this article so that you will use an example I have provided in a zip file OAM11g_LoadTest.zip [CLICK TO DOWNLOAD], and wrote the article in a format that explains all the important elements of the JMeter plan to make this all work for OAM11g.  I may not cover every detail about the elements because there are so many things, so please explore the Apache JMEter User Manual http://jmeter.apache.org/usermanual/ and look up how some of these elements work.  The Apache JMeter site has some pretty good documentation and if you google there are also a lot of great articles that help supplement the User Manual.

The JMeter OAM11g Load Test Example

If you have not already, download the example OAM11g JMeter Load Test plan I have provided and lets get started.  Note that included in the sample JMeter plan is a supporting comma separated file (CSV) file that holds two columns, username and password, more on this later.  Once you open up the plan the screen should look like the following graphic.

The Elements of the Plan

As promised this section covers all the important parts of the JMeter Load Test plan I provided.   Note that the most critical of the elements to make this work with the OAM11g login are the elements under the Portal Request branch using either the Regular Expression Extractor or XPath Extractor; more on these later.  So without further ado…

The Thread Group

·      The Thread Group provides a way to configure the number of threads and Users that are load tested against your IDM and Application system.  You can tweak how fast the total number of Users makes requests by changing the value of the Ram-Up Period.  As an example if you have 1500 Users, then set the Ramp-Up Period to 150 seconds, that starts the initial requests at 5 User logins per second (1500 Number of Threads or Users / 150 Ramp-Up Period in seconds = 5).   After 150 seconds all 1500 Users will have opened up requests that were created in the load test plan.  The Number of Threads are the total number of Users the test will run though, but if you set the Loop Count to 2 the load will then be 2 times the Number of Threads though not all at once hence the term loop.   Basically it just runs through the same test twice or what ever you set the Loop Count for.  What you configure the Thread Group for is solely based on the type of metrics you are trying to simulate.  It is best to start small and work up to larger volumes.  JMeter can consume a lot of member if there are hundreds or even thousands of requests.

Common Configuration Elements

The following are what I call common configuration elements.  They are located at the top of the load test plan because they provide common settings or functionality used across all the load test elements.

Common Configuration Elements

The following are what I call common configuration elements.  They are located at the top of the load test plan because they provide common settings or functionality used across all the load test elements.  

HTTP Cookie Manager

This configuration helps store cookies sent by OAM so that when JMeter gets the necessary OAMAuthnCookie or ObSSOCookie depending on whether it is a 11g or 10g Webgate respectively, all the necessary cookies will be stored in JMeter so that SSO can be accomplished.  Note in the plan I provided the option Clear cookies each iteration has been checked so that each new request simulates a User closing their browser thereby requiring to get new cookies on a subsequent new thread.

HTTP Cache Manager

This configuration will cache resource elements like URLs in JMeter’s memory so that it is not always reaching out for the same URL over and over.  The purpose of this is to simulate how many browsers improve performance by caching requests.  Note I have also checked Clear cache each iteration so that cache is not reused for each new thread request. 

Configurable Variables (User Defined Variables)

This configuration is used to simplify making hostname and port changes across all the HTTP Request samplers that use a host and port.  In my plan I  have a hostname for the SSO Login server and a hostname for a Portal Server, so you may need to change this including the ports to reflect your test environment.   If your environment hostname for the application and login are the same, then modify the test plan accordingly.  In any case, instead of tediously going through all the HTTP Request elements throughout the plan and updating the Server Name or IP and Port Number values individually one by one, you can easily make the change in one place.  The reason this works is because for all the HTTP Request elements those values are populated with a dynamic variable like ${sso_host} and ${app_host}, then for example the Host Variables table the variable Name “app_host” gets replaced by the actual Value you included in the configuration element.  This is a real time saver. 

User Data (CSV Data Set Config)

This configuration provides a way to include an external flat file that will hold things like Usernames, Passwords, etc.  In my plan I only include the Username.  The file name is entered in the Filename field; my example uses the included Users.csv file.  The User.csv file I provided has one column, username, but you could add other columns like password, employeenumber, etc.  In my plan I know that all the passwords for each User is the same,  so I added the password as a variable in the Configuration Variables element by adding the variable Name “password” and in the Value field the actual password.  Then I added the ${password} variable in the Login Submit element.  Feel free to experiment and learn. 

The Simple Controller Elements

Within the Logical Controller in JMeter is an option called the Simple Controller, which provides a way to organize all types of elements like HTTP Requests, Regular Expression Extractors, XPath Extracters, etc.  The Simple Controller really does nothing more than provide for a way to logically group elements of the load test plan.  For example I used three Simple Controllers and named them Login, AuthZ, and Logout to represent the three functional actions of my Load Test plan.   As you build the complexity of the plan you will appreciate breaking the functionality of the test plan into logical groups using this technique.   Feel free to expand on the idea and add more Simple Controllers if you feel it is necessary.  The following gives more details on each group I created to help better understand what I am trying to accomplish.

Login (Simple Controller)

The Login Simple Controller is meant for all actions that have to do with a User logging into OAM.  The following are explanations of the elements.

Portal Request (HTTP Request)

Modify the Path parameter URI to point to a protected HTTP resource.   The protected resource should redirect JMeter to the initial login page.

Extract QueryString from request_id (Regular Expression Extractor)

This comes from one of the Pre-Processors elements.  If you are using a custom form login, the Regular Expression Extractor is used to extract the OAM 11g querystring parameter called request_id; this was not required in OAM10g so if you happen to be load testing OAM10g you can disable this element.  The Regular Expression Extractor basically looks for the existence of the a HTTP querystring parameter named request_id and if found extracts the value.  The value changes on every request, which is why it needs to be extracted dynamically on every login attempt.  The request_id is a critical value required when posting to a custom form login in order to authenticate, without it the login would fail.  To learn more about this, Brian Eidelman wrote a great blog on this very topic, “External Custom Login Forms with Oracle Access Manager 11g”.

Extract Form Input from request_id (XPath Extractor)

Similar to the “Extract QueryString from request_id”, this element is used when you are using the OAM11g Out-Of-The-Box (OOTB) form login.  When the OAM11g OOTB form login is used the request_id does not present itself as a querystring variable, instead it shows up as a hidden form input field called “request_id”.  In order to get this value the XPath Extracter is used with a XPath query “//input[@name='request_id']/@value”, which grabs the value.  Then the Reference Name field is named “request_id” and used as a dynamic parameter ${request_id} in the Login Submit element to post the value.  By default I have disabled this element, but if you are using the OOTB form login, enable this element and disable the Extract QueryString from request_id element.

Extract Form Input from OAM_REQ (XPath Extractor)

Similar to the “Extract Form Input from request_id” element, this element is enabled when the serverRequestCacheType parameter in the oam-config.xml file is set to FORM.  If you are not sure what your parameter is set to look in $ORACLE_HOME/$FMW_HOME/ user_projects/domains/oam_domain/config/fmwconfig to find the oam-config.xml and search for serverRequestCacheType.  There are three optional modes, BASIC, COOKIE, or FORM.  When it is set to FORM, the request_id is sent as OAM_REQ as a hidden form input field.  So in this case the XPath query is set to “//input[@name='OAM_REQ']/@value”.  However the Reference Name is still request_id in order to keep the dynamic parameter the same to keep everything as consistent and simple as possible.

Login Submit (HTTP Request)

An important configuration for this element is the path, the value needs to be /oam/server/auth_cred_submit so that JMeter will submit all the credential parameters to the OAM11g login just like a person submitting their credentials in a login form.  Note that in the Parameters table all the additional elements required are included in the sample plan like username, password, and request_id.  Note that  displayLangSelection, and Languages are only added if required to change the local language. 

Authz (Throughput Controller)

This is another type of controller that can control the number of HTTP authorization requests that happen after login.  After the Login actions happen from the Login (Simple Controller), JMeter has the proper OAM SSO cookies that are temporarily stored in the HTTP Cookie Manager.  So in this logical controller, I make all my URL requests simulating authorizations.  In the example I provided there are only three sample HTTP Requests and each Path URI is simply “/”, so be sure to modify each HTTP Request per your load test setup and in fact do not be shy, add more and even elaborate on other options to simulate a User clicking on various pages in your application.  My sample includes the following.  Note that in each of the following elements is a Server Name or IP field that has the dynamic value ${app_host} that relates to the Configurable Variables element where you would have the real application hostname.  Feel free to experiment. 

• Portal Home (HTTP Request)
• My Account (HTTP Request)
• Info Page (HTTP Request)

Logout (Simple Controller)

This is another Simple Controller that contains actions required to logout and also verifies the logout was successful.  Since a typical User will complete a logout, it is important to include this logical controller as part of the load test life cycle.  Not that each HTTP Request element uses the Method GET.

Logout Request (HTTP Request)

Note that the Path parameter needs to point minimally to the /oam/server/logout URI, but in my example I have used the URI value /oam/server/logout?end_url=/logout.html.  The end_url is a special parameter that tells OAM to return the User to a custom landing page after the logout.  In my setup I have a custom logout.html page on the same host as the custom login.html page.  For your setup feel free to remove this extra parameter or modify it as necessary per your setup.  It is important to check your OAM11g setup to know how it has been configured to properly logout of the application from which you can then modify the JMeter plan as needed.

Logout Landing (HTTP Request)

This element requests the final logout page URL.   Note that underneath this element is a Response Assertion, which determines if the logout was successful, more on that below.

Validate Logout (Response Assertion)

If you look at this element, it has the following configurations selected, Main Sample Only, Document (text), Contains, and Patterns to Test where the pattern match is “You are now logged out” based on what is displayed in my logout.html page.  However, in your case you may want to look for something else that is seen in the final logout page.   I would use an HTTP trace tool like Fiddler, or FireFox HTTPFox, or any other HTTP trace tool to determine what is required.  Basically the Response Assertion can look at content seen in a page and if found it returns a TRUE, which means the logout was successful.  If FALSE, it will show up as red in the View Results Tree and provide a negative response.  Take a look at the options in the Response Assertion; it could be text in the HTML page, a URL, or other.  Please modify the Response Assertion element per your environment logout page action so that it can determine if the logout was successful.  If you want, you could also disable this element. 

Listeners

Listeners are elements that provide a way to monitor events during a load test.  In the load test sample I provided I just used two Listeners, but feel free to experiment with other Listeners to learn more.

View Results Tree

This element is very useful because it records the requests and responses from each Sampler event during a load test and displays exactly what the client would see in an Internet browser. 

Aggregate Report

This element records a nice comprehensive table of responses like errors, throughput, median, etc. seen in each thread request.  As you review the report you can look at each column of each row to help identify bottlenecks or successes.  There is even an option to save the results to a file so that each load test can have a separate report that can be compared to determine positive and negative improvements. 

Test the Load Script

Now that you have tweaked the example JMeter load test plan per your environment it is time to test.

CAUTION: Before running the test I want to point out some words of caution. JMeter will consume as much memory as it needs without concern of how much memory in the computer it is running on. Therefore, if you enter a high value in the Number of Threads, your desktop can potentially come to a grinding halt or worse freeze. If you want to send a high volume of requests to the system in the case you need to simulate thousands of requests, it is a good idea run several JMeter instances on more than one desktop and simultaneously run them all at the same time. Since JMeter is written in pure Java, you can run it on any platform that supports the Java JRE 1.6x or higher.

It is a good idea to run a light load from a single JMeter instance to test the plan.  Before kicking off the load test, select the View Results Tree, then click the green right arrow from the top menu to start the load test; see the illustration below.  You should now start to see requests populate the results tree.  Once the test is completed review the results in the Aggregate Report.  If you want to run the test again use the double broom button to clear everything.

I had mentioned early on that in the Thread Group element you could change various parameters to send various types of loads.  In Part 2 I will include more details on how this can be used with multiple Thread Groups along with making the test more random.

Beyond Apache JMeter

I saved the boring stuff for last, so don’t say I didn’t do you any favors.  I wanted to take a step back now and talk a little bit about load testing tips in general.  The following are things I am aware of based on years of experience in regard to good load testing.  Don’t take these as golden rules, just use them as good guidelines.

1. Use Realistic Metrics

Load tests should generally be based on real traffic to provide pretty good estimations on what to expect when you go live.   Do not guess on the metrics to load test; that is simply a bad idea and where customers usually get into trouble.  Try to simulate real load pattern seen in production.  For example average page views, session duration, page size, average loads, peak loads, and basically data to help simulate what an average user would do after logging in and completing some actions.  The metrics should also determine how many concurrent logins are seen on average during peak periods.  In fact average loads in general are good to know.  One reason is that average loads are important during longevity load tests because often a system may hold up gracefully during a peak load, but act differently over longer periods of time even at average loads.  Also be realistic, for example if there are peak loads of say 5000 unique concurrent logins an hour, don’t just load test with 100 unique concurrent logins per hour even if you push the requests at a higher rate because the software caching can skew the expected results and when you go to production the difference between 100 concurrent logins per minute versus 5000 concurrent logins per hour can make a difference.   If you don’t load test using real life loads and patterns, the result is you may end up being unprepared when going live and parts of the system start toppling over because someone tried to rush and not do their homework or maybe they were lazy.  Do yourself a favor and do the homework, it will make a difference between working toward success or gambling on having the stress of dealing with firefighting.  

2. Stress Test all Actions; Login, Authorization, and Logout

Good load testing should simulate the full life cycle of a User using the system.  For example login, authorization actions, and logout patterns.   I have seen to many times where authentication patterns are only stress tested, and think that is good enough to simulate a real production system test.   Based from my experience, as a general rule authorization requests account for approximately 75% percent of the load.  In fact I have seen clients that have discovered authorization requests can go even higher at more than 80% percent of the load and that load can even be more compounded by things like Policy Responses in the OAM Policies that send lots of header, cookie, etc.  All that additional data adds to the load of the system.  Therefore load tests should be made as close to real life as possible by iterating all the basic actions of a life-cycle, logging in, accessing pages, entering data if possible, registering, and logging out.  The closer to realistic load patterns that can be designed into the load test, the more chances issues will bubble up during your testing, which gives you time to correct for any problems before going live.  

3. Monitor All Components

Load testing is just a part of the testing phase so don’t think that just sending a big load against your system and seeing it hums right along with flying colors it is all done.  Taking the time to monitor the components from end-to-end is very important especially in cases where you identify bottle necks that help identify defects or even the need to do some tuning.  There are always cases where certain tiers may not be scaled enough or little problems bubble up and the only way to determine that is to monitor as much as possible.  If problems arise, the solution could be as simple as tuning, and in some cases scaling up or even out.  In a worse case scenario maybe the design needs to be reworked.  The good news is that proper load testing identifies as much as possible before going live. 

4. Document the Results

This may seem obvious, but documentation seems to be one of those items no one wants to do.  It is a good idea to create some type of document with tables to record necessary results in a historical manner.  For example using Excel to record results and changes.   This provides a way to present of all the negative and positive results to determine progress and if the system is ready for prime time.   The document should also have ways to record potential issues, bugs, etc. so that those things can be addressed and the load tests can be run again once the defects have been corrected.   Once all the issues are worked out you can at least show that due diligence has been done before going live.  As a benefit if issues do bubble up in production, you have a tool that can help potentially reproduce the problems in a non-production environment so that corrections can be tested versus trying to do it in a live system.

5. How Long will Load Testing Take?

This is the 10,000 dollar question.  It is also a tricky question to answer because the exercise of setting up the load test and executing can some what be estimated, but where the timeline starts dragging out is when problems bubble up during a load test.  It is difficult to determine in the future what problems will arise and even more challenging how long it will take to correct for the defect especially if it turns out to be a bug.   If possible, add enough buffer time to provide some comfort to account for future issues.  Good luck with that.  If buffer time is added in, and you are lucky enough to have very few issues (A big dream of mine.), then that time not used for testing can be gained back and you will look like a Rock Star.  My advise is to over estimate and over deliver than to under estimate and under deliver.  I realize this is easier said than done.  

Summary

In summary JMeter is a really great load testing tool for OAM11g.  It can not only provide a way to simulate a realistic load test, but also a way to stress the system while making tweaks to tuning parameters to determine positive and negative results.  JMeter can also be used for other things beyond OAM like the LDAP, Web Service Calls, JDBC, JMS, etc.  Most importantly it is a load testing tool when used properly, will provide a way to identify issues that bubble up during the testing and hopefully fix those issues before going live.   I hope the sample plan provides a great start and you take the time to go beyond what I have provided.  Happy testing and good luck!