Automated Policy Synchronization (APS) for OAM Clone Environment

Introduction

Since the introduction of MDC support in OAM 11g, Customers have been asking for Automated Synchronization between Master and Clone OAM Environments. It is supported in OAM R2PS2. Thanks to the development team! Before R2PS2, It required T2P process to keep all the data centers in synch which is manual process or customer had to write crone jobs to run T2P process at frequent intervals. Please note that T2P process is still supported with R2PS2 if that is the preference for some reason.

How does APS function?

APS is client side pull model. Each client (clone environment) pulls changes from the server (Master environment). A Replication thread runs after every pre-configured time Interval on clone environment and fetches changes from replication service (REST Service) running on master environment. For every cloned environment, Master environment keeps track of a change sequence number which indicates when cloned environment was last synched. Master environment also keeps track of change log which is list of changes (Create/Update/Delete) that have happened in the environment. Along with that, Master environment keeps track of mapping between cloned environment specific change sequence number and change log. Cloned environment while updating configuration can transform configuration parameter values to environment specific values depending on transformation rules. This is very interesting and determines success/failure of APS in an environment. Unfortunately it is not well documented. No worries. I will talk about transformation rules in little more details in this blog.  

Step-by-step process to configure APS

 

• While setting up clone environment, make sure latest configuration is copied from master environment to clone environment. That is running copyConfig.sh to generate new oamt2pConfig.jar file  every time (As documented in MDC Implementation blog post by Vinay Kalra from A Team) you start setting up a new clone environment instead of using old oamt2pConfig.jar.

• For APS to work, partnerInfo.properties file used for MDC configuration (While running addPartnerForMultiDataCentre WLST command) should have an additional REST endpoint property defined as described below where NYC-CLUSTER (Master/Primary environment) and LON-CLUSTER (Clone environment) are two cluster IDs. You may already have run addPartnerForMultiDataCentre before; however you need to run that again with RESTEndpoint parameter if that was not added in partnerInfo.properties file when you had last run that. You need to run addPartnerForMultiDataCentre commands as below on both Master and Clone environments.

[addPartnerForMultiDataCentre(propfile="ABSOLUTE_PATH_OF_PrimaryPartnerInfo.properties") addPartnerForMultiDataCentre(propfile="ABSOLUTE_PATH_OF_SecondaryPartnerInfo.properties")]

PrimaryPartnerInfo.properties for NYC-CLUSTER (Master environment)	SecondaryPartnerInfo.properties for LON-CLUSTER (Clone environment)
remoteDataCentreClusterId=NYC-CLUSTER oamMdcAgentId=MDC-NYC PrimaryHostPort=oam1-nyc.oracle.com:5575 SecondaryHostPort=oam2-nyc.oracle.com:5575 AccessClientPasswd=****** oamMdcSecurityMode=Open agentVersion=11g RESTEndpoint=http://oam1-nyc.oracle.com:7001	remoteDataCentreClusterId=LON-CLUSTER oamMdcAgentId=MDC-LON PrimaryHostPort=oam1-lon.oracle.com:5575 SecondaryHostPort=oam2-lon.oracle.com:5575 AccessClientPasswd=****** oamMdcSecurityMode=Open agentVersion=11g RESTEndpoint=http://oam1-lon.oracle.com:7001

• Make sure Master data center and clone data centers are marked as Master and Clone data centers respectively by running following WLST commands.

setMultiDataCenterType(DataCenterType="Master") on Master data center Admin server setMultiDataCenterType(DataCenterType="Clone") on Clone data center Admin server

• Now you have to add "-Doracle.oam.EnableMDCReplication=true" Java Property in setDomainEnv.sh file on both master and clone environments as shown below to enable APS service. After adding that Java property, restart admin servers

• After enabling APS service, execute following curl command to test if APS is enabled on both the data centers. If it is enabled, you will see {"ok":"true"} JSON response.

[oracle@oam1-nyc bin]$ curl -u weblogic 'http://oam1-nyc.oracle.com:7001/oam/services/rest/_replication/hello' Enter host password for user 'weblogic': {"ok":"true"} [oracle@oam1-lon bin]$ curl -u weblogic 'http://oam1-lon.oracle.com:7001/oam/services/rest/_replication/hello' Enter host password for user 'weblogic': {"ok":"true"}  

• Now you need to setup replication agreement between master and clone environment by running following curl command. Service responds back with JSON object that has identifier for the replication agreement. Save that identifier because you have to use that identifier to update or delete replication agreement.

Request: curl -u weblogic:****** -H 'Content-Type: application/json' -X POST 'http://oam1-nyc.oracle.com:7001/oam/services/rest/_replication/setup' -d  '{"name":"NYCtoLON", "source":"NYC-CLUSTER","target":"LON-CLUSTER","documentType":"ENTITY"}' Response: {"enabled":"true","identifier":"201409040157218184","ok":"true","pollInterval":"900","startingSequenceNumber":"101","state":"READY"}

I have highlighted some of the important parameters from JSON response.

First one is identifier. As mentioned above, it is a reference to replication agreement setup between NYC-CLUSTER and LON-CLUSTER data centers.

Second is polling Interval. Clone data center will poll for changes against master data center after every polling interval. Typically policy and configuration changes are not done very often and so this number can be as high as 900 seconds as setup by default by OAM. However you can change that if needed as we will do in next step.

Third one is startingSequenceNumber. That is the change sequence number that I talked about earlier in the blog post. Initial agreed upon sequence number is 101 in this example. Clone data center will start polling for changes that have happened after 101 sequence number. To understand more about sequence number and how it is used, refer to sequence diagram below.

• Now using replication agreement identifier number, you can make changes in the replication agreement configuration. Before doing that, Run first command as below to get current status of replication agreement. As shown below, we are changing poll Interval to 60 seconds. Service responds back with JSON object that is the status of replication agreement before making the change. You need to fetch replication agreement status again to see updated configuration.

 Fetching Status of Replication Agreement

Request: curl -u weblogic:****** -H 'Content-Type: application/json'  'http://oam1-nyc.oracle.com:7001/oam/services/rest/_replication/201409040157218184'

JSON Response: {"enabled":"true","identifier":"201409040157218184","ok":"true","pollInterval":"3600","startingSequenceNumber":"101","state":"ACTIVE"}

 Updating poll interval to 60 seconds

Request: curl -u weblogic:****** -H 'Content-Type: application/json'  -X PUT 'http://oam1-nyc.oracle.com:7001/oam/services/rest/_replication/201409040157218184' -d '{"pollInterval":"60","replicaType":"consumer"}'

JSON Response: {"enabled":"true","identifier":"201409040157218184","ok":"true","pollInterval":"3600","startingSequenceNumber":"101","state":"ACTIVE"}

 Fetching Status if Replication Agreement again to validate change

Request: curl -u weblogic:****** -H 'Content-Type: application/json'  'http://oam1-nyc.oracle.com:7001/oam/services/rest/_replication/201409040157218184'

JSON Response: {"enabled":"true","identifier":"201409040157218184","ok":"true","pollInterval":"60","startingSequenceNumber":"101","state":"ACTIVE"}

• After setting up replication agreement, restart admin servers on both master and clone data centers.

Transformation Rules

By default APS uses following transformation rules that you can change if you need to.

Out-Of-The box transformation rules

<?xml version="1.0" encoding="UTF-8"?>
<mdc-transform-rule>
     <changes-to-include entity-path="/DeployedComponent/Agent/WebGate/Instance">
         <replace attribute-match="/*/PrimaryServerList/*/host" value-match="(.*)">
             <replace-with n="1">${DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile/OAMSERVER/serverhost}</replace-with>
         </replace>
         <replace attribute-match="/*/logoutRedirectUrl" value-match="(.*)://(.*):(.*)/oam/server/logout">
             <replace-with n="1">${DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile/OAMSERVER/serverprotocol}</replace-with>
             <replace-with n="2">${DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile/OAMSERVER/serverhost}</replace-with>
             <replace-with n="3">${DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile/OAMSERVER/serverport}</replace-with>
         </replace>
     </changes-to-include>
     <changes-to-include entity-path="/DeployedComponent/Server/NGAMServer/Profile/AuthenticationModules"/>
     <changes-to-include entity-path="/DeployedComponent/Server/NGAMServer/Profile/oamproxy"/>
     <changes-to-include entity-path="/DeployedComponent/Server/NGAMServer/Profile/Sme/SessionConfigurations"/>
 </mdc-transform-rule>

Above transformation rules make couple of changes in WebGate definitions. Let's talk about each one of them.

WebGate ServerList transformation

<replace attribute-match="/*/PrimaryServerList/*/host" value-match="(.*)">
     <replace-with n="1">${DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile/OAMSERVER/serverhost}
     </replace-with>
 </replace>

It updates PrimaryServerList for all the WebGates and replaces them with OAM server host from Clone environment. The best way to understand transformation rule is to open oam-config.xml file in web browser and browse through the tree. Above mentioned transformation rule replaces PrimaryServerList with ${DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile/OAMSERVER/serverhost} value. If I browse that value in oam-config.xml, I get oam1-lon.oracle.com value as shown in the screen below so APS thread will update PrimaryServerList host for every webgate agent to oam1-lon.oracle.com.  

The drawback or limitation of this transformation rule is; it updates all the servers/hosts in the PrimaryList with oam1-lon.oracle.com. You can update transformation rule as below so that it updates servers in PrimaryServerList with respective servers from Cloned data center. We are updating 0th PrimaryServerList (oam1-nyc.oracle.com) with oam_server1 host (oam1-lon.oracle.com) from clone environment and 1st PrimaryServerList (oam2-nyc.oracle.com) with oam_server2 host (oam2-lon.oracle.com) from clone environment as shown and highlighted in the screen below. 

Note: This transformation rule assumes that you have two OAM server instances in both Primary and Cloned environment. Similar to this transformation rule, you can choose to update SecondaryServerList as well for all the WebGates. 

<changes-to-include entity-path="/DeployedComponent/Agent/WebGate/Instance">
     <replace attribute-match="/*/PrimaryServerList/0/host" value-match="(.*)">
         <replace-with n="1">${DeployedComponent/Server/NGAMServer/Instance/oam_server1/host}
         </replace-with>
     </replace>
     <replace attribute-match="/*/PrimaryServerList/1/host" value-match="(.*)">
         <replace-with n="1">${DeployedComponent/Server/NGAMServer/Instance/oam_server2/host}
         </replace-with>
     </replace>
 </changes-to-include>

It is recommended to use Load Balancer (LB) between WebGate and OAM server. If you are using it right, you do not have to update PrimaryServerList across data centers and so can remove this transformation rule from the XML. However you do need to update PrimaryServerList for IAMSuiteAgent and accessgate-oic unless you have configured those agents as well to talk to LB. Below change will make sure that PrimaryServerList is updated only for IAMSuiteAgent and accessgate-oic and not updated for any other webgate agent. 

<changes-to-include entity-path="/DeployedComponent/Agent/WebGate/Instance">
     <replace attribute-match="/IAMSuiteAgent/PrimaryServerList/*/host" value-match="(.*)">         <replace-with n="1">${DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile/OAMSERVER/serverhost}
         </replace-with>
     </replace>
     <replace attribute-match="/accessgate-oic/PrimaryServerList/*/host" value-match="(.*)">         <replace-with n="1">${DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile/OAMSERVER/serverhost}
         </replace-with>
     </replace>
</changes-to-include>

WebGate logout URL transformation rule

Second change in the OOB transformation rule is logout URL. For all the WebGate agents, as shown below, it updates logout URL protocol, host and port with respective values from clone environment. 

<replace attribute-match="/*/logoutRedirectUrl" value-match="(.*)://(.*):(.*)/oam/server/logout">     <replace-with n="1">${DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile/OAMSERVER/serverprotocol}
     </replace-with>
     <replace-with n="2">${DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile/OAMSERVER/serverhost}
     </replace-with>
     <replace-with n="3">${DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile/OAMSERVER/serverport}
     </replace-with>
</replace>

If you used global LB to define logout URL for all the WebGate agents in master environment, then you don't need to replace logoutURL in clone environment and can remove this transformation rule. If you are using DCC authentication scheme and using global LB hostname to define DCC login and logout URL, then also you don't need to replace login and logout URL in cloned environment and can remove the transformation rule. Now that you understand how to write custom transformation rules, it’s time to configure OAM to use custom transformation rules which unfortunately is not documented. You need to update setDomainEnv.xml file as shown in the below screen shot on clone environment. Each clone environment can use different transformation rules. Restart admin server in clone environment after changing transformation rule.

APS as I said in the beginning was a long pending feature and I hope this blog will help you setup MDC environment with APS. If you create environment specific transformation rules, please share it here so that everyone will benefit from it.