JGroups Cluster Groups

The JGroups communications module provides communications among groups that share a common name, multicast address, and multicast port. JGroups is installed with JBoss, but it can also be used without JBoss. The User Application includes a JGroups module in the User Application WAR to support caching in a cluster environment.

JBoss Cluster

JBoss clusters are implemented by JBoss using the JGroups communications module. The configuration of JGroups and session replication is defined by JBoss and depends on the version of JBoss you are using. See the JBoss Administration Guide for more detailed information on configuring JBoss clusters.

User Application Cluster Group

The Identity Manager User Application uses an additional cluster group solely to coordinate User Application caches in a clustered environment on either JBoss or WebSphere clusters.

The User Application cluster group is independent of the two JBoss cluster groups and does not interact with them. By default, the User Application cluster group and the two JBoss groups use different group names, multicast addresses, and multicast ports, so no reconfiguration is necessary.

By default, this cluster group uses a UUID name to minimize the risk of conflicts with other cluster groups that users might add to their servers. The default name is c373e901aba5e8ee9966444553544200. By default, the group uses multicast address 228.8.8.8 and runs on port 45654. This cluster isn't configured using a JBoss service file. Instead, the configuration settings are located in the directory and can be configured using the User Application administration features. If you are familiar with JGroups and JBoss clustering, you can adjust the User Application cluster configuration using this interface. Changes to the cluster configuration only take effect for a server node when that node is restarted.

User Application cluster group settings are shared by any Identity Manager application that shares the directory configuration. The purpose of the local settings option in the User Application administration interface is to allow an administrator to remove a node from a cluster, or change the membership of servers in a cluster. For example, you can disable clustering globally, then enable it locally for a subset of your servers sharing the directory configuration.

About Multiple Clusters on the Same Network

If you have more than one cluster running on a network, you must separate the clusters to prevent performance problems and anomalous behavior. You accomplish this by ensuring that each cluster uses a different partition name, multicast address, and multicast port. Even if you are not running multiple clusters on the same network, it’s a good idea to specify a unique partition name for the cluster, rather than using the default partition.

The following are important points:

For JBoss, you can find instructions about running more than one cluster on a network by using your browser to view Two Clusters Same Network.

Synchronizing Application Server Clocks

You must synchronize the clocks of the servers in a User Application cluster. If server clocks are not synchronized, sessions might time out early, causing HTTP session failover to not work properly. There are many time synchronization methods available. The method that you use depends on the needs of your organization. One common approach is to use the Network Time Protocol (NTP). For a discussion of using the xNTP protocol for time synchronization, see Time Synchronization using Extended Network Time Protocol (xntp).

Avoiding Multiple Browser Logins from the Same Browser Window in a Cluster

We do not recommend using multiple logins across browser tabs or browser sessions on the same host. Some browsers share cookies across tabs and processes, so using multiple logins might cause problems with HTTP session failover (in addition to risking unexpected authentication functionality if multiple users share a computer).

About the User Application Database

When you install the User Application using the User Application installation program, you designate an existing version of a supported database to use (for example, MySQL, Oracle or Microsoft SQL Server). The database is used to store User Application data and User Application configuration information.

When the User Application is installed in a cluster environment, all nodes in the JBoss cluster must access the same database instance. The User Application uses standard JDBC calls to access and update the database. The User Application uses a JDBC data source bound to the JNDI tree to open a connection to the database.

When you install the User Application into a JBoss cluster by using the User Application installation program, the data source is installed for you. The installation program creates a data source file named IDM-ds.xml, and places this file in the deploy directory (for example, server/IDM/deploy). The installation program also places the appropriate JDBC driver for the database specified during installation in the lib directory (for example, /server/IDM/lib). For more information about setting up the User Application database for a cluster, see Specifying the User Application Database.

(java.sql.SQLException: Data source rejected establishment of connection, message from server: "Too many connections.")

About the Server Configuration

JBoss comes with three different ready-to-use server configurations: minimal, default and all. Clustering is only enabled in the all configuration. A cluster-service.xml file in the /deploy folder describes the configuration for the default cluster partition. When you install the User Application and indicate to the installation program that you want to install into a cluster, the installation program makes a copy of the all configuration, names the copy IDM (this is the default; the installation program allows you to change the name), and installs the User Application into the this configuration.

Specifying the User Application Database

All nodes in the JBoss cluster must access the same database instance. When you use the User Application installation program, you are prompted to specify the database name, host and port:

Figure 2-3 Specifying the Database Host and Port

Make sure that you specify the same database parameters each time you install the User Application to a cluster node.

Selecting the Cluster (all) Option

When you use the User Application installation program, you are prompted to specify the IDM configuration:

Figure 2-4 Specifying the Cluster (all) Option and Engine ID

Select the clustering (all) option.

Configuring the Workflow Engine

Workflow engine clustering works independently of the User Application cache framework. There are several steps that you must perform to ensure that the workflow engine works correctly in a cluster environment.

You might want to combine the setting of this system property with the setting of other system properties (see Setting JBoss system properties in the JBoss startup script).

For information about managing running workflows, see Managing Workflows in a Cluster.

start run.bat -c IDM -Djboss.partition.name=Example_Partition -Djboss.partition.udpGroup=228.3.2.1 -Dcom.novell.afw.wf.engine-id=Engine1 

Using the Same Master Key for Each User Application in the Cluster

The Identity Manager User Application encrypts sensitive data (see Section 2.2.7, Encryption of Sensitive User Application Data). A master key is used to access encrypted data. All User Applications in a cluster must use the same master key. Follow these steps to ensure that all User Applications in a cluster use the same master key.

Starting the User Application Cluster Group

After the User Applications in your cluster have been installed, you must enable the cluster in the User Application cluster configuration.

Configuring the User Application Driver for Clustering

Clustering is the only scenario in which the same User Application driver is used by multiple User Applications. The User Application driver stores various kinds of information (such as workflow configuration and cluster information) that is application-specific. Therefore, a single instance of the User Application driver should be not shared among multiple applications.

The User Application stores application-specific data to control and configure the application environment. This includes JBoss application server cluster information and the workflow engine configuration. The only User Applications that should share a single User Application driver instance are those applications that are part of the same JBoss cluster.

In a cluster, the User Application driver must be configured to use the host name or IP address of the dispatcher or load balancer for the cluster. You create the User Application driver when you install the User Application (see the Roles Based Provisioning Module Installation Guide). You configure the User Application driver using iManager.

Specifying the User Application Cluster Group Caching Configuration

Users who are familiar with JGroups and JBoss clustering can modify the cluster group caching configuration, using the User Application administration user interface (see Cache Settings for Clusters). Changes to the cluster configuration only take effect for a server node when the server node is restarted.

In most cases you should use global settings when configuring a cluster. However, global settings present a problem if you need to use TCP, because the IP address of the server must be specified in the JGroups initialization string for each server. You can use local settings to specify a JGroups initialization string by checking Enable Local for Cluster Properties, then typing the JGroups initialization string in the Local field. For an example of a working JGroups TCP protocol stack, see JGroupsStackTCP.

Alternatively, you can add a token (for example, “IDM_HOST_ADDR”) to the global settings for the Cluster Properties. You can then edit the hosts file on each server in the cluster to specify the IP address for that server.

Configuring Logging in a Cluster

This section includes tips for configuring logging in a cluster. No tips are included for WebSphere or WebLogic.

<!--  Clustering logging 
  --> 
- <!-- 
 Uncomment the following to redirect the org.jgroups and
      org.jboss.ha categories to a cluster.log file.
   <appender name="CLUSTER" class="org.jboss.logging.appender.RollingFileAppender">
     <errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/>
     <param name="File" value="${jboss.server.home.dir}/log cluster.log"/>
     <param name="Append" value="false"/>
     <param name="MaxFileSize" value="500KB"/>
     <param name="MaxBackupIndex" value="1"/>
     <layout class="org.apache.log4j.PatternLayout">
       <param name="ConversionPattern" value="%d %-5p [%c] %m%n"/>
     </layout>
   </appender>
   <category name="org.jgroups">
     <priority value="DEBUG" />
     <appender-ref ref="CLUSTER"/>
   </category>
   <category name="org.jboss.ha">
     <priority value="DEBUG" />
     <appender-ref ref="CLUSTER"/>
   </category>
  --> 

Managing Workflows in a Cluster

The Identity Manager User Application workflow cluster implementation binds process instances to the engine on which they started. This is done by associating a workflow process instance with an engine-id and is maintained in the cluster database. When a workflow engine is started, it resumes process instances that are assigned to its engine-id. This prevents multiple engines in a cluster from resuming the same process instance. If a workflow engine fails, processes that were running on that engine are automatically resumed on another engine in the cluster.

You can manually reassign processes to other engines in the cluster. For example, an administrator could reassign processes back to a failed workflow engine when the workflow engine is brought back online, or redistribute processes to other engines when an engine is permanently removed from the cluster (see Section 17.2.7, Managing Workflow Processes in a Cluster).

When the workflow engine starts up it checks to see if its engine ID is already in use by another node in the cluster. When this is the case, the workflow engine checks the cluster database to see if the status of the engine is SHUTDOWN or TIMEDOUT. If it is, the workflow engine starts. If the status is STARTING or RUNNING, the workflow engine logs a warning, then waits for a heartbeat time out to occur. If the heartbeat time out occurs, that means that the other workflow engine with the same ID was not shut down properly, so it's safe to start. If the heartbeat timer is updated, that means another workflow engine with the same ID is running in the cluster, so the workflow engine cannot start. You can specify the heartbeat time out (the maximum elapsed time between heartbeats before a workflow engine is considered timed out) by setting the Heartbeat Interval and Heartbeat Factor properties in the User Application (see Configuring the Workflow Cluster).

Checking the Health of the Server

Most loadbalancers or dispatchers provide a healthcheck feature for determining whether an HTTP server is up and listening. The User Application contains a blank page that can be used for configuring HTTP healthchecks on your loadbalancer. The page can be addressed at this URL:

http://<HOSTNAME>/<CONTEXT>/jsps/healthcheck.jsp