Release: 2.6.2 Previous Releases Publish Date: July, 2008

rate-13500851-63985 Article Rating? 1 2 3 4 5

TODO: http://jira.terracotta.org/jira/browse/DOC-178

Introduction

The Terracotta server publishes a number of MBeans to provide information on the operation of the cluster. The Terracotta Console uses the JMX interface to provide real-time information, but the same interface can be used by any JMX client. As of Terracotta 2.3, there is also a new set of MBeans to report cluster events such as client nodes joining or leaving the cluster.

JMX Authentication

It is possible to configure the Terracotta Server to require authentication credentials before JMX communications take place. This will secure server shutdown, the Admin Console, JConsole, and any other processes attempting to connect via JMX. Server authentication will not provide secure message transmission once valid credentials are provided by a listening client. The JMX messages will not be encrypted. For this reason, we recommend that the Terracotta server be placed in a secure location on a private network and only be queried remotely via an encrypted tunnel such as those provided by stunnel or SSH.

Configuring JMX authentication with Terracotta requires that an authentication element be placed in the Terracotta configuration XML file, read by the Terracotta server instance that requires security. Terracotta relies on the standard Java security mechanisms for JMX which involve the creation of a .access and .password file with correct permissions set. The default location for these files is in (for JDK 1.5+) $JAVA_HOME/jre/lib/management.

Once configured properly, the Admin Console connection dialog will prompt you for credentials before completing a server connection. The stop-tc-server script located in the bin directory will also prompt for a username and password.

Configuration

See the Configuration Guide and Reference for information on configuring the Terracotta server to use JMX authentication.

Permissions

Setting permissions is done with the files jmxremote.password and jmxremote.access. Java security is very sensitive to the permission settings of these files so you will need to verify that they are properly set. One caveat to pay attention to is that the files must be owned by the system user who will be executing the Terracotta server. If the Terracotta server runs under the user "tcuser" then you must chown tcuser to change the ownership. The password file is most sensitive and requires:

# chmod 500 jmxremote.password 

-------------------------------------------------------------------------------
-rw-r--r-- 1 tcuser tcuser 2375 Feb 12 18:19 jmxremote.access
-r-x------ 1 tcuser tcuser 2873 Feb 12 18:19 jmxremote.password
-------------------------------------------------------------------------------

NOTE: Setting these permissions on Windows is possible but requires a special DOS command. This permission is only available on Windows drives with NTFS formatting. See http://java.sun.com/j2se/1.5.0/docs/guide/management/security-windows.html for more information.

Users/Groups

You may append to the default files located in $JAVA_HOME/jre/lib/management using the following commands:

# echo "myusername mypassword" >> jmxremote.password
# echo "myusername readwrite" >> jmxremote.access

'myusername' defines the desired username and 'mypassword' defines the desired password. In jmxremote.access, myusername is associated with the username defined in jmxremote.password and must declare readwrite permissions for that group.

...
# Following are two commented-out entries. The "measureRole" role has
# password "QED". The "controlRole" role has password "R&D".
#
# monitorRole QED
# controlRole R&D
myusername mypassword
...

...
# Default access control entries:
# o The "monitorRole" role has readonly access.
# o The "controlRole" role has readwrite access.

monitorRole readonly
controlRole readwrite
myusername readwrite
...

Cluster Events and Cluster Node Data

The Terracotta Cluster Events JMX interface lets a Terracotta client listen for cluster events such as client connect and disconnect from the server and other clients connecting and disconnecting from the cluster.

As of Terracotta 2.3, when a Terracotta client connects to the cluster, it publishes a local MBean to the server called "Terracotta Cluster Bean". You can register with that bean to listen to cluster events. The published events are:

• com.tc.cluster.event.thisNodeConnected—The local client (the current JVM) connecting to a Terracotta server
• com.tc.cluster.event.thisNodeDisconnected—The local client disconnecting from a Terracotta server
• com.tc.cluster.event.nodeConnected—Another client connecting to a Terracotta server
• com.tc.cluster.event.nodeDisconnected—Another client disconnecting to a Terracotta server

You can also query the MBean for the local client's cluster node id (which is a unique id given to each client when it connects to the cluster for the first time) and for an enumeration of the node id's of each currently connected client.

Using this data, you can keep a local model of the Terracotta cluster. You can also find out if the local client has been disconnected from the cluster and take any further action.

Here's some sample code that will locate the local MBeanServer and register for cluster events:

// get a reference to the local MBeanServer...
  final List servers = MBeanServerFactory.findMBeanServer(null);
  final MBeanServer server = (MBeanServer) servers.get(0);

  // Use these strings as the delegate name and cluster bean name
  final ObjectName delegateName = ObjectName.getInstance("JMImplementation:type=MBeanServerDelegate");
  final ObjectName clusterBeanName = new ObjectName("org.terracotta:type=Terracotta Cluster,name=Terracotta Cluster Bean");

  // Before you register a cluster bean listener, you have to wait for that cluster bean to be registered.
  // For an example of how to do that, see the ClusterBeanRegistrationListener in the tc-jmx-examples code
  waitForClusterBeanRegistration();

  // Now that we're sure the cluster bean has been registered, we can add listeners.
  // 
  // A cluster event listener should implement the interface:
  //
  //   javax.management.NotificationListener
  //
  // 
  // The callback method you implement is:
  //
  //    public void handleNotification(Notification notification, Object handback)
  //
  //  In that method, you can examine the type of the Notification object:
  //
  //    notification.getType();
  //
  //  Check that type to see if it is one of the four cluster event types described above:
  //
  //    com.tc.cluster.event.thisNodeConnected
  //    com.tc.cluster.event.thisNodeDisconnected
  //    com.tc.cluster.event.nodeConnected
  //    com.tc.cluster.event.nodeDisconnected
  //
  server.addNotificationListener(clusterBeanName, myClusterEventListener, null, null);
			
  // Now that the cluster bean is registered, we can find out what our node id is
  System.err.println("My cluster node id is: " + server.getAttribute(clusterBeanName, "NodeId"));

  // and I can get a snapshot of all the connected clients (including myself).
  String[] clusterNodeIds = (String[]) server.getAttribute(clusterBeanName, "NodesInCluster");
  for (int i=0; i<clusterNodeIds.length; i++) {
    System.err.println("Cluster node id: " + clusterNodeIds[i]);
  }

To see a sample usage of the Terracotta Cluster Events interface, check out the jmx-sample (http://svn.terracotta.org/svn/forge/projects/jmx-sample) project from the Terracotta forge.

Then, start the Terracotta server. Next, run the demo.ClusterEvents class (in Eclipse as a Java Application or from the command line) with these VM arguments:

Note:

• Change /path/to/your/terracotta/installation to wherever you installed Terracotta
• Change dso-boot-hotspot_osx_150_07.jar to the appropriate boot jar for your platform
• You can also run dso-env.sh or dso-env.bat to determine the proper values for these properties
• The property -Dcom.sun.management.jmxremote lets you use jconsole to inspect the published MBeans.

You should see something like this:

You can then start another Terracotta client (one of the samples that comes with the Terracotta kit will work) and the demo.ClusterEvents program will print the client connect event to the Eclipse console. If you quit that client, the demo.ClusterEvents program will print the client disconnect event to the console:

If you stop the Terracotta server, the demo.ClusterEvents program will print a notification that this node has been disconnected:

If the Terracotta server is configured to be persistent, you can start the server up again and the demo.ClusterEvents program will print a notification that this node has been connected:

For another example of the Terracotta Cluster Events interface in use, see the chatter sample in the Terracotta kit.

Further reading:
For more information on the Terracotta cluster architecture, see the Concept and Architecture Guide

Distributed Garbage Collection

TODO: Write me.
http://jira.terracotta.org/jira/browse/DOC-139

Rogue/Slow/Unhealthy Client Detection and monitoring

Rogue Client is a client which is for some reason not healthy e.g its having a lot of pending transactions or its slow.Terracotta server exposes a list of MBeans to monitor the performance of these clients (like pending transactions) so that one can determine the health of a client and can kill any particular unhealthy client.

Here is some sample code for getting client MBeans

MBeanServerConnection mbsc = null;
JMXConnectorProxy jmxc = new JMXConnectorProxy("localhost", Integer.valueOf(appConfig.getAttribute(JMX_PORT)));
try {
  mbsc = jmxc.getMBeanServerConnection();
} catch (IOException e) {
  throw new AssertionError(e);
}
DSOMBean dsoMBean = (DSOMBean) MBeanServerInvocationHandler
          .newProxyInstance(mbsc, L2MBeanNames.DSO, DSOMBean.class, false);

ObjectName[] clientObjectNames = dsoMBean.getClients();
DSOClientMBean[] clients = new DSOClientMBean[clientObjectNames.length];
for (int i = 0; i < clients.length; i++) {
  clients[i] = (DSOClientMBean) MBeanServerInvocationHandler.newProxyInstance(mbsc, clientObjectNames[i],DSOClientMBean.class, false);
}

These are the list of MBeans exposed to monitor these clients. The name of the APIs are self explanatory

• com.tc.stats.isTunneledBeansRegistered—Is the bean registered
• com.tc.stats.getL1InfoBeanName—Get the name(class ObjectName) of this bean
• com.tc.stats.getL1InfoBean—Get the MBean
• com.tc.stats.getInstrumentationLoggingBeanName—Get the name(class ObjectName) of the instrumentation logging MBean)
• com.tc.stats.getInstrumentationLoggingBean—Get the Instrumentation logging MBean
• com.tc.stats.getRuntimeLoggingBeanName—Get the name(Class ObjectName) of te runtime logging MBean
• com.tc.stats.getRuntimeLoggingBean—Get the run time logging MBean
• com.tc.stats.getRuntimeOutputOptionsBeanName—Get the name(class ObjectName) of the runtime output options MBean
• com.tc.stats.getRuntimeOutputOptionsBean—Get the runtime output options MBean
• com.tc.stats.getChannelID—Get the channel ID
• com.tc.stats.getRemoteAddress—Get the remote address of the channel
• com.tc.stats.getTransactionRate—Get the transaction rate for the client
• com.tc.stats.getObjectFaultRate—Get the object fault rate for the client
• com.tc.stats.getObjectFlushRate—Get the object flush rate for the client
• com.tc.stats.getPendingTransactionsCount—Get the pending transactions count for the client
• com.tc.stats.getStatistics—Get the statistics for the given properties
• com.tc.stats.killClient—Kill this client

  ---

  TODO: Document the rest of the MBeans.

Appendix

Contacting Terracotta

Web site:  http://www.terracotta.org
Online forums:  http://forums.terracottatech.com/forums/
Information:  info@terracottatech.com

Platform Support

See Platform Support for information on which platforms are supported by Terracotta.
See the Integrations space to see the status of integrations with third-party technologies.

Copyright Information

Copyright © 2005-2007
Terracotta, Inc.
All Rights Reserved

This publication (the "Documentation") and the Terracotta software which it describes (the "Software") are protected to the maximum extent permitted under applicable law, including but not limited to, the regulations set forth in Title 17 of the United States Code, and California law. This Documentation, or any parts thereof, may not be reproduced in any form, by any method, for any purpose, without the express written consent of Terracotta. Terracotta makes no warranty, either express or implied, including but not limited to any implied warranties of merchantability or fitness for a particular purpose, with respect to the Software discussed in this Documentation, and the Documentation itself (collectively, "the Materials"). The Materials are made available solely on an "as-is" basis. In no event shall Terracotta be liable to anyone for special, collateral, incidental, indirect, punitive, exemplary, or consequential damages in connection with, or arising from the purchase or use of, the Materials. Under no circumstances and regardless of the cause of action alleged, shall Terracotta's liability exceed the purchase price of the Software described herein. Terracotta reserves the right to revise and improve its Software and Documentation as it deems fit. The Documentation describes the state of the Software at the time of publication.

Trademarks
"Terracotta," the stylized "T" logo, and "Open Terracotta" are trademarks of Terracotta. All other brand names, product names, or trademarks belong to their respective holders. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respective holders.

Government Use
Use, duplication, or disclosure by the U.S. Government is subject to restrictions as set forth in FAR 12.212 (Commercial Computer Software-Restricted Rights) and DFAR 267.7202 (Rights in Technical Data and Computer Software), as applicable.