The AdminConsole maintains a set of references to Terracotta Clusters that you define. If no clusters are defined, a default cluster (host=localhost,jmx-port=9520) is created. You can define any number of cluster references, which are maintained across sessions. Create a new cluster reference by using the File|New cluster menu or by using the New cluster context-menu in the cluster list area. The cluster topology is determined from the server specified.

Further reading:
For more information on configuring your Terracotta server's jmx-port, see the Configuration Guide and Reference.

Connecting to a cluster

Once you have created a cluster reference you can attempt to connect to that cluster by selecting the cluster node in the navigation tree on the left side of the window. In the cluster panel, on the right side of the window, will be displayed the server connection information. Click the Connect button. If the server is active, after a short pause, the cluster elements (servers and clients) as well as cluster-wide information, such as the set of shared objects, is presented.

An alternate way to connect to a particular cluster is to bring up the context menu for that cluster node (usually a right-click operation) and select the Connect item. To automatically connect to the cluster, enable the Auto-connect item on the cluster context menu. With cluster auto-connect enabled, the AdminConsole is constantly attempting to connect to the cluster in the background, doing away with the need to explicitly connect.

Connecting to a Secured Cluster

The Terracotta Cluster can be secured for JMX access, requiring an identity and password. If the AdminConsole notices that a cluster is secured it will prompt for security credentials.

Further reading:
For more information on securing your cluster for JMX access, see the Configuration Guide and Reference.

Clustered Terracotta Servers

To avoid a single-point-of-failure Terracotta Servers can be setup in a cluster such that there is a single active and one or more passive members. The cluster can be configured such that the members either use a shared file-system or communicate server state over the network.

Shared File system-based Clusters

In the case of a cluster based on a shared file-system, passive members go into stand-by mode until a file lock held by the active members is released, at which time one of the passives takes over as the active. There are thus two states for a SAN-based cluster member:

started (yellow)
active (green)

Passive members in the started state are indicated by a yellow adornment in the AdminConsole while active members are indicated by a green adornment.

Network-based Clusters

In the case of a network-based cluster, passive members go through the following stages:

started (yellow)
passive-uninitialized (orange)
passive-standby (cyan)
active (green)

The following shows a two-member cluster with the active member named primary and the single passive member, standing-by, named secondary.

Servers that are not accessible for any reason are indicated by a red adornment.

Further reading:
For more information on setting up you cluster, see Creating a Terracotta Server Cluster.

Disconnecting from a Cluster

Once you have connected to a running cluster, you can disconnect your session by clicking the Disconnect button on the cluster panel or by selecting the Disconnect item from the cluster context menu.

The AdminConsole will automatically detect a cluster termination and disconnect.

Viewing the DSO Subsystem

Roots

Using DSO, you can specify that certain fields of your Java classes be transparently shared amongst multiple DSO-enabled applications. These shared fields in your classes are referred to as DSO roots. You can view the complete set of roots known to the server by selecting the Roots node.

In addition, each individual root is also broken out as a separate node under the Roots node. You can navigate through the root object graph and refresh the contents of the root by selecting the Refresh context menu on the root node. Simply selecting an object in the root graph will cause it's content to be refreshed.

Further reading:
For more information on roots, see the Concept and Architecture Guide

Classes

DSO allows for transparent, clustered object state synchronization. To accomplish this feature, some of your application classes will be adapted into new classes that are cluster-aware. The set of all such adapted classes known to the server are displayed in the Classes panel. The Tabular tab show all the adapted classes in a spreadsheet view, including the class name and a count of the number of instances of the class that have been created since the server started. The Tree tab shows a hierarchical, or Java package, view of the adapted classes. Finally, the TreeMap tab shows a presentation making it easy to quickly distinguish the most (and least) heavily used adapted classes. These views are a snapshots of the adapted classes known to the server. You can refresh these values by selecting the Refresh context menu on the Classes node.

Further reading:
For more information on object portability see the Concept and Architecture Guide.

Lock profiler

The Terracotta runtime system can be directed to gather statistics about the distributed locks that are realized as a result of the configuration you have created. Using these statistics can give you insight into the nature of your DSO application, letting you discover the code paths leading to highly-contented access to shared state.

The Lock Profiler panel lets you enable/disable lock statistics gathering, specify the lock code path trace depth for those statistics, and query the server to refresh the stats display.

Getting started with lock statistics:

Navigate to the Locks stats panel by clicking the Lock stats node in the main navigation tree
Enable statistics gathering by clicking the On button
Optionally set the desired trace depth (default=0)
Wait some period of time for some stats to be gathered and transmitted to the server
Click the Refresh button to update the display

The Locks Profiler panel is comprised of the Client and Server stats pages.

The client locks stats are based on the code paths in your DSO client applications that resulted in a lock being taken out. The Server page stats concern the cluster-wide nature of the distributed locks. Each lock has a corresponding identifier, the lock-id. For a named lock the lock-id is the lock name. For an autolock the lock-id is the server-generated id of the object on which that lock was taken out. An example of an autolock id is @1001. That autolock id corresponds to the shared object upon which distributed synchronization was carried out. You can use the Roots panel to view the state of shared object @1001.

A single lock-expression in the configuration can result in the creation of multiple locks by the use of wildcard patterns. A single lock may be arrived at through any number of different code paths. A trace depth of zero means that you don't care how the lock was arrived at, but by increasing the trace depth the client-side lock stats are broken out by the corresponding number of Java call-stack frames. For example, there may be 3 different call sequences that resulted in a particular lock being granted, with one of the paths rarely entered and another responsible for the majority of those lock grants. By setting the trace depth appropriately you can gain insight into the behavior of your application and how it may affect the performance of your clustered system. Note that trace stack frames will only include Java source line numbers if the code was compiled with debugging enabled, commonly done by passing the javac command the -g flag and in Ant by defining the javac task with the debug="true" attribute.

The Main Controls

Enable Lock Profiling

Use this control to turn lock profiling on or off.

Trace Depth

The trace depth control sets the number of client call-stack frames that are analyzed per lock event to record lock statistics. A depth of 0 gathers lock statistics without regard to how the lock event was arrived at.

A lock depth of 1 means that one call-stack frame will be used to disambiguate different code paths when the lock event occurred. A lock depth of 2 will use two frames, and so on. So for example, with a setting of 1, all locks will appear be recorded together, regardless of the call path, because the stack depth analyzed will always be just the method that resulted in the lock event to occur (in other words the surrounding method). However, beginning with a lock depth of 2, different call paths can be separated, since not only the surrounding method, but also the calling method, will be used to record different lock statistics.

This means that for example, when the trace depth setting is set to 1, a lock event that occurs within method Foo() will record all lock events occurring within Foo() as one single statistic, e.g. the number of lock requests, average held time and so on. But, with a trace depth setting of 2, the callers of Foo(), say Bar1() and Bar2(), will also be considered, so that a call path of Bar1() -> Foo() will be recorded separately from Bar2() -> Foo().

Refresh

Use this button to get an updated view of the statistics gathered so far.

The Data Table

Lock Statistic	Description
Times Requests	Number of times this lock was requested
Times Hopped	Times an acquired greedy lock was retracted from holding client and granted to another
Average Contenders	Average number of threads wishing to acquire this lock at the time it was requested
Average Acquire Time	Average time between lock request and grant
Average Held Time	Average time grantee held this lock
Average Nested Lock Depth	Average number of outstanding locks held by acquiring thread at grant time

Lock Element Details

The bottom portion of the Clients page displays detail on the selected lock element. The currently selected lock trace is shown on the left and the configuration element responsible for the creation of the selected lock is shown on the right.

A special note has to be made with respect to the Server lock stats. The Terracotta system employs the concept of greedy locks to help improve performance by limiting unnecessary lock hops. Once a client has been awarded a lock it is allowed to keep that lock until another client requests it. The assumption is that once a client obtains a lock it will likely want to request that same lock again shortly. So a lock is handed out to a client until such a time that a different client wants that lock. So if your cluster consisted of a single node that manipulated a single shared object repeatedly, the Server lock requests would be 1 until another client entered the cluster and began manipulating that object. It is likely that when you see Server stats that are undefined (na) it is due to the nature of greedy locks.

Distributed Garbage Collection

Objects in a DSO root object graph may become unreferenced at any time. DSO occasionally garbage collects such unreferenced objects. You can view the garbage collection statistics in the Garbage collection panel. Each time a collection occurs, that information is broadcast to the AdminConsole, so there is no need to refresh. You can view the gc iteration index, the start time of the iteration, the time elapsed during the sweep, the total number of shared objects held by the server, the number of shared objects that are candidates for collection, and finally the total number of shared objects that were actually collected.

Further reading:
For more information on configuring your Terracotta server's garbage collection, refer to the Concept and Architecture Guide.
For information on forcing the server to reclaim unused shared objects, see Terracotta Tools.

Thread dumps

The Thread dumps panel gives the ability to take a cluster-wide thread dump.

Cluster statistics recorder

The Cluster statistics recorder panel can generate recordings of selected cluster-wide statistics. This panel has controls to start, stop, view, and export recording sessions. You can use the Snapshot Visualization Tool (SVT) to view the recorded information.

For definitions of available statistics, see Cluster Statistics.

Servers

A Terracotta Cluster is comprised of one or more servers, configured to work together providing a highly-available shared heap. The set of clustered servers are displayed under the Servers node and in the Servers panel in tablular form.

Server Panel

Selecting a server node presents the server panel comprised of three tabs: Main, Environment, and Config.

The Main tab presents the server status and a list of properties such as the server host and JMX port.

The Environment tab presents the server JVM system properties.

The Config tab presents the Terracotta configuration under which the server is operating.

Each server node contains two child nodes: Runtime statistics and Thread dumps.

Runtime statistics

The Runtime statistics panel displays a set of server- and JVM-specific metrics sampled every Poll period seconds. The charts display History minutes worth of samples. Sampling begins automatically when the display is first viewed but can be started/stopped thereafter using the controls at the bottom of the panel.

Heap Usage

Shows the amount of available heap being used.

CPU Usage

Shows the CPU load.

Object Transaction Rate

The Object Transaction Rate statistic shows a measure of how many getfield/putfield calls have been invoked on shared objects. For instance, if you have defined a root that includes an int field, setting that fields value is a single transaction.

Cache Activity

The DSO server in-memory cache can be configured to hold some maximum number of objects from client root object graphs. When the number of shared objects known to the server exceeds the cache capacity, objects that have been Least-Recently-Used (LRU) are flushed from the in-memory cache to the disk cache. When a client requires an object that has been flushed to disk, that object is faulted from disk back into the in-memory cache, and the object state is transmitted to the client. The following two statistics relate to the DSO server cache activity.

Cache Miss Rate

The Cache Miss Rate statistic shows a measure of how many objects are being faulted from the disk-cache per second, as opposed to being retrieved from the server's in-memory cache.

Object Flush Rate

The Object Flush Rate statistic shows a measure of how many objects are being flushed out of the client memory. These objects are available in the server if needed at a later point in time.

Object Fault Rate

The Object Fault Rate statistic shows a measure of how many objects are being faulted into the client memory from the server.

Thread dumps

The Thread dumps panel provides a control for generating a thread dump.

Each thread dump is listed by the time it was acquired.

Clients

When a DSO client application connects to the cluster being monitored by the AdminConsole, a client node is created under the Clients node.

Selecting a client node presents the client panel comprised of three tabs: Main, Environment, Config, and Logging.

The Main tab presents of list of client properties such a host and DSO port.

The Environment tab presents the client JVM system properties.

The Config tab presents the Terracotta configuration under which the DSO client is operating.

The Logging tab presents controls corresponding the DSO client debugging section of the config. Refer to Configuration Guide and Reference for details on the various debug logging options.

When a DSO client disconnects from a server, that client node is automatically removed from the cluster hierarchy.

Update Checker

On a bi-weekly basis the AdminConsole will check, when first started, on updates to the Terracotta platform. By default, a notice informing you that an update check is about to be performed is displayed, allowing to ignore the immediate check, acknowledge and allow the check, or to disable further checking.

Should the update check be allowed, the AdminConsole will query the OpenTerracotta website (www.terracotta.org) and report on any new updates. Should the update checker feature be disabled, it can always be re-enabled via the Help|Update Checker... menu.

Appendix

Contacting Terracotta

Contact Terracotta at the following:

Web site: http://www.terracotta.org
Online forums: http://forums.terracottatech.com/forums/
Information: info@terracottatech.com

Platform Support

See Platform Support for information on which platforms are supported by Terracotta.
See the Integrations space to see the status of integrations with third-party technologies.

Copyright Information

This publication (the "Documentation") and the Terracotta software which it describes (the "Software") are protected to the maximum extent permitted under applicable law, including but not limited to, the regulations set forth in Title 17 of the United States Code, and California law. This Documentation, or any parts thereof, may not be reproduced in any form, by any method, for any purpose, without the express written consent of Terracotta. Terracotta makes no warranty, either express or implied, including but not limited to any implied warranties of merchantability or fitness for a particular purpose, with respect to the Software discussed in this Documentation, and the Documentation itself (collectively, "the Materials"). The Materials are made available solely on an "as-is" basis. In no event shall Terracotta be liable to anyone for special, collateral, incidental, indirect, punitive, exemplary, or consequential damages in connection with, or arising from the purchase or use of, the Materials. Under no circumstances and regardless of the cause of action alleged, shall Terracotta's liability exceed the purchase price of the Software described herein. Terracotta reserves the right to revise and improve its Software and Documentation as it deems fit. The Documentation describes the state of the Software at the time of publication.

Trademarks
"Terracotta," the stylized "T" logo, and "Open Terracotta" are trademarks of Terracotta. All other brand names, product names, or trademarks belong to their respective holders. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respective holders.

Government Use
Use, duplication, or disclosure by the U.S. Government is subject to restrictions as set forth in FAR 12.212 (Commercial Computer Software-Restricted Rights) and DFAR 267.7202 (Rights in Technical Data and Computer Software), as applicable.