Terracotta Logo (http://www.terracotta.org)

Terracotta Documentation Home

Terracotta 3.2.1 Documentation

Table of Contents •  Back  •  Forward

Terracotta 3.2.1

Support

Services

Training

Contact

Register

1.2 Distributed Ehcache Reference

This document contains reference information on the Terracotta Distributed Cache.

1.2.1 Ehcache Configuration File

The Terracotta Distributed Ehcache configuration file ( ehcache.xml by default) contains the configuration for one instance of a CacheManager (the Ehcache class managing a set of defined caches). This configuration file must be in your application's classpath to be found. When using a WAR file, ehcache.xml should be copied to WEB-INF/classes .

TIP: Naming the CacheManager
If you employ more than one Ehcache configuration files, use the name attribute in the <ehcache> element to identify specific CacheManagers in the cluster. The Terracotta Developer Console provides a menu listing these names, allowing you to choose the CacheManager you want to view.

1.2.1a Terracotta Clustering Configuration Elements

Certain elements in the Ehcache configuration control the clustering of caches with Terracotta.

<terracotta>

This element is an optional sub-element of <cache>. It can be set differently for each <cache> defined in ehcache.xml .

<terracotta> has the following attributes:

  • clustered – Enables ("true" DEFAULT) or disables ("false") Terracotta clustering of a specific cache. Clustering is enabled if this attribute is not specified.
  • valueMode – Sets the type of cache to serialization (DEFAULT, the standard Ehcache "copy" cache) or identity (Terracotta object identity). Identity mode is not available with the express installation . Identity mode can be used only with a custom Terracotta installation (see Express Versus Custom Installations).

    • TIP: Comparing Serialization and Identity Modes
    In serialization mode, getting an element from the cache gets a copy of that element. Changes made to that copy do not affect any other copies of the same element or the value in the cache. Putting the element in the cache overwrites the existing value. This type of cache provides high performance with small, read-only data sets. Large data sets with high traffic, or caches with very large elements, can suffer performance degradation because this type of cache serializes clustered objects. In read-write data sets, data coherency issues may arise because this type of cache cannot guarantee a consistent view of object values. Objects clustered in this mode _must be_ serializable.
    In identity mode, getting an element from the cache gets a reference to that element. Changes made to the referenced element updates the element on every node on which it exists (or a reference to it exists) as well as updating the value in the cache. Putting the element in the cache does not overwrite the existing value. This mode guarantees data coherence. It can be used only with a custom Terracotta Distributed Cache installation. Objects clustered in this mode must be portable and must be locked when accessed. If you require identity mode, you must use DSO (see Terracotta Custom Installation).
  • copyOnRead – Enables ("true") or disables ("false" DEFAULT) "copy cache" mode. If disabled, cache values are not deserialized on every read. For example, repeated get() calls return a reference to the same object(references are ==).

    • When enabled, cache values are deserialized (copied) on every read and the materialized values are not re-used between get() calls; each get()call returns a unique reference. When enabled, this setting allows Ehcache to behave as a component of OSGI, allows a cache to be shared by callers with different classloaders, and prevents local drift if keys/values are mutated locally without being put back into the cache. Enabling copyOnRead is relevant only for caches with valueMode set to serialization .

  • coherent – Enables ("true" DEFAULT) or disables ("false") cache coherence. When enabled, guarantees the consistency of the cache’s data across the cluster. When disabled, useful for optimizing certain cache operations, including bulk-loading of caches, by removing the requirement of locks and adding transaction batching. This attribute sets a cache’s initial coherence mode, which can be reset using the Terracotta Distributed Ehcache Cache Coherence API.
  • synchronousWrites – Enables ("true") or disables ("false" DEFAULT) synchronous writes from Terracotta clients (application servers) to Terracotta servers. Asynchronous writes (synchronousWrites="false") maximize performance by allowing clients to proceed without waiting for a "transaction received" acknowledgement from the server. Synchronous writes (synchronousWrites="true") maximize data safety by requiring that a client receive server acknowledgement of a transaction before that client can proceed. If coherence mode is disabled using configuration (coherent="false") or the coherence API, only asynchronous writes can occur (synchronousWrites="true" is ignored).
  • coherentReads – DEPRECATED. This attribute is superseded by the attribute coherent . Disallows ("true" DEFAULT) or allows ("false") "dirty" reads in the cluster. If set to "true", reads must be coherent on any node and returned data is guaranteed to be coherent. If set to false, local unlocked reads are allowed and returned data may be stale. Allowing dirty reads may boost the cluster’s performance by reducing the overhead associated with locking. Read-only applications, applications where stale data is acceptable, and certain read-mostly applications may be suited to allowing dirty reads.

The following attributes are used with Terracotta Distributed Ehcache for Hibernate:

  • localKeyCache – Enables ("true") or disables ("false" DEFAULT) a local key cache. Terracotta Distributed Ehcache for Hibernate can cache a "hotset" of keys on clients to add locality-of-reference, a feature suitable for read-only cases. Note that the set of keys must be small enough for available memory.
  • localKeyCacheSize – Defines the size of the local key cache in number (positive integer) of elements. In effect if localKeyCache is enabled. The default value, 300000, should be tuned to meet application requirements and environmental limitations.
  • orphanEviction – Enables ("true" DEFAULT) or disables ("false") orphan eviction. Orphans are cache elements that are not resident in any Hibernate second-level cache but still present on the cluster's Terracotta server instances.
  • orphanEvictionPeriod – The number of local eviction cycles (that occur on Hibernate) that must be completed before an orphan eviction can take place. The default number of cycles is 4. Raise this value for less aggressive orphan eviction that can reduce faulting on the Terracotta server, or raise it if garbage on the Terracotta server is a concern.
Default Behavior

By default, adding <terracotta /> to a <cache> block is equivalent to adding the following:

 <terracotta clustered="true" valueMode="serialization" coherent="true" copyOnRead="false" />
<terracottaConfig>

This element can not be used with a DSO installation (see Express Versus Custom Installations). It enables the client to identify the source of configuration. The client must load the configuration from a file or a Terracotta server. The value of the url attribute should contain a path to the file or the address and DSO port (9510 by default) of a server. In the example value, "localhost:9510" means that the Terracotta server is on the local host.

TIP: Terracotta Clients and Servers
In a Terracotta cluster, the application server is also known as the client.

For more information on client configuration, see the Clients Configuration Section in the Terracotta Configuration Guide and Reference .

Using an URL Attribute

Add the url attribute to the <terracottaConfig> element as follows:

<terracottaConfig url="<source>" />

where <source> must be one of the following:

  • A path (for example, url="/path/to/tc-config.xml" )
  • An URL (for example, url="http://www.mydomain.com/path/to/tc-config.xml )
  • A Terracotta host address in the form <host>:<dso-port> (for example, url="host1:9510" )

Note the following about using server addresses in the form <host>:<dso-port>:

  • The default DSO port is 9510.
  • In a multi-server cluster, you can specify a comma-delimited list (for example, url="host1:9510,host2:9510,host3:9510" ).
Embedding Terracotta Configuration

You can embed the contents of a Terracotta configuration file in ehcache.xml as follows:

    <terracottaConfig>
        <tc-config>
            <servers>
                <server host="server1" name="s1"/>
                <server host="server2" name="s2"/>
            </servers>
            <clients>
                <logs>app/logs-%i</logs>
            </clients>           
        </tc-config>
    </terracottaConfig>

Note that not all elements are supported. For example, the <dso> section of a Terracotta configuration file is ignored in an Ehcache configuration file.

1.2.1b Cache Events Configuration

The <cache> subelement <cacheEventListenerFactory>, which registers listeners for cache events such as puts and updates, has a notification scope controlled by the attribute listenFor . This attributed can have one of the following values:

  • local – Listen for events on the local node. No remote events are detected.
  • remote – Listen for events on other nodes. No local events are detected.
  • all – (DEFAULT) Listen for events on both the local node and on remote nodes.

In order for cache events to be detected by remote nodes in a Terracotta cluster, event listeners must have a scope that includes remote events. For example, the following configuration allows listeners of type MyCacheListener to detect both local and remote events:

<cache name="myCache" maxElementsInMemory="1000" 
       maxElementsOnDisk="10000" eternal="false" timeToIdleSeconds="3600" 
       timeToLiveSeconds="0" memoryStoreEvictionPolicy="LFU"> 
  <!-- Not defining the listenFor attribute for <cacheEventListenerFactory> is by default equivalent to listenFor="all". -->
  <cacheEventListenerFactory class="net.sf.ehcache.event.TerracottaCacheEventReplicationFactory" />
  <terracotta /> 
</cache>

You must use net.sf.ehcache.event.TerracottaCacheEventReplicationFactory as the factory class to enable cluster-wide cache-event broadcasts in a Terracotta cluster.

1.2.1c Incompatible Configuration

For any clustered cache, you must delete, disable, or edit configuration elements in ehcache.xml that are incompatible when clustering with Terracotta. Clustered caches have a <terracotta /> or <terracotta clustered="true"> element.

The following Ehcache configuration attributes or elements should be deleted or disabled:

  • DiskStore-related attributes overflowToDisk and diskPersistent .

    • The Terracotta server automatically provides a disk store.

  • Replication-related attributes such as replicateAsynchronously and replicatePuts .
  • The attribute MemoryStoreEvictionPolicy must be set to either LFU or LRU.

    • Setting MemoryStoreEvictionPolicy to FIFO causes the error IllegalArgumentException .

See the Ehcache documentation for more information on the elements in a standard Ehcache configuration file.

1.2.2 Terracotta Distributed Ehcache Cluster Events API

The Terracotta Distributed Ehcache cluster events API provides access to Terracotta cluster events and cluster topology.

1.2.2a Cluster Topology

The interface net.sf.ehcache.cluster.CacheCluster provides methods for obtaining topology information for a Terracotta cluster. The following table lists these methods.

 

Method
Definition
String getScheme()
Returns a scheme name for the cluster information. Currently TERRACOTTA is the only scheme supported. The scheme name is used by CacheManager.getCluster() to return cluster information (see Events API Example Code).
Collection<ClusterNode> getNodes()
Returns information on all the nodes in the cluster, including ID, hostname, and IP address.
boolean addTopologyListener(ClusterTopologyListener listener)
Adds a cluster-events listener. Returns true if the listener is already active.
boolean removeTopologyListener(ClusterTopologyListener)
Removes a cluster-events listener. Returns true if the listener is already inactive.

The interface net.sf.ehcache.cluster.ClusterNode provides methods for obtaining information on specific Terracotta nodes in the cluster. The following table lists these methods.

Method
Definition
getId()
Returns the unique ID assigned to the node.
getHostname()
Return the hostname on which the node is running.
getIp()
Return the IP address on which the node is running.

1.2.2b Cluster Events

The interface net.sf.ehcache.cluster.ClusterTopologyListener provides methods for detecting the following cluster events:

  • nodeJoined(ClusterNode)
  • nodeLeft(ClusterNode)
  • clusterOnline(ClusterNode)
  • clusterOffline(ClusterNode)

1.2.2c Events API Example Code

// Get cluster data
CacheManager mgr = new CacheManager(); // Local ehcache.xml exists, with at least one cache configured with Terracotta clustering.
CacheCluster cluster = mgr.getCluster("TERRACOTTA");

NOTE: Programmatic Creation of CacheManager
If a CacheManager instance is created and configured programmatically (without an ehcache.xml or other external configuration resource),
getCluster("TERRACOTTA") may return null even if a Terracotta cluster exists. To ensure that cluster information is returned in this case, get a cache that is clustered with Terracotta:
// mgr created and configured programmatically.
CacheManager mgr = new CacheManager(); 
// myCache has Terracotta clustering.
Cache cache = mgr.getEhcache("myCache"); 
// A Terracotta client has started, making available cluster information.
CacheCluster cluster = mgr.getCluster("TERRACOTTA");
 
// Get current nodes
Collection<ClusterNode> nodes = cluster.getNodes();
for(ClusterNode node : nodes) {
    System.out.println(node.getId() + " " + node.getHostname() + " " + node.getIp());
}
 
// Register listener
cluster.addTopologyListener(new ClusterTopologyListener() { 
    public void nodeJoined(ClusterNode node) { System.out.println(node + " joined"); }
    public void nodeLeft(ClusterNode node) { System.out.println(node + " left"); }
    public void clusterOnline(ClusterNode node) { System.out.println(node + " enabled"); }
    public void clusterOffline(ClusterNode node) { System.out.println(node + " disabled"); }
});

1.2.3 Terracotta Distributed Ehcache Cache Coherence API

The Distributed Ehcache cache coherence API can optimize certain cache operations, including bulk-loading of caches, by removing the requirement of locks and adding transaction batching. The cache coherence API allows applications to:

The initial coherence mode of a cache can also be set by configuration. See the attribute "coherent" in <terracotta>.

The following table lists the cache coherence API methods that are available by implementing the interface org.terracotta.modules.ehcache.coherence.CacheCoherence .

Method
Definition
public boolean isClusterCoherent()
Returns true if a cache is coherent throughout the cluster, or false if it is not coherent anywhere in the cluster.
public boolean isNodeCoherent()
Returns true if a cache is coherent on the current node, or false if it is not.
public void setNodeCoherent(boolean)
Sets a cache’s coherence mode to coherent (true) or not coherent (false) on the local node. No operation results if the cache is already in the mode specified by setNodeCoherent() .
public void waitUntilClusterCoherent()
Waits until a cache is coherent before returning. Changes are automatically batched and the cache is updated throughout the cluster. Returns immediately if a cache is coherent throughout the cluster.

Note the following about the cache coherence API:

1.2.3a Coherence API Example Code

The following example code shows how a clustered application with Terracotta Distributed Ehcache can use the cache coherence API to optimize a bulk-load operation.

The following is a simple example of a class that uses the cache coherence API to optimize loading a cache:

import net.sf.ehcache.Cache;
 
public class MyBulkLoader {
   CacheManager cacheManager = new CacheManager();  // Assumes local ehcache.xml.
   Cache cache = cacheManager.getEhcache("myCache"); // myCache defined in ehcache.xml.
   cache.setCoherent(false); // myCache is now not coherent.
  
   // Load data into myCache.
 
   cache.setCoherent(true); // Done, now set myCache back to being coherent.
}

On another node, application code that intends to touch myCache can run or wait, based on whether myCache is coherent or not:

...
   if (cache.isCoherent()) {
 
    // Do some work.
    }
  
   else {
 
     cache.waitUntilCoherent()
     // Do the work when waitUntilCoherent() returns.
}
...

Waiting may not be necessary if the code can handle potentially stale data:

...
   if (cache.isCoherent()) {
 
    // Do some work.
    }
  
   else {
    
    // Do some work knowing that data in myCache may be stale.
 
    }
...

1.2.3b Allowing Non-Coherent Reads From a Coherent Cache

Certain environments require coherent cached data while also desiring optimized reads of cache data. If applications that read data from a coherent cache can tolerate getting potentially stale data, an unlocked (non-coherent) reads view, UnlockedReadsView, can be created for Cache types. UnlockedReadsView requires Ehcache 2.1 or higher and requires the underlying cache have Terracotta clustering. For example:

<cache name="myCache"
      maxElementsInMemory="500"
      eternal="false"
      overflowToDisk="false"
    <terracotta clustered="true"/>
</cache>

You can create an unlocked view of myCache programmatically:

Cache cache = cacheManager.getEhcache("myCache");
UnlockedReadsView unlockedReadsView = new UnlockedReadsView(cache, "myUnlockedCache");

The following table lists the API methods that are available by with net.sf.ehcache.constructs.unlockedreadsview.UnlockedReadsView .

Method
Definition
public String getName()
Returns the name of the unlocked cache view.
public Element get(final Object key)
public Element get(final Serializable key)
Returns the data under the given key. Returns null if data has expired.
public Element get(final Object key)
public Element get(final Serializable key)
Returns the data under the given key without updating cache statistics. Returns null if data has expired.

1.2.4 Non-Blocking Disconnected Cache (NonStopCache) API

The NonStopCache API allows certain cache operations to proceed on clients that have become disconnected from the cluster. Applications can use this non-blocking API to create a local cache, based on an existing cache, in the following way:

Cache cache = cacheManager.getEhcache("myCache");
NonStopCache nonStopCache = new NonStopCache(cache, "myNonStopCache");

myNonStopCache is a local cache using the same backing store as myCache. To enable access to myNonStopCache in the CacheManager scope, add myNonStopCache to a CacheManager:

cacheManager.addDecoratedCache(nonStopCache);

1.2.4a Configuring Timeouts and Behaviors

NonStopCache types can be configured with the following attributes:

  • name – A unique name for the NonStopCache instance. REQUIRED.
  • timeoutMillis – Specifies the number of milliseconds an application waits for any cache operation before timing out. The default value is 5000 (five seconds). The behavior after the timeout occurs is determined by timeoutBehavior . This
  • timeoutBehavior – Determines the response after a timeout occurs ( timeoutMillis expires). The response can be set to one of the values listed in the following table:
    • Value
    Behavior
    exception
    (DEFAULT) Throw NonStopCacheException .
    noop
    Return null for gets. Ignore all other cache operations.
    localReads
    For caches with Terracotta clustering, allow non-coherent reads of cache data. Ignore all other cache operations. For caches without Terracotta clustering, throw an exception.
  • immediateTimeout – Enables ("true", DEFAULT) or disables ("false") an immediate timeout response if the Teracotta client detects a network interruption (the node is disconnected from the cluster). If enabled, this parameter overrides timeoutMillis , so that the option set in timeoutBehavior is in effect immediately.

You can tune the default timeout values and behaviors of NonStopCache to fit your environment.

For example, in an environment with regular network interruptions, consider disabling immediateTimeout and increasing timeoutMillis to prevent timeouts for most of the interruptions.

For a cluster that experiences regular but short network interruptions, and in which caches clustered with Terracotta carry read-mostly data or there is tolerance of potentially stale data, you may want to set timeoutBehavior to localReads .

In an environment where cache operations can be slow to return and data is required to always be in sync, increase timeoutMillis to prevent frequent timeouts. Set timeoutBehavior to noop to force the application to get data from another source or exception if the application should stop.

Configuring a NonStopCache in ehcache.xml

NonStopCaches are configured in a <cache> block using <cacheDecoratorFactory> elements. Use one <cacheDecoratorFactory> element for each NonStopCache you want to make available for a cache. In the following example, myCache has more than one NonStopCache configuration available:

<cache name="myCache" maxElementsInMemory="10000" eternal="false"
        overflowToDisk="false">
  <cacheDecoratorFactory
   class="net.sf.ehcache.constructs.nonstop.NonStopCacheDecoratorFactory"
   properties="name=nonStopCache" />
  <cacheDecoratorFactory
   class="net.sf.ehcache.constructs.nonstop.NonStopCacheDecoratorFactory"
   properties="name=nonStopNoopCache, timeoutBehavior=noop, timeoutMillis=10000, immediateTimeout=false" />
  <cacheDecoratorFactory
   class="net.sf.ehcache.constructs.nonstop.NonStopCacheDecoratorFactory"
   properties="name=nonStopLocalReadsCache, timeoutBehavior=localReads, timeoutMillis=4000, immediateTimeout=false" />
  <terracotta />
</cache>

Application nodes can instantiate any combination of the NonStopCaches configured for myCache.

<CacheDecoratorFactory> has the following attributes:

  • class – Specifies the creator class for NonStopCache.
  • properties – Specifies the configuration properties of the NonStopCache. The name property is required. The remaining properties use their default values if left unspecified.

    • NOTE: defaultCache
    The default cache cannot have <cacheDecoratorFactory> elements. However, the defaultCache can be programmatically configured with a NonStopCache.
Configuring a NonStopCache Programmatically

You can programmatically configure a NonStopCache either by using the NonStopCache API directly or by passing a standard Java Properties object.

Use NonStopCacheConfigImpl() to create the NonStopCache configuration and pass it to the NonStopCache:

NonStopCacheConfig config = new NonStopCacheConfigImpl(); // This creates a config with default values.
config.setTimeoutMillis(10000); // Customize other parameters similarly: config.setXXX(..)
NonStopCache nonStopCache = new NonStopCache(cache, "myNonStopCache", config); // Note that the name of the NonStopCache cannot be set using NonStopCacheConfig.

You can also pass a Properties object to NonStopCacheConfig:

Properties properties  = new Properties();
properties.setProperty("timeoutMillis", "1000"); // Customize other parameters similarly: properties.setProperty(...).
NonStopCacheConfig config = new NonStopCacheConfigImpl(properties);
NonStopCache nonStopCache = new NonStopCache(cache, "myNonStopCache", config); // Note that the name of the NonStopCache cannot be set using NonStopCacheConfig.

1.2.5 Write-Behind Queue in Distributed Ehcache

If your application uses the write-behind API with Ehcache and you cluster Ehcache with Terracotta, the write-behind queue automatically becomes a clustered write-behind queue. The clustered write-behind queue features the following characteristics:

The write-behind queue is enabled for a cache with the <cacheWriter /> element. See the Ehcache documentation for more information on the write-behind API.

1.2.6 Working With JTA

Terracotta Distributed Ehcache caches can participate in Java Transaction API (JTA) transactions with an isolation level of ReadCommitted. Terracotta Distributed Ehcache transactions are fully XA compliant and are tested with the most common transaction managers by Atomikos, Bitronix, JBoss, WebLogic, and others.

To allow Terracotta Distributed Ehcache to participate in JTA transactions, the following <cache> attribute must be set as shown:

In addition, the <cache> subelement <terracotta> must have the following attributes set as shown:

For more information on working with transactional caches in Terracotta Distributed Ehcache for Hibernate, see Setting Up Transactional Caches.

1.2.7 Working With OSGi

To allow Terracotta Distributed Ehcache to behave as an OSGi component, the <cache> subelement <terracotta> must have the following attributes set as shown:

Top of 1.2 Distributed Ehcache Reference