THIS IS ARCHIVE DOCUMENTATION FOR TERRACOTTA v. 2.7. For the current release, see the current DSO documentation »

Release: 2.7.3
Publish Date: February, 2009

Troubleshooting Guide

Two Common Errors Getting TCNonPortableObjectError? See [Resolving TCNonPortableObjectError]. Getting UnlockedSharedObjectException? See [Resolving UnlockedSharedObjectException].

The issues covered in this guide are divided into the following sections:

An expanded contents list showing every covered issue is included below.

If your question isn't answered by this guide, try one of the following resources:

• Frequently Asked Questions
• Terracotta Forums
• Users mailing list
• Reference documentation

Contents

EXCEPTIONS AND ERRORS...

On Windows, I get the following error message right after the first "preparing to install" dialog box finishes: Windows error 3 occurred while loading the Java VM. What does this mean and how do I get around this?

You probably have several versions of the Java SDK and runtime installed and something in the Windows registry entries on your machine is confusing the installer. (Even if the JAVA_HOME environment variable set correctly). To get past this issue, clean up your environment or add the "LAX_VM" parameter and the location of the correct Java executable to the command line, like this:

terracotta-windows-2.6.0.exe LAX_VM "c:/jdk1511/bin/java.exe"

I am getting a message that says The DSO boot JAR you need for this platform does not exist. You may need to run the 'make-boot-jar' script. What does that mean?

You need to create the boot JAR. Terracotta includes a set of standard boot JARS, but none may match your environment's JDK version and/or OS. The make-boot-jar tool fixes this error by creating the appropriate boot JAR for your environment. You should run make-boot-jar each time relevant changes, such as a JVM upgrade, occur in the Terracotta installation environment.

$TC_INSTALL_DIR%/common/lib/dso-boot/ is the default output location for make-boot-jar. To specify a different output location for the boot JAR, use the -o option (output).

%TC_INSTALL_DIR%\dso\bin\make-boot-jar.bat [-o My/Output/Location]

$TC_INSTALL_DIR/dso/bin/make-boot-jar.sh [-o My/Output/Location]

Make sure to delete the existing boot JAR files under TC_INSTALL_DIR/lib/dso-boot. The make-boot-jar tool may be unable to replace existing files.

See the Concept and Architecture Guide for more information about the boot JAR file.

Related Issues:

See the Terracotta Boot JAR section.

I am getting com.tc.exception.TCNonPortableObjectError. What next?

See [Resolving TCNonPortableObjectError].

A Terracotta server instance restarted, but the clients did not automatically reconnect.

If you've restarted Terracotta server instances and clients do not automatically reconnect without a restart, the Terracotta server instances are not configured for permanent store.

I am getting com.tc.object.tx.UnlockedSharedObjectException on my code. What next?

See [Resolving UnlockedSharedObjectException]. If the exception refers to third-party code, see this issue.

I get com.tc.object.tx.UnlockedSharedObjectException on a third party JAR and do not have access to code. Is Terracotta any good for me?

Use Sparingly While the following solution suggests the use of "named" locks, this type of lock is a last resort. There can be a hefty performance penalty to using named locks, so use named locks sparingly and only where absolutely necessary.

Yes. You can use "named" locks instead of autolocks. Named locks will create one global lock across the cluster for the named lock. This can cause contention, and can, in some cases, decrease throughput. When using named locks, you do not need to use the synchronize keyword to isolate the transaction because the method becomes the point of synchronization, and the name of the method becomes the name of the lock. Due to there only being one named lock per method, ALL object instances will serialize access through the single named lock.

Here is an example:

<locks>  
  <named-lock>
    <lock-name>mutateFoo</lock-name>
    <method-expression>void com.system.services.MutateService.mutate(Object)</method-expression>
    <lock-level>write</lock-level>
  </named-lock>
</locks>

See the Concept and Architecture Guide for more information about locks.
See the Configuration Guide and Reference for more information on how to configure locks.

What does java.lang.IllegalMonitorStateException mean and how can I fix it?

An IllegalMonitorStateException implies a lock is missing or cannot be applied to a shared object. If the code in question has synchronization, check for the following causes:

1. A lock-then-share operation occurred. A lock has been specified but was applied to an object before that object was shared. See this Gotcha for more information.
2. Read-level or named locks are being used where write-level locks or autolocks are necessary. Refactor your Terracotta configuration to use the appropriate locks.

I am getting java.io.IOException: No locks available, what next?

This is a problem with NFS lock mgr.

• Solution: Modify tc-config.xml to use different log files

Why am I getting a Hyperic (Sigar) exception and why is the Terracotta Administrator Console not showing certain metrics?

These two problems are related to starting Java from a location different than the value of JAVA_HOME. To avoid the Hyperic error and restore metrics to the Terracotta Administrator Console, invoke Java from the location specified by JAVA_HOME.

Unable to locate bundle dependency: I get the following exception which indicates one or more Terracotta modules or dependencies are missing. How do I get past this scenario?

Missing Terracotta Module 2008-01-18 09:47:55,240 FATAL - Unable to locate bundle dependency: 'jdk15_preinst_config', version '2.5.0', group-id 'org.terracotta.modules'; Tried to resolve the location using the following repositories: 'file:/www/terracotta/terracotta-2.5.0/bin/../modules/'

On Solaris platform:

This is a result of the way the default tar utility on Solaris works. Terracotta loads a few modules located in $TC_INSTALL_DIR/modules directory on startup and if you look in that directory, you should see a collection of .jar files. The default tar utility causes some of these file names to be truncated which results in the above scenario as Terracotta is unable to load these modules.

To get past this, use an alternative tar utility like gtar or download the patch from Sun which fixes this issue.

Links The patch for Solaris tar is Solaris 2.6 patch 105792-03. The SunSoft patch page is at: http://sunsolve.sun.com/pub-cgi/retrieve.pl?doc=fpatches/105792 You can download the Solaris version of GNUtar from: http://www.ibiblio.org/pub/packages/solaris/sparc/

More generic solution:

If it is not an unpacking issue and the kit is installed properly, check the following:

1. The user has read access to all files in the Terracotta installation (specifically the modules directory).
2. Make sure that the module specified in the tc-config.xml is actually located and read-accessible in the appropriate repository (default is $TC_INSTALL_DIR/modules).
3. Make sure that the module version number is specified appropriately in the tc-config.xml and there are no typos.

Why is the Terracotta Administrator Console timing out when it tries to connect to a Terracotta server?

If you've verified that your Terracotta cluster is up and running, but your attempt to monitor it remotely using the Terracotta Administrator Console is unsuccessful, a firewall may be the cause. Firewalls that block traffic from Terracotta servers' JMX ports prevent the Terracotta Administrator Console from seeing those servers. To avoid this and other connection issues that may also be attributable to firewalls, ensure that the JMX and DSO ports configured in Terracotta are unblocked on your network.

When I start a Terracotta server, why does it fail with a schema error?

You may get an error similar to the following when a Terracotta server fails to start:

Error Message:
Starting BootJarTool...
2008-10-08 10:29:29,278 INFO - Terracotta 2.7.0, as of 20081001-101049 (Revision 10251 by cruise@rh4mo0 from 2.7)
2008-10-08 10:29:30,459 FATAL -
*******************************************************************************
The configuration data in the file at '/opt/terracotta/conf/tc-config.xml' does not obey the Terracotta schema:
[0]: Line 8, column 3: Element not allowed: server in element servers

*******************************************************************************

This error occurs when there's a schema violation in the Terracotta configuration file, at the line indicated by the error text. To confirm that your configuration file follows the required schema, see the official Terracotta configuration schema.

I have a setup with one active Terracotta server instance and a number of standbys, so why am I getting errors because more than one active server comes up?

Due to network latency or load, the Terracotta server instances may not may be have adequate time to hold an election. Increase the <election-time> property in the Terracotta configuration file to the lowest value that solves this issue.

Why is a crashed Terracotta server instance failing to come up when I restart it?

If it's running in persistent mode, clear the crashed server instance's data directory.

SHARING IN-MEMORY OBJECTS...

It doesn't look like my application data is being shared. What do I do?

The object browsers available in the Terracotta Administrator Console allow you to view the shared object graphs in your cluster and in each client. The roots of each shared object graph in your cluster are shown as the roots of each graph displayed in the browser. Client object browsers also indicate which object graphs each client is sharing, and which are not being shared.

If clients appear in the Terracotta Administrator Console but certain roots are missing from the object browsers, ensure that the fully qualified name of each root is accurate. For example, if the fully qualified name of a class you want to cluster is org.system.services.foo, check that the Terracotta configuration file declares foo as a root using that full name. Because of dynamic class loading in Java, it's impossible for Terracotta to detect invalid roots due to issues such as missing package names, although scanning your project in Eclipse may locate missing roots.

If both clients and roots appear in the console, but problems with object sharing appear, check for the following causes:

• Static fields – A static field of a shared class is not shared unless that field is configured to be a Terracotta root. Optionally, make the field non-static if that is acceptable in your application.
• Differing configuration files – Clients using different Terracotta configuration files may not share objects properly. This may be the case, for example, if the configuration files do not have the same sets of root declarations and instrumented classes. Align the content in the configuration files and restart each application instance.
• Changed application code – If your application code has changed, the Terracotta configuration file may be out of date. Update the configuration file and restart each application instance.

  If your application code has changed, restarting your Terracotta server instances is also recommended. While this may not be necessary in all cases, it is safer to restart your entire cluster following code changes.

• Changed configuration file – If a Terracotta configuration file has changed, every client or server instances using that configuration file should be restarted to avoid clustering anomalies.
• Uninstrumented code – If uninstrumented code is writing or reading to shared objects, data may not be updated on all nodes. This can cause stale data to appear in some application servers. See the Uninstrumented Access Gotcha for more information.

How do I share Terracotta roots between a POJO application and a web application?

Object identity in Terracotta is established by joining the name of the object's classloader with the fully qualified name of the object reference. To share Terracotta roots between a POJO application and a web application, you must configure the POJO application to use the same classloader being used by the web application. Use the following option to name the default classloader for the POJO application:

-Dcom.tc.loader.system.name=<Any.class.loader.name>

For example, when Tomcat is the application server, use:

-Dcom.tc.loader.system.name=Tomcat.Catalina:localhost:/webapp_name

When Jetty is the application server, use:

-Dcom.tc.loader.system.name=Jetty.path:/webapp_name

If the POJO application's classes are loaded through the extension classloader (java/ext), use:

-Dcom.tc.loader.ext.name=<Any.class.loader.name>

How do I confirm that my application data is being shared?

The object browsers available in the Terracotta Administrator Console allow you to view the shared object graphs in your cluster and in each client.

How do I know if my POJOs are being shared or not?

Use the cluster object browser and the client object browser, available in the Terracotta Administrator Console. Cluster and client object browsers display the Terracotta root objects at the root of shared-object graphs.

How do I know if EHCache data is being shared? I can see a bunch of EHCache objects in the admin console but cannot find my application data.

First thing to look for is the terracotta root objects. Connect the TC Admin Console ($TC_INSTALL_DIR/bin/admin.sh) to the TC Server instance and look under the Roots. If you see the following roots, EHCache (Hibernate 2nd level cache) is being clustered properly:

net.sf.ehcache.CacheManager.ALL_CACHE_MANAGERS (java.util.Collections$SynchronizedRandomAccessList)
net.sf.ehcache.store.MemoryStoreEvictionPolicy.LRU (net.sf.ehcache.store.MemoryStoreEvictionPolicy)
net.sf.ehcache.store.MemoryStoreEvictionPolicy.LFU (net.sf.ehcache.store.MemoryStoreEvictionPolicy)
net.sf.ehcache.store.MemoryStoreEvictionPolicy.FIFO (net.sf.ehcache.store.MemoryStoreEvictionPolicy)
net.sf.ehcache.store.MemoryStoreEvictionPolicy.DSO (net.sf.ehcache.store.MemoryStoreEvictionPolicy)
net.sf.ehcache.CacheManager.singleton (net.sf.ehcache.CacheManager)

EHCache breaks down the key-value pair and stores the value object wrapped by Element objects in a fairly deep object graph, as it does a lot of bookkeeping with timestamps etc. I am breaking down below exactly where you should be looking for the actual application data stored in the cache:

net.sf.ehcache.CacheManager.ALL_CACHE_MANAGERS (java.util.Collections$SynchronizedRandomAccessList)
|
|_____ java.util.Collections$SynchronizedCollection.c (java.util.ArrayList) [1/1]
	   |
	   |_____ 0 (net.sf.ehcache.CacheManager)
	         |
	         |_____ net.sf.ehcache.CacheManager.caches (java.util.HashMap) [3/3]
	               |
	               |_____ 0 (MapEntry)
	                     |
	                     |_____ value (net.sf.ehcache.Cache)
	                           |
	                           |_____ net.sf.ehcache.Cache.memoryStore (net.sf.ehcache.store.TimeExpiryMemoryStore)
(continued)
|
|_____ net.sf.ehcache.store.MemoryStore.map (net.sf.ehcache.store.TimeExpiryMemoryStore$SpoolingTimeExpiryMap)
       |
       |_____ com.tcclient.ehcache.TimeExpiryMap.timeExpiryDataStore (com.tcclient.cache.CacheDataStore)
              |
              |_____ com.tcclient.cache.CacheDataStore.store (java.util.Map[])
                     |
                     |_____ 0 (java.util.HashMap) [1/1]
                            |
                            |_____ 0 (MapEntry)
                                  |
                                  |_____ key=XML_0K_1551
                                         value (com.tcclient.cache.CacheData)
                                         |
                                         |_____ com.tcclient.cache.CacheData.value (net.sf.ehcache.Element)
                                                |
	                                        |_____ net.sf.ehcache.Element.value (Value object type)
	                                               |
                                                       |_____ The value object put in the cache by the app

THE TERRACOTTA BOOT JAR...

Why is there a boot JAR file?

The boot JAR instruments classes that are loaded early in the JVM's startup. These classes can't be instrumented in the usual way because they are instantiated before normal instrumentation can take place. Terracotta provides a set of standard boot JARs for a variety of environments.

See the Concept and Architecture Guide for more information about the boot JAR file.

Related Issues:

If you get the error The DSO boot JAR you need for this platform does not exist. You may need to run the 'make-boot-jar' script., then non of the standard Terracotta boot JARs may be right for your environment. See this issue on creating a boot JAR that works in your environment.

Do I ever have to rebuild the boot JAR?

The boot JAR may have to be rebuilt under the following conditions:

• Before the first time you run Terracotta (see this issue for more information.)
• You've updated the <additional-boot-jar-classes> property in tc-config.xml
• You've made changes to Java locks that affect Java core classes such as java.util.Hashtable.

There is no need to rebuild the boot JAR if the changes you've made affect only application classes.

So I need to add a certain class to the boot JAR. How do I do it?

First, add the class in the configuration file in the <additional-boot-jar-classes> section:

<additional-boot-jar-classes>
    <include>java.awt.datatransfer.Clipboard</include>
</additional-boot-jar-classes>

Next, provide the configuration file with the above entry as a parameter to the make-boot-jar script:

%TC_INSTALL_DIR%\dso\bin\make-boot-jar.bat -f tc-config.xml

$TC_INSTALL_DIR/dso/bin/make-boot-jar.sh -f tc-config.xml

Make sure to delete the existing boot JAR files under TC_INSTALL_DIR/lib/dso-boot as the make-boot-jar script is sometimes unable to replace existing files.

TERRACOTTA FOR SPRING...

How is logical identity for the Spring application context maintained?

Terracotta for Spring creates DSO root for each Spring application context. Because Spring framework
does not provide unique Id for the application context, matching patterns are used to select web
application name and to specify matching config names to identify application contexts
that need to be clustered.

Name of the DSO root is calculated based on the web application name and names of all config
resources used to initialize application context. If any name of the Spring configs changes or
if new configs been added, the generated id will be different.
See the following method for more details com.tcspring.DistributableBeanFactoryMixin.addLocation(String location)

When different configs are used but contexts need to have the same identity, the root name can be
assigned manually using the following configuration element:

<application>
   <spring>
     <jee-application>
       <application-contexts>
         <application-context>
           <root-name>myClusteredSpringContext</root-name>
           ...

What if the Spring application context is not distributed?

Terracotta console or a web server logs shows that Spring application context is not distributed:

INFO [com.tcspring.ApplicationHelper] - Application name Tomcat.shared
INFO [com.tcspring.DistributableBeanFactoryMixin] - 09080A0C0905030C0E05070F0805050A Context is NOT distributed

To debug that you can enable DEBUG logging for org.springframework and or.tcspring categories using either
log4j settings for your web application, or if running on Tomcat global logging settings at <tomcat home>\conf\logging.properties

Then you can check the actual names for the web application and config resources. For example:

2007-12-27 18:33:43,507 INFO [com.tcspring.ApplicationHelper] - Application name Tomcat.context:/foo
 2007-12-27 18:33:43,804 INFO [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] - 
Loading XML bean definitions from ServletContext resource [/WEB-INF/spring/servlet.xml]
 2007-12-27 18:33:45,710 INFO [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] - 
Loading XML bean definitions from ServletContext resource [/WEB-INF/spring/dao.xml]
 2007-12-27 18:33:45,804 INFO [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] - 
Loading XML bean definitions from ServletContext resource [/WEB-INF/spring/services.xml]
 2007-12-27 18:33:46,273 INFO [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] - 
Loading XML bean definitions from ServletContext resource [/WEB-INF/spring/datasource.xml]
 2007-12-27 18:33:46,335 INFO [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] - 
Loading XML bean definitions from ServletContext resource [/WEB-INF/spring/cache.xml]
 2007-12-27 18:33:47,257 INFO [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] - 
Loading XML bean definitions from ServletContext resource [/WEB-INF/spring/captcha.xml]
 2007-12-27 18:33:47,320 INFO [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] - 
Loading XML bean definitions from ServletContext resource [/WEB-INF/spring/validators.xml]
 2007-12-27 18:33:47,351 INFO [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] - 
Loading XML bean definitions from ServletContext resource [/WEB-INF/spring/storedprocs.xml]
 2007-12-27 18:33:47,648 INFO [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] - 
Loading XML bean definitions from ServletContext resource [/WEB-INF/spring/pagers.xml]
 2007-12-27 18:33:47,695 INFO [com.tcspring.DistributableBeanFactoryMixin] - 09080A0C0905030C0E05070F0805050A Matching 
locations:[/WEB-INF/spring/servlet.xml, /WEB-INF/spring/dao.xml, /WEB-INF/spring/services.xml, 
/WEB-INF/spring/datasource.xml, /WEB-INF/spring/cache.xml, /WEB-INF/spring/captcha.xml, WEB-INF/spring/validators.xml, 
/WEB-INF/spring/storedprocs.xml, /WEB-INF/spring/pagers.xml]
 2007-12-27 18:33:47,695 INFO [com.tcspring.DistributableBeanFactoryMixin] 
- 09080A0C0905030C0E05070F0805050A Context is distributed
 2007-12-27 18:33:47,695 INFO [com.tcspring.ClassHierarchyWalker] - 09080A0C0905030C0E05070F0805050A 
registering include for com.planetpope.beans.SystemStatus
 2007-12-27 18:33:47,695 INFO [com.tcspring.DistributableBeanFactoryMixin] - 09080A0C0905030C0E05070F0805050A 
registering transient fields for systemStatus com.planetpope.beans.SystemStatus

In the above example application name is "foo" and corresponding configuration would look like this:

<application>
    <spring>
      <jee-application name="foo">
        <application-contexts>
          <application-context>
            <paths>


              <path>*/spring/servlet.xml
              <path>*/spring/dao.xml</path>
              <path>*/spring/services.xml</path>
              <path>*/spring/datasource.xml</path>
              <path>*/spring/cache.xml</path>
              <path>*/spring/captcha.xml</path>
              <path>*/spring/validators.xml</path>
              <path>*/spring/storedprocs.xml</path>
              <path>*/spring/pagers.xml</path>
            </paths>
            ...

When context is distributed Terracotta admin console will also show special spring_info root for each clustered
context where you could see the web application name, matching config names and also clustered Spring bean names.
For example:

tc:spring_info:05000C00060B090F090D040B0D0D0501 (java.util.ArrayList) [4/4]
   0 (com.tcspring.ApplicationContextEventProtocol)
       com.tcspring.ApplicationContextEventProtocol.appName (java.lang.String)=jmx
       com.tcspring.ApplicationContextEventProtocol.beanNames (java.util.ArrayList) [2/2]
         0 (java.lang.String)=clusteredHistory
         0 (java.lang.String)=clusteredCounder
       com.tcspring.ApplicationContextEventProtocol.configs (java.util.ArrayList) [1/1]
         0 (java.lang.String)=demo/jmx/applicationContext.xml
       com.tcspring.ApplicationContextEventProtocol.eventTypes (java.util.ArrayList) [0/0]

How can I share data between a standalone application and web application?

See this topic.

BEST PRACTICES...

Terracotta clients are not showing up after I start my application servers.

If you use a startup script (such as startup.sh or catalina.sh with Apache Tomcat), make sure it does not overwrite the JAVA_OPTS environment variable, which must have -Xbootclasspath for Terracotta to start properly. If your startup script must set JAVA_OPTS, be sure it includes -Xbootclasspath.

If you've restarted Terracotta server instances and clients do not automatically reconnect without a restart, the Terracotta server instances are not configured for permanent store.

Is there a faster way to declare configurations?

The Eclipse plugin shipped with Terracotta makes the task of creating the Terracotta configuration seamless. You can also leverage the AspectWerkz pattern language for a regex approach to this task. For example, define read locks around all getters in a Class:

<locks>
  <autolock>
    <method-expression>* com.system.services.MutateService.get*(..)</method-expression>
    <lock-level>read</lock-level>
  </autolock>
</locks>

My code uses the singleton pattern. I get exceptions on restart.

If your code uses the singleton pattern, make sure that the root is the field in the Singleton and not a reference to the Singleton elsewhere in your code. Static fields are not replicated by Terracotta unless you explicitly mark them as roots (given that static fields belong to a "class" and not an "object" instance).

I lost some data after my entire cluster lost power and went down. How can I ensure that all data persists through a failure?

First, if you've lost only some data, then you have configured Terracotta to persist data (which is highly recommended in most production environments). The cause for losing a small amount of data could be disk "write" caching on the machines running the Terracotta server instances. If every Terracotta server instance lost power when the cluster went down, data remaining in the disk cache of each machine was obliterated.

Turning off disk caching is not an optimal solution because the machines running Terracotta server instances will suffer a substantial performance degradation. A better solution is to ensure that power is never interrupted at any one time to every Terracotta server instance in the cluster. This can be achieved through techniques such as the use of uninterruptible power supplies and geographically subdividing cluster members.

HOW DO I...

How do I use my JRE instead of the one shipped with Terracotta?

Point the TC_JAVA_HOME environment variable to a supported JDK or JRE. You should point it to the top level installation directory of the JDK, which is the directory containing the bin directory.

How do I use Terracotta to cluster my POJOs? Is there an API?

There are three different ways to add clustering to your application using Terracotta:

• Add shared classes and to the Terracotta configuration file, tc-config.xml, and add locking to methods that read from or write to shared objects.
• Instrument classes directly in your code using annotations.
• Install a Terracotta Integration Module that includes the instrumented classes your application needs for clustering.

If your code is already optimized for data coherency by implementing synchronization designs provided by Java, Terracotta may already support those designs and little configuration may be necessary. For more information, see the Configuration Guide and Reference.

How do I confirm that my Terracotta servers and clients are up and running correctly?

Here are some ways to confirm that your Terracotta servers are running:

• Connect to the servers using the Terracotta Administrator Console.
• When you start a Terracotta server, look at the standard output messages to see that each server started and did not exit.
• Check each server's logs to see that each server started and did not exit.
• Use a tool such as wget to access the /config or /version servlet. For example, for a server running on localhost and using DSO port 9510, use the following wget command to connect to the version servlet:

  [PROMPT] wget http://localhost:9510/version
  

Here are some ways to confirm that your application servers are running with Terracotta:

• After connecting to the cluster using the Terracotta Administrator Console, check that the clients are displayed as expected.
• When you start your application, look at the standard output messages to see that each Terracotta client started and did not exit.
• Check each client's logs to see that each server started and did not exit.

If an exception appears in any of the logs, search for information on it in this document or elsewhere in the Terracotta documentation.

How can I deploy terracotta in my clustered environment to cluster my plain java application, my application server, servlet container?

Version 2.3 and higher This answer assumes that the Terracotta version being used is 2.3 and higher.

1. Install Terracotta (on all nodes). Set the TC_INSTALL_DIR environment variable to your installation directory and make sure that JAVA_HOME is set to the appriprate JDK/JRE that is being used for your application.

2. Create the boot JAR for your environment by running the make-boot-jar script. Make sure that the JAVA_HOME environment variable is pointing to the correct JDK/JRE.

$TERRACOTTA_INSTALL_DIR/bin/make-boot-jar.sh

%TERRACOTTA_INSTALL_DIR%\bin\make-boot-jar.bat

3. Create a Terracotta configuration (tc-config.xml).

• You can use the tc-config-*.xml samples located in TC_INSTALL_DIR\config-examples directory as a good starting point. Here is a good documentation of different properies in tc-config.xml: Configuration Guide and Reference.

• You can also refer to the tc-config-reference.xml located in the docs27 directory of the Terracotta installation.

Make sure that the server host property is set to the correct hostname/ip if you are running the Terracotta Server on a seprate box than your application.

<servers>
    <!-- Tell DSO where the Terracotta server can be found. -->
    <server host="localhost">
      <data>%(user.home)/terracotta/server-data</data>
      <logs>%(user.home)/terracotta/server-logs</logs>
    </server>
</servers>

<servers>
    <!-- Tell DSO where the Terracotta server can be found. -->
    <server host="host1">
      <data>%(user.home)/terracotta/server-data</data>
      <logs>%(user.home)/terracotta/server-logs</logs>
    </server>
</servers>

You can also use the %h wildcard which always uses the the local hostname of the machine on which the server started:

<servers>
    <!-- Tell DSO where the Terracotta server can be found. -->
    <server host="%h">
      <data>%(user.home)/terracotta/server-data</data>
      <logs>%(user.home)/terracotta/server-logs</logs>
    </server>
</servers>

4. Start the Terracotta Server on some machine (e.g. host1) passing the configuration created in step 3 like this:

$TC_INSTALL_DIR/bin/start-tc-server.sh -f tc-config.xml

%TC_INSTALL_DIR%\bin\start-tc-server.bat -f tc-config.xml

5. Set the environment/system properties and java options necessary to integrate Terracotta in your application. You can use the dso-env script that will set the necessary environment variables/java options for you (in the TC_JAVA_OPTS variable):

TERRACOTTA_SERVER=localhost
TC_CONFIG_PATH="${TERRACOTTA_SERVER}:9510"
. $TC_HOME/bin/dso-env.sh -q
export JAVA_OPTS="${TC_JAVA_OPTS}"
java $JAVA_OPTS MainClass

** Replace localhost above with the hostname/ip of the machine
** where the Terracotta Server is started (Step 4).

setlocal
set TERRACOTTA_SERVER_HOST=localhost
set TC_CONFIG_PATH=%TERRACOTTA_SERVER_HOST%:9510
call "%TC_INSTALL_DIR%\bin\dso-env.bat" -q
set JAVA_OPTS=%TC_JAVA_OPTS%
java %JAVA_OPTS% MainClass
endlocal

** Replace localhost above with the hostname/ip of the machine 
** where the Terracotta Server is started (Step 4).

TERRACOTTA_SERVER=localhost
TC_CONFIG_PATH="${TERRACOTTA_SERVER}:9510"
. $TC_HOME/bin/dso-env.sh -q
export JAVA_OPTS="${TC_JAVA_OPTS}"
startup.sh

** Replace localhost above with the hostname/ip of the machine where the 
** Terracotta Server is started (Step 4).

setlocal
set TERRACOTTA_SERVER_HOST=localhost
set TC_CONFIG_PATH=%TERRACOTTA_SERVER_HOST%:9510
call "%TC_INSTALL_DIR%\bin\dso-env.bat" -q
set JAVA_OPTS=%TC_JAVA_OPTS%
call startup.bat
endlocal

** Replace localhost above with the hostname/ip of the machine 
** where the Terracotta Server is started (Step 4).

Extra Information for the curious:

Alternatively, if you are someone who likes to know what is being set by the dso-env script to Terracotize your application, you can set it up manually also.

You will need to add the following three environment variables:

i. -Xbootclasspath/p:"%TC_HOME%\lib\dso-boot\dso-boot-hotspot_win32_150_08.jar" 

** Replace the name of the file above (dso-boot-hotspot_win32_150_08.jar)
**  with the file created in step 2.

** Make sure that the boot JAR file name reflects the same
** JDK/JRE major_minor version as the JDK/JRE you are 
** going to use to run the application (JAVA_HOME or in your PATH). 
** You can verify your java version by running  'java -version' on the command line.

ii. -Dtc.install-root="%TC_HOME%" 

** Assuming TC_HOME is set to your installation path **

iii. -Dtc.config="localhost:9510"

** Replace localhost with the hostname/ip of the machine where the 
** Terracotta Server was started in step 4 e.g. host1 or localhost if
** running on the same machine.

So, if your java invocation looks like this:

java -cp c:\lib\somejar.jar com.mypackage.MainClass

And it should look similar to this with Terracotta:

java -Xbootclasspath/p:"%TC_HOME%\lib\dso-boot\dso-boot-hotspot_win32_150_08.jar"
-Dtc.install-root="%TC_HOME%" 
-Dtc.config="localhost:9510" 
-cp c:\lib\somejar.jarcom.mypackage.MainClass

For your AppServer, you can just add the above java options/system properties to the JAVA_OPTS variable in the AppServer start script:

set JAVA_OPTS=-Xbootclasspath/p:"%TC_HOME%\lib\dso-boot\dso-boot-hotspot_win32_150_08.jar"
-Dtc.install-root="%TC_HOME%" 
-Dtc.config="localhost:9510";%JAVA_OPTS%

export JAVA_OPTS="-Xbootclasspath/p:$TC_HOME/lib/dso-boot/dso-boot-hotspot_win32_150_08.jar 
-Dtc.install-root=$TC_HOME -Dtc.config=localhost:9510":$JAVA_OPTS

TODO ConcurrentModificationException during dehydrate Hashtable and Vector aren't autolocked Null pointer exceptions and transience bean-shell on-load methods

How do I get failover to work properly with 2 nodes, especially after pulling the ethernet cable (plug)?

See Configuring Terracotta for High Availability.

How do I improve the performance of my application which is using clustered EHCache with Terracotta?

Add the following property in the tc.properties file under $TC_INSTALL_DIR/lib (just create one if it doesn't exist):

ehcache.concurrency = 32

This property sets the concurrency (number of striped locks per cache instance) vis-a-vis java.util.concurrent.ConcurrentHashMap. Depending on your application, you will want to play around with this number. E.g. If you have just one cache instance with 128 threads, you will want to use a big number like 64 (or even 128). But if you have hundreds of caches, you have to be careful as too high a number (200 (caches) * 200 (locks) = 40000 ) can add more overhead in terms of memory and lock management. 32 or 64 should be good starting point for most applications. The default value is 1.

How do I reduce planned downtime with Terracotta?

Terracotta persists objects in a new format (field name/value pairings), that does not use Java serialization. It is decoupled from specifics of class versioning. For example - adding/deleting a field to a class will require no flush of Terracotta caches. Additional use-cases in the table below.

How do I use TC Sessions with Tomcat?

It is basically three steps:

• Install Terracotta on all nodes. Set the TC_INSTALL_DIR environment variable to your installation directory and make sure that JAVA_HOME is set to the appriprate JDK/JRE that is being used for your application.

1. Declare your web application (war) name in the configuration xml file:

<web-applications>
  <web-application>your_web_app1</web-application>
  <web-application>your_web_app2</web-application>
</web-applications>

2. Start the terracotta server on hostname:

$TC_INSTALL_DIR/sessions/bin/start-tc-server.sh -f path/to/config/xml/file

3. Start the clients (Tomcat instances) using the follwing script:

TERRACOTTA_SERVER=localhost
TC_CONFIG_PATH="${TERRACOTTA_SERVER}:9510"


. $TC_HOME/bin/dso-env.sh -q

export JAVA_OPTS="${TC_JAVA_OPTS}"
$CATALINA_HOME/bin/startup.sh

setlocal

set TERRACOTTA_SERVER=localhost
set TC_CONFIG_PATH=%TERRACOTTA_SERVER%:9510 -q

call "%TC_INSTALL_DIR%\bin\dso-env.bat"

set JAVA_OPTS=%TC_JAVA_OPTS%
call %CATALINA_HOME%\bin\startup.bat

endlocal

• Replace the localhost in TERRACOTTA_SERVER=localhost with the appropriate IP or hostname where the Terracotta server is running in step 2 (for production mode) or set the TC_CONFIG_PATH with the physical path to the terracotta config file used to start the server in step 2 (for development mode).

• If it complains about the dso-boot-jar, create one (see this issue).

• Alternatively, you can use the TC Sessions Configurator that ships with the sessions product.

How do I use TC Sessions with JBoss Application Server?

It is basically three steps:

• Install Terracotta on all nodes. Set the TC_INSTALL_DIR environment variable to your installation directory and make sure that JAVA_HOME is set to the appriprate JDK/JRE that is being used for your application.

1. Declare your web application (war) name in the configuration xml file:

<web-applications>
  <web-application>your_web_app1</web-application>
  <web-application>your_web_app2</web-application>
</web-applications>

2. Start the terracotta server on hostname:

$TC_INSTALL_DIR/sessions/bin/start-tc-server.sh -f path/to/config/xml/file

3. Start the clients (JBoss instances) using the follwing script:

TERRACOTTA_SERVER=localhost
TC_CONFIG_PATH="${TERRACOTTA_SERVER}:9510"

. $TC_HOME/bin/dso-env.sh -q

export JAVA_OPTS="${TC_JAVA_OPTS}"
$JBOSS_HOME/bin/run.sh

setlocal

set TERRACOTTA_SERVER=localhost
set TC_CONFIG_PATH=%TERRACOTTA_SERVER%:9510 -q

call "%TC_INSTALL_DIR%\bin\dso-env.bat"

set JAVA_OPTS=%TC_JAVA_OPTS%
call %JBOSS_HOME%\bin\run.bat

endlocal

• Replace the localhost in TERRACOTTA_SERVER=localhost with the appropriate IP or hostname where the Terracotta server is running in step 2 (for production mode) or set the TC_CONFIG_PATH with the physical path to the terracotta config file used to start the server in step 2 (for development mode).

• If it complains about the dso-boot-jar, create one (see this issue).

How do I use TC Sessions with WebLogic Application Server?

It is basically three steps:

• Install Terracotta on all nodes. Set the TC_INSTALL_DIR environment variable to your installation directory and make sure that JAVA_HOME is set to the appriprate JDK/JRE that is being used for your application.

1. Declare your web application (war) name in the configuration xml file:

<web-applications>
  <web-application>your_web_app1</web-application>
  <web-application>your_web_app2</web-application>
</web-applications>

2. Start the terracotta server on hostname:

$TC_INSTALL_DIR/sessions/bin/start-tc-server.sh -f path/to/config/xml/file

3. Start the clients (Weblogic instances) using the follwing script:

TERRACOTTA_SERVER=localhost
TC_CONFIG_PATH="${TERRACOTTA_SERVER}:9510"

. $TC_HOME/bin/dso-env.sh -q

export JAVA_OPTS="${TC_JAVA_OPTS}"
export WLS_USER=admin
export WLS_PW=admin
export PRODUCT_MODE=false

${BEA_HOME}/weblogic81/server/bin/startWLS.sh

setlocal

set TERRACOTTA_SERVER=localhost
set TC_CONFIG_PATH=%TERRACOTTA_SERVER%:9510 -q

call "%TC_INSTALL_DIR%\bin\dso-env.bat"

set JAVA_OPTS=%TC_JAVA_OPTS%
set WLS_USER=admin
set WLS_PW=admin
set PRODUCT_MODE=false

call %BEA_HOME%\weblogic81\server\bin\startWLS.cmd

endlocal

• WLS_USER, WLS_PW and PRODUCT_MODE environment variables in the above script are optional and may vary. You can start your individual domain instance instead of

  ${BEA_HOME}/weblogic81/server/bin/startWLS.sh

  or

  %BEA_HOME%\weblogic81\server\bin\startWLS.cmd

• Replace the localhost in TERRACOTTA_SERVER=localhost with the appropriate IP or hostname where the Terracotta server is running in step 2 (for production mode) or set the TC_CONFIG_PATH with the physical path to the terracotta config file used to start the server in step 2 (for development mode).

• If it complains about the dso-boot-jar, create one (see this issue).

• Alternatively, you can use the TC Sessions Configurator that ships with the sessions product.

How do I use the TC Sessions Configurator for any container/app server?

Terracotta Sessions Configurator is preconfigured to use a bundled Apache Tomcat web container. Other web containers are available from the TC Sessions Configurator File | Servers menu option.

If you want to use a customized web container, do not use Sessions Configurator. Check the Integrations page to see if a setup guide is available for your web container.

Each web container generally requires two properties for successful operation. The first is the location of the product home directory (such as CATALINA_HOME or BEA_HOME) and the second is the location of the Java Development Kit (JDK) to use when running the web container (speciffied by the TC_JAVA_HOME property).

Apache Tomcat 5.5
If CATALINA_HOME is not set it will default to the bundled version. If TC_JAVA_HOME is not set it will default to the bundled version.

Apache Tomcat 5.0:
CATALINA_HOME must be set to a valid Tomcat 5.0 installation directory. TC_JAVA_HOME must be set to a valid JDK installation directory. Note that a JDK is required for Tomcat 5.0, as opposed to a JRE.

BEA WebLogic 8.1:
BEA_HOME must be set to a valid BEA 8.1 installation directory. TC_JAVA_HOME must be set to a valid JDK 1.4 installation directory. Note that WLS 8.1 does not work with Java 5.

Web applications that you import into the Configurator are deployed to the currently selected web container. The Terracotta provided sample web applications are pre-deployed to each of the supported web containers.

I know Ubuntu isn't a supported platform, but do you have any advice for running Terracotta on it?

The known issues when trying to run Terracotta on Ubuntu are:

1. Default shell is dash bash. Terracotta scripts don't behave under dash. You might solve this issue by setting your default shell to bash or changing /bin/sh in our scripts to /bin/bash.

1. The Ubuntu default JDK is from GNU. Terracotta supports only the Sun JDKs.

PERFORMANCE TUNING...

This section is under development

My application runs very slowly on Terracotta. What's going on?

One of the main reasons applications run slowly on Terracotta is due to lock configuration. You must protect critical sections of your code with distributed locks (i.e., synchronization or named locks). If you hold a lock around more code than you need, then you will have high contention for that lock and reduced concurrency. If that is the case, try to minimize the scope of the lock.

If you have too much synchronization or grab more locks than you need to, you will suffer from the overhead of grabbing distributed locks over the network. If that is the case, try to synchronize only where you need to to reduce the number of times you request distributed locks.

Why are the Terracotta server instances in my cluster running very slowly?

If you turn off disk "write" caching on the machines running Terracotta server instances, they'll suffer a substantial performance degradation. Disk caching is often on by default, but may have been turned off in response to this issue.