Troubleshooting Guide
If your question isn't answered here, try one of the following resources:
Contents
Exceptions and Errors...
On Windows, I get the following error message right after the first "preparing to install" dialog box finishes: Windows error 3 occurred while loading the Java VM. What does this mean and how do I get around this?
You probably have several versions of the Java SDK and runtime installed and something in the Windows registry entries on your machine is confusing the installer. (Even if the JAVA_HOME environment variable set correctly). To get past this issue, clean up your environment or add the "LAX_VM" parameter and the location of the correct Java executable to the command line, like this:
terracotta-windows-2.6.0.exe LAX_VM "c:/jdk1511/bin/java.exe"
I am getting a message that says "The DSO boot JAR you need for this platform does not exist. You may need to run the 'make-boot-jar' script." What does that mean?
You need to create the boot-jar. Terracotta ships with some standard boot-jars - this means though that the shipped boot jar does not match with the JDK version and/or OS your environment is running. The following command will create the appropriate boot-jar for your environment to fix this problem. You should have to run this only once, until something relevant in the environment changes (e.g. upgrade the JVM).
%TC_INSTALL_DIR%\dso\bin\make-boot-jar.bat
$TC_INSTALL_DIR/dso/bin/make-boot-jar.sh
 | Make sure to delete the existing boot-jar files under TC_INSTALL_DIR/lib/dso-boot as the make-boot-jar script is sometimes unable to replace existing files. |
The -o option (output) tells the script where to create the boot-jar and $TC_INSTALL_DIR%/common/lib/dso-boot/ is the default location. If you omit the -o option, the boot JAR will be generated in the default location.
See the Concept and Architecture Guide
for more information on what the boot JAR is.
I am getting com.tc.exception.TCNonPortableObjectError. What next?
|
For a detailed guide on how to configure Terracotta DSO for your application, including how to handle TCNonPortableObjectErrors, see Configuring Terracotta. |
There can be a number of reasons to encounter this exception. This exception basically means that one (or more) of the objects in the shared object graph reference a Non-Portable object. A Non-Portable object is an object that Terracotta cannot cluster or share. An object can be non-portable in one of the following scenarios:
1. Object in question is implicitly non-portable, i.e. cannot be clustered. In other words, it doesn't make sense to share these objects across JVMs - examples include objects that hold a resource native to the OS/Hardware (File Handle, Sockets etc.). If a JVM is holding on to a physical socket on one machine, this socket does not hold any significance to another JVM on another machine, and hence it does not make sense to share it.
- Solution: Mark these fields as "transient" in the Terracotta config. A transient field is an instance variable that is referenced by a shared object, but is NOT shared by Terracotta.
There are two ways to do this. If the field in question is already marked transient in the containing class, you can turn on transient behavior for the class (note that you can only do so for the entire class. You may want to fill in the on-load section as well - it behaves similarly to the readObject() Java Serialization API).
<include>
<class-expression>com.mycompany.pkga.MyClassOne</class-expression>
<honor-transient>true</honor-transient>
<!-- optional -->
<on-load>
<execute><![CDATA[...]]></execute>
</on-load>
</include>
If the field in question is not already marked transient, or you need more fine grained control for the fields in a particular class, you can name individual fields to be treated as transient:
<transient-fields>
<field-name>com.mycompany.pkga.MyClassOne.fieldA</field-name>
<field-name>com.mycompany.pkgb.subpkg2.fieldB</field-name>
</transient-fields>
 | Tip
Marking a field transient can lead to a NullPointerException in your clustered application. Let's look at the following scenario: An object, foo, is a shared object. Foo references an instance of java.io.File called file. You decide to mark file as transient. Now, in a clustered environment, when foo materializes on multiple JVMs, since file is not shared it is going to be null unless there is code to initialize file on each shared JVM. This can be done in the constructor of foo or by using the Bean Shell script in the Terracotta config. |
| Common Declared Transients
Here are some common classes whose instances, if referenced by shared objects, will most likely need to be declared transient: Loggers, File Handlers, Sockets, java.io., java.net., java.nio., java.sql.Connection, javax.net., javax.print.*, java.lang.reflect, java.lang.Thread, java.lang.Runtime. |
See the Concept and Architecture Guide for details on portability and transience.
2. The class that implements the field has not been instrumented.
- Solution: Include the class in the instrumentation section of the tc-config.xml file, as in the following example.
<instrumented-classes>
<include>
<class-expression>com.mycompany.pkga.*</class-expression>
</include>
</instrumented-classes>
3. The class that implements the field has not been instrumented and requires to be in the boot-jar.
- Solution: Add the class to the <additional-boot-jar-classes> section of the configuration file and regenerate the boot-jar (see Q9).
In some cases, Terracotta must add explicit support for a core Java class — simply adding to the boot-jar does not work. Before you contact Terracotta
, see Unsupported Classes for a list of unsupported classes.
I am getting com.tc.object.tx.UnlockedSharedObjectException. What next?
|
For a detailed guide on how to configure Terracotta DSO for your application, including how to handle UnlockedSharedObjectExceptions, see Configuring Terracotta. |
This exception means that some shared object is being mutated or modified without being protected. Preventing unlocked shared objects from being changed ensures that the cluster state is always coherent. In Terracotta, data integrity and cluster coherency is maintained by ensuring atomic operations on shared objects. This is achieved through plain and natural Java concurrency constructs — every mutation to a shared object must be done within a synchronized block of code. This synchronized block of code is the Terracotta transaction boundary. In coding terms, the following two steps will fix this exception:
1. Define a synchronization boundary around the object mutation.
For example, suppose the following code block is the cause of the exception, where foo is the object being mutated in the method mutate:
public void mutate(Object foo) {
foo.setProperty(property);
}
The exception can be prevented by making the following change:
public void mutate(Object foo) {
synchronized(foo) {
foo.setProperty(property);
}
}
2. Configure Terracotta to use an appropriate lock for the mutate transaction above by adding the following in the locks section of the Terracotta configuration file:
<locks>
<autolock>
<method-expression>void com.system.services.MutateService.mutate(Object)</method-expression>
<lock-level>write</lock-level>
</autolock>
</locks>
Locks are declared at the method level. Be sure to use write locks for write operations (for example, set()) and read locks for read operations (for example get()) for better performance.
I get com.tc.object.tx.UnlockedSharedObjectException on a third party jar and do not have access to code. Is Terracotta any good for me?
 | Use Sparingly
There can be a performance penalty to using named locks, so it highly recommended that you use named locks very sparingly and only where absolutely necessary. |
Yes. You can use "named" locks instead of autolocks demonstrated in the previous query. Named locks will create one global lock across the cluster for the named lock. This can cause contention, and can, in some cases, decrease throughput. When using named locks, you do not need to use the synchronize keyword to isolate the transaction (step 1 in the previous query is not needed) because the method becomes the point of synchronization, and the name of the method becomes the name of the lock. Due to there only being one named lock per method, ALL object instances will serialize access through the single named lock.
Here is an example:
<locks>
<named-lock>
<lock-name>mutateFoo</lock-name>
<method-expression>void com.system.services.MutateService.mutate(Object)</method-expression>
<lock-level>write</lock-level>
</named-lock>
</locks>
See the Concept and Architecture Guide for more information about locks.
See the Configuration Guide and Reference for more information on how to configure locks.
I am getting java.io.IOException: No locks available, what next?
This is a problem with NFS lock mgr.
- Solution: modify tc-config.xml to use different log files
Why am I getting a Hyperic (Sigar) exception and why is the Terracotta Administrator Console not showing certain metrics?
These two problems are related to starting Java from a location different than the value of JAVA_HOME. To avoid the Hyperic error and restore metrics to the Terracotta Administrator Console, invoke Java from the location specified by JAVA_HOME.
Unable to locate bundle dependency: I get the following exception which indicates one or more Terracotta modules or dependencies are missing. How do I get past this scenario?
| Missing Terracotta Module
2008-01-18 09:47:55,240 FATAL - Unable to locate bundle dependency: 'jdk15_preinst_config', version '2.5.0', group-id 'org.terracotta.modules'; Tried to resolve the location using the following repositories: 'file:/www/terracotta/terracotta-2.5.0/bin/../modules/' |
On Solaris platform:
This is a result of the way the default tar utility on Solaris works. Terracotta loads a few modules located in $TC_INSTALL_DIR/modules directory on startup and if you look in that directory, you should see a collection of .jar files. The default tar utility causes some of these file names to be truncated which results in the above scenario as Terracotta is unable to load these modules.
To get past this, use an alternative tar utility like gtar or download the patch from Sun which fixes this issue.
More generic solution:
If it is not an unpacking issue and the kit is installed properly, check the following:
- The user has read access to all files in the Terracotta installation (specifically the modules directory).
- Make sure that the module specified in the tc-config.xml is actually located and read-accessible in the appropriate repository (default is $TC_INSTALL_DIR/modules).
- Make sure that the module version number is specified appropriately in the tc-config.xml and there are no typos.
Clustering Issues...
So I need to add a certain class to the boot-jar. How do I do it?
See Q3 to create the boot-jar. The difference here is that we will provide the configuration file with the additional classes that need to be added to the boot-jar as a command line parameter.
First, add the class in the configuration file in the <additional-boot-jar-classes> section:
<additional-boot-jar-classes>
<include>java.awt.datatransfer.Clipboard</include>
</additional-boot-jar-classes>
Next, provide the configuration file with the above entry as a parameter to the make-boot-jar script:
%TC_INSTALL_DIR%\dso\bin\make-boot-jar.bat -f tc-config.xml
$TC_INSTALL_DIR/dso/bin/make-boot-jar.sh -f tc-config.xml
| Make sure to delete the existing boot-jar files under TC_INSTALL_DIR/lib/dso-boot as the make-boot-jar script is sometimes unable to replace existing files. |
I did everything I was supposed to do and nothing is being clustered or shared. What next?
Is the Terracotta Server running? Look at the client logs for exceptions. Chances are there was some exception which is most likely covered in this document or in our documentation section. If there are no exceptions and you see something similar to the following:
Terracotta, version 2.2 as of 20061203-151234
line in the client logs, check if the clients show up in the Admin Console (see point 2 in Q6). If clients are showing up but the roots are not, make sure that the fully qualified name of the root is accurate (e.g. org.system.services.foo). If you still have problems, please contact our mailing list (tc-users@lists.terracotta.org).
Why is the Terracotta Administrator Console timing out when it tries to connect to a Terracotta server?
If you've verified that your Terracotta cluster is up and running, but your attempt to monitor it remotely using the Terracotta Administrator Console is unsuccessful, a firewall may be the cause. Firewalls that block traffic from Terracotta servers' JMX ports prevent the Terracotta Administrator Console from seeing those servers. To avoid this and other connection issues that may also be attributable to firewalls, ensure that the JMX and DSO ports configured in Terracotta are unblocked on your network.
Terracotta for Spring...
How is logical identity for the Spring application context maintained?
Terracotta for Spring creates DSO root for each Spring application context. Because Spring framework
does not provide unique Id for the application context, matching patterns are used to select web
application name and to specify matching config names to identify application contexts
that need to be clustered.
Name of the DSO root is calculated based on the web application name and names of all config
resources used to initialize application context. If any name of the Spring configs changes or
if new configs been added, the generated id will be different.
See the following method for more details com.tcspring.DistributableBeanFactoryMixin.addLocation(String location)
When different configs are used but contexts need to have the same identity, the root name can be
assigned manually using the following configuration element:
<application>
<spring>
<jee-application>
<application-contexts>
<application-context>
<root-name>myClusteredSpringContext</root-name>
...
Spring application context is not distributed
Terracotta console or a web server logs shows that Spring application context is not distributed:
INFO [com.tcspring.ApplicationHelper] - Application name Tomcat.shared
INFO [com.tcspring.DistributableBeanFactoryMixin] - 09080A0C0905030C0E05070F0805050A Context is NOT distributed
To debug that you can enable DEBUG logging for org.springframework and or.tcspring categories using either
log4j settings for your web application, or if running on Tomcat global logging settings at <tomcat home>\conf\logging.properties
Then you can check the actual names for the web application and config resources. For example:
2007-12-27 18:33:43,507 INFO [com.tcspring.ApplicationHelper] - Application name Tomcat.context:/foo
2007-12-27 18:33:43,804 INFO [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] -
Loading XML bean definitions from ServletContext resource [/WEB-INF/spring/servlet.xml]
2007-12-27 18:33:45,710 INFO [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] -
Loading XML bean definitions from ServletContext resource [/WEB-INF/spring/dao.xml]
2007-12-27 18:33:45,804 INFO [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] -
Loading XML bean definitions from ServletContext resource [/WEB-INF/spring/services.xml]
2007-12-27 18:33:46,273 INFO [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] -
Loading XML bean definitions from ServletContext resource [/WEB-INF/spring/datasource.xml]
2007-12-27 18:33:46,335 INFO [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] -
Loading XML bean definitions from ServletContext resource [/WEB-INF/spring/cache.xml]
2007-12-27 18:33:47,257 INFO [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] -
Loading XML bean definitions from ServletContext resource [/WEB-INF/spring/captcha.xml]
2007-12-27 18:33:47,320 INFO [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] -
Loading XML bean definitions from ServletContext resource [/WEB-INF/spring/validators.xml]
2007-12-27 18:33:47,351 INFO [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] -
Loading XML bean definitions from ServletContext resource [/WEB-INF/spring/storedprocs.xml]
2007-12-27 18:33:47,648 INFO [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] -
Loading XML bean definitions from ServletContext resource [/WEB-INF/spring/pagers.xml]
2007-12-27 18:33:47,695 INFO [com.tcspring.DistributableBeanFactoryMixin] - 09080A0C0905030C0E05070F0805050A Matching
locations:[/WEB-INF/spring/servlet.xml, /WEB-INF/spring/dao.xml, /WEB-INF/spring/services.xml,
/WEB-INF/spring/datasource.xml, /WEB-INF/spring/cache.xml, /WEB-INF/spring/captcha.xml, WEB-INF/spring/validators.xml,
/WEB-INF/spring/storedprocs.xml, /WEB-INF/spring/pagers.xml]
2007-12-27 18:33:47,695 INFO [com.tcspring.DistributableBeanFactoryMixin]
- 09080A0C0905030C0E05070F0805050A Context is distributed
2007-12-27 18:33:47,695 INFO [com.tcspring.ClassHierarchyWalker] - 09080A0C0905030C0E05070F0805050A
registering include for com.planetpope.beans.SystemStatus
2007-12-27 18:33:47,695 INFO [com.tcspring.DistributableBeanFactoryMixin] - 09080A0C0905030C0E05070F0805050A
registering transient fields for systemStatus com.planetpope.beans.SystemStatus
In the above example application name is "foo" and corresponding configuration would look like this:
<application>
<spring>
<jee-application name="foo">
<application-contexts>
<application-context>
<paths>
<path>*/spring/servlet.xml
<path>*/spring/dao.xml</path>
<path>*/spring/services.xml</path>
<path>*/spring/datasource.xml</path>
<path>*/spring/cache.xml</path>
<path>*/spring/captcha.xml</path>
<path>*/spring/validators.xml</path>
<path>*/spring/storedprocs.xml</path>
<path>*/spring/pagers.xml</path>
</paths>
...
When context is distributed Terracotta admin console will also show special spring_info root for each clustered
context where you could see the web application name, matching config names and also clustered Spring bean names.
For example:
tc:spring_info:05000C00060B090F090D040B0D0D0501 (java.util.ArrayList) [4/4]
0 (com.tcspring.ApplicationContextEventProtocol)
com.tcspring.ApplicationContextEventProtocol.appName (java.lang.String)=jmx
com.tcspring.ApplicationContextEventProtocol.beanNames (java.util.ArrayList) [2/2]
0 (java.lang.String)=clusteredHistory
0 (java.lang.String)=clusteredCounder
com.tcspring.ApplicationContextEventProtocol.configs (java.util.ArrayList) [1/1]
0 (java.lang.String)=demo/jmx/applicationContext.xml
com.tcspring.ApplicationContextEventProtocol.eventTypes (java.util.ArrayList) [0/0]
Share data between standalone application and web application
This is not supported out of the box, but you could use an application wrapper to make
classloader names the same between standalone application and the web app.
TODO
Best Practices...
Is there a faster way to declare configurations?
The Eclipse plugin shipped with Terracotta makes the task of creating the Terracotta configuration seamless. You can also leverage the AspectWerkz pattern language for a regex approach to this task. For example, define read locks around all getters in a Class:
<locks>
<autolock>
<method-expression>* com.system.services.MutateService.get*(..)</method-expression>
<lock-level>read</lock-level>
</autolock>
</locks>
My code uses the singleton pattern. I get exceptions on restart.
In such a case please make sure that the root is the field in the Singleton and not a reference to the Singleton elsewhere in your code. Static fields are not replicated by Terracotta unless you explicitly mark them as roots (given that static fields belong to a "class" and not an "object" instance).
How do I...
How do I use my JRE instead of the one shipped with Terracotta?
Point the TC_JAVA_HOME environment variable to a supported JDK or JRE. You should point it to the top level installation directory of the JDK, which is the directory containing the bin directory.
How do I use Terracotta to cluster my POJOs? Is there an API?
There are three different ways to add clustering to your application using Terracotta:
- Add shared classes and to the Terracotta configuration file, tc-config.xml, and add locking to methods that read from or write to shared objects.
- Instrument classes directly in your code using annotations.
- Install a Terracotta Integration Module that includes the instrumented classes your application needs for clustering.
If your code is already optimized for data coherency by implementing synchronization designs provided by Java, Terracotta may already support those designs and little configuration may be necessary. For more information, see the Configuration Guide and Reference.
How do I know if my POJOs are being shared or not?
- The starting point would be that the client logs should display a line similar to this:
Terracotta, version 2.2 as of 20061203-151234
- A quick way to verify is to monitor your cluster using the Terracotta Admin Console (TC_INSTALL_DIR/dso/bin/tc-admin.bat(sh). In the frame on the left, you will see a tree display with one of the nodes being clients and another one being roots. When you bring up the client JVMs, your should see the IP of your client JVMs under clients and the root under roots.
How can I deploy terracotta in my clustered environment to cluster my plain java application, my application server, servlet container?
| Version 2.3 and higher
This answer assumes that the Terracotta version being used is 2.3 and higher. |
1. Install Terracotta (on all nodes). Set the TC_INSTALL_DIR environment variable to your installation directory and make sure that JAVA_HOME is set to the appriprate JDK/JRE that is being used for your application.
2. Create the boot-jar for your environment by running the make-boot-jar script. Make sure that the JAVA_HOME environment variable is pointing to the correct JDK/JRE.
$TERRACOTTA_INSTALL_DIR/bin/make-boot-jar.sh
%TERRACOTTA_INSTALL_DIR%\bin\make-boot-jar.bat
3. Create a Terracotta configuration (tc-config.xml).
- You can use the tc-config-*.xml samples located in TC_INSTALL_DIR\config-examples directory as a good starting point. Here is a good documentation of different properies in tc-config.xml: Configuration Guide and Reference.
- You can also refer to the tc-config-reference.xml located in the docs directory of the Terracotta installation.
Make sure that the server host property is set to the correct hostname/ip if you are running the Terracotta Server on a seprate box than your application.
<servers>
<!-- Tell DSO where the Terracotta server can be found. -->
<server host="localhost">
<data>%(user.home)/terracotta/server-data</data>
<logs>%(user.home)/terracotta/server-logs</logs>
</server>
</servers>
<servers>
<!-- Tell DSO where the Terracotta server can be found. -->
<server host="host1">
<data>%(user.home)/terracotta/server-data</data>
<logs>%(user.home)/terracotta/server-logs</logs>
</server>
</servers>
You can also use the %h wildcard which always uses the the local hostname of the machine on which the server started:
<servers>
<!-- Tell DSO where the Terracotta server can be found. -->
<server host="%h">
<data>%(user.home)/terracotta/server-data</data>
<logs>%(user.home)/terracotta/server-logs</logs>
</server>
</servers>
4. Start the Terracotta Server on some machine (e.g. host1) passing the configuration created in step 3 like this:
$TC_INSTALL_DIR/bin/start-tc-server.sh -f tc-config.xml
%TC_INSTALL_DIR%\bin\start-tc-server.bat -f tc-config.xml
5. Set the environment/system properties and java options necessary to integrate Terracotta in your application. You can use the dso-env script that will set the necessary environment variables/java options for you (in the TC_JAVA_OPTS variable):
TERRACOTTA_SERVER=localhost
TC_CONFIG_PATH="${TERRACOTTA_SERVER}:9510"
. $TC_HOME/bin/dso-env.sh -q
export JAVA_OPTS="${TC_JAVA_OPTS}"
java $JAVA_OPTS MainClass
** Replace localhost above with the hostname/ip of the machine
** where the Terracotta Server is started (Step 4).
setlocal
set TERRACOTTA_SERVER_HOST=localhost
set TC_CONFIG_PATH=%TERRACOTTA_SERVER_HOST%:9510
call "%TC_INSTALL_DIR%\bin\dso-env.bat" -q
set JAVA_OPTS=%TC_JAVA_OPTS%
java %JAVA_OPTS% MainClass
endlocal
** Replace localhost above with the hostname/ip of the machine
** where the Terracotta Server is started (Step 4).
TERRACOTTA_SERVER=localhost
TC_CONFIG_PATH="${TERRACOTTA_SERVER}:9510"
. $TC_HOME/bin/dso-env.sh -q
export JAVA_OPTS="${TC_JAVA_OPTS}"
startup.sh
** Replace localhost above with the hostname/ip of the machine where the
** Terracotta Server is started (Step 4).
setlocal
set TERRACOTTA_SERVER_HOST=localhost
set TC_CONFIG_PATH=%TERRACOTTA_SERVER_HOST%:9510
call "%TC_INSTALL_DIR%\bin\dso-env.bat" -q
set JAVA_OPTS=%TC_JAVA_OPTS%
call startup.bat
endlocal
** Replace localhost above with the hostname/ip of the machine
** where the Terracotta Server is started (Step 4).
Extra Information for the curious:
Alternatively, if you are someone who likes to know what is being set by the dso-env script to Terracotize your application, you can set it up manually also.
You will need to add the following three environment variables:
i. -Xbootclasspath/p:"%TC_HOME%\lib\dso-boot\dso-boot-hotspot_win32_150_08.jar"
** Replace the name of the file above (dso-boot-hotspot_win32_150_08.jar)
** with the file created in step 2.
** Make sure that the boot-jar file name reflects the same
** JDK/JRE major_minor version as the JDK/JRE you are
** going to use to run the application (JAVA_HOME or in your PATH).
** You can verify your java version by running 'java -version' on the command line.
ii. -Dtc.install-root="%TC_HOME%"
** Assuming TC_HOME is set to your installation path **
iii. -Dtc.config="localhost:9510"
** Replace localhost with the hostname/ip of the machine where the
** Terracotta Server was started in step 4 e.g. host1 or localhost if
** running on the same machine.
So, if your java invocation looks like this:
java -cp c:\lib\somejar.jar com.mypackage.MainClass
And it should look similar to this with Terracotta:
java -Xbootclasspath/p:"%TC_HOME%\lib\dso-boot\dso-boot-hotspot_win32_150_08.jar"
-Dtc.install-root="%TC_HOME%"
-Dtc.config="localhost:9510"
-cp c:\lib\somejar.jarcom.mypackage.MainClass
For your AppServer, you can just add the above java options/system properties to the JAVA_OPTS variable in the AppServer start script:
set JAVA_OPTS=-Xbootclasspath/p:"%TC_HOME%\lib\dso-boot\dso-boot-hotspot_win32_150_08.jar"
-Dtc.install-root="%TC_HOME%"
-Dtc.config="localhost:9510";%JAVA_OPTS%
export JAVA_OPTS="-Xbootclasspath/p:$TC_HOME/lib/dso-boot/dso-boot-hotspot_win32_150_08.jar
-Dtc.install-root=$TC_HOME -Dtc.config=localhost:9510":$JAVA_OPTS
| TODO
- ConcurrentModificationException during dehydrate
- Hashtable and Vector aren't autolocked
- Null pointer exceptions and transience
- bean-shell
- on-load methods
|
How do I get failover to work properly with 2 nodes, especially after pulling the ethernet cable (plug)?
See Configuring Terracotta for High Availability.
How do I improve the performance of my application which is using clustered EHCache with Terracotta?
Add the following property in the tc.properties file under $TC_INSTALL_DIR/lib (just create one if it doesn't exist):
ehcache.concurrency = 32
This property sets the concurrency (number of striped locks per cache instance) vis-a-vis java.util.concurrent.ConcurrentHashMap. Depending on your application, you will want to play around with this number. E.g. If you have just one cache instance with 128 threads, you will want to use a big number like 64 (or even 128). But if you have hundreds of caches, you have to be careful as too high a number (200 (caches) * 200 (locks) = 40000 ) can add more overhead in terms of memory and lock management. 32 or 64 should be good starting point for most applications. The default value is 1.
How do I know if EHCache data is being shared? I can see a bunch of EHCache objects in the admin console but cannot find my application data.
First thing to look for is the terracotta root objects. Connect the TC Admin Console ($TC_INSTALL_DIR/bin/admin.sh) to the TC Server instance and look under the Roots. If you see the following roots, EHCache (Hibernate 2nd level cache) is being clustered properly:
net.sf.ehcache.CacheManager.ALL_CACHE_MANAGERS (java.util.Collections$SynchronizedRandomAccessList)
net.sf.ehcache.store.MemoryStoreEvictionPolicy.LRU (net.sf.ehcache.store.MemoryStoreEvictionPolicy)
net.sf.ehcache.store.MemoryStoreEvictionPolicy.LFU (net.sf.ehcache.store.MemoryStoreEvictionPolicy)
net.sf.ehcache.store.MemoryStoreEvictionPolicy.FIFO (net.sf.ehcache.store.MemoryStoreEvictionPolicy)
net.sf.ehcache.store.MemoryStoreEvictionPolicy.DSO (net.sf.ehcache.store.MemoryStoreEvictionPolicy)
net.sf.ehcache.CacheManager.singleton (net.sf.ehcache.CacheManager)
EHCache breaks down the key-value pair and stores the value object wrapped by Element objects in a fairly deep object graph, as it does a lot of bookkeeping with timestamps etc. I am breaking down below exactly where you should be looking for the actual application data stored in the cache:
net.sf.ehcache.CacheManager.ALL_CACHE_MANAGERS (java.util.Collections$SynchronizedRandomAccessList)
|
|_____ java.util.Collections$SynchronizedCollection.c (java.util.ArrayList) [1/1]
|
|_____ 0 (net.sf.ehcache.CacheManager)
|
|_____ net.sf.ehcache.CacheManager.caches (java.util.HashMap) [3/3]
|
|_____ 0 (MapEntry)
|
|_____ value (net.sf.ehcache.Cache)
|
|_____ net.sf.ehcache.Cache.memoryStore (net.sf.ehcache.store.TimeExpiryMemoryStore)
(continued)
|
|_____ net.sf.ehcache.store.MemoryStore.map (net.sf.ehcache.store.TimeExpiryMemoryStore$SpoolingTimeExpiryMap)
|
|_____ com.tcclient.ehcache.TimeExpiryMap.timeExpiryDataStore (com.tcclient.cache.CacheDataStore)
|
|_____ com.tcclient.cache.CacheDataStore.store (java.util.Map[])
|
|_____ 0 (java.util.HashMap) [1/1]
|
|_____ 0 (MapEntry)
|
|_____ key=XML_0K_1551
value (com.tcclient.cache.CacheData)
|
|_____ com.tcclient.cache.CacheData.value (net.sf.ehcache.Element)
|
|_____ net.sf.ehcache.Element.value (Value object type)
|
|_____ The value object put in the cache by the app
How do I reduce planned downtime with Terracotta?
Terracotta persists objects in a new format (field name/value pairings), that does not use Java serialization. It is decoupled from specifics of class versioning. For example - adding/deleting a field to a class will require no flush of Terracotta caches. Additional use-cases in the table below.
| Type of Upgrade |
Currently Supported |
Future Support |
Notes |
| Server JVM Upgrade |
No |
No |
Servers always run with the supplied JVM version. (Shut primary L2 and upgrade it, while live traffic goes to secondary L2 and vice-versa) |
| Client JVM Upgrade |
Yes |
Yes |
No issues here, the server is agnostic to the client JVM. The client JVM must be a supported platform. (Some testing required if the implementation of the core-class changes between 1 JVM and another - quite rare) |
| Terracotta Version Upgrade |
No |
Yes |
Future Terracotta versions will include rolling upgrade support. |
| Class Versioning - add/delete/modify methods |
Yes |
Yes |
Terracotta is only concerned with changes to the attributes stored in a class |
| Class Versioning - add a field |
Yes |
Yes |
Supported today. If you need to initialize the field you can use the onLoad feature. |
| Class Versioning - delete a field |
Yes |
Yes |
Supported today. If your class no longer references a field this does not cause any problems. (Note that the data is preserved).i.e. if Node1 has class with field f1 and Node2 has class without field f1, Node2 sees no f1 |
| Class Versioning - modify a field name |
No |
Yes |
This is similar to adding a new field - the onLoad mechanism will be used, however support will be needed to be able to retrieve the value of the old field (Also means old field is deleted) |
| Class Versioning - modify a field type |
Partial |
Yes |
This will work if the types are compatible, for example an int to an Integer, an int to a String etc. This will be fully supported in the future using the onSet feature. |
| Class Versioning - modify a field to a static |
No |
Yes |
No answer |
| Class Versioning - modify a field from a static |
Yes |
Yes |
This is equivalent to adding a new field. |
| Class Versioning - change the name of a class |
No |
Yes |
No answer |
| Change the type of a root |
No |
Yes |
No answer |
How do I use TC Sessions with Tomcat?
It is basically three steps:
- Install Terracotta on all nodes. Set the TC_INSTALL_DIR environment variable to your installation directory and make sure that JAVA_HOME is set to the appriprate JDK/JRE that is being used for your application.
1. Declare your web application (war) name in the configuration xml file:
<web-applications>
<web-application>your_web_app1</web-application>
<web-application>your_web_app2</web-application>
</web-applications>
2. Start the terracotta server on hostname:
$TC_INSTALL_DIR/sessions/bin/start-tc-server.sh -f path/to/config/xml/file
3. Start the clients (Tomcat instances) using the follwing script:
TERRACOTTA_SERVER=localhost
TC_CONFIG_PATH="${TERRACOTTA_SERVER}:9510"
. $TC_HOME/bin/dso-env.sh -q
export JAVA_OPTS="${TC_JAVA_OPTS}"
$CATALINA_HOME/bin/startup.sh
setlocal
set TERRACOTTA_SERVER=localhost
set TC_CONFIG_PATH=%TERRACOTTA_SERVER%:9510 -q
call "%TC_INSTALL_DIR%\bin\dso-env.bat"
set JAVA_OPTS=%TC_JAVA_OPTS%
call %CATALINA_HOME%\bin\startup.bat
endlocal
- Replace the localhost in TERRACOTTA_SERVER=localhost with the appropriate ip or hostname where the terracottaserver is running in step 2 (for production mode) or set the TC_CONFIG_PATH with the physical path to the terracotta config file used to start the server in step 2 (for development mode).
- If it complains about the dso-boot-jar, create one (see Q3).
- Alternatively, you can use the configurator (see Q15) that ships with the sessions product.
How do I use TC Sessions with JBoss Application Server?
It is basically three steps:
- Install Terracotta on all nodes. Set the TC_INSTALL_DIR environment variable to your installation directory and make sure that JAVA_HOME is set to the appriprate JDK/JRE that is being used for your application.
1. Declare your web application (war) name in the configuration xml file:
<web-applications>
<web-application>your_web_app1</web-application>
<web-application>your_web_app2</web-application>
</web-applications>
2. Start the terracotta server on hostname:
$TC_INSTALL_DIR/sessions/bin/start-tc-server.sh -f path/to/config/xml/file
3. Start the clients (JBoss instances) using the follwing script:
TERRACOTTA_SERVER=localhost
TC_CONFIG_PATH="${TERRACOTTA_SERVER}:9510"
. $TC_HOME/bin/dso-env.sh -q
export JAVA_OPTS="${TC_JAVA_OPTS}"
$JBOSS_HOME/bin/run.sh
setlocal
set TERRACOTTA_SERVER=localhost
set TC_CONFIG_PATH=%TERRACOTTA_SERVER%:9510 -q
call "%TC_INSTALL_DIR%\bin\dso-env.bat"
set JAVA_OPTS=%TC_JAVA_OPTS%
call %JBOSS_HOME%\bin\run.bat
endlocal
- Replace the localhost in TERRACOTTA_SERVER=localhost with the appropriate ip or hostname where the terracottaserver is running in step 2 (for production mode) or set the TC_CONFIG_PATH with the physical path to the terracotta config file used to start the server in step 2 (for development mode).
- If it complains about the dso-boot-jar, create one (see Q3).
- Alternatively, you can use the configurator (see Q15) that ships with the sessions product.
How do I use TC Sessions with WebLogic Application Server?
It is basically three steps:
- Install Terracotta on all nodes. Set the TC_INSTALL_DIR environment variable to your installation directory and make sure that JAVA_HOME is set to the appriprate JDK/JRE that is being used for your application.
1. Declare your web application (war) name in the configuration xml file:
<web-applications>
<web-application>your_web_app1</web-application>
<web-application>your_web_app2</web-application>
</web-applications>
2. Start the terracotta server on hostname:
$TC_INSTALL_DIR/sessions/bin/start-tc-server.sh -f path/to/config/xml/file
3. Start the clients (Weblogic instances) using the follwing script:
TERRACOTTA_SERVER=localhost
TC_CONFIG_PATH="${TERRACOTTA_SERVER}:9510"
. $TC_HOME/bin/dso-env.sh -q
export JAVA_OPTS="${TC_JAVA_OPTS}"
export WLS_USER=admin
export WLS_PW=admin
export PRODUCT_MODE=false
${BEA_HOME}/weblogic81/server/bin/startWLS.sh
setlocal
set TERRACOTTA_SERVER=localhost
set TC_CONFIG_PATH=%TERRACOTTA_SERVER%:9510 -q
call "%TC_INSTALL_DIR%\bin\dso-env.bat"
set JAVA_OPTS=%TC_JAVA_OPTS%
set WLS_USER=admin
set WLS_PW=admin
set PRODUCT_MODE=false
call %BEA_HOME%\weblogic81\server\bin\startWLS.cmd
endlocal
- WLS_USER, WLS_PW and PRODUCT_MODE environment variables in the above script are optional and may vary. You can start your individual domain instance instead of
${BEA_HOME}/weblogic81/server/bin/startWLS.sh
%BEA_HOME%\weblogic81\server\bin\startWLS.cmd
- Replace the localhost in TERRACOTTA_SERVER=localhost with the appropriate ip or hostname where the terracottaserver is running in step 2 (for production mode) or set the TC_CONFIG_PATH with the physical path to the terracotta config file used to start the server in step 2 (for development mode).
- If it complains about the dso-boot-jar, create one (see Q3).
- Alternatively, you can use the configurator (see Q15) that ships with the sessions product.
How do I use the TC Sessions Configurator for any container/app server?
Terracotta Sessions Configurator is preconfigured to use a bundled Apache Tomcat web container. Other web containers are available from the File | Servers menu option.
|
If you want to use a customized web container, do not use Sessions Configurator. Check the Integrations page to see if a setup guide is available for your web container. |
Each web container generally requires two properties for successful operation. The first is the location of the product home directory (such as CATALINA_HOME or BEA_HOME) and the second is the location of the Java Development Kit (JDK) to use when running the web container (speciffied by the TC_JAVA_HOME property).
Apache Tomcat 5.5
If CATALINA_HOME is not set it will default to the bundled version. If TC_JAVA_HOME is not set it will default to the bundled version.
Apache Tomcat 5.0:
CATALINA_HOME must be set to a valid Tomcat 5.0 installation directory. TC_JAVA_HOME must be set to a valid JDK installation directory. Note that a JDK is required for Tomcat 5.0, as opposed to a JRE.
BEA WebLogic 8.1:
BEA_HOME must be set to a valid BEA 8.1 installation directory. TC_JAVA_HOME must be set to a valid JDK 1.4 installation directory. Note that WLS 8.1 does not work with Java 5.
Web applications that you import into the Configurator are deployed to the currently selected web container. The Terracotta provided sample web applications are pre-deployed to each of the supported web containers.
Performance Tuning
| This section is under development |
My application runs very slowly on Terracotta. What's going on?
One of the main reasons applications run slowly on Terracotta is due to lock configuration. You must protect critical sections of your code with distributed locks (i.e., synchronization or named locks). If you hold a lock around more code than you need, then you will have high contention for that lock and reduced concurrency. If that is the case, try to minimize the scope of the lock.
If you have too much synchronization or grab more locks than you need to, you will suffer from the overhead of grabbing distributed locks over the network. If that is the case, try to synchronize only where you need to to reduce the number of times you request distributed locks.
