Terracotta Logo (http://www.terracotta.org)

Terracotta 3.2.1 Documentation

Table of Contents •  Back  •  Forward

Terracotta 3.2.1

Support

Services

Training


The content you have selected is available exclusively to Terracotta, Ehcache, and Quartz community members. If you are already a member, please login. If not, please take a moment to become one. Please sign up to join the Terracotta, Ehcache, and Quartz communities. Membership is free and only requires you to complete the form below. In addition to full documentation access, you will also be eligible to attend product briefings, download select early access product releases before the general public, post forum questions, file bug reports, submit patches and receive other member benefits still to come.

Login

 
Login

Register

All fields in bold are required.
(eight characters max)
Title:
City:
State/Province:
Country:
Phone:
Which technologies are you interested in or are already using?
 
Using
 
Hibernate
Ehcache
Distributed Caching
Quartz
Web Sessions Clustering
DSO (Distributed Shared Objects)
Please contact me
about Terracotta products, services, or training:
Please keep me updated on important Terracotta news:
 
Register
registering Register

Account Activated

Congratulations, your account has been activated. Thanks for joining the Terracotta, Ehcache, and Quartz community. You may now login to all of the the community resources, including:

  • forums
  • project issue trackers
  • community content

Account Registration Successful

(But you're not quite done—you still need to validate your account. Please check your email!)

Thanks for joining the Terracotta and Ehcache community. You will recieve an e-mail shortly containing instructions for validating your account and downloading Ehcache EX.

After you validate your account, you will have full access to all Terracotta and Ehcache community resources, such as:

6.3 Terracotta Server Arrays

This document shows you how to architect a Terracotta server array to add cluster reliability, availability, and scalability.

Terracotta server arrays can vary from a basic two-node tandem to a multi-node array providing configurable scale, high performance, and deep failover coverage.

The main features of a Terracotta server array include:

6.3.1 Definitions and Functional Characteristics

The major components of a Terracotta server array are:

6.3.1a Figure 1: A Server Array With 3 Mirror Groups

Figure 1 illustrates a Terracotta cluster with three mirror groups. Each mirror group has an active server and a standby, and manages one third of the shared data in the cluster.

A Terracotta cluster has the following functional characteristics:

  • Each mirror group automatically elects one active Terracotta server instance.
  • There can never be more than one active server instance per mirror group, but there can be any number of standbys. In Figure 1: A Server Array With 3 Mirror Groups, Mirror Group 1 could have two standbys, while Mirror Group 3 could have four standbys.

  • Every mirror group in the cluster must have a Terracotta server instance in active mode before the cluster is ready to do work.
  • The shared data in the cluster is partitioned and distributed to the mirror groups.
  • The number of partitions equals the number of mirror groups. In Figure 1: A Server Array With 3 Mirror Groups, each mirror group has one third of the shared data in the cluster.

  • Mirror groups CAN NOT provide failover for each other.
  • Failover is provided within each mirror group, not across mirror groups. This is because mirror groups provide scale by managing discrete portions of the shared data in the cluster -- they do not replicate each other. In Figure 1: A Server Array With 3 Mirror Groups, if Mirror Group 1 goes down, the cluster must pause (stop work) until Mirror Group 1 is back up with its portion of the shared data intact.

  • Active servers are self-coordinating among themselves.
  • No additional configuration is required to coordinate active server instances.

  • Only standby server instances can be hot-swapped in an array.
  • In Figure 1: A Server Array With 3 Mirror Groups, the L2 PASSIVE servers can be shut down, replaced, or joined by new L2 PASSIVE servers with no affect on cluster functions. However, to add or remove an entire mirror group, the cluster must be brought down.

6.3.2 Server Array Configuration Tips

To successfully configure a Terracotta server array using the Terracotta configuration file, note the following:

For more information on Terracotta configuration files, see:

6.3.2a Backing Up Persisted Shared Data

Certain versions of Terracotta provide tools to create backups of the Terracotta server array's disk store. See the Terracotta Operations Center and the Database Backup Utility (backup-data) for more information.

6.3.3 Terracotta Cluster in Development

6.3.3a Persistence: No | Failover: No | Scale: No

In a development environment, persisting shared data is often unnecessary and even inconvenient. It puts more load on the server, while accumulated data can fill up disks or prevent automatic restarts of servers, requiring manual intervention. Running a single-server Terracotta cluster without persistence (see Figure 2: Single Server – No Persistence of Shared Data) is a good solution for creating a more efficient development environment.

6.3.3b Figure 2: Single Server – No Persistence of Shared Data

By default, a single Terracotta server is in "temporary-swap-mode", which means it lacks persistence. Its configuration could look like the following:

<?xml version="1.0" encoding="UTF-8" ?>
<tc:tc-config xmlns:tc="http://www.terracotta.org/config"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.terracotta.org/schema/terracotta-5.xsd">
  <servers>
    <server name="Server1">
      <data>/opt/terracotta/server1-data</data>
      <l2-group-port>9530</l2-group-port>
    </server>
   <servers>
  ...
</tc:tc-config>

If this server goes down, all application state placed in the clustered shared heap is lost.

6.3.4 Terracotta Cluster With Reliability

6.3.4a Persistence: Yes | Failover: No | Scale: No

The "unreliable" configuration above may be advantageous in development, but if shared in-memory data must be persisted, the server's configuration must be expanded:

<?xml version="1.0" encoding="UTF-8" ?>
<tc:tc-config xmlns:tc="http://www.terracotta.org/config"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.terracotta.org/schema/terracotta-5.xsd">
  <servers>
    <server name="Server1">
      <data>/opt/terracotta/server1-data</data>
      <l2-group-port>9530</l2-group-port>
      <dso>
      
   <!-- The persistence mode is "temporary-swap-only" by default, so it must be changed explicitly. -->
        <persistence>
          <mode>permanent-store</mode>
        </persistence>
        
      </dso>
    </server>
  </servers>
  ...
</tc:tc-config>

By changing the value of the <persistence> element's <mode> element to "permanent-store" (it's "temporary-swap-only" by default), the server now backs up all shared in-memory data to disk (see Figure 3: Reliable Single Server – Shared Data Persisted to Disk).

6.3.4b Figure 3: Reliable Single Server – Shared Data Persisted to Disk

6.3.5 Terracotta Server Array with High Availability

6.3.5a Persistence: Yes | Failover: Yes | Scale: No

The example illustrated in Figure 3: Reliable Single Server – Shared Data Persisted to Disk presents a reliable but not _highly available_ cluster. If the server fails, the cluster fails. There is no redundancy to provide failover. Adding a standby server adds availability because the standby is ready to take over immediately (see Figure 4: Available and Reliable – Failover, Persistence).

6.3.5b Figure 4: Available and Reliable – Failover, Persistence

In this array, if the active Terracotta server instance fails then the standby instantly takes over and the cluster continues functioning. No data is lost.

The following Terracotta configuration file demonstrates how to configure this two-server array:

<?xml version="1.0" encoding="UTF-8" ?>
<tc:tc-config xmlns:tc="http://www.terracotta.org/config"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://www.terracotta.org/schema/terracotta-5.xsd">
  <servers>
    <server name="Server1">
      <data>/opt/terracotta/server1-data</data>
      <l2-group-port>9530</l2-group-port>
      <dso>
        <persistence>
          <mode>permanent-store</mode>
        </persistence>
      </dso>
    </server>
    <server name="Server2">
      <data>/opt/terracotta/server2-data</data>
      <l2-group-port>9530</l2-group-port>
      <dso>
        <persistence>
          <mode>permanent-store</mode>
        </persistence>
      </dso>
    </server>
     <ha>
        <mode>networked-active-passive</mode>
           <networked-active-passive>
               <election-time>5</election-time>
           </networked-active-passive>
       </ha>
  </servers>
  ...
</tc:tc-config>

Terracotta automatically syncs the two servers. The active and passive servers must always be synced, allowing the passive to mirror the state of the active. The recommended <mode> in the <ha> section is "networked-active-passive" because it allows the active and passive servers to synchronize directly, without relying on a disk.

You can add more standby servers to this configuration by adding more <server> sections.

If the server instances are started at the same time, one is elected as active server while the others serve as standbys. If a server instance is started while an active server instance is present, it syncs up state from (or replicates shared data of) the active server instance before becoming a standby. A standby cannot become an active server instance during a failure until its state is fully synced up. If the active fails and more than one standby is present, an election determines the new active.

TIP:

Standbys can be hot swapped.

The active server instance carries the load of sending state to the standby during the sync process. The time taken to sync up is dependent on the amount of data to be synced up and on the current load on the cluster. The active server instance and standbys should be run on similarly configured machines for better throughput, and should be started together to avoid unnecessary sync ups.

Terracotta server instances acting as standbys can run either in persistent mode or non-persistent mode. If an active server instance running in persistent mode goes down, and a standby takes over, the data directory must be cleared before bringing back the crashed server. Removing the data is necessary because the cluster state could have changed since the crash. The new state is taken from the new active server instance when the crashed server comes back up. A crashed standby running in persistent mode, however, automatically recovers.

NOTE:

Under certain circumstances, standbys may fail to automatically clear their data directory and fail to restart, generating errors. In this case, the data directory must be manually cleared.

6.3.6 Scaling the Terracotta Server Array

6.3.6a Persistence: Yes | Failover: Yes | Scale: Yes

For capacity requirements that exceed the capabilities of an two-server active-passive setup, expand the Terracotta server array using a mirror-groups configuration. Using mirror groups with multiple coordinated active Terracotta server instances adds scalability to server array.

Scalable server arrays are available in enterprise versions of Terracotta.

6.3.6b Figure 5: Available, Reliable, and Scalable – Failover, Persistence, Capacity

Mirror groups are specified in the <servers> section of the Terracotta configuration file. Mirror groups work by assigning group memberships to Terracotta server instances. The following snippet from a Terracotta configuration file shows a mirror-group configuration with four servers:

...
 <servers>
   <server name="server1">
   ...
   </server>
   <server name="server2">
   ...
   </server>
   <server name="server3">
   ...
   </server>
   <server name="server4">
   ...
   </server>
   <ha>
    <mode>networked-active-passive</mode>
    <networked-active-passive>
       <election-time>5</election-time>
    </networked-active-passive>
   </ha>
   <mirror-groups>
     <mirror-group group-name="group1">
       <members>
         <member>server1</member>
         <member>server2</member>
       </members>
     </mirror-group>
     <mirror-group group-name="group2">
       <members>
         <member>server3</member>
         <member>server4</member>
       </members>
     </mirror-group>
   </mirror-groups>
 </servers>
...

In this example, the cluster is configured to have two active servers, each with its own standby. If server1 is elected active in group1, server2 becomes its standby. If server3 is elected active in group2, server4 becomes its standby. server1 and server2 automatically coordinate their work managing Terracotta clients and shared data across the cluster.

See the mirror-groups section in the Configuration Guide and Reference for more information on mirror-groups configuration elements.

In a Terracotta server array designed for multiple active Terracotta server instance, the server instances in each mirror group participate in an election to choose the active. Once every mirror group has elected an active server instance, all the active server instances in the cluster begin cooperatively managing the cluster. The rest of the server instances become standbys for the active server instance in their mirror group. If the active in a mirror group fails, a new election takes place to determine that mirror group's new active. Clients continue work without regard to the failure.

When a server instance is started in a mirror group while an active server instance is present, it syncs up state from the active server instance before becoming a standby. A standby cannot become an active server instance during a failure until its state is fully synced up.

TIP:

Hot standbys can be hot swapped.

The active server instance in a mirror group carries the load of sending state to the standby during the sync process. The time taken to sync up is dependent on the amount of data to be synced up and on the current load on the cluster. The active server instance and standbys should be run on similarly configured machines for better throughput, and should be started together to avoid unnecessary sync ups.

Terracotta server instances acting as standbys can run either in persistent mode or non-persistent mode. If an active server instance running in persistent mode goes down, and a standby takes over, the data directory must be cleared before bringing back the crashed server. Removing the data is necessary because the cluster state could have changed since the crash. The new state is taken from the new active server instance when the crashed server comes back up. A crashed standby running in persistent mode, however, automatically recovers.

NOTE:

Under certain circumstances, standbys may fail to automatically clear their data directory and fail to restart, generating errors. In this case, the data directory must be manually cleared.

High-availability configuration can be set per mirror group. See the mirror-groups section in the Configuration Guide and Reference for more information.


Top of 6.3 Terracotta Server Arrays