This site hosts historical documentation. Visit www.terracotta.org for recent product information.

Commercial Products
Open Source

Securing Terracotta Clusters with SSL

Introduction

Secure Sockets Layer (SSL) encryption and certificate-based authentication can be used to secure enterprise versions of Terracotta clusters.

Security in a Terracotta cluster includes both server-server connections and client-server connections. Note that all connections in a secured cluster must be secure; no insecure connections are allowed, and therefore security must be enabled globally in the cluster. It is not possible to enable security for certain nodes only.

Overview

Security is set up using Terracotta configuration, tools provided in the Terracotta kit, standard Java tools, and public key infrastructure (via standard digital X.509 digital certificates).

Each Terracotta server utilizes the following types of files to implement security:

  • Java keystore – Contains the server's private key and public-key certificate. Protected by a keystore/certificate-entry password.
  • Truststore – A keystore file containing only the certificates' public keys. Needed only if using self-signed certificates, not a Certificate Authority (CA).
  • Keychain – Stores passwords, including to the server's keystore and to entries in these other files. The tools for creating and managing the Terracotta keychain file are provided with the Terracotta kit.
  • Authorization – A .ini file with password-protected user accounts and their roles for servers and clients that connect to the server.
TIP: Secure cacerts File
The standard Java cacerts file, located in ${JAVA_HOME}java.home/lib/security by default, is a system-wide repository for CA root certificates included with the JDK. These certificates can play a part in certificate chains. Java documentation recommends that the cacerts file be protected by changing its default password and file permissions.

Each Terracotta client also has a keychain file that stores the password it uses to authenticate with the server.

All files are read on startup. Changes made to the files after startup cannot be read unless the cluster is restarted.

The following diagram illustrates the flow of security information—and that information's origin in terms of the files described above—during initial cluster connections:

IMAGE: Security flow

From a Terracotta server's point of view, security checks take place at startup and at the time a connection is made with another node on the cluster:

  1. At startup, server1 requires a password to be entered directly from the console to complete its startup process.
  2. A connection request from server2 initiates the process of establishing a secure connection using SSL.
  3. Server1 authenticates server2 using stored credentials. Credentials are also associated with a role that authorizes server2. The process is symmetrical: server2 authenticates and authorizes server1.
  4. A connection request from a Terracotta client initiates the process of establishing a secure connection using SSL.
  5. Server1 authenticates and authorizes the client using stored credentials and associated roles. Because clients may be communicating with any active server in the cluster during their lifetimes, the client must be able to authenticate with any active server. Clients should be able to authenticate against all servers in the cluster, since active servers may fail over to mirror servers.

From a Terracotta client's point of view, security checks occur at the time the client attempts to connect to an active server in the cluster:

  1. The client uses a server URI that includes the client username.

    A typical (non-secure) URI is <server-address>:<port>. A URI that initiates a secure connection takes the form <client-username>@<server-address>:<port>.

  2. A secure connection using SSL is established with the server.

  3. The client sends a password fetched from a local keychain file. The password is associated with the client username.

The following configuration snippet is an example of how security could be set up for the servers in the illustration above:

<tc:tc-config xmlns:tc="http://www.terracotta.org/config">
...
  <servers secure="true"> 
    <server host="172.16.254.1" name="server1">
    ...
      <security>
        <ssl>
          <certificate>jks:server1alias@/the/path/keystore-file.jks</certificate>
        </ssl>
        <keychain>
          <class>com.terracotta.management.keychain.FileStoreKeyChain</class>
          <url>file:///%(user.dir)/server1keychain.tkc</url>
        </keychain>
        <auth>
          <realm>com.tc.net.core.security.ShiroIniRealm</realm>
          <url>file:///%(user.dir)/my_auth.ini</url>
          <user>server1username</user>
        </auth>
      </security>
   ...
   </server>
    <server host="172.16.254.2" name="server2">
    ...
      <security>
        <ssl>
          <certificate>jks:server2alias@/the/path/keystore-file.jks</certificate>
        </ssl>
        <keychain>
          <class>com.terracotta.management.keychain.FileStoreKeyChain</class>
          <url>file:///%(user.dir)/server2keychain.tkc</url>
        </keychain>
        <auth>
          <realm>com.tc.net.core.security.ShiroIniRealm</realm>
          <url>file:///%(user.dir)/my_auth.ini</url>
          <user>server2username</user>
        </auth>
      </security>
   ...
   </server>
   ...
 </servers>
...
</tc:tc-config>

See the configuration section below for more information on the configuration elements in the example.

Setting Up Server Security

To set up security on a Terracotta server, follow the steps in each of these procedures:

NOTE: Script names in the examples given below are for *NIX systems. Equivalent scripts are available for Microsoft Windows in the same locations. Simply substitute the .bat extension for the .sh extension shown and convert path delimiters as appropriate.

Create the Server Certificate and Add It to the Keystore

Each Terracotta server must have a keystore file containing a digital certificate and the associated private key. Your organization may already have policies and procedures in place regarding the generation and use of digital certificates and certificate chains, including the use of certificates signed by a CA.

IMPORTANT SECURITY CONSIDERATION!: Self-signed certificates might be less safe than CA-signed certificates because they lack third-party identity verification and do not carry a digital signature from an official CA. Your organization might already have policies and procedures in place regarding the generation and use of digital certificates and certificate chains, including the use of certificates signed by a Certificate Authority (CA). To follow your organization's policies and procedures regarding using digital certificates, you might need to adjust the procedures outlined in this document.

When used for a Terracotta server, the following conditions must be met for certificates and their keystores:

  • The keystore must be a Java keystore (JKS) compatible with JDK 1.6 or higher.
  • The certificate must be keyed with the alias named in the value of the <certificate> element of the server's configuration.
  • The Common Name (CN) field in the Distinguished Name must contain the hostname of the server, as configured in the server's configuration.
  • The password securing the certificate must match the keystore's main password. In other words, the store password and key passwords must be identical.
  • When using a self-signed certificate (not one signed by a trusted CA), create a custom truststore for storing public keys.
  • If security is not the most important consideration, you can set the following Java system properties as shown:
    • -Dtc.ssl.trustAllCerts=true (use any certificates)
    • -Dtc.ssl.disableHostnameVerifier=true (bypass hostname verification)

If you have a keystore in place, but the server certificate is not already stored in the keystore, you must import it into the keystore. If the keystore does not already exist, you must create it.

Creating Self-Signed Certificates Using Java Keytool

For testing purposes, or if you intend to use self-signed certificates, use the Java keytool command to create the public-private key pair. You also use this command to create keystores and truststores, but note that keytool refers to truststores as "keystores" since there is only a logical difference.

Specifying a Custom Truststore
Note that if you are not using cacerts, the default Java truststore, the custom truststore must be specified with the javax.net.ssl.trustStore system property. In this case, you can choose to reset the custom truststore's default password with javax.net.ssl.trustStorePassword.

The following could be used to create both public-private keys (including a certificate) and a keystore file for the server called "server1" in the configuration example above:

keytool -genkey -keystore keystore-file.jks -dname "CN=172.16.254.1, OU=Terracotta, O=SAG, L=San Francisco, S=California, C=US" -alias server1alias -storepass server1pass -keypass server1pass

Note that the values passed to -storepass and -keypass match. Also, the field designating the Common Name (CN) must match the server's hostname, which matches the value entered in the server's configuration. This hostname can be an IP address or a resolvable domain name. If the -dname option is left out, a series of identity prompts (distinguished-name fields based on the X.500 standard) will appear before the server's entry is created in the keystore. The CN prompt appears as shown:

What is your first and last name?
   [Unknown]:

There are a number of other keytool options to consider, including -keyalg (cryptographic algorithm; default is DSA) and -validity (number of days until the certificate expires; default is 90). These and other options are dependent on your environment and security requirements. See the JDK documentation for more information on using the keytool.

Create a keystore and entry on each Terracotta server.

Exporting and Importing Certificates

Each server should have a copy of each other server's public-key certificate in its truststore.

The following could be used to export the certificate of the server called "server1" in the configuration example above:

keytool -export -alias server1alias -keystore keystore-file.jks -file server1SelfSignedCert.cert

This "cert" file can now be used to import server1's certificate into the truststore of every other server. For example, to create a truststore and import server1's certificate on server2, copy the cert file to the working directory on server2 and use the following command:

keytool -import -alias server1alias -file server1SelfSignedCert.cert -keystore truststore.jks

After the password prompt, information about the certificate appears, and you are prompted to trust it. You must repeat this process until every server has a truststore containing the public-key certificate of every other server in the cluster.

TIP: Use a Single Truststore
Instead of recreating the truststore on every server, create a single truststore containing every server's public key, then copy that to every server. This same truststore can also be used for clients..

Set Up the Server Keychain

The keystore and each certificate entry are protected by passwords stored in the server keychain file. The location of the keychain file is specified in the value of the <url> element under the <keychain> element of the server's configuration file.

For example, with this server configuration, when the server starts up, the keychain file would be searched for in the user's (process owner's) home directory. In the configuration example, a keychain file called server1keychain.tkc is searched for when server1 is started.

The keychain file should have the following entries:

  • An entry for the local server's keystore entry.
  • An entry for every server that the local server will connect to.

Entries are created using the keychain script found in the Terracotta kit's bin directory.

Creating an Entry for the Local Server

Create an entry for the local server's keystore password:

bin/keychain.sh <keychain-file> <certificate-URI>

where <keychain-file> is the file named in the server configuration's <keychain>/<url> element, and <certificate-URI> is the URI value in the server configuration's <ssl>/<certificate> element. Note: The <certificate-URI> must match the server configuration's <ssl>/<certificate> element exactly, including the path to the keystore.

If the keychain file does not exist, add the -c option to create it:

bin/keychain.sh -c <keychain-file> <certificate-URI>

You will be prompted for the keychain file's password, then for a password to associate with the URI. For the URI, you must enter the same password used to secure the server's certificate in the keystore.

For example, to create an entry for server1 from the configuration example above, enter:

bin/keychain.sh server1keychain.tkc jks:server1alias@/the/path/keystore-file.jks

Terracotta Management Console - Keychain Client
Open the keychain by entering its master key:   xxxxxxx
Enter the password you wish to associate with this URL:   server1pass
Confirm the password to associate with this URL:   server1pass
Password for jks:server1alias@/the/path/keystore-file.jks successfully stored

Creating Entries for Remote Servers

Entries for remote servers have the format tc://<user>@<host>:<group-port>. Note that the value of <user> is specified in each server configuration's <security>/<auth>/<user> and is not related to the user running as process owner. If a value for <security>/<auth>/<user> is not specified, the username "terracotta" is used by default.

For example, to create an entry for server2 in server1's keychain, use:

bin/keychain.sh server1keychain.tkc tc://server2username@172.16.254.2:9530

If the keychain file does not exist, use the -c option:

bin/keychain.sh -c server1keychain.tkc tc://server2username@172.16.254.2:9530

You will be prompted for the keychain file's password, then for a password to associate with the entry server2username@172.16.254.2:9530.

An entry for server1 must also be added to server2's keychain:

bin/keychain.sh server2keychain.tkc tc://server1@172.16.254.1:9530

To use a copy of the same keychain file on every server, be sure to create an entry for the local server as well as every server that will connect to it.

Set Up Authentication

Servers and clients that connect to a secured server must have authentication (usernames/passwords) and authorization (roles) defined. By default, this is set up using the usermanagement script, located in the Terracotta kit's bin directory. This script also creates the .ini file that contains the required usernames and roles (associated passwords are stored in the keychain file).

All nodes in a secured Terracotta cluster must have an entry in the server's .ini file:

  • The local server itself
  • All other servers
  • All clients

Use the usermanagement script with the following format:

bin/usermanagement.sh -c <file> <username> terracotta

where -c is required only if the file does not already exist. For servers, the <username> will be used as the value configured in <security>/<auth>/<user>. For clients, the username must match the one used to start the client.

Note: While the "terracotta" role is appropriate for Terracotta servers and clients, the "admin" role is necessary for performing system functions such as stopping servers. For more information about roles, refer to the User Roles section.

For example:

# Create the .ini file and add a server username and role.
bin/usermanagement.sh -c my_auth.ini server1username terracotta

# Add another server.
bin/usermanagement.sh my_auth.ini server2username terracotta

# Add a client.
bin/usermanagement.sh my_auth.ini client1username terracotta

# Add a user with an admin (read/write) role.
bin/usermanagement.sh my_auth.ini admin1username admin

# Add a user with an operator (read) role.
bin/usermanagement.sh my_auth.ini console1username operator

The correct Shiro realm must be specified in the server configuration, along with the path to the .ini file:

...
<auth>
  <realm>com.tc.net.core.security.ShiroIniRealm</realm>
  <url>file:///%(user.dir)/my_auth.ini</url>
  <user>server1username</user>
</auth>
...

Configure Server Security

Security for the Terracotta Server Array is configured in the Terracotta configuration file (tc-config.xml by default). The following is an example security configuration:

<tc:tc-config xmlns:tc="http://www.terracotta.org/config">
...
  <servers secure="true"> 
    <server host="172.16.254.1" name="server1">
    ...
      <security>
        <ssl>
          <certificate>jks:server1alias@/the/path/keystore-file.jks</certificate>
        </ssl>
        <keychain>
          <class>com.terracotta.management.keychain.FileStoreKeyChain</class>
          <url>file:///%(user.dir)/server1keychain.tkc</url>
        </keychain>
        <auth>
          <realm>com.tc.net.core.security.ShiroIniRealm</realm>
          <url>file:///%(user.dir)/my_auth.ini</url>
          <user>server1username</user>
        </auth>
      </security>
   ...
   </server>
   ...
 </servers>
...
</tc:tc-config>

Every server participating in an SSL-based secured cluster must have a <security> block wherein security-related information is encapsulated and defined. The keystore, keychain, and .ini files named in the configuration must be available to every server in the cluster.

The following table defines selected security-related elements and attributes shown in the configuration example.

NameDefinitionNotes
secure Attribute in <servers> element. Enables SSL security for the cluster. DEFAULT: false. Enables/disables SSL-based security globally.
certificate Element specifying the location of the server's authentication certificate and its containing keystore file. The format for the certificate-keystore location is jks:alias@/path/to/keystore. "alias" must match the value used to key the certificate in the keystore file. Only the JKS type of keystore is supported.
class Element specifying the class defining the keychain file. If a class is not specified, com.terracotta.management.keychain.FileStoreKeyChain is used.
url The URI for the keychain file (when under <keychain>) or for the Realm configuration (.ini) file (when under <auth>). These URIs are passed to the keychain or realm class to specify the keychain or authentication file, respectively. These files are created and managed with the keychain and usermanagement scripts.
user The username that represents this server and is authenticated by other servers. This name is part of the credentials stored in the .ini file. Defaults to "terracotta".

Enabling SSL on Terracotta Clients

Terracotta clients do not require any specific configuration to enable SSL connections to a Terracotta Server Array.

NOTE: Script names in the examples given below are for *NIX systems. Equivalent scripts are available for Microsoft Windows in the same locations. Simply substitute the .bat extension for the .sh extension shown and convert path delimiters as appropriate.

To enable SSL security on the client, prepend the client username part to the Terracotta server URL the client uses to connect to the cluster. This should be the URL of an active server that has the client's credentials in its .ini file. The format of this security URL is <client-username>@<host>:<dso-port>. Prepending the username automatically causes the client to initiate an SSL connection.

If the client has username tcclient, for example, and attempts to connect to the server in the configuration example, the URI would be:

tcclient@172.16.254.1:9510

This type of URL replaces the non-secure URL (<host>:<dso-port>) used to start clients in non-SSL clusters.

Both the client username and the corresponding password must match the those in the server's .ini file. The username is included in the URL, but the password must come from a local keychain-file entry that you create.

NOTE: If Terracotta servers are using self-signed certificates (not certificates signed by a well-known CA), then you must specify a truststore for the client which contains the public key of every server in the cluster.

Create a Keychain Entry

The Terracotta client should have a keychain file with an entry for every Terracotta server in the cluster. The format for the entry uses the "tc" scheme:

tc://<client-username>@<host>:<dso-port>

An entry for the server in the example configuration should look like the following:

tc://client1@172.16.254.1:9510

Use the keychain script in the Terracotta kit to add the entry:

bin/keychain.sh clientKeychainFile tc://client1@172.16.254.1:9510

If the keychain file does not already exist, use the -c flag to create it:

bin/keychain.sh -c clientKeychainFile tc://client1@172.16.254.1:9510

If creating the keychain file, you will be prompted for a master password.

This entry in the client's keychain file serves as the key for the client's password, and will be provided to the server along with the client username ("client1" in the example). These credentials must match those in the server's .ini file.

The Terracotta client searches for the keychain file in the following locations:

  • %(user.home)/.tc/mgmt/keychain
  • %(user.dir)/keychain.tkc
  • The path specified by the system property com.tc.security.keychain.url

When you run the keychain script, the following prompt should appear:

Terracotta Management Console - Keychain Client
KeyChain file successfully created in clientKeychainFile
Open the keychain by entering its master key:

Enter the master key, then answer the prompts for the secret to be associated with the server URI:

Enter the password you wish to associate with this URL: 
Password for tc://client1@172.16.254.1:9510 successfully stored

Note that the script does not verify the credentials or the server address.

Using a Client Truststore

If Terracotta servers are using self-signed certificates (not certificates signed by a well-known CA), create a truststore on the client and import each server's public-key certificate into that truststore.

If you have already created a truststore for a server in the TSA, you can copy that file to each client after first importing that server's public-key certificate into the copy.

For the client to find the truststore, you must set the Java system property javax.net.ssl.trustStore to the location of the truststore file. In this case, note the existing secrets for opening the truststore and accessing each certificate.

TIP: Changing the Truststore Password
To change the existing truststore master password, use the Java system property javax.net.ssl.trustStorePassword.

Restricting Clients to Specified Servers (Optional)

By default, clients are not restricted to authenticate a specific set of servers when responding to REST requests. However, it is possible to explicitly list the servers that a client may respond to by using the <managementRESTService> element's securityServiceLocation attribute in the Ehcache configuration.

When this attribute is empty (or missing), no such restriction exists and the client will authenticate against any server in the cluster that meets the established security requirements. This is the recommended setting, as SSL connections and the authentication/authorization mechanism provide sufficient security.

In the case where an extra layer of security is required for the client's REST service, you can configure a list of allowed servers as follows:

<managementRESTService ... 
  securityServiceLocation=" https://my-l2-node1/tmc/api/assertIdentity ,  
     https://my-l2-node2/tmc/api/assertIdentity ">

where my-l2-node1 and my-l2-node2 are the servers' hostnames. Note, however, that any of the servers in a client's cluster can forward a REST request to that client at any time, and so all servers should be listed if this feature is used.

Running a Secured Server

Start a server in a secure Terracotta cluster using the start-tc-server script as usual. After initial startup messages, you will be prompted for the password to the server's keychain. Once the password is entered, the server should continue its startup as normal.

Confirm Security Enabled

You can confirm that a server in a number of ways:

  • Look for the startup message Security enabled, turning on SSL.
  • Search for log messages containing "SSL keystore", "HTTPS Authentication enabled", and "Security enabled, turning on SSL".
  • Attempt to make JMX connections to the server—these should fail.

Stopping a Secured Server

Stop a server in a secure Terracotta cluster using the stop-tc-server script with the following arguments:

  • -f <tc-config-file> — A valid path to the self-signed certificate must have been specified in the server's configuration file.
  • -u <username> — The user specified must have the "admin" role.
  • -w <password>

For more information about the stop script, refer to Start and Stop Server Scripts.

Troubleshooting

You may encounter any of the following errors at startup:

RuntimeException: Couldn't access a Console instance to fetch the password from!

This results from using "nohup" during startup. The startup process requires a console for reading password entry.

TCRuntimeException: Couldn't create KeyChain instance ...

The keychain file specified in the Terracotta configuration cannot be found. Check for the existence of the file at the location specified in <keychain>/<url>.

RuntimeException: Couldn't read from file ...

This error appears just after an incorrect password is entered for the keychain file.

RuntimeException: No password available in keyChain for ...

This error appears if no keychain password entry is found for the server's certificate. You must explicitly store the certificate password in the keychain file.

This error could also appear if the resolved hostname or IP address is different from the one in the keychain entry:

  • tc://terracotta@localhost:9530 is the entry, but when the server configuration is read then localhost is resolved to an IP address. The entry searched for becomes tc://terracotta@<a.certain.ip.address>:9530.

  • tc://terracotta@<a.certain.ip.address>:9530 is the entry, but when the server configuration is read then <a.certain.ip.address> is resolved to a host name. The entry searched for becomes tc://terracotta@my.host.com:9530.

Two Active Servers (Split Brain)

Instead of an active-mirror 2-server stripe, both servers assert active status after being started. This error can be caused by the failure of the SSL handshake. An entry similar to the following may appear in the server log:

2013-05-17 12:10:24,805 [L2_L2:TCWorkerComm # 1_W] ERROR com.tc.net.core.TCConnection - SSL     handshake error: unable to find valid certification path to requested target, closing           connection.

For each server, ensure that all keychain entries are accurate, and that the required certificates are available from the appropriate truststores.

No Messages Indicating Security Enabled

If servers start with no errors, but there are no messages indicating that security is enabled, ensure that the <servers> element contains secure="true".

Visit www.terracotta.org for most recent product information.