High Availability With 2 Nodes

The data in this document is outdated as of Terracotta 2.6. Please check back on this document as we will update it with proper information with respect to Terracotta 2.6

It is possible to run Terracotta with only 2 nodes. In this configuration, there are a minimum of 4 processes that must be running for High Availability with Terracotta.

Terminology

L1 : The Terracotta Client. This is where the application resides. Often this application is a server application, however from Terracotta's point of view, it is a Client.

L2 : The Terracotta Server. Terracotta Servers are JVM processes.

2x2 : This configuration may be referred to as "2x2" as it involves placing one L1 and one L2 together onto one machine, and another L1 and L2 on another machine.

Configuration

These processes are one L1 and one L2 on one machine, and one L1 and one L2 on the other machine. For the purposes of discussion, we shall name these processes:

Machine A: The first machine. Assume at steady state it is the "Active" L2.
L1A : The L1 process on machine A
L2A : The L2 process on machine A
L1B : The L1 process on machine B
L2B : The L2 process on machine B

Timeouts

There are 2 timeouts that affect the configuration of systems in a 2x2 configuration. They are:

TCP Timeout

TCP does not typically timeout. The configuration of timeouts is host dependent. The most common configuration for TCP timeouts is approximately 2 hours.

Without a TCP timeout configured, a host operating system, and thus Terracotta, will not know that a TCP connection has been severed due to some kind of network failure, such as a severed link, or simply the ethernet cable being removed from the host.

Note that some operating systems will detect the ethernet cable being pulled, and may or may not behave appropriately. For example, it appears that Mac OS 10.5 does detect the ethernet cable being pulled (link level loss) but does not by default terminate the TCP connection. However it should be noted that Mac OS 10.5 does lose the interface upon which the IP address was associated, which can be problematic for any clients that are connected to the interface - in particular Terracotta L2s can have a problem with this behavior.

Terracotta Reconnect Window

During a failover procedure, Terracotta expects that all clients previously connected to the cluster will reconnect to the newly active server.

The Reconnect Window setting controls how long Terracotta will wait for all clients to reconnect. If all clients connect to the newly active Terracotta server, cluster operations can proceed immmediately. If they do not all connect, cluster operations are paused awaiting the connection of all clients.

If all clients do not connect within the Reconnect Window period, the remaining clients will be banned from the cluster.

Configuring Timeouts correctly

It is important to ensure that the TCP keepalive and timeout window is configured to be shorter than the Terracotta Reconnect Window.

In a typical environment, this is not critical, however in a 2x2 it is critical, because pulling the ethernet cable to one host means that both the L1 and L2 of that host are affected.

Let's look at some typical scenarios.

Pulling the cable from an active cluster (TCP Timeout > Reconnect)

This scenario assumes that the TCP timeout is greater than the Terracotta Reconnect Window.

• Assume that we have a cluster with L1A, L2A, L1B and L2B operating in a steady state. L2A is the Active server.
• We pull the ethernet cable on L2A
• L2A notices the disconnect of L2B because of its heartbeat detection.
• L2B notices the disconnect of L2A because of its heartbeat detection
• L2A is already active, so it stays active
• L2B has no other active servers nearby, so it elects itself active
  (Note that this has setup a classic split-brain scenario, but since machine A is disconnected from the network, we expected this when we pulled the cable)
• L2A will no longer be able to successfully write to L1B, since the connection is severed, but L2A does not know this. It will begin to push back on L1A who will eventually stop since all available transaction buffers will be used.
• L1B will no longer successfully write to L1A, since the connection is severed, but L1B does not know this. It will eventually stop because it's transaction buffers will be used.
• Meanwhile, L2B has begun the L1 Reconnect Window
• Eventually, L2B expires the L1 Reconnect Window. Since no L1s attached to it in the pre-configured time period, the cluster moves on with no L1s attached. Any L1s that wish to talk to L2B cannot have been from the previous cluster (before pulling the cable) since their state is now out of date wrt the cluster.
• Eventually, L1B notices that its TCP connection is severed, and tries to failover to L2B. L2B rejects this client as it no longer accepts clients from the previous cluster. In other words, L1B is permanently orphaned.
• Eventually, L2A notices that the TCP connection to L1B is severed, and drops L1B from the cluster. L1A can now proceed.

Note that if your operating system drops the interface or the ip address of the interface when the cable is pulled, the events as described above for L1A and L2A may not happen exactly as described - those two processes may stop working alltogether since prior to the cable being pulled, they may have been communicating on the interface/IP address that was dropped.

Pulling the cable from the passive cluster (TCP Timeout > Terracotta Reconnect)

TBD

Pulling the cable from an active cluster (TCP Timeout < Terracotta Reconnect)

This scenario assumes that the TCP timeout is less than the Terracotta Reconnect Window.

• Assume that we have a cluster with L1A, L2A, L1B and L2B operating in a steady state. L2A is the Active server.
• We pull the ethernet cable on L2A
• L2A notices the disconnect of L2B because of it's heartbeat detection.
• L2B notices the disconnect of L2A because of it's heartbeat detection
• L2A is already active, so it stays active
• L2B has no other active servers nearby, so it elects itself active
  (Note that this has setup a classic split-brain scenario, but since machine A is disconnected from the network, we expected this when we pulled the cable)
• L2A will no longer be able to successfully write to L1B, since the connection is severed, but L2A does not know this. It will begin to push back on L1A who will eventually stop since all available transaction buffers will be used.
• L1B will no longer successfully write to L1A, since the connection is severed, but L1B does not know this. It will eventually stop because it's transaction buffers will be used.
• Meanwhile, L2B has begun the L1 Reconnect Window
• Eventually, the TCP Timeout expires for L1B to L2A connection (on the L1B side). This means L1B will attempt to reconnnect to L2B, which succeeds since the Terracotta Reconnect Window has not expired.
• Eventually, the TCP Timeout expires for the L2A to L1B connection (on the L2A side). The L2A simply removes L1B from the cluster which frees up L1A to operate correctly.
• Eventually, the Terracotta Reconnect Window expires, which means the L2B and L1B cluster now move forward without L1A.

Note that if your operating system drops the interface or the ip address of the interface when the cable is pulled, the events as described above for L1A and L2A may not happen exactly as described - those two processes may stop working alltogether since prior to the cable being pulled, they may have been communicating on the interface/IP address that was dropped.

Pulling the cable from the passive cluster (TCP Timeout < Terracotta Reconnect Window)

TBD

OS Specific Settings for TCP

General Information: http://tldp.org/HOWTO/html_single/TCP-Keepalive-HOWTO/

More info, and OS specific commands: http://www.gnugk.org/keepalive.html

CentOS (and probably many others, including RedHat)

net.ipv4.tcp_keepalive_intvl = 1 
net.ipv4.tcp_keepalive_probes = 8 
net.ipv4.tcp_keepalive_time = 1 
net.ipv4.tcp_retries2 = 3

These should also work, depending on your needs:

net.ipv4.tcp_keepalive_time = 1 
net.ipv4.tcp_keepalive_intvl = 1 
net.ipv4.tcp_keepalive_probes = 3 
net.ipv4.tcp_retires2 = 3

Gotchas

Firewall

Both Windows and Mac OS come with firewalls that can pose problems with inbound TCP connections. Make sure these are disabled, or will let connections through on port 9530 (the default keepalive port)

TBD

• Feature Discussion
  • L1 - L2 Heartbeat
  • Configurable "Quorum"
  • Automatic detection of co-resident L1s with L2