Red5 + Terracotta POC

Tim Eck and I (Orion Letizi) worked on a proof-of-concept (POC) of clustering Red5 with Terracotta. The ultimate goal is to provide Red5 with a simple, fast, reliable, and open-source way to add scalability and high-availability. Enabled with Terracotta, Red5 can be deployed in a multiple-server scenario so that, as client load increases, more Red5 servers can be added to handle the load.

The proximate goal of this POC was to get the Red5 SharedObjects facility clustered so that flash clients connected to different Red5 server instances can communicate with each other via the SharedObject mechanism as if they were connected to the same Red5 server.

We used the BallControl sample flash application as our test case. We wanted to be able to:

1. Run two Red5 servers
2. Connect one BallControl sample instance to one Red5 server
3. Connect another BallControl sample instance to the other Red5 server
4. Make changes in the position of one BallControl sample instance
5. See those changes reflected in the other BallControl instance

We took two separate paths to this:

1. Brute Force Style: Naively cluster the entire Scope tree, starting with GlobalScope.
2. Refactored Style: Selectively cluster only the data that must be shared.

Brute Force Style

The first approach has the advantage of very little changes to the Red5 server code. However, I wasn't ever really expecting it to work, since the Scope data structure just doesn't lend itself to be shared very well. I got as far as getting the Scope tree to become a shared object graph in Terracotta, but, there would have been a lot more work (and probably a fair amount of ugly refactoring) to get it to actually work the way it's supposed to.

Refactored Style

The second approach we took actually works (with some caveats), but it requires a little more surgery to the Red5 server code (although, not really that much).

The basic idea is that we refactored the SharedObjectScope so that the shared object data is actually in a separate data structure. We made that separate data structure into a Terracotta shared object. The SharedObjectScopes essentially use their path to find the actual shared object data.

We also added a distributed method invocation that plugs into the data change notification system so that all connected flash clients will know that data has changes and that they should update their display, etc.

Advantages of This Approach

• It actually works
• It didn't take very long (~four hours—it would have been less had we actually known anything about the Red5 server code before we started)
• There's much less clustered data

Caveats to This Implementation

This is by no means complete. There's still a lot of stuff to do.

• There are lifecycle issues that we ignored.
• There was some shared lock stuff that we ignored. I'm pretty sure that we didn't get all the locking completely right.
• We didn't do any extensive testing. The only thing we tried was the BallControl demo.
• Our actual implementation is a hack. We don't know very much about the Red5 server code, so our solution isn't very elegant. Someone with more intimate knowledge of Red5 should help us with a refactoring that follows the design paradigms of Red5. This shouldn't be too hard, but we (Terracotta) can't really do it alone.
• The TerracottaService shouldn't be static
• The lookup to the TerracottaService needs to use a real path with something unique to the idenity of the red5 shared object. Right now the code is using a hard coded string for the path (SharedObjectScope.java:106)
• The "hashes" fields in red5 code might be problematic given the inherent non-portability of hashcode values.

Next Steps

• Work with Red5 team to architect a more elegant version of this POC and to work out the caveats listed above

HOWTO

Instructions for trying this POC are as follows.

You really need at least two machines for this, since we couldn't figure out how to get the various ports that Red5 listens to to not conflict. Logically, there are three separate machines that I'll refer to as red5-1, red5-2, and terracotta. Red5-1 and red5-2 will run red5 servers. The terracotta machine will run the Terracotta server.

1. Download Terracotta 2.4 or later.
2. Install Terracotta on all the machines (red5-1, red5-2, and terracotta). On *nix, just untar it in a convenient place.
3. Download the Terracotta + Red5 POC patch (this isn't actually a patch file... it's just a tarball with the modified files in it)
4. Check out Red5 on both red5 machines (we used Revision 2146, so you should probably check this version out as well):

   %> svn co -r 2146 http://svn1.cvsdude.com/osflash/red5/java/server/trunk red5-2146
   

5. Untar the changes to Red5 on both red5 machines

   %> cd red5-2146
   %> tar xfvz <path/to/downloaded/patch>/red5-patch.tar.gz
   

   You should see output that looks like this:

   host-144-0-0:~/Documents/workspace/red5-2146 orion$ tar xfvz ~/Downloads/red5-patch.tar.gz 
   tc-config.xml
   src/org/red5/server/so/TerracottaSharedObject.java
   src/org/red5/server/so/TerracottaService.java
   src/org/red5/server/so/SharedObjectScope.java
   src/org/red5/server/so/SharedObject.java
   

6. Build red5 on both red5 machines

   %> ant
   

7. Run dso-env.sh on each Red5 machine from the red5 server directory. This will build a boot jar for your platform if necessary and also output the JVM arguments you need to add to the red5.sh shell script.

   red5-n:~/red5-2146%> ~/terracotta/bin/dso-env.sh terracotta:9510
   2007-07-02 15:19:51,092 INFO - Terracotta 2.4-stable1, as of 20070622-150631 (Revision 3946 by cruise@rh4mo0 from 2.4)
   2007-07-02 15:19:52,366 INFO - Configuration loaded from the server at 'terracotta:9510'.
   Creating boot JAR at '/Users/orion/terracotta/bin/../lib/dso-boot/dso-boot-hotspot_osx_150_06.jar...
   Successfully created boot JAR file at '/Users/orion/terracotta/bin/../lib/dso-boot/dso-boot-hotspot_osx_150_06.jar'.
   -Xbootclasspath/p:/Users/orion/terracotta/bin/../lib/dso-boot/dso-boot-hotspot_osx_150_06.jar  -Dtc.install-root=/Users/orion/terracotta/bin/..  -Dtc.config=terracotta:9510
   

8. Copy the last line of output and prepend it to the JVM arguments in the red5.sh script. When you're done, the red5.sh script should look something like this:

   #!/bin/bash
   
   for JAVA in "$JAVA_HOME/bin/java" "/usr/bin/java" "/usr/local/bin/java"
   do
     if [ -x $JAVA ]
     then
       break
     fi
   done
   
   if [ ! -x $JAVA ]
   then
     echo "Unable to locate java. Please set JAVA_HOME environment variable."
     exit
   fi
   
   # start red5
   exec $JAVA -Xbootclasspath/p:/Users/orion/terracotta/bin/../lib/dso-boot/dso-boot-hotspot_osx_150_06.j\
   ar  -Dtc.install-root=/Users/orion/terracotta/bin/..  -Dtc.config=terracotta:9510 -Djava.security.manager -\
   Djava.security.policy=conf/red5.policy -cp red5.jar:conf:$CLASSPATH org.red5.server.Standalone
   

9. Copy tc-config.xml to the terracotta machine. For this example, I'll copy it to my home directory on the terracotta machine:

   %> scp tc-config.xml terracotta:~/
   

10. Start the Terracotta server on the terracotta machine (I'll assume that you've installed terracotta in your home directory and named it "terracotta")

    %> ssh terracotta
    terracotta:~/ %> terracotta/bin/start-tc-server.sh -f ~/tc-config.xml
    2007-07-02 15:16:47,517 INFO - Terracotta 2.4-stable1, as of 20070622-150631 (Revision 3946 by cruise@rh4mo0 from 2.4)
    2007-07-02 15:16:48,632 INFO - Configuration loaded from the file at '/Users/orion/tc-config.xml'.
    2007-07-02 15:16:48,839 INFO - Log file: '/Users/orion/terracotta/server-logs/terracotta-server.log'.
    2007-07-02 15:16:49,843 INFO - JMX Server started. Authentication OFF - Available at URL[service:jmx:rmi:///jndi/rmi://localhost:9520/jmxrmi]
    2007-07-02 15:16:51,068 INFO - Terracotta Server has started up as ACTIVE node on port 9510 successfully, and is now ready for work.
    

11. Start the Red5 servers:

    red5-n:~/red5-2146 %> red5.sh
    

    You should see output that looks like this at the beginning. This is an indication that the Terracotta client code is running in Red5:

    2007-07-02 15:29:47,526 INFO - Terracotta 2.4-stable1, as of 20070622-150631 (Revision 3946 by cruise@rh4mo0 from 2.4)
    

12. Open two browser windows and point them at the two different Red5 servers: http://red5-1:5080/demos/BallControl.swf and http://red5-2:5080/demos/BallControl.swf
13. In both browser windows, click the plug icon to connect the BallControl demo to the Red5 server.
14. When both BallControl instances are connected, move the Red5 ball around with your mouse in one browser window; you should see the other Red5 ball move in tandem. There you go: clustered Red5!

to

Look at the Terracotta Console

For kicks, run the Terracotta console and look at the data that's shared. You should see something like this:

You can also look at the transaction rate graphs as you move the Red5 ball and see the shared object transaction rate go up and down as you make changes:

Pretty neat, huh?