|
Performance Testing an Examinator InstallationIntroductionThis document shows how we set up a testbed to verify an installation of Examinator. We tested performance against the service level in the Quantitative Requirements section of the Examinator SDLC document. The test involves a load-generating application simulating user interactions which are distributed to a cluster of application servers using a load balancer. All application servers can write and read from the same database instance. A Terracotta server instance in the cluster provides data coherence and persistence of shared in-memory data. The basic steps for setting up a performance test of Examinator are:
Testbed SetupThe following summarizes the recommended testbed to run performance tests on Examinator. Specific references to hardware, operating systems, and software versions are intended to give you guidance on the computing power and operating characteristics under which our Examinator tests were conducted.
The testbed setup is illustrated in the following figure. Configuring Testbed ComponentsThe following sections show you how to configure certain components in the testbed. TomcatThe following script configures the Tomcat environment, then starts Tomcat with Terracotta. The script assumes that Tomcat and the Terracotta client are installed in the subdirectories tomcat and terracotta. You can edit the script to adapt it to your particular installation. # Remove previous version of folder/files from the local dir rm -rf <path_to_test_install>/tomcat rm -rf <path_to_test_install>/terracotta # Copy shared tomcat profile to local dir echo "Creating TOMCAT Local Profile..." cp -r /myDir/tomcat <path_to_test_install> # Set TC_HOME TC_HOME=/<path_to_test_install>/terracotta TC_INSTALL_DIR=$TC_HOME # Set TC_CONFIG -- path to Terracotta configuration file TC_CONFIG_PATH=<path_to_test_install>/terracotta/tc-config.xml export TC_INSTALL_DIR TC_CONFIG_PATH TC_HOME echo "TC_HOME = $TC_HOME" # Create DSO jar for tomcat echo "Installing DSO..." . ${TC_INSTALL_DIR}/bin/dso-env.sh -q echo "DSO Installed successfully..." export JAVA_OPTS="$TC_JAVA_OPTS $JAVA_OPTS" # Set tomcat related env variables CATALINA_HOME=<path_to_test_install>/tomcat TOMCAT_HOME=<path_to_test_install>/tomcat CATALINA_BASE=<path_to_test_install>/tomcat export CATALINA_BASE CATALINA_HOME # The next part of the script sets jvmRoute to the last byte of the application server's IP address. IP=`/sbin/ifconfig | grep inet | grep '10.0' | cut -d '.' -f 4 | cut -d ' ' -f 1` echo "Changing jvmRoute to $IP" sed -i 's/IPCONFIG/'"$IP"'/g' <path_to_test_install>/tomcat/conf/server.xml # At this point in the script, server.xml in each application server # should have an Engine element with the following format: # # <Engine defaultHost="Catalina" name="<string>" jvmRoute="<xyz>"> # # where defaultHost is the application server's hostname, name is # name of the Tomcat instance, and jvmRoute is the last byte of # server's IP address. jvmRoute is included in cookies sent by # the Tomcat sever and is then used by the load balancer to route # traffic back to this Tomcat server for sticky sessions. # Set JVM properties and start tomcat server echo "Starting local tomcat server at port 8080..." export CATALINA_OPTS="-Duse.async.processing=true -Dasync.concurrency=5 -Dcom.tc.l1.cachemanager.enabled=false -Duse.pojoizer=true -Xms1024m -Xmx1024m -server -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -verbose:gc" $TOMCAT_HOME/bin/startup.sh -Dcatalina.base=$CATALINA_BASE echo "Check logs in <path_to_test_install>/tomcat/logs ..." Additional ConfigurationTo optimize the Tomcat JSP compiler for a production-like environment, download this configuration snippet and add it to the web.xml file used for each Tomcat server. To optimize the Apache Tomcat HTTP Connect, add the following to the server.xml file used for each Tomcat server: <Connector port="8080" protocol="HTTP/1.1" enableLookups="false" maxThreads="500" minSpareThreads="100" maxKeepAliveRequests="1" acceptCount="200" connectionTimeout="20000" redirectPort="8443" />
Load BalancerThe testbed uses the Apache HTTP server mod_proxy_balancer module as load balancer. The Apache server must be configured to use sticky sessions.
Modifying conf/httpd.confAfter installing the Apache HTTP server, enable the following modules in conf/httpd.conf: LoadModule proxy_module modules/mod_proxy.so LoadModule proxy_balancer_module modules/mod_proxy_balancer.so LoadModule proxy_http_module modules/mod_proxy_http.so LoadModule rewrite_module modules/mod_rewrite.so Also add the following configuration section: <VirtualHost *> RewriteEngine On UseCanonicalName On # To reduce logging on the http server ErrorLog "/var/log/apache2/rewrite.log" LogLevel error CustomLog /dev/null combined # Edit the mapped request shown in this example of Proxypass (/examinator to //myCluster/examinator) to match your setup. # Sticky sessions are maintained using JSESSIONID. # No failover is enabled. ProxyPass /examinator balancer://myCluster/examinator stickysession=JSESSIONID nofailover=On maxattempts=0 timeout=1800 ProxyPassReverse / balancer://cluster/ ProxyPreserveHost On ProxyTimeout 1800 # Note that balancer:// in Proxy and ProxyPass must # have the same value. <Proxy balancer://myCluster> # List the application servers here, one on each line: # BalancerMember http://<hostname>:8080 route=<last byte of application server's IP address> # For example, an application with hostname myAppServer and IP address 123.45.6.789 would have the following entry: # BalancerMember http://myAppServer:8080 route=789 # The value of route for myAppServer must match the value of # myAppServer's jvmRoute attribute in its server.xml. </Proxy> </VirtualHost> Load GenerationApache JMeter is a Java application designed for performing functional and performance tests on various types of applications. In this load test, JMeter simulates a number of virtual users that log into Examinator at an escalating rate and take an exam. The following parameters define the load:
The virtual users follow a particular pattern that exercises the user interface much like a real user is expected to do:
You can create your own JMeter test plan based these parameters and user behavior, or download the provided test plan. See the JMeter documentation for more information about JMeter test plans. Using the Provided Test PlanThe provided test plan works in tandem with a CSV file containing <username,password> pairs. These <username,password> pairs correspond to user accounts on the Examinator database. To set up the test plan, follow these steps:
JMeter LoggingTo improve the accuracy of your performance measurements, reduce the load created by the JMeter instances themselves by setting the following logging properties in the jmeter.properties file: jmeter.save.saveservice.assertions=false jmeter.save.saveservice.response_data.on_error=true // helpful for error debugging Starting JMeter InstancesUse the same startup command for each JMeter instance. For example, if you differentiate the JMeter test plans by appending a number to each one's filename, you could start JMeter with the following command: <JMeter Home>/bin/jmeter -n -t <path_to_testplan>/JMeterTestPlan<jmeter_instance_number>.jmx & This startup command causes a log file called JMeterTestPlan<jmeter_instance_number>.log to be created in the local directory. To introduce a realistic load, the JMeter instances should all be started at approximately the same time. This can be done by batching each JMeter instance's startup command in a script. DatabaseIf you are setting up the Examinator database manually, see the Examinator Maven Project for directions.
No default data is created when you set up the Examinator database. You must create the following data:
TerracottaYou can download a Terracotta configuration file (tc-config.xml) that has been modified for use with Examinator. The application server setup script expects to find tc-config.xml at the directory of the Terracotta installation. For more information on working with Terracotta, see the Terracotta reference documentation. LinuxIf the error "too many open files" occurs, increase the hard limit for the "open files" parameter shown by ulimit -aH to the maximum value (65536). MonitoringYou should monitor the following performance aspects:
Network PerformanceTo analyze performance of your network, use the free nmon tool to track network metrics. Using the following command, you can have nmon record system statistics and save them to a file: nmon -fT -s 5 -c 120 where -s is the sampling time (in seconds) and -c is the number of samples taken during that time. The output file appears in the current directory and has the extension .nmon. You can analyze the output file using the nmon analyzer tool. Measuring Server LatencyThe speed with which an application server processes a request can introduce a delay, or latency, into the overall performance of Examinator. More precisely, server latency is the time between the point a request is received by a server and the point a response is sent. You can measure server latency by introducing a response-time tracking valve to the Tomcat application servers in the testbed, capturing the response times in the log file responseTime.log. The purpose of this customized valve (ResponseTimeTrackingValve) is to record response time for a page and store response-time statistics for that page.
To implement, download the ResponseTimeTrackingValve JAR file from https://svn.terracotta.org/repo/forge/projects/exam-perf-test/ResponseTimeTrackingValve and deploy it in each Tomcat server's server/lib directory. You must also register the valve by adding the following statement to Tomcat's server.xml just before the closing tag: <Valve className="org.tc.performance.ResponseTimeTrackingValve" exclude="*.png,*.css,*.js,*.gif,*.ico,/,index.html" prnCount="1000" suffix=".log" prefix="responseTime" lastPageToken="e1s31"/> You can configure the following attributes in this <Valve> statement:
In a production environment, you should disable the response-time tracking valve by removing its JAR file. In addition, to prevent the printing of request and response times, remove the execution ID (execution=e1s31) from test URLs. Response-Time Tracking ServletYou can view the current data stored by ResponseTimeTrackingValve by installing a tracking servlet. Current response-time values are displayed on a web page with regular updates based on a specified time interval. The servlet provides a method to reset these values. The statistics are shown in a table for each Request URI. The code for the servlet can be downloaded from https://svn.terracotta.org/repo/forge/projects/exam-perf-test/TrackingWAR. Run the TestOnce the testbed components are ready, you can begin the test.
|
Exam App Testing
(None)