The final test

In the evening, we identified a bottleneck in our software and fixed it to improve the synchronisation between all apllication server instances. This should lead to another performance improvement beside our new configuration.

The situation for the final test is as follows:
4 instances of Tomcat on the Sun T2000 (+1 instance of Tomcat for our chat server)
1 instance of Tomcat on the AMD Opteron

The load average on the Sun T2000 is 4 (constantly compared to yesterday) and CPU usage went down to 15% (this is mainly because of fixing the bottleneck of synchronisation).

The load average on the AMD Opteron is 1,5 with a CPU usage of 55%. Under comparable circumstances to the configuration on the Sun T2000, the AMD Opteron shows a load average of 12 with a CPU usage 99,9%.

The Sun T2000 (8 cores, 16 GB) still has enough resources to install another 4 instances of the application server. This is, what we were looking for!!

The real heavy load test

It's time to see the performance at peak times.
In the morning (1.800 users online), the results are comparable to saturday.
In the afternoon (1:00 PM), there are more than 3.600 users concurrently online.

The measurements show the following results:
Sun T2000: load average of 4 with a CPU usage of 30%
AMD Opteron: load average of 12 with a CPU usage of 99,9%

The Sun T2000 scales very good and still has enough resources. The next step will be to install 2 more instances of Tomcat on the Sun T2000 (4 instances altogether) and to deactivate one instance of Tomcat on the AMD Opteron. The test will go on tomorrow!

The enhanced test

After a lot of research work in the past 2 days, we would like to try out the new cognitions from our researches in different areas (application server settings, Apache setting and the loadbalancer settings (mod_jk)). As our first test was in the middle of the week, where our users are most active, we try out the new settings today (Saturday), when less users visit the platform.

The situation is as follows:
Webserver layer: Apache with new mod_jk settings as webserver and load balancer on the Sun E420R
Application Server layer:
2 instances of Tomcat 5.5.16 on the Sun T2000
2 instances of Tomcat 5.5.16 on the AMD Opteron
Database layer: MySQL 5.0.19 on the Intel Xeon

And here are the results:
With about 2000 concurrent users online at peak times, the load average of the Sun T2000 is at 2,5 with a CPU usage of about 10%. At the same time, the AMD Opteron machine has a load average of about 2 with a CPU usage of 99,9%. The performance on both sides is good and page delivery time is about 1 second.

Wow!!! This is the performance, we were seraching for!
But how will it behave on monday, when most of our users connect to our platform.
Will the performance be the same? We will see.

The first real-life test

In order to be able to analyze the behaviour of the Sun T2000, we decided to launch it directly in the production environment. So we switched it online parallel to the Opteron machine, also in order to have a direct comparison.

Our test scenario:
Tier 1: 1 instance of Apache with mod_jk as loadbalancer (on the Sun E420R)
Tier 2: 1 instance of Tomcat (on the AMD Opteron machine) and
1 instance of Tomcat (on the Sun Fire T2000)
Tier 3: 1 instance of MySQL (on the Xeon machine)

The first impression:
During the test, the load average went up on all machines (Opteron machine from 1.4 to 12, Sun T2000 from 2.5 to over 20 and even the Sun E420R from 0.5 to about 5) compared to a situation with the same amount of requests! Something must be wrong... but what is it?
There are too many possible reasons: the mod_jk configuration, the Tomcat configuration, the MySQL configuration? We have to try!

Installation of Tomcat and mod_jk

Since OPenPKG ist installed, we are able to install different versions of all software at the same time and switch to the desired version very easily without noteworthy downtimes.

Until now, we only had one instance of Apache, one instance of Tomcat and one instance of MySQL. The Sun T2000 is planned for the middle tier, the application server layer. In order to balance load among the multiple application servers, we use mod_jk, which is a plug-in of the Apache webserver using sticky sessions to send a user to the same application server instance which he connected the first time when enabling the session.

Installation of OpenPKG

OpenPKG (www.openpkg.org) is a cross-platform multiple-instance Unix software packaging system and is a collaboration effort with the goal of creating and maintaining portable and easy to install software packages for use on all major Unix server platforms — especially on the popular Linux and Solaris flavors.

OpenPKG is linked into the kernel, thus it took some time to compile the packages!

The first impression

It took only 8 days from submitting the order to receiving the brand new Sun T2000 server.















What we found very impressive:

• up to 4 hot swappable 2,5" HDDs and redundant hot swappable power supplys whithin only 2 HEs
• almost no cables/wires in the air flow area
• very efficient air deflector on the top cover
• the new T1 processor: 8 cores in whithin only 3,5"x3,5"

Next step: Installation of OpenPKG as the desired packaging tools for the Software.