lokalisten blog for the Sun Fire T2000

The final test

2006-05-16T15:35:00.000-07:00

In the evening, we identified a bottleneck in our software and fixed it to improve the synchronisation between all apllication server instances. This should lead to another performance improvement beside our new configuration.

The situation for the final test is as follows:
4 instances of Tomcat on the Sun T2000 (+1 instance of Tomcat for our chat server)
1 instance of Tomcat on the AMD Opteron

The load average on the Sun T2000 is 4 (constantly compared to yesterday) and CPU usage went down to 15% (this is mainly because of fixing the bottleneck of synchronisation).

The load average on the AMD Opteron is 1,5 with a CPU usage of 55%. Under comparable circumstances to the configuration on the Sun T2000, the AMD Opteron shows a load average of 12 with a CPU usage 99,9%.

The Sun T2000 (8 cores, 16 GB) still has enough resources to install another 4 instances of the application server. This is, what we were looking for!!

The real heavy load test

2006-05-15T18:46:00.000-07:00

It's time to see the performance at peak times.
In the morning (1.800 users online), the results are comparable to saturday.
In the afternoon (1:00 PM), there are more than 3.600 users concurrently online.

The measurements show the following results:
Sun T2000: load average of 4 with a CPU usage of 30%
AMD Opteron: load average of 12 with a CPU usage of 99,9%

The Sun T2000 scales very good and still has enough resources. The next step will be to install 2 more instances of Tomcat on the Sun T2000 (4 instances altogether) and to deactivate one instance of Tomcat on the AMD Opteron. The test will go on tomorrow!

The enhanced test

2006-05-13T17:07:00.000-07:00

After a lot of research work in the past 2 days, we would like to try out the new cognitions from our researches in different areas (application server settings, Apache setting and the loadbalancer settings (mod_jk)). As our first test was in the middle of the week, where our users are most active, we try out the new settings today (Saturday), when less users visit the platform.

The situation is as follows:
Webserver layer: Apache with new mod_jk settings as webserver and load balancer on the Sun E420R
Application Server layer:
2 instances of Tomcat 5.5.16 on the Sun T2000
2 instances of Tomcat 5.5.16 on the AMD Opteron
Database layer: MySQL 5.0.19 on the Intel Xeon

And here are the results:
With about 2000 concurrent users online at peak times, the load average of the Sun T2000 is at 2,5 with a CPU usage of about 10%. At the same time, the AMD Opteron machine has a load average of about 2 with a CPU usage of 99,9%. The performance on both sides is good and page delivery time is about 1 second.

Wow!!! This is the performance, we were seraching for!
But how will it behave on monday, when most of our users connect to our platform.
Will the performance be the same? We will see.

The first real-life test

2006-05-10T09:16:00.000-07:00

In order to be able to analyze the behaviour of the Sun T2000, we decided to launch it directly in the production environment. So we switched it online parallel to the Opteron machine, also in order to have a direct comparison.

Our test scenario:
Tier 1: 1 instance of Apache with mod_jk as loadbalancer (on the Sun E420R)
Tier 2: 1 instance of Tomcat (on the AMD Opteron machine) and
1 instance of Tomcat (on the Sun Fire T2000)
Tier 3: 1 instance of MySQL (on the Xeon machine)

The first impression:
During the test, the load average went up on all machines (Opteron machine from 1.4 to 12, Sun T2000 from 2.5 to over 20 and even the Sun E420R from 0.5 to about 5) compared to a situation with the same amount of requests! Something must be wrong... but what is it?
There are too many possible reasons: the mod_jk configuration, the Tomcat configuration, the MySQL configuration? We have to try!

Installation of Tomcat and mod_jk

2006-05-09T09:03:00.000-07:00

Since OPenPKG ist installed, we are able to install different versions of all software at the same time and switch to the desired version very easily without noteworthy downtimes.

Until now, we only had one instance of Apache, one instance of Tomcat and one instance of MySQL. The Sun T2000 is planned for the middle tier, the application server layer. In order to balance load among the multiple application servers, we use mod_jk, which is a plug-in of the Apache webserver using sticky sessions to send a user to the same application server instance which he connected the first time when enabling the session.

Installation of OpenPKG

2006-05-08T08:53:00.000-07:00

OpenPKG (www.openpkg.org) is a cross-platform multiple-instance Unix software packaging system and is a collaboration effort with the goal of creating and maintaining portable and easy to install software packages for use on all major Unix server platforms — especially on the popular Linux and Solaris flavors.

OpenPKG is linked into the kernel, thus it took some time to compile the packages!

The first impression

2006-05-04T12:45:00.000-07:00

It took only 8 days from submitting the order to receiving the brand new Sun T2000 server.

What we found very impressive:

up to 4 hot swappable 2,5" HDDs and redundant hot swappable power supplys whithin only 2 HEs
almost no cables/wires in the air flow area
very efficient air deflector on the top cover
the new T1 processor: 8 cores in whithin only 3,5"x3,5"

Next step: Installation of OpenPKG as the desired packaging tools for the Software.

Introduction and background information to www.lokalisten.de

2006-04-25T18:57:00.000-07:00

Prior to the technical details behind our social community platform www.lokalisten.de, I would like to introduce the platform and it's history:

www.lokalisten.de emerged from the idea of the 5 founders, who wanted to develop a platform for their own circle of friends. The platform was launched in May 2005 and we began to invite our friends. We started with a typical low cost configuration (Apache, Tomcat, MySQL) on a single processor Intel Xeon machine for the first time.

Within less than 1 month, the community had more than 1.000 members and increased exponential. Due to the exponential growth (which still exists), after only 3 month, we had to expand the hardware architecture and bought another server, which was an AMD Dual Opteron machine. But the problems just began!

Though the Dual Opteron machine had enough calculating capacity, the IO performance of the Intel and the AMD machine was not sufficent to meet the increased requirements.
The solution brought an old Sun E420R with only 4 MB as the webserver (with Apache).
This machine had the desired IO performance to fulfill the http-requests and to deliver the content in time.

The lokalisten community platform (www.lokalisten.de) is one of the fastest growing community platforms in germany (if not the fastest one), with a daily growth of more than 1% of all members. Today, more than 35.000 members are using the platform every day with a peak amount of about 4.500 users concurrently. The members are really heavy users with more than 35 page views per session (see the Alexa ranking diagrams for more details). The current throughput is about 2 TB of dynamic content every month. Here is the growth of page views:

As traffic and usage grew, the Xeon and the Opteron server were not sufficent any more. We were looking for a more scalable (and even more reliable) platform. The Xeon server began to overheat, which once even caused a shutdown by the server itself.
Looking at the alternatives, we only had 2 options:

installing 1 Sun Fire T2000 server
installing another 2-3 dual Opteron servers

The decision was pretty clear: The Sun Fire T2000 Server (8 cores, 16 GB)
also the reasons herefore:

2 HEs of the Sun T2000 instead of 8 HEs for the 4 Opteron machines in the 19" rack
scalability of the Sun T1 processor (constant response time with growing load)
reliable hardware with very good IO performance (our experience from the webserver)

So we ordered the Sun Fire T2000 (8 cores, 16 GB) through Sun's 60-day try & buy program!