The command line

I made a series of test with different values of the parameters. All of them are for reference purpose only. As with any benchmark, a lot of factors affect the results. The advice is, try to isolate your environment so that the results are meaningful

My machine is

• Intel(R) Core(TM)2 Duo CPU T6400 @ 2.00GHz
• Cache: 2048 KB
• RAM: 4GB

with a lot of processes running and the same machine hosting lighttpd, JMeter and the images.

There are two series of tests. The first one test the seaside.example.com. That is the application that stores everything on memory in each image. So this will be very fast.
The second one test the magma.example.com. So each request is accessing the Magma database. As we increase the number of images, the database will be the bottleneck. Keep that in mind when comparing results. Also, don’t bash Magma as magma is very fast. It is just that the SeasideMagmaTester isn’t optimized yet. It is just a simple application for measuring a *simple* Magma-Seaside integration. In a real production environment you’ll put the database in the most powerful server (at least for writing performance), and for read performance, you can add several servers to the magma node and get a lot of reads/sec. But that is another problem. We just want to test the SeasideProxyTester as is. Optimize your application as you see fit.

First the seaside.example.com results:

Now the magma.example.com results:

Those results as I said, are just a reference. YMMV.

Comments:

The seaside.example.com results are very varying. With one seaside image, you get 113 requests/second. Thats a lot of requests. Really. I hope someday I have a site that receive that number of requests. But have in mind that the seaside.example.com application it is just storing the counters in memory. Also, the server (that is, my laptop) is just handling 1 process for the magma image (not used), 1 process for the seaside image (heavily used), 1 process for the lighttpd server (heavily used) and 1 process for JMeter. Not a lot of work for the cpu and the Linux process scheduler.
But if you see the results for 2 seaside images, the best you get is 140 requests/second without getting errors. That is unexpected, because if 1 image can handle 113, 2 images should handle at least 200 request. That is even more notorious when you use 10 or 30 Seaside images. The best you get is 154 requests/second. As I said before a lot of things affect this results. First my CPU isn’t as powerful as the ones from real servers. My laptop is doing a lot of other processes (webbrowser, gaim, JMeter on GUI mode, the GNOME desktop, the wireless, the music). In a dedicated server more resources are reserved for the Seaside images. In the worst case, with 30 Seaside images (each of them doing a lot of work by itself) the laptop CPU is doing a lot of process context switching giving each image a slice of processor time. Each image, in turn, is doing its own process scheduling between Komanche, Seaside and the others processes that run in a Pharo image. If you consider this you can explain why there isn’t a linear scaling in the requests/second as you increment the number of images. The best, appears to be, is to use different servers for the webserver and for the images. Also, distributing the images on two or more small servers (as my laptop is), can get the best of the images and from the balancer.

Now for the Magma results. They are ugly and disappointing. But, remember, isn’t optimized yet. It is just the simplest way of getting Magma and Seaside working. For example, if your app reads a lot more than writes to the database (as most application are, unless you are storing the results of subatomic collisions ;)) you can add more read only server to a magma node to improve the read performance. Besides, you can use different read strategies and use Magma Collections to store your data. The PROBLEM WITH THIS PARTICULAR APPLICATION is that all the images are trying to write to the same slots on the dictionary that holds the counters. This, when you have a lot of processes trying to write, necessarily results in a lot of commit errors. Suppose session 1 reads the current counter value in order to increase it. Before it can commit the new value (current value + 1) the Pharo scheduler switches to other session on the same image or the OS scheduler switches to other Pharo image. The new scheduled session (session 2) reads the current value (not yet updated by the first session) and if not uncheduled like the session 1, successfully commits the new value. Some time later the session 1 get scheduled again and tries to resume from the exact same place where it left. So update the value and send the commit to magma. Magma notes that the value has changed since it was read and marks a dirty object (that is, the client must do an abort to get the new value) and a magma conmit conflict. The error is arrives to the final user and is counted by JMeter as an error because of the status 503 from the headers.
In a real application, the common scenario is that each user writes to its own section of the database or to different parts of a common collection, this is handled very well by Magma, even better if you use Magma Collections. So in a more realistic scenario, you won’t have that many commit conflicts, if any. But that is Magma optimization and you know better than anybody your own application. Maybe Chris Muller (Magma creator) or Keith Hodges (Magma seasideHelper creator) can replicate this results and suggest better ways to test Magma and use Magma seasideHelper. I repeat: the apparent errors are a consequence of the application tested and not from Magma. How do you know? Because in every case we get a response from the Magma server, that is, a commit conflict error response. So the server is alive and healthy, responding appropriately every request made by a Seaside image. Keep that in mind before bashing Magma.
One better way to test this application is to give each session its own counter on the database (as if each user were getting its own private data) and all of those private counters being held on a Magma Collection (that is, a collection of user data). This way each session will update its own data and that by its own nature, won’t produce commit errors. But that is left as an exercise to the reader.

So, to test your apps.

We have a working deployment setup. It runs a Magma image and several Seaside images. The Seaside images are all of them running copies of a single Seaside image. This Seaside image has a copy of the application we want to deploy. In this tutorial the application is the SeasideProxyTester from the SeasideExamples of squeaksource.com.

The SeasideProxyTester is very simple because it doesn’t want to show any particular Seaside feature. It is a tool to test a setup of proxied Seaside images behind a proxy webserver. It consists of three classes:

• SPTApplication. It is a Seaside application registered as seasideProxyTester.
• SPTApplicationMagma. It is a Seaside application registered as magmaProxyTester
• SPTDatabase. It is used as the root of the domain objects in the Magma repository. It is used by SPTApplicationMagma to store the number of requests made to the application

SPTApplication and SPTApplicationMagma are very simple. When each application is accessed a simple page is rendered, showing the number of requests made so far to the application. The request can be counted in three ways:

1. By Seaside session. Number of requests made as part of the same Seaside session (that is, _s is the same for all of them).
2. By Seaside image. Number of requests made to the same image, no matter if they are from different Seaside sessions (that is, the requests have the same value for the server cookie: app9001, app9002, etc)
3. By application. Number of requests made to the application, no matter if they are from different Seaside sessions or Seaside images (that is is a global counter of the requests made to any of the images in any of the created Seaside sessions).

The magmaProxyTester works exactly as the previous points explain. The Magma database is used to store the global request number and a request number for each image.

The seasideProxyTester can’t, by their nature, keep track of the third counter, that is the global request counter or application request counter. This is because each image is an autonomous entity that has no way to communicate between them to share and update the global request counter. So, although each SPTApplication in each image has a class instance variable to track the number of requests, this is scoped to the image and only can track the number of request that reached the image is part of. So each image has it very own global counter that tracks the requests that has processed. If you sum the global counter of each Seaside image for the SPTApplication, you can get the real global request counter for the seasideProxyTester application. In the SPTApplicationMagma app, the magma database is the external (to the image) holder of those counters, and is also responsible of mediating between them when updating the counters.

So with this warning out of the way. What can you get from this SeasideProxyTester. Lets begin with the magma.example.com application. The first time I access it it shows:

This shows that the request was served by the first Seaside image listening on port 9001.  This request has been correctly counted. If I reload the page I create a new Seaside session but i remain in the same Server (as my browser has accepted the cookie with value app9001). Now the page shows:

As you can see you are on the same server (the one listening on port 9001) and the global counters have been updated accordingly.

Now, I delete the cookies of my browser so that the next reload it doesn’t include the server cookie. lighttpd will use round-robin to proxy to the next available server, that is, the one on port 9002. Now I see:

You are now on a different server, the one listening on port 9002. Well, you get the idea.

Now, there are two links on the page. The first one, “Reset database”, as you can guess, will reset the global counters. But, as this is also a request, the counters won’t be 0 but 1 the next time the page is rendered.

The second one “Make a new request inside this session” will make a new request (by means of a Seaside callback) that is part of the same session (same _s parameter). This, as always, changes the global counters, but also increments the session request counter.

So far so good.

The seasideProxyTester works very similar, the only problem, as I have said is that there is no way for all the Seaside images to update a group of shared global counters (this, in the end, will need a external database, the role Magma is playing in the magmaProxyTester), so each image has a class instance variable that tries to keep track of the global counters but this is only valid inside the image scope. Repeat the same requests made with the magma.example.com using the seaside.example.com and you’ll note that each time you change of Seaside server you get a different global counter. Only if you sum all the global counters you will have the real global request counter.

Now you can put your stress loading tools to work by pointing them to your new setup and measure the results.

Or, wait for the next post and I will show you one way to do load testing on Seaside applications

You have a working squeak vm install. Now we will create the directories we’ll use.

Create directories

export WORK=/home/miguel/work
mkdir -p $WORK

export DEPLOY=/home/miguel/example
mkdir -p $DEPLOY/{pharo,magma,backup,logs,scripts,website}

We will use two directories, put them where you want. I chose to put them on my home directory but you can use any other if you wish. The important thing is to have the environment variables correctly assigned. This will ease the following steps.

The work directory is to hold temporary files as we setup the deploy directory. At the end will be discarded.

The deploy directory is what we will populate with the images and other useful scripts to host our Seaside application and data. As you can see has directories for the images (pharo), for the database (magma), for the magma backups (backups), for the logs (currently only has the output of the nohups used to start the images), for the scripts (guess, scripts!) and for the static content of your application, that you wisely have the webserver to serve and not the Seaside server (website). More on this later.

Download PharoCore

Now go to the Pharo download page look for the section “Sources files” and download the Sources zip file. At the moment is:

SqueakV39.sources.zip

Now go to the bottom of the page and follow the link that says “Pharo-core images and other files”. Find the most recent PharoCore zip file. Currently is:

PharoCore-1.0-10451-BETA.zip

but any newer will do.

Unzip this two zip files. The PharoCore will create a directory and inside it will be the image and changes files. The Sources zip contains one file. Now, copy this three files to the $WORK directory:

cp PharoCore-1.0-10451-BETA.image $WORK
cp PharoCore-1.0-10451-BETA.changes $WORK
cp SqueakV39.sources $WORK

So far so good. You have a PharoCore image with its changes file and a sources file. This, together with the virtual machine gives you a complete Pharo environment to work. You can try it:

cd $WORK
$VM PharoCore-1.0-10451-BETA.image

You should see the PharoCore image running. Quit the image WITHOUT saving.

Save scripts

Save the following scripts to the $WORK directory.

magma-image.st:

“Install Magma Server on a PharoCore image”

“Set some preferences”
Preferences enable: #fastDragWindowForMorphic.
Preferences disable: #windowAnimation.
Preferences enable: #updateSavesFile.
Preferences disable: #windowAnimation.

“Update from pharo update stream (only works/recomended for PharoCore)”
Utilities updateFromServer.

“Your name, like MiguelCoba or VincentVanGogh, dont  use spaces or accents, just ASCII”
Author fullName: ‘FirstnameLastname’.

“Install Installer”
ScriptLoader
loadLatestPackage: ‘Installer-Core’ fromSqueaksource: ‘Installer’.

“RFB”
Installer lukas project: ‘unsorted’;
install: ‘RFB’.

“Magma server”
Installer ss project: ‘Magma’;
install: ‘1.0r42 (server)’.

“Configure the packages”
RFBServer current
initializePreferences;
allowEmptyPasswords: false;
allowLocalConnections: true;
allowRemoteConnections: false;
allowInteractiveConnections: true;
connectionTypeDisconnect;
configureForMemoryConservation;
setFullPassword: ‘useyourownpasswordhere’.

“Save with a new name”
SmalltalkImage current saveAs: ‘magma’.
SmalltalkImage current snapshot: true andQuit: true.

This script when executed on a PharoCore image, will set some preferences, apply updates from the Pharo project if available and then install some packages directly from their repositories. The packages installed are RFBServer (a VNC server for Squeak and Pharo) and the Magma server. Finally it configures the packages and saves the image with a new name: magma. Be sure to change your full name and the RFBServer before saving the file.

magma-run.st:

“When a file named magma.shutdown is found on the same directory as the image
this process is triggered and the image is shutdown without saving”
[
[
[ 60 seconds asDelay wait.
(FileDirectory default fileOrDirectoryExists: 'magma.shutdown')
ifTrue: [ SmalltalkImage current snapshot: false andQuit: true ].
(FileDirectory default fileOrDirectoryExists: ‘magma.startvnc’)
ifTrue: [ Project uiProcess resume.  RFBServer start:0 ].
(FileDirectory default fileOrDirectoryExists: ‘magma.stopvnc’)
ifTrue: [ RFBServer stop. Project uiProcess suspend ].
] on: Error do: [ :error | error asDebugEmail ]
] repeat
] forkAt: Processor systemBackgroundPriority.
“To save CPU cycles”
Project uiProcess suspend.

I will explain this script later.

seaside-image.st:

“Install Seaside on a PharoCore image”

“Install packages”

“Comanche”
Installer ss project: ‘KomHttpServer’;
install: ‘DynamicBindings’;
install: ‘KomServices’;
install: ‘KomHttpServer’.

“Seaside”
Installer ss project: ‘Seaside’;
answer: ‘.*username.*’ with: ‘admin’;
answer: ‘.*password.*’ with: ’seaside’;
install: ‘Seaside2.8a1′;
install: ‘Scriptaculous’.

“Seaside Jetsam”
Installer ss project: ‘Jetsam’;
install: ‘Seaside28Jetsam-kph.67′.

“Seaside helper”
Installer ss project: ‘MagmaTester’;
answer:’username’ with:’admin’;
answer:’password’ with:’seaside’;
install: ‘Magma seasideHelper’.

“SeasideProxyTester”
Installer ss project: ‘SeasideExamples’;
install: ‘SeasideProxyTester’.

“Configure the packages”
“Start Seaside”
WAKom startOn: 9001.

“Unregister example apps”
WADispatcher default trimForDeployment.

“Unregister deployed apps”
WADispatcher default
unregister: (WADispatcher default entryPointAt: ‘/browse’);
unregister: (WADispatcher default entryPointAt: ‘/config’).

“Save with a new name”
SmalltalkImage current saveAs: ’seaside’.
SmalltalkImage current snapshot: true andQuit: true.

This script install Seaside 2.8, Magma seasideHelper and the SeasideProxyTester app that we will use to test the setup. Besides, start Seaside on port 9001, unregister unnecessary apps from the Seaside dispatcher and save the image as seaside.

seaside-run.st:

“When a file named seaside.shutdown is found on the same directory as the image
this process is triggered and the image is shutdown without saving”
[
[
[ 60 seconds asDelay wait.
(FileDirectory default fileOrDirectoryExists: 'seaside.shutdown')
ifTrue: [ SmalltalkImage current snapshot: false andQuit: true ]
] on: Error do: [ :error | error asDebugEmail ]
] repeat
] forkAt: Processor systemBackgroundPriority.
“To save CPU cycles”
Project uiProcess suspend.

I will explain this script later.

start_app.sh:

#!/bin/sh

HOME=”/srv/example”
NOHUP=”/usr/bin/nohup”
VM=”/opt/pharo/squeak -mmap 100m -vm-sound-null -vm-display-null”
IMAGES=”$HOME/pharo”
SCRIPTS=”$HOME/scripts”
LOGS=”$HOME/logs”
START_PORT=9001
END_PORT=9004

# Delete command files

[ -f $IMAGES/magma.shutdown ] && rm $IMAGES/magma.shutdown
[ -f $IMAGES/magma.startvnc ] && rm $IMAGES/magma.startvnc
[ -f $IMAGES/magma.stopvnc ] && rm $IMAGES/magma.stopvnc
[ -f $IMAGES/seaside.shutdown ] && rm $IMAGES/seaside.shutdown

# Start the Magma image
echo “Starting Magma image”
$NOHUP $VM $IMAGES/magma.image $SCRIPTS/magma-run.st >> $LOGS/magma.nohup &

# To give Magma time to open the repository
sleep 5

# Start the Seaside images
for PORT in `seq $START_PORT $END_PORT`; do
echo “Starting Seaside image on port: $PORT”
$NOHUP $VM $IMAGES/seaside.image $SCRIPTS/seaside-run.st port $PORT >> $LOGS/seaside$PORT.nohup &
done

I will explain this script later.

Prepare images

Make sure that the previous scripts are saved to the $WORK directory. Then build the images:

cd $WORK

# Build magma image from PharoCore image
$VM PharoCore-1.0-10451-BETA.image $WORK/magma-image.st

# Build seaside image from the magma image
$VM magma.image $WORK/seaside-image.st

This will take the PharoCore image and, by using the scripts given, will build the magma image. Then, using the magma image, will build the seaside image.

The build scripts are based on the scripts included in the pharo-dev and pharo-web images created by Damien Cassou.

The magma-run.st and seaside-run.st scripts are based on the ones Ramon Leon posted on his blog.

This is the first of a several post about Seaside.

The goal: to deploy a Seaside application to the real world.

Ingredients:

• Debian GNU/Linux
• PharoCore
• Seaside
• Magma
• Magma seasideHelper
• Lighttpd 1.4.23.
• Automated image setup
• ab
• Gnuplot

I must say that this is not the only way to do this kind of deployment. Ramon Leon’s post explain other setup and also the Seaside book has a section on scaling Seaside. This is the way I do it.

Some things to discuss first. I will be using Debian GNU/Linux Squeeze but the procedure is the same for other versions of Debian and for other distributions. I think that this setup can be done in Windows too but I don’t plan to check. I will be using Seaside 2.8 because is the current stable version. When the 3.0 release is announced I will update the instructions accordingly. The Magma version will be the 1.0r42 (server) that was recently released and that since this version runs also in Pharo. I will be using Magma seasideHelper that although for the moment isn’t being mantained, the version used here works very well. This is used for easy integration of Magma and Seaside. I will use lighttpd because that is the webserver I use on my sites but the equivalent Apache configuration it is very easy and sometimes shorter that the ones I show. nginx or pound can be used too. The webserver will be used to proxy the requests from the users to the Seaside server images. It will be doing load balancing too. In order to test the setup a simple Seaside application has been developed and uploaded to the SeasideExamples project on SqueakSource. This can be used to verify that the webserver is doing a sticky session kind of load balancing. This is necessary because for a setup of proxied Seaside images a session must be routed always to the same image where it was created. This is not necessary on applications deployed to Gemstone/S or GLASS because on them the session data is persisted to the OODB and restored on subsecuent request no matter what stone the request is handled by. But, in a setup that doesn’t use Gemstone/S the proxy/load balancing server must guarantee that a session is always routed to the same image. Of course, on the first request of a session, the server choose a random image to handle the session so that the users are distributed uniformly between the images.

The setup will be tested with ab, the tool from the apache project for load testing websites. This is a very simple test, as only request the first page of the application. This shows how many new sessions can be created in the cluster of Seaside images. It doesn’t test inner navigation of the application. Of course, the same setup can be tested with tools like Selenium, JMeter, httperf, siege or jcrawler. This is left as an exercise to the reader. One cool feature of ab is that can output the results in a format understood by GNUPlot so that they can be plotted and visualized more easily.

Scripts to automate image setup are provided too. They update a PharoCore image and prepare Magma and Seaside images with just the required packages. Also, they configure an RFBServer (VNC server) so you can connect to the Magma image to do routine tasks as database backup (this at least until you integrate that tasks to your app so you can launch them from Seaside without entering the image).

Finally a script to start the images and procedures to stop them on demand are provided.

So, a lot of steps, but at the end you’ll have a fully deployed app in a server using a domain name and suitable of scaling by adding more images or physical servers behind the proxy/load balancing server. Also, with the current high availability features of Magma you can also increase the persistence performance by adding more magma servers to the setup.

Ready? Lets go…  to the Seaside.