PIE Benckmarking Tests

The PIE has had an unfortunate history of speed problems. It is reliant on a variety of background systems, which makes it difficult to diagnose these problems. The current system, for example, utilises third party software including the Apache web server and the Postgres database; these are on different machines connected by a fast Ethernet network. The CGI script on the first machine uses the perl DBI library to contact the database on the second. There are clearly several places where bottlenecks might occur, not all of which are under the control of the PIE installation.

Some benchmark measures of the options for the various components might be obtainable from general studies, e.g. comparisons of relational databases. However, it is frequently the case that the results of benchmarking tests have been disputed by the authors of the software which comes out less well as a result, and some reports have been accused of being based on studies skewed to favour particular developers, even when the benchmark tests involved have been specified by established standards (e.g. the ANSI SQL Benchmark Standard). See the following Slashdot articles for discussions related to this: Postgres beats MySQL, Interbase, and Proprietary DBs, MySQL Developer Contests PostgreSQL Benchmarks and MS Squashes SQL Benchmarks.

In addition, the PIE is a heavy user of particular functionality in its components, as can be seen from the table which gives for each function the approximate number of each type of SQL statement executed, assuming the eventual presentation of a page with seven items on it. (These values are approximate because the number of times particular lines of code are called depends on the value of parameters passed to the function and information already in the database as well as the function itself.)
FunctionSimple SelectComplex SelectInsertUpdateDelete
Add_new6011811
Addtolist6311915
Change_default5511531
Createpage61112721
Createweblink60/TD>111111
Customise5911515
Edit681111329
Editconfig5611511
Edithelp5711633
Editpage5611864
Editresource5511622
Execute5611522
Exit270111
Expcon5811521
Feedback5511511
Help5811612
Logon63112831
Pageview5511916
Present5511511
Search671215513

Furthermore, some at least of the slowness of the PIE is likely to come from the interaction between components, from network connections or the time required to spawn a child process, as well as from the functionality tested by a normal software benchmark.

For all these reasons, it makes sense to come up with benchmark tests which are tailored to the PIE as a whole, and these will be of most use in improving the user experience if they mimic that experience. It is therefore suggested that the benchmark tests to be developed consist of a series of HTTP requests, amounting to complete PIE sessions. This structure also has the advantage that it would be easily reusable to access any (fairly simple) WWW based system.

Benchmark Script Description

The script will use a configuration file which contains a series of URLs (though usernames and passwords will be replaced by the fragments and ). The file will also contain a list of user names and passwords to plug into these URLs. The script will spawn child processes to call the series of URLs for each username in parallel, so that testing of the loading caused by multiple users can be carried out. An additional configuration parameter will determine the number of times the series of URLs is passed through. For each URL, the script will obtain the entire HTML page (not just calling the HTTP status) but will not download graphics from the page. If there is an HTTP error with any of the URLs, the child process will die; it will also parse the output from an authentication action to check that the username and password it has used is valid and die if authentication fails.

When the processing is complete, the script will report the number of trials, and for each URL the shortest/average/longest times for retrieval as well as the same statistics for the overall process.