The internals of the selftest system

This currently very unstructured and messy document describes the internal workings of the selftest system, both the parser/compiler and the runtime.

The runtime

Initializing and running the testsuite

TestSuite::main( int *argc, char *argv[] )

This function should be called just after opera has been initialized, usually from main, WinMain or equivalent function.

It parses test-suite specific arguments.

If now is supplied as part of one of them, it runs all tests that does not require initialization and then tries to exit opera. In that case this function will not return.

If there is a test argument, but no now part, a flag is set, the testsuite arguments are removed from argv, and argc is decremented to match.

void TestSuite::run( )

This function should be called when opera has been fully initialized, just before the messageloop is entered.

If the testsuite is to be run, this function will send a message to itself that causes the TestSuite::step() function to be called.

void TestSuite::step()

This function is called from the message handler in opera.

It runs one step of the testsuite, which basically means that it runs the current test, or a part of the current test if it uses delay, or requires a document and no window with a document is available.

Testsuite structure

This information is totally useless unless you plan to change how the testsuite works.

Each testgroup is internally represented as a class implementing the TestGroup interface.

class TestGroup
{
protected:
	int TS_next_test;
	int TS_skip_group;

public:
	virtual int TS_step() = 0;
	virtual TestGroup *TS_next() = 0;
	virtual void TS_init() = 0;
	virtual void TS_exit() = 0;
};
The very first time TestSuite::step is called it creates an instance of the first testgroup, and calls TS_init in it.

The rest of the time it keeps calling TS_step in the current testgroup until it returns 0, then it calls TS_exit() in the current group, then TS_next() to get the next group, and if it did get a new group it calls TS_init() in it and sets it as the current group, and deletes the old group.

If there are no more groups TestSuite::step prints a summary of the testresults and returns 0, signaling that the message loop should exit. This is currently done in mhpi.cpp by calling exit() directly.

If TestSuite::step() returns 1, mhpi.cpp immediately posts a new MSG_TESTSUITE_STEP message, which will cause it to call step() again, if any other number is returned mhpi.cpp will simply return, this is used to implement asynchronous tests, they will post a MSG_TESTSUITE_STEP message once they are done. html, ecmascript and manual tests are always asynchronous

The parser

The parser is a program written in Pike that locates all files with an extension .ot in the source directories, parses them and then generates a C++-file using a template file.

Locating source files

The program starts by recursively locating all files with a name that ends with .ot but does not start with .. It then parses each file in the order they were located (the order is, however, largely irrelevant).

Parsing

Tokenization

The first pass of the parsing is a simple tokenization of the input data, done using the Parser.Pike module in pike. This is only done if the .ot-file has changed since last time it was parsed. Since pike allows all tokens that are allowed in C++, this works surprisingly well.

Each token-object is annotaded with the filename and the linenumber where it was found.

This code is imported from the pike runtime library, see the directory <pikedir>/lib/modules/Parser.pmod/, files Pike.pmod and C.pmod

Building the tree

The tokens are then grouped in such a way that lists ({}, (), [] etc) are grouped, recursively, into a single node, with subnodes representing each element in the list. This is done since it greatly simplifies the other passes. This code recides in the function group() in NodeUtil.pmod

String comments

Then the parsetree is recursively searched to locate all "string comments", that is, comments on the form //! <text here> which are converted to strings (in this case the string "<text here>\n") Look no further than parse_tests.pike convert_special_comments

Comment removal

Then all comments are removed This one is in NodeUtil.pmod remove_comments

After this stage the parsed tokens are serialized and saved to disk in a cache-file.

Table, foreach and friends

After the grouping and stringification, the code is run through a multi-pass parser that handles the table, foreach and iterate keywords. It also search for classnames the be used for the transformation of the global secion.

The preprocessor is run once for each time a foreach or iterate is expanded in the parsetree, to allow foreach and iterate inside other foreach and iterate.

tables are removed from the tree completely, and added to a global list of tables

foreach is expanded by looping over the table, and then duplicating the nodes in the foreach block once for each loop, with the identifiers used in the foreach statement replaced with the values for the current table row, and the special syntax $(identifier) replaced anywhere (inside any other tokens) with the same value.

iterate() is basically replaced with a for loop, and as a side-effect sets a flag in the table that causes it to be included it in the generated C++-code.

The main code can be found in the function preprocess in parse_tests.pike The token replacement code for foreach can be found in the NodeUtil.pmod::replace_identifier and NodeUtil.pmod::replace_macro functions.

Parsing the tree

The top-level of the parsetree is then parsed to locate all other things handled by the selftest system, such as tests, group definitions, include statements etc. See the manual for a list of the supported keywords.

Once a test or group has been located, a new Group or Test object is created, the latter added to the list of tests in the current group, and the group added to a toplevel list of groups. This code can be found in parse_tests.pike::parse_file (the while(1) loop). This function is also where the calls the passes above recides

Transformations

The transformations are mainly performed when code is added to the test or group (as in the case of verify or the global section)

Global section

Moving definitions of class methods

Move out all TYPE ClassName::FunctionName( TYPE ) { } blocks to the toplevel scope, to be compatible with booth vc++ and gcc.

In the process the ClassName has to be fixed to include the classname of the testgroup class.

parse_tests.pike::recursively_fix_class_names

Rewriting toplevel variables with initialization assignments

Code such as int q = 10; cannot be included directly in the global section, since each testgroup is converted to a class, and that kind of variable initialization is not allowed for class-local variables.

They are thus moved to the init section (thus basically is moved to the constructor of the class). The function parse_tests.pike::move_initializers

Locating and moving extern declarations and defines

All #define directives and extern declarations are moved to the start of the test-group, to make it possible to use them in the tests.

Test sections

verify

Code generation

Not up to date

The template file

A file named template.cpp is read. In this file the following strings (Note: not identifiers this time, this is a low-level replace without parsing) are replaced:

__YEAR__

The year when the file was generated. This is used to update the copyright header in the template file, e.g.: "Copyright (C) 2002-__YEAR__ Opera Software AS"

__TESTS__

All the classes that makes up the test-groups, and in them all that methods that make make up the tests, and their subtests. Also all methods defined in classes in the tests, and the success variables used to indicate if a file has succeeded or not.

__DEFINES__

All defines and extern declarations found in the tests.

__FIRST_GROUP__

The name of the class that is the first testgroup. Used in the runtime to get it all started.

__INCLUDES__

All includes generated from include statements, with associated #line indicators that show where the include was found.

__TABLES__

All the tables used in the iterate. The code that outputs code can be found in various places in parse_tests.pike, the easiest way to locate it is probably to go to the main method in parse_tests.pike and start from there.

However, if you look for define and call methods in the various classes in parse_tests.pike you will find most of the code.

Dependencies

The testgroups and their tests are reordered in such a way that any dependencies are resolved, that is, if test A depends on test B, it will be outputted after B. In the same way, if a test in group A depends on a test in group B, the group B will be called before group A, and currently also placed before group A in the file. parse_tests.pike::resolv

Writing the result

The final code is compared to the exising optestsuite.cpp (if any). If there are no changes, nothing is written. parse_tests.pike::main

A note about #line

If you open optestsuite.cpp after it has been generated, you will notice that there are a lot of #line statements in it, unless you are running the parse_test.pike program in debug mode (see parse_tests.pike::main, this mode is intended to be used to debug the testsuite generation process, not to debug tests)

The testsuite runtime system