The command line

Running the tests normally

Add the argument -test to the opera commandline

Running tests in a specific module

Add the argument -test-module=MODULENAME to the opera commandline ta

Running only tests that does not require initialization

Add the argument -test-now to the opera commandline

Running all normal tests, and also tests that require manual interaction

Add the argument -test-manual to the opera commandline

Running with output to file

Add the argument -test-output=FILENAME to the opera commandline

Running with different testdata source directory

Add the argument -test-testdata=DIRECTORY to the opera commandline. The specified directory is the root directory of all source (ex: code-2-gogi)

Running only named tests

You can add =GLOB after any of the arguments above to select only some testgroups to be run. As an example, if you only want to run the tests that test the testsuite system, specify -test='Testsuite*'. If you want to also run the manual tests, add -test-manual='Testsuite*'

You can also specify that only some of the tests in a group should be run by appending ':GLOB' to the argument.

As an example, to run the manual tests in the Testsuite tests (which are all named something including manual), you can use -test-manual='Testsuite:*manual*'

Using selftest - Windows

First you have to install pike. You almost certainly want the latest stable-pike windows installer from pike.ida.liu.se

Then start opera using the arguments described above by adding them to the 'Debug' section of the MSVC settings panel for the Opera project.

The output will end up in the same place where compilation errors normally go.

Using selftest - Linux

First you have to install pike. Pike is available from various sources, such as debian packages and binary and source tarballs from http://pike.ida.liu.se/download/. The _minimum_ version required for the testsuite is Pike 7.4 (7.2 might work, but it has not been tested, some versions of 7.3 does not work due to a bug in those pike versions). Pike 7.6 is the currently recommended version, but any stable release after 7.4 should work.

Compile opera normally, but verify that you have SELFTEST = YES in your user settings.

Then start opera using the arguments described above. The output will be sent to stderr.

Using selftest - Macintosh

First you have to install pike. Pike is available from various sources, but this is the easiest to install: http://pike.darwinports.com/. Just follow the instructions on the page

First you need to set 'Build Self Test Sources' as the target and build that. This will create the self test sources. Afterwards compile opera normally, but verify that you have the feature #define FEATURE_SELFTEST = YES. Remeber to check both features.h and quick-features.h.

Then start opera using the arguments described above. The output will be sent to stderr.

Preprocessor

The .ot-files are first run through a preprocessor that recognizes the following expressions:

#include X

Includes the file X in the .ot-file.

Example:
inc.oth:
test ("This is an included test")
{
    verify (TRUE);
}

anyfile.ot:
...
test("This is a test") { }
#include "inc.oth"
test("This is another test") { }
...

#string X

It's possible to use #string like this:
uni_char *x = UNI_L(#string "foo");
Converts the file X to a string, and inserts it.

constant X=Y;

Basically equivalent to #define X Y, but can only handle constant values, not substitutions like define. Note that define is not available (or, rather, it's passed on directly to the output when it's in a section that allows C++-code such as global)

Strings in comments

This feature exists mainly to make it easier to write html blocks to be set as the current document.

Level: any

Syntax:

//! String data here

Example:

//! This is a comment converted to a string.
//! Quote characters, '\' and '"' are quoted.
is equivalent to
"This is a comment converted to a string.\nQuote characters, '\\' and '\"' are quoted.\n"

Utility functions

C++

void output( char *fmt, ...)

Works just like printf in C/C++, but outputs to the same place as the normal selftest output.

Mainly intended to be used for debugging.

char *ST_down( uni_char *x )

Convert 'x' to an 8bit char* string from a unicode string. Will not allocate any memory, maximum returned length is 100 bytes, the memory returned will be overwritten every 3 calls to ST_down. Characters that can not be represented in iso-8859-1 will be quoted as \[x????] where ???? is the character code.

HTML_Element *find_element( uni_char *elem, int nelm=1 )
HTML_Element *find_element( char *elem, int nelm=1 )

Tries to find the nelem:th element of the specified type in the current document, and return it.

template<class Type>
inline OP_STATUS ST_delete_after_selftest(Type* object)

Simple util macro for managing temporary heap allocated objects in asynchronous tests. As async tests don't support finally clause proper deletion can become tricky. This macro simplifies it by scheduling all objects passed to it for deletion after completing test group. Example:

SomeClass* object = OP_NEW(SomeClass, ());
// Will be dealocated after selftest group finishes
ST_delete_after_selftest(object);

If this macro fails(OOM) then the object will be deleted immidietly and the macro will fail the test. This macro also fails if object is NULL, so NULL check can be ommited. As deallocation uses OP_DELETE object must be allocted by OP_NEW(not malloc, or array new allocator).

Ecmascript

void output(fmt, ...)

Works just like printf in C/C++, but outputs to the same place as the normal selftest output.

Mainly intended to be used for debugging.

bool binary_compare_files(pathA, pathB)

Compares the byte contents of two files.

Takes two string arguments, both file paths. Returns true if they are equal, otherwise false.

ST_paint()

Paints the current document to an offscreen bitmap, which is then discarded. Forces paint events, even if the document is hidden.

Takes no arguments. Throws an exception on error.

ST_mouse_event(x, y, type, button)

Sends a mouse event to the current document, regardless of visibility.

Takes four arguments, a document x coordinate, a document y coordinate, a type string, and a mouse button index. The type string is "mousedown", "mouseup" or "mousemove", while the button index is 1 for the left mouse button, 2 for the right mouse button, and 3 for the middle mouse button.

ST_key_event(target, type, key)

Sends a keyboard event to the current document, regardless of visibility.

Takes three arguments, an HTML element to target, a type string, and an Opera key code. The type string is one of "keydown" and "keyup". The function throws an exception on error.

Syntax levels

A syntax level is defined for each keyword. Instead of defining where a keyword can be used, we use the syntax level to tell where the keyword can be used. A keyword might be allowed on more than one syntax level.

common_test

After 'test', 'test_equal' or 'test_nequal', and before the actual code of the test. Example, test("Test name") require BIG_ENDIAN;. require has syntax level common_test.

group

The syntax level is in a group, that is, always after the first group statement in the file.

test

After 'test', like test("Test name") delay 3;. Delay has syntax level test. The common_test syntax level is a subset of the test syntax level.

test_body

Inside the body of a 'test' or a 'subtest'. iterate is an example of a keyword with the syntax level test_body.

top

The syntax level is in the top of the file, that is, not inside a group. group and disabled is on the top level.

Keywords

allowexceptions

Description: Exceptions in ecmascript threads in asynchronous selftests are detected, and the test executing will fail regardless. However there are cases where the exceptions might be expected. Use allowexceptions to prevent the selftest system from treating exceptions as failures.

Level: test

Syntax:

test("test name") async;

Example async:

group "Group 1";

test("Test 1.1") async;// this test will fail
{

  setTimeout(function(){
    throw new Error('Hello');
  },10);

}

test("Test 1.2") async; fails;// this test will fail also
{

  setTimeout(function(){
    throw new Error('Hello');
  },10);

}

test("Test 1.3") async;
  allowexceptions;// this test will pass
{

  setTimeout(function(){
    throw new Error('world!');
  },10);

  setTimeout(function(){
    ST_passed();
  },20);

}

async

Description: Makes a specific test asynchronous. An asynchronous test will not run if "now" is given to selftest on the command line. A test is synchronous by default. An asynchronous test will end and the next test start after the test calls one of the functions ST_passed() or ST_failed("error message"). These two functions work for both c++ and ecmascript tests.

'verify', 'verify_success', 'verify_trap', 'verify_status' and 'verify_string' cannot be used in asynchronous C++ tests.

Level: test

Syntax:

test("test name") async;

Example async:

group "Group 1";

test("Test 1.1") async; // this test will run asynchronously
{
  // Do something
}

test("Test 1.2") // this test will run synchronously
{
  // Do something
}

data

Description: As html, but you must specify the type and specifying a URL is not supported.

Level: group

Syntax:

data "TYPE" { DATA }
For a detailed description, see html

Example data:

data "text/plain" {
//! Plaintext document follows
}

delay

Delay causes the test to be asynchronous, it can thus not be used for tests that should run before opera has been initialized.
Only use delay if it's really, really nessesary.

Description: Adds a delay between tests. By default, 'post' will be used, to add a delay after the test has been run. 'pre' means that the delay will be before the test starts. The delay is given in seconds as a floating point number.

Level: test

Syntax:

test("test name") delay [pre | post]? FLOAT;

Example delay:

group "Group 1";

test("Test 1.1") delay 3; // after this test is finished, selftest will rest for 3 seconds.
{
  // Do something
}

test("Test 1.2") delay post 3.5; // after this test is finished, selftest will rest for 3.5 seconds.
{
  // Do something
}

test("Test 1.3") delay pre 1; // before this test starts, selftest will rest for 1 second.
                              // Since "Test 1.2" had a post delay of 3.5 seconds, selftest will
                              // rest for a total time of 4.5 seconds between "Test 1.2" and "Test 1.3".
{
  // Do something
}

disabled

This will cause the code of the test (or the file) to be totally ignored. No checking will be done. This can thus be used to comment out tests that does not compile.

Description: Disables tests. Will disable all tests in a file or a specific test, depending on where it's put.

Toplevel 'disabled' can have a condition, the rest of the file will only be parsed if the condition is true. This is useful when you use features that are not present in older selftest versions.

Level: top or common_test

Syntax:

disabled; // On top level
disabled if(!TS_CAP_SANE_TESTEQ); // On top level, with condition
test("test name") disabled; {} // On test level

Example disabled on top level:

disabled; // None of the tests in this file will be run.

syntax error ignored

group "Test group 1";

test("Test 1.1")
{
  verify(2 == 3);
}

Example disabled on common_test level:

group "Test group 1";

test("Test 1.1") disabled; // "Test 1.1" will not run.
{
  verify(2 == 3);
}

test("Test 1.2") // This one will, however
{
  verify(2 == 2);
}

exit

exit is only supported for C++-tests.

Description: A function that will run after all tests in a group. Should do all the cleanup necessary. Normally used to cleanup the things initialized in setup.

Level: group

Syntax: exit { c++ statements }

Example exit:

group "Test group 1";

global
{
    char* str;
}

setup
{
    str = new char[10];
    strcpy(str, "Hello")
}

exit
{
    delete [] str;
}

test("Test 1.1")
{
    verify(strcmp(str, "Hello") == 0); // Will most likely crash if setup section was removed.
}

fails

Description: This test should fail to succeed, that is, at least one verify() in it should fail.

Level: common_test

Syntax: test("test name") fails;

Example fails:

group "Test group 1";

test("Test 1.1") fails; // This test will succeed, since one of its verify's fail.
{
    verify(2 == 2);
    verify(2 == 3);
    verify(3 == 4);
}

test("Test 1.1") fails; // This test will fail, since all of its verify's succeed.
{
    verify(6 == 6);
    verify(3 == 3);
    verify(5 == 5);
}

failure

See require

file

Description: File is used to declare any external files used by the test. This allowes the selftest parser to extract a list of external files needed by the tests. It also creates a full path for you that can be used to access the file. Note that files in tables created with filelist automatically gets recognized and does not need to be declared further.

Level: test

Syntax: file [uni] identitifier "filename";

Example file:


group "Test group 1";

test("Test 1.1")
    file uni    testfile_uni    "file.txt";
    file        testfile        "file.txt";
{
    OpString filename;

    // testfile is expanded to a char* version of a full path to 'file.txt'
    filename.Set(testfile);

    // testfile_uni is a uni_char* string.
    verify(uni_strcmp(testfile_uni, filename.CStr()) == 0);

    OpFile file;
    verify(OpStatus::IsSuccess(file.Construct(testfile_uni)));
    verify(OpStatus::IsSuccess(file.Open(OPFILE_READ)));
    verify(OpStatus::IsSuccess(file.Close()));
}

filelist

See table

finally

finally is only supported for C++-tests.

Description: Finally adds some code to a test, that will always run, even if a verify fails. You cannot use verify inside the finally block.

Level: test_body (special)

Syntax: test("test name") { c++ statements } finally { c++ statements }

Example finally:

group "Test group 1";

test("Test 1.1") // A test that fail will print "Finally", but will not print "Success".
{
    verify(2 == 3);
    output("Success");
}
finally
{
    output("Finally");
}

test("Test 1.2") // A test that succeed will print first "Success" and then print "Finally".
{
    verify(2 == 2);
    output("Success");
}
finally
{
    output("Finally");
}

foreach

foreach is a macro construct, and can basically be used anywhere. However, it's recommended that it's only used on group-level.

Description: Will expand everything inside its body to variants with data from the supplied table. A way to create a multiple of tests. Good if you want to test with different data, without risking to stop testing just because one iteration failed (as in iterate). Right now you can also use foreach inside a test, but its usage is discuraged.

Level: group

Syntax: foreach (IDLIST) from [ID | { TABLE_CELL_LIST }] { GROUP_CONTENT }

Example foreach:

group "Test group 1";

table Table1(int, int, char*)
{
  { 3, 3, "Hello" },
  { 4, 54, "Hi" },
  { 5, 65, "Hej" },
  { 1, 100, "Hejsan" }
}

foreach (x, y, tooltip) from Table1
{
  test("Test 1.1 " tooltip)
  {
    verify(x == y); // This will create 4 tests, with the first test succeeding, and the rest is failing.
  }
}

foreach (x, y, tooltip) from Table1 // this will create 4 versions of Test 2.1, and 4 versions
                                    // of Test 2.2. The first version of Test 2.1 will succeed,
                                    // the rest of the Test 1.1 versions will fail. Test 2.2
                                    // will only fail in the first version.
{
  test("Test 2.1 " tooltip)
  {
      verify(x == y);
  }

  test("Test 2.2 " tooltip)
  {
      verify(x != y);
  }
}

foreach (x, y, tooltip) from Table1 // this will create 16 versions of Test 3.1.
                                    // Warning, it is easy to create so many tests
                                    // that the system will fail due to limitations in VC++.
{
  foreach (x1, y1, tooltip1) from Table1
  {
    test("Test 3.1 " tooltip ", " tooltip1)
    {
        verify(x == x1 && y == y1);
    }
  }
}

foreach (x, y, tooltip) from { { 5, 5, "Hello" }, { 4, 4, "HelloHi" } }
// foreach can also have a table included directly.
{
  test("Test 4.1 " tooltip)
  {
    verify(x == y);
  }
}

foreach (x, y, test_name) from { { 5, 5, Hello }, { 4, 4, HelloHi } }
{
  test("$(test_name)") // The first test will be called "Hello", the second on "HelloHi".
  {
    verify(x == y);
  }

  test("Inverted test $(test_name)")// The first test will be called "Inverted test Hello",
                                    // the second on "Inverted test HelloHi".
  {
    verify(y == x);
  }
}

from

See foreach and iterate

global

global is only supported for C++-tests.

Description: Includes a number of functions, classes, variables, etc. that is normal C++ code, and can be used by all tests in a group. It supports an optional string, before the code block, which is the name of a class the test group will inherit, allowing multiple selftest groups in different files sharing the same set of utilities.

Level: group

Syntax: global { c++ statements } | global BASE { c++ statements }

Example global:

group "Test group 1";

global
{
    class Value
    {
    public:
        void SetValue(int val);
        int GetValue() { return value; }

    private:
        int value;
    };

    void Value::SetValue(int val)
    {
        val.value = val;
    }
}

test("Test 1.1") // Using the Value class defined in the global section
{
    int nr = 3;
    Value value;
    value.SetValue(nr);
    verify(nr == value.GetValue());
}

group "Test group 2";

test("Test 2.1") // Cannot use Value class, since this test is in a
                 // different group that has no global section
{
}

group "Test group 3"

include "utilities.h"

global "BaseUtilities"
{
    // More things can be declared here.
    BOOL UseUtility2(){ return UseUtility(g_variable); }
}

setup
{
    g_variable = SOME_VALUE;
}

test("Test 3.1")
{
    verify(UseUtility());
    verify(UseUtility2());
}

// in utilities.h
class BaseUtilities
{
    BOOL UseUtility(int input){ ... }
    int g_variable;
};

group

Description: Starts a new group of tests. The group is ended by the end of the file or by another group statement. The description of a test will include the group name and the test name.

Level: top

Syntax: group "text string";

Example: group "OpString tests";

html

Description: You can use a html { STRING } or html URL { STRING } block to set the contents of the current document. You can use comment-to-string parsing for the document contents. The contents of the block are parsed as text/html. The URL is optional and will be used as the document's effective url, affecting too the scripting origin. Note, if an URL is specified, then there must be some content in the block, else the empty document would result in a network request. This rule is enforced by the selftest parser.

Level: top

Syntax: html { STRING } | html URL { STRING }

Example html:

group "Test group 1";

html {
   //! <body bgcolor="white" text="black">
   //!   <h1>Hello world!</h1>
   //! </body>
}

// equivalent to
html {
   " <body bgcolor=\"white\" text=\"black\">\n"
   "   <h1>Hello world!</h1>\n"
   " </body>\n"
}

// or
html " <body bgcolor=\"white\" text=\"black\">\n   <h1>Hello world!</h1>\n </body>\n"

// example: spoofing an url
html "http://www.google.com/" { "<h1>We're evil</h1>" }

test("Evilness?")
    language ecmascript;
{
    verify(location.href == 'http://www.google.com/'); // Passes
    verify(document.getElementsByTagName('h1')[0]);
}

init

See require

include

Only && works in the expressions. You can simulate || by having multiple includes with different expressions.

Description: Includes a header file for use inside a test group. Please use this include command instead of using #include. The reason is that all include files will be placed in the top of the generated test file, avoiding include duplicates.

Optionally there can be a conditional expression that causes the file to only be included if certain requirements (in addition to any toplevel requires in the file) are met.

The syntax for the conditional expression is if(CONDITIONAL [&& CONDITIONAL [...]]). Using '(' and ')' in the expressions is optional.

CONDITIONAL is one of the following:

exists
Only include the file if it exists
defined X
Only include the file if X is defined
undefined X
Only include the file if X is undefined
X
Short version of defined(X)

Level: group

Syntax: include [CONDITIONAL] "header file";

Example:

group "Test group 2";

include "modules/util/simset.h"; // simset.h can be used in all of the group.
include if exists "legacy.h"; // include legacy.h if it exists
include if(exists) "legacy.h"; // Identical to the above
include if(defined(X)&&undefined(Y)) "legacy.h"; // include legacy.h if X is defined and Y is not defined
include
    if(
       exists
       && defined(X)
       && undefined(Y)
      )
  "legacy.h"; // same as above

// And, finally, since preprocessing is done before the include tokens
// are parsed, you can do some rather odd things:
constant include_condition=(defined(FOO) && undefined(BAR));
include if(include_condition) "include_file";

test("Test 1.1")
{
    Head h;
}

import

Description: Allows you to import tests from another test group into the current group. Use it responsibly.

Level: group

Syntax: import "group[,group,...]";

Example import:

group "selftest.basic";

import "selftest.tables";   // No we will run 'selftest.tables' whenever 'selftest.basic' is run

iterate

iterate is only supported for C++-tests.

Description: Iterates the contents in a table. See desciption of table for details of how the table content can look. This is basically a for() loop over table data. The table used must be typed.

Level: test_body

Syntax: iterate (IDLIST) from ID { BODY }

Example iterate:

group "Test group 1";

table Table1(int, int, char*)
{
  { 3, 3, "Hello" }
  { 4, 54, "Hi" }
  { 5, 65, "Hej" }
  { 1, 100, "Hejsan" }
}

test( "Test 1.1" )
{
  iterate (x, y, tooltip) from Table1
  {
    verify(x == y); // This test will fail on the second table row, since 4 != 54.
  }
}



language

C is currently synonymous with C++. This might not be the case in the future, though.

Description: Defines which language that is used to write tests. The language can be one of c, c++, or ecmascript. Language can be defined for the following tests in a group, or defined for a specific test. Default language is c++.

Level: group, test

Syntax:

language lang; // On group level
test("test name") language lang; // On test level

Example language:

group "Test group 1";

language ecmascript;

test("Test 1.1"); // Written in ecmascript.
{
    verify(2 == 3);
}

test("Test 1.2") // Written in ecmascript.
{
    verify(2 == 2);
}

group "Test group 2";

test("Test 2.1") // Written in c++
{
    verify(1 == 3);
}

language ecmascript;

test("Test 2.2") language c++; // Written in c++.
{
    verify(1 == 2);
}

test("Test 2.3") // Written in ecmascript.
{
    verify(1 == 2);
}

language c++;

test("Test 2.4") // Written in c++.
{
    verify(1 == 2);
}

test("Test 2.5") // Written in c++.
{
    verify(1 == 2);
}

leakcheck

Leakcheck does not work on all platforms. Currently only windows and linux are supported.

Description: Indicates that leakchecking should be performed for the test

Level: common_test

Syntax: test("test name") leakcheck;

Example:

group "Test group 1";

test("Test 1.1") leakcheck; // Will report that this test leak memory.
{
    char* str = new char[10];
}

test("Test 1.2") // Will not report that this test leak memory.
{
    char* str = new char[10];
}

test("Test 1.3") leakcheck; // Will not report that this test leak memory, since it doesn't leak.
{
    char* str = new char[10];
    delete [] str;
}

manual

Manual tests are only run if -test-manual is used to start the testsuite.

Manual tests are always asynchronous, and thus can not be used when opera has not been initialized

Description: Indicates that a test requires manual interaction

Level: group, test

Syntax: test("test name") manual "message";

Syntax: manual("test name", "message")

Example, test level:

test( "Manual question" )
   manual "Are you there?";
{
 // We can place code here, and use modifiers above. The code is run
 // before the question is put to the user.
}

Example, group level:

manual( "Manual question", "Are you there?" );
// Shorthand for the very common case where you have an empty test
// body. Most commonly this statement is preceded by html {}

multi

Description: multi is a way to run multiple tests from a table without using foreach. It's basically identical to doing iterate on a table, but multiple tests are run instead of just one.

Level: test

Syntax: multi TABLENAME(IDLIST)

Example usage:

// Create the table test_file from a file, with char*, double and int columns.
// Multi requires that the columns are typed (since it's a runtime, not a macro, construct)

table test_file(char *, double, int) read "test_table.txt";

// All $(X), where X comes from the IDLIST in multi, are replaced with the current value.
test( "table test : $(text)" )
    multi test_file(text,f,i);
{
 // test, f and i will take their values from the table.
    verify( text )
    verify( i );
    verify( i == op_ceil(f) );
}

name

See table

noinit

See require

nopch

This should only be used when writing testing windows platform code. On other systems this is a no-op.
Description: Declare the file should no be compiled with precompiled headers. Usable when testing platform code.

Level: top

Syntax: nopch;

or

See table

post

See delay

pre

See delay

read

See table

repeat

It is almost always preferable to use a normal loop in the test.

Description: Defines the number of times a test should run. Default is 1. If the test fails in the n:th run, the test will not run any more time.

Level: test

Syntax: test "test name" repeat INTEGER;

Example repeat:

group "Test group 1";

test("Test 1.1") repeat 3; // Will run 3 times.
{
    verify(a == b);
}

test("Test 1.2"); // Will only run once.
{
    verify(b == c);
}

require

A disabled test counts as a failed one for the purpose of require success and require failure.

Description: Defines a requirement for a test or a group.

require init; This is a test or group that needs to be run after Opera has started. If the command line argument given to selftest included now, this test will not run.

require noinit; This is a test or group that needs to be run before Opera has started. If the command line argument given to selftest did not include now, this test will not run.

require success and require failure are not supported for ecmascript tests yet.

require success "test name"; Requires that the test "test name" succeeded, otherwise this test will not run. Does not work on group level.

require failure "test name"; Requires that the test "test name" fails, otherwise this test will not run. Does not work on group level.

require undefined ID; Requires that the define ID is not defined, otherwise this test or group will not run.

require ID; Requires that the define ID is defined, otherwise this test or group will not run.

Level: common_test

Syntax:

  group "Group";
  require ID; // group level
  test("testname") require ID; // test level

Example require:

group "Test group 1";
require init; // The whole group requires initialization

test("Test 1.1") require FOOBAR_SUPPORT;
// Will only run if FOOBAR_SUPPORT is defined.
{
    verify(a == b);
}

test("Test 1.2") require undefined FOOBAR_SUPPORT;
// Will only run if FOOBAR_SUPPORT is not defined.
{
    verify(b == c);
}

group "Test group 2";
test("Test 2.1") require init;
// Will only run if now is not defined on the command line argument given to selftest.
{
    verify(b == c);
}

test("Test 2.2") require noinit;
// Will only run if now is defined on the command line argument given to selftest.
{
    verify(b == c);
}

test("Test 2.3") require success "Test 2.1";
// Will only run if "Test 2.1" has been run and succeeded.
{
    verify(b == c);
}

test("Test 2.4") failure "Test*:Test 1.2";
// Will only run if all tests named "Test 1.2" in groups whose
// name starts with Test has been run and failed.
{
    verify(b == c);
}

test("Test 1.7") success "Test 2.1";
                 failure "Test 2.2";
// Will only run if "Test 2.1" was run and succeeded, and "Test 2.2" was not run or failed.
{
    verify(b == c);
}

setup

setup is only supported for C++-tests.

Description: A function that will run before all tests in a group. Should setup all global things that the tests need. If you need to run setup code that might fail in a way that affects the selftests after, like OOM, then use a test block instead, and make the subsequent tests depend on the former, using require success "test name".

Level: group

Syntax: setup { c++ statements }

Example setup:

group "Test group 1";

global
{
    int var;
}

setup
{
    var = 3;
}

test("Test 1.1")
{
    verify(var == 3); // Will most likely fail if setup section was removed.
}

subtest

subtest is only supported for C++-tests.

Description: A subtest is code written in c++. It takes a number of parameters and is used by tests as subtests. A subtest can as a test include verify and iterate statements. If a subtest fails, it returns 0, if the subtest passes, it returns 1. To use a subtest in a synchronous test, call it as verify(MySubTest()); thus the test calling it will fail. A subtest can have no modifiers, and must be called by a test written in c++.

Level: group

Syntax: subtest testname(parameters) { body }

Example subtest:

group "Test group 1";

subtest Subtest1(int a)
{
    verify(a == 2);
}

subtest Subtest2(int a, int b)
{
    verify(a == b);
}

subtest Subtest3(int a) // A subtest can call another subtest
{
    verify(Subtest2(a, 2));
}

test("Test 1.1")
{
    verify(Subtest1(2)); // This test will succeed, since SubTest1 succeeds.
}

test("Test 1.2")
{
    verify(Subtest1(1)); // This test will fail, since SubTest1 fails.
}

test("Test 1.3")
{
    verify(Subtest2(2, 2)); // This test will succeed, since SubTest2 succeeds.
}

test("Test 1.4")
{
    verify(Subtest2(2, 3)); // This test will fail, since SubTest2 fails.
}

test("Test 1.5")
{
    verify(Subtest3(2)); // This test will succeed, since SubTest3 succeeds.
}


success

See require

svg

Description: As html, but the type is image/svg+xml

Level: group

Syntax:

svg { DATA } | svg URL { DATA }
For a detailed description, see html

Example svg:

svg {
//! <?xml version='1.0' charset='utf-8'?>
//! <svg xmlns="www.w3.org/2000/svg">
//!    <!--SVG document-->
//! </svg>
}

table

Even when the table data is the contents of a file, it is computed compile-time, not run time. The same is true when you use a list of files.

Description: A table is an array of value arrays. We can use foreach and iterate on a table. A table is very useful to supply data to tests. A table's content is normally an array of literal data. A table can also be a tab separated text file with table data (when using the read modifier). Last, but not least, it can be a simple list of directories and filename globs.

The type list is describing the types in the cells, and can be used in combination with all table variants. It's needed if iterate() will be used on the table, and provides hints for the file parser (e.g., a char* or uni_char* cell will always be a string, even if it's not a string in the table file, and uni_char * strings will have UNI_L() added)

The DIRECTORIES and GLOBS are a list of or-separated string ("foo" or "bar" or "gazonk")

Level: group

Syntax:

table ID[(TYPE_LIST)]? { TABLE_CELL_LIST }
table read STRING;
table filelist DIRECTORIES [name GLOBS]? [recursively]? [directories]?

Example table:

group "Test group 1";

table Table1(int, int, char*)
{
  { 3, 3, "Hello" }
  { 4, 54, "Hi" }
  { 5, 65, "Hej" }
  { 1, 100, "Hejsan" }
}

table Table2 // it is ok to use an untyped table.
{
  { 3, 3, "Hello" }
  { 4, 54, "Hi" }
  { 5, 65, "Hej" }
  { 1, 100, "Hejsan" }
}

// table.txt, content (the list only needs to be space separated, I
// used tab separated above in the text since it is a more common description.
// 3 3   "Hello"
// 4 54  "Hi"
// 5 65  "Hej"
// 1 100 "Hejsan"

table Table3(int, int, char*) read "table.txt";
// Read a table from table.txt. Will be the same table content as in Table1.

table Table4 read "table.txt";
// Read a table from table.txt. Will be the same table content as in
// Table2, but untyped. This table can not be used for iterate, but
// can be used for foreach

table Table5(char*) filelist "." name "*.h";
// Let the content be all file names ending with .h found in the current directory

table Table6 filelist "." name "*.h";
// Same content type as Table5 since char* is the default cell type for the file name.

table Table7 filelist "." name "*.h" or "*.cpp"
// Let the content be all file names ending with .h or .cpp found in the current directory.

table Table8 filelist "." name "*.h" recursively;
// Let the content be all file names ending with .h in the current directory
// and its subdirectories.

table Table9 filelist "." directories;
// Let the content be all file and directory names in the current directory.

table Table10 filelist "." or "../.." name "Makefile*";
// All files whose name starts with Makefile in the current directory
// or the directory two levels up in the directory hierarchy

test

Description: A syntax for writing tests. Depending on defined language, the test can include c, c++, or ecmascript code. The test can also include verify statements and iterate statements. The test can have a number of modifiers with syntax level test or common_test. Finally, the test can have a finally clause. A test can also call subtests.

Level: group

Syntax: test("test name") { body }

Example test:

group "Test group 1";

test("Test 1.1")
{
    verify(2 == 2); // This test will succeed.
}

test("Test 1.2")
{
    int i = 3;
    verify(2 == i); // This test will fail.
}

test_equal

If the capability TS_CAP_SANE_TESTEQ is not defined, test_equal will have a different syntax test_equal( NAME, FORMAT, VAL1, VAL2 ), and is really only useful for comparison of integers (format == "%d").

Description: A syntax for writing simple tests that only verify that two expressions return the same value. test_equal can as modifiers have all keywords with level common_test.

Level: group

Syntax: test_equal("test name", expression_1, expression_2);

Example test_equal:

group "Test group 1";

test_equal("Test 1.1", 2, 2);
// compares if 2 and 2 is equal, will succeed

test_equal("Test 1.2", 2, 3);
// compares if 2 and 3 is equal, will fail

test_equal("Test 1.3", 2.5, 2.5);
// compares if 2.5 is equal to 2.5, will succeed

test_equal("Test 1.4", "Hello", "Hello");
// compares if the string "Hello" has the same content as "Hello", will succeed

test_equal("Test 1.6", GetValue1() + 1, GetValue2());
// Will call the functions GetValue1() and
// GetValue2(), and test if GetValue1() + 1 == GetValue2().


The extra ';' after the options is nessesary, since each option is ended with a ;, as is the test declaration.
test_equal("Test 1.7", GetValue1() + 1, GetValue2()) leakcheck; ; // an example of using a modifier on the test. Will report if we have // leaked memory by running this test.

test_nequal

If the capability TS_CAP_SANE_TESTEQ is not defined, test_nequal will have a different syntax test_nequal( NAME, FORMAT, VAL1, VAL2 ), and is really only useful for comparison of integers (format == "%d").

Description: A syntax for writing simple tests that only verify that two expressions return different values. test_nequal can only have the keywords allowed in common_test as modifiers.

Level: group

Syntax: test_nequal("test name", expression_1, expression_2);

Example test_nequal:

group "Test group 1";

test_nequal("Test 1.1", 2, 2); // compares if 2 and 2 is not equal, will fail

test_nequal("Test 1.2", 2, 3); // compares if 2 and 3 is not equal, will succeed

test_nequal("Test 1.3", 2.5, 2.5); // compares if 2.5 is not equal to 2.5, will fail

test_nequal("Test 1.4", "Hello", "Hello"); // compares if the string "Hello" has different
                                                // content than "Hello", will fail

test_nequal("Test 1.5", p1, p2);
// compares if the pointer p1 points to a different address compared to p2

The extra ';' after the options is nessesary, since each option is ended with a ;, as is the test declaration.
test_nequal("Test 1.5", p1, p2) fails;; // Silly way to write test_equal( "Test 1.5", p1, p2 ) test_nequal("Test 1.6", GetValue1() + 1, GetValue2()); // Will call the functions GetValue1() and // GetValue2(), and test if GetValue1() + 1 != GetValue2().

text

Description: As html, but the type is text/plain

Level: group

Syntax:

text { DATA } | text URL { DATA }
For a detailed description, see html

Example text:

text {
//! If a<b, then b>a
}

timer

Description: Measures the time a test takes to execute. This is usable with both asynchronous and synchronous test. Note that the time is measured with the system timer so don't expect to much accuracy.

Level: test

Syntax:

test("test name") timer; { ... }

Example text:

test("t")
    repeat 1000;
    timer;
{
    /* Do something */
}

undefined

See require

verify

The parser recognizes several simple expressions following the pattern X OPERATOR Y and tries to write easy to read error messages.

Description: Verify that an expression evaluates to true. A test will stop running the test body immediately if a verify statement fails and a subtest will return to its calling test if a verify statement fails. The optional message is appended to the output and allows the test to provide extra info about what should happen if the test fails. The message needs not to be wrapped in quotes, and if so, can span multiple lines.

Level: test_body

Syntax: test("test name") { verify(boolean_expression[, message]); }

Example:

group "Test group 1";

test("Test 1.1") // Will succeed and print "Success1" and "Success2"
{
    verify(2 == 2);
    output("Success1");
    verify(3 == 3);
    output("Success2");
}

test("Test 1.2") // Will fail and not print anything
{
    verify(2 == 3);
    output("Success1");
    verify(3 == 3);
    output("Success2");
}

test("Test 1.3") // Will fail after printing "Success1"
{
    verify(2 == 2);
    output("Success1");
    verify(2 == 3);
    output("Success2");
}

subtest SubTest(int a, int b)
{
    verify(a == b);
}

test("Test 1.4") // Will succeed and print "Success", since SubTest() will succeed.
{
    verify( SubTest(2, 2) );
    output("Success");
}

test("Test 1.5") // Will fail and not print anything, since SubTest() will fail.
{
    verify( SubTest(2, 3) );
    output("Success");
}

test("Test 1.6") // Will fail if a != b and append the optional message in the error output
{
    verify(a == b, a did not have the same value as b);
}

verify_files_equal

Description: verify_files_equal has the exact same semantics as verify. It checks if two files are equal, byte for byte. It means that the files' endings must also match.

The file paths should be absolute. In most cases the 'file' directive will be used to provide full absolute paths to files in the selftest data directory.
Since the 'file' directive doesn't work in EcmaScripts you have to be inventive there. One of the solutions is to create a pseudo test in C++ that adds variables to EcmaScript runtime before the actual test is run.

Just like verify, verify_files_equal also supports an optional error message

Level: test_body

Syntax: test("test name") { verify_files_equal(reference_file_path, tested_file_path[, message]); }

Example:

group "Test group 1";

test("Test 1.1")
    file uni reference_file_str "reference_file_contents.txt";
    file uni test_file_str "output_file.txt";
{
    /* generate some data in file output_file.txt */
    verify_files_equal(reference_file_str, test_file_str); // succeeds if files have identical content
}

verify_not_oom

Description: verify_not_oom has the exact same semantics as verify. It checks if a pointer has a NULL value, and if so, reports an Out-of-memory error, similar to verify_status. This can be used to check that allocations during testing succeed.

Just like verify, verify_not_oom also supports an optional error message

Remarks: this function is only available in C++ tests.

Level: test_body

Syntax: test("test name") { verify_not_oom(expr[, message]); }

Example:

group "Test group 1";

test("Test 1.1")
{
    Object x = new Object();
    verify_not_oom(x); //succeeds if allocation does not fail
    delete x;
}

test("Test 1.1") fails; //test succeeds
{
    verify_not_oom(NULL);
}

verify_string

Description: verify_string has the exact same semantics as verify, but allows easy and quick comparison of strings. When the strings are not equal, they are output, together with the error log. You can combine the following four types in any of the arguments.

Just like verify, verify_string also supports an optional error message

Remarks:

This function is only available in C++ tests.

This function treats an uninitialized OpString as equal to an empty string, and not equal to NULL.
Consider this:

verify_string(UNI_L(""), OpStringC()); // passes 
verify_string(NULL, OpStringC()); // fails

Level: test_body

Syntax: test("test name") { verify_string(lhs, rhs[, message]); }

Example:

group "Test group 1";

test("Test 1.1") //succedds
{
    verify_string("hello", "hello");

    OpStringC16 str_1(UNI_L("hello"));
    verify_string("hello", str_1);

    verify_string(NULL, NULL);
}

test("Test 1.2") //fails and prints the two compared strings in the output
{
    verify_string("not", "equal");
}

verify_success, verify_status, verify_trap

Description: verify_success(), verify_status() and verify_trap() have the exact same semantics as verify, but slightly different use cases. verify_success(expression) and verify_status(expected_status, expression) test the result of an expression with the return value of an OP_STATUS, verify_trap(FunctionL()) can be used to verify the success of function call that may LEAVE.
verify_success() and verify_trap() take only one argument and test if the value returned is success, resp. if the called function did not LEAVE (it calls Opstatus::IsSuccess() implicitly).
verify_status(expected_status, expression) takes two arguments and tests if the value returned by the expression (2nd argument) equals the reference value (1st argument). If the conditions fail, the resulting error code, from the 2nd argument, is output, and for some known values, like those in OpStatus, it's converted into a readable string.
These functions, just like verify, also support an optional error message.

Remarks: these functions are only available in C++ tests. It's highly recommended to use this function to check for status codes, because foreign conditions like out of memory can be flagged in SPARTAN as ERRORs, instead of FAILs.

Level: test_body

Syntax: test("test name") { verify_success(expression[, message]); verify_trap(expression with a function that may leave[, message]); verify_status(error_code, expression[, message]); }

Example:

group "Test group 1";

global {
	int MyFunctionL(const char* arg) { if (!arg) LEAVE(OpStatus::ERR_NULL_POINTER); return 17; }
	OP_STATUS MyFunction(void* arg){ return arg ? OpStatus::OK : OpStatus::ERR_NULL_POINTER; }
	OP_STATUS GetOOM(){ return OpStatus::ERR_NO_MEMORY; }
}

test("Test 1.1") //prints Success if Myfunction returns OpStatus::OK
{
    int x;
    verify_success(MyFunction(&x));
    output("Success");
}

test("Test 1.2") //prints Success if Myfunction returns OpStatus::ERR_NULL_POINTER
{
    verify_status(OpStatus::ERR_NULL_POINTER, MyFunction(NULL));
    output("Success");
}

test("Test 1.3") //if this test runs on spartan, it can tell that it failed due to oom, and not some unknown error
{
    verify_success(GetOOM());
}

test("Test 1.4")
{
    int x;
    verify_trap(x = MyFunctionL("foo")); // passes
    verify_trap(x = MyFunctionL(0)); // fails, because MyFunctionL() leaves
}

xml

Description: As html, but the type is text/xml

Level: group

Syntax:

xml { DATA } | xml URL { DATA }
For a detailed description, see html

Example xml:

xml {
//! <?xml version='1.0' charset='utf-8'?>
//! <document>
//!    XML document
//! </document>
}