# module.sources

$Id$

- Introduction
- Adding Source-File Options
- Options
  - Precompiled header
  - Jumbo compile
  - Allow System Includes
- Sources Setup
  - Sources Setup for module developer
  - Sources Setup for platform developer
- Overriding Configuration Locally
- References

## Introduction

Each module has a file `module.sources` which lists all sources in the module. Each source-file is listed with its full path relative to the module directory on a single line. The path-separator must be a slash "/".

Example:

Suppose you have a module `foo` which has the source files `foo_module.cpp`, `src/foo_file_1.cpp` and `src/foo_file_2.cpp`, then the corresponding `module.sources` should be

    foo_module.cpp
    src/foo_file_1.cpp
    src/foo_file_2.cpp

Empty lines are ignored and everything after a hash character ("#") is regarded as a comment.

## Adding Source-File Options

Some source-files need special attention. They must be compiled with different options. The `module.sources` file allows to specify a set of options to one single source-file or to a group of source-files.

The options are specified as a semicolon (";") separated lists of key-value pairs. A single key-value pair is specified as "key" or "key=value". If the value is omitted, it is assumed to be "1". So the option "key" is the same as "key=1". The options are specified in square-brackets after a hash character:

    # [option_1;option_2=some_value;option_3=0]

Note:

- There may only be whitespace between the hash character ("#") and the open square bracket ("[") if there is any text, the options are regarded as comments: `# this [option] is a comment`
- Whitespaces in options are significant, i.e. `# [foo=bar]` and `# [ foo= bar ]` define two different keys ("foo" and " foo") and two different values ("bar" and " bar ").

Options can be specified for a single file by adding the option specification "`# [options]`" to the line with the source-file:

    src/foo_file_1.cpp # [foo=bar]

Options can also be specified for a group of files by adding the option specification "`# [options]`" to a single line before the files:

    # no option for file_1.cpp:
    file_1.cpp
    # specify option_1 and option_2 for file_2.cpp and file_3.cpp:
    # [option_1;option_2=value_1]
    file_2.cpp
    file_3.cpp
    # specify only option_2 for file_4.cpp and file_5.cpp
    # [option_2=value_2]
    file_4.cpp
    file_5.cpp

Note:

- Each time you specify a new set of group-options in a single line, all previous options are reset. In the example above the "option_1" is not set for `file_4.cpp` and `file_5.cpp`.
- Options for a single line can override group options. In the following example all files have "option_1" and "option_2", but for `file_1.cpp` "option_2" has value "2" (as defined in the single-file option) and for `file_2.cpp` "option_2" has value "1" (as defined in the group option) and only `file_1.cpp` has "option_3":

      # [option_1;option_2=1]
      file_1.cpp # [option_2=2;option_3]
      file_2.cpp

## Options

The following options are supported by core:

- pch or no-pch: Precompiled header
- jumbo or no-jumbo: Jumbo compile
- system_includes: Allow System Includes

### Precompiled header (pch)

- pch: use a precompiled header (default for all files).
- no-pch: don't use a precompiled header.

Opera's source-code has many header files that are included in every source file. The time the compiler takes to process these header files over and over again can account for nearly all of the time required to build the project. To make builds faster, many compilers allow to "precompile" a header file; then, if builds can use the precompiled header file they will be much faster.

Opera's precompiled header is `core/pch.h`. Each source-file starts with including the precompiled header:

    /**
     * Some copyright header...
     */
    #include "core/pch.h"
    ...

For some source files it might be necessary for some reason to not use a precompiled header. Those files should have the option no-pch. The option pch (= use precompiled header) is default.

Note:

- For gcc the option "no-pch" or "pch" is not needed, because gcc uses a precompiled header automatically if it finds the header (see [1]).
- Visual Studio projects need to configure in the project file which precompiled header to use. See also the sources script in ... TODO

### Jumbo compile (jumbo)

- jumbo: include this source-file into the jumbo-compile-unit `{module}_jumbo.cpp` for jumbo compile (where `{module}` is the name of the module). This option is the default for all source-files.
- jumbo=compile_unit.cpp: include this source-file into the specified jumbo-compile-unit `compile_unit.cpp` for jumbo compile.
- no-jumbo: don't include this source-file into any jumbo-compile unit. Instead this file will be compiled as a single file.

A big overhead on compiling source-files is to start the compiler and load the (precompiled) header files for each source-file. One way to optimize compile time is to use a precompiled header. Another step is to combine multiple source files of one module into one jumbo-compile-unit by `#include`ing the source-files. In this case the jumbo-compile-unit uses a different pre-compiled header file `core/pch_jumbo.h`, because some compilers don't support to include the same precompiled header file twice.

The jumbo-compile-units are automatically generated by the operasetup script. In the simplest case all source-files of a module are included in one jumbo-compile-unit.

Example: If the module "foo" defines the `foo/module.sources` file as:

    src/file_1.cpp
    src/file_2.cpp
    src/file_3.cpp

Then a single jumbo-compile unit `foo/foo_jumbo.cpp` is created:

    /**
     * Some copyright header...
     */
    #include "core/pch_jumbo.h"

    #include "modules/foo/src/file_1.cpp"
    #include "modules/foo/src/file_2.cpp"
    #include "modules/foo/src/file_3.cpp"

It is possible to exclude a single file (or multiple files) from a jumbo-compile-unit by adding the option "jumbo=0" or "no-jumbo".

It is also possible to split several files into different jumbo-compile-units, e.g. to hide some declarations from some other source-files. This is done by specifying a filename as the value of the "jumbo" option.

Example: If the module "foo" defines the `foo/module.sources` file as:

    src/file_1.cpp # [no-jumbo]
    # [jumbo=src/foo_jumbo_1.cpp]
    src/file_2.cpp
    src/file_3.cpp
    # [jumbo=src/foo_jumbo_2.cpp]
    src/file_4.cpp
    src/file_5.cpp

Then two jumbo-compile units `foo/src/foo_jumbo_1.cpp` and `foo/src/foo_jumbo_2.cpp` are created:
`foo/src/foo_jumbo_1.cpp`:

    /**
     * Some copyright header...
     */
    #include "core/pch_jumbo.h"

    #include "modules/foo/src/file_2.cpp"
    #include "modules/foo/src/file_3.cpp"

`foo/src/foo_jumbo_2.cpp`:

    /**
     * Some copyright header...
     */
    #include "core/pch_jumbo.h"

    #include "modules/foo/src/file_4.cpp"
    #include "modules/foo/src/file_5.cpp"

Note:

- The precompiled header of a jumbo-compile-unit is set to `"core/pch_jumbo.h"`. That header file is basically the same as `"core/pch.h"`, but some compilers can only handle one precompiled header in a compile-unit and all included files already include `"core/pch.h"`.
- Only files with the same "pch" and "system_includes" options can be joined into one jumbo-compile-unit.
- If all files in a jumbo-compile-unit have the option "no-pch" or "pch=0", then the jumbo-compile-unit will not be compiled with precompiled headers.
- If all files in a jumbo-compile-unit have the option system_includes, then the jumbo-compile-unit will not use precompiled headers and it will include `"core/pch_system_includes.h"` instead of `"core/pch_jumbo.h"`.

### Allow System Includes

- system_includes: allow the source-file to include system header files.

If the option system_includes is set, then the source-file includes `core/pch_system_includes.h` instead of `core/pch.h`. The only difference between those two files is, that `core/pch_system_includes.h` defines `ALLOW_SYSTEM_INCLUDES`.

This is currently used for different purposes:

- The module lea_malloc sets the option system_includes for some source-files and defines the symbol `COMPILING_LEA_MALLOC`. Depending on the platforms' configuration of lea_malloc (see TWEAK_LEA_MALLOC_PLATFORM_CONFIG), the platform may require to include some system header files to configure lea_malloc; this may e.g. include access to the function `malloc()`.
  No core-code (with the above exception) may use the functions `malloc()`, `free()` and friends directly. Instead the corresponding `op_`* functions (like `op_malloc()`) or the macros `OP_NEW` and friends must be used.
  This allows the platform's `system.h` to add some compile-time check that `malloc` and `free` are not used.

  Example:
  With the following defines in the platform's system.h, the compiler prints an error message that the symbol `dont_use_...` is not available if `malloc` or `free` are used directly.

      /* platforms/my_platform/config/system.h */
      ...
      /* get errors if malloc, free and friends are used */
      #define free    dont_use_free
      #define calloc  dont_use_calloc
      #define realloc dont_use_realloc
      #if !defined(COMPILING_LEA_MALLOC)
      /* lea_malloc's MORECORE is configured to use the system's malloc,
       * so if we compile lea_malloc we don't want to get this compile
       * error (see my_lea_malloc_config.h): */
      # define malloc  dont_use_malloc
      #endif // !COMPILING_LEA_MALLOC
      ...

  Only lea_malloc may use malloc, because the platform's lea_malloc-configuration uses malloc as backend:

      /* platforms/my_platform/config/my_lea_malloc_config.h */
      ...
      /* Use system malloc as backend for lea_malloc */
      #define MORECORE malloc
      ...

- The file modules/stdlib/stdlib_pi.cpp has the option system_includes set and defines the symbol `COMPILING_STDLIB_PI`. Depending on the system's configuration, `stdlib_pi.cpp` may need to include some system header files and use some system functions (like `sscanf()` or `snprintf()`). No other core-code may use the stdlib functions directly. Instead the corresponding `op_`* functions (like `op_sprintf`) must be used. This allows the platform's `system.h` to add some compile-time check that the stdlib functions are not used.

  Example:
  With the following defines in the platform's system.h, the compiler prints an error message that the symbol `dont_use_...` is not available if `sprintf()` etc are used directly.

      /* platforms/my_platform/config/system.h */
      ...
      // first define the system's capabilities:
      #define SYSTEM_SNPRINTF   YES // or NO
      #define SYSTEM_SPRINTF    YES // or NO
      #define SYSTEM_STRTOD     YES // or NO
      #define SYSTEM_SSCANF     YES // or NO
      ...

      #if defined(COMPILING_STDLIB_PI) && !defined(STDLIB_DTOA_CONVERSION)
      // If this platform doesn't use thirdparty dtoa code, the stdlib module
      // needs to implement the some functions in
      // modules/stdlib/stdlib_pi.cpp, so if we include this header file
      // while compiling the file stdlib_pi.cpp, we might need (depending on
      // the above system definitions) require some system functions:

      # if !SYSTEM_STRTOD && SYSTEM_SSCANF
      // stdlib's StdlibPi::StringToDouble() uses the system's sscanf()
      // (not op_sscanf()) which is provided by including stdio.h:
      #  define INCLUDE_STDIO_H
      # else // SYSTEM_STRTOD || !SYSTEM_SSCANF
      // Either StdlibPi::StringToDouble() is not defined because the
      // system's strtod() can be used (SYSTEM_STRTOD == YES), or stdlib's
      // StdlibPi::StringToDouble() uses an approximate implementation which
      // does not need the system's sscanf() but is good enough in many
      // cases (SYSTEM_STRTOD == NO and SYSTEM_SSCANF == NO)
      # endif // !SYSTEM_STRTOD && SYSTEM_SSCANF

      # if SYSTEM_SNPRINTF
      // stdlib's StdlibPI::SPrintfDouble() uses the system's snprintf() to
      // print a double, so include stdio.h to provide the declaration of
      // snprintf()
      #  define INCLUDE_STDIO_H
      # elif SYSTEM_VSNPRINTF
      // stdlib's StdlibPI::SPrintfDouble() uses the system's va_list,
      // va_start(), va_end() and vsnprintf() to print a double, so
      // include stdio.h to provide the declaration of vsnprintf and include
      // stdarg.h to provide the declaration of va_start() etc.
      #  define INCLUDE_STDIO_H
      #  define INCLUDE_STDARG_H
      # elif SYSTEM_SPRINTF
      // if there is no snprintf nor vsnprintf, stdlib's
      // StdlibPI::SPrintfDouble() uses the system's sprintf() to print a
      // double, so include stdio.h to provide the declaration of sprintf()
      #  define INCLUDE_STDIO_H
      # elif SYSTEM_VSPRINTF
      // if there is no snprintf nor vsnprintf, stdlib's stdlib's
      // StdlibPI::SPrintfDouble() uses the system's va_list, va_start(),
      // va_end() and vsprintf() to print a double, so include stdio.h to
      // provide the declaration of vsprintf and include stdarg.h to provide
      // the declaration of va_start() etc.
      #  define INCLUDE_STDIO_H
      #  define INCLUDE_STDARG_H
      # else // !SYSTEM_S(N)PRINTF && !SYSTEM_VS(N)PRINTF
      // stdlib cannot provide an implementation for
      // StdlibPI::SprintfDouble(), so the platform must do it:
      # endif // !SYSTEM_S(N)PRINTF || !SYSTEM_VS(N)PRINTF
      #endif // COMPILING_STDLIB_PI && !defined(STDLIB_DTOA_CONVERSION)

      ...
      #ifdef INCLUDE_STDIO_H
      # include
      #else // !INCLUDE_STDIO_H
      // generate a compile error if anybody still uses the system stdio
      // functions instead of the op_... functions:
      # define sscanf dont_use_sscanf
      # define snprintf dont_use_snprintf
      # define sprintf dont_use_sprintf
      # define vsnprintf dont_use_vsnprintf
      # define vsprintf dont_use_vsprintf
      ...
      #endif // INCLUDE_STDIO_H
      #undef INCLUDE_STDIO_H

      #ifdef INCLUDE_STDARG_H
      # include
      #endif // INCLUDE_STDIO_H
      #undef INCLUDE_STDARG_H
      ...

- The file modules/encodings/testsuite/charconv/utility.cpp has the option system_includes set and defines the symbol `COMPILING_ENCODINGS_CHARCONV_TESTSUITE`. On a windows or linux platform some system headers are needed if FEATURE_SELFTEST is enabled.

  On a windows platform (i.e. `WIN32` is defined), the header file `<windows.h>` is included to use the functions `::VirtualAlloc()` and `::VirtualFree()`.

  On a linux platform (i.e. `linux` is defined), the header file `<sys/mman.h>` is included to use the function `mprotect()`.

  The platform's `system.h` might need to prepare for these system header files to be included to avoid conflicting declarations.

- The file modules/sqlite/generated/sqlite_sqlite3.c has the option system_includes set and defines the symbol `COMPILING_SQLITE`.

  If FEATURE_3P_SQLITE is enabled, that file includes several system header files like `<stdarg.h>`, `<stdio.h>`, `<windows.h>` (on windows systems) ...

  The platform's `system.h` might need to prepare for these system header files to be included to avoid conflicting declarations.

- The sources files in modules/libopeay/ are a wrapper around the openssl implementation. Some files of that module need to include some system header files (depending on system and configuration) like `<assert.h>`, `<stdlib.h>`, `<stdio.h>`, `<string.h>`, `<unistd.h>`, `<errno.h>`, `<time.h>`, `<io.h>`, `<math.h>`, (windows like systems:) `<tlhelp32.h>`, `<windows.h>` `<tchar.h>` `<wincrypt.h>`, `<lcmons.h>`, `<lmstats.h>`, ...

  The libopeay sources are only compiled if `FEATURE_3P_OPENSSL` is enabled or if some parts of the code is used (e.g. see `TWEAK_CRYPTO_ENCRYPTION_AES_USE_CORE_IMPLEMENTATION` which is enabled by default for all profiles).

  The platform's `system.h` might need to prepare for these system header files to be included to avoid conflicting declarations.

- The source files in platforms/vega_backend/ is the hardware backend implementation for libvega. There are e.g. implementations for Direct 3d and OpenGL.

  The source files in platforms/media_backends/ provide a default implemetnation of OpMediaPlayer.

  Those implementations need to include several system headers. The platform's `system.h` might need to prepare for these system header files to be included to avoid conflicting declarations.

## Sources Setup

The operasetup script parses all `module.sources` files and generates list of files depending on the options pch, jumbo and system_includes in `modules/hardcore/setup/plain/sources/` and `modules/hardcore/setup/jumbo/sources/`.

In each of those directories the files `sources.all`, `sources.nopch`, `sources.pch`, `sources.pch_jumbo` and `sources.pch_system_includes` are generated. The format of these files is simple:
The files are plain text files, each line of the file contains a single path to a source file. The path is relative the source root. The file contains no comments.

- sources.all contains a list of all source files.
- sources.nopch contains a list of all source files that have the option no-pch set, i.e. which should be compiled without a precompiled header.
- sources.pch contains a list of all source files which should be compiled with `core/pch.h` as precompiled header, i.e. the option pch is set but no jumbo-compile (no-jumbo) nor system includes (no-system_includes) are set.
- sources.pch_jumbo contains a list of all source files which should be compiled with `core/pch_jumbo.h` as precompiled header, i.e. the options pch and jumbo are set but system includes (no-system_includes) is not set.
- sources.pch_system_includes contains a list of all source files which should be compiled with `core/pch_system_includes.h` as precompiled header, i.e. the options pch and system includes (no-system_includes) are set, but jumbo-compile (jumbo) is disabled.

The generated sources files are used by the update_vcproj tool to update the list of sources in Visual Studio project files.

The following table shows to which source.* files a source-file is added, depending on the different combinations of the pch, jumbo and system_includes options.

pch: | yes | yes | yes | yes | no | no | no | no
--- | --- | --- | --- | --- | --- | --- | --- | ---
jumbo: | yes | yes | no | no | yes | yes | no | no
system_includes: | yes | no | yes | no | yes | no | yes | no
plain/sources.all | + | + | + | + | + | + | + | +
plain/sources.nopch | - | - | - | - | + | + | + | +
plain/sources.pch | - | + | - | + | - | - | - | -
plain/sources.pch_jumbo | - | - | - | - | - | - | - | -
plain/sources.pch_system_includes | + | - | + | - | - | - | - | -
jumbo/sources.all | + | + | + | + | + | + | + | +
jumbo/sources.nopch | + | - | - | - | + | + | + | +
jumbo/sources.pch | - | - | - | + | - | - | - | -
jumbo/sources.pch_jumbo | - | + | - | - | - | - | - | -
jumbo/sources.pch_system_includes | - | - | + | - | - | - | - | -

Note: the intersection of the source-files listed in sources.nopch, sources.pch, sources.pch_jumbo and sources.pch_system_includes is empty, because a single source file may only be compiled with a single preceompiled header.

Note: the union of the source-files listed in sources.nopch, sources.pch, sources.pch_jumbo and sources.pch_system_includes is equal to the set of source-files listed in sources.all.

### Sources Setup for module developer

A module developer who needs to maintain a module should set the options according to the following rules:

Usually a source-file does not need an option. It should be compiled with the default options (i.e. pch, jumbo and no-system_includes). The file should `#include "core/pch.h"`. And the file will be added to the module's default jumbo-compile-unit.

If a source-file needs to include some system header file (like `<sdtio.h>`) then the option system_includes should be set and the source-file should `#include "core/pch_system_includes.h"`.
Note: Files with different system_includes options cannot be compiled in the same jumbo-compile-unit, so you may have to use the option jumbo to organize the source files of the module into different jumbo-compile-units.

If a source-file cannot be compiled with a precompiled header file (neither with `core/pch.h` nor with `core/pch_system_includes.h`), e.g. because the source-file needs to `#define` something before #including the pch header file, then the source-file should get the option no-pch.
Note: In addition the option system_includes may (or may not) be set, see the paragraph above. Note: Files with different pch options cannot be compiled in the same jumbo-compile-unit, so you may have to use the option jumbo to organize the source files of the module into different jumbo-compile-units.

### Sources Setup for platform developer

A platform developer who needs to compile all opera sources can use the sources setup to get a list of all files to compile.

First the platform developer should consider whether to use jumbo compile or plain compilation:
Jumbo compilation is usually faster than plain compilation because the compiler does not need to be started and it does not need to load the header files for each single file.
Jumbo compilation was tested with gcc and visual studio, but there may be compilers for which the source files grow too big.

If jumbo compilation should be used, then the sources.* files from `modules/hardcore/setup/jumbo/sources/` should be used.
If plain compilation should be used, then the sources.* files from `modules/hardcore/setup/plain/sources/` should be used.

If a compiler supports precompiled headers, then

- Create precompiled headers for `core/pch.h`, `core/pch_jumbo.h` and `core/pch_system_includes.h`.
  Note: These header files are automatically generated by operasetup in the directory `modules/hardcore/setup/*/include/`. Either add that path to the compiler's include directory or copy the file to a place where they can be found.
- Compile all files in `sources.nopch` without precompiled headers.
  In Visual Studio a single source file can be configured to not use precompiled headers.
  gcc doesn't use a precompiled header if it was compiled with a different commandline option.
- Compile all files in `sources.pch` with precompiled header `"core/pch.h"`.
- Compile all files in `sources.pch_jumbo` with precompiled header `"core/pch_jumbo.h"`.
  Note: this file is empty if you choose plain-compilation.
- Compile all files in `sources.pch_system_includes` with precompiled header `"core/pch_system_includes.h"`.

If a compiler does not support precompiled headers, then comile all files in `sources.all`.

## Overriding Configuration Locally

If a module has a file `module.sources.override` in its root directory, this file fully overrides the contents of the `module.sources` file; that is, it is read instead and the `module.sources` file is completely ignored. The file format and interpretation of the contents is identical.

The purpose of this exception is to allow developers to locally override the configuration in the `modules.sources` file without having to have locally modified version controlled files. An example of changes a developer might want to do locally is to fully or partially disable jumbo compilation for some modules. Since the idea is to avoid having locally modified version controlled files, a `modules.sources.override` file should of course never be checked in.

## References

- [1] GCC online documentation: "Using Precompiled Headers" http://gcc.gnu.org/onlinedocs/gcc/Precompiled-Headers.html
- [2] MSDN: MSDN Library: Development Tools and Languages: Visual Studio 2008: Visual Studio: Visual C++: Building a C/C++ Program: C/C++ Building Reference: Compiling a C/C++ Program: "Creating Precompiled Header Files" http://msdn.microsoft.com/en-us/library/szfdksca.aspx