Flower: Introduction

Flower is a build tool designed to replace GNU make for building Opera products on UNIX-like systems. Flower is written in Python and included in the Opera source tree, so no installation is required.

Why Flower?

There are several issues with GNU make that make it unsuitable for Opera. These issues can be worked around more or less successfully, as demonstrated by the existence of the unix-build and minimake modules. However, having the right tool for the job is better than forcing the wrong one to do it. The high complexity and maintenance costs of unix-build demonstrate that, and although minimake is clearly a step in the right direction, it's still restricted by the limitations of GNU make.

Make builds the full dependency graph before it starts building.
This is the biggest and most fundamental limitation of make. Also, pretty much every alternative build tool mirrors this design decision.
Make operates in two phases. First, it parses the makefiles and builds a graph of all targets describing how they depend on each other; second, it traverses the graph, executing commands on the way to update the targets. This is true even though make has some ways to express generic rules, such as pattern rules. In Opera products, we have certain parts of the dependency tree unknown until some setup steps have run. For example, the list of all source files is produced by a hardcore setup step.
One possible solution to this is to run all such steps during parsing of the makefiles (first phase), which is slow because it's not always necessary, and ugly because we can't use make's dependency engine to express when they need to be run and when they don't. Another approach is to invoke make recursively after executing the setup steps (this is the approach taken by minimake). This is somewhat nicer because unnecessary invocations of the setup steps can be spared, but still a workaround with unnecessary complexity which could be avoided if we didn't have to fight make's architecture.
Flower solves this by not requiring the entire dependency graph to be known before it begins traversing it. Instead, a node references and brings to life its dependencies when asked to build itself.
Make's configuration system is a flat namespace of string variables.
There are no lists, dictionaries or even boolean values. As a result, all we can do with those variables is string manipulations, which are often difficult to read and write, and are picked out of make's quite limited set of operations.
Flower's configuration system is hierarchical and can use all data types available in Python, so that strings don't have to be abused for expressing structured information.
Make's output log is difficult to read.
Make doesn't really have proper logging; it just lets the commands it runs write directly to the standard output and standard error, and optionally echoes each command before invoking it. When running many commands in parallel, this results in an unreadable mess.
Flower has an extensible logging facility that captures and reorders the output of individual commands so that it doesn't intermix during parallel execution.
One has to read the makefiles to figure out how to invoke make for the desired result.
Sophisticated makefiles contain hundreds of targets and configuration variables, some of which are internal and some meant to be invoked or configured by the end user wishing to build the project. Well-documented makefiles help to a certain point.
In Flower, goals and options exposed to the end user are explicitly declared as such, and become part of the self-documented command line syntax. Invoking Flower with --help generates a description of all such goals and options dynamically.

Flower contains two major subsystems that can be described separately: the flow subsystem and the configuration subsystem.

Flow subsystem

This is what roughly corresponds to the target and dependency system of GNU make.

Nodes and flows

Flower operates on nodes that can be thought of as relatives of targets in GNU make. A node is an abstract entity that does not necessarily correspond to a file. Every node is associated with an algorithm (a piece of Python code) that describes how to make it; this algorithm is called a flow. This notion gave Flower its name.

One flow can be associated with a single unique node, but more commonly it's associated with a family of similar nodes. For example, there are many similar nodes sharing the same flow that compiles a source file into an object file; the only difference between them is the actual source (and therefore also target) file the flow operates on. Such a flow is written once and used for many similar nodes, somewhat like when a class in an object-oriented programming language is written once and then instantiated many times.

Flower looks for flows in every module listed in a readme.txt file. Any module can contain a module.build directory at its top level. Flower looks for flow files named flow.py or *.flow.py in these directories. They are essentially Python source code and are all read as if they were was one long Python module. The order in which flows are defined doesn't matter. There is just one flat namespace for all flows.

A close look at a flow

Here is an example of a flow that compiles a source file. (The actual compileSource flow is a little more complex.)

@flow
def compileSource(self, source):
    self['target'] = config.targetPlatform.compiler.target
    yield sourceSetup()
    if self < (util.readDepFile(config.targetPlatform.compiler.depFile) or [source]):
        util.makeDirs(self)
        yield command(config.targetPlatform.compiler.compileCommand, "Compiling %(source)s")

A flow is essentially a Python function wrapped in the @flow decorator that adds a bit of magic around it. The first formal argument of the flow is always self which will refer to the node object when the flow runs. The remaining formal arguments are keyword arguments that distinguish one such node from all other nodes using the same flow. In this case, the flow is used once per source file, and the name of that source file is the parameter that makes it possible to identify the exact node. If another flow was to refer to a particular node using this flow, it would use an expression like compileSource(source='dir/file.cpp'). Note the absence of the self argument here: it will be supplied by the decorator when it instantiates the node.

The source argument in the flow above is a node parameter. Every node has a dictionary of such parameters. They are available through the array subscript syntax on the node object, as demonstrated by the first line of the flow above: it adds a node parameter target, which by convention is used to store the name of the output or timestamp file produced by the flow. A node is not required to have a target.

The call to util.makeDirs makes sure that the destination directory is created before running the compiler (again, the target node parameter is used to determine the location of the output file).

The long dotted expressions starting with config are queries into the configuration subsystem, which has the next section dedicated to it. The configuration system knows which node is querying it, and tailors its answers to the particular node based on its node parameters. For example, the config.targetPlatform.compiler.target expression returns the name of the object file that will be produced by compiling the source file, according to the naming conventions used by the particular compiler on the particular platform (with typical UNIX C/C++ compilers, this means taking the source node parameter and replacing the filename suffix with .o).

Depending on other nodes

Flows typically depend on other nodes' flows having completed. For example, the compileSource flow requires that the sourceSetup() node has completed. If not for the parallelism, flows could have called each other as ordinary functions, but Flower supports parallel execution, so it requires that flows yield execution (they are technically Python generators) instead of calling other flows. This is what the flow above does: it yields a the sourceSetup() node. A yield statement in a flow can pass a single node or a sequence of nodes (a sequence is any container that has a length and is indexable, such as a list or a tuple).

To get a reference to a node you want made, you invoke its flow as a function. Doing so does not in itself cause the flow to be executed. In our example, sourceSetup is a flow corresponding to one unique node, that's why it's called with no arguments; if it was a flow that's used for many nodes, keyword arguments would have to be specified, narrowing it down to a single node. If you evaluate such an expression several times, you'll get references to the same object every time, because a node with a particular flow and each particular combination of node parameters is unique.

A yield statement in the body of a flow with one or more references to other nodes means that the execution of this flow should be suspended until each of these nodes completes its flow. By the time such a statement is reached, some of the nodes listed might be already made, some might be in progress because of the parallelism, and some might not yet be started upon. The current flow will be suspended, and other flows will be scheduled for execution. The Python code itself is not multi-threaded; flows only multitask by yielding, which allows the scheduler to let a different flow start or continue.

Checking if the target needs updating

Next, the flow checks if the target file is older than the files used to produce it. The source is obviously one such file. The util.readDepFile function is used to parse a .d file produced by the compiler as a side effect (e.g. enabled with gcc's -MMD flag), if one already exists, and returns a list of dependencies mentioned in it. Obviously, this list includes source. If the .d file does not exist, or None is passed instead of its name, the function returns an empty list, and [source] is used as fallback. (Note that parentheses are necessary to enforce the desired evaluation order.) There is no need to run any dependency generation in advance: if a .d file does not exist, this means the file hasn't been compiled before, and the target won't exist either, forcing the remake.

The comparison self < deps uses the overloaded < operator of the node object. The operator should be read as older than in this context. On the right-hand side, it accepts a single filename or node or a sequence of those, and returns true if the target needs to be remade. For the node on the left-hand side and for any nodes listed on the right-hand side, the operator uses the target node parameter as the name of the file whose modification time to use. Absent target file forces a remake, and so does an absent dependency.

Invoking a command

Finally, another yield statement is executed, this time with a command object. This means that the current flow is to be suspended, and the command should be scheduled for execution. It might not be started right away because of the configured limit on the number of processes to run in parallel. When the command completes, the current flow will be resumed (in our case, only to succeed immediately). The scheduler might let other flows run while the current flow is blocked on a command object.

The first argument of the command constructor is a list of words making up the command. In this case, the configuration system supplies the compiler command, which can be quite long, but if we were to hardcode a minimal gcc invocation here, it would look like this: yield command(['g++', '-c', source, '-o', self['target'], '-MMD'], "Compiling %(source)s"). The command is invoked directly without using the shell, so you don't have to escape any special characters, for example, in filenames. The optional second argument to the constructor is the progress message that should appear in the output describing what's being done. Named substitutions in it can refer to the parameters of the current node. If the command fails to start or returns a non-zero exit code, the flow does not continue, and the whole build fails (this behavior can be overridden for a particular command object, if necessary).

How nodes are identified

Note how the name of the flow is a verb phrase rather than a noun phrase referring to the output file, such as objectFile. Unlike GNU make, in Flower the node is primary, and the file it creates or updates is secondary (and doesn't always exist). The whole build process is a flow invoking other flows invoking other flows, each of which accomplishes something like compiling a source file or linking a binary. Also, note that among all the nodes using this flow, a particular one is identified using the source file name rather than the output file name. This is an arbitrary design decision made for this particular flow, in this case because when we think of a build step such as compiling a source file, what we really care about is which source file we are compiling, not where the output is written. It also makes it simpler to refer to all such nodes after reading a list of all source files to be compiled. Here is another simplified example that produces the Opera binary file:

@flow
def linkBinary(self):
    self['target'] = config.binaryTarget
    scan = scanSources()
    yield scan
    sources = [compileSource(source=s) for s in scan['sources']]
    yield sources
    if self < sources:
        util.makeDirs(self)
        yield command(config.targetPlatform.linker.linkCommand, "Linking %(target)s")

This example assumes that only one binary is ever produced, so the linkBinary flow does not expect any node parameters. The real linkBinary flow takes a binary parameter which specifies which binary we are building (opera, operapluginwrapper-native etc).

First, a scanSources() node is brought up to date. Its flow makes sure that all necessary hardcore scripts have been run that produce the source list, and then reads the list and makes it available as the sources node parameter on itself. Then, we use a list comprehension (Python FTW) to build a list of nodes for compiling each of the source files. We yield on the list, making sure all the source files are compiled before the next line of our flow is reached. Then, we reuse the sources list to check if any of the object files are newer than the binary (remember that for each node in the list, its target node parameter is used). The linker command is again supplied by the configuration system, but if we were to hardcode it, it might have looked like this: yield command(['g++', '-o', self['target']] + [s['target'] for s in sources], "Linking %(target)s"). (In practice, such linker commands can grow too long, so the actual linkBinary flow writes the list of object files into a text file and refers to that using the @ character on the command line.)

Flow variants

There is a pattern-matching system for flows that allows one of several flow variants to be selected based on the specified node parameters. For example, there may be one way to compile C files and another to compile C++ files. In most places outside the compileSource flow we don't care, so we'd like the expressions compileSource(source='file1.c') and compileSource(source='file2.cpp') to select the right flow variant automatically. This can be accomplished by providing more than one flow with the same name and specifying some matching patterns as arguments to the @flow decorator:

@flow(source=util.hasSuffix('.c'))
def compileSource(self, source):
    # Compile a C source file...

@flow(source=util.hasSuffix('.cpp'))
def compileSource(self, source):
    # Compile a C++ source file...

Here, util.hasSuffix is a handy matcher that checks if a string has the specified suffix. A constant string, a compiled regular expression object, or a function returning a boolean-like value can be used as a matcher; see the reference manual for details. The right flow variant will be selected automatically. The order of declaration doesn't matter. No match or more than one match is a runtime error.

In the example above, most of the code in both flow variants would be the same. To avoid duplication, flow variants can be chained. In the example below, both flow variants set some node parameters on self and chain to the common flow variant containing the shared code, which then uses those parameters.

@flow(1, source=util.hasSuffix('.c'))
def compileSource(self):
    self['stem'] = util.removeSuffix(source, '.c')
    self['lang'] = 'c'
    yield self

@flow(1, source=util.hasSuffix('.cpp'))
def compileSource(self):
    self['stem'] = util.removeSuffix(source, '.cpp')
    self['lang'] = 'c++'
    yield self

@flow
def compileSource(self, source):
    # Compile a C or C++ source file...
    # using self['stem'] and self['lang'] where necessary

The first positional argument to @flow is the priority of the variant; variants with different priorities are allowed to match at the same time, and the highest-priority one is selected. A priority is a signed integer and defaults to 0.

The special yield self statement is a magical instruction to Flower that the execution should continue to a matching flow variant with the next highest priority. The current flow variant never continues beyond yield self. In the example above, the low-priority flow variant (with the default priority of 0) matches every invocation of compileSource. Because the variants matching .c and .cpp files have a higher priority, one of them will be executed, but then will chain to a matching flow variant with the next highest priority, which is the bottom one in the example. It is a runtime error if yield self fails to find a match with a lower priority, or encounters a tie between two or more matching flow variants.

Goals

A flow can be declared a goal that can be invoked from the Flower command line. This is done with the @goal decorator:

@goal('all', 'Build the entire product.')
@flow
def buildAll(self):
    yield linkBinary()

With this declaration, when flower all command is run, Flower will execute the buildAll flow. The first argument of @goal defines the keyword to be specified on the command line to invoke this goal. Flower's command line syntax is self-documenting, and when flower --help is invoked, all will be listed as a valid goal name, and the second argument of @goal will be used as its description. Most flows, however, are internal and not meant to be invoked by the end user; these are not declared as goals, which helps keep the command line syntax clean. When no goal is specified on the command line, config.goal configuration query is used to select the default, which will typically be all.

Goals can declare arguments that will be translated to node parameters (see the reference manual for details):

@goal('compile', 'Compile a single source file.',
      {'arg': 'source', 'help': 'The source file to compile.'})
@flow
def compileSource(self, source):
    ...

Configuration subsystem

This roughly corresponds to variables in GNU make.

Queries and answers

Flower's configuration subsystem provides information used at various stages of the build process. The configuration system is best described as providing answers to questions (queries) at runtime. Examples of such questions are: What compiler flags do we need? Do we want to build with debug information or not? Is the target platform little-endian or big-endian?

A configuration query cannot require significant computations to produce an answer, and cannot depend on flows being run. For example, the list of all source files to compile would not be a valid configuration query because this list is not known before hardcore setup runs. Instead, such information should be obtained from nodes whose flows perform the non-trivial processing. However, configuration queries are allowed to invoke simple commands that take insignificant time to complete, such as pkg-config or gcc --version.

The configuration system uses appropriate Python types for its answers. Strings are never abused to represent structured data; lists, dictionaries, sets and objects are used where appropriate. For primitive values, strings, numbers and booleans are used. None is used to represent the absence of a meaningful primitive value or an object, but not of a structure like a list or dictionary because those can simply be empty.

Providing a simple answer

The configuration subsystem is accessed from flows and other Python code using the config global variable. Read-only properties of this object provide answers to configuration queries. For example, config.debugSymbols yields true if debug information is to be produced, and false otherwise. In the configuration files, which are essentially Python source files, answers are provided by declaring functions with the names of configuration queries. For example:

def debugSymbols():
    return config.debugMode

This example returns true whenever another configuration query, config.debugMode, returns true. Note that, even though debugSymbols is a function, any code making the query uses the property access syntax (config.debugSymbols) rather than the method call syntax (config.debugSymbols()): the configuration subsystem takes care of calling the function. It may implement some caching of answers in the future, but at the moment, Flower's performance looks decent without any caching, so none was implemented. Anyway, the configuration functions should always be written in such a way that they yield the same result on repeated invocations in the same context (see below). It is illegal for a configuration query to depend on its own answer directly or indirectly; this will cause a stack overflow.

If our answer did not depend on another configuration query and was in fact a constant, we could have used this shorthand form instead of declaring a function:

debugSymbols = True

The above is equivalent to declaring a function always returning True.

It's common to use util.runOnce to run utilities producing interesting standard output. The results are cached, so the same command won't be run more than once. For example, util.runOnce(['gcc', '--version']) returns the output of gcc --version or raises errors.CommandFailed if the command didn't succeed (return exit code 0). The shell is not used to invoke the command, so nothing should be escaped.

Configuration files

The order in which the configuration files are read matters: a configuration file overrides anything declared by the files already read. The exact order in which configuration files are read if found is documented in the reference manual.

A configuration function overriding a setting in another configuration file can use the global name default to access the value being overridden:

def includePaths():
    return default.includePaths + ['/usr/local/include']

Here, the function takes the value it's overriding, which is a list of include paths, and appends one more item to it. Unfortunately, shorthand syntax like includePaths = default.includePaths + ['/usr/local/include'] cannot be used because the computation can depend on the context and needs to be run at flow execution time rather than parse time.

The default global object should only be used to refer to the setting the function is overriding; all other configuration queries should be made using the config global object because they may be overridden by the subsequently read configuration files.

Configuration objects

Some values returned by configuration queries are configuration objects through which further queries can be made by accessing their read-only properties, leading to hierarchical queries like config.targetPlatform.compiler.compileCommand. This is accomplished by declaring classes in configuration files and returning instances of such classes from queries. Here is a real example:

import sys

def hostPlatform():
    return config.AutoPlatform()

def targetPlatform():
    return config.hostPlatform

class AutoPlatform(object):
    # ...
    def bigendian(self):
        return sys.byteorder == 'big'
    # ...

The query config.hostPlatform returns a platform configuration object providing information about the platform Flower is running on. This default configuration returns an instance of the AutoPlatform class (where auto refers to auto-detection of platform properties). The platform object is expected to answer a number of queries about the platform, one of which is bigendian (whether or not the platform is big-endian). Queries are defined on the platform object rather than in the global configuration scope if the answers to them can change depending on whether we are cross-compiling. config.targetPlatform defaults to config.hostPlatform (they'll differ when cross-compiling) but can be overridden to return an instance of some other class providing different answers to the same set of queries.

Note how hostPlatform accesses the AutoPlatform class through the config global object: this allows all overrides of the AutoPlatform class to take effect.

The same magic applies to the properties of the configuration objects as to top-level configuration queries: the caller uses the property access syntax, and the configuration subsystem automatically calls methods of the configuration object. Shorthand syntax for constant answers can be used in classes as well.

Note that configuration classes must be new-style Python classes, that is, they must subclass object rather than nothing.

Another prominent example of a configuration object is the compiler property of a platform configuration object. An expression like config.targetPlatform.compiler returns an object describing the compiler that runs on the host platform and produces files for the target platform. By default, AutoPlatform returns instances of the GCC class.

Classes can be overridden in the configuration files, too. This is accomplished by subclassing the class with the same name that you are overriding:

class AutoPlatform(default.AutoPlatform):
    def compiler(self):
        return config.LLVM()

This overrides the compiler property of AutoPlatform while leaving all other properties unmodified. If the overriding method needs to refer to the method being overridden, it should use super and the property access syntax:

class GCC(default.GCC):
    def optimizeFlags(self):
        return super(GCC, self).optimizeFlags + ['-funsafe-math-optimizations']

Answers depending on the context

The answer to any configuration query can depend on the parameters of the node whose flow issues the query. That node is called the context in which the query is initiated. To use the node parameters, a top-level configuration function or a method of a configuration class should accept formal parameters with names corresponding to the node parameters:

def optimizeSize(source):
    return not source.startswith('modules/ecmascript/')

This means that source files under modules/ecmascript should be optimized for speed, while everything else should be optimized for size. This query will cause a runtime error if issued from a flow where the node doesn't have a source parameter; if it's desirable that the query is answerable even then, the function should declare a default value for the argument. A configuration function or method can also use the **kwargs syntax to capture a dictionary of all node parameters in the current context.

Note that even a function that doesn't declare explicit named arguments can end up depending on the context because it can make other configuration queries that do.

Some configuration queries are invoked with an empty context by the system itself, such as config.processQuota that controls how many parallel processes can be run. All such queries are initially defined in platforms/flower/default.py. However, the vast majority of the configuration queries are entirely user-defined and are only used by flows and other configuration queries.

It's possible to ask the configuration subsystem a what-if question by invoking the config global object as a function with one or more keyword arguments. For example, the expression config(source='file1.cpp').optimizeFlags evaluates to what config.optimizeFlags would have returned had the source parameter in the current context been 'file1.cpp'. The rest of the node parameters in such an expression are unchanged from the current context.

Options

A top-level configuration function (but not a method of a configuration class) can be declared as a command-line option using the @option decorator:

@option('--debug-symbols', 'Produce debug information.')
def debugSymbols():
    return False

This declares that --debug-symbols on the command line should override config.debugSymbols with a true value, and supplies the help text to be used for the option in the output of flower --help. The configuration subsystem queries the answer to the query before parsing the command line and documents that as the default setting in the help text; it also uses the type of the default to restrict the values that can be specified. For a boolean option, its negation like --no-debug-symbols will also be documented and recognized unless this behavior is explicitly suppressed. Note that @option cannot be used with the shorthand syntax like debugSymbols = False because of the restrictions of the decorator syntax in Python.

Other design decisions

This section outlines other notable design decisions made when developing Flower.

The top-level source directory is the origin

Flower changes into the top-level source directory before doing anything else. All files are referred to by their canonical names relative to that location, such as platform/unix/product/main.cpp. This simplifies references to files in the source tree compared to the unix-build module which refers to files using paths relative to its own location, resulting in the annoying and unreadable use of ../.. in many paths.

Always distinct buildroot and simple cleaning

All files produced by the build are always put in a dedicated directory called the buildroot. This is mandatory; unlike the unix-build module, Flower does not allow the buildroot to coincide with the top of the source tree. The default buildroot location is build (a subdirectory at the top level of the source tree), but it can be located outside the source tree, too.

All hardcore setup and other scripts run during the build process have been patched to support a buildroot that does not coincide with the top of the source tree.

As a result, Flower doesn't need to keep track of any generated files to clean. The clean goal simply removes the buildroot directory. The only files generated during the build that end up outside the buildroot are .pyc files generated by Python when compiling modules, since the location of these files cannot be controlled. These are not removed by the clean goal and are harmless (cannot disrupt a subsequent build in any way); git clean -fx can be used if removal of the .pyc files is nevertheless desired.

Note that you should never share a buildroot directory between unix-build and Flower. It's best to delete the old directory upon transition to Flower.

Extensible logging

All output to the console and log files is handled by logger objects. They are essentially listeners that can react to various events such as a command starting, command completing etc. More than one logger can be active at the same time and react to the same events. A logger can ignore some events while reacting to others, and different logging formats and levels of detail can be achieved through add-on logging classes.

The standard output and standard error of each command is captured via pipes by a background thread, which is the only thread other than the main thread that Flower uses. The output of the commands is split into lines. A logger has a chance to react to a command's output on two occasions: first immediately when a line of output is fully read from the pipe (real-time logging), and later when the command eventually completes or fails and the list of all lines it has written to the standard output and standard error is available to the logger as an attribute of the command object.

Flower contains two fully functional logger classes at the moment: a console logger and a text file logger.

An instance of the console logger is created to log to the standard output and standard error. The level of detail is controlled by several configuration queries and the corresponding command-line options such as -v and -q. The console logger uses real-time logging to avoid delaying the output of the commands as they run, which can lead to some intermixing of the output when parallel processes are active. This logger uses ANSI control sequences to color different kinds of output such as progress messages, commands' standard output and standard error, etc, unless coloring is disabled or the output is redirected. Coloring was introduced especially so that it becomes the single universally hated feature of Flower, and thus all the other dubious design decisions go unnoticed.

At the same time, a text file logger is created to log to a file (build.log by default). This logger rotates the log files to keep a configurable number of old log files under names such as build.log.1, build.log.2 etc. The text file logger uses a different format from the console logger. In particular, it does not log the commands' output in real time, and instead writes the entire output of a command into the log when the command completes or fails. This leads to a more readable log file where the output of the commands that ran in parallel does not intermix.

Because the output goes to two places (the console and the text log file) by default, the user will seldom need to redirect Flower's output.

No user-defined code such as flows or configuration queries should write to the standard output or standard error. The logging system takes care of progress reporting automatically, and if something goes wrong, user-defined code should raise an exception (which will be reported). Warnings tend to be treated as noise and ignored, so everything should either work or fail. In a future version of Flower, the logging system may provide a way to issue warnings with support for “known issues” that don't need to be reported to everyone building the product.

Parallel whenever possible

Flower defaults to running as many parallel commands as the machine has CPU cores. There is seldom any reason to disable that, especially because the log file is always readable even when many parallel processes are run. However, the config.processQuota configuration query can be overridden on any level, including with the command-line option -j.

Various hardcore setup steps are run separately, which allows most of them to be run at the same time (however, some of them depend on others, which introduces some sequencing).

Compilation of each selftest is also run separately, resulting in a massive improvement of the build time because of parallelization. The hardcore step to generate selftests is not used, and the Pike script is invoked directly for every selftest. This script has been modified to allow invocation for an individual selftest.

Dependency tracking in hardcore scripts

Hardcore setup scripts have been patched to accept the new --timestamp option. Running a hardcore script with --timestamp FILE results in creation or touching of an empty FILE (which is touched even if no output files have been updated). In addition, FILE.d is created or overwritten. The latter is a GNU makefile snippet that declares FILE as depending on every file or directory that the script has read or tried to read (for files that are currently absent but whose presence would have affected the output, $(wildcard ...) syntax is used). This means that if one of these dependencies becomes newer than the timestamp FILE, the hardcore step needs to be re-run, but not otherwise. Similar modifications have been made to the selftest compiler.

This can be used in a build system based on GNU make as well. In Flower, the util.readDepFile function understands just as must of the GNU make syntax as is necessary to read these dependency files (and those generated by gcc -MMD, too). The flows that run hardcore steps read the dependency files to avoid running the hardcore scripts needlessly.

As a result, Flower successfully avoids running any commands unless necessary. When invoked the second time after a complete, successful build, Flower does not run any commands, and only spends an order of a second on a modern developer workstation to detect that everything is up-to-date. (Only 50 ms of that is system time, so there is room for optimization.)

Core and products

The core delivery, once it hopefully incorporates Flower, will contain flow and configuration files in the module.build directories of the relevant modules such as hardcore.

Some of the configuration files will be indifferent to the order in which they are read, for example because they add items to lists they override, such as the list of include paths. Such configuration files should have normal names like, for example, gcc.conf.py for a file overriding some aspects of the GCC configuration class.

Other configuration files will establish configuration queries and classes that can be overridden by other modules. It's important that these are read before any files overriding them are, therefore such configuration files should go early in the lexicographical order. By convention, names like 00-gcc.conf.py should be used in these cases.

Products using the core delivery should override aspects of Flower configuration in their own modules rather than by patching Core. Core should define enough configurables to allow the degree of flexibility required by the products without resorting to patching.

The product is expected to define a default goal specific to the project. For example, in Desktop, the default goal produces operating system packages.

In the rare cases when a product needs to override a particular flow in Core, it should define a flow variant with a priority of 10 or more.

Currently UNIX-only

This is not actually a design goal (doesn't sound like an admirable one, in any case). Technically, little stands in the way of applying Flower to platforms that don't currently use GNU make. However, as it happens, Windows and Mac users prefer the build process to be controlled by the IDE, and their platform-specific scripts generate project files for the IDE rather than running the build directly. While Flower could be used to build Opera on these platforms, it's questionable whether anybody needs this. Therefore, the current development of Flower focuses on providing good support for the platforms currently using GNU make.

Currently, Flower uses very little UNIX-specific code (actually, POSIX-specific, so it will work on MacOS X, too). Porting to Windows would require a few changes in the process scheduler because it uses a POSIX-specific way of waiting for completing child processes and reacting to signals, in the output-capturing background thread.

There are also some areas where, although Flower is not directly incompatible with Windows, it assumes a design philosophy biased towards UNIX. For example, / is always used as the separator of file path components (which is so much simpler to read and write than using something like os.path.join everywhere). The command line syntax is typical of UNIX tools. ANSI control sequences are used to color the console output. The dependency tracking and some aspects of the compiler configuration make assumptions about the compiler that may only be true for GCC and compilers imitating GCC, which most compilers on UNIX do. Some output-handling code may be careless about the line-break conventions.