Flower: Reference Manual

This is a reference manual on the user-exposed API of Flower. See Introduction for a more general discussion of the project's architecture.

Flow subsystem

The flow subsystem resides in the flow module. The system automatically makes the names flow (the decorator), command and goal available in the global namespace in the context of which the flow files are interpreted. Flow files do not need to import this module explicitly unless they need to access other names than the ones mentioned above.

Flower looks for flow files named module.build/flow.py and module.build/*.flow.py in every module listed in a readme.txt file. The order in which flow files are loaded does not matter.

@flow (decorator)

The use of this decorator is mandatory when defining a flow. The argument list is optional. The decorator can be applied to more than one function with the same name, in which case they are considered variants of the same flow. Which one is used in each case will be decided through pattern matching and priorities. The decorator accepts one optional positional argument, priority, and zero or more keyword arguments declaring the patterns to match against the node parameters specified when instantiating the node. In case of a tie when two or more flow variants with the same priority match the specified node parameters, errors.AmbiguousMatch is raised. If no flow variants match, errors.NoMatch is raised.

priority (integer, default: 0)
Defines the priority of the flow variant. Signed. Higher values are preferred when looking for a match.

Each keyword argument describes a matcher for the node parameter of the same name. All matchers specified as arguments to @flow must match in order for the flow variant to match the instantiation of a node. The possible matcher types are:

Callable
If a matcher is a callable, it will be called with a single argument containing the value of the node parameter. The match is considered successful if the matcher returns a true value, and unsuccessful otherwise. The util module provides several useful matchers of this type.
Compiled regular expression
If a matcher is a compiled regular expression object as produced by re.compile from the standard Python library, it is matched against the value of the node parameter, which must be a string, using the search method of the regular expression object. In fact, anything with a search method will be treated this way.
Constant
If neither of the above cases apply, the value of the node parameter will be tested for equality to the matcher.

The function being decorated (flow function) must declare a first positional argument, by convention called self. When the function gets called by the flow subsystem, this argument receives the current node object. The flow function may accept any number of additional keyword arguments, which will receive the values of the node parameters with corresponding names. If an argument does not specify a default value, the node parameter by that name is mandatory when instantiating nodes. Node parameters that are listed neither among the patterns as arguments of the @flow decorator nor among the keyword arguments of the function being decorated must not be specified when instantiating a node. If the flow function uses the **kwargs syntax to accept arbitrary keyword arguments, arbitrary node parameters are allowed.

If an uncaught exception of any type escapes the function, the flow is considered to have failed, and so are all flows directly or indirectly waiting for the current one to complete. If a node's flow fails, and the node has a target parameter, the file named by the parameter is deleted as a precaution to prevent incompletely generated output files from being considered as valid by the next build. The same happens when a build is interrupted with a signal.

Any finally blocks in a flow whose corresponding try blocks have been entered are guaranteed to run. However, the order in which such blocks in flows of different nodes will run is undefined.

yield (statement)

When the yield statement is used in a flow, the following expressions can be supplied.

Node
If the node's flow has not already been started, it is scheduled to be started. The current flow is suspended until the specified node's flow completes. If the flow of the specified node is already in progress, the current flow joins the set of flows blocked by it. If the flow of the specified node has already completed, the current flow is not suspended. If the flow of the specified node fails, or has already failed, with an exception, or because it yielded on a node or a command that failed, the current flow is considered to have failed as well, and is not resumed. While a flow is suspended, other flows that are not blocked by anything will run. If a flow yielding on a node produces a circular dependency, errors.CircularDependency is raised.
Command
The command is scheduled to be started. The current flow is suspended until the command completes. By default, if the command fails to start or returns a non-zero exit code, the command is considered to have failed, in which case the current flow fails as well and is not resumed. This behavior can be customized when creating the command object.
A sequence of Node and/or Command objects
(A sequence is any container that has a length and is indexable; tuples and lists are typical.) The same as described above happens for each node and command in the sequence, and the current flow is suspended until all listed nodes and commands complete. The flows and commands are not guaranteed to be executed in the specified order; some of the nodes' flows might already be in progress or completed, and some or all of the commands may be run in parallel. If any of the listed flows or commands fails, the current flow is considered to have failed as well and is not resumed.
self
If a flow yields a reference to the current node using the yield self statement, the execution chains to a matching flow variant with the next highest priority. The current flow variant is not resumed.

User-defined code should not spawn any long-lived child processes by any means other than yielding a Command object. Short-lived processes are allowed, that is, if their lifespan doesn't extend past a yield or a return from a flow, and the user-defined code waits for its completion (util.runOnce is an example of how short-lived child processes can be run safely). Also, user-defined code is not allowed to call os.wait (to wait on all child processes) directly or indirectly; doing so will disrupt the scheduler.

Node (class)

Represents a node.

The constructor is not public. Instances of the class are obtained by invoking flows as functions, specifying zero or more node parameters as keyword arguments. No positional arguments are allowed. The keyword arguments both are used for pattern matching to decide which flow variant to use, and become the initial set of the node parameters. Parameters from the current context are not inherited. Repeated invocation of the same flow with the same set of keyword arguments produces references to the same node. Each flow function receives a reference to the current node as the first and only positional argument.

A node object conforms to the dictionary protocol and supports dictionary-like indexing with string keys, membership testing, and iteration. When used as a dictionary, the node represents the set of node parameters. The node parameters are dynamic and can be added, modified or deleted at any time. However, the set of parameters used to instantiate the node is frozen for purposes of obtaining a reference to the same node by invoking the flow with the same set of keyword arguments.

A node object also supports the context manager protocol and, when used in a with statement, becomes the current context for the duration of the statement's body.

Other members defined on node instances (all properties described below are read-only):

state (integer)
The state of the node's flow, one of the following:
Node.PENDING
The node's flow is not currently running, but is scheduled to be run (not blocked on anything).
Node.RUNNING
The node's flow is currently running. There is at most one such node at any given time.
Node.BLOCKED
The node's flow is suspended until one or more Node and/or Command objects complete.
Node.COMPLETED
The node's flow has completed successfully. The state never changes after this.
Node.FAILED
The node's flow has failed. The state never changes after this.
error (tuple)
A tuple (type, value, traceback) (in the same format as returned by sys.exc_info) with information about the exception that caused the flow to fail, or a tuple of three None values if no exceptions have occurred in the flow. For some internally generated exceptions such as errors.CommandFailed, traceback will be None.
< (operator)
This operator is used to check whether a node needs updating. With the Node on the left-hand side, the operator accepts a string, a Node object, or a sequence of strings and/or Node objects. (A sequence is any container that has a length and is indexable.) The operator returns true if any of the following conditions are true: For Node objects appearing on the right-hand side, the optional changed parameter serves as an override. If it's true, the dependency is considered to be newer than the target; if false, it's considered to be older. In either case, the existence and timestamp of the target are not checked. Also note that, as follows from the above, when the right-hand operand is an empty sequence, the operator returns true if and only if the file named by the left-hand operand's target parameter does not exist.

Command (class)

Represents a command.

Instances are obtained by calling the global command function. Unlike flows, repeated invocations of command with the same arguments produces distinct command objects.

Arguments of command:

args (list or string)
A sequence of command words (the command and its arguments) if shell is false, or a single command string if shell is true.
message (string, default: None)
The progress message to display when starting the command. The string is interpreted as a new-style Python format string (as used with str.format) into which the paremeters of the current node are interpolated as named items. How, when, and whether the progress message actually appears is up to the active loggers. None means no progress message.
fail (boolean or callable, default: True)
Specifies whether the command should be considered failed if it fails to start or completes with a non-zero exit code. If a callable, will be called under the described circumstances with the command object as the only argument to obtain a boolean-like value.
shell (boolean, default: False)
Whether the command should be passed to the shell. If false, the command is invoked directly, and the arguments don't have to be escaped.
cwd (string, default: None)
If not None, the directory which should be made current before executing the command. Flower always operates with the top-level source directory as current, so None as cwd leaves that unchanged. A cwd value affects only the particular process but not any subsequently spawned child processes or Python code.
env (dictionary, default: None)
A dictionary of environment variables to pass to the child process. If None, the environment is inherited from the Flower's main process.

Members defined on command instances (all properties are read-only):

state (integer)
The state of the process. Can be one of the following:
Command.PENDING
The process is not currently running, but is scheduled to be run.
Command.RUNNING
The process is currently running.
Command.COMPLETED
The process has completed successfully. The state never changes after this.
Command.FAILED
The process has failed. The state never changes after this.
pid (integer)
The ID of the child process or None if not yet started.
returncode (integer)
The exit code of the child process or None if still running or not yet started.
error (tuple)
A tuple (type, value, traceback) (in the same format as returned by sys.exc_info) with information about the exception that caused the command to fail, or a tuple of three None values if no exceptions have occurred. For some internally generated exceptions such as errors.CommandFailed, traceback will be None.
command (string)
The command string produced from args, with special characters escaped in arguments so that the string is usable as a shell command. If shell is true, the shell command is returned unchanged. In either case, the string includes a cd command (if cwd is not None) and environment assignments (if env is not None).
args (list)
The value of args used to create the command.
message (string)
The value of message after interpolation, or None if there is no message.
shell (boolean)
The value of shell used to create the command.
cwd (string)
The value of cwd used to create the command. Can be None.
env (dictionary)
The value of env used to create the command. Can be None.
output (list)
A list of output lines written by the command so far, in the chronological order. Each line is an object with the following properties:
text (string)
The content of the line, without the terminating newline character.
stderr (boolean)
True if the line was written to the standard error, false if it was written to the standard output.

@goal (decorator)

Declares a flow as a goal that can be selected from the command line.

The decorator should be used above the @flow decorator. If the flow has more than one variant, the decorator should be used on exactly one of them, does not matter which one.

The decorator takes two mandatory positional arguments:

name (string)
The keyword that identifies this goal on the command line. Must be unique for every goal. Case-sensitive. Stick to printable ASCII.
desc (string)
A one-line description of the goal that will appear in the help text.

In addition, any number of extra positional arguments is accepted, each describing a positional non-option goal argument that can be specified on the command line and will translate into a node parameter. Each such argument is a dictionary with the following entries:

arg (string)
The name of the node parameter to which this argument maps. This is the only mandatory entry in the dictionary.
help (string)
A one-line description of the parameter that will appear in the help text.
default (string or integer)
The default value to be used in case the argument is omitted from the command line. If None (the default), the argument is mandatory. A mandatory argument cannot follow an optional argument.
type (string, default: 'string')
The type of the argument, either 'string' or 'int'.
metavar (string, default: the value of arg in upper case)
The meta-variable that will be used to refer to the parameter in the help text, such as FILE.
value (any type)
If this is specified (even if None), the argument merely specifies a hardcoded value for the node parameter to be used when instantiating the goal. Such an argument does not consume anything when parsing the command line, and is not mentioned in the help text.

If the goal's flow sets the result node parameter on self to a string, the string will be logged when the goal completes. This should include a one-line explanation of what has been done and where the output file(s) were placed.

Configuration subsystem

The configuration subsystem resides in the config module. The system automatically makes the name config available in the global namespace in the context in which flow files are interpreted, and the names config, default (if available) and option in the global namespace in the context in which configuration files are interpreted. User-defined code does not need to import this module explicitly.

Configuration file loading order

Loading order matters for configuration files: those loaded later override anything before them. Everything listed below, except for platforms/flower/default.py, is optional.

  1. platforms/flower/default.py;
  2. module.build/conf.py and module.build/*.conf.py in every module listed in a readme.txt file (sorted by the filename rather than module name, so a name like 00-gcc.conf.py can be used to make sure the file is read before the files overriding its contents in other modules)
  3. files listed in config.confFiles by any of the configuration files read so far, and by the configuration files read in this step, repeated until no files are listed that haven't been read before (each particular file is only ever read once); by default, this includes:
    1. /etc/opera.conf.py, /usr/local/etc/opera.conf.py (can be used to set options specific to a build server);
    2. the file named by the FLOWER_USER_CONF environment variable, or ~/.opera.conf.py by default (a user can set their preferred options here);
    3. *.conf.py at the top level of the source tree (a user can set per-checkout options here; listed in .gitignore);
  4. options specified on the command line are treated like a configuration file read at this point;
  5. extra configuration files mentioned on the command line, and the files they list in config.confFiles.

config (object)

This global object is the primary entry point into the configuration subsystem.

The object exposes a read-only property for every top-level configuration query defined in any of the configuration files read. Reading the property evaluates the query in the context of the currently running flow. If several configuration files define the property, the most recently read one is selected.

Evaluation of the property depends on how the configuration file defines it.

Class
The class is returned after applying the same transformation to all its methods and properties whose names don't start with an underscore as described here for top-level properties.
Function
The function (or method of a configuration object) is invoked. If the function declares keyword arguments, they receive the corresponding node parameters from the current context. If a keyword argument does not have a default value, the configuration query cannot be evaluated in a context without the corresponding node parameter. If the function uses the **kwargs syntax, all node parameters are passed to it.
Constant value
A deep copy of the value is returned. The copying makes sure that a mutable value such as a list is not modified.

In addition to the read-only properties for top-level configuration queries, the config object provides the following operators:

() (operator)
When invoked as a function with one or more keyword arguments, the config object returns a temporary object with a similar public interface, which answers configuration queries in the context modified by the specified arguments. Node parameters not modified in this way are left unchanged from the current context. This operation can be applied repeatedly. If one of the parameters specified in this way is named softFallback, and it is set to true, the temporary object will apply soft fallback logic when evaluating queries: if an exception is raised during evaluation, instead of failing, the previous overridden configuration function is used, and so on. The soft fallback logic does not apply to configuration class methods.
+= (operator)
Add another configuration file, whose settings take precedence over those from any files added so far. The right-hand operand must be a callable or a Python file object. If a callable, it will be called with the dictionary representing the configuration file namespace as the only positional argument; it is expected to insert overrides into the dictionary just as loading a file would.

default (object)

This global object is available to all configuration files except for the first one read. The interface is similar to that of config, except that the += operator is not available.

The object answers configuration queries using only the configuration files read before the current one. The only purpose of default is to allow a function overriding a configuration query to access the value it overrides (typically to append items to a list). The default object should only be used to access the value being overridden, not to make any other configuration queries (use config for that).

@option (decorator)

Declares a configuration function as an option that can be set from the command line.

The decorator can only be used with a function, not with a class or a constant assignment.

The decorator takes two mandatory positional arguments, opt and desc, and a number of optional keyword arguments:

opt (string or sequence of strings)
A string or a sequence of strings defining the command-line options. (A sequence is any container that has a length and is indexable.) The options can be short (a dash followed by one letter) or long (two dashes followed by a word). Case sensitive.
desc (string)
A one-line description of the option that will appear in the help text.
value (boolean, default: None)
For a boolean option, the value that this option sets when specified. Up to two @option decorators can be put on the same function with with different value arguments. The default behavior for a boolean option when value is not specified is that the option sets a true value, and the opposite option name is generated automatically (--with- is swapped for --without-, --enable- for --disable-, and --no- is added otherwise). Short options are not inverted. The inversion is not performed when value is specified.
default (any type convertible to string, default: None)
The default value to display in the help text. This is only for documentation purposes and does not affect the actual default used when the option is not specified. The default behavior when default is not specified is to use the actual value obtained from reading the configuration files up to the point of parsing the command line.
metavar (string, default: None)
The meta-variable that will be used to refer to the option's value in the help text, such as FILE. The default is N for integer options, STRING for strings and lists, and KEY=VALUE for dictionaries.
replace (boolean, default: False)
Only for list or dictionary options. If true, the first use of the option on the command line clears the list or dictionary; subsequent uses add items to it. If false, every use of the option adds an item.

The type of the option is determined by its default value obtained from reading the configuration files up to the point of parsing the command line (soft fallback logic is used to evaluate the configuration queries for the purpose of obtaining defaults). The default should not be None. The allowed types are boolean, integer, string, dictionary mapping strings to strings, and list of strings. In the latter two cases, the option can be repeated on the command line to specify more list items or key-value pairs. If the value of a dictionary option does not contain a = character, the value of the key-value pair is set to None.

otree (module)

The module exposes properties that describe the modules comprising the Opera source tree. The information is obtained by reading the readme.txt files.

The module is automatically made available as the name otree in the global namespaces in the context of which the flow files and configuration files are interpreted.

modsets (dictionary)

A read-only dictionary mapping the name of each module set (there are currently four: modules, data, adjunct and platforms) to a module set object with the following read-only properties:

name (string)
The name of the module set.
modules (dictionary)
A dictionary mapping the short name of each module in the set to a module object (see below).

modules (dictionary)

A dictionary mapping the full name of each module to a module object with the following read-only properties:

name (string)
The short name of the module (not containing a slash).
fullname (string)
The full name of the module (the module set name, a slash, and the module name).
file(name)
A method taking a file name relative to the module's directory, opening the file for reading and returning a Python file object or None if the file does not exist. Useful for reading optional files like module.sources.
glob(pattern)
A method taking a glob pattern relative to the module's directory and returning a list of files matching the pattern, also relative to the module's directory.

util (module)

An assortment of useful functions and classes.

The module is automatically made available as the name util in the global namespaces in the context of which the flow files and configuration files are interpreted.

ensureList(items)

The function checks whether items is a sequence other than a string, and if it isn't, wraps items in a single-element tuple, ensuring that the result is always a sequence. A sequence is any container that has a length and is indexable. If items is None, an empty tuple is returned.

touch(file)

The function creates an empty file with the specified name or updates the last modification time of the existing file, imitating the UNIX touch command. The file argument can be the name of the file or a Node whose target parameter names the file. The latter behavior was introduced especially so that you could write touch(self) in your flow code and maintain an innocent look on your face.

hasSuffix(suffix)

Returns a matcher usable with the @flow decorator that checks whether the value has the specified suffix.

hasPrefix(prefix)

Returns a matcher usable with the @flow decorator that checks whether the value has the specified prefix.

removeSuffix(s, suffix)

Returns s with suffix removed from the end. Note that s must end with suffix, must be at least one character longer, and the character preceding the suffix must not be / (the function contains an assertion, but you should not rely on that check — use hasSuffix first if you need to).

makeDirs(item, item, ...)

Any number of arguments can be specified. If an item is a string, treat it as a file path and create all its parent directories that do not yet exist, not including the final filename itself. If an item is a Node object, it must have a target parameter, and that will be used as the file path.

readDepFile(filename)

Read the makefile snippet in the GNU make format and return a list of all dependencies mentioned there. Only a very limited subset of the makefile language is supported. No commands are allowed. Only one target can have prerequisites. The only supported substitutions are $$ and $(wildcard ...), implemented as in GNU make.

Parse errors are logged and cause an empty list to be returned. If the file does not exist, or if None is passed instead of the file name, an empty list is returned as well.

runOnce(command, input=None)

Runs the command (specified as a list of non-escaped arguments and invoked directly without a shell), optionally feeding the input string to its standard input, waits for its completion, and returns the text the command has written to the standard output. Only very short-lived and lightweight commands, such as --version checks, should be invoked this way. If the command fails to start, the OS exception will escape this function. If the command returns a non-zero exit code, errors.CommandFailed will be raised.

The result of successful invocation of a command is cached, and the same command won't be invoked twice with the same value of input.

readOnce(filename)

Reads the specified text file and returns the list of lines in it. The contents of the file are cached, and the same file won't be read twice.

shellEscape(text)

Returns text with special characters escaped so that it appears as a single word under the POSIX shell syntax.

writeIfModified(filename, text)

Write text to file unless the file exists and contains the very same text. An existing file will be overwritten. Directories containing the file will be created if they don't exist.

Library (class)

The base class for configuration objects representing libraries. All properties are read-only, and the lists and dictionaries they refer to must not be modified, either. This base class provides empty containers as fallback values for all properties.

defines (dictionary)
A dictionary that maps preprocessor macro names to the values they should be given when using the library. A name given value None will simply be defined, without specifying a value.
includePaths (list)
A list of include paths to specify when using the library.
libraryPaths (list)
A list of library paths to specify when using the library.
libraryNames (list)
A list of library names (without the lib prefix or any standard suffixes) to link with when using the library.

StandardLibrary (class)

This class extends Library and represents a library with a well-known name that does not need special configuration to be linked. An example of such a library is pthread.

The constructor takes a single argument:

name (string)
The library name. This value ends up the only item in the library object's libraryNames list.

PkgConfig (class)

This class extends Library and represents a library about which the pkg-config tool provides information.

The constructor takes one mandatory positional argument, name, and a number of optional keyword arguments.

name (string)
The name of the library package to be passed to pkg-config.
link (boolean, default: True)
If false, the library won't be linked, however, the include paths and preprocessor macros will be used.
pkgConfig (string, default: 'pkg-config')
The name of the pkg-config tool to use.

errors (module)

Custom exception classes.

The module is automatically made available as the name errors in the global namespaces in the context of which the flow files and configuration files are interpreted.

AmbiguousMatch (exception)

Raised when instantiating a node or chaining in case of a tie between two or more matching flow variants with the highest priority (next highest, in case of chaining).

Attributes:

name (string)
The name of the flow.
attrs (dictionary)
The dictionary of node parameters specified.

NoMatch (exception)

Raised when instantiating a node or chaining in case when no matching flow variants are found.

Attributes:

name (string)
The name of the flow.
attrs (dictionary)
The dictionary of node parameters specified.

CommandFailed (exception)

Raised when a command fails to start or returns a non-zero exit code.

command (Command)
The Command that failed.
error (object)
A tuple (type, value, traceback) (in the same format as returned by sys.exc_info) with information about the exception that was raised when the command failed to start, or a tuple of three None values if the command didn't fail to start, but rather returned a non-zero exit code.

CircularDependency (exception)

Raised when a node indirectly waits for its own completion.

BuildInterrupted (exception)

Raised when the build is interrupted with a signal.

Attributes:

signal (integer)
The signal with which the build was interrupted.

CommandLineError (exception)

Raised when parsing of the command-line arguments fails.

output (module)

Logging facilities.

The module is automatically made available as the name output in the global namespaces in the context of which the flow files and configuration files are interpreted.

Logger (class)

The base class for loggers. In the base class, all the methods below have dummy implementations that do nothing.

systemMessage(text)
A sytem-wide event described by text has occurred. An example is a message about the time elapsed while building, which Flower issues before exiting.
goalStarting(goal)
The Goal goal (a Node object) is starting.
goalCompleted(goal)
The Goal goal (a Node object) has completed successfully.
goalFailed(goal, errorList)
The Goal goal (a Node object) has failed. errorList is a list of failed nodes, commands, and exceptions that are not related to any particular node or command.
nodeStarting(node)
The flow of node is starting.
nodeCompleted(node)
The flow of node has completed successfully.
nodeFailed(node)
The flow of node has failed.
processStarting(process)
The command process is starting.
processCompleted(process)
The command process has completed successfully.
processFailed(process)
The command process has failed.
realtimeStdout(process, text)
The command process has written text to its standard output. The text can contain several lines or just a part of a line. No newline characters are stripped from it.
realtimeStderr(process, text)
The command process has written text to its standard error. The text can contain several lines or just a part of a line. No newline characters are stripped from it.

ConsoleLogger (class)

A subclass of Logger providing real-time logging to the standard output and standard error. The constructor accepts the following arguments:

colorize (boolean)
Whether to colorize the output with ANSI control sequences (this is only performed if the outupt stream is a TTY).
echoMessage (boolean)
Whether to print the progress messages of commands that have them.
echoCommands (boolean)
Whether to print the commands that are about to be executed.
echoStdout (boolean)
Whether to display the standard output of the invoked commands.
echoStderr (boolean)
Whether to display the standard error of the invoked commands.

FileLogger (class)

A subclass of Logger and a base class for file-based loggers.

The constructor accepts two arguments:

filename (string)
The name of the log file to write into.
keep (integer, default: 0)
The number of old log files to keep. The constructor rotates the existing files so that build.log becomes build.log.1, build.log.1 becomes build.log.2 and so on. If keep is 0, no old log files are kept, and the log file is truncated every time logging is started.

The class defines two internal methods to be used by subclasses:

_print(text)
Write text into the file.
_println(text)
Write text and a newline character into the file.

TextFileLogger (class)

A subclass of TextFileLogger providing logging into a text file.

The standard output and standard error of each command are logged at the moment the command completes rather than in real time. This prevents intermixing of output when several commands run in parallel.

log (object)

A global object with a public interface similar to that of Logger. Invoking Logger methods on output.log makes all active loggers receive the events.

In addition to the Logger methods, the object provides an operator:

+= (operator)
Accepts a Logger object on the right-hand side. Adds a new logger to the list of active loggers.