This is a reference manual on the user-exposed API of Flower. See Introduction for a more general discussion of the project's architecture.
The flow subsystem resides in the flow module. The system automatically makes the names flow (the decorator), command and goal available in the global namespace in the context of which the flow files are interpreted. Flow files do not need to import this module explicitly unless they need to access other names than the ones mentioned above.
Flower looks for flow files named module.build/flow.py and module.build/*.flow.py in every module listed in a readme.txt file. The order in which flow files are loaded does not matter.
The use of this decorator is mandatory when defining a flow. The argument list is optional. The decorator can be applied to more than one function with the same name, in which case they are considered variants of the same flow. Which one is used in each case will be decided through pattern matching and priorities. The decorator accepts one optional positional argument, priority, and zero or more keyword arguments declaring the patterns to match against the node parameters specified when instantiating the node. In case of a tie when two or more flow variants with the same priority match the specified node parameters, errors.AmbiguousMatch is raised. If no flow variants match, errors.NoMatch is raised.
priority (integer, default: 0)Each keyword argument describes a matcher for the node parameter of the same name. All matchers specified as arguments to @flow must match in order for the flow variant to match the instantiation of a node. The possible matcher types are:
util module provides several useful matchers of this type.re.compile from the standard Python library, it is matched against the value of the node parameter, which must be a string, using the search method of the regular expression object. In fact, anything with a search method will be treated this way.The function being decorated (flow function) must declare a first positional argument, by convention called self. When the function gets called by the flow subsystem, this argument receives the current node object. The flow function may accept any number of additional keyword arguments, which will receive the values of the node parameters with corresponding names. If an argument does not specify a default value, the node parameter by that name is mandatory when instantiating nodes. Node parameters that are listed neither among the patterns as arguments of the @flow decorator nor among the keyword arguments of the function being decorated must not be specified when instantiating a node. If the flow function uses the **kwargs syntax to accept arbitrary keyword arguments, arbitrary node parameters are allowed.
If an uncaught exception of any type escapes the function, the flow is considered to have failed, and so are all flows directly or indirectly waiting for the current one to complete. If a node's flow fails, and the node has a target parameter, the file named by the parameter is deleted as a precaution to prevent incompletely generated output files from being considered as valid by the next build. The same happens when a build is interrupted with a signal.
Any finally blocks in a flow whose corresponding try blocks have been entered are guaranteed to run. However, the order in which such blocks in flows of different nodes will run is undefined.
When the yield statement is used in a flow, the following expressions can be supplied.
Nodeerrors.CircularDependency is raised.CommandNode and/or Command objectsselfyield self statement, the execution chains to a matching flow variant with the next highest priority. The current flow variant is not resumed.User-defined code should not spawn any long-lived child processes by any means other than yielding a Command object. Short-lived processes are allowed, that is, if their lifespan doesn't extend past a yield or a return from a flow, and the user-defined code waits for its completion (util.runOnce is an example of how short-lived child processes can be run safely). Also, user-defined code is not allowed to call os.wait (to wait on all child processes) directly or indirectly; doing so will disrupt the scheduler.
Represents a node.
The constructor is not public. Instances of the class are obtained by invoking flows as functions, specifying zero or more node parameters as keyword arguments. No positional arguments are allowed. The keyword arguments both are used for pattern matching to decide which flow variant to use, and become the initial set of the node parameters. Parameters from the current context are not inherited. Repeated invocation of the same flow with the same set of keyword arguments produces references to the same node. Each flow function receives a reference to the current node as the first and only positional argument.
A node object conforms to the dictionary protocol and supports dictionary-like indexing with string keys, membership testing, and iteration. When used as a dictionary, the node represents the set of node parameters. The node parameters are dynamic and can be added, modified or deleted at any time. However, the set of parameters used to instantiate the node is frozen for purposes of obtaining a reference to the same node by invoking the flow with the same set of keyword arguments.
A node object also supports the context manager protocol and, when used in a with statement, becomes the current context for the duration of the statement's body.
Other members defined on node instances (all properties described below are read-only):
state (integer)Node.PENDINGNode.RUNNINGNode.BLOCKEDNode and/or Command objects complete.Node.COMPLETEDNode.FAILEDerror (tuple)(type, value, traceback) (in the same format as returned by sys.exc_info) with information about the exception that caused the flow to fail, or a tuple of three None values if no exceptions have occurred in the flow. For some internally generated exceptions such as errors.CommandFailed, traceback will be None.< (operator)Node on the left-hand side, the operator accepts a string, a Node object, or a sequence of strings and/or Node objects. (A sequence is any container that has a length and is indexable.) The operator returns true if any of the following conditions are true:
target parameter does not exist.target parameter.target parameter of any of the nodes specified on the right-hand side names a file that does not exist or is newer than the file named by the left-hand operand's target parameter.Node objects appearing on the right-hand side, the optional changed parameter serves as an override. If it's true, the dependency is considered to be newer than the target; if false, it's considered to be older. In either case, the existence and timestamp of the target are not checked. Also note that, as follows from the above, when the right-hand operand is an empty sequence, the operator returns true if and only if the file named by the left-hand operand's target parameter does not exist.Represents a command.
Instances are obtained by calling the global command function. Unlike flows, repeated invocations of command with the same arguments produces distinct command objects.
Arguments of command:
args (list or string)shell is false, or a single command string if shell is true.message (string, default: None)str.format) into which the paremeters of the current node are interpolated as named items. How, when, and whether the progress message actually appears is up to the active loggers. None means no progress message.fail (boolean or callable, default: True)shell (boolean, default: False)cwd (string, default: None)None, the directory which should be made current before executing the command. Flower always operates with the top-level source directory as current, so None as cwd leaves that unchanged. A cwd value affects only the particular process but not any subsequently spawned child processes or Python code.env (dictionary, default: None)None, the environment is inherited from the Flower's main process.Members defined on command instances (all properties are read-only):
state (integer)Command.PENDINGCommand.RUNNINGCommand.COMPLETEDCommand.FAILEDpid (integer)None if not yet started.returncode (integer)None if still running or not yet started.error (tuple)(type, value, traceback) (in the same format as returned by sys.exc_info) with information about the exception that caused the command to fail, or a tuple of three None values if no exceptions have occurred. For some internally generated exceptions such as errors.CommandFailed, traceback will be None.command (string)args, with special characters escaped in arguments so that the string is usable as a shell command. If shell is true, the shell command is returned unchanged. In either case, the string includes a cd command (if cwd is not None) and environment assignments (if env is not None).args (list)args used to create the command.message (string)message after interpolation, or None if there is no message.shell (boolean)shell used to create the command.cwd (string)cwd used to create the command. Can be None.env (dictionary)env used to create the command. Can be None.output (list)text (string)stderr (boolean)Declares a flow as a goal that can be selected from the command line.
The decorator should be used above the @flow decorator. If the flow has more than one variant, the decorator should be used on exactly one of them, does not matter which one.
The decorator takes two mandatory positional arguments:
name (string)desc (string)In addition, any number of extra positional arguments is accepted, each describing a positional non-option goal argument that can be specified on the command line and will translate into a node parameter. Each such argument is a dictionary with the following entries:
arg (string)help (string)default (string or integer)None (the default), the argument is mandatory. A mandatory argument cannot follow an optional argument.type (string, default: 'string')'string' or 'int'.metavar (string, default: the value of arg in upper case)FILE.value (any type)None), the argument merely specifies a hardcoded value for the node parameter to be used when instantiating the goal. Such an argument does not consume anything when parsing the command line, and is not mentioned in the help text.If the goal's flow sets the result node parameter on self to a string, the string will be logged when the goal completes. This should include a one-line explanation of what has been done and where the output file(s) were placed.
The configuration subsystem resides in the config module. The system automatically makes the name config available in the global namespace in the context in which flow files are interpreted, and the names config, default (if available) and option in the global namespace in the context in which configuration files are interpreted. User-defined code does not need to import this module explicitly.
Loading order matters for configuration files: those loaded later override anything before them. Everything listed below, except for platforms/flower/default.py, is optional.
platforms/flower/default.py;module.build/conf.py and module.build/*.conf.py in every module listed in a readme.txt file (sorted by the filename rather than module name, so a name like 00-gcc.conf.py can be used to make sure the file is read before the files overriding its contents in other modules)config.confFiles by any of the configuration files read so far, and by the configuration files read in this step, repeated until no files are listed that haven't been read before (each particular file is only ever read once); by default, this includes:
/etc/opera.conf.py, /usr/local/etc/opera.conf.py (can be used to set options specific to a build server);FLOWER_USER_CONF environment variable, or ~/.opera.conf.py by default (a user can set their preferred options here);*.conf.py at the top level of the source tree (a user can set per-checkout options here; listed in .gitignore);config.confFiles.This global object is the primary entry point into the configuration subsystem.
The object exposes a read-only property for every top-level configuration query defined in any of the configuration files read. Reading the property evaluates the query in the context of the currently running flow. If several configuration files define the property, the most recently read one is selected.
Evaluation of the property depends on how the configuration file defines it.
**kwargs syntax, all node parameters are passed to it.In addition to the read-only properties for top-level configuration queries, the config object provides the following operators:
() (operator)config object returns a temporary object with a similar public interface, which answers configuration queries in the context modified by the specified arguments. Node parameters not modified in this way are left unchanged from the current context. This operation can be applied repeatedly. If one of the parameters specified in this way is named softFallback, and it is set to true, the temporary object will apply soft fallback logic when evaluating queries: if an exception is raised during evaluation, instead of failing, the previous overridden configuration function is used, and so on. The soft fallback logic does not apply to configuration class methods.+= (operator)This global object is available to all configuration files except for the first one read. The interface is similar to that of config, except that the += operator is not available.
The object answers configuration queries using only the configuration files read before the current one. The only purpose of default is to allow a function overriding a configuration query to access the value it overrides (typically to append items to a list). The default object should only be used to access the value being overridden, not to make any other configuration queries (use config for that).
Declares a configuration function as an option that can be set from the command line.
The decorator can only be used with a function, not with a class or a constant assignment.
The decorator takes two mandatory positional arguments, opt and desc, and a number of optional keyword arguments:
opt (string or sequence of strings)desc (string)value (boolean, default: None)@option decorators can be put on the same function with with different value arguments. The default behavior for a boolean option when value is not specified is that the option sets a true value, and the opposite option name is generated automatically (--with- is swapped for --without-, --enable- for --disable-, and --no- is added otherwise). Short options are not inverted. The inversion is not performed when value is specified.default (any type convertible to string, default: None)default is not specified is to use the actual value obtained from reading the configuration files up to the point of parsing the command line.metavar (string, default: None)FILE. The default is N for integer options, STRING for strings and lists, and KEY=VALUE for dictionaries.replace (boolean, default: False)The type of the option is determined by its default value obtained from reading the configuration files up to the point of parsing the command line (soft fallback logic is used to evaluate the configuration queries for the purpose of obtaining defaults). The default should not be None. The allowed types are boolean, integer, string, dictionary mapping strings to strings, and list of strings. In the latter two cases, the option can be repeated on the command line to specify more list items or key-value pairs. If the value of a dictionary option does not contain a = character, the value of the key-value pair is set to None.
The module exposes properties that describe the modules comprising the Opera source tree. The information is obtained by reading the readme.txt files.
The module is automatically made available as the name otree in the global namespaces in the context of which the flow files and configuration files are interpreted.
A read-only dictionary mapping the name of each module set (there are currently four: modules, data, adjunct and platforms) to a module set object with the following read-only properties:
name (string)modules (dictionary)A dictionary mapping the full name of each module to a module object with the following read-only properties:
name (string)fullname (string)file(name)name relative to the module's directory, opening the file for reading and returning a Python file object or None if the file does not exist. Useful for reading optional files like module.sources.glob(pattern)pattern relative to the module's directory and returning a list of files matching the pattern, also relative to the module's directory.An assortment of useful functions and classes.
The module is automatically made available as the name util in the global namespaces in the context of which the flow files and configuration files are interpreted.
The function checks whether items is a sequence other than a string, and if it isn't, wraps items in a single-element tuple, ensuring that the result is always a sequence. A sequence is any container that has a length and is indexable. If items is None, an empty tuple is returned.
The function creates an empty file with the specified name or updates the last modification time of the existing file, imitating the UNIX touch command. The file argument can be the name of the file or a Node whose target parameter names the file. The latter behavior was introduced especially so that you could write touch(self) in your flow code and maintain an innocent look on your face.
Returns a matcher usable with the @flow decorator that checks whether the value has the specified suffix.
Returns a matcher usable with the @flow decorator that checks whether the value has the specified prefix.
Returns s with suffix removed from the end. Note that s must end with suffix, must be at least one character longer, and the character preceding the suffix must not be / (the function contains an assertion, but you should not rely on that check — use hasSuffix first if you need to).
Any number of arguments can be specified. If an item is a string, treat it as a file path and create all its parent directories that do not yet exist, not including the final filename itself. If an item is a Node object, it must have a target parameter, and that will be used as the file path.
Read the makefile snippet in the GNU make format and return a list of all dependencies mentioned there. Only a very limited subset of the makefile language is supported. No commands are allowed. Only one target can have prerequisites. The only supported substitutions are $$ and $(wildcard ...), implemented as in GNU make.
Parse errors are logged and cause an empty list to be returned. If the file does not exist, or if None is passed instead of the file name, an empty list is returned as well.
Runs the command (specified as a list of non-escaped arguments and invoked directly without a shell), optionally feeding the input string to its standard input, waits for its completion, and returns the text the command has written to the standard output. Only very short-lived and lightweight commands, such as --version checks, should be invoked this way. If the command fails to start, the OS exception will escape this function. If the command returns a non-zero exit code, errors.CommandFailed will be raised.
The result of successful invocation of a command is cached, and the same command won't be invoked twice with the same value of input.
Reads the specified text file and returns the list of lines in it. The contents of the file are cached, and the same file won't be read twice.
Returns text with special characters escaped so that it appears as a single word under the POSIX shell syntax.
Write text to file unless the file exists and contains the very same text. An existing file will be overwritten. Directories containing the file will be created if they don't exist.
The base class for configuration objects representing libraries. All properties are read-only, and the lists and dictionaries they refer to must not be modified, either. This base class provides empty containers as fallback values for all properties.
defines (dictionary)None will simply be defined, without specifying a value.includePaths (list)libraryPaths (list)libraryNames (list)lib prefix or any standard suffixes) to link with when using the library.This class extends Library and represents a library with a well-known name that does not need special configuration to be linked. An example of such a library is pthread.
The constructor takes a single argument:
name (string)libraryNames list.This class extends Library and represents a library about which the pkg-config tool provides information.
The constructor takes one mandatory positional argument, name, and a number of optional keyword arguments.
name (string)pkg-config.link (boolean, default: True)pkgConfig (string, default: 'pkg-config')pkg-config tool to use.Custom exception classes.
The module is automatically made available as the name errors in the global namespaces in the context of which the flow files and configuration files are interpreted.
Raised when instantiating a node or chaining in case of a tie between two or more matching flow variants with the highest priority (next highest, in case of chaining).
Attributes:
name (string)attrs (dictionary)Raised when instantiating a node or chaining in case when no matching flow variants are found.
Attributes:
name (string)attrs (dictionary)Raised when a command fails to start or returns a non-zero exit code.
command (Command)Command that failed.error (object)(type, value, traceback) (in the same format as returned by sys.exc_info) with information about the exception that was raised when the command failed to start, or a tuple of three None values if the command didn't fail to start, but rather returned a non-zero exit code.Raised when a node indirectly waits for its own completion.
Raised when the build is interrupted with a signal.
Attributes:
signal (integer)Raised when parsing of the command-line arguments fails.
Logging facilities.
The module is automatically made available as the name output in the global namespaces in the context of which the flow files and configuration files are interpreted.
The base class for loggers. In the base class, all the methods below have dummy implementations that do nothing.
systemMessage(text)text has occurred. An example is a message about the time elapsed while building, which Flower issues before exiting.goalStarting(goal)goal (a Node object) is starting.goalCompleted(goal)goal (a Node object) has completed successfully.goalFailed(goal, errorList)goal (a Node object) has failed. errorList is a list of failed nodes, commands, and exceptions that are not related to any particular node or command.nodeStarting(node)node is starting.nodeCompleted(node)node has completed successfully.nodeFailed(node)node has failed.processStarting(process)process is starting.processCompleted(process)process has completed successfully.processFailed(process)process has failed.realtimeStdout(process, text)process has written text to its standard output. The text can contain several lines or just a part of a line. No newline characters are stripped from it.realtimeStderr(process, text)process has written text to its standard error. The text can contain several lines or just a part of a line. No newline characters are stripped from it.A subclass of Logger providing real-time logging to the standard output and standard error. The constructor accepts the following arguments:
colorize (boolean)echoMessage (boolean)echoCommands (boolean)echoStdout (boolean)echoStderr (boolean)A subclass of Logger and a base class for file-based loggers.
The constructor accepts two arguments:
filename (string)keep (integer, default: 0)build.log becomes build.log.1, build.log.1 becomes build.log.2 and so on. If keep is 0, no old log files are kept, and the log file is truncated every time logging is started.The class defines two internal methods to be used by subclasses:
_print(text)text into the file._println(text)text and a newline character into the file.A subclass of TextFileLogger providing logging into a text file.
The standard output and standard error of each command are logged at the moment the command completes rather than in real time. This prevents intermixing of output when several commands run in parallel.
A global object with a public interface similar to that of Logger. Invoking Logger methods on output.log makes all active loggers receive the events.
In addition to the Logger methods, the object provides an operator:
+= (operator)Logger object on the right-hand side. Adds a new logger to the list of active loggers.