Protobuf tutorial

Introduction

The protobuf system is an implementation of Google Protobuf.

The system is used to generate C++ code based on proto files. It creates classes which represent the messages in the proto files. In addition it can either create a service which is used in the scope module, or create OpMessage classes which are used for multiprocess communication.

Script

The protobuf system is contained in modules/protobuf/cppgen and is written in Python (2.4 or higher). The main script is modules/protobuf/cppgen/script.py This system is also integrated with operasetup which means that files will be created or updated each time a build is started. The part in operasetup can be executed with:

> python modules/hardcore/scripts/generate_protobuf.py
For working with just the protobuf system and examining changes execute:
> python modules/protobuf/cppgen/script.py

Example

The first time a module makes use of the protobuf system a file called module.protobuf must be created. This file contains the names of proto files to use, the filenames are relative to the module directory. It is also possible to specify a directory, all files which end with .proto will then be included.

Let's create a file called messages.proto and add it to the module.protobuf file. The contents of the file will be:

message Area
{
    required int32 x = 1;
    required int32 y = 2;
    required int32 width = 3;
    required int32 height = 4;
}

The message names should use CamelCase, the fields uses the same but does not capitalize the first word (ie. camelCase). This is required for the generation of class/method names.

Normally the classes and required files are created by the operasetup process but it is also possible to inspect in more detail what is created. This is useful if when customizing the fields with options (more on that later) or if one just wants to see what gets created. To see what will be created or updated execute the Python script modules/protobuf/cppgen/script.py. Calling it will not do any changes to the system, only show what will be modified.

For instance:

> python modules/protobuf/cppgen/script.py

It is possible to see the actual changes by adding the --diff option. This will print out a contextual diff.

To apply the changes add the -r option (or --replace). This will then write back any changes to the c++ files. This will then create or update files per module that has protobuf files defined, in addition it updates some global files.

Once the files are created they can be used in C++ code by including the the generated file. The file is located in the module under the sub-directory src/generated. The protobuf code is placed in g_proto_<module>_<filename>.h/.cpp.

It is now time to create the files, run:

> python modules/protobuf/cppgen/script.py -r

So far the system has not created the files for OpMessage classes. This must be specified in the proto file by adding the option cpp_opmessage, e.g:

option cpp_opmessage = true;

Now re-run script.py with the -r option and it will create the OpMessages code in g_message_<module>_<filename>.h/.cpp, ie. run:

> python modules/protobuf/cppgen/script.py -r

The code is now ready to be used, all that is needed is to include the file that defines the OpMessage files. For instance if the file messages.proto was placed in the cache module the file to include would be:

modules/cache/src/generated/g_message_cache_messages.h

A class named OpAreaMessage has been created from the message Area. To create a new instance call OpAreaMessage::Create(...). For instance:


OpAreaMessage *area = OpAreaMessage::Create();
area->SetX(10);
area->SetY(10);
area->SetWidth(50);
area->SetHeight(200);

Alternatively set them all in Create():


OpAreaMessage *area = OpAreaMessage::Create(10, 10, 50, 200);

For the most part the fields can be initialized in the Create() call in the order they were defined, however there are some cases where it will not be possible, or they cannot have default parameters. That makes it important to check the parameters of Create(), especially when modifying an existing message.

Customization

It is possible to customize the code that is generated, for instance to change which methods are created for a specific field. This is handled via options in the proto file, it is possible to add options to each item in the file, ie. to the package, any message, field or enum. For the exact syntax for each item see the Google Protobuf documentation. It also possible to place the options in a config file (named cpp.conf), this is mostly useful for the scope module where the proto files are publicly available and it is better to keep such implementation details hidden. But for internal usage (such as the OpMessage code) it is easier and more convenient to place the options directly in the proto file.

For instance to add more helper methods for string fields one could set:

required string someField = 1 [cpp_method = "SetString, SetUniStringPtr, SetStringPtr"];

For more details on which options and how they work see the file modules/protobuf/cpp.conf