Module: util
Copyright © 1995-2012 Opera Software ASA. All rights reserved.
This file is part of the Opera web browser. It may not be distributed
under any circumstances.
About this module
util is a collection of utility- functions, classes and macros.
It includes string functionality, file handling routines, hashtable,
exception handling macros and more. There is no real definition of
what goes into the util module, but the goal is to keep it down to
what the Opera core uses.
The API
See the API documentation
Here follow a description of the largest and most important API's exposed by the util module.
String handling
There are three different ways of dealing with strings, using the util module. They are:
OpString is found in opstring.h,
TempBuffer in tempbuf.h.
See also
the stdlib module.
File handling
There is a virtual base class for all file objects which is called OpFileDescriptor. Inheriting from that are:
All file handling functionality resides in opfile/.
OOM handling
util contains a group of macros for OOM handling. They include (not complete):
- TRAP
- TRAPD
- LEAVE
- LEAVE_IF_ERROR
- LEAVE_IF_NULL
- LEAVE_IF_FATAL
- RETURN_IF_ERROR
- RETURN_IF_MEMORY_ERROR
- RETURN_IF_LEAVE
- ANCHOR
- ANCHOR_ARRAY
These macros reside in excepts.h.
List, trees and hash tables
There are many trees, almost a whole forest:
Host resolver
The class PiOpHostResolver can be used to resolve hostnames in a platform independent way.
OOM handling
Used OOM policies
We have a number of oom policies. Examples are returning OP_STATUS, TRAP/LEAVE, setting a flag in the object, etc. Each module should have a short resume with the oom policies used.
Both OP_STATUS and TRAP/LEAVE policies are used throughout the module. There are classes using both techniques (OpString) and those only using one (OpFile).
Who is handling OOM?
When an OOM occurs, who will be responsible for handling it? The module itself, another specific module?
The util module will never handle OOM situations itself, but return or leave to the caller.
Description of flow
To be able to understand OOM, the module must describe its code flow. Then we will be able to find strengths and weaknesses in the module's OOM strategy.
In almost all cases the util module is used to call a function, get a result, and that's it. That means that there are no deep and complicated code paths inside the module.
Heap memory usage
How much heap memory is used by the module? Describe complexity based on indata. Describe normal case and worst case.
No idea.
Describe how many allocations and deallocations that will happen in a module for certain scenarios, and the size of these allocations.
OpString
The OpString class allocates as much as needed to hold the string it contains. There is also a possibility to allocate an arbitrary large block of memory with the Reserve function. When using methods such as Append, memory for holding both the new and old string will be needed while copying the old string. The old memory will then be freed.
When an OpString object is destroyed all memory is freed.
OpVector
OpVector allocates memory in steps that can be set via the SetAllocationStepSize function. The default stepsize is 10.
The memory allocated by OpVector is freed when the object is freed, but if the elements in the vector are pointers to other objects they will not be freed. If the class OpAutoVector is used, also the objects contained in the vector will be freed.
TempBuffer
A TempBuffer can allocate memory in two different modes. The first allocates exactly as much memory as needed to hold the string the second mode allocate twice as much as the old size or in case the new string is larger than that the size of the new string in each step.
OpFile
WriteUTF8Line allocates storage for a temporary copy of the input string which is encoded in UTF16. The temporary copy is freed when the data has been written.
An OpMemFile stores its entire contents in memory and it is deallocated when the file is closed.
OpZip
When opening a zip-file there is memory allocated for the central directory and for the local header of each file contained in the archive. This memory is freed when the object is freed.
The function GetFile or CreateOpZipFile are used to open a file contained in the archive. Then a buffer to hold the decompressed data is created and it is the callers responsibility to free this memory.
OpHashTable
Memory is allocated in blocks of growing size defined in hashTableSizes the first three are 31, 61 and 127. The hashtable grows when 70% of it is used. OpHashTable also allocates an array to hold the links between the elements in a chained hash table this table also grows in steps. The first three are 21, 42, 88.
When the hashtable is 70% full, a rehash is done. This means that a new hashtable of approximately the double size of the old one is allocated. When the old elements have been added the old table is freed.
Stack memory usage
How much stack memory is used by the module? Describe normal case and worst case.
Functions which use large stack arrays are: SetFromEncodingL and SetToEncoding in OpString uses a stack array of size 1024 bytes. PiOpHostResolverManager::OnSocketDataReady uses a 1024 bytes large stack array. OpFile::Print uses a 2048 bytes large array.
Static memory usage
Describe how much static memory the module is using.
Four objects are allocated in UtilModule::InitL. These are OpFolderManager, PiOpHostResolverManager (if FEATURE_INTERNAL_HOST_RESOLVER is enabled), OpGUIDManager (if API_UTIL_GENERATE_GUID is imported) and ZipCache (if TWEAK_UTIL_ZIP_CACHE is on). They are freed when UtilModule::Destroy is called.
Caching and freeing memory
Describe how your module caches data. Describe how we can free memory in the module by an explicit function call, and when we are allowed to call this function.
A cache for OpZip objects when using OpZipFolder may be used via the tweak TWEAK_UTIL_ZIP_CACHE. The elements are cached until timeout and as long as they are used. The timeout is set be the tweak TWEAK_UTIL_ZIP_CACHE_TIME.
It is also possible to free unused cached objects before timeout via the function ZipCache::FlushUnused. It may be called at any time.
Freeing memory on exit
Describe how the memory is freed when exiting. All memory must be released, it is a coding requirement because on some platforms, the system will not clear the memory for us.
The zip cache is freed when UtilModule::Destroy is called.
Other than that it's up to the users of the module to free memory allocated by objects from the util module.
Temp buffers
Is the module using our shared temp buffers? How is it guaranteed that noone else is using the temp buffer at the same time? Is it possible to rewrite the code, so the risk of using the same temp buffer as another module is eliminated?
Not using temp buffers internally, but exporting an API for temp buffers.
Memory tuning
Is it possible to tune the memory usage of the module? How is that done? Features, ini files or something else?
No, not possible.
Tests
Describe the tests for memory usage that the module provides.
No tests available.
Coverage
Describe how we can get a high coverage in the module, by using a relatively small set of steps. This should include as much of the code that is handling oom errors.
There are no easy ways of trigging code in the util module. To tigger ALL code extremely many test cases must be run. Coverage data from the footprint projects gives the util module a coverage of about 40%, where other modules normally end up with about 70% of the lines of code being executed at least once. The best way is probably to reduce and clarify the API. There is duplicate code and the same thing can be done is several ways. One reason for coverage not being high in the util module is that there is much OOM handling code, which was never triggered during the test runs in the footprint project.
Design choices
Which design choices have been made in the module, balancing memory usage, footprint, performance, and design? Do the module have explicit design choices?
No design choices whatsoever.
Suggestions of improvements
How could we improve this module from a memory perspective? Does the module owner have a plan?
- Rewrite the OpFile class to get a memory safe API (almost done)
- Use the same OOM technique throughout the whole module