Welcome to my first blog series on one of my larger-scale parts of my code library I built up over the last few years of freelance work:

調べ-Build (Shirabe-Build)

Shirabe-Build is my take on a reusable build framework based on CMake and Bash/Batch, aiming to simplify the management of large scale and complex C++ projects for multiple platforms, toolchains, compilers and IDEs.

Let me introduce you to the fundamental concepts.

(調べ, jap. Shirabe means inquiry, exploration, and is the broad term over all my projects and spare-time work, which I consider “playground”-environments to learn and progress).

Demarcation

These articles are not meant to teach you CMake or bash scripting in depth.
I will explain tricky intricacies where necessary and provide links to resource useful for understanding the point of discussion, though!

Why an own build framework?

Managing your C++ project can be tough, especially, if multiple platforms and build-toolchains and/or IDEs are required and the project reaches large scale.

Working with larger clients as a freelancer had me set-up C++ build environments repeatedly, which became a frustrating task. Not rarely there was just few time allocated for the task, which caused tailor made build environments to be hacked together.

At some point it annoyed me so much that I learned modern CMake from ground up and created a build-environment, which permitted simple extension and quick integration of projects, subprojects, external repositories and prebuilt binaries.

This was the day, on which 調べ-Build was born.

I reiterated over it frequently and optimized here and there, added several features, usually based on the requirements of my 3D-engine development of 調べ-Engine (https://github.com/BoneCrasher/ShirabeEngine).

The latest addition has been a set of bash-scripts, which facilitated the automated versioning, pulling and building of all required third party libraries required.

I will elaborate on the entire system step by step, but first, let’s take a look into the underlying concepts of 調べ-Build.

Core components

There are two large subsystems, which, joined together, will enable project build management in a quick and simple way.

CMake Build-Framework

The CMake-based build framework is used to manage the actual application code written and is run against a hierarchically structured codebase with one root-project and zero or more levels of sub-projects.

It consists of several .cmake-files, which process the configuration made in the CMakeLists.txt-files of the various project directories against the code base and setup the build-files for the desired toolchain.

To forestall three most important ones:

  1. …/integration/<project>.environment.cmake
    Workspace setup-file, which will perform basic configuration of the workspace directories and references and defines many global build-framework variables.
    Will also include global helper-files.
  2. project_setup.cmake
    Core entry point for each project to cause the interpretation of the project file and derivation of configuration of the build.
    This script will invoke many other .cmake-files in proper order.
  3. apply.cmake
    CMake-integration script, which will project all build-framework configuration to effective CMake-commands.

Bash Script Collection

The bash-script collection is used to maintain and automatically pull, configure, build and deploy the third party libraries according to the project reuqirements.

Fundamentally, the scripts prescribe:

  1. Where to get the sources of the thirdparty libraries from
  2. In which order to pull them and where to pull them to
  3. Which interdependencies between libraries need to be considered
  4. How to setup the global build environment common to all thirdparty libraries
  5. Where to find the buildscript of each library to be built
  6. How to build a set of libraries
  7. And where and how to deploy the results to

The critical part of interconnection is the <project>.environment.cmake-file’s configuration of the thirdparty deployment directory, since this has to point to the location defined by the deployment-bash script.

Workspace structure

In order to have it all work, there’s a prescribed workspace structure to be respected, when using the system.

<workspace_root_dir>/
        buildsystem/
        <root_project_dir>/
        build_extension/
                thirdparty/
        thirdparty/
                buildscripts/
        CMakeLists.txt

<workspace_root_dir>/
The workspace root directory will be the enclosing entity in the system, which contains all buildsystem files and all projects to be created.

It is mandatory to have the directories buildsystem and thirdparty inside it.

buildsystem/
The buildsystem directory contains all buildsystem files, which are project independent and can be considered the build engine.

thirdparty/
The thirdparty directory contains all bash-scripts required to build the thirdparty libraries, including the directory buildscripts where individual buildscripts for each library required are to be stored.

<root_project_dir>/
The <root_project_dir> is where all your project (source) files go to.

It is required to have a build_extension-directory underneath. It should contain a directory thirdparty, which in turn contains 1..N file w/ names in the scheme link_<dependency>.cmake. These files permit the fine grained definition of how to include an external dependency and under which conditions.

CMakeLists.txt
Last but not least, the CMakeLists.txt-file is the workspace root file used by any kind of CMake integration or invocation, performing initial setup, configuration and inclusion of the root project’s CMake-structure.

Project structure

Each project is required to obey a specific structure as well, since the project_setup.cmake-file relies on it in order to properly setup and derive configuration.

It is defined as follows:

<sub_project_dir>/
        code/
                include/
                source/
        resources/ (optional)
        ui/ (optional)
        documentation/ (optional)
        CMakeLists.txt

The <sub_project_dir>/ is the root directory of a project to be added to the workspace.

It only has two mandatory contents, which are the project’s CMakeLists.txt-file as well as the code/-directory.

The CMakeLists.txt-file, containing a specific structure to be explained in another post is necessary to have the project be detected by the system and included in the build-process.

The code/-directory contains all source and header files required to build.
Each code/-directory requires two mandatory directories, namely include/ and source/.

include/ contains all PUBLIC or INTERFACE header files to be provided with static or shared libraries.
source/ contains all headers and sources which are considered PRIVATE.

The system will automatically glob all files in these directories, except those explicitly excluded in the CMakeLists.txt.

The optional resources/-directory contains all images, videos, asset files or whatever is kind of resource files are required for the application to build and/or run.

The optional ui/-directory contains all ui-related files, like QT5-.ui files or .xaml.

The optional documentation/-directory contains all documentation files, which should be installed with your projected and is considered to be PUBLIC.
For any private documentation, create whichever directory you want as long as it is NOT part of this documentation directory.

Conventions

There are a few more conventions to be considered, when working with the buildsystem, except for the workspace and project structures.

Temporary Directories
All temporary directories created are prefixed with an underscore.

There are currently the subsequent directories automatically created by the system:

  1. _build
    Contains all temporary build files created by the CMake-System or the bash-scripts. Will be located either in the root project directory or the thirdparty directory.
  2. _deploy
    Contains all deployed data like applications, shared or static libraries, object files, resources, headers and documentation. Will be located either in the root project directory or the thirdparty directory.
  3. _gen
    Contains all generated data such as import/export headers, precompiled headers, generated code, … Will be located either in the root project directory or the thirdparty directory.
  4. _documentation
    Contains all generated documentation from tools like doxygen. Will be part of the documentation installation. Will be located either in the root project directory or the thirdparty directory.

Deployment Structure
All libraries and applications will be deployed in the subsequent scheme:

_deploy/
    <platform><address_model>/
        <configuration>/
            bin/
            lib/
            include/
            documentation/

Examples:
    _deploy/windows32/debug/...
    _deploy/linux64/release/...

Conclusion/Outlook

Writing a one to rule most of them-build framework was a boon for my productivity, which I want to share with you.

One of the most important aspects to keep it consistent has been to select a small amount of technology to deal with and to define constraints on the filesystem structures and dependencies.

Based on the descriptions above, I will dive deep into how the whole thing works. I will go into detail on these topics (list not complete, I guess):

  1. Abstracting the toolchains and compilers and their configuration
  2. Handling automatic thirdparty linkage in order
  3. Building applications and different types of libraries
  4. How to automate thirdparty dependency handling
  5. Defining individual projects and creating ordering dependencies between them
  6. More and more….

I hope you like the first of an undefined number of articles on the topic!

Leave your feedback! I’m looking forward to it!

Post Views: 4,198