Author: admin

It was not easy at all to master MediaPipe. We thought little in C++ could surprise us. MP did. They say Google libraries do not work outside of Google. We can confirm this is the truth. The ways Google uses the C++ language are highly unusual from our point of view.

How is C++ normally used?

Normally (at least where we come from) people use CMake, a nice cross-platform build system, for C++ projects. Other somewhat common build systems for C++ include Autotools (aka configure+make, mostly Linux/Unix), qmake, and Visual Studio projects (Windows+Visual Studio only). These build systems are similar in the way they handle dependencies. Libraries needed by your projects are typically downloaded and installed system-wide, and not attached to any particular project (as they do in Java or JavaScript worlds). In Linux, macOS and MSYS2 you typically use the system package manager (e.g. ‘sudo apt install libopencv-dev’). For Windows+Visual Studio, you can use vcpkg. If a library is not in the package manager repo, you can download it by hand (as a binary), or, in the worst case, build from the source. By the way, in the latter case, we always install it in a user’s home directory in Linux (e.g. “/home/mickeymouse/opencv-cuda”), we never do “sudo make install”.

What is an installed C/C++ library (by ‘sudo apt install’ or otherwise)? It is a bunch of  headers (.h or .hpp files); and one or more static (.a/.lib) or more often dynamic (.so/.dll) library files. In any case, an “installed library” is compiled once, then used as a binary, which is a good idea, since building a large library like OpenCV, FFMpeg or Boost from the sources takes significant time even on modern PCs. As a C++ developer, you rarely (if ever) have to deal with building standard libraries from the source.

But how do you use installed libraries in your C++ project? First, your project must find the libraries. CMake has a find_package() command for CMake packages, and pkg-config packages can be found by both CMake and Autotools projects on Linux. Things are a bit worse in Windows, but CMake find_package() still mostly works, if used properly.

How does MediaPipe use C++? Part 1.

MP logic is very different. MP does not use CMake. It uses a different build system called Bazel. We’ll tell you in a moment what it is. MP also has tons of dependencies. Namely:

• Source downloaded from github (Non-google): Bazel-skylib, EasyExif, pybind11, Ceres
• Source downloaded from github (Google): Abseil, GoogleTest, Benchmark, GLog, GFlags, Protobuf, libyuv, AudioTools, TensorFlow
• The choice between building from source or using system libraries: OpenCV, ffmpeg

Below we will explain the “downloading and building from source” part. It is practically impossible to build MP in any other way (e.g. with CMake). Maybe a C++ professional could solve this, given time, but the sheer number of dependencies would make it very hard. Definitely not a project for beginners.

What is Bazel?

Bazel is a multi-language build system, which Google uses for many C++ projects, MP included. Probably there are production-related reasons for this, but for us (we are not Google professionals) our experience with Bazel was predominantly negative.

A Bazel project root directory has a file named WORKSPACE, which can be empty. What is a minimal Bazel project? It has an empty WORKSPACE file and a subdirectory fun1. This subdirectory has a file hello.cpp with a project file called BUILD:

load(“@rules_cc//cc:defs.bzl”, “cc_binary”)

cc_binary(

name = “hello”,

srcs = [“hello.cpp”],

)

Note that a project has only one WORKSPACE file, but it can have multiple BUILD files, usually as a hierarchical subdirectory structure. To build the target hello, type (in the project root):

bazel build //fun1:hello

It builds the target and creates 4 directories, which are actually symbolic links to somewhere in ${HOME}/.bazel (tricky !): bazel-bin, bazel-out, bazel-hello and bazel-testlogs. Or, if you want to build and run, type:
bazel run //fun1:hello

How does Bazel treat dependencies? First, there are internal dependencies, other targets of the same project, this is not interesting. Second, there are external dependencies, both Bazel and non-Bazel. Bazel dependencies must be Bazel projects built from the source. Non-Bazel dependencies, in theory, can be the binary libraries, combinations of *.h+.so files. All external dependencies must be listed in the WORKSPACE file.

Here the trouble starts. First, Bazel cannot look for CMake packages. It cannot even find pkg-config packages (we saw a library on GitHub which is supposed to do this, but it did not work for us, at least with OpenCV). We don’t think Bazel can even use standard system paths for libraries in include files (in Linux), you must specify an exact path to each and every library in WORKSPACE and its headers. And even this is nontrivial. Just look at the third_party directory of the MediaPipe repo to see how ugly things can get.

The preferred way in the Bazel world (or at least for Google projects like MP), is to download each and every dependency as a source code (and a Bazel project), and include it as an external Bazel dependency. Bazel has a macro called http_archive() for downloading, but you still must supply an URL. No, there is no “Bazel code repo”, it’s not like Gradle for Java or PIP for Python. Bazel does not manage any “packages”, it can only download stuff from the internet, even CMake can do that (with probably less boilerplate code).

And even such a model does not work properly, as Bazel does not understand “dependency of dependency”. Suppose your project P depends on library A, which in turn depends on B, C, D, E, F, do you add A as the external dependency in P? No, you must add A, B, C, D, E, F, or otherwise P will not build. And don’t forget that building all your dependencies from the source takes time, to say the least, especially if your dependencies are large libraries like OpenCV.

Is there any reason for using Bazel in C++ projects? We did not see any. However, in production, it might be good to download all dependencies from the internet and not rely on the Linux version and APT package versions, for example. 

Another odd thing: suppose executable target A depends on a library target B. Then, if you build target A, Bazel compiles all source files (including the ones belonging to B) to .o, and links the executable A, but never actually links library B (as an .a or .so file). Only if you build target B explicitly, will the library be built.

Finally, how well is Bazel supported by IDEs? Our answer: Not at all. A CLion plugin was announced, but it is incompatible with recent CLion versions. VS Code plugin did not work either, giving very weird error messages, something about Android, while running on Linux desktop. We don’t know enough Bazel or VS Code to fix it. 

To summarize, while Bazel documentation says how great Bazel is, our impression is quite the opposite.

How does MediaPipe use C++? Part 2.

Disclaimer: When we say “impossible” in this chapter, it actually means “impossible, unless you are a highly skillful C++ professional ready to devote a lot of effort to the task”.

Google MediaPipe is a Bazel project. What does it mean? It means it cannot be installed with “sudo apt install libmediapipe-dev”. And it cannot be installed as a pre-built binary library (.h and .so files). Can you build it from the source? Again, the answer is no, at least if you want .h and .so files you can use in your project. So, for all practical purposes (see the disclaimer above), MP can be only used in Bazel C++ projects. Moreover, MP itself has to be built from the source.

How does MP handle dependencies? As we explained above, it downloads >10 dependencies from the internet as source Bazel projects. An exception is made only for OpenCV and FFMpeg, where you can choose between source and system libraries (in the latter case you must specify full paths). Can you use MP as an external Bazel dependency of your project? Basically no, or at least it is very hard (we saw an example in GitHub though). The reason is the “dependencies of dependency” issue, you will need to specify basically all MP dependencies in your project, and not only MP itself.

So the only way (at least for beginners) to use MP is to make your projects not only Bazel projects but parts of the MP project, located inside the mediapipe/ directory, just like MP examples. From our point of view, this is extremely ugly. And not using any IDE does not make coding in C++ any easier.

If this is not enough for you, there are many other ways MP complicates things unnecessarily. For example:

1. You cannot build anything without the –define MEDIAPIPE_DISABLE_GPU=1 flag. The default is a GPU build that fails for rather obscure reasons. 
2. MP examples use GLog logger a lot instead of cout and will not work without GLOG_logtostderr=1
3. The same examples require command line arguments with paths to graph, and will not work if called from a different directory. 
4. MP creates its own wrappers for OpenCV headers and other dependencies, instead of using these libraries as they are.

We promised the final verdict by the end of the series of articles, but actually, we can put it here: MediaPipe would be very nice, if not for Bazel. Bazel (and all related issues) makes you think twice before deciding to use MediaPipe in your C++ project. In particular, if something like GStreamer is suitable for you, it is a much better choice, as it does not require Bazel.

What about using non-C++ wrappers? As we explained before, writing custom calculators requires rebuilding MP from C++ sources. Once again, you will have to deal with Bazel, and also an additional complication of integrating Bazel with Python or Android or whatever.

Google Libraries

MP uses a lot of Google libraries and some non-google ones, which it builds from sources as Bazel projects. What are those libraries? A few Google examples:

• TensorFlow: If you are reading this, you should know what it is 😉
• GLog: A pretty standard logger, and probably the worst logger we have seen. By default, it logs to files in some obscure locations (instead of console), and it’s hard to override. 
• GFlags: Google library for parsing command line arguments, and another reason why MP examples are so hard to read.
• GTest: A well-known unit test library for C++.
• Abseil: A Google’s answer to Boost, and a “thousand useful things for C++” type of library. It can be actually installed with apt and used in CMake projects (but not the latest version). It can be pretty nice, but as far as we know, MP uses only the error codes from Abseil.
• Protobuf: The only library we genuinely liked. We devote a whole section to it.

Google Protocol Buffers (Protobuf)

What is Protobuf? It is a cross-language and cross-platform library from Google for class definition and serialization. Where is it used? TensorFlow and MediaPipe and probably many other things. 

What does it all mean? Let’s do a simple example. Suppose we want to define a data type (or “message” in the Protobuf lingo) Hero in hero.proto:

syntax = "proto3"; // Language version: proto2, proto3
package goblin;  // Becomes C++ namespace
message Hero{
	string name = 1;
	int32 age = 2;
}

“Package” corresponds to a python or Java package, or a C++ namespace. “proto3” is the language version, there are 2 and 3 (they are incompatible). “=1”, “=2” are NOT defaults, but the field unique IDs, they are compulsory. 

Next, we must compile the .proto file to the class definition of your language of choice. For C++, it is:

protoc --cpp_out=. hero.proto

It generates C++ files hero.pb.h and hero.pb.cc containing a C++ class Hero. It’s very important that Hero is not a “simple C++ data class of 2 fields”, but a monster class with lots of obscure methods that requires the Protobuf C++ library. However, it’s not a big problem, as Protobuf can be installed by APT and included in CMake projects easily. Then you can use this class in your own code, with getters and setters and such:

// Create a goblin::Hero object and set fields
goblin::Hero h1;
h1.set_name("Brianna");
h1.set_age(18);
// Can be copied by value (clone aka deep copy, expensive !)
goblin::Hero h2 = h1;
// Print it
cout << "h1: name=" << h1.name() << ", age=" << h1.age() << endl;
// Or like this
cout << h1.DebugString() << endl;

Classes like Hero (but not non-Protobuf classes) can be serialized in both binary and text formats. Such serialization is efficient, cross-language, cross-platform and immune to little/big-endian and 32/64-bit issues.

// Serialize to binary, then deserialize
string buf; // Here std::string is used for BINARY data !
bool ret = h1.SerializeToString(&buf);
goblin::Hero h2;
ret = h2.ParseFromString(buf);

// Serialize to text, then deserialize
string buf;
bool ret = google::protobuf::TextFormat::PrintToString(h1, &buf);
goblin::Hero h2;
ret = google::protobuf::TextFormat::ParseFromString(buf, &h2);
// Text format looks like this:
name: "Brianna"
age: 18

The binary serialization is, well, binary, even if it is contained in an std::string. Why use Protobuf? We think its potential is enormous. TensorFlow uses it to serialize models (.pb files). MediaPipe uses text format to define graphs. And you can use it in your own projects. Every time you see JSON, XML, YAML, TOML and such, Protobuf would probably be better. Binary serialization is efficient, while text serialization is human-readable, and good for e.g. config files.

Let’s now move to our next article and see how MediaPipe works in practice!