SDK for Augmented Reality Applications
Project Overview
Our client’s goal was to enhance various printed media (magazines, posters, banners, etc.) with interactive experience using augmented reality. With AR, certain areas on the reading materials can be overlayed with digital information of a different kind: from videos, images, and 3D models to weather information and buttons that bring additional functionality, etc. Imagine, for example, a cooking video popping up when you hover your phone over its recipe in a journal or an instant 3D view on a model’s outfit you liked in the catalog along with the information where to buy it. This AR experience makes printed products more fun, exciting, emotional, and interactive, and ultimately requires robust computer vision algorithms. That is why the client was looking for a partner with solid experience in computer vision to create mobile AR SDK.
AR SDK: Key Components
The developed AR SDK provides a complete list of tools for the creation of augmented reality experience. This makes it very simple and ready to use. The components built are presented in Fig. 1:
Fig. 1. Key components of the developed AR SDK.
They include:
- AR tools (Web and desktop versions)
- AR engine including object detection, tracking and image retrieval modules
- Native part including high-level APIs for iOS and Android
AR tools are used to provide a simple user interface for digital content creation. In particular, the user can upload the target images (magazine pages, banners, restaurant menu, photos, etc.). Built-in image analysis algorithms automatically determine the image quality, estimate the level of its suitability, and enhance its content for better AR experience. Another important role of AR tools is to provide an easy way to manage the digital layers, i.e. change the layout and geometrical properties of AR models (videos, images, 3D models). As a result, users can see how exactly the AR content will look like on mobile devices.
AR engine is a core of the system. It contains a set of custom computer vision algorithms and solutions for analysis and recognition of video stream from the mobile camera. The AR engine comprises three major modules:
- Marker detection and tracking modules provide robust real-time image recognition. The stability of this part is a key to the smooth augmentation of the digital AR layer.
- An additional visual search module allows adding AR experience to large image collections.
AR engine is written in C++, which is additionally optimized for real-time performance directly on mobile devices. This means that all algorithms work on the edge without an internet connection.
The data flow within AR SDK is illustrated in Fig. 2.
Fig. 2. AR SDK data flow.
Firstly, the user selects the image targets which will be used as triggers in the mobile AR application. Secondly, the digital layer is uploaded and easily managed using the Web AR tool. All auxiliary data is automatically generated and stored in the cloud infrastructure. Finally, once the mobile application is installed, all necessary data are downloaded from the server and we are ready to enjoy the AR immersion directly on the device.
Here is an example of AR SDK usage on different markers:
If you are looking for more technical details, check our blog post on marker-based augmented reality or the research paper on an advanced planar tracking approach for augmented reality applications.
Value delivered
Developed AR SDK opened a number of possibilities to apply advanced computer vision algorithms in a seamless manner. Created AR tools can be used without any additional expertise in AR and computer vision making the process of adding digital layers to the product really simple.
Do you have a plan on how augmented reality could bring new functionality to your app or software? Use the form below to get in touch with us and discuss your idea.
