An augmented reality solution for urban infrastructure that you can rely on

Over the past 10 years, many potential applications of augmented reality (AR) for urban infrastructure have been proposed: visualization of subsurface utilities, asset identification and query, excavation and construction planning, etc.  Unfortunately, the implementation of AR apps that would be sufficiently accurate and robust for use in infrastructure engineering is hard – because accurately augmenting the physical world is challenging. 

For a long time, the main difficulty that AR developers had to face is the real time estimation of the position of the tablet or phone doing the augmentation.  Tracking is fundamental to AR, and inaccurate position estimation leads to inaccurate augmentation.  Although recent augmentation devices such as the Google Tango or the Microsoft HoloLens pretty much solved the tracking problem in indoor environments, accurate outdoor augmentation is still challenging.  Most tracking techniques rely on tracking visual features in a video stream captured by a phone or tablet.  Vehicle movements, wind, changing shadows caused by clouds, and uniform surfaces all affect the number of reliable features to be tracked, which results in shaky augmentations, for which the accuracy can hardly be estimated, let alone be predicted.  Infrastructure workers cannot rely on such low accuracy augmentation for their work. 

Working on the outdoor augmentation problem, our team came up with a solution to the accuracy issue: instead of trying to track the physical world, we "capture" it.  Watch our system in action:


Of course, our solution is not true AR.  I guess we could call it mesh augmentation, or augmentation of mirror worlds.  However in the context of urban infrastructure engineering, it has several advantages over true AR systems:

  • Augmentations are temporally stable – no shaking whatsoever, regardless of user’s movements or environment changes, as the augmentations have been pre-aligned with a static 3D mesh.
  • Augmentation accuracy can not only be estimated, but it can also be certified – it depends solely only on data and mesh quality.  That means users can rely on mesh augmentations, unlike true AR for which accuracy is hard to predict. 
  • Augmentation sessions can be done during the night, or when there is snow cover – as they do not rely on any visual features to be tracked. 
  • Augmentation can be done remotely, e.g. from one’s office, when planning tomorrow’s visit to the site, unlike true live AR that requires one’s presence on site, pointing his tablet to the object to be augmented.

However, our technique also has disadvantages compared to true AR:

  • Our augmentations are static – any live event taking place on site will not be visible on the tablet. Our method is therefore more adapted to the augmentation of objects that do not move, such as: infrastructure…
  • Our method requires the pre-capture of 3D photo-realistic meshes, which could be challenging for operational use, as entire cities would need to be modeled.
  • The realism of the augmentations depends on the resolution of the images used to produce the meshes, which in turn depends on altitude – drones photos will likely produce more detailed meshes than aerial photos.
  • A mesh update program would be required for assets construction and demolition.

Some may question the viability of such a technique, considering the requirement of using an up to date high resolution reality mesh of an entire city.  Such meshes, however, are becoming more common every year – take for instance this mesh of the city of Paris, that was obtained in a few months from aerial photos, for an area over 1000 km2

I am convinced that in the future, most major cities will use such photorealistic meshes for urban management and planning.

The method we proposed is not as flexible as true AR.  It is actually a tradeoff that has the advantage of offering accurate augmentations that users can rely on, that could be developed today using existing technology, and that would require, on site, nothing more than a tablet or a phone…