Offline spatial panoramic video augmentation

The term “Augmented Reality” is interpreted in various ways.  But the concept of “live” augmented reality usually means that the physical world is augmented from your current location, at the moment you are viewing the world – that means here and now.  That is what most of us have heard of – you take your smart phone, aim at something, its image is displayed on screen, and the augmentation is overlaid.  That is very cool.

But regardless of how nice and useful such a technology could be in the engineering world, it is not always something to be wished for.  For instance, live augmented reality requires the user to be on site for augmentation – that can be inefficient if the site to be augmented is far away, or if it is a dangerous place like inside a nuclear reactor.  Since live augmentation has to be done now, it precludes the review of “past” augmentations (which could be useful for documenting a site visit).  Live augmentation accuracy is also limited by the user’s movements – who has to be tracked in real time, which is hard to do accurately.  Finally, live augmentation may not always be compatible with authored communications within environments, which can be used to guide, clarify, instruct, and affirm specifically what people should see, and do.  In live augmentation, the environment cannot be guaranteed (as we never know where the user will be augmenting from).  So we thought we needed something more – an augmentation solution that would provide an answer to those problems.

Instead of augmenting live, we have proposed augmenting the world on pre-recorded media.  To provide realistic augmentations, we chose a media that is as realistic as possible: panoramic images, captured along a path.  We visited a building, used a panoramic video camera to capture video along a path inside and outside that building, and aligned the 3D model of that building with each frame of the video.  Consequently, the augmentation is no longer constrained by specific locations (as in Ref1, Ref2) but could be done anywhere along the path where the camera was moved.  The panoramic video, 3D model and path location were stored in a distributable “pre-recorded augmentation package”.  Since no tracking is required, the resulting augmentation is stable, repeatable, and authorable (actionable clarifying instructions can be authored within it), and uses only a small amount of CPU power.

We implemented our system, and tested it in the Paddy Wagon Irish Pub in Richmond, Kentucky, using a Ladybug panoramic camera installed on a tripod and dolly, and aligned each captured video frame with a detailed 3D model of the building.  The package viewer offers a view split in 2 parts: the top part shows the augmented scene, the bottom part shows the 3D map, the camera path(s) (white line) and current camera position and orientation.  The result is shown in the following video:

Augmentation of pre-recorded media will never replace live augmentation.  Such pre-recorded packages can offer no live augmentation, and any recorded environment is, by definition, out of date.  So both types of augmentation are different, but they also are complementary: augmenting pre-recorded media offers jitter free augmentation (since no camera tracking is required), it is 100% deterministic, and it offers some navigation freedom in spite of the fact that it runs on a pre-recorded environment.  The resulting augmentation package is compatible with authorship, offsite augmentation, and with reviewing past augmentations.  The system could be used in operations (for identifying locations of hidden assets), renovations (locating structure), design (showing a model in its physical world context), and in general as a nice way of accessing information related with a building (the interface being the physical world itself).  And although the system can be used away from the augmented area, it could also be used on site, in a similar fashion to typical live augmented reality apps, but this time offering jitter free augmentation. 

Exploring the world of AR, and talking with users in the field, we realize the best solution for them is not always what we would expect.  Technology and market are taking us somewhere, but that is not necessarily always compatible with the needs of infrastructure professionals: accuracy, authorship, reliability – those have to be taken into account in application design.  Our goal is to make sure the AR tools we develop for them are not just toys, that they become useful and  totally reliable.  So far our results show we are making some progress in that direction…

Want to read more?  Check our paper:

Côté S., Barnard J., Snyder R., Gervais R., 2013.  Offline  spatial  panoramic  video  augmentation  for  visual communication in the aec industry.  Proceedings of the 13th International Conference on Construction Applications of Virtual Reality, London, November 2013.  PDF

Many thanks to Chuck Fields, owner of the Paddy Wagon Irish Pub in Richmond, Kentucky, for giving us access to his building and permission to share our results!