Nowadays there has been tremendous interest in extracting the 3D models of a building façade from a sparse sets of 2d images. The interest is fueled by the intensive need of users to explore the earth online naturally and interactively. To satisfy their need, many applications such as Bing Map have been developed to offer the functionality of viewing the 2D photos about the places of interest. Specifically Bing Map allows the users to explore certain places from both aerial view and street view. Although it cannot be denied that all these services  have provided users great experience of exploring the world, there remains space to improve by integrating the 3D models behind all these 2D images.

The bad news is that, provided we only have access to a sparse set of images, reconstructing the 3D model for a general scene is like task impossible.

Fortunately, if we restrict the scenario to urban scenes, where all kinds of regular structures that provides strong hint for the 3D models behind the 2D textures are abundant, we’ve found a close solution. Leveraged on some of the mathematical breakthroughs in high-dimensional data analysis, we’ve developed an algorithm named TILT which could bridges such gap from 2D images to 3D models. This tool gives us not only the geometry of each individual façade in every image, but also the exact frontal view of them. Utilizing its power, we’ve implemented a pipeline to recover the 3D models of these regular building facades.

Navigation in Urban Scenes

With this additional 3D information, we can realize many interesting applications in practice. One representative example of such applications is 3D navigation in urban scenes. Currently Bing map enable the users to explore the aerial photos of an urban region online. Now, we could build a virtual 3D environment in the urban areas and let the user really dive into the world behind the 2D photos. For instance, users can appreciate their place of interest from their preferable view point instead of the fixed one preset by the camera that takes the photos.

Augmented Reality in Urban Scenes

Aside from enjoying the beauty of the buildings in urban areas, users are given an extra option of exploring more information about these buildings. Let’s take the advertisement display as an example. Without 3D information of buildings, we can only paste the advertisement onto the location of the façade in the image and could not let the advertisement tightly attached to the surface of the façade. Fortunately TILT enables us to overcome this problem and generate the effect as if the advertisement is really on the building surface. Actually this applies to many other scenarios more than advertisement. We could display the name or the logo of the building of a famous company or bank exactly on the façade, and let the user have fun by freely painting anything they like on the façade, to name a few.