Physical, Virtual Worlds Converge
March 6, 2012 8:45 AM PT

When a corporate research organization has nine locations worldwide, with more than 850 researchers exploring more than 60 areas of computer science, how does it share innovations effectively with product groups?

At Microsoft Research, technology transfer is a priority. It occurs through formal mechanisms such as a dedicated technology-transfer team, product incubations, and intellectual-property licensing, as well as through informal relationships that nurture collaboration between research and product teams.

Then there’s Microsoft Research’s annual TechFest, a three-day event that has been described by the Seattle Times as “a science fair on steroids.” Think booths with demos run by computer-science Ph.D.s and aisles full of representatives of Microsoft product teams hungry for new technologies.

Each year, Microsoft Research invites employees and selected guests to see some of the most exciting projects brewing in the labs and to network with the researchers responsible for these novel technologies. This popular event has been an effective forum for fostering connections between research and product development—and even between researchers.

TechFest 2012 will be held on Microsoft’s Redmond, Wash., campus, beginning with a session on March 6 during which special guests from academia and external organizations will be able to see some of the TechFest projects; on March 7 and 8, Microsoft employees will fill the event’s Microsoft Conference Center venue.

Merging the Physical with the Virtual

This year’s TechFest will include new types of natural user interfaces (NUI) that merge the physical and virtual worlds. Users expect a more seamless blending of the two and some of these technologies will change the user experience—and our lives—profoundly, through interactions we are only beginning to understand.

The assortment of such demos during TechFest is fascinating; they range from augmented reality to next-generation webcams as researchers show prototypes that display intuitive, human-like behavior, sensing their surroundings and users through gesture, voice recognition, environmental awareness, adaptive behavior, and context.

Three of these demos show how advances in hardware and software can contribute to simpler, more natural integration of the physical and virtual worlds.

Gesture Recognition with a Next-Generation Webcam

With Windows 8 in mind, Chunshui Zhao, associate researcher at Microsoft Research Asia, and his teammates addressed the challenge of building a next-generation webcam prototype, as well as software that takes advantage of the new webcam’s capabilities. When asked about the single most challenging aspect of this project, Zhao replied without hesitation, “getting high-resolution depth imaging out of a very low cost device.”

Gesture Recognition with a Next-Generation Webcam
Using hand gestures and a webcam to play a 3-D game.

The new prototype has a significantly wider view angle than traditional webcams and includes a hardware accelerator and a new, image-sensor design. It captures stereo movies and high-accuracy depth images simultaneously and enables users to chat in stereoscopic video. The cost is similar to that of current webcams and has the potential to be miniaturized as a mobile-device camera.

The project did not stop at hardware design. A cross-discipline team—including Zhao’s Microsoft Research Asia colleagues Jiawei Gu, research designer; Yasuyuki Matsushita, lead researcher; and Jing Yan, assistant researcher; as well as Sean Song, firmware engineer at the Microsoft Asia Center for Hardware—also used the new webcam to develop highly accurate depth-image-processing software that supports not only Kinect-style scenarios on a PC, but also new kinds of gesture recognition.

TechFest attendees will be able to try out the webcam prototype with activities such as playing a 3-D game and chatting in stereoscopic video.


Previously, collaborative systems shared digital objects only. At Microsoft Research Redmond, a new kind of remote shared experience has been enabled by Sasa Junuzovic, post-doctorate researcher; Kori Inkpen Quinn, research manager and principal researcher; Tom Blank, hardware-engineering manager; Anoop Gupta, Microsoft distinguished scientist; and engineers Bruce Cleary and Jeff Herron.

IllumiShare enables remote users to share physical or digital objects. The card on the lower left is a projected image from the remote workspace.

“Sharing digital objects is only half the story,” Junuzovic says. “We wanted to enable sharing of both physical and digital objects, and to do so for any surface, to enable natural and seamless interactions.”

IllumiShare consists of a camera-projector combination that looks like a desk lamp. The area it illuminates is the shared workspace and includes objects moving within the workspace, such as hands or pens. IllumiShare captures video of the local workspace and sends it to a remote space, and at the same time, projects video of the remote workspace onto the local space. Users can see, draw on, or point to objects on the workspace and interact much as though they were in the same place.

Behind every seemingly simple, intuitive system is complex technology, and IlumiShare is no exception. One of the major technical challenges for Junuzovic and colleagues was that of “visual-echo cancellation”—avoiding the visual-feedback loop created when the camera captures the projected image. They solved this through time multiplexing, so that camera and projector alternate being on or off. The most difficult problem was achieving this using low-cost, lightweight hardware while avoiding flicker and color imbalance.

TechFest visitors will be able to experience IllumiShare firsthand by sketching in a shared workspace, playing tic-tac-toe and card games, or using it to share a whiteboard.


Holoflector is an augmented reality mirror that creates an interactive, immersive display. Rather than overlay graphics onto a digitized video feed, it takes a new approach to melding the real and digital worlds by rendering graphics onto the real world—or, at least, onto a reflection of the real world.

Sensor fusion of a mobile phone and Kinect enables real-time response and accurate orientation of this digital paddle board and ball.

“The configuration of Holoflector,” says Andy Wilson, research manager and principal researcher at Microsoft Research Redmond, “consists of a large LED display three feet behind a translucent mirror. This has the advantage of allowing graphics rendered on the panel to appear at the correct focal length and to support multiple users simultaneously.”

A Kinect camera tracks bodies positioned in front of the mirror so that images and effects on the display respond in real time to user movements, creating a seamless, convincing rendering of graphics in the “real” world.

Another novel interaction is something Wilson calls “sensor fusion,” which integrates data from a mobile phone with Kinect to incorporate the phone’s position into the graphics. A team—including Wilson, along with summer intern Mahsan Rofouei, Ph.D. candidate at UCLA; A.J. Bernheim Brush, senior researcher at Microsoft Research Redmond; and Stewart Tansley, a director with Microsoft Research Connections—realized that Kinect data provides good absolute position, but not good orientation, while data from the mobile phone’s accelerometer provides fast, accurate orientation information.

The result is a system that infers the position of a phone and renders graphics accordingly, which promises to make Holoflector a popular TechFest attraction. Wilson will invite attendees to pass around virtual objects, including a miniature image of Wilson, and play with a virtual paddle and ball.

Natural Integration of Computers and Daily Life

These three examples are just a few of the ways Microsoft Research personnel are examining scenarios for integrating computers into our daily lives.

“I want to emphasize,” Junuzovic says, “that although visual echo cancellation was a big technical challenge, that is not the important thing about the IllumiShare project. It is about enabling a new kind of shared experience between people in remote locations.”

How humans communicate with machines and vice versa defines a tectonic shift in our relationship with technology. The possibilities and impact are immense, and the diverse approaches, new experiences, and exciting technology previews on display during TechFest are part of Microsoft Research’s ongoing explorations into this realm.