By Janie Chang, Writer, Microsoft Research
In August 2010, Bing Maps rolled out the first images from its Global Ortho program, a multiyear mapping project launched that spring to provide the Bing Maps platform and consumer website with consistently high-quality aerial-photo images of the continental United States and western Europe.
The largest imagery project ever undertaken, the project will be completed in 2012 and will cover about 4 million square miles (10.5 million square kilometers) at a resolution significantly higher than that of commercial imaging satellites.
Microsoft Research Podcast
Collaborators: Renewable energy storage with Bichlien Nguyen and David Kwabi
Dr. Bichlien Nguyen and Dr. David Kwabi explore their work in flow batteries and how machine learning can help more effectively search the vast organic chemistry space to identify compounds with properties just right for storing waterpower and other renewables.
Neel Joshi
Also in August 2010, a team from Microsoft Research Redmond presented a paper called Image Deblurring using Inertial Measurement Sensors during the 37th International Conference and Exhibition on Computer Graphics and Interactive Techniques. The paper received extremely positive reviews. Researcher Neel Joshi and his co-authors—Sing Bing Kang, principal researcher; Larry Zitnick, senior researcher; and Richard Szeliski, Microsoft distinguished scientist—presented a novel way to correct images by measuring camera motion at the time a photo is taken. The team attached inexpensive gyroscopes and accelerometers to a consumer-quality camera to take inertial-measurement data during camera activity and then removed the blur from photos through algorithms that used the data to reverse the effects of motion from the images.
Image deblurring remains an open problem of perennial interest to researchers. It’s difficult to solve mathematically because the goal is to recover the photo that would have been taken if the camera had not moved. In theory, if the distortion factor of the photo is known, then it can be applied to the photo to recover the true image.
Some of the work in this field assumes spatially invariant scenarios, where the same kind of blur applies equally to all parts of the photo. In practice, that hardly ever happens. As a simple example, if the camera rotates slightly, the center of the image will not appear to be blurred significantly, but blurring at the edges will be distinctly noticeable. That requires a spatially variant solution.
“When you look at the actual numbers and the math behind it, it feels like you’re trying to do the impossible,” Joshi says. “You’re trying to come up with an image that’s not blurred, but you don’t know anything about what caused that blurring.”
Joshi’s idea was to get more information about how the image became distorted by using sensors on the camera to capture motion data and then applying the result to recover the intended photo.
This camera’s images can be deblurred.
“We wanted to get information that was realistically available,” he says, “and accelerometers and gyroscopes are now part of every mobile phone, so we’re using data that we know we can get from real products. We did the experiment by adding gear onto a normal camera mostly because we wanted ready access to the data. Hacking into a phone was not the point of the project.”
The team was aiming its research at consumer photography, so Joshi was surprised after returning home from the conference to find email asking him to contact Wolfgang Schickler, senior program manager at Bing Imagery Technologies (BIT) in Boulder, Colo.
Photography at a Different Scale
Joshi and Schickler discussed possibilities, but it was not until early 2011, when Schickler visited Redmond and was able to meet face-to-face with Joshi, that discussions about the potential for technology transfer took off.
The Global Ortho program employs several airplanes mounted with computing equipment and highly efficient digital aerial cameras. The planes fly at the same time, about five miles apart, shooting 220 megapixels every two seconds at 17,000 feet.
The motion-blurred image (top) is sharpened dramatically (bottom).
“If any of the photos don’t meet quality thresholds, they have to get the planes to fly that sequence again,” Joshi says. “There’s a huge cost each time that happens. If we could help them salvage a little bit through image-deblurring technology, even just 10 percent, it would be a worthwhile cost savings.”
Schickler was interested in the approach Joshi’s team had taken of using sensor data to remove blur because the equipment in each plane also included a tracking platform that collected data from gyroscopes, altimeters, and other sensors. At the time, this data was being used to determine whether there had been excessive motion or whether the plane was flying at the wrong orientation when the camera was taking an exposure.
“This data allowed them to detect when an image would have been blurry,” Joshi explains, “and told them they had to send up the planes again. But until they saw our work, I don’t think they had considered taking that next step to use the data to recover the image.”
Mapping Data for Maps
Joshi was excited when he realized how compatible his research was to the project’s aerial-photography process because BIT was already collecting the data he needed to deblur the aerial images.
“It was one of those things that happened organically,” he recalls. “We had some discussions, they sent some sample images and data, we did some initial exploration, and decided it was a good fit. Then we got started.
The original aerial image (top) shows distinct improvement once the deblurring is applied (bottom).
“I saw pretty quickly that if we could map their data directly into our framework, it would work. The key was to figure out how the coordinate systems BIT used were different from those we had used in our project. Once we understood how to map their coordinates to ours, it turned out to be a pretty smooth process of literally mapping A to B and C to D.”
Joshi wrote additional code to enable use of the original research code. He was able to generate sample images fairly quickly to show to BIT. That team was impressed with the results and soon informed Joshi that it wanted to make image deblurring part of its aerial-imaging pipeline.
BIT assigned developers Chintan Shah and Yuxiang Liu to help Joshi package the code to meet the needs of the production process.
“BIT works with really large images,” Joshi says. “They take chunks of these images, run them through the deblurring code, then stitch them all back together. The volume of data is amazing. Each individual strip of image data represents miles and miles of actual ground in the real world. Even though they chop up a strip into another hundred blocks to process, each one still represents gigabytes of pixel data.”
From the Lab to the Real World
Joshi has been interested in image deblurring for a long time, even when he was an intern with Microsoft Research Redmond in 2007. When he began working full time at that lab in 2008, this image-deblurring project was the first one he tackled, and he’s thrilled that it has led to his first experience in working with a product group.
“Image deblurring is a very active research area,” he says. “The fact that this work is being used in an actual product scenario is pretty cool and a big success, because it is still an open problem.”
He recalls his excitement when Bing Maps finally put some of “his” images online.
“I was constantly pinging my contacts in Boulder during the summer of 2011,” he recalls. “There’s so much processing they have to do before the images go online, not just the deblurring. One day, I got email that said, ‘OK, go to Bing Maps, and enter these coordinates,’ and I finally got to see an area where the images had been processed using my algorithm.”
What has been the most interesting thing about this technology transfer?
“The fact that when we worked on this paper, it was aimed at consumer photography,” Joshi says. “I never expected technology transfer to an aerial-mapping application. In a sense, the hard part was in connecting the two dots, when someone at BIT who knew about the gyroscopic data they were collecting thought there might be a way to apply our research to their situation. From there on, it was a matter of getting everyone on the same page.”
Joshi muses that image deblurring is only one of many Microsoft Research tech transfers that have enhanced Microsoft products behind the scenes.
“There are many research-to-product transfers that take place behind the scenes at Microsoft that are not apparent to consumers,” he says. “I wish people had more visibility into these collaborations, which take place more often than people might expect, because it makes the product story more interesting.”
Graphics and multimedia