The moment mobile-phone manufacturers added cameras to their devices, they stopped being just mobile phones. Not only have lightweight phone cameras made casual photography easy and spontaneous, they also have changed the way we record our lives. Now, with help from Microsoft Research, the Office team is out to change how we document our lives in another way—with the Office Lens (opens in new tab) app for Windows Phone 8 (opens in new tab).
Office Lens, now available in the Windows Phone Store (opens in new tab), is one of the first apps to use the new OneNote Service API (opens in new tab). The app is simple to use: Snap a photo of a document or a whiteboard, and upload it to OneNote, which stores the image in the cloud. If there is text in the uploaded image, OneNote’s cloud-based optical character-recognition (OCR) software turns it into editable, searchable text. Office Lens is like having a scanner in your back pocket. You can take photos of recipes, business cards, or even a whiteboard, and Office Lens will enhance the image and put it into your OneNote Quick Notes for reference or collaboration. OneNote can be downloaded for free (opens in new tab).
OCR accuracy depends on the quality of the image being scanned. Camera-phone users capture images under more diverse conditions than those putting documents through a desktop scanner. Office Lens users might be taking photos from an angle, and photos could be over- or under-exposed, blurred, or suffer from glare from a whiteboard’s reflective surface. This is where the product team’s collaboration with Microsoft Research improved results for Office Lens.
Microsoft Research Podcast
Collaborators: Holoportation™ communication technology with Spencer Fowers and Kwame Darko
Spencer Fowers and Kwame Darko break down how the technology behind Holoportation and the telecommunication device being built around it brings patients and doctors together when being in the same room isn’t an easy option and discuss the potential impact of the work.
“We keep things simple for the user,” says Chris Yu, principal group program manager for Office. “Office Lens automatically performs image correction and cleanup to photos on your phone before the files are uploaded to OneNote for storage and text conversion. Office Lens detects the edges of the document or whiteboard, or you have the option to set the border manually. You take a photo, and Office Lens cleans up the image and saves it to OneNote, where the OCR software in the cloud does the text-recognition work that allows you to have a searchable digital file.”
From Whiteboard to OCR
The technological starting point for Office Lens came from a research project originally called Whiteboard It (opens in new tab). Whiteboards provide effective, economical, free-form collaboration, but their contents are difficult to archive and share with colleagues not in the room. Whiteboard image capture and data conversion pose unique challenges. Just ask Zhengyou Zhang (opens in new tab), principal researcher and research manager of the Multimedia, Interaction, and Communication (opens in new tab) group.
“Our ultimate goal was to reproduce whiteboard content as a faithful, yet enhanced, electronic document,” Zhang says. “The original project, Whiteboard It, was something I worked on with research engineer Li-wei He (opens in new tab).”
Office Lens can transform a whiteboard image taken at an angle and exhibiting glare …
Whiteboard It identified key challenges in working with whiteboards, such as compensating for perspective distortion when users take photos from an angle, edge detection to localize the board’s boundaries, white balancing to deliver a uniformly white background, and strong color saturation for pen strokes. All these issues and more must be addressed to deliver a crisp image that can be integrated with any Office document.
Yu learned about Whiteboard It through senior research program manager PD Singh, the liaison between Microsoft Research and Microsoft Office product groups.
“Chris’ team tested the code and very quickly committed to integrating it with Office Lens,” Zhang recalls. “His group is based in Japan, so we had quite a few online meetings, as well as face-to-face meetings in Redmond, to look into use cases and brainstorm solutions.
“Even though Whiteboard It is a core technology of Office Lens, it’s just one component of the whole product. For example, the Office Lens product team had to design the best user experience possible for the general public. That’s a lot of work: thinking, designing, and testing. They looked at ways to cope with a person being part of a snapshot of a whiteboard and how to remove that person from the equation. Our technology prototype didn’t include such considerations. I’ve been really impressed with them.”
Yu, too, is impressed.
“Zhengyou had already envisioned Whiteboard It many years ago,” Yu says. “He saw the need for an intuitive way to enhance networked meetings. That was back when webcams had just started becoming affordable. Today, device technologies and the cloud have evolved and provide a new context for his work. We are delighted to be delivering his vision.”
Begin with High-Quality Images
Office Lens also incorporates algorithms from Advanced Image Editor (AIE), a project that researcher Lu Yuan (opens in new tab) of Microsoft Research’s Visual Computing Group (opens in new tab) and his colleagues—principal researcher Jian Sun, research development engineer Jiangyu Liu (opens in new tab), and researcher Kaiming He (opens in new tab)—built to showcase different technologies for improving the quality of consumer photos.
… into a correctly oriented image without any glare.
“The main goal of AIE,” Yuan says, “was to provide a platform and user interface that made our core technologies easy to present to product teams. In addition, we used AIE to get feedback from internal users and product groups. The feedback they gave us further improved our techniques and drove new ideas. In fact, it was AIE that introduced our advanced image-processing algorithms to the Office team.”
Within a week of trying out AIE, a product team from Office was back in touch with Yuan to give feedback and to discuss features that would be relevant to their use cases. Yuan and Liu worked closely with product developers to integrate algorithms and code. Then followed an intense second stage of collaboration to test and improve the code, which became part of the Camera Scan (opens in new tab) feature in OneNote for Windows 8.1 (opens in new tab), released in November 2013. Their work improved existing camera image-capturing functionality for documents and whiteboards via automatic rotation, straightening, cropping, sharpening, and shadow removal.
“Then we continued our relationship by working with the Office Lens team,” Yuan says. “We contributed our latest techniques and helped them implement the code.”
Office Lens automatically applies some of the key image-enhancement algorithms from AIE to correct image tone and exposure, improve contrast, and reduce “noise.” This might sound like a repeat of Camera Scan, but consider that all of this image-processing functionality had to run efficiently on a mobile device—one of the biggest challenges for Office Lens developers and researchers.
Perfect Alignment
“Working with a product group is very encouraging,” Yuan says. “It gives us passion and motivation, and the open dialog helps us understand what’s required for a better user experience. We want to develop techniques in computational photography that help users achieve high-quality photos, so our research goals are in perfect alignment with Office Lens goals.”
Zhengyou Zhang
Zhang agrees.
“I am very impressed with the efficacy of Chris’ group,” he says. “Our collaboration with his team was very smooth. They’re open to suggestions and work really hard. I’m thrilled that the technology for Whiteboard It has been productized, and I want to thank the Office Lens team. I truly believe Office Lens will help improve productivity and collaboration.”
From the product side of the effort, Yu also is thrilled.
“The collaboration we’ve had with Microsoft Research for Office Lens has been a natural fit,” he says. “Microsoft Research has such deep expertise in so many aspects of image enhancement. For example, we were able to consult with Neel Joshi (opens in new tab) and Sebastian Nowozin (opens in new tab) about best practices for blur measurement and Piotr Dollar (opens in new tab) on edge detection.”
What have early users been saying about Office Lens?
“Feedback has been incredibly positive,” Yu says. “They rave about how much time it saves and how it’s changed the way they take notes, now that they know they can just take a photo of a document and find it later in the OneNote cloud. We couldn’t have done this without Microsoft Research.”
Graphics and multimedia