Microsoft Research
Computational User Experiences

Mobile Phone Interaction Techniques

With their small screen and limited input abilities, mobile phones demand novel methods for inputting, viewing and interacting with data. Here we highlight some of our explorations into improving data input and access on mobile phones.


Project Team

Multi-Modal Text Entry

Multi-Modal Text Entry     

Rich text tasks are increasingly common on mobile devices, requiring the user to interleave typing and selection to produce the text and formatting she desires. However, mobile devices are a rich input space where input does not need to be limited to a keyboard and touch. We studied the use of four different input modalities to perform selection tasks in support of text entry on a mobile device: selecting the formatting command by touching it directly with the finger (Touch), tilting the device (Tilt), speaking the command (Voice), and tapping the foot (Foot). Overall, we found that Touch and Voice supported higher text throughput rates and were more preferred than Tilt and Foot.


Phrase Builder

Phrase Builder     

As users enter web queries, real-time query expansion (RTQE) interfaces offer suggestions based on an index garnered from query logs. In selecting a suggestion, users can potentially reduce keystrokes, which can be very beneficial on mobile devices with deficient input means. Unfortunately, RTQE interfaces typically provide little assistance when only parts of an intended query appear among the suggestion choices. We designed and developed an RTQE interface called Phrase Builder that reduces keystrokes by facilitating the selection of individual query words and by leveraging back-off query techniques to offer completions for out-of-index queries.


Phrase Builder Video [.wmv (2.7MB), .mov (3.2MB)]

Search Vox


In light of the challenges of mobile voice search, we developed Search Vox, a multimodal interface that tightly couples speech with touch and text in two directions; users can not only use touch and text to refine their queries whenever speech fails, but they can also use speech whenever text entry becomes burdensome. We facilitate this tight coupling through interaction techniques that leverage wildcard queries; that is, search queries that utilize wildcards (*) to match zero or more characters. Wildcard queries allow users to take advantage of any partial knowledge they may have about the words in the business listing. For example, a user may only remember that the listing starts with a word beginning with “s” and also contains “avenue”. Likewise, the user may only remember “saks something”, where “something” is used to express uncertainty about what words follow.


Search Vox Video for UIST 2008 [.wmv (2.2MB), .mov (32.7MB)]


User Experiences with Activity-Based Navigation on Mobile Devices

A.J. Brush, Amy K. Karlson, James Scott and the Menlo Team

Proceedings of MobileHCI 2010

Multi-Modal Selection During Text Entry on Mobile Devices

David Dearman, Amy Karlson, Brian Meyers and Benjamin Bederson

Proceedings of GI 2010

Designing Phrase Builder: A Mobile Real-Time Query Expansion Interface

Tim Paek, Bongshin Lee, Bo Thiesson

Proceedings of MobileHCI 2009

Investigating the Use of Voice and Ink for Mobile Micronote Capture

Adrienne Andrew, Amy K. Karlson, A.J. Brush

Proceedings of MobileHCI 2009

Can I Borrow Your Phone? Understanding Concerns When Sharing Mobile Phones

Amy Karlson, A.J. Brush, Stuart Schechter

Proceedings of CHI 2009, pp. 1647-1650

Search Vox: Leveraging Multimodal Refinement and Partial Knowledge for Mobile Voice Search

Tim Paek, Bo Thiesson, Y.C. Ju, Bongshin Lee

Proceedings of UIST 2008, pp.141-150

FaThumb: A facet-based interface for mobile search

Amy Karlson, George Robertson, Daniel Robbins, Mary Czerwinski, and Greg Smith

Proceedings of CHI 2006

Contact Us Terms of Use Trademarks Privacy Statement ©2010 Microsoft Corporation. All rights reserved.Microsoft