Rich text tasks are increasingly common on mobile devices, requiring the user to interleave typing and selection to produce the text and formatting she desires. However, mobile devices are a rich input space where input does not need to be limited to a keyboard and touch. We studied the use of four different input modalities to perform selection tasks in support of text entry on a mobile device: selecting the formatting command by touching it directly with the finger (Touch), tilting the device (Tilt), speaking the command (Voice), and tapping the foot (Foot). Overall, we found that Touch and Voice supported higher text throughput rates and were more preferred than Tilt and Foot.

As users enter web queries, real-time query expansion (RTQE) interfaces offer suggestions based on an index garnered from query logs. In selecting a suggestion, users can potentially reduce keystrokes, which can be very beneficial on mobile devices with deficient input means. Unfortunately, RTQE interfaces typically provide little assistance when only parts of an intended query appear among the suggestion choices. We designed and developed an RTQE interface called Phrase Builder that reduces keystrokes by facilitating the selection of individual query words and by leveraging back-off query techniques to offer completions for out-of-index queries.

Phrase Builder Video [.wmv (2.7MB), .mov (3.2MB)]

In light of the challenges of mobile voice search, we developed Search Vox, a multimodal interface that tightly couples speech with touch and text in two directions; users can not only use touch and text to refine their queries whenever speech fails, but they can also use speech whenever text entry becomes burdensome. We facilitate this tight coupling through interaction techniques that leverage wildcard queries; that is, search queries that utilize wildcards (*) to match zero or more characters. Wildcard queries allow users to take advantage of any partial knowledge they may have about the words in the business listing. For example, a user may only remember that the listing starts with a word beginning with “s” and also contains “avenue.” Likewise, the user may only remember “saks something,” where “something” is used to express uncertainty about what words follow.

Search Vox Video for UIST 2008 [.wmv (2.2MB), .mov (32.7MB)]