Who Needs a Stylus When You Have Fingers?
By Rob Knies
April 30, 2007 10:00 AM PT

It must be the ultimate clichéd description of the caffeinated Seattleite: coffee in right hand, personal digital assistant in left. The image is so stereotypically simple that it can’t be accurate—unless you look around the city and learn that, yes, as a matter of fact, it is.

That particular practice has spread far and wide, of course, and in each such scenario, there is a stylus, sheathed and inaccessible because human beings have but two hands. Humans are, however, an adaptable lot, and what invariably happens is the stylus stays tucked away, and the human thumb becomes the input apparatus.

Problem solved—almost. As known by anybody who has tried to take such advantage of a device’s touch screen, thumbs and fingers are not the most adept tools for navigating through a handful of small-display pixels. Time to set down that coffee cup, right?

Wait just a minute, suggests Patrick Baudisch, researcher for Microsoft Research Redmond.

Baudisch, working with former intern Dan Vogel, has devised a solution, called Shift, that enables accurate, finger-based input to operate mobile devices effectively. That, it seems, is what people on the go—and those who need to be productive while quaffing their periodic jolt—really need.

The research project is described in a paper entitled Shift: A Technique for Operating Pen-Based Interfaces Using Touch. The paper received a Best Paper award for CHI, the annual Conference on Human Factors in Computing Systems, which began April 28 in San José, Calif., and runs through May 3.

“While we tend to think of the PDA as a stylus-based device,” he says, “many people use it with their bare hands. When you watch people use their PDAs, they’re actually using the thumb of the same hand that’s holding the device to operate it.”

It’s important, Baudisch observes, to cater to the way people actually use technology, not merely to the way product designers expect them to use it.

“In many ways,” he says, “it doesn’t matter what we think we’re designing for. This is what users have decided they want this device to be. They’ve decided they want it to be a touch device.”

The Shift project examines the challenges, the techniques, and the opportunities inherent in that user preference.

The functionality is there; the touch screen is what enables people to use a stylus with such devices. And the need for both hands is not the only reason why people sometimes eschew the stylus. Retrieving it takes time, an effort that can seem impractical when only a brief interaction is desired, such as checking a calendar. It’s much easier to just use a finger.

The stylus, Baudisch notes, does have its advantages.

“The stylus itself is a really interesting piece,” he says. “Initially, it was there to deliver ink. But it has two other interesting properties. One is that it provides a well-defined tip. If you don’t use a stylus, if you use a finger, the fleshy part of the finger is kind of amorphous, kind of soft, and you don’t really know when you touch the screen what part of your finger the computer considers the hot spot. That’s what you lose when you move away from the stylus.

“The second thing is, the stylus gives me a way to remove my hand from the display. If I use my finger, not only is it unclear what part really touched the screen, it also occludes the target at the same time, so I’ve got no visual feedback. The stylus gives me both of these things.

“In many ways,” Baudisch says, “Shift is like a software stylus. It puts back the functionality that we lose when we give up using the stylus.”

Many people “give up using the stylus” for the most basic of reasons. It tends to get misplaced.

That, obviously, is not an issue with Shift. When a user touches a screen, an adjacent circle appears, showing the blocked portion of the touch screen. The circle includes a pointer that the user can manipulate via finger motion to the preferred screen target. Once the pointer is in the proper position, the user lifts the finger to select the target.

Shift explanatory images
When small targets are occluded by a user's thumb (left), Shift displays a circular callout above the thumb (center). The callout can appear elsewhere on the screen (right), if necessary.

It’s a simple solution. It’s easy. It’s intuitive. And, Baudisch says, it adheres to one of his primary research concepts.

“The big theme I’m working on,” he explains, “is what I call escalation. My main assumption is that people will always use the most available device, which is somewhat different from the most powerful device.

“Maybe you’ve got a smartwatch, you’ve got a phone, a PDA, a notebook computer, a desktop computer. Not everybody will have all of these, but many people will have some subset. We will not always use the most powerful device. Today, you might log in to your computer to look up a phone number or get driving directions. In the future, people will use the most available device, even if it does not give them the biggest screen or the most compelling usage scenario.

“The idea of escalation is to enable people to start a task with the most available device and, when it fails [to deliver what is needed], to provide a compelling way of escalating to the next, more complex solution. In a lot of the work I do, that means escalating to the form factor with the next larger display.”

Displays, though, are just one manner of escalation. There’s also the example of moving from casual usage to complex, particularly with small-screen devices.

“Shift is one step in this,” Baudisch says, “insofar that you allow people to start using a device as a touch device, and only when this fails do we require people to use the stylus.”

Baudisch and Vogel started with a hypothesis: that using touch input for a small screen would be more effective for large screen “buttons,” but that effectiveness would diminish as the buttons shrink, because they would tend to be blocked from view by the fingertip. Then they moved into a testing phase, a process the pair found invaluable.

“Repeated testing was really important,” Baudisch says. “The final design is cleaner and simpler than what I initially envisioned.”

That doesn’t mean they didn’t encounter surprises. For one thing, the original design just didn’t work as expected.

“We tried out a bunch of different techniques,” he recalls. “Initially, we thought of creating many callouts, one for each potential target. But it turned out that it’s better if you just copy a piece of the screen as a whole. That has many benefits, because it means we can be agnostic about what’s underlying the fingertip. We can handle any underlying geometry.”

Before settling on the adjacent-circle technique, though, an earlier, seemingly logical approach had to be scrapped.

“One of the things we did in the beginning,” Baudisch says, “was, when you touched the screen, the whole screen would just shift up. According to everything we had learned, that seemed the right way of doing it, because it provided a maximum amount of context: You could see everything [blocked by the finger].”

The best-laid plans …

“People felt like the content was escaping from them,” he laughs ruefully. “They’d try to hit a button, and they would feel the whole screen was moving away from them, and so they’d try to catch up and trap the content—which was not, obviously, what we wanted.”

The project also had to account for the amount of occlusion depending on whether the user was using a fingernail or a fingertip. Research determined that nails were more accurate, fingertips less so, but that the amount of blocking that occurred for each was pretty consistent across users, regardless of finger size. Fingernails generally blocked between three and six millimeters, fingertips between five and 10. This sort of detail could be of great value to a user-interface designer; buttons at least 48 pixels high and somewhat wider should provide problem-free fingertip input.

Then there was the challenge of target selection. When fingers are lifted off the screen, they tend to move a bit, thus repositioning the pointer and decreasing its accuracy. The solution was to apply filters that would tell Shift to ignore the final milliseconds of the input.

The project also provided insights into what Baudisch terms “dwell time.” Shift waits 300 milliseconds before providing its circular view, thus avoiding the intrusion of acting as an unwanted helper. If you can make your desired selection in a reasonable amount of time, you don’t see anything. It’s only when a hesitation suggests possible difficulty that the technology provides assistance. As such, it can be useful even when a person is using a stylus.

“It’s a very natural way of doing it,” Baudisch says, “because if people do need help, time is the most observable evidence of the need for help.”

Taking that concept a bit further, if Shift can learn about the size of available targets, it can act appropriately. With a tiny target or a collection of small targets, it offers quicker assistance. And if a button is sufficiently large, Shift realizes that no help is required and isn’t displayed at all.

This sort of practicality is what makes Shift so appealing. The problem it addresses has been recognized for years. In the 1980s, a technique called “offset cursor” was proposed. Whenever users touch the screen, the pointer appears about an inch above the actual touch location. The offset-cursor technique was considered an optimal solution—but was never implemented in the real world, because it was counterintuitive.

“If you try to hit a light switch,” Baudisch explains, “you don’t want to hit below the light switch. If you tried to do this in a public system, you first would have to educate people about it. A user interface that you need to explain is not a user interface.”

Shift avoids such issues and could well become a harbinger of a focus on touch-based design.

“We will reserve the stylus for those scenarios where we have something complicated to do,” Baudisch predicts. “Our devices will largely become touch devices.”

In the meantime, Shift offers technology that can leveraged right now. It comes with other functionalities, such as a magnifying function.

“If you don’t have the time to get out the stylus,” Baudisch says, “maybe you also don’t have time to get out your glasses.

“I think this has a real practical impact of making PDAs more useful in everyday life. It’s something we can put in a product today. Our customers would enjoy using it.”

He credits much of Shift’s user-friendliness to his collaboration with Vogel.

“He did a fantastic job,” Baudisch says, “and the most impressive thing is what he didn’t do. We had lots of discussion about design, and we agreed on just keeping it simple. I think we succeeded in doing that, and Dan was really instrumental.”

That, in itself, was one of the Shift project’s great rewards.

“When designing these types of user interfaces,” Baudisch concludes, ”the biggest success is if people don’t notice that you are continuously helping them. You just want people to feel that they are skilled, that they have succeeded. I think, with Shift, we’ve accomplished that.”