Share on Facebook Tweet on Twitter Share on LinkedIn Share by email
Bringing Robotics to the Surface

Publication date: December 8, 2011

Robotics technology has the potential to play a major role in search-and-rescue missions, particularly in disaster-ravaged areas that are dangerous for humans. Rescue teams need to be able to control robots with precision when they send them to explore a hazardous area. Traditionally, robots have been operated by using a remote-control device that resembles a videogame joystick. Researchers at the University of Massachusetts, Lowell, have been exploring ways to integrate Microsoft Surface and its natural user interface into search-and-rescue robotics to improve the ease and precision by which the robots can be maneuvered.

Facilitating Rescue Efforts with Robots

“Ideally, users should be able to walk up to the controller and know intuitively how to control the robot—no instructions required. The Surface makes this goal achievable.

— Holly Yanco, professor of Computer Science, University of Massachusetts, Lowell

The goal of this research is not to create a robotic replacement for human search-and-rescue workers, explains Holly Yanco, professor of Computer Science at the University of Massachusetts, Lowell. Rather, it’s to enhance the human rescue team’s capabilities through technology. Robots may be used to enter dangerous areas to investigate a hazardous material spill or to search for survivors in areas that are too dangerous for humans to enter. In the case of a disaster, a robot may help save the life of both the victim and the rescue workers who might otherwise place themselves in harm’s way to search for survivors.

As recently as a decade ago, managing a robot would have involved a lot of joysticks, dials, and switches, creating a very elaborate controller with many moving parts. Today, robotics labs are turning to natural user interface (NUI) technology—to develop a more fluid experience.

The DREAM Controller

The DREAM controller uses Microsoft Surface for the user interfaceThe DREAM Controller uses Microsoft Surface for the user interfaceNUI technology was one of the inspirations for the Lowell lab’s Dynamically Resizing Ergonomic and Multi-Touch (DREAM) Controller, which has been in development since 2008. The team chose two Microsoft technologies as a basis for the DREAM Controller: the Microsoft Robotics Developer Studio to simulate the robotics hardware side and Microsoft Surface for the user interface.

What stood out for us in the [Lowell] project was the amazing combination of a Surface computer together with a robotics in a rescue situation. That seemed to be a very magic combination for us.”

— Stewart Tansley, director of Natural User Interface, Microsoft Research Connections

Microsoft Surface is a coffee table-sized device with a computer inside and a touch-sensitive interface on top. The Surface features a NUI that allows multiple users to interact with the computer simultaneously by using just hand gestures. Users can interact with the Surface by using whole hand gestures and multiple fingers, giving a higher level of precision than traditional robotics controllers—a critical feature for search-and-rescue teams. In addition, the Surface permits more than one robot to be controlled at a time—something that is not possible with a single mouse pointer.

To use the DREAM Controller, users simply place their hands on the Surface interface. The DREAM Controller identifies each of the user’s fingers and thumbs and paints a “joystick” beneath each of their hands. The user then uses their thumb to manipulate the virtual joystick.

Improving Future First-Response Efforts

“What stood out for us in the [Lowell] project was the amazing combination of a Surface computer together with a robotics in a rescue situation. That seemed to be a very magic combination for us,” said Stewart Tansley, director of Natural User Interface, Microsoft Research Connections. “I think the DREAM Controller project is really showing the way for what a better first-response system could look like in the near future.”

With refined controls and robotics technology, the DREAM Controller has the potential to make future disaster response safer for rescue workers and to enable them to broaden their searches in disaster areas. Both possibilities point to its capacity to help save lives.