Share on Facebook Tweet on Twitter Share on LinkedIn Share by email
Making Sense of the SenseCam
May 10, 2010 11:00 AM PT

How acute is your power of observation? How well do you capture and process the visual information that surrounds your every waking moment? When you look, do you really see?

You might be surprised at the answers to these questions. I was.

During Enabling Innovation Through Research 2010, an event held May 5 at Microsoft Research Cambridge, I got a chance to sample the Microsoft SenseCam, new technology from the host lab that offers exciting potential for assisting persons experiencing memory loss.

That’s inspiring, indeed, and could make a huge difference for patients worldwide, some of them with Alzheimer’s disease. In clinical tests with people with severe memory impairment, the SenseCam has been shown to enhance autobiographical recall.

I don’t feel that I am among those who have trouble remembering things—though, on occasion, my wife might disagree. I’m pretty good at recalling birthdays. I don’t neglect paying the bills. I know where I leave my keys.

But, truth be told, it took barely a half-hour with a SenseCam before I began to wonder about my powers of recall.

Vicon Revue
The Vicon Revue.

In preparation for the event, I got an opportunity to borrow a Vicon Revue, a production model of the SenseCam licensed by Vicon, a subsidiary of the Oxford (U.K.) Metrics Group, a developer of motion-capture products for the life-science, engineering, and entertainment industries.

Vicon’s motion-capture systems already are being used by medical patients, so, naturally, the company was interested in the SenseCam technology. For the moment, though, the emphasis is on further examination of potential uses for the product.

“We’re focusing on the research community,” said Hayley Roberts, a public-relations spokesperson for Vicon, “and developing our relationship with them.”

My bit of personal research was simply structured: I was to wander around the 22 demos on display during the event, wearing the Vicon Revue device—a camera, affixed to a lanyard worn around the neck, that shoots photos unobtrusively, triggered by internal sensors. The camera’s wide-angle, fish-eye lens takes a digital photo every 30 seconds that approximates the view the user would see. Certain environmental changes, such as temperature, color and light intensity, and motion also generate an automatic photo.

Once the photos are downloaded to a PC, Vicon software enables the user to review the images, singly or as a time-lapse video of sorts. In a number of academic projects conducted across the United Kingdom and funded by Microsoft, the process of reviewing the photos has been shown to help memory-loss patients remember things that otherwise likely would slip from their consciousness.

Don’t forget, though: My memory is perfectly fine, thank you. Still, the idea of revisiting a set of demos I had just experienced seemed like it might be interesting, so off I went.

First stop: a pair of digital-living demonstrations in a small room decorated like a household, complete with homey touches like wall hangings. In one corner of the room, Xiang Cao, a post-doctoral researcher in the Computer-Mediated Living group at Microsoft Research Cambridge, was telling a few listeners about TellTable, an interactive, storytelling scenario for the Microsoft Surface that enables children to devise digital characters and use photos of real-world objects, people, and environments to weave a multitouch-enabled tale.

Making Memories

Across the room, John Helmes, an interaction designer in the same group, was discussing A Memory Making System, another Surface-based scenario, this one organized around the concept of a family archive that enables family members to capture, manipulate, create, and store new kinds of digital memorabilia. Users can upload photos or videos and scan in physical memorabilia to create rich, multimedia digital scrapbooks.

“This system functions as a central hub for the family to gather all its digital content,” Helmes said. “But we take it a few steps further, because it communicates with devices within the home, like picture frames, and creates novel artifacts. It also allows you to digitize physical objects, to create digital mementos, and to look at novel ways to encapsulate that content.”

One of those novel ways is a digital piñata that, which when physically struck by the hand, breaks into pieces, spilling its contents—photos, videos, a scan of a baby’s first pair of shoes—onto the display, where they can be resized, moved, activated, or otherwise manipulated. It’s pretty memorable.

But not completely so, apparently. When I reviewed the time-lapse video of the still photos my Vicon Revue had captured, I noticed a few things that had escaped my attention. For one, a woman I watched attempt to crack open the digital piñata took four whacks at it before getting it to break. Somehow, I had forgotten that. And while I remembered the room looking homey, I had neglected to note the brightly designed wallpaper and the presence of a wall clock, a slender bookcase, a couple of lamps, and a birdcage. These things simply hadn’t registered with me. How do you forget a birdcage?

The Path of Go

Oh, well. Next, I headed into the Inference area in another demo room, inference being one of the themes that had been emphasized by Andrew Herbert, managing director of Microsoft Research Cambridge, during his keynote speech for the event. I strode purposefully toward David Stern, an applied researcher in the Online Services and Advertising group, who was talking about a project on which he has worked, The Path of Go, designed to let Xbox Live Arcade gamers tackle the game of Go.

“Go is an ancient Chinese game,” Stern explained. “It’s a game that appears simple and has simple rules, but it’s given enormous challenges to people who have tried to use computers to play the game. I’ve been doing research on how we could create a computer-game player, and this game is an embodiment of that.”

I was happy to chat with Stern, because I had been planning to learn more about The Path of Go during my visit to Cambridge, and after we discussed the game for a while, we made plans to get together the next day. It was a brief, but fruitful, discussion.

Imagine my surprise, then, when I watched the Vicon Revue time-lapse video and noticed that he was sporting a full, closely cropped beard. I’d just spent five minutes with Stern, time in which we had talked about reconvening for further discussion, but I had absolutely no recollection that he had a beard covering a good third of his face. Hmm.

At this point, I began to wonder: Is it just me? Or is it possible that we all experience continuous sensory overload, that our brains are only able to process so much data at any one time? And if so, could it be that SenseCam technology might prove beneficial to all people, not just those experiencing memory loss?

‘Vivid Details’

Luckily, answers were readily at hand, in the form of Emma Berry at the SenseCam demo booth. While working as a neuropsychologist at Addenbrooke’s Hospital in Cambridge, she had an opportunity to observe firsthand the dramatic improvement the SenseCam provided to memory-loss patients. Now, as a vendor working with Microsoft Research Cambridge’s Computer-Mediated Living group, she remains an advocate for the technology. I asked her about the possibility of the SenseCam serving to augment anybody’s memory of an event.

“Absolutely. It’s exactly that,” Berry said. “We forget rapidly. And what happens if we are able to recollect that stuff? Because generally, we’re not. We know that digital photographs aren’t a good way of remembering. There have been some studies around that. What happens if we are able to recollect those vivid details of our lives?”

The technology seems to hold rich potential. Other potential uses include diet monitoring, automatic diaries, and understanding the events that precede instances of intense anxiety or anger.

For now, though, Berry noted, giving patients a tool to help retrieve memories feared lost constitutes sufficient value.

“We found,” she said, “that for our patients with severe memory impairment—clinically diagnosed patients who’ve had an accident or had a viral infection and have very, very poor memory; after three days, they would forget something highly significant that happened to them—this was a fantastically powerful cue for memory.

“When they look through the images, they are able to recollect an event they otherwise would have entirely forgotten. The event comes back to them. They have all the visual imagery you have when you recollect an event, and if they look through the images every day for a few weeks, they’re then able to remember the event without having to review the images.”

That’s the kind of research that’s hard to forget, and thanks to my brief foray with the Vicon Revue, I certainly won’t.