Models of Human Memory
By Suzanne Ross
August 6, 2004 12:00 AM PT

Ancient storytellers kept their audience spellbound, not only because it was the only form of entertainment around, but also because they gave people an engaging way to remember past events and pass them on to new generations.

In those days, there wasn't a lot to remember. One day was almost like another. Only births, deaths, weddings and the occasional skirmish broke the monotony. Between air travel and the Internet, the world has grown more complex. We have more choices, more information, and more things to remember. Our brains are buffeted with multitudes of events, and we can't recall important information, let alone where we left our keys.

Eric Horvitz, the research manager for Adaptive Systems and Interaction, has been researching models of memory to help us remember.

"Memory is a core aspect of intelligence that gives us an ability to review the past and anticipate the future. I've been pursuing methods and models that show promise for giving computers insights about what people will remember and forget," said Horvitz. "Models of memory can be used in applications that help people remember-as well as to help them to search or browse through large amounts of content."

Predicting Memorable Events
Studies of episodic memory show that people use 'landmarks' to guide their recall. These memory markers include personal and public events, such as the birth of a child or 9/11 for Americans.

Horvitz and his colleagues believed that they could build a system that identified key memory landmarks by learning and predicting which events would come to serve as memorable "handles" into activities of the past.

They focused their initial efforts on events stored in users' online calendars. To build a memory model, they developed a calendar event crawler that analyzes a user's calendar. The crawler creates a library of events and the properties associated with them. It extracts numerous properties for each event, including date, duration, subject, location, and relationships among people involved with the event. The crawler also considers the statistics of properties over time for the events in a user's life.

"We consider all that Outlook knows about events as well as other properties of events that we compute from long-term statistics and information from other sources such as organizational relationships," said Horvitz. "We can harvest a great deal of information about events, and we wanted to see what properties came out as the most predictive for people."

They found that certain patterns of properties indicate that an event is likely a memorable landmark. If a meeting invitation comes into your inbox, but you don't reply, the meeting probably isn't important to you, and therefore you probably won't consider that event a memory landmark, or perhaps not even remember the event several months later. However, if the organizer is Bill Gates, or the location is a beach on Maui, then you'll more likely remember the event as a landmark in time-given that you don't drink one rum cola too many. Events that recurred were unlikely to serve as useful landmarks in time. Such repetitive events tend to blur together.

From this research, they learned more about what makes events memory landmarks. They wanted to integrate these insights into easy to use applications and services that could help people remember and to more effectively find things. Their first task was to pursue additional models of memory that connected images to landmark events.

A Bayesian Walk Down Memory Lane
When the storyteller says, "All the trouble started the day we hunted for buffalo just over the river and near the foot of the mountains," he is building a picture in the minds of his listeners to help them remember. We use digital images in the same way.

Horvitz and his team knew that images help us remember events. They developed an image crawler that analyzes photos in a user's personal photo library. The image crawler collects properties of the images from information stored by digital cameras, and also by automatically extracting features of pictures with image analysis algorithms. This data is used to build a Bayesian model of which pictures would likely serve as memory landmarks.

"The program looks at metadata from the camera that reveals clues to the picture's importance. This data includes properties such as focal distance, aperture settings, and whether you used a flash. We also analyze the images for patterns of color and such higher-level abstractions as whether people are in the image. Beyond the image properties, we consider a user's actions such as their picture taking behavior. For example, if you took a lot of pictures close together, that can indicate that you were excited about the subject, so the pictures are more likely to be memory landmarks," said Horvitz.

Then they used Bayesian network learning methods to identify the connections between the variables and to understand the likelihood of events captured in photos as being memory landmarks. One of the resulting applications from the research is a slide show assembled by Bayesian analysis, called MemoryLens Episodes.

MemoryLens Episodes identifies important photo sessions and automatically selects sets of images from that memory episode for display on a computer monitor. Episodes is designed as an ambient display application that shares digital memories whether deployed on a dedicated display or as a screensaver.

"People take a lot of pictures, but don't have time to look at them. One way to experience photo memories is through ambient displays that allow users to visit the best of their photo libraries with a glance," said Horvitz.

"Episodes allows users to automatically key pictures to the current context. I can set up my ambient display to show past memorable holiday pictures in December, or vacation pictures in July," said Horvitz.

Another application, called MemoryLens Carousel, uses the Bayesian models to identify a best set of slides from an episode. The application helps you show off your vacation pictures to an impatient neighbor. If you want to show her some pictures to illustrate the fun you had in Acapulco, you can select the number of pictures you want to display. The program automatically picks the best ones that span the episode. You won't be embarrassed by fingers in the frame or ten pictures of the sky taken by your six year old.

Both photo applications allow people to fine-tune the choices for memorable pictures. They can move a slider from 'most memorable' to "least memorable" to match their own concept of a memorable moment. They can also tailor the display based on context.

Memory Timelines
Horvitz wanted to use his research in memory landmarks to help people to find what they are looking for within their growing personal stores of information. He and his team developed a rich timeline of different types of landmarks that Horvitz refers to as a "memory backbone" for navigating content.

They combined the predictive models for calendar events and images and added a user's computer activities, such as files created or edited and Web pages visited. They also added public news events. They organized the results into a timeline browser they call LifeBrowser.

The LifeBrowser interface allows you to search your memories just as you can search the Internet. You can access a news event on the timeline such as "Seattle Earthquake," or personal events, such as "Travel to DC" or "Group Off-site," and view emails that you sent or received, documents that you worked on, and Web sites that you visited at these times. A memorability slider allows users to control the detail displayed in the memory backbone. You can display just a few 'most memorable' events or include a larger number of events, some of which will fall into the 'less memorable' category.

"It could be used to help people find things more easily, or jog memories when the details are starting to fade. People can access information at the edge of memory via a "landmark" of something that happened near something else," said Horvitz.

"We're fascinated by cognitive psychology and all that it has revealed about our nature and limitations. Guided by insights from psychology, we're working to mesh learning and reasoning methods with application design to develop new computing experiences. I'm excited about the prototypes and the possibilities."