Turn Your Photos Into Movies
By Suzanne Ross
January 17, 2005 12:00 AM PT

Researchers from the lab in Beijing have developed a system that can take your still photographs and automatically convert them into motion clips.

"A single photograph contains extremely rich content," said Xian-Sheng Hua, the lead researcher on the project. "We wanted to use this content to build entertaining videos from still photos."

In the Photo2Video project, the researchers used three key steps to automatically convert photos into video, taking their cues from traditional filmmaking techniques. The steps are story generation, framing scheme, and video composition.

Before completing these steps, the system selects a set of photos to include in the video, by browsing your archive of photos and clustering the images. It identifies photos that belong together, such as photos from the same event or trip, by analyzing semantic features.

The program uses face detection to identify similar faces, whether the photos were taken indoors or outdoors, in the city or a landscape such as mountains, the sea, or a meadow. It also determines which photos belong together by extracting meta data, such as the timestamp, from the photograph.

In the selection phase, duplicate photos are eliminated, as are pictures of low quality, such as overexposed or blurry photos. Now that the set of photos has been selected, the program can move on to the fun part - generating the story.

During the selection phase, the user has identified "lead actors" that will be the focus of attention in the video clips. If the event was a birthday party, for instance, you might identify the birthday boy or girl as a lead actor, and also chose additional lead actors, such as close family members. The lead actors in a set of photos help generate a story, as the movie will follow them throughout the sequence of frames.

Another important part of telling the story is the music. Music adds emotional resonance. The researchers, who include Hua, Lie Lu, and Hong-Jiang Zhang, observed that professionally edited videos that match the camera motion patterns with the tempo of the background music are more expressive. They designed Photo2Video to match the average zooming and panning motion speed to the tempo of the selected music.

The second step simulates the process of shooting or photographing in filmmaking. In this step, the key frames are designed for each photograph based on the content. The program automatically determines the important areas of the photo, based on user's patterns of attention. The researchers found that the most popular focus of attention is people's faces. For photos of scenery or objects, the program estimates the most attractive areas in the image, such as a waterfall in a landscape. These become the second popular focuses of attention.

They also studied the typical framing strategy for movies. There are three primary types of shots: full scene, partial or medium scene, and the close-up. Filmmakers mix these shots up to evoke emotion or decrease emotion. For a more emotional effect, they focus first on the entire scene, and then move into the close-up. To decrease the emotion in a scene, they do the opposite, close-up first, then gradually pan out to a full scene.

Combining the three primary shots with the ability to zoom in or out, do a light pan or a full pan, the researchers identified 14 typical key frame motion patterns, which are applied to each photograph differently. The best key frame motion pattern for the photo is determined by how many areas of focus are in the image, where the image is taken, and whether it is of people, scenery or objects.

The video composition step mimics the process of film editing. This step connects the individual clips with appropriate transitions based on the content of the photograph. To make the transitions appear as smooth as possible, the program determines the similarity of the photos, and the tempo of the background music. The transitions are varied so that the motion pattern isn�t predictable and boring.

Documentaries will often present a series of still photos as though they were movies by simulating the viewer looking at details of a certain image. The researchers have developed Photo2Video to simulate a documentary style of filmmaking. This type of presentation could become a new medium to tell the story embedded in your still photos and make them come alive for your viewers.