Face annotation technology is important for a photo management system. In this project, we proposed a learning framework to automate the face annotation in family photograph albums. Beyond this, we further investigated following problems: 1) propagate common person names from a group of photographs selected by users, 2) reduce user annotation work load by combining unsupervised learning and interactive learning technologies.

1. Automated annotation of human faces in family albums

The core of this framework is a set of salient features to define face similarity and the learning algorithm to refine automatic face annotation result. Firstly, methodologies of content-based image retrieval and face recognition are seamlessly integrated to achieve automated annotation. Secondly, face annotation is formulated in a Bayesian framework, in which the face similarity measure is defined as maximum a posteriori (MAP) estimation. Thirdly, to deal with the missing features, marginal probability is used so that samples which have missing features are compared with those having the full feature set to ensure a non-biased decision.

Fig. 1  automated face annotation

Beyond single face annotation, we further proposed and investigated a new user scenario for face annotation, in which users are allowed to multi-select a group of photographs and assign names to these photographs. The system will then attempt to propagate names from photograph level to face level, i.e. to infer the correspondence between name and face.

Given the face similarity measure which combines methodologies from face recognition and content-based image retrieval, we formulate name propagation as an optimization problem. We define the objective function as the sum of similarities between each pair of faces of the same individual in different photographs, and propose an iterative optimization algorithm to infer the optimal correspondence. To make the propagation result reliable, a reject scheme is adopted to reject those with low confidence scores. Furthermore, we investigated the combination and alternation of browsing mode for propagation and viewer mode for annotation, so that each mode can benefit from additional inputs from the other mode.

2. Easy album: an interactive photo annotation system

 We develop several innovative interaction techniques for semi-automatic photo annotation. Compared with traditional annotation systems, our approach provides the following new features: “cluster annotation” puts similar faces or photos with similar
scene together, and enables user label them in one operation; “contextual re-ranking” boosts the labeling productivity by guessing the user intention; “ad hoc annotation” allows user label photos while they are browsing or searching, and improves system performance progressively through learning propagation. Our results show that these technologies provide a more user friendly interface for the annotation of person
name, location, and event, and thus substantially improve the annotation performance especially for a large photo album.

Fig. 2  Face partial clustering for group annotation

 

 Fig. 3 The EasyAlbum face labeling User Interface

 

Project Member

Lei Zhang, Rong Xiao, Longbin Chen, Yuxiao Hu

Publication

1. Lei Zhang, Yuxiao Hu, Mingjing Li, Weiying Ma, Hongjiang Zhang, Efficient Propagation for Face Annotation in Family Albums, in Proc. ACM Multimedia, New York, US, 2004
2. Lei Zhang, Longbin Chen, Mingjing Li, Hongjiang Zhang, Bayesian face annotation in family albums (demo), in Proc. International Conference on Computer Vision (ICCV), Nice, France, 2003.
3. Lei Zhang, Longbin Chen, Mingjing Li, Hongjiang Zhang, Automated annotation of human faces in family albums, in Proc. ACM Multimedia, Berkeley, CA USA, 2003.
4. Longbin Chen, Baogang Hu, Lei Zhang, Mingjing Li, Hongjiang Zhang, Face annotation for family photo album management, International Journal of Image and Graphics (IJIG), Special Issue on Multimedia Data Storage and Management, Vol.3(1),2003
5. Rong Xiao, Wujun Li, Yuandong Tian, and Xiaoou Tang, Joint Boosting Feature Selection for Robust Face Recognition, in Proc. Computer Vision and Pattern Recognition (CVPR), New York, US, March 2006
6. Deli Zhao, Zhouchen Lin, Rong Xiao, and Xiaoou Tang, Linear Laplacian Discrimination for Feature Extraction, in Proc. Computer Vision and Pattern Recognition (CVPR), Minneapolis, US, 2007
7. Yuandong Tian, Wei Liu, Rong Xiao, Fang Wen, and Xiaoou Tang, A Face Annotation Framework with Partial Clustering and Interactive Labeling, in Proc. Computer Vision and Pattern Recognition (CVPR), Minneapolis, US, 2007
8. Jingyu Cui, Fang Wen, Rong Xiao, Yuandong Tian, and Xiaoou Tang, EasyAlbum: An Interactive Photo Annotation System Based on Face Clustering and Re-ranking, In Proc. SIGCHI conference on Human factors in computing systems, San Jose, US, April 2007