Expectation-Propagation for the Generative Aspect Model

Thomas Minka and John Lafferty
Proceedings of the 18th Conference on Uncertainty in Artificial Intelligence, pp. 352-359
2002

The generative aspect model is an extension of the multinomial model for text that allows word probabilities to vary stochastically across documents. Previous results with aspect models have been promising, but hindered by the computational difficulty of carrying out inference and learning. This paper demonstrates that the simple variational methods of Blei et al (2001) can lead to inaccurate inferences and biased learning for the generative aspect model. We develop an alternative approach that leads to higher accuracy at comparable cost. An extension of Expectation-Propagation is used for inference and then embedded in an EM algorithm for learning. Experimental results are presented for both synthetic and real data sets.

PDF (100K)
Talk slides (425K)

Also check out some comparisons of VB and EP.

Tomonari Masada has written up some derivations of the equations in this paper.

Software

A Matlab implementation of VB and EP is available: minka-aspect.zip
It requires lightspeed and fastfit, so install these first.


Thomas P Minka
Last modified: Thu Mar 22 14:49:03 GMT 2007