A roadmap to research on EP

Expectation Propagation (EP) is a family of algorithms for approximate inference in Bayesian models. This page summarizes research conducted on EP (including variants such as ADATAP and EC).

Tutorials

Video lectures on Approximate Inference

Expectation Propagation: CUED Tutorial slides

A family of algorithms for approximate Bayesian inference
How to construct EP algorithms, illustrated with a variety of different approximations and factor grouping schemes.

From Belief Propagation to Expectation Propagation K. Murphy, 2001
A quick skim of thesis Chapter 3, with some more derivations.

EP Summary E. Sudderth, 2002
An even shorter summary.

EP: A quick reference
A list of equations useful for constructing Gaussian EP algorithms.

Notes on Minka's Expectation Propagation for Gaussian Process classification M. Seeger, 2002

EP in practice
Extending expectation propagation for graphical models Y. Qi, 2004
An overview of techniques used to implement EP in practice.

Expectation Propagation for Exponential Families M. Seeger, 2007

Theoretical developments

The impact of different divergence measures
Divergence measures and message passing

Raising factors to powers
Power EP

Kikuchi approximation
Structured Region Graphs: Morphing EP into GBP

Convergence control
Damping and skipping: Expectation-Propagation for the Generative Aspect Model
Double-loop: Expectation propagation for approximate inference in dynamic Bayesian networks T. Heskes and O. Zoeter, UAI'2002

EP within EM
Expectation-Propagation for the Generative Aspect Model
Predictive Automatic Relevance Determination by Expectation Propagation

Sparse approximation
Gaussian Processes - Iterative Sparse Approximations L. Csato, 2002

Gaussian quadrature based expectation propagation
O. Zoeter and T. Heskes, AISTATS 2005

The objective function
Expectation Propagation for approximate Bayesian inference
The EP energy function and minimization schemes
TAP Gibbs Free Energy, Belief Propagation and Sparsity L. Csato, M. Opper, and O. Winther, NIPS'2001
Expectation propagation for approximate inference in dynamic Bayesian networks T. Heskes and O. Zoeter, UAI'2002
Approximate Inference Techniques with Expectation Constraints T. Heskes, M. Opper, W. Wiegerinck, O. Winther and O. Zoeter, Journal of Statistical Mechanics: Theory and Experiment, 11015 (2005)

ABC-EP: Expectation Propagation for Likelihood-free Bayesian Computation
S. Barthelmé and N. Chopin, ICML 2011
 

Uses of EP

Perceptrons, Gaussian process classifiers
Assessing Approximate Inference for Binary Gaussian Process Classification M. Kuss and C. E. Rasmussen, JMLR 2005
Predictive Automatic Relevance Determination by Expectation Propagation
Fast Sparse Gaussian Process Methods: The Informative Vector Machine N. Lawrence, M. Seeger, and R. Herbrich, NIPS'2002
Gaussian Processes - Iterative Sparse Approximations L. Csato, 2002
A family of algorithms for approximate Bayesian inference
Gaussian Processes for Classification: Mean Field Algorithms M. Opper and O. Winther, Neural Computation 12: 2655-2684, 2000
Bayes Machines for Binary Classification D. Hernández-Lobato and J.M. Hernández-Lobato, Pattern Recognition Letters 29(10): 1466-1473, 2008
Expectation Propagation for Microarray Data Classification D. Hernández-Lobato, J.M. Hernández-Lobato, and A. Suárez, Pattern Recognition Letters 31(12): 1618-1626, 2010
Robust Multi-Class Gaussian Process Classification D. Hernández-Lobato, J.M. Hernández-Lobato, and P. Dupont, NIPS 2011
Expectation Propagation for Bayesian Multi-task Feature Selection D. Hernández-Lobato, J.M. Hernández-Lobato, T. Helleppute, and P. Dupont, PKDD 2010

Ordinal regression
Gaussian processes for ordinal regression W. Chu and Z. Ghahramani, JMLR 2005

Bayesian Conditional Random Fields

Discrete Bayes nets and Markov random fields
Expectation Consistent Approximate Inference M. Opper and O. Winther, JMLR, to appear
Tree-structured approximations by expectation propagation

Density estimation with Gaussian processes
Gaussian Processes - Iterative Sparse Approximations L. Csato, 2002

Neural networks, Multilayer perceptrons
Computing with Finite and Infinite Networks O. Winther, NIPS'2001 (ADATAP)
Expectation Propagation for Neural Networks with Sparsity-promoting Priors P. Jylänki, A. Nummenmaa and A. Vehtari, JMLR (2013)

Independent Components Analysis (ICA)
TAP Gibbs Free Energy, Belief Propagation and Sparsity L. Csato, M. Opper, and O. Winther, NIPS'2001
Mean Field Approaches to Independent Component Analysis P. A.d.F.R. Hřjen-Sřrensen, O. Winther, and L. K. Hansen, Neural Computation 14: 889-918 (2002)

Text modeling, latent Dirichlet allocation
Expectation-Propagation for the Generative Aspect Model

Hybrid dynamic systems (continuous + discrete state)
Expectation propagation for approximate inference in dynamic Bayesian networks T. Heskes and O. Zoeter, UAI'2002
Window-based expectation propagation for adaptive signal detection in flat-fading channels (Fixed-lag smoothing with EP)
Nonlinear dynamic systems
Gaussian quadrature based expectation propagation O. Zoeter and T. Heskes, AISTATS 2005
Expectation propagation for inference in non-linear dynamical models with Poisson observations B. M. Yu, K. V. Shenoy, M. Sahani, NSSPW'2006
Efficient computation of the maximum a posteriori path and parameter estimation in integrate-and-fire and more general state-space models S. Koyama and L. Paninski, J Comp Neuroscience 2009
Iterated extended Kalman smoothing with Expectation-Propagation A. Ypma and T. Heskes, NNSP'2003

Bayesian Inference for Sparse Generalized Linear Models
M. Seeger, S. Gerwinn, M. Bethge, ECML 2007

Bayesian inference for Plackett-Luce ranking models
J. Guiver, E. Snelson, ICML 2009

A Distributed Message Passing Algorithm for Sensor Localization
M. Welling and J. J. Lim, ICANN 2007

Expectation Propagation for Continuous Time Bayesian Networks
U. Nodelman, D. Koller, and C. R. Shelton, UAI 2005

Visualization of time-series data
Hierarchical visualization of time-series data using switching linear dynamical systems O. Zoeter and T. Heskes, PAMI 2003

Expectation Propagation for Infinite Mixtures


Tom Minka
Last modified: Tue Mar 20 18:53:36 GMT 2007