Sparse Bayesian Learning and the Relevance Vector Machine

Mike Tipping

Note: Mike Tipping has left Microsoft Research. However, as this is an established URL for the Relevance Vector Machine and since this work was done while Mike was at MSR, this page and its associated material will remain. A possibly more up-to-date page can be found at www.relevancevector.com .

Introduction

Sparse Bayesian learning is the application of Bayesian automatic relevance determination (ARD) to models linear in their parameters. The "relevance vector machine" is a special case of the technique, applied to linear kernel models of the same form as the popular "support vector machine".

Slides from my lectures at the 2003 Tübingen "Machine Learning Summer School" are now available in ".ps.gz" format:

  1. Introduction to Bayesian Inference [180 KB]
  2. Bayesian Inference: Marginalisation [147 KB]
  3. Sparse Bayesian Models and the "Relevance Vector Machine" [1.18 MB]
  4. Sparse Bayesian Models: Analysis, Optimisation and Applications [2.68 MB]

Papers

A comprehensive paper on sparse Bayesian learning from the Journal of Machine Learning Research:

bullet Sparse Bayesian learning and the relevance vector machine. Journal of Machine Learning Research  1, 211–244. [Abstract] [Available from JMLR]

There are a couple of (very minor) typos in the above [corrections here]

Two early conference publications on the Relevance Vector Machine:

bullet The Relevance Vector Machine. In S. A. Solla, T. K. Leen, and K.-R. Müller (Eds.), Advances in Neural Information Processing Systems 12, pp.  652–658. Cambridge, Mass: MIT Press. [Abstract] [gzipped PostScript]
bullet Variational relevance vector machines. In Proceedings of the 16th Conference in Uncertainty in Artificial Intelligence. Accepted to appear. [Abstract] [gzipped PostScript]

Exploiting the sparse Bayes methodology to realise "sparse kernel PCA":

bullet Tipping, M. E. (2001). Sparse kernel principal component analysis. In Advances in Neural Information Processing Systems 13. MIT Press. [Abstract] [gzipped PostScript]

Robust sparse Bayesian regression:

bullet Faul, A. and M. E. Tipping (2001). A variational approach to robust regression. In G. Dorffner, H. Bischof, and K. Hornik (Eds.), Proceedings of ICANN'01, pp.  95–102. Springer. [Abstract] [gzipped PostScript]

Some theoretical analysis of marginal likelihood optimisation and sparsity:

bullet Faul, A. and M. E. Tipping (2002). Analysis of Sparse Bayesian Learning. In Proceedings of NIPS*01 [gzipped PostScript]

An accelerated learning algorithm:

bullet Tipping, M. E. and A. Faul (2003). Fast marginal likelihood maximisation for sparse Bayesian models. In  Proceedings of Artificial Intelligence and Statistics '03  [gzipped PostScript]

 

A Matlab implementation of "SparseBayes V1.0" is available

Some simple Matlab code to implement sparse Bayesian regression and classification models (e.g. like the RVM) is now available: [gzipped tar file]

Note that the software is supplied subject to an "end-user license agreement" (included in the package) which is standard for all freely-distributed Microsoft code. For a preview: click here

Machine Learning and Perception—Machine Learning—Relevance Vector Machine