Informative Vector Machines

We have developed a framework for sparse Gaussian process methods which uses forward selection with criteria based on information-theoretical principles, previously suggested for active learning. In contrast to most previous work on sparse GPs, our goal is not only to learn sparse predictors (which can be evaluated in O(d) rather than O(n), d<<n, n the number of training points), but also to perform training under strong restrictions on time and memory requirements. The scaling of our method is at most O(n d^2), and in large real-world classification experiments we show that it can match prediction performance of the popular support vector machine (SVM), yet it requires only a fraction of the training time. In contrast to the SVM, our approximation produces estimates of predictive probabilities (`error bars'), allows for Bayesian model selection and is less complex in implementation.

References

• Neil Lawrence, Matthias Seeger and Ralf Herbrich. Fast Sparse Gaussian Process Methods: The Informative Vector Machine. Advances in Neural Information Processing Systems 15, 625--632, 2003. (Gzipped Postscript).
• Neil Lawrence, John C. Platt and Michael I. Jordan. Extensions of the informative vector machine. In J. Winkler, N. D. Lawrence and M. Niranjan (eds) Proceedings of the Sheffield Machine Learning Workshop, Springer-Verlag, Berlin. 2005. (Gzipped Postscript)
• Matthias Seeger, Chris Williams and Neil Lawrence. Fast Forward Selection to Speed Up Sparse Gaussian Process Regression.
  Workshop on AI and Statistics 9. 2003. (Gzipped Postscript).