Bayes Point Machines for Classification Learning

From a Bayesian perspective Support Vector Machines choose the hypothesis corresponding to the largest possible hypersphere that can be inscribed in version space, i.e. in the space of all consistent hypotheses given a training set. Those boundaries of version space which are tangent to the hypersphere define the support vectors. An alternative and potentially better approach is to construct the hypothesis using the whole of version space. This is achieved by using a Bayes Point Machine which finds the midpoint of the region of intersection of all hyperplanes bisecting version space into two halves of equal volume (the Bayes point). It is known that the centre of mass of version space approximates the Bayes point. We investigate estimating the centre of mass by averaging over the trajectory of a billiard ball bouncing in version space. Experimental results indicate that Bayes Point Machines consistently outperform Support Vector Machines.

References

• Ralf Herbrich, Thore Graepel, and Colin Campbell. Bayes Point Machines. Journal of Machine Learning Research, 1:245-279, 2001. (Gzipped PostScript) 
• Ralf Herbrich, Thore Graepel, and Colin Campbell. Bayesian Learning in Reproducing Kernel Hilbert Spaces. Technical report, Technical University of Berlin, 1999. TR 99-11. (Gzipped PostScript).
• Ralf Herbrich, Thore Graepel, and Colin Campbell. Bayes Point Machines: Estimating the Bayes Point in Kernel Space. In Proceedings of IJCAI Workshop Support Vector Machines, pages 23-27, 1999. (Gzipped PostScript).
• Thore Graepel and Ralf Herbrich. The Kernel Gibbs Sampler. In Advances in Neural Information System Processing 13, 2001. (Gzipped PostScript).
• Ralf Herbrich and Thore Graepel. Large Scale Bayes Point Machines. In Advances in Neural Information System Processing 13, 2001. (Gzipped PostScript)