Infer.NET

Infer.NET user guide : Tutorials and examples

Tutorial 6: Mixture of Gaussians

This tutorial shows how to make a mixture of multivariate Gaussians and fit it to some data. It will introduce the switch block which allows variable-sized mixtures to be constructed.

You can run the code in this tutorial either using the Examples Browser or by opening the Tutorials solution in Visual Studio and executing MixtureOfGaussians.cs.

Parameters priors

Being Bayesian, we need to define priors on each of the parameters of our model. To do this, we start by defining a range for the number of mixture components:


Range k = new Range(2);

Using this range, we create an array for the means of our mixture components and give them priors:

VariableArray<Vector> means = Variable.Array<Vector>(k); means[k] = Variable.VectorGaussianFromMeanAndPrecision(new Vector(0.0,0.0), PositiveDefiniteMatrix.IdentityScaledBy(2,0.01)).ForEach(k);

The first line creates an array of Vector variables of size k. The second line assigns a multivariate Gaussian prior to each element of the array. Notice the use of ForEach(k) to duplicate the prior across the array.

We define an array of PositiveDefiniteMatrix precision variables in a similar fashion but using Wishart priors instead of Gaussian:


VariableArray<PositiveDefiniteMatrix> precs = Variable.Array<PositiveDefiniteMatrix>(k);
precs[k] = Variable.WishartFromShapeAndScale(100.0, PositiveDefiniteMatrix.IdentityScaledBy(2,0.01)).ForEach(k);

The final parameter is the vector of mixture weights, which has a Dirichlet prior.


Variable<Vector> weights = Variable.Dirichlet(k,new double[] { 1, 1 });

Notice that we pass the range k as the first parameter. This tells Infer.NET that the size of the vector weights is given by the range k.

Mixing it together

Now that we have defined variables for each of the parameters of the model, we are almost ready to construct the mixture model. But first, we must create a VariableArray for the data:


Range n = new Range(300);
VariableArray<Vector> data = Variable.Array<Vector>(n);

and for each data point, we need a variable to indicate which mixture component the data point came from


VariableArray<int> z = Variable.Array<int>(n);

Now we can construct the mixture model. We use a ForEach block to loop over z and give each element of z a prior given by the weights. Then we use a Switch block to assign each data point to be drawn from the Gaussian with the mean and precision given by the element of z.


using (Variable.ForEach(n)) {
  z[n] = Variable.Discrete(weights);
  using (Variable.Switch(z[n])) {
    data[n] = Variable.VectorGaussianFromMeanAndPrecision(means[z[n]], precs[z[n]]);
  }
}

And there we have it - a fully Bayesian multivariate Gaussian mixture model.

Breaking symmetry

We can now attach some data and run inference. For example data, we use a function which generates data from a known mixture of Gaussians model (see the tutorial file for the code of this function). We then use Variational Message Passing to infer posterior distributions over the weights, means and precisions.


// Attach some generated data
data.ObservedValue = GenerateData(n.SizeAsInt);

// The inference
InferenceEngine ie = new InferenceEngine(new VariationalMessagePassing());
Console.WriteLine("Dist over pi=" + ie.Infer(weights));
Console.WriteLine("Dist over means=\n" + ie.Infer(means));
Console.WriteLine("Dist over precs=\n" + ie.Infer(precs));

If we run this code, the result is:


Dist over pi=Dirichlet(151 151)
Dist over means=
[0] VectorGaussian(4.052 3.739, 0.02212  0.007127)
                                0.007127 0.00796
[1] VectorGaussian(4.052 3.739, 0.02212  0.007127)
                                0.007127 0.00796
Dist over precs=
[0] Wishart(175, 0.00242   -0.002167)
                 -0.002167 0.006726
[1] Wishart(175, 0.00242   -0.002167)
                 -0.002167 0.006726

This is not a good result - the two mixture components are identical! The reason for this is that our model is symmetrical - all the mixture components are defined in exactly the same way and so they are not distinguished during inference.

This problem can be resolved by introducing small random perturbations in the initial messages used in message passing, thus breaking the symmetry between mixture components. Infer.NET does not do this automatically, because it normally guarantees to give the same result each time you perform inference - introducing random initialisation would break this contract. Instead, you are required to break symmetry explicitly. In this example, we can achieve this using InitialiseTo() to randomise the initial message for z, as follows:

Discrete[] zinit = new Discrete[n.SizeAsInt]; for (int i = 0; i < zinit.Length; i++) zinit[i]=Discrete.PointMass(Rand.Int(k.SizeAsInt), k.SizeAsInt); z.InitialiseTo(Distribution<int>.Array(zinit));

If we now run the inference again, we get the following result (note that results may be different on different runs due to the randomness that we have introduced):


Dist over pi=Dirichlet(180 122)
Dist over means=
[0] VectorGaussian(2.033 2.897, 0.003775   -9.646e-05)
                                -9.646e-05 0.004307
[1] VectorGaussian(7.041 4.986, 0.006446   -6.828e-05)
                                -6.828e-05 0.00586
Dist over precs=
[0] Wishart(189.5, 0.007813 0.000175)
                   0.000175 0.006848
[1] Wishart(160.5, 0.007988  9.308e-05)
                   9.308e-05 0.008788

Now we can see that the two mixture components are different and that we have a good solution.

Inference using InferAll()

In the above, we made separate calls to infer posterior over weights, means and precisions. This can be inefficient since Infer does not know the complete set of variables that will be requested, and therefore must generate code conservatively to infer marginals for variables that may never be requested. To avoid this (usually small) overhead, we can make use of the method InferAll to specify a specific set of variables to perform inference for. InferAll takes an array of type IVariable that holds all the variables to infer. It doesn't return anything but caches all the computed marginals and you can retrieve them using Infer as normal. Hence, all you need to do is insert InferAll(<list of variables>) before you call Infer and your code should be more efficient. In this case, we just add the following line just before the first call to Infer.

ie.InferAll(weights, means, precs);

Last modified at 8/4/2009 12:08 PM by John Winn