Infer.NET

Infer.NET user guide : Tutorials and examples

Sparse Multi-class Bayes Point Machine

Here, we present the details for implementing a sparse multiclass Bayes point machine using Infer.NET. Sparse, in this context, represents the fact every data point has only a subset of relevant features. Hence, a data point is represented by a set of indices (into the feature set) and the corresponding values for these features.

During the training phase, we have labeled data, and the goal is to find the (distribution over) weight vectors for each class. During the testing phase, given unlabeled data, the goal is to find the posterior distribution over the class labels for the data. Hence, we specify two models, a trainModel and a testModel that can be used during training and testing, respectively.

Model for training

During training, all examples are labeled. So, we group training examples by their label. For a particular class, we have the variables nItem, range over these items, number of features in each item (represented using a VariableArray), the range over these features, and of course, the values and the indices of the features. Note the use of jagged arrays for xValues and xIndices; each item is represented by a VariableArray (with range itemFeature) of data values given by xValues[item] and the corresponding feature indices, xIndices[item]. The examples themselves are in a VariableArray (with range item). We have such a construct for every class (not shown for clarity).

Variable<int> nItem = Variable.New<int>().Named("nItem_" + c + s);
Range item = new Range(nItem).Named("item_" + c + s);

VariableArray<int> xValueCount = Variable.Array<int>(item).Named("xCount_" + c + s);
Range itemFeature = new Range(xValueCount[item]).Named("itemFeature_" + c + s);

VariableArray<VariableArray<double>, double[][]> xValues = Variable.Array(Variable.Array<double>(itemFeature), item).Named("xValues_" + c + s);
VariableArray<VariableArray<int>, int[][]> xIndices = Variable.Array(Variable.Array<int>(itemFeature), item).Named("xIndices_" + c + s);

Weights corresponding to each feature have an independent Gaussian prior that is set during the start of training phase. We require this independence assumption since we want to work with only a subset of these features for each training example.

Range feature = new Range(nFeatures);
VariableArray<double>[] w = new VariableArray<double>[nClass];
VariableArray<Gaussian>[] wInit = new VariableArray<Gaussian>[nClass];
for (int c = 0; c < nClass; c++)
{
wInit[c] = Variable.Array<Gaussian>(feature);
w[c][feature] = Variable.Random<double, Gaussian>(wInit[c][feature]);
}

A data point belonging to a particular class, k, should have maximal score under that class. To ascertain this, we constrain that the examples have maximal score only under the class it is being labeled to belong. In the snippet below, BPMUtils.ComputeClassScores, computes the scores of the example under all classes. BPMUtils.ConstrainArgMax constrains that this score is maximum for the class the example belongs to.

for (int c = 0; c < nClass; c++)
{
using (Variable.ForEach(item[c]))
{
Variable<double>[] score = BPMUtils.ComputeClassScores(w, xValues[item], xIndices[item], itemFeature);
BPMUtils.ConstrainArgMax(c, score);
}
}

Computing class scores for sparse model

Since, only relatively small number of features are pertinent for any given example, score can be efficiently computed. We can obtain the weights, wSparse, for pertinent features of an item using Variable.SubArray. (Variable.SubArray is efficient when there are unique indices to retrieve. For non-unique indices, use Variable.GetItems). We compute the score by summing (using Variable.Sum) the element-wise product between wSparse and observed feature values.

Variable<double>[] score = new Variable<double>[nClass];
for (int c = 0; c < nClass; c++)
{
VariableArray<double> wSparse = Variable.Subarray<double>(w[c], xIndices);
VariableArray<double> product = Variable.Array<double>(itemFeature);
product[itemFeature] = xValues[itemFeature] * wSparse[itemFeature];
score[c] = Variable.Sum(product);
}

Training the model

The above model can be nicely encapsulated as trainModel, which can then be used to train, either in batch mode, or incrementally.

Training the model in batch mode

We set the prior over the weights to be standard Gaussians. Then, use the observed data to set the observed values for the variables. Then, inference is performed to obtain the posterior over w. We also reset the prior to the inferred posterior distribution so that we can perform incremental training, and use at the testing time.

// Infer the weights
object[] wInfer = (trainModel.ie).InferAll(trainModel.w);
Gaussian[][] wInferred = new Gaussian[nClass];

// Reset the priors to support incremental training
for ( int c = 0; c < nClass; c++)
{
wInferred[c] = Distribution.ToArray<Gaussian[]>(wInfer[c]));
trainModel.wInit[c].ObservedValue = wInferred[c];
}

Training the model in incremental mode

In this case, we use the posterior distribution inferred during the previous learning stage as the prior distribution for the w. Perform inference to obtain updated posterior distribution over w.

Using Shared variables

We can easily convert the sparse Bayes Point Machine to make use of shared variables. See Multi-class Bayes Point machine using shared variables for an example.

Last modified at 7/28/2009 6:26 PM by Elise Langham (Elanit)