Learning to Generalize for Complex Selection Tasks

People: Sumit Basu (MSR Redmond) and Alan Ritter (University of Washington)

Summary: Selection tasks are common in modern computer interfaces: we are often required to select a set of files, emails, data entries, and the like.  File and data browsers have sorting and block selection facilities to make these tasks easier, but for complex selections there is little to aid the user without writing complex search queries. We propose an interactive machine learning solution to this problem called ďsmart selection,Ē in which the user selects and deselects items as inputs to a selection classifier which attempts at each step to correctly generalize to the userís target state (see Figure 1).  Furthermore, we take advantage of our data on how users perform selection tasks over many sessions, and use it to train a label regressor that models their generalization behavior: we call this process learning to generalize.  We then combine the userís explicit labels as well the label regressor outputs in the selection classifier to predict the userís desired selections (see Figure 2 below).  We show that the selection classifier alone takes dramatically fewer mouse clicks than the standard file browser, and when used in conjunction with the label regressor, the predictions of the classifier are significantly more accurate with respect to the target selection state.

Demo: Watch this video explanation and demonstration of our method.

Reference:  Ritter, A. and S. Basu:  "Learning to Generalize for Complex Selection Tasks."  Proceedings of the Conference on Intelligent User Interfaces (IUI'09).  February, 2009.

Figure 1: the prototype file browser with smart selection enabled

Figure 2: Learning to generalize.  For a given session, the selection classifier only has labels for that particular task.  However, the label regressor uses data from many sessions of users doing selection tasks in order to form a model of how to best generalize selections.  The selection classifier then combines the labels from the user with the predictions of the label regressor in order to come up with the best prediction of the userís desired selection.