ADA-IML'09

Workshop on Analysis and Design of Algorithms
for Interactive Machine Learning at NIPS 2009
Hilton Black Tusk Room, Whistler, BC
Saturday, December 12, 2009

Organizers: Sumit Basu (sumitb at microsoft dot com)
and Ashish Kapoor (akapoor at microsoft dot com)

Introduction and Goals of the Workshop

The traditional role of the human operator in machine learning problems is that of a batch labeler, whose work is done before the learning even begins. However, there is an important class of problems in which the human is interacting directly with the learning algorithm as it learns. Canonical problem scenarios which fall into this space include active learning, interactive clustering, query by selection, learning to rank, and others. Such problems are characterized by three main factors:

the algorithm requires input from the human during training, in the form of labels, feedback, parameter guidance, etc.
the user cannot express an explicit loss function to optimize, either because it is impractical to label a large training set or because they can only express implicit preferences.
the stopping criterion is performance that's "good enough" in the eyes of the user.

The goal of this workshop is to focus on the machine learning techniques that apply to these problems, both in terms of surveying the major paradigms and sharing information about new work in this area. Through a combination of invited talks, discussions, and posters, we hope to gain a better understanding of the available algorithms and best practices for this space, as well as their inherent limitations.

Call for Abstracts

We invite all researchers interested in presenting at the workshop to submit a one-page abstract of their work. The presentation format will be a spotlight summary talk along with a poster session later in the afternoon. We encourage presentations on new, developing ideas, as well as previously published work the authors would like to discuss in this forum. Feel free to email us if you are concerned about whether your work is appropriate for the workshop. Note that there will not be formal proceedings for the workshop, so authors need not be concerned about publishing work they present here at a later venue.

[The deadline for submissions (Oct. 31) has now passed]

Invited Speakers and Attendees

We will have four invited talks (titles coming soon):

Jerry Zhu (University of Wisconsin, Madison)
Carlos Guestrin (CMU)
Rich Caruana (Microsoft Research)
Pushmeet Kohli (MSR Cambridge, Cambridge University)

Schedule

Morning (7:30-10:30)

7:30-8:00 - Introduction; Developing a Syllabus for Interactive Machine Learning*
8:00-8:30 - Invited Talk: Rich Caruana,The Need for User Interaction and Feedback in Clustering
8:30-9:00 - Invited Talk: Carlos Guestrin, Using Personalization to Tame Information Overload
9:00-9:30 - Coffee Break
9:30-10:30 - Poster preview talks

Afternoon (3:30-6:30)

3:30-4:00 - Invited Talk: Pushmeet Kohli, Learning and Evaluating Interactive Segmentation Systems
4:00-5:00 - Poster Session (1.5 hrs, including coffee break)
5:00-5:30 - Coffee Break
5:30-6:00 - Invited Talk: Jerry Zhu, Human Machine Co-Learning
6:00-6:30 - Open Problems, Challenges, Opportunities

*Note that the syllabus-in-progress will be left on the board so that participants may continue to contribute to it during the poster session, final discussion, etc.

Posters

Saleema Amershi, James Fogarty, Ashish Kapoor, and Desney Tan. Designing for End-User Interactive Concept Learning in CueFlik.
Pranjal Awasthi, Maria-Florina Balcan, and Avrim Blum. Clustering with Interactive Feedback.
Vineeth Balasubramanian, Shayok Chakraborty, and Sethuraman Panchanathan. Online Active Learning Using Conformal Predictions.
Scott Blunsden and Cristina Versino. Interactively Reviewing Large Image Sets.
Krzysztof Gajos. Beyond Feature Relevance: Incorporating Rich User Feedback Into Interactive Machine Learning Applications.
Ajay J. Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos. Multi-Class Active Learning with Binary User Feedback.
Goo Jun, Alexander Liu, and Joydeep Ghosh. Interactive Learning on Multiple Binary Classification Problems.
Karrie Karahalios and Tony Bergstrom. Leveraging People and Computers for NLP.
W. Bradley Knox and Peter Stone. Interactively Shaping Agents via Human Feedback: TheTAMER Framework.
Joshua M. Lewis. Software, Psychophysics, and Selection: Towards Anthropocentric Data Analysis.
Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. A Contextual-Bandit Approach to Personalized News Article Recommendation.
Xu Miao, Rajesh P. N. Rao, and Shin'ichi Satoh. Interactive Structural Learning for Image and Video Analysis.
Stephane Ross and J. Andrew Bagnell. Achieving Small Regret Using an Interactive Learning Approach to Imitation Learning.
Kevin Small and Dan Roth. Interactive Feature Space Construction.
Yisong Yue. Online Gradient Descent Using Interactive User Feedback.
Alice X. Zheng, John Dunagan, and Ashish Kapoor. Actively Cutting Graphs: Think Globally, Cut Locally.

About the Organizers:

Sumit Basu is a researcher in the Knowledge Tools group of Microsoft Research, where he investigates interactive machine learning problems in a variety of applications domains. His focus is on techniques which help the user achieve their goals in scenarios where they may not be able to express an explicit cost function, but can show examples or express preferences. He is on the Senior PC of the Intelligent User Interfaces conference, and also works in the area of systems and machine learning: he was co-organizer of the MLSys Workshop at NIPS 2007, as well as co-chair of the SysML Workshop at OSDI 2008.

Ashish Kapoor is a Researcher in the Adaptive Systems and Interaction Group at Microsoft Research where his research interests are centered around interactive machine learning. His recent work focused on systems that often involve humans in the loop and have the ability to adapt and learn over long periods of time. Ashish also co-organized related workshops at AAAI 2009 spring symposium on Human Behavior Modeling and IJCAI 2009 workshop on Intelligence and Interaction. These workshops focused mostly on the interaction facet of IML and the proposed NIPS workshop would complement the research efforts in the area with discussion on formal machine learning methods.