Aggregating Ordinal Labels from Crowds by Minimax Conditional Entropy

Dengyong Zhou, Qiang Liu, John C. Platt, and Christopher Meek

Abstract

We propose a method to aggregate noisy ordinal labels collected from a crowd of workers or annotators. Eliciting ordinal labels is important in tasks such as judging web search quality and rating products. Our method is motivated by the observation that workers usually have difficulty distinguishing between two adjacent ordinal classes whereas distinguishing between two classes which are far away from each other is much easier. We formulate our method as minimax conditional entropy subject to constraints which encode this observation. Empirical evaluations on real datasets demonstrate significant improvements over existing methods.

Details

Publication typeInproceedings
Published inProceedings of the 31st International Conference on Machine Learning (ICML)
> Publications > Aggregating Ordinal Labels from Crowds by Minimax Conditional Entropy