Share on Facebook Tweet on Twitter Share on LinkedIn Share by email
Worker types and personality traits in crowdsourcing relevance labels

Gabriella Kazai, Jaap Kamps, and Natasa Milic-Frayling

Abstract

Crowdsourcing platforms offer unprecedented opportunities for creating evaluation benchmarks, but suffer from varied output quality from crowd workers who possess different levels of competence and aspiration. This raises new challenges for quality control and requires an in-depth understanding of how workers' characteristics relate to the quality of their work. In this paper, we use behavioral observations (HIT completion time, fraction of useful labels, label accuracy) to define five worker types: Spammer, Sloppy, Incompetent, Competent, Diligent. Using data collected from workers engaged in the crowdsourced evaluation of the INEX 2010 Book Track Prove It task, we relate the worker types to label accuracy and personality trait information along the `Big Five' personality dimensions. We expect that these new insights about the types of crowd workers and the quality of their work will inform how to design HITs to attract the best workers to a task and explain why certain HIT designs are more effective than others.

Details

Publication typeInproceedings
Published inProceedings of the 20th ACM Conference on Information and Knowledge Management, CIKM 2011, Glasgow, United Kingdom, October 24-28, 2011
Pages1941-1944
ISBN978-1-4503-0717-8
PublisherACM
> Publications > Worker types and personality traits in crowdsourcing relevance labels