Crowdsourcing Gold-HIT Creation at Scale: Challenges and Adaptive Exploration Approaches (Position Paper)

  • Ittai Abraham ,
  • Omar Alonso ,
  • Vasilis Kandylas ,
  • Rajesh Patel ,
  • Steven Shelford ,
  • ,
  • Hai Wu

MSR-TR-2013-106 |

Gold HITs — Human Intelligence Tasks with known answers — are commonly used to measure worker performance and data quality in industrial applications of crowdsourcing. We suggest adaptive exploration as a promising approach for automated, scalable Gold HIT creation. We substantiate this with initial experiments in a stylized model.