Share on Facebook Tweet on Twitter Share on LinkedIn Share by email
Crowdsourcing and Human Computation

We are working toward a theoretic foundation of developing large-scale human-machine systems that combine the intelligence of human and the computing power of machine to solve the problems that are difficult to solve by either human or machine alone.

People involved in such systems usually have incentives and diverse expertise. They may learn from their experiences, communicate and collaborate with others.  All these things together pose great scientific and engineering challenges in building an efficient human-machine system.

People

Publications

Talks

  • Incentives in Human Computation. Microsoft TechFest, March 26, 2015.
  • Double or Nothing: Unique Mechanism to Incentivize MTurkers to Skip. Computer Science and Engineering Department, University of Washington. Hosted by Prof. Anna Karlin. March 6, 2015. (slides)
  • Double or Nothing: Unique Mechanism to Incentivize MTurkers to Skip. Facebook, March 6, 2015.
  • Double or Nothing: Unique Mechanism to Incentivize MTurkers to Skip. Microsoft Bing, November 5, 2014.
  • Algorithmic Crowdsourcing. NIPS Workshop on Crowdsourcing: Theory, Algorithms and Applications, December 9, 2013. (slides)
  • Learning from the Wisdom of Crowds by Minimax Entropy. Amazon, July 25, 2013.
  • Learning from the Wisdom of Crowds by Minimax Entropy. UC Berkeley, Neyman Seminar, March 15, 2013. (slides)
  • Learning from the Wisdom of Crowds by Minimax Entropy. Facebook, March 14, 2013.
  • Learning from the Wisdom of Crowds by Minimax Entropy. Joint UW-Microsoft Research Machine Learning Workshop. Oct 26, 2012.

Crowdsourcing datasets

Software

Matlab code: minimax conditional entropy for both multiclass and ordinal labels together with cross validation for choosing regularization parameters.  Majority voting and the Dawid-Skene estimator are also implemented as the baselines.