Share on Facebook Tweet on Twitter Share on LinkedIn Share by email
Algorithmic Crowdsourcing

We are working toward a theoretic foundation of developing large-scale human-machine systems that combine the intelligence of human and the computing power of machine to address tasks that are difficult to complete by either human or machine alone.

People involved in such systems usually have incentives and diverse expertise. They may learn from their experiences, communicate and collaborate with others.  All these things together pose great scientific and engineering challenges in building an efficient human-machine system.




  • Double or Nothing: Multiplicative Incentive Mechanisms for Crowdsourcing. NIPS'14 Workshop on Transactional Machine Learning and E-Commerce. (slides)
  • Aggregating Ordinal Labels from Crowds by Minimax Conditional Entropy. ICML'14 workshop:  Crowdsourcing and Human Computing, June 25, 2014. (slides)
  • Algorithmic crowdsourcing. NIPS'13 workshop:  Crowdsourcing: Theory, Algorithms and Applications, December 9, 2013. (slides)
  • Learning from the Wisdom of Crowds by Minimax Entropy. UC Berkeley, Neyman Seminar, March 15, 2013. (slides)
  • Learning from the Wisdom of Crowds by Minimax Entropy. Facebook, March 14, 2013. (slides)
  • A Minimax Entropy Method for Learning the Wisdom of Crowds. Joint UW-Microsoft Research Machine Learning Workshop. Oct 26, 2012. (slides)

Software and Datasets

  • Web search data: judging the relevance of query-URL pairs with a 5-level rating scale
  • Dog data: recognizing 4 breeds of dogs in a set of images
  • Bluebird data: image dataset of two breeds of bluebirds
  • Age data: estimate the age of a person in a face image
  • Matlab code: minimax conditional entropy  (also include Dawid-Skene)