Share on Facebook Tweet on Twitter Share on LinkedIn Share by email
Algorithmic Crowdsourcing

We are working toward a theoretic foundation of developing large-scale human-machine systems that combine the intelligence of human and the computing power of machine to address tasks that are difficult to complete by either human or machine alone.

People involved in such systems usually have incentives and diverse expertise. They may learn from their experiences, communicate and collaborate with others.  All these things together pose great scientific and engineering challenges in building efficient human-machine systems.

People

Publications

Invited Talks

  • Aggregating Ordinal Labels from Crowds by Minimax Conditional Entropy. ICML'14 workshop:  Crowdsourcing and Human Computing, June 25, 2014. (slides)
  • Algorithmic crowdsourcing. NIPS'13 workshop:  Crowdsourcing: Theory, Algorithms and Applications, December 9, 2013. (slides)
  • Learning from the Wisdom of Crowds by Minimax Entropy. UC Berkeley, Neyman Seminar, March 15, 2013. (slides)
  • Learning from the Wisdom of Crowds by Minimax Entropy. Facebook, March 14, 2013. (slides)
  • A Minimax Entropy Method for Learning the Wisdom of Crowds. Joint UW-Microsoft Research Machine Learning Workshop. Oct 26, 2012. (slides)

Software and Datasets

  • Web data: judging the relevance of query-URL pairs with a 5-level rating scale
  • Dog data: recognizing 4 breeds of dogs in a set of images
  • Bluebird data: image dataset of two breeds of bluebirds
  • Matlab code: minimax conditional entropy  (also include Dawid-Skene)