Share on Facebook Tweet on Twitter Share on LinkedIn Share by email
Experiences Surveying the Crowd: Reflections on Methods, Participation, and Reliability

Catherine C. Marshall and Frank M. Shipman

Abstract

Crowdsourcing services such as Amazon's Mechanical Turk (MTurk) provide new venues for recruiting participants and conducting studies; hundreds of surveys may be offered to workers at any given time. We reflect on the results of six related studies we performed on MTurk over a two year period. The studies used a combination of open-ended questions and structured hypothetical statements about story-like scenarios to engage the efforts of 1252 participants. We describe the method used in the studies and reflect on what we have learned about identified best practices. We analyze the aggregated data to profile the types of Turkers who take surveys and examine how the characteristics of the surveys may influence data reliability. The results point to the value of participant engagement, identify potential changes in MTurk as a study venue, and highlight how communication among Turkers influences the data that researchers collect.

Details

Publication typeInproceedings
Published inProceedings of WebSci 2013
PublisherACM
> Publications > Experiences Surveying the Crowd: Reflections on Methods, Participation, and Reliability