Anticipating Experimental Risks Using Surveys

We introduce a survey instrument for anticipating otherwise-unforeseen risks resulting from research experiments. We present experiments hypothetically, then ask: “If someone you cared about were a candidate participant for this experiment, would you want that person to be included as a participant?” (Q1) and “Do you believe the researchers should be allowed to proceed with this experiment?’ (Q2). Having honed this approach over multiple studies, and multiple years, we have aborted proposed studies due to survey respondents’ concerns. In this paper, we test this instrument by presenting five past (real) experiments, posed as hypotheticals, to 3,539 workers on Amazon’s Mechanical Turk. These experiments include Indiana University’s social phishing study, University of California’s `spamalytics’ study, and Facebook’s emotional contagion experiment. We reveal what researchers behind controversial experiments might have foreseen had our instrument been available to them prior to conducting their experiments.