Large-Scale Behavioral Data: Potential and Pitfalls

Over the last decade, the rise of web services has made it possible to gather traces of human behavior in situ at a scale and fidelity previously unimaginable. Large-scale behavioral data enables researchers and practitioners to detect adverse drug reactions and interactions, to understand how information diffuses through social networks, how people browse and search for information, how individual learning strategies are related to educational outcome, etc. Using examples from search, I will highlight how observational logs provide a rich new lens onto the diversity of searchers, tasks, and interactivity that characterize information systems today, and how experimental logs have revolutionized the way in which web-based systems are designed and evaluated. Although logs provide a great deal of information about what people are doing, they provide little insight about why they are doing so or whether they are satisfied. Complementary methods from observations, laboratory studies and panels are necessary to provide a more complete understanding of and support for search which is increasingly a core fabric of people’s everyday lives. The CHI community should lead the way in shaping best practices and policy in behavioral log studies.