WSCD2012: Workshop on Web Search Click Data

Including the results of the Relevance Prediction Challenge
Held in conjunction with WSDM 2012
February 12th, 2012

9:00am – Morning Session I

• Introduction (10 min)
• Invited talk (40 min)

  Practical and Reliable Retrieval Evaluation Through Online Experimentation
  Yisong Yue – Carnegie Mellon University – Abstract and bio – Slides

  Abstract
  Online evaluation is amongst the few evaluation techniques available to the information retrieval community that is guaranteed to reflect how users actually respond to improvements developed by the community.

  Broadly speaking, online evaluation refers to any evaluation of retrieval quality conducted while observing user behavior in a natural context. For example, in interleaving experiments, the rankings computed by two retrieval functions are interleaved together and then presented to the user. Afterward, clicks can be interpreted as a relative preference of one retrieval function over the other. The effectiveness of such approaches lie in their ability to directly measure how real users perceive retrieval quality, as well as their ability to preemptively control for various sources of bias or noise.

  In this talk, I will briefly overview the interleaving approach, and showcase how it can reliable and practical even at smaller scales, such as in academic settings. I will conclude by presenting recent advancements in learning more sensitive test statistics for interleaving experiments, as well as interactive model optimization via sequential online interleaving experiment design.

  Demo scripts are available at: http://www.yisongyue.com/downloads/sigir_tutorial_demo_scripts.tar.gz

  Bio
  Yisong Yue is a postdoctoral researcher in the iLab and Machine Learning Department at Carnegie Mellon University. His research interests lie primarily in structured prediction and interactive systems, with an application focus in information retrieval and access. He received a Ph.D. from Cornell University and B.S. from the University of Illinois at Urbana-Champaign. He is the author of the SVM-map software package for optimizing mean average precision using support vector machines. His current research focuses on machine learning approaches to online experiment design, diversified retrieval, and interactive information retrieval.

• 2 short talks (15+15 min)

  Investigating the Effect of Duration, Page Size and Frequency on Next Page Recommendation with Page Rank Algorithm
  Banu Deniz Gunel and Pinar Senkul

  Detecting Search Sessions Using Document Metadata and Implicit Feedback
  Tomas Kramar and Maria Bielikova

10:20am – Morning Break

10:40am – Morning Session II

• 1 short talk (15 min)

  Analyzing Related Real and Virtual Networks of Geographical Names Using Query Logs
  Kazuhiro Kazama

• 2 regular talks (25+25 min)

  What Are We Searching For? Analyzing User Objectives When Searching Relational Data
  Joel Coffman and Alfred C. Weaver

  QRU-1: A Public Dataset for Promoting Query Representation and Understanding Research
  Hang Li, Gu Xu, W. Bruce Croft, Michael Bendersky, Ziqi Wang and Evelyne Viegas

• Yahoo dataset (20 min)

  Description of New Yahoo Data Release
  Georges Dupret

• Challenge Overview and Summary of Results (25 min)

  Overview of the Relevance Prediction Challenge
  Pavel Serdyukov

12:30pm – Lunch

1:40pm – Afternoon Session I

• 2 challenge talks from industry participants (25+25 min)

  Microsoft Participation
  Mikhail Parakhin

  Yandex Participation
  Andrey Gulin

• 2 Challenge talks from university participants (25+25 min)

  Improving Relevance Prediction by Addressing Biases and Sparsity in Web Search Click Data
  Qi Guo, Dmitry Lagun, Denis Savenkov, Qiaoling Liu

  Linear combination of random forests for the Relevance Prediction Challenge
  Michael Figurnov and Alexander Kirillov

3:20pm – Afternoon Break

3:40pm – Afternoon Session II

• 1 Challenge talk from a university participant (30 min)

  Learning from Click Model and Latent Factor Model for Relevance Prediction Challenge
  Botao Hu, Nathan N. Liu and Weizhu Chen

• Panel Discussion (50 min)

  Q&A and Discussion: Current and Future Data Releases
  The workshop included presentations on current data releases by Yandex, Yahoo and Microsoft. This panel is a chance to discuss the impact of these releases, ideas for future releases and what we can do as a community to enable research on log data.

5:00pm – End of Workshop