Paul N. Bennett

Researcher, Context, Learning, and User Experience for Search (CLUES), Microsoft Research

Research Activities:

I am interested in the development, improvement, and analysis of machine learning methods with a focus on systems that can aid in the automatic analysis of natural language as components of adaptive systems or information retrieval systems. My current focus is on contextual and personalized search. I am also actively engaged in enriched information retrieval, active sampling and learning, hierarchical and large-scale classification, and human computation and preferences.

My past work has examined a variety of areas — primarily ensemble methods, calibrating classifiers, search query classification and characterization, and redundancy and diversity, but also extending to transfer learning, machine translation, recommender systems, and knowledge bases. In addition to my research, I engage in a variety of professional service activities for the machine learning, data mining, and information retrieval communities.

While at Microsoft Research, I have been lucky enough to work with a variety of great interns — either as primary mentor or in broader collaborations.

Before coming to Microsoft, I obtained my Ph.D. from the Computer Science Department at Carnegie Mellon under Jaime Carbonell and John Lafferty.   Prior to that I worked with Ray Mooney and Robert Causey during my undergraduate days in the Computer Science, Philosophy, and Plan II Honors departments at the University of Texas at Austin.

 With colleagues from Microsoft and elsewhere, I am currently a Track Co-coordinator of the TREC 2013 Web Track, a co-organizer of the ICML 2013 workshop on where Machine Learning Meets Crowdsourcing, and a co-organizer of the CIKM 2013 workshop on Exploiting Semantic Annotations for Information Retrieval (ESAIR '13).

Publications and presentations by year (or by primary subject).

2013

•  Karthik Raman, Paul N. Bennett, and Kevyn Collins-Thompson (2013).  Toward Whole-Session Relevance: Exploring Intrinsic Diversity in Web Search.   To Appear in Proceedings of the 36th Annual ACM SIGIR Conference (SIGIR 2013).  Dublin, Ireland.  July-August 2013.
•  Milad Shokouhi, Ryen White, Paul N. Bennett, Filip Radlinski (2013).  Fighting Search Engine Amnesia: Reranking Repeated Results.   To Appear in Proceedings of the 36th Annual ACM SIGIR Conference (SIGIR 2013).  Dublin, Ireland.  July-August 2013.
•  Carsten Eickhoff, Kevyn Collins-Thompson, Paul N. Bennett, and Susan Dumais (2013).   Designing Human-Readable User Profiles for Search Evaluation.  Short-paper in Proceedings of the 35th ACM European Conference on Information Retrieval (ECIR '13).  Moscow, Russia.  March 2013.
•  Xi Chen, Paul N. Bennett, Kevyn Collins-Thompson, and Eric Horvitz (2013).   Pairwise Ranking Aggregation in a Crowdsourced Setting.  In Proceedings of the 6th ACM International Conference on Web Search and Data Mining (WSDM '13).  Rome, Italy.  February 2013.
•  Kira Radinsky and Paul N. Bennett (2013).   Predicting Content Change on the Web.  In Proceedings of the 6th ACM International Conference on Web Search and Data Mining (WSDM '13).  Rome, Italy.  February 2013.
•  Carsten Eickhoff, Kevyn Collins-Thompson, Paul N. Bennett, and Susan Dumais (2013).   Personalizing Atypical Web Search Sessions.  In Proceedings of the 6th ACM International Conference on Web Search and Data Mining (WSDM '13).  Rome, Italy.  February 2013.

2012

• Mining and Using Contextual Information from Large-Scale Web Search Logs. Invited talk at Emory University (Nov 2012). Also presented at Georgia Institute of Technology.
• J.J. Pfeiffer, J. Neville, P.N. Bennett (2012).  Active Sampling of Networks.   In Proceedings of the ICML 2012 Workshop on Mining and Learning with Graphs.  Edinburgh, Scotland.  July 2012.
• P.N. Bennett, R.W. White, W. Chu, S.T. Dumais, P. Bailey, F. Borisyuk, and X. Cui (2012).  Modeling the Impact of Short- and Long-Term Behavior on Search Personalization.   In Proceedings of the 35th Annual ACM SIGIR Conference (SIGIR 2012).  Portland, WA.  August 2012.
•  L. Wang, P.N.Bennett, K. Collins-Thompson (2012).  Robust Ranking Models via Risk-Sensitive Optimization.   In Proceedings of the 35th Annual ACM SIGIR Conference (SIGIR 2012).  Portland, WA.  August 2012. (Honorable Mention Best Paper)
• E. Agichtein, R.W. White, S.T. Dumais, and P.N. Bennett (2012).  Search, Interrupted: Understanding and Predicting Search Task Continuation.   In Proceedings of the 35th Annual ACM SIGIR Conference (SIGIR 2012).  Portland, WA.  August 2012.
• M. Bilgic and P.N. Bennett (2012).  Active Query Selection for Learning Rankers.   Poster-Paper in Proceedings of the 35th Annual ACM SIGIR Conference (SIGIR 2012).  Portland, WA.  August 2012.
• D.J. Liebling, P.N. Bennett, and R.W. White (2012).  Anticipatory Search: Using Context to Initiate Search.   Poster-Paper in Proceedings of the 35th Annual ACM SIGIR Conference (SIGIR 2012).  Portland, WA.  August 2012.
• Improving Classification in Large Taxonomies. Invited talk at Amazon Machine Learning Lunch Seminar (May 2012). PDF version of slides are here.
• Jagadeesh Jagarlamudi, Paul N. Bennett, and Krysta Svore (2012).   Leveraging Interlingual Classification to Improve Web Search.  In Poster-Track of the Proceedings of the 21st Annual International World Wide Web Conference (WWW '12).  Lyon, France.  April 2012.
• David Sontag, Kevyn Collins-Thompson, Paul N. Bennett, Ryen White, Susan Dumais, and Bodo von Billerbeck (2012).   Probabilistic Models for Personalizing Web Search.  In Proceedings of the 5th ACM International Conference on Web Search and Data Mining (WSDM '12).  Seattle, Washington.  February 2012.
• Jinyoung Kim, Kevyn Collins-Thompson, Paul N. Bennett, and Susan Dumais (2012).   Characterizing Web Content, User Interests, and Search Behavior by Reading Level and Topic.  In Proceedings of the 5th ACM International Conference on Web Search and Data Mining (WSDM '12).  Seattle, Washington.  February 2012.

2011

• Paul N. Bennett, Khalid El-Arini, Thorsten Joachims, Krysta M. Svore (2011).   Enriching Information Retrieval (Summary of the SIGIR 2011 Workshop).  In SIGIR Forum. pp. 60-65. Dec 2011.
• Paul N. Bennett, Ece Kamar, Gabriella Kazai (2011).   MSRC at TREC 2011 Crowdsourcing Track.  In Proceedings of the Twentieth Text REtrieval Conference (TREC 2011).  Gaithersburg, Maryland.  November 2011. Related talk slides: Consensus Relevance with Topic and Worker Conditional Models.
• Kevyn Collins-Thompson, Paul N. Bennett, Ryen White, Sebastian de la Chica, and David Sontag (2011).   Personalizing Web Search Results by Reading Level.  In Proceedings of the 20th ACM International Conference on Information and Knowledge Management (CIKM '11).  Glasgow, Scotland.  October 2011.
• Paul N. Bennett, Filip Radlinski, Ryen W. White, and Emine Yilmaz (2011).   Inferring and Using Location Metadata to Personalize Web Search.  In Proceedings of the 34th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2011).  Beijing, China.  July 2011.
• Alex Kotov, Paul N. Bennett, Ryen W. White, Susan Dumais, and Jaime Teevan (2011).   Modeling and Analysis of Cross-Session Search Tasks.  In Proceedings of the 34th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2011).  Beijing, China.  July 2011.
• Edith Law, Paul N. Bennett, and Eric Horvitz (2011).   The Effects of Choice in Routing Relevance Judgments.  In Proceedings of the 34th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2011).  Beijing, China.  July 2011.
• Jagadeesh Jagarlamudi and Paul N. Bennett (2011).   Fractional Similarity: Cross-lingual Feature Selection for Search.  In Proceedings of the 33rd Annual European Conference on Information Retrieval (ECIR 2011).  Dublin, Ireland.  April 2011.
• Christopher J.C. Burges, Krysta M. Svore, Paul N. Bennett, Andrzej Pastusiak, and Qiang Wu (2011).   Learning to Rank using an Ensemble of Lambda-Gradient Models.  In Journal of Machine Learning Research: Workshop and Conference Proceedings, Yahoo! Learning to Rank Challenge. vol. 14, pp. 25-35, Journal of Machine Learning Research.  2011.
• F. Radlinski, P.N. Bennett, E. Yilmaz (2011).   Detecting Duplicate Web Documents using Clickthrough Data.  In Proceedings of the 4th ACM International Conference on Web Search and Data Mining (WSDM '11).  Hong Kong.  Feb 2011.
• Preference, Consensus, and Choice in Crowdsourced Relevance. Invited talk at UT-Austin, Information School, Crowdsourcing Seminar (March 2011). PDF version of slides are here.

2010

• R. White, P.N. Bennett, S. Dumais (2010).   Predicting Short-term Interests Using Activity-Based Search Contexts.  In Proceedings of the 19th ACM International Conference on Information and Knowledge Management (CIKM '10).  Toronto, Canada.  Oct 2010.
• P.N. Bennett, V. Carvalho (2010).   Online Stratified Sampling: Evaluating Classifiers at Web-Scale.  Short-paper in Proceedings of the 19th ACM International Conference on Information and Knowledge Management (CIKM '10).  Toronto, Canada.  Oct 2010.
• P.N. Bennett, K. Svore, S. Dumais (2010).  Classification-Enhanced Ranking.  In Proceedings of the 19th Annual International World Wide Web Conference (WWW '10).  Raleigh, NC.  April 2010.
• K. Collins-Thompson, P.N. Bennett (2010).  Predicting Query Performance via Classification.  In Proceedings of the 32nd Annual European Conference on Information Retrieval (ECIR 2010).  Milton Keynes, UK.  March 2010.
• Class-Based Contextualized Search. Invited talk at Carnegie Mellon, Language Technologies Institute Colloquium (Oct 2010). Talk abstract. Also, presented at the University of Texas at Austin, Purdue University, and UIUC. PDF of recent slides are here.
• ECIR 2010 Tutorial on Machine Learning in IR: Recent Successes and New Opportunities.

2009

• F. Radlinski, P.N. Bennett, B. Carterette, T. Joachims (2009).   Redundancy, Diversity, and Interdependent Document Relevance, a summary of the SIGIR 2009 workshop.  In SIGIR Forum. pp. 46-52. Dec 2009.
• P.G. Ipeirotis, R. Chandrasekar, P.N. Bennett, E. Law, M. Chickering, A. Mityagin, F. Provost, L. von Ahn (2009).   A report on the Human Computation Workshop (HComp 2009) .  In KDD Explorations. Vol 11:2 pp. 80-83. Dec 2009.
• P.N. Bennett, N. Nguyen (2009).   Refined Experts: Improving Classification in Large Taxonomies.  In Proceedings of the 32nd Annual ACM SIGIR Conference (SIGIR 2009).  Boston, MA.  July 2009. (Talk powerpoint slides and PDF slide deck)
• K. Collins-Thompson, P.N. Bennett (2009).   Estimating Query Performance using Class Predictions..  Appears as a Poster-Paper in Proceedings of the 32nd Annual ACM SIGIR Conference (SIGIR 2009).  Boston, MA.  July 2009.
• P.N. Bennett, D.M. Chickering, A. Mityagin (2009).   Picture This: Preferences for Image Search..  Demo-paper in the Human Computation 2009 Workshop at KDD ’09 (see WWW'09 paper for primary version).  Paris, France.  June 2009.
• P.N. Bennett, D.M. Chickering, A. Mityagin (2009).   Learning Consensus Opinion: Mining Data from a Labeling Game.  In Proceedings of the 18th Annual International World Wide Web Conference (WWW '09).  Madrid, Spain.  April 2009.
• ICML 2009 Tutorial on Machine Learning in IR

2008

• P.N. Bennett, B. Carterette, O. Chapelle, T. Joachims (2008).   Beyond Binary Relevance: Preferences, Diversity, and Set-Level Judgments, a summary of the SIGIR 2008 workshop.  In SIGIR Forum. pp 53-58. Dec 2008.
• B. Carterette, P.N. Bennett (2008).   Evaluation Measures for Preference Judgments .  Appears as a Poster-Paper in Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.  Singapore.  July 2008.
• B. Carterette, P.N. Bennett, O. Chapelle (2008).   A Test Collection of Preference Judgments.  In Proceedings of the SIGIR 2008 Beyond Binary Relevance: Preferences, Diversity, and Set-Level Judgments Workshop .   Singapore.  July, 2008.
• B. Carterette, P.N. Bennett, D.M. Chickering, S.T. Dumais (2008).   Here or There: Preference Judgments for Relevance .  In Proceedings of ECIR 2008.  Glasgow, Scotland.  March-April 2008.

2007

• P.N. Bennett (2007).   Neighborhood-Based Local Sensitivity .  In Proceedings of ECML 2007.  Warsaw, Poland.  September 2007.
• P. Donmez, J.G. Carbonell, and P.N. Bennett (2007).   Dual Strategy Active Learning .  In Proceedings of ECML 2007.  Warsaw, Poland.  September 2007.
• P.N. Bennett and J. Carbonell (2007).   Combining Probability-Based Rankers for Action-Item Detection.  In Proceedings of HLT-NAACL 2007.  Rochester, New York.  April 2007. (Talk powerpoint slides)
• J.M. Adams, P.N. Bennett, and A. Tomasic (2007).  Combining Personalized Agents to Improve Content-Based Recommendations.  CMU-LTI-07-015, Language Technologies Institute, School of Computer Science, Carnegie Mellon University, December 2007.

2006

• P.N. Bennett (2006).  Building Reliable Metaclassifiers for Text Learning.  CMU-CS-06-121, Ph.D. Thesis, Computer Science Department, School of Computer Science, Carnegie Mellon University.  May 2006. See errata.

2005

• P.N. Bennett, S.T. Dumais, and E. Horvitz (2005).  The Combination of Text Classifiers Using Reliability Indicators.  Information Retrieval.  8(1):67-100.  Kluwer Academic Publishers.  January, 2005. Initial Submission: 12/19/2002; Acceptance (Pending Revision): 7/21/2003; Final Version Acceptance: 1/6/2004.  (This paper revises and extends work originally presented in [Bennett, Dumais, & Horvitz; SIGIR 2002].)
• P.N. Bennett and J. Carbonell (2005).  Feature Representation for Effective Action-Item Detection. In the Beyond Bag-of-Words Workshop at the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.  pp. 63-70.  Salvador, Brazil.  2005.  (This extends work presented as a SIGIR 2005 poster.)
• P.N. Bennett and J. Carbonell (2005).  Detecting Action-Items in E-mail.  Appears as a Poster-Paper in Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.  Salvador, Brazil.  August 2005.

2004

• Using Asymmetric Distributions to Improve Classifier Probability Estimates.  Presentation at the NIPS 2004 Workshop on Calibration and Probabilistic Prediction in Supervised Learning.

2003

• P.N. Bennett, S.T. Dumais, E. Horvitz (2003).  Inductive Transfer for Text Classification using Generalized Reliability Indicators. In Proceedings of the ICML-2003 Workshop on The Continuum from Labeled to Unlabeled Data.  Washington DC, U.S.A.  August 2003.
• P.N. Bennett (2003).  Using Asymmetric Distributions to Improve Text Classifier Probability Estimates.  In Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.  Toronto, Canada.  July-August 2003.  See errata. (This paper revises work that originally appeared as CMU-CS-02-126.)
• Leveraging Locality for Text Classifier Combination.  Presentation at the IJCAI 2003 Doctoral Consortium.
• R. Hutchinson, P.N. Bennett, J. Carbonell, P. Jansen, R. Brown (2003).  Maximal Lattice Overlap in Example-Based Machine Translation.  CMU-CS-03-138, Computer Science Department, School of Computer Science, Carnegie Mellon University, June 2003.  (Also listed as CMU-LTI-03-174. Shares significant overlap in content with publication listing for MT Summit IX, 2003.)
• R. Brown, R. Hutchinson, P.N. Bennett, J. Carbonell, P. Jansen (2003).  Reducing Boundary Friction Using Translation-Fragment Overlap.  In Proceedings of the Machine Translation Summit IX.  New Orleans, U.S.A.  September 2003.  (Shares significant overlap in content with publication listing for CMU-CS-03-138, 2003.)

2002

• P.N. Bennett, S.T. Dumais, and E. Horvitz (2002).  Probabilistic Combination of Text Classifiers Using Reliability Indicators: Models and Results.  In Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Tampere, Finland, August 2002. (An extended and revised version of this work appears in Information Retrieval, 2005.)
• P.N. Bennett (2002).  Using Asymmetric Distributions to Improve Classifier Probabilities: A Comparison of New and Standard Parametric Methods.  CMU-CS-02-126, Computer Science Department, School of Computer Science, Carnegie Mellon University, April 2002.  See errata. (A revised version of this work appears in SIGIR 2003.)

2000

• P.N. Bennett and A.D. Brown, eds. (2000).  The Guide to Living in Pittsburgh, 9th Edition, 299 pages, Pittsburgh, PA.  Dec/5, Inc.  2000.
• P.N. Bennett (2000).  Assessing the Calibration of Naive Bayes' Posterior Estimates.  CMU-CS-00-155, Computer Science Department, School of Computer Science, Carnegie Mellon University.  September 2000.

1998

• R.J. Mooney, P.N. Bennett, and L. Roy (1998).  Book Recommending Using Text Categorization with Extracted Information.  In the AAAI-98/ICML-98 Workshop on Learning for Text Categorization and the AAAI-98 Workshop on Recommender Systems.  Madison, WI.  July 1998.
• P.N. Bennett (1998).  Text Categorization Through Probabilistic Learning: Applications to Recommender Systems. Undergraduate Honor Thesis, Department of Computer Sciences, University of Texas at Austin. May 1998. (Also appears as AI TR 98-270.)
   

Professional Service:

• Conference Organization:
  • TREC 2013 Web Track Co-organizer.
  • CIKM 2012 Co-Demo Chair.
  • WSDM 2012 Tutorial Chair.
  • Co-organizer of the SIGIR 2011 Workshop on Enriching Information Retrieval.
  • Co-organizer of the SIGIR 2009 Workshop on Redundancy, Diversity, and Interdependent Document Relevance.
  • Co-organizer of the KDD 2009 Workshop on Human Computation.
  • Co-organizer and Chair of the SIGIR 2008 Workshop on Beyond Binary Relevance: Preferences, Diversity, and Set-Level Judgments.
  • Publication Chair of ICML, 2006.
• Senior Program Committees:
  • CIKM SPC 2011-2012.
  • SIGIR 2013 Area Chair.
• Conference Program Committees:
  • AAAI Doctoral Consortium 2008-2010.
  • ACL 2008.
  • CHI 2009.
  • CIKM 2010, CIKM SPC 2011.
  • ECIR 2008-2010.
  • ECML 2008 Workshop on Preference Learning.
  • ECML 2009-2010.
  • EMNLP 2012.
  • HLT-NAACL 2007.
  • HCOMP 2009-2011.
  • ICML 2010-2012.
  • IIIX 2012.
  • SDM 2013.
  • SIGIR 2004-2011.
  • SIGIR 2011 Crowdsourcing for IR Workshop
  • SIGKDD 2011.
  • WSDM 2011, 2013.
  • WWW 2010-2013.
• Journal Reviewing
  • Reviewer for the Journal of Artificial Intelligence Research (JAIR).
  • Reviewer for the Journal of the American Society for Information Science and Technology (JASIST).
  • Reviewer for the Journal of Machine Learning Research (JMLR).
  • Reviewer for ACM Transactions on Informations Systems (TOIS).
  • Reviewer for ACM Transactions on Internet Technology (TOIT).
• Co-organizer IR Seminar Series, Summer 2004 - Summer 2005.

Former Intern Collaborators

• Mustafa Bilgic
• Fedor Borisyuk
• Ben Carterette
• Xi Chen
• Carsten Eickhoff
• Jagadeesh Jagarlamudi
• Jinyoung Kim
• Alex Kotov
• Edith Law
• Nam Nguyen
• Kira Radinsky
• Karthik Raman
• Lidan Wang
• Rongjing Xiang

Contact Us Terms of Use Trademarks Privacy Statement ©2010-2013 Microsoft Corporation. All rights reserved.