Paul N. Bennett

Researcher, Context, Learning, and User Experience for Search (CLUES), Microsoft Research

Research Activities:

I am interested in the development, improvement, and analysis of machine learning methods with a focus on systems that can aid in the automatic analysis of natural language as components of adaptive systems or information retrieval systems. My current focus is on contextual and personalized search, enriched information retrieval, hierarchical and large-scale classification, search query classification and characterization, classification-aided IR ranking, diversity in retrieval results, preferences, and data mining via human computation applications.

My past work has examined a variety of areas — primarily text learning, ensemble methods, active learning, and obtaining reliable probability estimates, but also extending to machine translation, recommender systems, and knowledge bases.

We are looking for great demos for CIKM 2012.

Before coming to Microsoft, I obtained my Ph.D. from the Computer Science Department at Carnegie Mellon under Jaime Carbonell and John Lafferty.   Prior to that I worked closely with Ray Mooney and Robert Causey during my undergraduate days in the Computer Science, Philosophy, and Plan II Honors departments at the University of Texas at Austin.

Workshops, Collaborations and Papers:

•   P.N. Bennett, R.W. White, W. Chu, S.T. Dumais, P. Bailey, F. Borisyuk, and X. Cui (2012).  Modeling the Impact of Short- and Long-Term Behavior on Search Personalization.   To appear in Proceedings of the 35th Annual ACM SIGIR Conference (SIGIR 2012).  Portland, WA.  August 2012.
•   L. Wang, P.N.Bennett, K. Collins-Thompson (2012).  Robust Ranking Models via Risk-Sensitive Optimization.   To appear in Proceedings of the 35th Annual ACM SIGIR Conference (SIGIR 2012).  Portland, WA.  August 2012.
•   E. Agichtein, R.W. White, P.N. Bennett, and S.T. Dumais (2012).  Search, Interrupted: Understanding and Predicting Search Task Continuation.   To appear in Proceedings of the 35th Annual ACM SIGIR Conference (SIGIR 2012).  Portland, WA.  August 2012.
•   M. Bilgic and P.N. Bennett (2012).  Active Query Selection for Learning Rankers.   To appear as a Poster-Paper in Proceedings of the 35th Annual ACM SIGIR Conference (SIGIR 2012).  Portland, WA.  August 2012.
•   D.J. Liebling, P.N. Bennett, and R.W. White (2012).  Anticipatory Search: Using Context to Initiate Search.   To appear as a Poster-Paper in Proceedings of the 35th Annual ACM SIGIR Conference (SIGIR 2012).  Portland, WA.  August 2012.
•   David Sontag, Kevyn Collins-Thompson, Paul N. Bennett, Ryen White, Susan Dumais, and Bodo von Billerbeck (2012).   Probabilistic Models for Personalizing Web Search .  In Proceedings of the 5th ACM International Conference on Web Search and Data Mining (WSDM '12).  Seattle, Washington.  February 2012.
•   Jinyoung Kim, Kevyn Collins-Thompson, Paul N. Bennett, and Susan Dumais (2012).   Characterizing Web Content, User Interests, and Search Behavior by Reading Level and Topic .  In Proceedings of the 5th ACM International Conference on Web Search and Data Mining (WSDM '12).  Seattle, Washington.  February 2012.
•   Paul N. Bennett, Khalid El-Arini, Thorsten Joachims, Krysta M. Svore (2011).   Enriching Information Retrieval (Summary of the SIGIR 2011 Workshop) .  In SIGIR Forum. pp. 60-65. Dec 2011.
•   Paul N. Bennett, Ece Kamar, Gabriella Kazai (2011).   MSRC at TREC 2011 Crowdsourcing Track .  To Appear in Proceedings of the Twentieth Text REtrieval Conference (TREC 2011).  Gaithersburg, Maryland.  November 2011. Related talk slides: Consensus Relevance with Topic and Worker Conditional Models.
• Kevyn Collins-Thompson, Paul N. Bennett, Ryen White, Sebastian de la Chica, and David Sontag (2011).   Personalizing Web Search Results by Reading Level .  In Proceedings of the 20th ACM International Conference on Information and Knowledge Management (CIKM '11).  Glasgow, Scotland.  October 2011.
• Paul N. Bennett, Filip Radlinski, Ryen W. White, and Emine Yilmaz (2011).   Inferring and Using Location Metadata to Personalize Web Search .  In Proceedings of the 34th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2011).  Beijing, China.  July 2011.
• Alex Kotov, Paul N. Bennett, Ryen W. White, Susan Dumais, and Jaime Teevan (2011).   Modeling and Analysis of Cross-Session Search Tasks .  In Proceedings of the 34th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2011).  Beijing, China.  July 2011.
• Edith Law, Paul N. Bennett, and Eric Horvitz (2011).   The Effects of Choice in Routing Relevance Judgments .  In Proceedings of the 34th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2011).  Beijing, China.  July 2011.
• Jagadeesh Jagarlamudi and Paul N. Bennett (2011).   Fractional Similarity: Cross-lingual Feature Selection for Search .  In Proceedings of the 33rd Annual European Conference on Information Retrieval (ECIR 2011).  Dublin, Ireland.  April 2011.
• Learning to Rank using an Ensemble of Lambda-Gradient Models .  In Journal of Machine Learning Research: Workshop and Conference Proceedings, Yahoo! Learning to Rank Challenge. vol. 14, pp. 25-35, Journal of Machine Learning Research.  2011.
• F. Radlinski, P.N. Bennett, E. Yilmaz (2011).   Detecting Duplicate Web Documents using Clickthrough Data .  In Proceedings of the 4th ACM International Conference on Web Search and Data Mining (WSDM '11).  Hong Kong.  Feb 2011.
• R. White, P.N. Bennett, S. Dumais (2010).   Predicting Short-term Interests Using Activity-Based Search Contexts .  In Proceedings of the 19th ACM International Conference on Information and Knowledge Management (CIKM '10).  Toronto, Canada.  Oct 2010.
• P.N. Bennett, V. Carvalho (2010).   Online Stratified Sampling: Evaluating Classifiers at Web-Scale .  Short-paper in Proceedings of the 19th ACM International Conference on Information and Knowledge Management (CIKM '10).  Toronto, Canada.  Oct 2010.
• P.N. Bennett, K. Svore, S. Dumais (2010).  Classification-Enhanced Ranking.  In Proceedings of the 19th Annual International World Wide Web Conference (WWW '10).  Raleigh, NC.  April 2010.
• K. Collins-Thompson, P.N. Bennett (2010).  Predicting Query Performance via Classification.  In Proceedings of the 32nd Annual European Conference on Information Retrieval (ECIR 2010).  Milton Keynes, UK.  March 2010.
• F. Radlinski, P.N. Bennett, B. Carterette, T. Joachims (2009).   Redundancy, Diversity, and Interdependent Document Relevance, a summary of the SIGIR 2009 workshop.  In SIGIR Forum. pp. 46-52. Dec 2009.
• P.G. Ipeirotis, R. Chandrasekar, P.N. Bennett, E. Law, M. Chickering, A. Mityagin, F. Provost, L. von Ahn (2009).   A report on the Human Computation Workshop (HComp 2009) .  In KDD Explorations. Vol 11:2 pp. 80-83. Dec 2009.
• P.N. Bennett, N. Nguyen (2009).   Refined Experts: Improving Classification in Large Taxonomies.  In Proceedings of the 32nd Annual ACM SIGIR Conference (SIGIR 2009).  Boston, MA.  July 2009. (Talk powerpoint slides and PDF slide deck)
• K. Collins-Thompson, P.N. Bennett (2009).   Estimating Query Performance using Class Predictions..  Appears as a Poster-Paper in Proceedings of the 32nd Annual ACM SIGIR Conference (SIGIR 2009).  Boston, MA.  July 2009.
• P.N. Bennett, D.M. Chickering, A. Mityagin (2009).   Picture This: Preferences for Image Search..  Demo-paper in the Human Computation 2009 Workshop at KDD ’09 (see WWW'09 paper for primary version).  Paris, France.  June 2009.
• P.N. Bennett, D.M. Chickering, A. Mityagin (2009).   Learning Consensus Opinion: Mining Data from a Labeling Game.  In Proceedings of the 18th Annual International World Wide Web Conference (WWW '09).  Madrid, Spain.  April 2009.
• P.N. Bennett, B. Carterette, O. Chapelle, T. Joachims (2008).   Beyond Binary Relevance: Preferences, Diversity, and Set-Level Judgments, a summary of the SIGIR 2008 workshop.  In SIGIR Forum. pp 53-58. Dec 2008.
• B. Carterette, P.N. Bennett (2008).   Evaluation Measures for Preference Judgments .  Appears as a Poster-Paper in Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.  Singapore.  July 2008.
• B. Carterette, P.N. Bennett, O. Chapelle (2008).   A Test Collection of Preference Judgments .  In Proceedings of the SIGIR 2008 Beyond Binary Relevance: Preferences, Diversity, and Set-Level Judgments Workshop .   Singapore.  July, 2008.
• B. Carterette, P.N. Bennett, D.M. Chickering, S.T. Dumais (2008).   Here or There: Preference Judgments for Relevance .  In Proceedings of ECIR 2008.  Glasgow, Scotland.  March-April 2008.
• P.N. Bennett (2007).   Neighborhood-Based Local Sensitivity .  In Proceedings of ECML 2007.  Warsaw, Poland.  September 2007.
• P. Donmez, J.G. Carbonell, and P.N. Bennett (2007).   Dual Strategy Active Learning .  In Proceedings of ECML 2007.  Warsaw, Poland.  September 2007.
• P.N. Bennett and J. Carbonell (2007).   Combining Probability-Based Rankers for Action-Item Detection.  In Proceedings of HLT-NAACL 2007.  Rochester, New York.  April 2007. (Talk powerpoint slides)
• P.N. Bennett, S.T. Dumais, and E. Horvitz (2005).  The Combination of Text Classifiers Using Reliability Indicators.  Information Retrieval.  8(1):67-100.  Kluwer Academic Publishers.  January, 2005. Initial Submission: 12/19/2002; Acceptance (Pending Revision): 7/21/2003; Final Version Acceptance: 1/6/2004.  (This paper revises and extends work originally presented in [Bennett, Dumais, & Horvitz; SIGIR 2002].)
• P.N. Bennett and J. Carbonell (2005).  Feature Representation for Effective Action-Item Detection. In the Beyond Bag-of-Words Workshop at the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.  pp. 63-70.  Salvador, Brazil.  2005.  (This extends work presented as a SIGIR 2005 poster.)
• P.N. Bennett and J. Carbonell (2005).  Detecting Action-Items in E-mail.  Appears as a Poster-Paper in Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.  Salvador, Brazil.  August 2005.
• R. Brown, R. Hutchinson, P.N. Bennett, J. Carbonell, P. Jansen (2003).  Reducing Boundary Friction Using Translation-Fragment Overlap.  In Proceedings of the Machine Translation Summit IX.  New Orleans, U.S.A.  September 2003.  (Shares significant overlap in content with publication listing for CMU-CS-03-138, 2003.)
• P.N. Bennett, S.T. Dumais, E. Horvitz (2003).  Inductive Transfer for Text Classification using Generalized Reliability Indicators. In Proceedings of the ICML-2003 Workshop on The Continuum from Labeled to Unlabeled Data.  Washington DC, U.S.A.  August 2003.
• P.N. Bennett (2003).  Using Asymmetric Distributions to Improve Text Classifier Probability Estimates.  In Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.  Toronto, Canada.  July-August 2003.  See errata. (This paper revises work that originally appeared as CMU-CS-02-126.)
• P.N. Bennett, S.T. Dumais, and E. Horvitz (2002).  Probabilistic Combination of Text Classifiers Using Reliability Indicators: Models and Results.  In Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Tampere, Finland, August 2002. (An extended and revised version of this work appears in Information Retrieval, 2005.)
• R.J. Mooney, P.N. Bennett, and L. Roy (1998).  Book Recommending Using Text Categorization with Extracted Information.  In the AAAI-98/ICML-98 Workshop on Learning for Text Categorization and the AAAI-98 Workshop on Recommender Systems.  Madison, WI.  July 1998.
   

Tutorials:

• ICML 2009 Tutorial on Machine Learning in IR
• ECIR 2010 Tutorial on Machine Learning in IR: Recent Successes and New Opportunities.

Other Papers and Talks:

• Preference, Consensus, and Choice in Crowdsourced Relevance. Invited talk at UT-Austin, Information School, Crowdsourcing Seminar (March 2011). PDF version of slides are here.
• Class-Based Contextualized Search. Invited talk at Carnegie Mellon, Language Technologies Institute Colloquium (Oct 2010). Talk abstract. Also, presented at the University of Texas at Austin, Purdue University, and UIUC. PDF of recent slides are here.
• J.M. Adams, P.N. Bennett, and A. Tomasic (2007).  Combining Personalized Agents to Improve Content-Based Recommendations.  CMU-LTI-07-015, Language Technologies Institute, School of Computer Science, Carnegie Mellon University, December 2007.
• P.N. Bennett (2006).  Building Reliable Metaclassifiers for Text Learning.  CMU-CS-06-121, Ph.D. Thesis, Computer Science Department, School of Computer Science, Carnegie Mellon University.  May 2006. See errata.
• Using Asymmetric Distributions to Improve Classifier Probability Estimates.  Presentation at the NIPS 2004 Workshop on Calibration and Probabilistic Prediction in Supervised Learning.
• Leveraging Locality for Text Classifier Combination.  Presentation at the IJCAI 2003 Doctoral Consortium.
• R. Hutchinson, P.N. Bennett, J. Carbonell, P. Jansen, R. Brown (2003).  Maximal Lattice Overlap in Example-Based Machine Translation.  CMU-CS-03-138, Computer Science Department, School of Computer Science, Carnegie Mellon University, June 2003.  (Also listed as CMU-LTI-03-174. Shares significant overlap in content with publication listing for MT Summit IX, 2003.)
• P.N. Bennett (2002).  Using Asymmetric Distributions to Improve Classifier Probabilities: A Comparison of New and Standard Parametric Methods.  CMU-CS-02-126, Computer Science Department, School of Computer Science, Carnegie Mellon University, April 2002.  See errata. (A revised version of this work appears in SIGIR 2003.)
• P.N. Bennett and A.D. Brown, eds. (2000).  The Guide to Living in Pittsburgh, 9th Edition, 299 pages, Pittsburgh, PA.  Dec/5, Inc.  2000.
• P.N. Bennett (2000).  Assessing the Calibration of Naive Bayes' Posterior Estimates.  CMU-CS-00-155, Computer Science Department, School of Computer Science, Carnegie Mellon University.  September 2000.
• P.N. Bennett (1998).  Text Categorization Through Probabilistic Learning: Applications to Recommender Systems. Undergraduate Honor Thesis, Department of Computer Sciences, University of Texas at Austin. May 1998. (Also appears as AI TR 98-270.)
   

Other Activities:

• CIKM 2012 Co-Demo Chair.
• WSDM 2012 Tutorial Chair.
• Co-organizer of the SIGIR 2011 Workshop on Enriching Information Retrieval.
• Co-organizer of the SIGIR 2009 Workshop on Redundancy, Diversity, and Interdependent Document Relevance.
• Co-organizer of the KDD 2009 Workshop on Human Computation.
• Co-organizer and Chair of the SIGIR 2008 Workshop on Beyond Binary Relevance: Preferences, Diversity, and Set-Level Judgments.
• Publication Chair of ICML, 2006.
• Conference Reviewing:
  • Reviewer for AAAI Doctoral Consortium 2008-2010.
  • Reviewer for ACL 2008.
  • Reviewer for CHI 2009.
  • Reviewer for CIKM 2010, CIKM SPC 2011.
  • Reviewer for ECIR 2008-2010.
  • Reviewer for ECML 2008 Workshop on Preference Learning.
  • Reviewer for ECML 2009-2010.
  • Reviewer for HLT-NAACL 2007.
  • Reviewer for HCOMP 2009-2011.
  • Reviewer for ICML 2010-2011.
  • Reviewer for SIGIR 2004-2011.
  • Reviewer for SIGIR 2011 Crowdsourcing for IR Workshop
  • Reviewer for SIGKDD 2011.
  • Reviewer for WSDM 2011.
  • Reviewer for WWW 2010-2012.
• Journal Reviewing
  • Reviewer for the Journal of Artificial Intelligence Research.
  • Reviewer for JMLR.
  • Reviewer for TOIS.
  • Reviewer for ACM Transactions on Internet Technology.
• Co-organizer IR Seminar Series, Summer 2004 - Summer 2005.

Contact Us Terms of Use Trademarks Privacy Statement ©2010-2011 Microsoft Corporation. All rights reserved.