Nick Craswell

Nick Craswell
PRINCIPAL RESEARCH MANAGER
.

I am an Applied Researcher at Bing in Bellevue Washington, working on core relevance. I also lead a small team of Applied Researchers who are embedded in Microsoft Research Cambridge in the UK.

I'm a coordinator of the TREC-2009 Web Track, which evaluates search relevance on a 1 billion page web crawl [ ClueWeb09 ]. This year we are conducting a "TREC adhoc" evaluation and a diversity-aware evaluation.

Workshop: Web Search Click Data 2009 was held in conjunction with WSDM2009 in Barcelona, February 2009.

Research Overview

I am interested in Web search evaluation. I built the VLC, VLC2, WT2g and .GOV test collections, which have been made available to research groups around the world. David Hawking and I coordinated the TREC Web Track experiments. Some publications: Book chapter preprint (pdf), IR'01 (citeseer) and CSIRO'01 (pdf).

I also work on effective Web search, which means making use of information in pages, link structure and URL structure to generate more useful Web search results. Some papers: SIGIR'05 (pdf), SIGIR'01 (pdf), TOIS'03 (pdf) (copying is by permission of ACM, Inc.) and ADCS'03 (pdf).

My PhD was in distributed information retrieval (thesis pdf) which means building a system on top of multiple engines/databases that already exist. My recent work in the area has considered whether (or when) DIR is really practical. Some papers: ADC'99 (ps), DL'00 (pdf), ADC'03 (pdf) and ADC'04 (pdf).

Publications

2013

Ahmed Hassan, Xiaolin Shi, Nick Craswell, and Bill Ramsey, Beyond Clicks: Query Reformulation as a Predictor of Search Satisfaction, ACM International Conference on Information and Knowledge Management (CIKM), October 2013

Nick Craswell, Bodo Billerbeck, Dennis Fetterly, and Marc Najork, Robust Query Rewriting using Anchor Data, in 6th ACM International Conference on Web Search and Data Mining (WSDM), ACM, February 2013

Filip Radlinski and Nick Craswell, Optimized Interleaving for Online Retrieval Evaluation, in ACM International Conference on Web Search and Data Mining (WSDM) - Best Paper Award, ACM, February 2013

2012

Gabriella Kazai, Nick Craswell, Emine Yilmaz, and Seyed M. M. Tahaghoghi, An analysis of systematic judging errors in information retrieval, in 21st ACM International Conference on Information and Knowledge Management, CIKM'12, Maui, HI, USA, October 29 - November 02, 2012, ACM, 2012

Emine Yilmaz, Gabriella Kazai, Nick Craswell, and Seyed M. M. Tahaghoghi, On judgments obtained from a commercial search engine, in The 35th International ACM SIGIR conference on research and development in Information Retrieval, SIGIR '12, Portland, OR, USA, August 12-16, 2012, ACM, 2012

2011

Bodo Billerbeck, Nick Craswell, Dennis Fetterly, and Marc Najork, Microsoft Research at TREC 2011 Web Track, in Proc. of the 20th Text Retrieval Conference (TREC), National Institute of Standards and Technology , November 2011

Nick Craswell, Dennis Fetterly, and Marc Najork, The Power of Peers, in 33rd European Conference on IR Research (ECIR), Springer Verlag, April 2011

Daniel Sheldon, Milad Shokouhi, Martin Szummer, and Nick Craswell, LambdaMerge: Merging the Results of Query Reformulations, in Conf. Web search and data mining (WSDM), ACM, February 2011

2010

Nick Craswell, Dennis Fetterly, and Marc Najork, Microsoft Research at TREC 2010 Web Track, in Proc. of the 19th Text Retrieval Conference (TREC), National Institute of Standards and Technology , November 2010

Filip Radlinski and Nick Craswell, Comparing the Sensitivity of Information Retrieval Metrics, in Proceedings of SIGIR, Association for Computing Machinery, Inc., July 2010

Filip Radlinski, Martin Szummer, and Nick Craswell, Metrics for Assessing Sets of Subtopics, in SIGIR Conf. Research and Development in Information Retrieval, Association for Computing Machinery, Inc., July 2010

Tetsuya Sakai, Nick Craswell, Ruihua Song, Stephen Robertson, Zhicheng Dou, and Chin-Yew Lin, Simple Evaluation Metrics for Diversified Search Results, in The Third International Workshop on Evaluating Information Access (EVIA), National Institute of Informatics, June 2010

Filip Radlinski, Martin Szummer, and Nick Craswell, Inferring Query Intent from Reformulations and Clicks, in Proc. 19th Annual International World Wide Web Conference (WWW '10)., Association for Computing Machinery, Inc., April 2010

Emine Yilmaz, Milad Shokouhi, Nick Craswell, and Stephen Robertson, Expected browsing utility for web search evaluation, in 19th International Conference on Information and Knowledge Management, Association for Computing Machinery, Inc., 2010

Emine Yilmaz, Milad Shokouhi, Nick Craswell, and Stephen Robertson, Expected browsing utility for web search evaluation, in Proceedings of the 19th ACM international conference on Information and knowledge management , ACM, 2010

2009

Nick Craswell, Dennis Fetterly, Marc Najork, Stephen Robertson, and Emine Yilmaz, Microsoft Research at TREC 2009: Web and Relevance Feedback Tracks, in Proc. of the 18th Text Retrieval Conference (TREC), National Institute of Standards and Technology , November 2009

Dennis Fetterly, Nick Craswell, and Vishwa Vinay, The Impact of Crawl Policy on Web Search Effectiveness, in Proceedings of the 32nd annual international ACM SIGIR conference on Research and development in information retrieval, Association for Computing Machinery, Inc., 19 July 2009

Dennis Fetterly, Nick Craswell, and Vishwa Vinay, Measuring the Search Effectiveness of a Breadth-First Crawl , in Proceedings of the 31st European Conference on Information Retrieval (ECIR) , Springer Verlag, 9 April 2009

Marijn Koolen, Gabriella Kazai, and Nick Craswell, Wikipedia Pages as Entry Points for Book Search, in Proceedings of the Second ACM International Conference on Web Search and Data Mining (WSDM'09), Association for Computing Machinery, Inc., February 2009

Emine Yilmaz, Milad Shokouhi, Nick Craswell, and Stephen Robertson, Incorporating user behavior information in IR evaluation, in Understanding the user - Logging and interpreting user interactions in information retrieval. Workshop in Conjunction with the ACM SIGIR Conference on Information Retrieval., 2009

2008

Yunbo Cao, Jingjing Liu, Shenghua Bao, Hang Li, and Nick Craswell, A Two-Stage Model for Expert Search, no. MSR-TR-2008-143, October 2008

Marc Najork and Nick Craswell, Efficient and Effective Link Analysis with Precomputed SALSA Maps, in 17th ACM Conference on Information and Knowledge Management (CIKM), Association for Computing Machinery, Inc., October 2008

Onno Zoeter, Michael Taylor, Ed Snelson, John Guiver, Nick Craswell, and Martin Szummer, A Decision Theoretic Framework for Ranking using Implicit Feedback, in SIGIR 2008 Workshop on Learning to Rank for Information Retrieval, July 2008

Martin Szummer and Nick Craswell, Behavioral Classification on the Click Graph, in World Wide Web Conference, April 2008

Nick Craswell, Onno Zoeter, Michael Taylor, and Bill Ramsey, An experimental comparison of click position-bias models, in WSDM '08: Proceedings of the international conference on Web search and web data mining, ACM, New York, NY, USA, 2008

Dennis Fetterly, Nick Craswell, and Vishwa Vinay, Search Effectiveness with a Breadth-First Crawl, in Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, 2008

Arjen P. Vries, Anne-Marie Vercoustre, James A. Thom, Nick Craswell, and Mounia Lalmas, Overview of the INEX 2007 Entity Ranking Track, pp. 245–251, Springer-Verlag, Berlin, Heidelberg, 2008

Peter Bailey, Nick Craswell, Ian Soboroff, Paul Thomas, Arjen P. de Vries, and Emine Yilmaz, Relevance assessment: are judges exchangeable and does it matter, in SIGIR '08: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, ACM, New York, NY, USA, 2008

2007

Nick Craswell and Martin Szummer, Random walks on the click graph, in Proceedings of SIGIR 2007, July 2007

Peter Bailey, Nick Craswell, Ian Soboroff, and Arjen P. de Vries, The CSIRO enterprise search test collection, in SIGIR Forum, vol. 41, no. 2, pp. 42–45, ACM, New York, NY, USA, 2007

Jun Xu, Yunbo Cao, Hang Li, Nick Craswell, and Yalou Huang, Searching Documents Based on Relevance and Type, in ECIR 2007, 2007

2006

Thanh Tang, Nick Craswell, David Hawking, Kathleen Griffiths, and Helen Christensen, Quality and Relevance of Domain-specific Search: A Case Study in Mental Health, in Information Retrieval, vol. 9, no. 2, pp. 207–225, 2006

Michael Taylor, Hugo Zaragoza, Nick Craswell, Stephen Robertson, and Chris Burges, Optimisation methods for ranking functions with multiple parameters, in Proceedings of CIKM 2006, ACM Press, January 2006

2005

Charles Clarke, Nick Craswell, and Ian Soboroff, The TREC terabyte retrieval track, in SIGIR Forum, vol. 39, no. 1, pp. 25–25, 2005

Nick Craswell, Stephen Robertson, Hugo Zaragoza, and Michael Taylor, Relevance weighting for query independent evidence, in Proceedings of ACM SIGIR 2005, ACM Press, Salvador, Brazil, January 2005

Thanh Tang, David Hawking, Nick Craswell, and Kathleen Griffiths, Focused crawling for both topical relevance and quality of medical information, in Proceedings of CIKM 2005, Bremen, Germany, 2005

David Hawking and Nick Craswell, Very Large Scale Retrieval and Web Search, in Chapter in TREC: Experiment and Evaluation in Information Retrieval, MIT Press, 2005

Nick Craswell, Hugo Zaragoza, and Stephen Robertson, Microsoft Cambridge at TREC-14: Enterprise Track, in Proceedings of TREC-2005, 2005

2004

Nick Craswell and David Hawking, Overview of the TREC-2004 Web Track, in Proceedings of TREC-2004, Gaithersburg, Maryland USA, November 2004

Charles Clarke, Nick Craswell, and Ian Soboroff, Overview of the TREC-2004 Terabyte Track, in Proceedings of TREC-2004, Gaithersburg, Maryland USA, November 2004

David Hawking, Nick Craswell, Francis Crimmins, and Trystan Upstill, How Valuable is External Link Evidence when Searching Enterprise Webs?, in Proceedings of the Australasian Database Conference ADC2004, Dunedin, New Zealand, January 2004

Nick Craswell, Francis Crimmins, David Hawking, and Alistair Moffat, Performance and Cost Tradeoffs in Web Search, in Proceedings of the Australasian Database Conference ADC2004, Dunedin, New Zealand, January 2004

Hugo Zaragoza, Nick Craswell, Michael Taylor, Suchi Saria, and Stephen Robertson, Microsoft Cambridge at TREC 2004: Web and HARD track, in Proceedings of TREC-2004, Gaithersburg, MD: NIST, January 2004

David Hawking, Trystan Upstill, and Nick Craswell, Toward Better Weighting of Anchors, in Proceedings of ACM SIGIR 2004 (poster abstract), Sheffield UK, 2004

Yasuhiro Yamada, Nick Craswell, Tetsuya Nakatoh, and Sachio Hirokawa, Testbed for Information Extraction from Deep Web, in WWW2004 Poster Proceedings, New York, USA, 2004

2003

Trystan Upstill, Nick Craswell, and David Hawking, Predicting Fame and Fortune: PageRank or Indegree?, in Proceedings of the Australasian Document Computing Symposium, ADCS2003, Canberra, Australia, December 2003

Nick Craswell, David Hawking, Ross Wilkinson, and Mingfang Wu, Overview of the TREC-2003 Web Track, in Proceedings of TREC-2003, Gaithersburg, Maryland USA, November 2003

Nick Craswell, David Hawking, Trystan Upstill, Alistair McLean, Ross Wilkinson, and Mingfang Wu, TREC12 Web and Interactive Tracks at CSIRO, in Proceedings of TREC-2003, Gaithersburg, Maryland USA, November 2003

Trystan Upstill, Nick Craswell, and David Hawking, Query-independent evidence in home page finding, in ACM Transactions on Information Systems (TOIS), vol. 21, no. 3, pp. 286–313, ACM Press, 2003

Ian Soboroff, Ellen Voorhees, and Nick Craswell, Summary of the SIGIR 2003 workshop on defining evaluation methodologies for terabyte-scale test collections, in SIGIR Forum, vol. 37, no. 2, pp. 55–58, ACM Press, 2003

Peter Bailey, Nick Craswell, and David Hawking, Engineering a multi-purpose test collection for Web retrieval experiments, in Information Processing and Management, vol. 39, no. 6, pp. 853-871, 2003

Jared Cope, Nick Craswell, and David Hawking, Automated Discovery of Search Interfaces on the Web, in The Fourteenth Australasian Database Conference, Adelaide, Australia, 2003

2002

Nick Craswell, David Hawking, James Thom, Trystan Upstill, Ross Wilkinson, and Mingfang Wu, TREC11 Web and Interactive Tracks at CSIRO, in Proceedings of TREC-2002, Gaithersburg, Maryland USA, November 2002

Nick Craswell and David Hawking, Overview of the TREC-2002 Web Track, in Proceedings of TREC-2002, Gaithersburg, Maryland USA, November 2002

David Hawking, Nick Craswell, Francis Crimmins, and Trystan Upstill, Enterprise search: What works and what doesn't, in Proceedings of the Infonortics Search Engines Meeting, San Francisco, April 2002

Anne-Marie Vercoustre, James A. Thom, Alexander Krumpholz, Ian Mathieson, Peter Wilkins, Mingfang Wu, Nick Craswell, and David Hawking, CSIRO INEX experiments: XML search using PADRE, in INEX 2002 Workshop, Dagstuhl, Germany, 2002

Trystan Upstill, Nick Craswell, and David Hawking, Buying bestsellers online: A case study in Search & Searchability, in 7th Australasian Document Computing Symposium, Sydney, Australia, 2002

Nick Craswell, David Hawking, Alexander Krumpholz, Ian Mathieson, James A. Thom, Anne-Marie Vercoustre, Peter Wilkins, and Mingfang Wu, XML Document Retrieval with PADRE, in 7th Australasian Document Computing Symposium, Sydney, Australia, 2002

2001

Nick Craswell, David Hawking, Ross Wilkinson, and Mingfang Wu, TREC10 Web and Interactive Tracks at CSIRO, in Proceedings of TREC-2001, Gaithersburg, Maryland USA, November 2001

David Hawking and Nick Craswell, Overview of the TREC-2001 Web Track, in Proceedings of TREC-2001, Gaithersburg, Maryland USA, November 2001

David Hawking, Nick Craswell, and Kathleen Griffiths, Which search engine is best at finding online services?, in Poster Proceedings of WWW10, Hong Kong, May 2001

Nick Craswell, David Hawking, and Stephen Robertson, Effective site finding using link anchor information, in Proceedings of ACM SIGIR 2001, New Orleans, 2001

Nick Craswell, David Hawking, Anne-Marie Vercoustre, and Peter Wilkins, Panoptic Expert: Searching for experts not just for documents, in Ausweb Poster Proceedings, Queensland, Australia, 2001

David Hawking, Nick Craswell, Peter Bailey, and Kathleen Griffiths, Measuring search engine quality, in Journal of Information Retrieval, vol. 4, no. 1, pp. 33-59, 2001

Trystan Upstill, Raj Nagappan, and Nick Craswell, Visual Clustering of Image Search Results, in SPIE Visual Data Exploration and Analysis VIII, San Jose, California USA, 2001

Nick Craswell, David Hawking, and Kathleen Griffiths, Which search engine is best at finding airline site home pages?, no. 01/45, 2001

2000

Nick Craswell, Peter Bailey, and David Hawking, Server Selection on the World Wide Web, in Proceedings of the ACM Digital Libraries Conference, San Antonio, Texas USA, June 2000

Peter Bailey, Nick Craswell, and David Hawking, Dark matter on the Web, in WWW-9 Poster Proceedings, Amsterdam, The Netherlands, 2000

David Hawking, Peter Bailey, and Nick Craswell, Efficient and flexible search using text and metadata, 2000

Peter Bailey David Hawking and Nick Craswell, An intranet reality check for TREC ad hoc, 2000

Nick Craswell Peter Bailey and David Hawking, Chart of darkness: Mapping a large intranet, 2000

Nick Craswell, Methods for Distributed Information Retrieval, 2000

1999

David Hawking, Ellen Voorhees, Nick Craswell, and Peter Bailey, Overview of TREC-8 Web track, in Proceedings of TREC-8, Gaithersburg, Maryland USA, November 1999

David Hawking, Peter Bailey, and Nick Craswell, ACSys TREC-8 experiments, in Proceedings of TREC-8, Gaithersburg, Maryland USA, November 1999

David Hawking, Nick Craswell, Paul Thistlewaite, and Donna Harman, Results and challenges in Web search evaluation, Elsevier, Toronto, Canada, 1999

Nick Craswell, David Hawking, and Paul Thistlewaite, Merging Results from Isolated Search Engines, in Proceedings of the 10th Australasian Database Conference, Springer-Verlag, Auckland, NZ, 1999

Nick Craswell, Peter Bailey, and David Hawking, Is it fair to evaluate Web systems using TREC ad hoc methods?, in ACM SIGIR '99 Workshop on Web Retrieval, Berkeley, California USA, 1999

1998

David Hawking, Nick Craswell, and Paul Thistlewaite, ACSys TREC-7 experiments, in TREC-7, NIST special publication 500-242, Gaithersburg, Maryland USA, 1998

David Hawking, Nick Craswell, and Paul Thistlewaite, Overview of TREC-7 Very Large Collection Track, in TREC-7, NIST special publication 500-242, Gaithersburg, Maryland USA, 1998

1997

David Hawking, Paul Thistlewaite, and Nick Craswell, ANU/ACSys TREC-6 experiments, in TREC-6, NIST special publication 500-242, Gaithersburg, Maryland USA, 1997

Jason Haines, Brendan Humphreys, Chris Johnson Nick Craswell, and Paul Thistlewaite, Aglets: A good idea for spidering?, in 4th IDEA Workshop Proceedings, Queensland, Australia, 1997