Larry Heck

Larry Heck

Brief Bio

Larry Heck is a Distinguished Engineering in Microsoft Research. His research area is natural conversational interaction, focusing on open-domain NLP and dialog, machine learning, multimodal NUI, and inference/reasoning under uncertainty.  In 2009, Dr. Heck started the conversational understanding (CU) effort at Microsoft. The CU effort established the early scientific foundations for Cortana, Microsoft’s personal assistant for Windows Phone. From 2005 to 2009, he was Vice President of Search & Advertising Sciences at Yahoo!, responsible for the creation, development, and deployment of the algorithms powering Yahoo! Search, Yahoo! Sponsored Search, Yahoo! Content Match, and Yahoo! display advertising. From 1998 to 2005, he was with Nuance Communications and served as Vice President of R&D, responsible for natural language processing, speech recognition, voice authentication, and text-to-speech synthesis technology. He began his career as a researcher at the Stanford Research Institute (1992-1998), initially in the field of acoustics and later in speech research with the Speech Technology and Research (STAR) Laboratory. Dr. Heck received the PhD in Electrical Engineering from the Georgia Institute of Technology in 1991.

Current Research Interests

  • Natural Language Processing (NLP): Open-Domain (Web-Scale)
  • Leveraging Knowledge Graphs for Unsupervised NLP
  • Deep Learning/Deep Neural Networks from Structured Knowledge
  • Multi-modal Situated Conversational Interactions
  • Inference and Reasoning

Work Experience

  • 2012 - Present:  Microsoft, Mountain View, CA
    Microsoft Research
    Distinguished Engineer
  • 2009 - 2012: Microsoft, Mountain View, CA
    Speech at Microsoft
    Chief Scientist
  • 2005-2009:  Yahoo, Sunnyvale, CA
    Head of Search and Advertising Sciences, Yahoo Labs
    Vice President
  • 1998-2005:  Nuance Communications, Menlo Park, CA
    Head of Nuance Research and Development
    Vice President
  • 1992-1998:  SRI International, Menlo Park, CA
    Speech Technology and Research (STAR) Lab
    Acoustics and Radar Technology Lab (ARTL)
    Senior Research Engineer


  • Georgia Institute of Technology
    PhD Electrical Engineering
  • Georgia Institute of Technology
    MS Electrical Engineering
  • Texas Tech University
    BS Electrical Engineering
Deep Learning Neural Networks
Conversational Knowledge Graphs
Weakly Supervised Learning from Query Click Logs
Conversational NUI
Task Learning and Dialog Modeling
Spoken Language Understanding (General)
Speaker Recognition Papers (since 2000)
Other Recent Publications


  1. "Method for establishing handset-dependent normalizing models for speaker recognition", US Patent #5950157, Sept. 1999 (with M. Weintraub)
  2. "Voice authentication system having cognitive recall mechanism for password verification,” US Patent #6671672, Dec. 2003.
  3. "Method and system for on-line unsupervised adaptation in speaker verification,” US Patent #6,804,647, October 2004 (with N. Mirghafori)
  4. "Method and system for improving robustness in a speech system ,” US Patent #7203652, April 2007.
  5. “Detection of Characteristics of Human-Machine Interactions for Dialog Customization and Analysis,” US Patent #7263489, August 2007 (with M.Cohen, J. Balogh, J. Riseman, N. Mirghafori)
  6. “Adapting a Context-Independent Relevance Function for Identifying Relevant Search Results,” US Patent Application, 2007.
  7. “Automated Learning from a Question and Answering Network of Humans,” US Patent #7809664, 2007.
  8. “Methods and Apparatuses for Adapting a Ranking Function of a Search Engine For Use with a Specific Domain,” US Patent #8126839, 2008 (with K. Chen, R. Lu, C.K. Wong, G. Sun, B. Tseng)
  9. “Maintaining State Transition Data for a Plurality of Users, Modeling, Detecting, and Predicting User States and Behavior", US Patent #7941383, 2011.
  10. "Deep Linking from Task List Based on Intent", US Patent App 13/401876, 2012 (with AK Sullivan, L Stifelman, K Lee, M Klee, G Tur, D Hakkani-Tur)
  11. "Contextual Query Adjustments Using Natural Action Input", US Patent App 13/549503, 2012 (with M Chinthakunta, R Iyer)
  12. "Task Driven User Intents", U.S. Patent No. 20,120,253,791, 2012 (with M Chinthakunta, D Mitby, L Stifelman)
  13. "Personalization of Queries, Conversations, and Searches" U.S. Patent No. 2012135791, 2012 (with M Chinthakunta, D Mitby, L Stifelman)
  14. "Combined Activation for Natural User Interface Systems", U.S. Patent No. 20,120,254,810, 2012 (with M Chinthakunta, D Mitby, L Stifelman)
  15. "Location-Based Conversational Understanding", U.S. Patent No. 2012135210, 2012 (with M Chinthakunta, D Mitby, L Stifelman)
  16. "Augmented Conversational Understanding Agent", U.S. Patent No. 20,120,253,788, 2012 (with M Chinthakunta, D Mitby, L Stifelman)
  17. "Augmented Conversational Understanding Architecture", U.S. Patent No. 20,120,254,227, 2012 (with M Chinthakunta, D Mitby, L Stifelman)
  18. "Conversational Dialog Learning and Correction", U.S. Patent No. 2012135229, 2012 (with M Chinthakunta, D Mitby, L Stifelman)
  19. "Sentence Simplification for Spoken Language Understanding", U.S. Patent No. 20,120,290,290, 2012 (with G Tur, D Hakkani-Tur, S Parthasarathy)
  20. "Exploiting Query Click Logs for Domain Detection in Spoken Language Understanding", U.S. Patent No. 20,120,290,293, 2012 (with D Hakkani-Tur, G Tur)
  21.  "Training Statistical Dialog Managers in Spoken Dialog Systems with Web Data", U.S. Patent No. 20,120,290,509, 2012 (with D Hakkani-Tur, R Iyer, G Tur)
  22. "Suggesting Intent Frame(s) for User Request(s)", US Patent No. 20,130,158,980, 2012 (with SJ Landry, AK Sullivan, L Stifelman, AD Elman, S Parthasarathy)
  23. "Translating Natural Language Utterances to Keyword Search Queries", US Patent Filed 2012 (with D Hakkani-Tur, G Tur, R Iyer)
  24. "Probability-Based State Modification for Query Dialogues", MS#336765.01 filed on 12/14/2012 (with Dilek Hakkani-Tür, Gokhan Tur, A Celikyilmaz, A Fidler)
  25. "Collaborative Learning through User Generated Knowledge", MS#337949.01 filed on 1/17/2013.
  26. "Prosodic and Lexical Addressee Detection", MS# 337999.01, filed on 1/31/2013. (with Elizabeth Shriberg, Andreas Stolcke, and Larry Heck)
  27. "Satisfying Specified Intent(s) Based on Multimodal Request(s)", US Patent 20,130,159,001, 2013 (with L Stifelman, AK Sullivan, AD Elman, ST Tryphonas, KR Zargahi, KH Thai)
  28. "Unsupervised Relation Detection Model Training", filed US Patent, 2013 (with D Hakkani-Tur, G Tur)
  29. "Exploiting the Semantic Web for Unsupervised Spoken Language Understanding", MS# 338085.01, filed US Patent 2/18/2013 (with D Hakkani-Tur, G Tur)
  30. Exploiting the Semantic Web for Unsupervised Natural Language Semantic Parsing. MS# 337845.01, filed US Patent 2/27/2013 (with Gokhan Tur, Minwoo Jeong, and Ye-Yi Wang)
  31. "Using Human Wizards in A Conversational Understanding System", MS# 338086.01, filed US Patent 3/4/2013 (with Lisa Stifelman, Madhu Chinthakunta, and Larry Heck)
  32. "Active Learning Using Different Knowledge Sources", MS#339016.01, filed US Patent 6/2013 (with R Sarikaya, T Anastasakos, D Boies).
  33. "Environmentally Aware Dialog Policies and Response Generation", MS#339015.01, filed US Patent 6/2013 (with R Sarikaya, T Anastasakos, D Boies).
  34. "Building Conversational Understanding Systems Using a Toolset", MS#339005.01, filed US Patent 6/2013 (with R Sarikaya, T Anastasakos, D Boies).
  35. "A Deep Structured Semantic Model Produced Using Click-Through Data", MS#339669.01, filed with US Patent on 9/2013 (with X He, J Gao, A Acero, L Deng, P Huang).
  36. "Multi-user, Multi-domain Dialog System", MS#339845.01, filed US Patent 12/2013 (with D Hakkani-Tur, G Tur, L.P. Heck, D Wang).
  37. "Unsupervised Relation Detection Model Training, MS#339846.01, filed US Patent 12/2013 (with D Hakkani-Tur, G Tur).
  38. "Model-Based Approach for On-Screen Item Selection and Disambiguation", MS#340453.01, filed US Patent 3/2014 (with D Hakkani-Tur, G Tur, Z Feizollahi, A Celikyilmaz, R Sarikaya).
  39. "Discriminating Between Natural Language and Keyword Language Items", US Patent Filed, 2014 (with D Hakkani-Tur, A Celikyilmaz)
  40. "Techniques for Inferring Unknown Intents of Linguistic Items", MS#340356.01, filed US Patent 2/2014 (with Y Ji, D Hakkani-Tur, A Celikyilmaz, G Tur).
  41. "Transferring Information Across Language Understanding Model Domains", MS#340827.01, filed US Patent 4/2014 (with Derek Liu, Ruhi Sarikaya, Dilek Hakkani-Tur, Gokhan Tur)
  42. "Facilitating Conversations with Automated Location Mapping", MS#340001.01", filed 5/2014 (with L Stifelman, M Chinthakunta, J Odell, D Dole).
  43. "Knowledge Source Personalization to Improve Language Models", MS #340646.01, filed 5/2014 (with M Akbacak, G Tur, B Dumoulin, D Hakkani-Tur).
  44. "Language Modeling for Conversational Understanding Domains Using Semantic Web Resources", MS#340623.01, filed 5/2014 (with M Akbacak, G Tur, B Dumoulin, D Hakkani-Tur).
  45. "Session Context Modeling for Language Models in Conversational Understanding Systems using Knowledge Graphs and User Session Logs", MS #340625.01, filed 4/2014, (with M Akbacak, G Tur, D Liu, D Hakkani-Tur).
  46. "Deep Learning for Semantic Parsing Including Semantic Utterance Classification", MS #340534.01, filed 4/2014 (with Y Dauphin, G Tur, D Hakkani-Tur)