Larry Heck

Larry Heck

Brief Bio

Larry Heck is a Distinguished Engineer at Microsoft Research. His research area is natural conversational interaction, focusing on open-domain NLP and dialog, machine learning, multimodal NUI, and inference/reasoning under uncertainty.  In 2009, Dr. Heck started the conversational understanding (CU) effort at Microsoft. The CU effort established the early scientific foundations for Cortana, Microsoft’s personal assistant for Windows Phone. From 2005 to 2009, he was Vice President of Search & Advertising Sciences at Yahoo!, responsible for the creation, development, and deployment of the algorithms powering Yahoo! Search, Yahoo! Sponsored Search, Yahoo! Content Match, and Yahoo! display advertising. From 1998 to 2005, he was with Nuance Communications and served as Vice President of R&D, responsible for natural language processing, speech recognition, voice authentication, and text-to-speech synthesis technology. He began his career as a researcher at the Stanford Research Institute (1992-1998), initially in the field of acoustics and later in speech research with the Speech Technology and Research (STAR) Laboratory. Dr. Heck received a PhD in Electrical Engineering from the Georgia Institute of Technology in 1991.

Current Research Interests

  • Natural Language Processing (NLP): Open-Domain (Web-Scale)
  • Leveraging Knowledge Graphs for Unsupervised NLP
  • Deep Learning/Deep Neural Networks from Structured Knowledge
  • Multi-modal Situated Conversational Interactions
  • Inference and Reasoning

Work Experience

  • 2012 - Present:  Microsoft, Mountain View, CA
    Microsoft Research
    Distinguished Engineer
  • 2009 - 2012: Microsoft, Mountain View, CA
    Speech Products
    Chief Scientist
  • 2005-2009:  Yahoo, Sunnyvale, CA
    Head of Search and Advertising Sciences, Yahoo Labs
    Vice President
  • 1998-2005:  Nuance Communications, Menlo Park, CA
    Head of Nuance Research and Development
    Vice President
  • 1992-1998:  SRI International, Menlo Park, CA
    Speech Technology and Research (STAR) Lab
    Acoustics and Radar Technology Lab (ARTL)
    Senior Research Engineer


  • Georgia Institute of Technology
    PhD/MS Electrical Engineering
Conversational Knowledge Graphs
Deep Learning Neural Networks
Weakly Supervised Learning from Query Click Logs
Conversational NUI
Task Learning and Dialog Modeling
Spoken Language Understanding (General)
Speaker Recognition

Active Noise and Vibration Control

  • K. Naghshineh, L.P. Heck, J. Olkin, and J. Kamman, "Evaluation of an Actuator Placement Method for Active Noise Control", Journal of Vibration and Acoustics, Transactions of the ASME, to appear 1998.
  • L.P. Heck, K. Naghshineh, J. Olkin, "Transducer Placement for Broadband Active Vibration Control Using a Novel Multidimensional QR Factorization," Journal of Vibration and Acoustics, Transactions of the ASME, to appear 1998
  • L.P. Heck and K. Naghshineh, "Optimal Interpolation of Structural Responses for Active Noise Control," IEEE Trans. on Speech and Audio Processing , Vol. 5, No. 6, pp. 560-565, November 1997.
  • K. Naghshineh, L.P. Heck, J. Olkin, "Evaluation of the Multidimensional QR Algorithm for Automated Actuator Placement in Active Control Applications", International Mechanical Engineering Congress and Exposition, Atlanta, GA., Nov. 1996.
  • J. Olkin, L.P. Heck, K. Naghshineh, "Automatic Placement of Transducers for Active Noise Control: Performance Measures," Proc. IEEE Intern. Conf. on Acoustics, Speech, and Signal Proc., Atlanta, GA., May 1996.
  • D.S. Flamm, L.P. Heck, P.J. Titterton, W.C. Nowlin, J.A. Olkin, K.C. Chou, ``Control System Design for the SPICES Smart Structure Demonstrations,'' Proc. of IEEE Conference on Decision and Control New Orleans, LA., Dec. 1995.
  • L.P. Heck, "Automated Transducer Placement for the SPICES Active Vibration Control Demonstrations," International Mechanical Engineering Congress and Exposition, San Francisco, CA., Nov. 1995.
  • L.P. Heck, "A Multidimensional Householder QR Algorithm for Transducer Placement in Active Noise Control Systems", IEEE Workshop on Appl. of Signal Proc. to Audio and Acoustics, New Paltz, NY, Oct. 1995.
  • L.P. Heck, ``Broadband Sensor and Actuator Selection for Active Control of Smart Structures,'' Proc. of SPIE Symposium on Smart Materials and Structures, San Diego, CA., March 1995.
  • L.P. Heck and K. Naghshineh, ``Large-scale, Broadband Actuator Selection for Active Noise Control,'' Proc. of Intern. Conf. on Noise Control for Industry, Ft. Lauderdale, Florida, May 1994.
  • L.P. Heck, K. Naghshineh, and J.F. Stach, ``Interpolation of Forced Structural Responses From Non-uniform Sparse Measurements,'' Proc. of IEEE Workshop on Audio and Acoustics, Mohonk, NY, October 1993.

Machinery Health Monitoring

  • L.P. Heck, "Monitoring the Health of Mechanical Systems with Smart Materials: Signal Processing Issues" in Digital Signal Processing Technology, ed. P. Papamichalis and R. Kerwin, Vol. CR57, SPIE Optical Engineering Press, Bellingham, Washington, USA.
  • K.C. Chou and L.P. Heck, ``A Multiscale Stochastic Modeling Approach to the Monitoring of Mechanical Systems,'' Proc. of the IEEE Intern. Symposium on Time-Frequency/Time-Scale Analysis, Philadelphia, Pennsylvania, October 1994.
  • L.P. Heck and K.C. Chou, ``Gaussian Mixture Model Classifiers for Machine Monitoring,'' Proc. of IEEE Intern. Conf. on Acoustics, Speech, and Signal Processing , Adelaide, Australia, May 1994.
  • L.P. Heck, ``Signal Processing Research in Tool Wear Monitoring,'' Invited Paper in Proc. of the IEEE Intern. Conf. on Acoustics, Speech, and Signal Processing, Minneapolis, MN, April 1993.
  • Larry Heck, "A Subspace Approach to the Automatic Design of Pattern Recognition Systems for Mechanical Systems Monitoring," PhD Dissertation, School of Electrical Engineering, Georgia Institute of Technology, December 1991.
  • L.P. Heck and J.H. McClellan, ``Mechanical System Monitoring Using Hidden Markov Models,'' Proc. of IEEE Intern. Conf. on Acoustics, Speech, and Signal Processing, Toronto, Canada, May 1991.

Other Publications

  • Keke Chen, Rongqing Lu, CK Wong, Gordon Sun, Larry Heck, and Belle Tseng, Trada: Tree Based Ranking Function Adaptation, ACM International Conference on Information and Knowledge Management (CIKM), October 2008
  • Larry Heck and Ananth Sankar, Acoustic Clustering and Adaptation for Robust Speech Recognition, Eurospeech, September 1997
  • A. Sankar, L.P.Heck, and A.Stolcke, "Acoustic Modeling for the SRI Hub4 Partitioned Evaluation Continuous Speech Recognition System," Proc. of 1997 DARPA Speech Recognition Workshop, Chantilly, VA., 1997.
  • S. J. Reeves and L. P. Heck, ``Selection of observations in signal reconstruction,'' IEEE Transactions on Signal Processing, vol. 43, no. 3, pp. 788-791, March 1995.
  • L.P. Heck and J.H. McClellan, ``Subspace Techniques for Large-Scale Feature Selection,'' Proc. of the IEEE Intern. Conf. on Acoustics, Speech, and Signal Processing, Minneapolis, MN, April 1993.
  • S.J. Reeves and L.P. Heck, ``Observation Selection in Signal Reconstruction,'' Proc. of the IEEE Intern. Conf. on Acoustics, Speech, and Signal Processing, Minneapolis, MN, April 1993.
  • T.H. Fox, J.R. Galt, K.J. Kearfott, E.B. Bartlett, T.L. Wilson, L.P. Heck, M.N. Clemenshaw, ``Interactive Image Correlation for SPECT Evaluation of The Cerebral Cortex Using MR Images To Reveal Finite Resolution Effects,'' Proceedings of the American Association of Physicists in Medicine, San Francisco, California, July 1991.
  • B.M. Kim and L.P. Heck, ``Automatic Design of Parallel Implementations of DSP Algorithms,'' Proc. of IEEE Intern. Conf. on Acoustics, Speech, and Signal Processing, Albuquerque, New Mexico, April 1990.
  • L.P. Heck, D.A. Schwartz, R.M. Mersereau, and J.H. McClellan, ``Symbolic Simplification of Digital Signal Processing Software,'' Proc. of IEEE Intern. Symp. on Circuits and Systems, Portland, Oregon, May 1989.


  1. "Method for establishing handset-dependent normalizing models for speaker recognition", US Patent #5950157, Sept. 1999 (with M. Weintraub)
  2. "Voice authentication system having cognitive recall mechanism for password verification,” US Patent #6671672, Dec. 2003.
  3. "Method and system for on-line unsupervised adaptation in speaker verification,” US Patent #6,804,647, October 2004 (with N. Mirghafori)
  4. "Method and system for improving robustness in a speech system ,” US Patent #7203652, April 2007.
  5. “Detection of Characteristics of Human-Machine Interactions for Dialog Customization and Analysis,” US Patent #7263489, August 2007 (with M.Cohen, J. Balogh, J. Riseman, N. Mirghafori)
  6. “Adapting a Context-Independent Relevance Function for Identifying Relevant Search Results,” US Patent Application, 2007.
  7. “Automated Learning from a Question and Answering Network of Humans,” US Patent #7809664, 2007.
  8. “Methods and Apparatuses for Adapting a Ranking Function of a Search Engine For Use with a Specific Domain,” US Patent #8126839, 2008 (with K. Chen, R. Lu, C.K. Wong, G. Sun, B. Tseng)
  9. “Maintaining State Transition Data for a Plurality of Users, Modeling, Detecting, and Predicting User States and Behavior", US Patent #7941383, 2011.
  10. "Deep Linking from Task List Based on Intent", US Patent App 13/401876, 2012 (with AK Sullivan, L Stifelman, K Lee, M Klee, G Tur, D Hakkani-Tur)
  11. "Contextual Query Adjustments Using Natural Action Input", US Patent App 13/549503, 2012 (with M Chinthakunta, R Iyer)
  12. "Task Driven User Intents", U.S. Patent No. 20,120,253,791, 2012 (with M Chinthakunta, D Mitby, L Stifelman)
  13. "Personalization of Queries, Conversations, and Searches" U.S. Patent No. 2012135791, 2012 (with M Chinthakunta, D Mitby, L Stifelman)
  14. "Combined Activation for Natural User Interface Systems", U.S. Patent No. 20,120,254,810, 2012 (with M Chinthakunta, D Mitby, L Stifelman)
  15. "Location-Based Conversational Understanding", U.S. Patent No. 2012135210, 2012 (with M Chinthakunta, D Mitby, L Stifelman)
  16. "Augmented Conversational Understanding Agent", U.S. Patent No. 20,120,253,788, 2012 (with M Chinthakunta, D Mitby, L Stifelman)
  17. "Augmented Conversational Understanding Architecture", U.S. Patent No. 20,120,254,227, 2012 (with M Chinthakunta, D Mitby, L Stifelman)
  18. "Conversational Dialog Learning and Correction", U.S. Patent No. 2012135229, 2012 (with M Chinthakunta, D Mitby, L Stifelman)
  19. "Sentence Simplification for Spoken Language Understanding", U.S. Patent No. 20,120,290,290, 2012 (with G Tur, D Hakkani-Tur, S Parthasarathy)
  20. "Exploiting Query Click Logs for Domain Detection in Spoken Language Understanding", U.S. Patent No. 20,120,290,293, 2012 (with D Hakkani-Tur, G Tur)
  21.  "Training Statistical Dialog Managers in Spoken Dialog Systems with Web Data", U.S. Patent No. 20,120,290,509, 2012 (with D Hakkani-Tur, R Iyer, G Tur)
  22. "Suggesting Intent Frame(s) for User Request(s)", US Patent No. 20,130,158,980, 2012 (with SJ Landry, AK Sullivan, L Stifelman, AD Elman, S Parthasarathy)
  23. "Translating Natural Language Utterances to Keyword Search Queries", US Patent Filed 2012 (with D Hakkani-Tur, G Tur, R Iyer)
  24. "Probability-Based State Modification for Query Dialogues", MS#336765.01 filed on 12/14/2012 (with Dilek Hakkani-Tür, Gokhan Tur, A Celikyilmaz, A Fidler)
  25. "Collaborative Learning through User Generated Knowledge", MS#337949.01 filed on 1/17/2013.
  26. "Prosodic and Lexical Addressee Detection", MS# 337999.01, filed on 1/31/2013. (with Elizabeth Shriberg, Andreas Stolcke, and Larry Heck)
  27. "Exploiting the Semantic Web for Unsupervised Spoken Language Understanding", MS# 338085.01, filed US Patent 2/18/2013 (with D Hakkani-Tur, G Tur)
  28. Exploiting the Semantic Web for Unsupervised Natural Language Semantic Parsing. MS# 337845.01, filed US Patent 2/27/2013 (with Gokhan Tur, Minwoo Jeong, and Ye-Yi Wang)
  29. "Using Human Wizards in A Conversational Understanding System", MS# 338086.01, filed US Patent 3/4/2013 (with Lisa Stifelman, Madhu Chinthakunta, and Larry Heck)
  30. "Satisfying Specified Intent(s) Based on Multimodal Request(s)", US Patent 20,130,159,001, 2013 (with L Stifelman, AK Sullivan, AD Elman, ST Tryphonas, KR Zargahi, KH Thai)
  31. "Unsupervised Relation Detection Model Training", filed US Patent, 2013 (with D Hakkani-Tur, G Tur) 
  32. "Active Learning Using Different Knowledge Sources", MS#339016.01, filed US Patent 6/2013 (with R Sarikaya, T Anastasakos, D Boies).
  33. "Environmentally Aware Dialog Policies and Response Generation", MS#339015.01, filed US Patent 6/2013 (with R Sarikaya, T Anastasakos, D Boies).
  34. "Building Conversational Understanding Systems Using a Toolset", MS#339005.01, filed US Patent 6/2013 (with R Sarikaya, T Anastasakos, D Boies).
  35. "A Deep Structured Semantic Model Produced Using Click-Through Data", MS#339669.01, filed with US Patent on 9/2013 (with X He, J Gao, A Acero, L Deng, P Huang).
  36. "Multi-user, Multi-domain Dialog System", MS#339845.01, filed US Patent 12/2013 (with D Hakkani-Tur, G Tur, L.P. Heck, D Wang).
  37. "Unsupervised Relation Detection Model Training, MS#339846.01, filed US Patent 12/2013 (with D Hakkani-Tur, G Tur).
  38. "Model-Based Approach for On-Screen Item Selection and Disambiguation", MS#340453.01, filed US Patent 3/2014 (with D Hakkani-Tur, G Tur, Z Feizollahi, A Celikyilmaz, R Sarikaya).
  39. "Discriminating Between Natural Language and Keyword Language Items", US Patent Filed, 2014 (with D Hakkani-Tur, A Celikyilmaz)
  40. "Techniques for Inferring Unknown Intents of Linguistic Items", MS#340356.01, filed US Patent 2/2014 (with Y Ji, D Hakkani-Tur, A Celikyilmaz, G Tur).
  41. "Transferring Information Across Language Understanding Model Domains", MS#340827.01, filed US Patent 4/2014 (with Derek Liu, Ruhi Sarikaya, Dilek Hakkani-Tur, Gokhan Tur)
  42. "Facilitating Conversations with Automated Location Mapping", MS#340001.01", filed 5/2014 (with L Stifelman, M Chinthakunta, J Odell, D Dole).
  43. "Knowledge Source Personalization to Improve Language Models", MS #340646.01, filed 5/2014 (with M Akbacak, G Tur, B Dumoulin, D Hakkani-Tur).
  44. "Language Modeling for Conversational Understanding Domains Using Semantic Web Resources", MS#340623.01, filed 5/2014 (with M Akbacak, G Tur, B Dumoulin, D Hakkani-Tur).
  45. "Session Context Modeling for Language Models in Conversational Understanding Systems using Knowledge Graphs and User Session Logs", MS #340625.01, filed 4/2014, (with M Akbacak, G Tur, D Liu, D Hakkani-Tur).
  46. "Deep Learning for Semantic Parsing Including Semantic Utterance Classification", MS #340534.01, filed 4/2014 (with Y Dauphin, G Tur, D Hakkani-Tur)