Learning to Understand Natural Language in Physically-Grounded Environments
We all want computers to understand natural language. Whether it be to command a robot, or answer a question by reading the web, language understanding is a fundamental problem for natural language processing. Physically-grounded settings are an important special case of this problem, with applications in robotics and interactions with embodied systems.
This talk presents Logical Semantics with Perception (LSP), a model for understanding natural language statements within a physically-grounded environment. For example, given an image, LSP can map a description such as “the blue mug to the left of the monitor,” to a set of image segments containing blue mugs left of monitors. Importantly, LSP can be trained directly from natural language / object pairs, which is a natural form of supervision that can be easily obtained from human interaction. I will present experiments applying LSP to several domains, including image understanding (using Microsoft Kinect data) and geographical question answering.
The talk may also include some additional related work on semantic parsing at web-scale for information extraction and question answering. (time permitting)
Speaker Details
Jayant Krishnamurthy is a Ph.D. student in the Computer Science Department at Carnegie Mellon University. Prior to attending Carnegie Mellon, he received an M.Eng and S.B. from the Massachusetts Institute of Technology. Jayant’s research is on machine learning and natural language processing, with a focus on understanding the semantics of natural language. His work is part of the Never-Ending Language Learner (NELL) project at Carnegie Mellon, directed by Tom Mitchell.
- Series:
- Microsoft Research Talks
- Date:
- Speakers:
- Jayant Krishnamurthy
- Affiliation:
- Carnegie Mellon University
-
-
Jeff Running
-
Series: Microsoft Research Talks
-
-
-
-
Galea: The Bridge Between Mixed Reality and Neurotechnology
Speakers:- Eva Esteban,
- Conor Russomanno
-
Current and Future Application of BCIs
Speakers:- Christoph Guger
-
Challenges in Evolving a Successful Database Product (SQL Server) to a Cloud Service (SQL Azure)
Speakers:- Hanuma Kodavalla,
- Phil Bernstein
-
Improving text prediction accuracy using neurophysiology
Speakers:- Sophia Mehdizadeh
-
-
DIABLo: a Deep Individual-Agnostic Binaural Localizer
Speakers:- Shoken Kaneko
-
-
Recent Efforts Towards Efficient And Scalable Neural Waveform Coding
Speakers:- Kai Zhen
-
-
Audio-based Toxic Language Detection
Speakers:- Midia Yousefi
-
-
From SqueezeNet to SqueezeBERT: Developing Efficient Deep Neural Networks
Speakers:- Sujeeth Bharadwaj
-
Hope Speech and Help Speech: Surfacing Positivity Amidst Hate
Speakers:- Monojit Choudhury
-
-
-
-
-
'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo Project
Speakers:- Peter Clark
-
Checkpointing the Un-checkpointable: the Split-Process Approach for MPI and Formal Verification
Speakers:- Gene Cooperman
-
Learning Structured Models for Safe Robot Control
Speakers:- Ashish Kapoor
-
-