Share on Facebook Tweet on Twitter Share on LinkedIn Share by email
Learning to Understand Natural Language in Physically-Grounded Environments

Speaker  Jayant Krishnamurthy

Affiliation  Carnegie Mellon University

Host  Hoifung Poon

Duration  01:13:54

Date recorded  16 October 2013

We all want computers to understand natural language. Whether it be to command a robot, or answer a question by reading the web, language understanding is a fundamental problem for natural language processing. Physically-grounded settings are an important special case of this problem, with applications in robotics and interactions with embodied systems.

This talk presents Logical Semantics with Perception (LSP), a model for understanding natural language statements within a physically-grounded environment. For example, given an image, LSP can map a description such as "the blue mug to the left of the monitor," to a set of image segments containing blue mugs left of monitors. Importantly, LSP can be trained directly from natural language / object pairs, which is a natural form of supervision that can be easily obtained from human interaction. I will present experiments applying LSP to several domains, including image understanding (using Microsoft Kinect data) and geographical question answering.

The talk may also include some additional related work on semantic parsing at web-scale for information extraction and question answering. (time permitting)

©2013 Microsoft Corporation. All rights reserved.
> Learning to Understand Natural Language in Physically-Grounded Environments