The Conversational Architectures project aims to build natural, human-like systems and interfaces for engaging in conversation with a computer.  This involves exploring not only various theoretical frameworks for supporting human-computer dialog, but also computational architectures that apply these frameworks.  To better equip computers for conversation, we have endowed systems with inferential and sensory capabilities that give them the ability to see what the user is paying attention to, recognize words, grammatically parse utterances, and even, recover in a natural, human-like manner from misunderstandings when they occur.  To implement these capabilities, we have integrated leading-edge components in vision technology, speech recognition, text-to-speech generation, and natural language parsing, along with advanced machine learning techniques such as Bayesian inference for optimizing decisions over time.

Our efforts include research on:

• Multi-modal reasoning: Combining probabilities from multiple sources of information such as vision and speech.
• Intention recognition: Inferring the intentions of the user from an utterance or action.
• Dialog repair: Anticipating and recovering from misunderstanding in conversation through grounding.
• Identifying speech targets: Distinguishing when speech is directed towards the system or is simply overheard.

Further developments are underway.  We have applied these capabilities to handling receptionist tasks, navigating Microsoft PowerPoint, and directing clarification dialog in command-and-control speech recognition.  For more information, see our Systems web page.

 Bayesian Receptionist Revisited

Lately, we have been refurbishing the Bayesian Receptionist, a system designed to perform tasks typically handled by front desk receptionists at the Microsoft corporate campus.  The Bayesian Receptionist serves two purposes: first, to provide receptionist services to users, and second, to provide a testing environment for new research developments.  A prior version of the Bayesian Receptionist required users to manually input contextual observations wherever available.  The new version attempts to automate the evidence gathering process and to integrate multi-modal information through the Quartet system.

 Wizard-of-Oz Studies

To get an idea of how users interact with an automated system, and to build up a corpus of training data for computational modeling, we have been collecting spoken dialogs between users and a mockup system.  Using a synthetic voice to maintain the illusion of communicating with a computer, a human "wizard" assists users in accomplishing various personal information management (PIM) tasks.

 Demos

• Quartet and DeepListener at ACM1: Beyond Cyberspace, San Jose Conference Center, March 10 - 14, 2001.
• Microsoft TechFest, Redmond campus, January 22, 2001. See internal videoclips.