Hierarchical Representations for Learning and Inferring Office Activity from Multiple Sensory Channels

We present the use of hierarchical probabalistic representations for sensing, learning, and doing inference from multiple sensory streams at multiple levels of temporal granularity and abstraction. The methods show robustness to subtle changes in the environment and enable adaptation with minimal retraining. The approach centers on the use of a Hierarchy of Hidden Markov Models (HHMMs), whose parameters are learned from data. We have found that HHMMs are a promising means for making inferences about context and activity from perceptual signals. After presenting key properties of the representation, we review experiments with HHMMs applied in an office-awareness real-time multimodal prototype.