Fine Grained Permissions for Augmented Reality Applications

Augmented reality (AR) applications sense the environment, then create virtual objects overlaid on human senses. Examples include games such as Kinect Adventures! and tablet applications that show virtual furniture in the context of a real room. No current OS has special support for such applications, so they must perform all sensing and object recognition themselves. As a result, permissions for AR applications are necessarily coarse grained: applications must ask for access to raw sensor feeds, such as video and audio. Unfortunately, these raw feeds expose significant additional information over and above what applications need, such as location, identity of the user, or sensitive information written on documents.

To meet the needs of AR applications, we introduce a new OS abstraction: the recognizer. A recognizer takes as input raw sensor streams, then outputs higher-level objects, such as a skeleton or a face. We introduce a novel fine-grained permission system where AR applications can obtain the output of recognizers, but do not see the raw sensor data. We tackle the problem of permissions with noisy data, as recognizers may have false positives. We introduce a trusted OS component for managing noisy data in a prototype using the Kinect for Windows SDK. We show how recognizers in the OS eliminate duplication of heavyweight object recognition across applications, and we quantify the resulting performance improvement. Finally, we introduce privacy goggles, which allows users to inspect what data the application can see, both at install time and afterwards. We show that a small set of recognizers are sufficient to support all shipping Kinect applications, and we quantify users’ privacy attitudes toward the output of these recognizers.