Microsoft Research
Computational User Experiences

AnatOnMe: Facilitating Doctor-Patient Communication Using a Projection-Based Handheld Device

Patient non-compliance with medical treatment plans for chronic conditions is estimated at between 30-50 percent, presenting a serious threat to successful medical outcomes. Effective doctor-patient communication has been identified as one of the most important factors to increasing compliance. We set out to understand what kinds of computing systems might positively enhance such communication and what functions they would need to support. We talked to several doctors from a variety of specialties to understand general communication challenges and design requirements. Then, focusing on physical therapy as a particularly appropriate sub-specialty, we built and studied a handheld projection system that flexibly supports the key aspects of information exchange: injury assessment via photo-capture, annotation, and re-projection for shared discussion; education via on-body projection of anatomy imagery, and documentation of personalized therapeutic exercises via photo capture and the generation of personalized take-home instructions. An expert review and formal lab study established that both physical therapists and patients perceive that the system provides high value and a more engaging and informative experience than what is available today.


Project Team

Tao Ni


Amy Karlson


Daniel Wigdor






AnatOnMe in use during a physical therapy consultation

To explain the underlying cause of an injury, a physical therpaist can use the AnatOnMe system to project relevant anatomy imagery directon onto the site of injury on the patient. A pointing device held in the dominant hand can be used to point at important locations on the image, as well as to draw or annotate directly over the projected image. At any point the therapist can take a picture of the image and any annotations, so that the patient can review and recall the educational discussion later.

Device Components

The AnatOnMe system consists of a handheld projector/camera unit (left) for image projection/capture and laser tracking, and a pointing device (right) used for camera functions, menu interaction, pointing, and image annotation.

Surface Tradoffs

To help guide the operational use of AnatOnMe, we conducted a formal lab study to evaluate patient learning and preference tradoffs when educational material was taught on their own body (A), a physical model of a human (B), or a nearby wall (C). The body was preferred by a majority of participants for being the most engaging of the three surfaces, the best for helping the participant understand the medical condition, and the best for understanding the precise location of anatomy on their own body. Projection on the model was also well liked, as it allowed participants to see the material from multiple angles.


AnatOnMe: Facilitating Doctor-Patient Communication Using a Projection-Based Handheld Device

Tao Ni, Amy K. Karlson, Daniel Wigdor

Proceedings of ACM CHI 2011, May 2011



Contact Amy Karlson for questions about our work in this area.

Contact Us Terms of Use Trademarks Privacy Statement ©2010 Microsoft Corporation. All rights reserved.Microsoft