Microsoft Research
Computational User Experiences

SketchInsight: Natural Data Exploration and Presentation through Sketch-based Interaction

Microsoft continues to innovate in the NUI space, continuously providing new modalities for interacting with our computing systems. We are leveraging our expertise in NUI and developing a radically new way for people to create information visualizations and interact with their data through simple interactive sketches. We aim to make manipulating and analyzing data as easy as creating napkin sketches, and much more powerful.

We assert that removing some of the interaction friction present in traditional desktop interfaces enables users to more effectively explore, understand, and communicate data. We choose sketch as the grounding modality because we believe that sketching itself promotes thinking, insight, and inspiration. Such a system will enable levels of interactive analysis not previously possible, individually or collaboratively, which is critical as our data sources continue to get bigger and more complex.


Project Team


SketchInsight Video



Even though communication of insights is one of the main goals of data exploration, presenting visual information to other people is underexplored. To create a new and engaging form of storytelling with data, we explore how to leverage and extend the narrative storytelling attributes of whiteboard animation with interactive information visualization techniques. SketchStory is an interactive whiteboard system for storytelling with data through real-time sketching. It facilitates the creation of personalized and expressive data charts quickly and easily, where the presenter sketches example icon and SketchStory automatically completes the chart by synthesizing from example sketches. Furthermore, SketchInsight allows the presenter to interact with the data charts.


SketchStory Video for InfoVis 2013 [.wmv (43MB)]

Understanding Pen and Touch Interaction for Data Exploration on Interactive Whiteboards


Current interfaces for common information visualizations such as bar graphs, line graphs, and scatterplots usually make use of the WIMP (Windows, Icons, Menus and a Pointer) interface paradigm with its frequently discussed problems of multiple levels of indirection via cascading menus, dialog boxes, and control panels. Recent advances in interface capabilities such as the availability of pen and touch interaction challenge us to re-think this and investigate more direct access to both the visualizations and the data they portray. We conducted a Wizard of Oz study to explore applying pen and touch interaction to the creation of information visualization interfaces on interactive whiteboards without implementing a plethora of recognizers. Our wizard acted as a robust and flexible pen and touch recognizer, giving participants maximum freedom in how they interacted with the system. Based on our qualitative analysis of the interactions our participants used, we found several insights about pen and touch interactions in the context of learnability and the interplay between pen and touch gestures.



When faced with the task of understanding complex data, it is common for people to work on whiteboards, where they can collaborate with others, brainstorm lists of important questions, and sketch simple visualizations. However, these sketched visualizations seldom contain real data. We address this gap by extending these sketched whiteboard visualizations with the actual data to be analyzed. Guided by an iterative design process, we developed a better understanding of the challenges involved in bringing sketchbased interaction to data analysis. In this work we contribute insights into the design challenges of sketch-based charting, and we present SketchVis, a system that leverages hand-drawn input for exploring data through simple charts.


SketchVis Video for ITS 2011 [.wmv (1.9MB)]


SketchStory: Telling More Engaging Stories with Data through Freeform Sketching

Bongshin Lee, Rubaiat Habib Kazi, Greg Smith

IEEE TCVG (InfoVis 2013), Vol. 19, No. 2, pp. 2416-2425, 2013

Data Visualization on Interactive Surfaces: A Research Agenda from the DEXIS Workshop

Petra Isenberg, Tobias Isenberg, Tobias Hesselmann, Bongshin Lee, Ulrich von Zadow, Anthony Tang

IEEE CG&A, Vol. 33, No. 2, pp. 16-24, 2013 (Visualization Viewpoints)

Beyond Mouse and Keyboard: Expanding Design Considerations for Information Visualization Interactions

Bongshin Lee, Petra Isenberg, Nathalie Henry Riche, Sheelagh Carpendale

IEEE TVCG (InfoVis 2012), Vol. 18, No. 12, pp. 2689-2698, 2012

Understanding Pen and Touch Interaction for Data Exploration on Interactive Whiteboards

Jagoda Walny, Bongshin Lee, Paul Johns, Nathalie Henry Riche, Sheelagh Carpendale

IEEE TVCG (InfoVis 2012), Vol. 18, No. 12, pp. 2779-2788, 2012

Data Analysis on Interactive Whiteboards through Sketch-based Interaction

Jeffrey Browne, Bongshin Lee, Sheelagh Carpendale, Nathalie Riche, Timothy Sherwood

Proceedings of ITS 2011, pp. 154-157, 2011

Contact Us Terms of Use Trademarks Privacy Statement ©2010 Microsoft Corporation. All rights reserved.Microsoft