From multimodal to natural interactions

  • Kuansan Wang

W3C Workshop on Multimodal Interaction |

Since the pointing device was introduced alongside with the keyboard to the graphical user interface (GUI), users have been interacting with computers in a multimodal fashion: the pointing device enables the user to point and click on icons that are given specific meanings by the application developers, while the keyboard allows the users to enter more unconstrained text. Despite the successful adoption of GUI, computers are still regarded as too difficult to use for the mass. Although the graphical nature offers great flexibility for innovative presentation, GUI still puts the burden on users to discover how to express their intents by interacting with the graphical objects in a way foreseen by the application designers. As the computer becomes feature rich to meet the ever increasing and diverse needs, feature discoverability becomes a critical issue to improve usability. Users are frustrated when the computer cannot fulfill their wishes. Quite often, it is not the application does not offer the capabilities the user wanted, but users have hard time discovering and remembering how to use them.