Spoken Language Support for Software Development

Many programmers who suffer from repetitive strain injuries (RSI) and other more severe motor impairments have difficulty staying productive in a work environment that all but requires long hours typing code into a computer. My dissertation work helps to lower these productivity barriers by enabling developers to use speech to reduce their dependence on typing. Many disabled programmers already try to use voice recognition with existing programming environments. However, commercial voice recognition tools do not map well onto programming tasks because they are designed for natural languages. Our research adapts these tools for software development by exploiting the domain-specific nature of programming to constrain the domain of recognition. By applying programming language-based analyses to a verbalized form of authoring, editing and navigating through code, we hope to alleviate many of the common problems of voice recognition for programmers.

Our solution consists of Spoken Java, a semantically isomorphic dialect of Java that is more naturally verbalized by human developers, and a command and control speech grammar designed to enable programmers to find pieces of code and modify them in high- level linguistic ways. Spoken Java is considerably more lexically and syntactically ambiguous than a traditional programming language because verbalized programs contain homophones (words that sound alike but are spelled differently), unpronounceable words, abbreviated words, and inconsistently verbalized punctuation. The major technical challenge in our work is to resolve this new input form’s ambiguities using new algorithms for interacting lexical, syntactic, semantic, and program-specific analyses. The main artifact of our work is a software development system that can understand spoken program authoring, editing and navigation. Our evaluation employs user studies of programmers composing and editing code to learn how well the system performs, and understand how well software developers adapt to voice-based programming. In addition, our software can be used as a platform to learn about more natural modes of interaction than those found in traditional programming environments.