A selection of systems that weave language-learning experiences into the fabric of our everyday lives by leveraging our natural abilities and interests. These systems include a Windows Phone 7 game for learning Mandarin sounds, a Microsoft Kinect game for learning spatial language, and web flashcards that integrate skill-based microgames. Each represents part of a vision in which personalized learning services orchestrate connected learning experiences across devices, activities, and contexts.
Mastering a second language is a rewarding and lifelong activity. It can help you to think in new ways, experience new cultures, and communicate with new people. For many, it is also critical for their livelihood or career progression. However, the process of learning a second language is hard, especially for people struggling to find time to study in the face of competing demands from school or work, as well as family and social life.
The research challenge is to find new ways of motivating people to spend time learning their chosen second language, as well as more effective ways to make use of this time. In the Human Computer Interaction group at Microsoft Research Asia, we want to create learning experiences that are personalized to learners’ abilities, interests, and lifestyles. We are especially interested in how microlearning – learning in fragments of free time throughout the day – can provide a more convenient, contextual, and learner-driven alternative to traditional classroom teaching.
Our project on language-learning games uses explicit game-goals and game-mechanics to motivate gameplay that is closely aligned with language-learning objectives. In each of the three games we have created, gameplay is broken down into fast (and sometimes furious!) rounds of microlearning that can be repeated as desired to fill any time available. Together, they represent a vision of how language-learning technologies can put the learner first, developing his or her language knowledge and skills through games that engage the full range of auditory, kinesthetic, and visual senses.
Tip Tap Tones: Mobile Game for Learning Mandarin Sounds
For language learners, the main difficulty in understanding speech is the perception of words. This difficulty is especially pronounced for learners of tonal languages such as Mandarin Chinese, in which the meaning of a sound is inferred from both its changing pitch and its context of use. Since most people who learn Mandarin do so from non-tonal native languages and lack the vocabulary to work out what words mean based on how they are used, they often find tones the most confusing and frustrating aspect of acquiring the language.
To address the challenge of mastering the Mandarin sound system, we have developed Tip Tap Tones: a Windows Phone 7 game that helps language learners to identify the sounds and tones of Mandarin Chinese in a way that is fast, fun, and effective. In the game, the learner must listen to a Mandarin sound before tapping the button representing the tone and syllable they thought they heard. The game begins with just a single syllable and four tone options, but as the learner progresses by giving correct responses, the sounds get faster and the number of possible syllables increases. Initially, this is to two syllables that are easily confusable, but on the final game screen a full set of four confusable syllables and four (similarly confusable) tones presents the greatest challenge. You have 60 seconds to complete as many game screens as you can by giving fast and accurate responses, earning greater scores as your skills improve.
Overall, Tip Tap Tones represents a new way to “sharpen your ears” and retrain your brain to identify foreign sounds, in a game format that is challenging and enjoyable for learners of all levels. It is available free in the Windows Phone marketplace.
SpatialEase: Kinect Game for Learning the Language of Space
Learning words in a second language is hard, but mastering grammatical constructions that require a change in native language thought patterns is even harder. For example, in the case of native English speakers learning Chinese, the command “move your left hand to the right” can translate to a word order of “to the right, move your left hand”. Understanding the language of space and motion in real-time is a critical skill that we develop as children through repeated interactions with others, but similar opportunities for “learning by doing” are rare for second-language learners.
To support “learning by doing” that helps learners to process spatial language in real-time, we have created SpatialEase: a Microsoft Kinect game for language learning through body motion. In the game, the learner sees and hears second-language commands that they must follow by moving his or her body in space. If the learner does not move or moves incorrectly, the game draws arrows showing how they should move along with the native-language translation of the command. This unsuccessful command is then scheduled for future tests, extending the number of commands that must be followed to complete the game, all while the remaining game time is running out.
Overall, SpatialEase builds on the theories of embodied cognition (that interacting through our bodies affects how we think) and kinesthetic learning (that moving our bodies can express what we think), using repeated bodily interactions to create rich interconnections between second-language structures and learner thought-patterns.
Polyword Flashcards: Web App for Developing Vocabulary Skills
Language-learning games can motivate learners to spend time engaging with the language, but only for as long as the game content remains relevant. Similarly, games based on fixed language items might help learners to master those items, but ultimately they will be forgotten if they aren’t reviewed at appropriate time intervals. New approaches are needed to translate short-term gains from gaming into long-term retention and real-world use.
To always keep language-learning games fresh and relevant while supporting long-term language retention, we have created Polyword Flashcards. These mobile flashcards ask the learner to recall the translations of vocabulary they have chosen to learn, with the adaptive scheduling of future tests based on the learner’s history of responses. Over time, this adaptive flashcard algorithm aims to converge on a consistently high recall success rate even when the learner is restricted to short, sparse sessions of microlearning. Each flashcard also contains a set of badges representing language skills, each of which links to a microgame that tests the skill for both that vocabulary item and other vocabulary items in need of review. These microgames are modeled on the Tip Tap Tones system described above as a way to develop key language skills in short bursts of addictive gameplay.
Overall, Polyword Flashcards uses the vocabulary you are studying with flashcards to populate microgames with language content, and the mechanics of microgames themselves to suggest new vocabulary to learn. This combined approach to the development of language knowledge and skills helps learners direct their own learning with experiences that match their abilities, interests, and lifestyles.
Related Systems & Services
Both in our work and the work of Microsoft Research and Microsoft as a whole, there is a strong desire to build technologies that bridge languages and cultures in ways that help people worldwide to communicate with and learn from one another.
We developed the MicroMandarin system to help investigate how mobile phones can support microlearning of language where it is used in the real-world. For example, in a Chinese restaurant our system would help the learner remember contextually-relevant words like “soy sauce” and “chopsticks”, as well as culturally-appropriate phrases for key actions like asking for the menu and the check. More details can be found in the paper:
MicroMandarin: Mobile Language Learning in Context (2011). Darren Edge, Elly Searle, Kevin Chiu, Jing Zhao & James A. Landay. In proceedings of the CHI 2011 Conference on Human Factors in Information Systems.
In Polyword Flashcards, vocabulary definitions are downloaded using the AJAX API of the Bing Dictionary web application.
The adaptive flashcard algorithm in Polyword Flashcards is also currently driving the flashcards component of Bing Dictionary Desktop (in Chinese only).
Microsoft Lync 2010 Conversation Translator allows you to send and receive instant messages with Microsoft Lync using the Microsoft Bing translation service. Bilingual users can provide translation feedback to help improve the service.
Microsoft Translator Hub
Microsoft Translator Hub is an extension of the Microsoft Translator platform and service that brings better and specialized translation quality to estabilished languages as well as native languages that have not previously been well supported. The service helps you to build a superior translation system within a private website by combining your translated documents with the power of Microsoft Translator’s big data back end.
For more information about the Language Learning Games project, contact Darren Edge via http://research.microsoft.com/en-us/people/daedge/. For more information about the Human Computer Interaction group at Microsoft Research Asia, see our website at http://research.microsoft.com/hci/.