Abstract

The tutorial will give an introduction to the emerging area of applying machine learning to computer games and of using computer games as test beds for machine learning. One of the most important challenges in computer games is the creation of agents driven by artificial intelligence (AI) that interact with the player in believable and entertaining ways. As a consequence a substantial part of the tutorial will consider adaptive and learning game AI based on supervised and reinforcement learning. However, computer games also offer a great variety of other challenges including problems in graphics, sound, networking, player rating and matchmaking, interface design, narrative generation etc. Selected problems from some of these areas will be discussed together with machine learning approaches to solve them. Since this is an application area, the tutorial will focus on past and recent applications, open problems and promising avenues for future research. It will also provide resources available to people who would like to work in this fascinating and fun research space.

Background

Let us begin with a provocative question: In which area of human life is artificial intelligence currently applied the most? The answer, by a large margin, is computer games. This is essentially the only big area in which people interact with AI controlled agents on a regular basis. And the market for video games is growing, with sales in 2007 of $17.94 billion marking a 43% increase over 2006. However, growth is not only in sales but also in the diversity of content offered, ranging from educational games to first-person shooters.

So, what do games offer to machine learning researchers? From a research point of view, video games offer fascinating toy examples that capture the complexity of real-world situations while maintaining the controllability and traceability of computer simulations. As an example, consider the problem of driving a racing car under realistic race conditions. While the full problem is too complex and expensive to be tackled right now because it involves problems around limited actuators and noisy sensors in addition to the AI problem, important aspects can be tackled working inside a state-of-the-art racing game simulation.

One of the key problems in computer games is the creation of AI driven agents that interact with the player so as to create a great interactive gaming experience. These agents can take a variety of roles such as player’s opponents, teammates or other non-player characters. The framework of reinforcement learning , and in particular, multi-agent reinforcement learning is very well suited to this kind of task and we anticipate that people from the RL community will have a lot to contribute to the field of computer games once the wealth of opportunities in this space has been understood.

In addition, computer games offer a great variety of other challenges including problems in graphics, sound, networking, player rating and matchmaking, interface design, narrative generation etc. All of these areas would benefit from various machine learning techniques that may include supervised, unsupervised and other learning paradigms.

Finally, the holy grail of an interaction between computer games and machine learning might be the emergence of a new genre of games that has adaptive agents at the core of its game play. Games like Creatures and Black & White have attempted to build entire games around the concept of teaching behaviour to adaptive AI agents. It is an exciting question to ask if these ideas could be taken further to put machine learning at the centre of computer games.

Topics

Material

• Slide Deck (PowerPoint 2007)
• Videos
  • Tao Feng - Repetetive Beahviour
  • Tao Feng - Dancing Behaviour
  • Gran Turismo - Racing Chaing
  • Gran Turismo - Stubborn Racing
  • Q Learning (Choose Action, Get Reward)
  • Tao Feng - Exile Intro, Wulong Intro
  • Tao Feng - Early Training Agressive Reward, Late Training Agressive Reward
  • Tao Feng- Early Training Aikido Reward, Late Training Aikido Reward
  • Halo 2 - Jumping and Walking Based on Motion Capture
  • Learning to Walk - Hopper Training, Hopper Trained
  • Learning to Walk - Biped Training, Biped Trained
  • Quake - Bayesian Programming
  • TORCS - WCCI 2008 Final Race

Presenters

Ralf Herbrich and Thore Graepel together lead the Applied Games (APG) group at Microsoft Research Cambridge. The group’s mission is to apply machine learning to games. These include both recreational games such as Go, poker, and video games as well as abstract decision games such as auctions and negotiations played in the real world. The group developed the TrueSkill™ system used millions of times each day in Microsoft’s Xbox Live service for ranking and matchmaking players according to their skills and which is also the backbone of the ranking system of the blockbuster game Halo 3. APG also developed the Drivatar™ AI for the Xbox title Forza Motorsport™.

Thore Graepel studied physics in Hamburg, London and Berlin. He received both his diploma in physics and his PhD in statistics from the Technical University of Berlin. He worked on machine learning and large scale optimisation at the Swiss Federal Institute of Technology Zurich and on kernel methods and learning theory at Royal Holloway, University of London. Since 2002 he has been a researcher in the Machine Learning and Perception group at Microsoft Research Cambridge where he co-founded the Applied Games group in 2006. Thore has published over 50 technical papers on topics around machine learning including probabilistic models, statistical learning theory, kernel methods, clustering, stochastic optimisation and reinforcement learning with applications including computer go, video games, computer networks and ranking.

Ralf Herbrich studied computer science in Berlin. He received both his diploma in computer science and his PhD in statistics from the Technical University of Berlin. He worked on machine learning, kernel methods and learning theory during his Post-doc time at Microsoft Research. Since 2001 he has been a researcher in the Machine Learning and Perception group at Microsoft Research Cambridge where he co-founded the Applied Games group in 2006. Ralf has published over 50 technical papers on topics around machine learning including probabilistic models, statistical learning theory, kernel methods, and reinforcement learning with applications including computer Go, video games, computer networks and ranking. He also published a book on “Learning Kernel Classifiers” (MIT Press) in 2001.