Share on Facebook Tweet on Twitter Share on LinkedIn Share by email
Playing Machines: Machine Learning Applications in Computer Games

ICML 2008 Tutorial - 5 July 2008, Helsinki, Finland This tutorial will explore the exciting research area of applying machine learning to computer games.

Abstract

The tutorial will give an introduction to the emerging area of applying machine learning to computer games and of using computer games as test beds for machine learning. One of the most important challenges in computer games is the creation of agents driven by artificial intelligence (AI) that interact with the player in believable and entertaining ways. As a consequence a substantial part of the tutorial will consider adaptive and learning game AI based on supervised and reinforcement learning. However, computer games also offer a great variety of other challenges including problems in graphics, sound, networking, player rating and matchmaking, interface design, narrative generation etc. Selected problems from some of these areas will be discussed together with machine learning approaches to solve them. Since this is an application area, the tutorial will focus on past and recent applications, open problems and promising avenues for future research. It will also provide resources available to people who would like to work in this fascinating and fun research space.

Background

Let us begin with a provocative question: In which area of human life is artificial intelligence currently applied the most? The answer, by a large margin, is computer games. This is essentially the only big area in which people interact with AI controlled agents on a regular basis. And the market for video games is growing, with sales in 2007 of $17.94 billion marking a 43% increase over 2006. However, growth is not only in sales but also in the diversity of content offered, ranging from educational games to first-person shooters.

So, what do games offer to machine learning researchers? From a research point of view, video games offer fascinating toy examples that capture the complexity of real-world situations while maintaining the controllability and traceability of computer simulations. As an example, consider the problem of driving a racing car under realistic race conditions. While the full problem is too complex and expensive to be tackled right now because it involves problems around limited actuators and noisy sensors in addition to the AI problem, important aspects can be tackled working inside a state-of-the-art racing game simulation.

One of the key problems in computer games is the creation of AI driven agents that interact with the player so as to create a great interactive gaming experience. These agents can take a variety of roles such as player’s opponents, teammates or other non-player characters. The framework of reinforcement learning , and in particular, multi-agent reinforcement learning is very well suited to this kind of task and we anticipate that people from the RL community will have a lot to contribute to the field of computer games once the wealth of opportunities in this space has been understood.

In addition, computer games offer a great variety of other challenges including problems in graphics, sound, networking, player rating and matchmaking, interface design, narrative generation etc. All of these areas would benefit from various machine learning techniques that may include supervised, unsupervised and other learning paradigms.

Finally, the holy grail of an interaction between computer games and machine learning might be the emergence of a new genre of games that has adaptive agents at the core of its game play. Games like Creatures and Black & White have attempted to build entire games around the concept of teaching behaviour to adaptive AI agents. It is an exciting question to ask if these ideas could be taken further to put machine learning at the centre of computer games.

Topics

  • History of computer games with emphasis on AI and machine learning. Here we will also discuss how other aspects of the game like sound, graphics, physics and network play affect the role of AI and pose new challenges.
  • Overview of machine learning frameworks ranging from the general multi-agent framework of Stochastic Games to the more familiar notions of Markov Decision Processes. This section will include different learning paradigms and how they are or may be applied to computer games.
  • Example applications of machine learning in commercial games including a discussion of games that shipped with a machine learning component such as Black & White, Colin McRae Rally 2.0, and Forza Motorsport.
  • Machine learning projects using computer games as test beds including reinforcement learning in Tao Feng and Project Gotham Racing 3, Bayes nets for Quake bots and machine learning approaches in RoboCup.
  • Machine learning approaches to other aspects of games including a discussion of the Bayesian skill ranking system TrueSkill™, predictive algorithms for network latency mitigation, and dimensionality reduction for animation control based on motion capture data. 
  • Pointers to resources to get started including available code bases for simulations and games, game programming frameworks, papers, competitions and books.
  • Presenters

    Ralf Herbrich and Thore Graepel together lead the Applied Games (APG) group at Microsoft Research Cambridge. The group’s mission is to apply machine learning to games. These include both recreational games such as Go, poker, and video games as well as abstract decision games such as auctions and negotiations played in the real world. The group developed the TrueSkill™ system used millions of times each day in Microsoft’s Xbox Live service for ranking and matchmaking players according to their skills and which is also the backbone of the ranking system of the blockbuster game Halo 3. APG also developed the Drivatar™ AI for the Xbox title Forza Motorsport™.

    Thore Graepel studied physics in Hamburg, London and Berlin. He received both his diploma in physics and his PhD in statistics from the Technical University of Berlin. He worked on machine learning and large scale optimisation at the Swiss Federal Institute of Technology Zurich and on kernel methods and learning theory at Royal Holloway, University of London. Since 2002 he has been a researcher in the Machine Learning and Perception group at Microsoft Research Cambridge where he co-founded the Applied Games group in 2006. Thore has published over 50 technical papers on topics around machine learning including probabilistic models, statistical learning theory, kernel methods, clustering, stochastic optimisation and reinforcement learning with applications including computer go, video games, computer networks and ranking.

    Ralf Herbrich studied computer science in Berlin. He received both his diploma in computer science and his PhD in statistics from the Technical University of Berlin. He worked on machine learning, kernel methods and learning theory during his Post-doc time at Microsoft Research. Since 2001 he has been a researcher in the Machine Learning and Perception group at Microsoft Research Cambridge where he co-founded the Applied Games group in 2006. Ralf has published over 50 technical papers on topics around machine learning including probabilistic models, statistical learning theory, kernel methods, and reinforcement learning with applications including computer Go, video games, computer networks and ranking. He also published a book on “Learning Kernel Classifiers” (MIT Press) in 2001.

    People