 |
Playing
Machines:
Machine Learning Applications in Computer Games
ICML 2008 Tutorial - 5 July 2008, Helsinki, Finland
This tutorial will explore the exciting research area of applying
machine learning to computer games.
|
|
Abstract
The tutorial will give an introduction to the emerging area of
applying machine learning to computer games and of using computer
games as test beds for machine learning. One of the most important
challenges in
computer games is the creation of agents driven by artificial
intelligence (AI) that interact
with the player in believable and entertaining ways. As a consequence a substantial part of the tutorial will
consider adaptive and learning game AI based on supervised and
reinforcement learning. However, computer games also offer a great
variety of other challenges including problems in graphics, sound,
networking, player rating and matchmaking, interface design,
narrative generation etc. Selected problems from some of these areas
will be discussed together with machine learning approaches to solve
them. Since this is an application area, the tutorial will focus on
past and recent applications, open problems and promising avenues
for future research. It will also provide resources available to
people who would like to work in this fascinating and fun research
space.
|
|
Background
Let us begin with a provocative question: In which area of human
life is artificial intelligence currently applied the most? The
answer, by a large margin, is computer games. This is essentially
the only big area in which people interact with AI controlled agents on
a regular basis. And the market for video games is growing, with
sales in 2007 of $17.94 billion marking a 43% increase over 2006.
However, growth is not only in sales but also in the diversity of
content offered, ranging from educational games to first-person
shooters.
So, what do games offer to machine learning researchers? From a research
point of view, video games offer fascinating toy examples that
capture the complexity of real-world situations while maintaining
the controllability and traceability of computer simulations. As an
example, consider the problem of driving a racing car under
realistic race conditions. While the full problem is too complex and
expensive to
be tackled right now because it involves problems around limited
actuators and noisy sensors in addition to the AI problem, important
aspects can be tackled working inside a state-of-the-art racing game
simulation.
One of the key problems in computer games is the creation of AI
driven agents that interact with the player so as to create a great
interactive gaming experience. These agents can take a variety of
roles such as player’s opponents, teammates or other non-player
characters. The framework of
reinforcement learning
, and in particular,
multi-agent reinforcement learning is very well suited to this kind
of task and we anticipate that people from the RL community will
have a lot to contribute to the field of computer games once the
wealth of opportunities in this space has been understood.
In addition, computer games offer a great variety of other challenges
including problems in graphics, sound, networking, player rating and
matchmaking, interface design, narrative generation etc. All of
these areas would benefit from various machine learning techniques
that may include supervised, unsupervised and other learning
paradigms.
Finally, the holy grail of an interaction between computer games
and machine learning might be the emergence of a new genre of games
that has adaptive agents at the core of its game play. Games like
Creatures and Black & White have attempted to build entire games
around the concept of teaching behaviour to adaptive AI agents. It
is an exciting question to ask if these ideas could be taken further to put
machine learning at the centre of computer games.
|
|
Topics
-
History of computer games with emphasis on AI
and machine learning. Here we will also discuss how other aspects of
the game like sound, graphics, physics and network play affect the
role of AI and pose new challenges.
- Overview of machine learning frameworks ranging
from the general multi-agent framework of Stochastic Games to the
more familiar notions of Markov Decision Processes. This section
will include different learning paradigms and how they are or may be
applied to computer games.
-
Example applications of machine learning in commercial
games including a discussion of games that shipped with a
machine learning component such as Black & White, Colin McRae Rally
2.0, and Forza Motorsport.
- Machine learning projects using computer games as
test beds including reinforcement learning in Tao Feng and
Project Gotham Racing 3, Bayes nets for Quake bots and machine
learning approaches in RoboCup.
- Machine learning approaches to other aspects of games
including a discussion of the Bayesian skill ranking system
TrueSkill™, predictive algorithms for
network latency mitigation, and dimensionality reduction for
animation control based on motion capture data.
- Pointers to resources to get started including
available code bases for simulations and games, game programming
frameworks, papers, competitions and books.
|
|
Format
-
The tutorial is planned for 2.5 hours including a ten minute
break.
-
We will aim at showing many videos and demos to illustrate
our points as well as to make the tutorial an entertaining
experience.
-
The two presenters will take turns presenting different parts
of the tutorial and will work together when more complex demos
require it.
|
Presenters
Ralf
Herbrich
and Thore Graepel together
lead the Applied
Games (APG) group at Microsoft Research Cambridge. The group’s
mission is to apply machine learning to games. These include both
recreational games such as Go, poker, and video games as well as
abstract decision games such as auctions and negotiations played in
the real world. The group developed the TrueSkill™ system used
millions of times each day in Microsoft’s Xbox Live service for
ranking and matchmaking players according to their skills and which
is also the backbone of the ranking system of the blockbuster game
Halo 3. APG also developed the Drivatar™ AI for the Xbox title Forza
Motorsport™.
Thore Graepel
studied physics in Hamburg, London and Berlin. He received both his
diploma in physics and his PhD in statistics from the Technical
University of Berlin. He worked on machine learning and large scale
optimisation at the Swiss Federal Institute of Technology Zurich and
on kernel methods and learning theory at Royal Holloway, University
of London. Since 2002 he has been a researcher in the Machine
Learning and Perception group at Microsoft Research Cambridge where
he co-founded the Applied Games group in 2006. Thore has published
over 50 technical papers on topics around machine learning including
probabilistic models, statistical learning theory, kernel methods,
clustering, stochastic optimisation and reinforcement learning with
applications including computer go, video games, computer networks
and ranking.
Ralf
Herbrich studied computer science in Berlin. He received
both his diploma in computer science and his PhD in statistics from
the Technical University of Berlin. He worked on machine learning,
kernel methods and learning theory during his Post-doc time at
Microsoft Research. Since 2001 he has been a researcher in the
Machine Learning and Perception group at Microsoft Research
Cambridge where he co-founded the Applied Games group in 2006. Ralf
has published over 50 technical papers on topics around machine
learning including probabilistic models, statistical learning
theory, kernel methods, and reinforcement learning with applications
including computer Go, video games, computer networks and ranking.
He also published a book on “Learning Kernel Classifiers” (MIT
Press) in 2001.
|
|
|
|
