Professional Documents
Culture Documents
Marl Book
Marl Book
This license applies only to the work in full and not to any components included with
permission. Subject to such license, all rights are reserved. No part of this book may be
used to train artificial intelligence systems without permission in writing from the MIT
Press.
The MIT Press would like to thank the anonymous peer reviewers who provided
comments on drafts of this book. The generous work of academic experts is essential for
establishing the authority and quality of our publications. We acknowledge with
gratitude the contributions of these otherwise uncredited readers.
This book was set in Times New Roman by Stefano V. Albrecht, Filippos Christianos,
and Lukas Schäfer. Printed and bound in the United States of America.
ISBN:
10 9 8 7 6 5 4 3 2 1
Für A.H.A. und R.J.A. – träumt groß, strebt nach euren Zielen, geht voraus.
For A.E.A. – this book would not be here without you.
–S.V.A.
Für Jonas, Daniel, Annette und Stefan – für eure Liebe, Unterstützung und die
Neugierde, die ihr in mir geweckt habt.
–L.S.
Contents
Preface xiii
List of Figures xx
1 Introduction 1
1.1 Multi-Agent Systems 2
1.2 Multi-Agent Reinforcement Learning 6
1.3 Application Examples 8
1.3.1 Multi-Robot Warehouse Management 8
1.3.2 Competitive Play in Board Games and Video Games 10
1.3.3 Autonomous Driving 11
1.3.4 Automated Trading in Electronic Markets 11
1.4 Challenges of MARL 12
1.5 Agendas of MARL 13
1.6 Book Contents and Structure 15
2 Reinforcement Learning 19
2.1 General Definition 20
2.2 Markov Decision Processes 22
2.3 Expected Discounted Returns and Optimal Policies 24
2.4 Value Functions and Bellman Equation 26
2.5 Dynamic Programming 29
2.6 Temporal-Difference Learning 32
2.7 Evaluation with Learning Curves 36
References 341
respectively. Readers who are already familiar with reinforcement learning and
deep learning, and who want to quickly get going with recent deep learning-
based MARL algorithms, may read Chapter 3 and then skip to Chapter 9 and
onward. To aid lecturers in adopting this book, we have developed lecture slides
(available from the book’s website), which can be modified as required to suit
the course’s needs.
MARL has become a large field of research, and this book does not cover
all aspects of MARL. For instance, there is a growing body of work on using
communication in MARL, which is not covered in this book. This includes
questions such as how agents can learn to communicate robustly when commu-
nication channels are noisy and unreliable; and how agents may use MARL to
learn specialised communication protocols or languages for a given task. While
this book does not focus on communication in MARL, the models introduced
in this book are general enough to also represent communication actions (as
described in Section 3.5). There has also been research on using evolutionary
game theory for multi-agent learning, which is not covered in this book (we
recommend the excellent survey of Bloembergen et al. (2015)). Finally, with the
steep rise of activity in MARL in recent years, it would be futile to write a book
that tries to keep up with new algorithms. We instead focus on the foundational
concepts and ideas in MARL, and refer to survey papers (including those listed
in Appendix A) for a more complete list of algorithm developments.
Acknowledgements: We are grateful to many people who have worked with
us or provided feedback during the writing of this book. Special thanks
go to Elizabeth Swayze and Matthew Valades from the MIT Press who
diligently guided us through the publication process. Many colleagues have
provided valuable feedback and suggestions, and we are grateful to all of
them (in alphabetical order): Christopher Amato, Marina Aoyama, Ignacio
Carlucho, Georgios Chalkiadakis, Sam Dauncey, Alex Davey, Bertrand
Decoster, Mhairi Dunion, Kousha Etessami, Aris Filos-Ratsikas, Elliot
Fosong, Amy Greenwald, Dongge Han, Josiah Hanna, Leonard Hinckeldey,
Sarah Keren, Mykel Kochenderfer, Marc Lanctot, Stefanos Leonardos,
Michael Littman, Luke Marris, Elle McFarlane, Trevor McInroe, Mahdi
Kazemi Moghaddam, Frans Oliehoek, Georgios Papoudakis, Tabish Rashid,
Michael Rovatsos, Rahat Santosh, Raul Steleac, Massimiliano Tamborski,
Kale-ab Tessera, Callum Tilbury, Jeroen van Riel, Zhu Zheng. We also
thank the anonymous reviewers who reviewed the book for MIT Press.
The Mars Rover MDP from Figure 2.3 is based on a similar MDP created
by Elliot Fosong and Adam Jelley for the Reinforcement Learning course
at Edinburgh University. The images in Figure 4.4 and Figure 4.5b were
October, 2023
Stefano V. Albrecht
Filippos Christianos
Lukas Schäfer
General
R set of real numbers
∝ proportional to
x> transpose of a vector x
X> transpose of a matrix X
Pr probability
Pr(x | y) conditional probability of x given y
Ep[x] expectation of x under probability distribution p
x∼p x sampled according to probability distribution p
x←y assign value y to variable x
D training data set
∂f
∂x derivative of function f with respect to x
∇ gradient
ha, b, c, ...i concatenation of inputs a, b, c, ... into tuple (a, b, c, ...)
[x]1 indicator function: returns 1 if x is true, otherwise returns 0
Game Model
I set of agents
i, j agent subscripts
–i subscript to denote the tuple <all agents except agent i>
S, S̄ state space, set of terminal states
s state
O, Oi (joint) observation space, observation space of agent i
o, oi (joint) observation, observation of agent i
A, Ai (joint) action space, action space of agent i
a, ai (joint) action, action of agent i
Deep Learning
θ network parameters
f (x; θ) function f over input x with parameters θ
L loss
B training data batch
B batch size, i.e. number of samples in a batch
Agent Agent
Observations
Observations
Actions
Actions
Environment
1. The level-based foraging example will appear throughout this book. We provide an open-source
implementation of this environment at https://github.com/uoe-agents/lb-foraging. See Section 11.3.1
for more details on this environment.
2. Note that the skill levels of robots and items are not included in the state, since these are assumed
constant. However, if the skill levels can change between different episodes, then the state may
additionally include the skill levels of the robots and items.
Figure 1.2: A level-based foraging task in which a group of three robots, each
controlled by an agent, must collect all items (shown as apples). Each robot
and item has an associated skill level, shown inset. A group of one or more
robots can collect an item if they are located next to the item, and the sum of
the robots’ levels is greater than or equal to the item’s level.
from the set {up, down, left, right, collect, noop} to control its robot. The first
four actions in this action set modify the x/y-position of the robot in the state
by moving the robot into the respective direction (unless the robot is already at
the edge of the grid world, in which case the action has no effect). The collect
action attempts to collect an item that is located adjacent to the robot; if the item
is collected, the action modifies the binary existence variable corresponding to
the item. The noop (no operation) action has no effect on the state.
Notice that in the above description, we used the terms “robot” and “agent”
to refer to two distinct concepts. In level-based foraging, the word “robot” is a
label for an object which is explicitly represented via the x/y-position variables.
Similarly, the word “item” refers to an object in level-based foraging which is
represented by its x/y-position variables and its binary existence variable. In
contrast to these object labels, the term “agent” refers to an abstract decision-
making entity which observes some information from the environment and
chooses values for certain action variables; in this case, the action for a robot. If
there is a direct one-to-one correspondence between agents and certain objects,
such as agents and robots in level-based foraging, then it can be convenient
to use the terms interchangeably. For example, in level-based foraging, we
may say “the skill level of agent i” when referring to the skill level of the robot
controlled by agent i. Unless the distinction is relevant, in this book we will
often use the term “agent” synonymously with the object that it controls.
The defining characteristic of a multi-agent system is that the agents must
coordinate their actions with (or against) each other in order to achieve their
goals. In a fully cooperative scenario, the agents’ goals are perfectly aligned so
that the agents need to collaborate towards achieving a shared goal. For example,
in level-based foraging, all agents may receive a reward of +1 whenever an
item is successfully collected by any of the agents. In a competitive scenario,
the agents’ goals may be diametrically opposed so that the agents are in direct
competition with each other. An example of such a competitive scenario is two
agents playing a game of chess, in which the winning player gets a reward of +1
and the losing player gets a reward of –1 (or both get 0 for a drawn outcome). In
between these two extremes, the agents’ goals may align in some respects while
differing in other respects, which can lead to complex multi-agent interaction
problems that involve both cooperation and competition to varying degrees. For
example, in the actual implementation of level-based foraging we use in this
book (described in Section 11.3.1), only those agents which were involved in the
collection of an item (rather than all agents) will receive a positive normalised
reward. Thus, the agents have a motivation to maximise their own returns (sum
of rewards), which can lead them to try and collect items before other agents
can do so; but they may also need to collaborate with other agents at certain
times in order to collect an item.
The above concepts of states, actions, observations, and rewards are for-
mally defined within game models. Different types of game models exist, and
Chapter 3 introduces the most common game models used in MARL, includ-
ing normal-form games, stochastic games, and partially observable stochastic
games. A solution for a game model consists of a set of policies for the agents
which satisfies certain desired properties. As we will see in Chapter 4, there
exist a range of solution concepts in the general case. Most solution concepts
are anchored in some notion of equilibrium, which means that no individual
agent can deviate from its policy in the solution to improve its outcome.
Research in multi-agent systems has a long history in artificial intelligence
and spans a vast range of technical problems (Shoham and Leyton-Brown 2008;
Wooldridge 2009). These include questions such as how to design algorithms
which enable agents to choose optimal actions towards their specified goals; how
to design environments to incentivise certain long-term behaviours in agents;
how information is communicated and propagated among agents; and how
norms, conventions and roles may emerge in a collective of agents. This book is
concerned with the first of these questions, with a focus on using RL techniques
to optimise and coordinate the policies of agents in order to maximise the
rewards they accumulate over time.
Agent 1 action
observation
& reward Agent 2 action
observation
& reward joint action
Agent n action
observation
& reward
Environment
joint action modifies environment state
3. This book uses a literal definition of the MARL term in which we learn policies for multiple
agents. This is in contrast to learning a policy for a single agent that operates in a multi-agent
system, in which we have no control over the other agents.
such applications, MARL algorithms can learn agent policies which can be
executed in a decentralised fashion.
MARL algorithms can be categorised based on a number of dimensions,
as shown in Figure 1.4. For example, this includes assumptions about the
agents’ rewards (e.g. fully cooperative, competitive, or mixed), what type of
solution concept the algorithm is designed to achieve (e.g. Nash equilibrium),
and what agents can observe about their environment. Algorithms can also be
categorised based on assumptions made during the learning of agent policies
(“training”) versus assumptions made after learning (“execution”). Centralised
training and execution assumes that both stages have access to some centrally
shared mechanism or information, such as sharing all observations between
agents. For example, a single central agent may receive information from
all other agents and dictate the actions to the agents. Such centralisation can
help to improve coordination between agents and alleviate issues such as non-
stationarity (discussed in Section 1.4). In contrast, decentralised training and
execution assumes no such centrally shared information, and instead requires
that the learning of an agent’s policy as well as the policy itself only use the local
information of that agent. The third major category, centralised training with
decentralised execution, aims to combine the benefits of the two aforementioned
approaches, by assuming that centralisation is feasible during training (e.g. in
simulation) while producing policies that can be executed in a fully decentralised
way. These ideas will be further discussed in Chapter 9.
Centralisation & Can agents coordinate their actions via a central controller
Communication or coordinator? Or do agents learn fully independent poli-
cies? Can agents share/communicate information during
learning and/or after learning? Is the communication chan-
nel reliable, or noisy and unreliable?
(Chapters 3, 5, 6 and 9)
Figure 1.4: Dimensions in MARL and relevant book chapters for further details.
10 Chapter 1
et al. 2022; Pérolat et al. 2022; Silver et al. 2018; Tesauro 1994; Vinyals et
al. 2019).
agendas which have been pursued in MARL research.4 The agendas differ in
their motivations and goals for using MARL, as well as the criteria by which
progress and success are measured. In their article, Shoham et al. made the
important point that one must be clear about the purpose and goals when using
MARL. We describe the main agendas as follows:
Computational The computational agenda uses MARL as an approach to
compute solutions for game models. A solution consists of a collection of
decision policies for the agents that satisfy certain properties (e.g. Nash
equilibrium and other solution concepts, discussed in Chapter 4). Once
computed, a solution could be deployed in an application of the game to
control the agents (see Section 1.3 for some examples), or it may be used
to conduct further analysis of the game. Thus, in the computational agenda,
MARL algorithms compete with other direct methods to compute game
solutions (Nisan et al. 2007; Roughgarden 2016). Such direct methods can
be significantly more efficient than MARL algorithms for certain types of
games (such as the linear programming methods discussed in Sections 4.3.1
and 4.6.1), but they typically require full knowledge of the game, including
the reward functions of all agents. In contrast, MARL algorithms are
usually designed to learn solutions without full knowledge of the game.
Prescriptive The prescriptive agenda focuses specifically on the behaviours
and performance of agents during learning, and asks how they should learn
to achieve a given set of criteria. Different criteria have been proposed in
this regard. One possible criterion is that the average reward received by
a learning agent should not fall below a certain threshold during learning,
regardless of how other agents may be learning. An additional criterion
might be that the learning agent should learn optimal actions if the other
agents come from a certain class of agents (such as static, non-learning
agents), and otherwise should not fall below a certain performance threshold
(Powers and Shoham 2004). Such criteria focus on the behaviours of agents
during learning, and may leave open whether the collective learning process
converges to a particular equilibrium. Thus, convergence to a particular
solution concept (e.g. equilibrium) is not necessarily the goal in this agenda.
Descriptive The descriptive agenda uses MARL to study the behaviours of
agents, including natural agents such as humans and animals, when learning
4. The article of Shoham, Powers, and Grenager (2007) was part of a special issue called “Founda-
tions of Multi-Agent Learning” which was published in the Artificial Intelligence journal (Vohra
and Wellman 2007). This special issue contains many interesting articles from some of the early
contributors in MARL research, including responses to Shoham et al.’s article.
1. Other learning objectives can be specified, such as maximising the average reward (Sutton and
Barto 2018) and different types of “risk-sensitive” objectives (Mihatsch and Neuneier 2002).
observation
Agent action
& reward
Environment
action modifies environment state
2. Technically, if we assume that the agent can observe the time step t, then termination after T time
steps can be modelled by including t in the information contained in the state st , and defining the
set of terminal states S̄ to contain all states st with t ≥ T.
Immobile
p = 0.1
r = -3
p = 0.2
r = -3
p = 0.8
Site A right Site B
p = 0.9 r = -1
r = -1
right
left r=-1 left
r = +10
left
r = -1
p = 0.5
Start right Base
r = +10
p = 0.5
r = -10
Destroyed
Figure 2.3: Mars Rover MDP. States are shown as circles, where Start is the
initial state. Each non-terminal state (white) has two possible actions, right and
left, which are shown as directed edges with associated transition probabilities
(p) and rewards (r). Grey shaded circles mark terminal states.
Site B) before arriving at the base station. However, it takes a day to arrive at
each site (–1 reward) and there is a probability of 0.3 that the rover will get
stuck on the rocky ground and be immobilised (–3 reward). Arriving at the base
station gives a reward of +10.
The term “Markov” comes from the Markov property, which states that the
future state and reward are conditionally independent of past states and actions,
given the current state and action:
Pr(st+1 , rt | st , at , st–1 , at–1 , ..., s0 , a0 ) = Pr(st+1 , rt | st , at ) (2.3)
This means that the current state provides sufficient information to choose
optimal actions in an MDP – past states and actions are not relevant.
The most common assumption in RL is that the dynamics of the MDP, in
particular the transition and reward functions T and R, are a priori unknown to
the agent. Typically, the only parts of the MDP which are assumed to be known
are the action space A and the state space S.
While Definition 1 defines MDPs with finite, or discrete, states and actions,
MDPs can also be defined with continuous states and actions, or a mixture of
both discrete and continuous elements. Moreover, while the reward function R
as defined here is deterministic, MDPs can also define a probabilistic reward
function such that R(s, a, s0 ) gives a distribution over possible rewards.
The multi-armed bandit problem (or simply “bandit problem”) is an important
special case of the MDP in which each episode terminates after one time step
(i.e. T = 1), there is only a single state in S and no terminal states (i.e. |S| = 1
and S̄ = ∅), and the reward function R is probabilistic and unknown to the agent.
Thus, in such a bandit problem, each action produces a reward from some
unknown reward distribution, and the goal is effectively to find the action that
yields the highest expected reward as quickly as possible. Bandit problems
have been used as a basic model to study the exploration-exploitation dilemma
(Lattimore and Szepesvári 2020).
An inherent assumption in MDPs is that the agent can fully observe the
state of the environment in each time step. In many applications, an agent
only observes partial and noisy information about the environment. Partially
observable Markov decision processes (POMDPs) (Kaelbling, Littman, and
Cassandra 1998) generalise MDPs by defining a decision process in which the
agent receives observations ot rather than directly observing the state st , and
these observations depend in a probabilistic or deterministic way on the state.
Thus, in general, the agent will need to take into account the history of past
observations o0 , o1 , ..., ot in order to infer the possible current states st of the
environment. POMDPs are a special case of the partially observable stochastic
game (POSG) model introduced in Chapter 3; namely, a POMDP is a POSG
in which there is only one agent. We refer to Section 3.4 for a more detailed
definition and discussion of partial observability in decision processes.
We have defined the MDP as a model of decision processes, but we have not
yet defined the learning objective in an MDP. The next section will introduce
the most common learning objective used in MDPs: maximising expected
discounted returns. As discussed earlier (Figure 2.1), together, an MDP and
learning objective specify an RL problem.
Due to the stochastic nature of the MDP, it may not be possible to maximise
this return in all episodes. This is because some actions may lead to different
outcomes with certain probabilities, and these outcomes are outside the control
of the agent. Therefore, the agent can maximise the expected return given by
Eπ r0 + r1 + ... + rT–1
(2.5)
where the expectation assumes that the initial state is sampled from the initial
state distribution (i.e. s0 ∼ µ), that the agent follows policy π to select actions
(i.e. at ∼ π(· | st )), and that successor states are governed by the state transition
probabilities (i.e. st+1 ∼ T (· | st , at )).
The above definition of total returns is guaranteed to be finite for a terminating
MDP. However, in non-terminating MDPs the total return may be infinite, in
which case returns may not be informative to distinguish the performance of
different policies that achieve infinite returns. The standard approach to ensuring
finite returns in non-terminating MDPs is to use a discount factor γ ∈ [0, 1],
based on which we define the discounted return3
"∞ #
0 X
1 2 2 t t
Eπ r + γr + γ r + ... = Eπ γr . (2.6)
t=0
For γ < 1 and assuming that rewards are constrained to lie in a finite range
[rmin , rmax ], the discounted return is guaranteed to be finite
∞ ∞
X X 1
γ t rt ≤ rmax γ t = rmax (2.7)
1–γ
t=0 t=0
where the right-hand fraction in the above equation is the closed form of the
P∞
geometric series given by t=0 γ t .
The discount factor has two equivalent interpretations. One interpretation
is that (1 – γ) is the probability with which the MDP terminates after each
time step. Thus, the probability that the MDP terminates after a total of T > 0
time steps is γ T–1 (1 – γ), where γ T–1 is the probability of continuing (i.e. not
terminating) in the first T – 1 time steps, multiplied by the probability (1 – γ) of
terminating in the following time step. For example, in the Mars Rover MDP
shown in Figure 2.3, we can specify a discount factor of γ = 0.95 to model
the fact that the rover’s battery may fail with a probability of 0.05 after each
state transition. The second interpretation is that the agent gives “weight” γ t to
reward rt received at time t. Thus, a γ close to 0 leads to a myopic agent that
cares more about near-term rewards, while a γ close to 1 leads to a farsighted
agent that also values distant rewards. In either case, it is important to note that
3. Note that rt refers to the reward at time t, while γ t denotes the exponentiation operation (γ to the
power of t). This is a standard notational convention in RL literature.
the discount rate is part of the learning objective, and not a tunable algorithm
parameter; γ is a fixed parameter specified by the learning objective. In the
remainder of this book, whenever we refer to returns, we specifically mean
discounted returns with some discount factor γ.
Finally, the above definition of discounted returns can work for both termi-
nating and non-terminating MDPs via the convention of absorbing states. If an
episode reaches a terminal state or a maximum number of time steps, we define
the final state in the episode to be absorbing in that any subsequent actions in
this state will transition the MDP into the same state with probability 1 and give
a reward of 0 to the agent. Thus, once the MDP reaches an absorbing state, it
will forever remain in that state and the agent will not accrue any more rewards.
Based on the above definition of discounted returns and absorbing states, we
can now define the solution to the MDP as the optimal policy π ∗ that maximises
the expected discounted return.
= rt + γut+1 . (2.9)
π
Given a policy π, the state-value function V (s) gives the “value” of state
s under π, which is the expected return when starting in state s and following
policy π to select actions, formally
V π (s) = Eπ ut | st = s
(2.10)
= Eπ r + γut+1 | st = s
t
(2.11)
X X
T (s0 | s, a) R(s, a, s0 ) + γEπ ut+1 | st+1 = s0
= π(a | s) (2.12)
a∈A s0 ∈S
X X
T (s0 | s, a) R(s, a, s0 ) + γV π (s0 )
= π(a | s) (2.13)
a∈A s0 ∈S
where V π (s) = 0 for terminal (i.e. absorbing) states s ∈ S̄. The recursive equation
given in Equation 2.13 is called the Bellman equation in honour of Richard
Bellman and his pioneering work (Bellman 1957).
The Bellman equation for V π defines a system of m linear equations with m
variables, where m = |S| is the number of states in the finite MDP:
X X
V π (s1 ) = T (s0 | s1 , a) R(s1 , a, s0 ) + γV π (s0 )
π(a | s1 ) (2.14)
a∈A s0 ∈S
X X
V π (s2 ) = T (s0 | s2 , a) R(s2 , a, s0 ) + γV π (s0 )
π(a | s2 ) (2.15)
a∈A s0 ∈S
..
.
X X
V π (sm ) = T (s0 | sm , a) R(sm , a, s0 ) + γV π (s0 )
π(a | sm ) (2.16)
a∈A s0 ∈S
where V π (sk ) for k = 1, ..., m are the variables in the equation system, and
π(a | sk ), T (s0 | sk , a), R(sk , a, s0 ), and γ are constants. This equation system has
a unique solution given by the value function V π for policy π. If all elements
of the MDP are known, then one can solve this equation system to obtain V π
using any method to solve linear equation systems (such as Gauss elimination).
Analogous to state-value functions, we can define action-value functions
Qπ (s, a) which give the expected return when selecting action a in state s and
then following policy π to select actions subsequently,
Qπ (s, a) = Eπ ut | st = s, at = a
(2.17)
= Eπ r + γut+1 | st = s, at = a
t
(2.18)
X
T (s0 | s, a) R(s, a, s0 ) + γV π (s0 )
= (2.19)
s0 ∈S
X X
T (s0 | s, a) R(s, a, s0 ) + γ π(a0 | s0 )Qπ (s0 , a0 )
= (2.20)
s0 ∈S a0 ∈A
where Qπ (s, a) = 0 for terminal states s ∈ S̄. For the jump from Equation 2.19 to
2.20, notice in Equation 2.13 that the second sum (over states s0 ) is an application
of Equation 2.19, hence it can be replaced by Qπ (s0 , a0 ). Equation 2.20 admits
a system of linear equations that has a unique solution given by Qπ .
A policy π is optimal in the MDP if the policy’s (state or action) value
function is the optimal value function of the MDP, defined as
0
V ∗ (s) = max
0
V π (s), ∀s ∈ S (2.21)
π
0
Q∗ (s, a) = max
0
Qπ (s, a), ∀s ∈ S, a ∈ A (2.22)
π
4. In fact, if multiple actions have maximum value, then there will be an infinite number of optimal
policies since the set of possible probability assignments to these actions spans a continuous
(infinite) space.
By the policy improvement theorem (Sutton and Barto 2018), we know that
if for all s ∈ S
X X
π 0 (a | s)Qπ (s, a) ≥ π(a | s)Qπ (s, a) (2.40)
a∈A a∈A
= V π (s) (2.41)
then π 0 must be as good as or better than π:
0
∀s : V π (s) ≥ V π (s). (2.42)
If after the policy improvement task, the greedy policy π 0 did not change
0
from π, then we know that V π = V π and it follows for all s ∈ S:
0
V π (s) = max Eπ rt + γV π (st+1 ) | st = s, at = a
(2.43)
a∈A
h 0
i
= max Eπ0 rt + γV π (st+1 ) | st = s, at = a (2.44)
a∈A
X h 0
i
= max T (s0 | s, a) R(s, a, s0 ) + γV π (s0 ) (2.45)
a∈A
s0 ∈S
∗
= V (s) (2.46)
Therefore, if π 0 did not change from π after policy improvement, we know that
π 0 must be the optimal policy of the MDP, and policy iteration is complete.
The above version of policy iteration, which uses iterative policy evaluation,
makes multiple complete update sweeps over the entire state space S. Value
iteration is a DP algorithm that combines one sweep of iterative policy evalua-
tion and policy improvement in a single update equation, by using the Bellman
optimality equation as an update operator:
X
V k+1 (s) ← max T (s0 | s, a) R(s, a, s0 ) + γV k (s0 ) , ∀s ∈ S
(2.47)
a∈A
s0 ∈S
learning rate, while a constant learning rate αk (s, a) = c does not. Nonetheless,
in practice it is common to use a constant learning rate since learning rates that
meet the above conditions, while theoretically sound, can lead to slow learning.
In order for Sarsa to learn the optimal value function Q∗ and optimal policy
∗
π , it must gradually modify the policy π in a way that brings it closer to
the optimal policy. To achieve this, π can be made greedy with respect to
the value estimates Q, similarly to the DP policy improvement task defined in
Equation 2.39. However, making π fully greedy and deterministic would violate
the first condition of trying all state-action combinations infinitely often. Thus,
a common technique in TD algorithms is to instead use an -greedy policy that
uses a parameter ∈ [0, 1], defined as6
if a ∈ arg maxa0 ∈A Q(s, a0 )
1 – + |A|
π(a | s) = . (2.55)
|A| otherwise
Thus, the -greedy policy chooses the greedy action with probability 1 – , and
with probability chooses a random other action. In this way, if > 0, the
requirement to try all state-action combinations an infinite number of times is
ensured. Now, in order to learn the optimal policy π ∗ , we can gradually reduce
the value of to 0 during learning, such that π will gradually converge to π ∗ .
Pseudocode for Sarsa using -greedy policies is given in Algorithm 2.
6. Equation 2.55 assumes that there is a single greedy action with maximum value under Q. If
multiple actions with maximum value exist for a given state, then the definition can be modified to
distribute the probability mass (1 – ) among these actions.
While Sarsa is based on the Bellman equation for Qπ , one can also construct
a TD update target based on the Bellman optimality equation for Q∗ ,
X
Q∗ (s, a) = T (s0 | s, a) R(s, a, s0 ) + γ max
0
Q ∗ 0 0
(s , a ) . (2.56)
a ∈A
s0 ∈S
7. Recall that we use the term “return” as a shorthand for discounted return when the learning
objective is to maximise discounted returns.
8. In policy gradient RL algorithms, discussed in Chapter 8, the algorithm learns a probabilistic
policy for which there usually is no “greedy” version. In this case, the evaluation return just uses
the current policy without modification.
4 4
2 3
0 2
2 1
Optimal policy (V * (s0) = 4.1)
Sarsa
Q-Learning
0 100 200 300 400 500 0 100 200 300 400 500
Environment time steps Environment time steps
(a) Average evaluation returns for Sarsa (b) Average episode length for Sarsa and
and Q-learning Q-learning
6 6
4 4
Evaluation returns
Evaluation returns
2 2
0 0
Figure 2.4: Results when applying the Sarsa and Q-learning algorithms to the
Mars Rover problem (Figure 2.3, page 23) with γ = 0.95, averaged over 100
independent training runs. The shaded area shows the standard deviation over
the averaged discounted returns from each training run. The dashed horizontal
line marks the optimal value of the initial state (V ∗ (s0 )), computed via value
iteration. In Figures 2.4a and 2.4b, the algorithms use a scheduled learning rate
of αk (s, a) = 1k where k is the number of times state s has been visited and action
a was applied. The exploration rate is linearly decayed from = 1 to = 0 over
1
time steps t = 0, ..., 500 (i.e. t = 1 – t 500 ).
38 Chapter 2
Note that the x-axis of the plots show the cumulative training time steps across
episodes. This is not to be confused with the time steps t within episodes. The
reason that we show cumulative time steps instead of the number of completed
episodes on the x-axis is that the latter can potentially skew the comparison of
algorithms. For instance, suppose we show training curves for algorithms A and
B with number of episodes on the x-axis. Suppose both algorithms eventually
reach the same performance with their learned policies, but algorithm A learns
much faster than algorithm B (i.e. the curve for A rises more quickly). In
this case, it could be that in the early episodes, algorithm A explored many
more actions (i.e. time steps) than algorithm B before finishing the episode,
which results in a larger number of collected experiences (or training data) for
algorithm A. Thus, although both algorithms completed the same number of
episodes, algorithm A will have done more learning updates (assuming updates
are done after each time step, as in the TD methods presented in this chapter),
which would explain the seemingly faster growth in the learning curve. If we
instead showed the learning curve for both algorithms with cumulative time
steps across episodes, then the learning curve for A might not grow faster than
the curve for B.
Recall that at the beginning of this chapter (Figure 2.1, page 20), we defined
an RL problem as the combination of a decision process model (e.g. MDP) and
a learning objective for the policy (e.g. maximising the expected discounted
return in each state for a specific discount factor). When evaluating a learned
policy, we want to evaluate its ability to achieve this learning objective. For
example, our Mars Rover problem specifies a discounted return objective with
the discount factor γ = 0.95. Therefore, our learning curves in Figure 2.4a show
exactly this objective on the y-axis.
However, in some cases, it can be useful to show undiscounted returns (i.e.
γ = 1) for a learned policy, even if the RL problem specified a discounted return
objective with a specific discount factor γ < 1. The reason is that undiscounted
returns can sometimes be easier to interpret than discounted returns. For exam-
ple, suppose we want to learn an optimal policy for a video game in which the
agent controls a space ship and receives a +1 score (reward) for each destroyed
enemy. If in an episode the agent destroys 10 enemies at various points in
time, the undiscounted return will be 10 but the discounted return will be less
than 10, making it more difficult to understand how many enemies the agent
destroyed. More generally, when analysing a learned policy, it is often useful to
show additional metrics besides the returns, such as win rates in competitive
games, or episode lengths as we did in Figure 2.4b.
When showing undiscounted returns and additional metrics such as episode
lengths, it is important to keep in mind that the evaluated policy was not actually
This is the simplified Bellman equation for MDPs using a reward function
R(s, a). Going the reverse direction from R(s, a) to R(s, a, s0 ), we know that by
defining R(s, a) to be the expected reward under the state transition probabilities
X
R(s, a) = T (s0 | s, a)R(s, a, s0 ) (2.63)
s0 ∈S
and plugging this into Equation 2.62, we recover the original Bellman equation
that uses R(s, a, s0 ), given in Equation 2.59. Analogous transformations can
be done for the Bellman equation of the action value function Qπ , as well as
the Bellman optimality equations for V ∗ /Q∗ . Therefore, one may use either
R(s, a, s0 ) or R(s, a).
In this book, we use R(s, a, s0 ) for the following pedagogical reasons:
1. Using R(s, a, s0 ) can be more convenient when specifying examples of
MDPs, since it is useful to show the differences in reward for multiple pos-
sible outcomes of an action. In our Mars Rover MDP shown in Figure 2.3,
when using R(s, a) we would have to show the expected reward (Equa-
tion 2.63) on the transition arrow for action right from the initial state Start
(rather than showing the different rewards for the two possible outcomes),
which would be less intuitive to read.
2. The Bellman equations when using R(s, a, s0 ) show a helpful visual con-
gruence with the update targets used in TD algorithms. For example, the
Q-learning target, rt + γ maxa0 Q(st+1 , a0 ), appears in a corresponding form
in the Bellman optimality equation (within the [] brackets):
X
∗ 0 0 ∗ 0 0
Q (s, a) = T (s | s, a) R(s, a, s ) + γ max
0
Q (s , a ) (2.64)
a ∈A
s0 ∈S
And similarly for the Sarsa target and the Bellman equation for Qπ (see
Section 2.6).
Finally, we mention that most of the original MARL literature presented in
Chapter 6 in fact defined reward functions as R(s, a). To stay consistent in our
notation, we will continue to use R(s, a, s0 ) in the remainder of this book, noting
that equivalent transformations from R(s, a) to R(s, a, s0 ) always exist.
2.9 Summary
This chapter has provided a concise introduction to the basic concepts of single-
agent RL. The most important concepts are the following:
• The Markov decision process (MDP) is the standard model to define the
environment in which the agent chooses actions over a number of time steps.
It defines the possible states of the environment, the actions available to the
agent, how the environment state changes in response to the different actions,
and the rewards received by the agent. The agent uses a policy which assigns
probabilities to the available actions in each state.
• A learning problem in RL is defined by the combination of a decision process
model (e.g. MDP) and a learning objective. The most common learning
objective used in single-agent RL is to find a decision policy for the agent
that maximises the expected discounted return, defined as the sum of rewards
received over time and weighted by a discount factor.
• The Markov property in MDPs means that the future states and rewards are
independent of past states and actions, given the current state and action. This
property allows us to define recursive value functions for policies, known as
Bellman equations, that return the “value” of a policy in each possible state
of the environment. The value is the expected return when starting in the
given state and following the policy to select actions. Value functions can
also be defined for state-action pairs, where a given action is first chosen and
subsequent actions are chosen by the policy.
• Dynamic programming (DP) and temporal-difference learning (TD) are two
families of algorithms that can learn optimal policies in MDPs. DP algorithms
require complete knowledge of the MDP specification and use procedures
such as value iteration to learn optimal value functions. TD algorithms build
on DP theory but do not need complete knowledge of the MDP, instead
learning optimal policies by repeatedly exploring actions in the environment
and observing the outcomes.
• RL algorithms are evaluated via learning curves that show improvement in
expected returns over an increasing number of agent-environment interactions
during learning.
The following chapters in this part of the book will build on the above
concepts in multiple ways. First, Chapter 3 will extend the MDP model by
introducing game models in which multiple agents interact in a shared envi-
ronment. Chapter 4 will use the notion of discounted returns to define a range
of solution concepts for multi-agent games. Finally, Chapters 5 and 6 will
introduce several families of MARL algorithms that build on and extend the DP
and TD algorithms introduced in this chapter.
1. There exists another type of game model, not covered in this chapter, called “extensive-form
game”. The primary distinction is that in extensive-form games, agents choose actions in turns (one
after another) while in the games we introduce, agents choose actions simultaneously (at the same
time). We focus on simultaneous-move games since most MARL research uses this type of game
model, and they are a more natural extension of the MDP-type models used in RL. Transformations
between extensive-form games and (sequential) simultaneous-move games are possible (Shoham
and Leyton-Brown 2008).
Stochastic Game
n agents
m states - fully observed
Repeated Markov
Normal-Form Game Decision Process
n agents 1 agent
1 state m states
Figure 3.1: Hierarchy of game models used in this book. Partially observable
stochastic games (POSGs) include stochastic games as a special case in which
the states and agents’ chosen actions are fully observable by all agents. Stochas-
tic games include repeated normal-form games as a special case in which there
is only a single environment state, and they include Markov decision processes
(MDPs) as a special case in which there is only a single agent.
probability πi (ai ) given by its policy. The resulting actions of all agents form a
joint action, a = (a1 , ..., an ). Finally, each agent i receives a reward based on its
reward function and the joint action, ri = Ri (a).
Normal-form games can be classified based on the relationship between the
reward functions of agents:
• In a zero-sum game, the sum of the agents’ rewards is always 0, i.e.
2
P
i∈I Ri (a) = 0 for all a ∈ A. In zero-sum games with two agents, i and
3
j, one agent’s reward function is simply the negative of the other agent’s
reward function, i.e. Ri = –Rj .
• In a common-reward game, all agents receive the same reward, i.e. Ri = Rj
for all i, j ∈ I.
• In a general-sum game, there are no restrictions on the relationship of reward
functions.
Normal-form games with two agents are also referred to as matrix games
because, in this case, the reward function can be compactly represented as a
matrix.4 Figure 3.2 shows three example matrix games. Agent 1’s action is to
choose the row position, and agent 2’s action is to choose the column position.
The values (r1 , r2 ) in the matrix cells show the agents’ rewards for each possible
joint action. Figure 3.2a shows the Rock-Paper-Scissors game in which each
agent chooses one of three possible actions (R,P,S). This is a zero-sum game
(i.e. r1 = –r2 ) since for each action combination, one agent wins (+1 reward)
and the other agent loses (-1 reward), or there is a draw (0 reward to both).
Figure 3.2b shows a “coordination” game with two actions (A,B) for each agent,
and agents must select the same action to receive a positive reward. This game
is common-reward (i.e. r1 = r2 ), hence it suffices to show a single reward in
the matrix cells which both of the agents receive. Lastly, Figure 3.2c shows a
widely studied game called the Prisoner’s Dilemma, which is a general-sum
game. Here, each agent chooses to either cooperate (C) or defect (D). While
mutual cooperation would give both agents the second-highest reward, each
agent is individually incentivised to defect since this is the “dominant action”,
meaning that D always achieves higher rewards compared to C. What makes
2. Zero-sum games are a special case of constant-sum games, in which the agents’ rewards sum to
a constant.
3. For games with two agents, we often use i and j to refer to the two agents in formal descriptions,
and we use 1 and 2 to refer to the agents in specific examples.
4. Some game theory literature uses the term “bimatrix game” to specifically refer to general-sum
games that require two matrices to define the agents’ reward functions, and “matrix game” for
games that require only a single matrix to define the reward functions. This book uses “matrix
game” in both cases.
R P S
R 0,0 -1,1 1,-1 A B C D
P 1,-1 0,0 -1,1 A 10 0 C -1,-1 -5,0
S -1,1 1,-1 0,0 B 0 10 D 0,-5 -3,-3
(a) Rock-Paper-Scissors (b) Coordination Game (c) Prisoner’s Dilemma
Figure 3.2: Three normal-form games with two agents (i.e. matrix games).
these and many other games interesting is that one agent’s reward depends on
the choices of the other agents, which are not known in advance.
Much effort has been devoted to developing classifications of normal-form
games and understanding the relationship between different games (Bruns 2015;
Kilgour and Fraser 1988; Marris, Gemp, and Piliouras 2023; Rapoport and
Guyer 1966; Robinson and Goforth 2005; Walliser 1988). For instance, in
Section 11.2 we provide a comprehensive listing of all structurally distinct and
strictly ordinal 2 × 2 normal-form games (meaning games with two agents and
two actions each), of which there are 78 in total.
In the remainder of the book, we will use “normal-form game” when making
general statements about n-agent normal-form games, and we will sometimes
use “matrix game” when discussing specific examples of normal-form games
for two agents.
actions before the current time step t (for t = 0, the history is the empty set).
Given the joint action at = (at1 , ..., atn ), each agent i receives a reward rit = Ri (at ).
Besides the time index t, the important addition in repeated normal-form
games is that policies can now make decisions based on the entire history of
past joint actions, giving rise to a complex space of policies that agents can use.
Typically, policies are conditioned on an internal state which is a function of the
history, f (ht ). For example, a policy may be conditioned on just the most recent
joint action at–1 , or on summary statistics such as the action counts of other
agents in the history. “Tit-for-Tat” is a famous policy for the game of repeated
Prisoner’s Dilemma that simply conditions on the most recent action of the
other agent, choosing to cooperate if the other agent cooperated and defecting
if the other agent defected (Axelrod and Hamilton 1981).
It is important to note that a game with finite repetitions is not in general
equivalent to the same game with infinite repetitions. In finitely repeated games,
there can be “end-game” effects: if the agents know that a game will finish after
T time steps, they may choose different actions closer to the end of the game
compared to earlier in the game (see Section 6.3.3 for an example in Prisoner’s
Dilemma). For infinitely repeated games, one may specify a probability with
which each time step terminates the game. This termination probability is
related to the discount factor γ ∈ [0, 1] used in the discounted-return learning
objective in RL, in that 1 – γ specifies the probability of termination in each
time step (see the discussion in Section 2.3). For γ < 1, the game still counts
as “infinite” since any finite number T > 0 of time steps will have a non-zero
probability of occurring.
In the remainder of this book, we will use the term non-repeated normal-form
game to refer to the special case when T = 1, while repeated normal-form game
refers to the case when T > 1.
probability function T . For example, two agents that together collect an existing
item will modify the state by switching the binary flag associated with that
item (meaning that the item has been collected and no longer exists). In a
common-reward version of the game, every agent will receive a reward of +1
whenever any of the items has been collected by any agents. A general-sum
version may specify individual rewards for agents, such as +1 reward for agents
which were actually involved in the collection of an item and 0 reward for all
other agents. The game terminates after a fixed number of time steps, or when
a terminal state has been reached in which all items have been collected.
Stochastic games include repeated normal-form games as a special case in
which there is only a single state in S and there are no terminal states (i.e.
S̄ = ∅). More generally, if we define reward functions as Ri (s, a),7 then each
state s ∈ S in the stochastic game can be viewed as a non-repeated normal-form
game with rewards given by Ri (s, ·), as shown in Figure 3.3. It is in this sense
that normal-form games form the basic building block of stochastic games.
Stochastic games also include MDPs as the special case in which there is only
a single agent in I. And, like MDPs, while the state and action sets in our
definition of stochastic games are finite (as per the original definition of Shapley
(1953)), a stochastic game can be analogously defined for continuous states and
actions. Finally, the classification of normal-form games into zero-sum games,
common-reward games, and general-sum games also carry over to stochastic
games (Section 3.1). That is, stochastic games may specify zero-sum rewards,
common rewards, or general-sum rewards.
Chapter 11 gives many more examples of state-based multi-agent environ-
ments, many of which can be modelled as a stochastic game.
7. For a comment on the equivalence of Ri (s, a, s0 ) and Ri (s, a), see Section 2.8.
a = (2,1)
r = (2,2) p = 0.8
p = 0.5 a = (2,1)
r = (4,1)
p = 0.5
s1
1,1 0,1
a = (2,2)
r = (1,1) 2,2 1,1
p = 1.0
p = 0.8
p = 0.5 a = (2,1)
r = (4,1)
s2 a = (1,2) p = 0.2 s3
r = (0,4)
1,3 0,4 1,1 0,0
4,0 3,1 4,1 3,2
(c) POSG
Figure 3.3: Normal-form games are the basic building block of all game models
described in Chapter 3. This figure shows one example game for each type of
game model, shown as a directed cyclic graph. Each game is for two agents with
two actions each. Each node in the graph corresponds to a state and shows a
normal-form game being played in that state. We show one possible joint action
choice for each state, along with the rewards received and the probabilities of
reaching the respective next state in the outgoing arrows. For POSG, the states
are dashed to represent that agents do not directly observe the current state of
the game; instead, agents receive partial/noisy observations about the state.
Games: Models of Multi-Agent Interaction 51
robot control tasks, or strategic games where players have private information
not seen by other players (e.g. card games).
In its most general form, a POSG defines state-observation probabilities
Pr(st , ot | st–1 , at–1 ), where ot = (ot1 , ..., otn ) is the joint observation at time t which
contains the agents’ individual observations oti . However, it is often the case
that observations only depend on the new environment state st and the joint
action at–1 that led to this state (and not on the previous state st–1 ). Thus, it is
common to define for each agent i an individual observation function Oi which
specifies probabilities over the agent’s possible observations oti given the state
st and joint action at–1 . We give the full definition of POSGs below:
Definition 4 (Partially observable stochastic game) A partially observable
stochastic game (POSG) is defined by the same elements of a stochastic game
(Definition 3) and additionally defines for each agent i ∈ I:
• Finite set of observations Oi
• Observation function Oi : A × S × Oi → [0, 1] such that
X
∀a ∈ A, s ∈ S : Oi (a, s, oi ) = 1 (3.4)
oi ∈Oi
multi-agent planning (see, for example, the textbook by Oliehoek and Amato
(2016)). POSGs can also be defined analogously with continuous-valued (or
mixed discrete-continuous) observations.
POSGs include stochastic games as the special case in which oti = (st , at–1 ).
They also include POMDPs as the special case in which there is only a single
agent in I. In general, the observation functions in POSGs can be used to
represent diverse observability conditions of interest. Examples include:
Unobserved actions of other agents Agents may observe the state and their
own previous action, but not the previous actions of the other agents, i.e.
oti = (st , at–1
i ). An example of this scenario is robot soccer, in which the
robots may observe the full playing field but do not directly communicate
their chosen actions to other players (especially players from the opponent
team). In this case, agents may need to infer with some uncertainty the pos-
sible actions of other agents based on changes in the observed environment
state (e.g. inferring a pass action between two players based on location,
direction, and velocity of the ball). Another example is a market setting in
which the asset prices are observed by all agents but the agents’ buy/sell
actions are private.
Limited view region Agents may observe a subset of the state and joint action,
i.e. oti = (s̄t , āt ) where s̄t ⊂ st and āt ⊂ at . Such a scenario may arise if agents
have a limited view of their surrounding environment, such that an agent
may only see those parts of the state st and joint action at–1 which are (or
took place) within its view region. For example, in the partially observable
version of level-based foraging shown in Figure 3.4, the agents have local
vision fields and can only see items, other agents, and their actions within
their own vision fields. Another example is agents playing a real-time
strategy game in which parts of the game map are concealed by “fog of
war” (e.g. Vinyals et al. 2019).
Observation functions can model uncertainty in observations (i.e. noise) by
assigning non-zero probability to multiple possible observations. For example,
such uncertainty may be caused by imperfect sensors in robotics applications.
Observation functions can also model communication between agents which
may be limited by range and can be unreliable. A communication message
can be modelled as a multi-valued vector (e.g. a bit vector) which does not
modify the environment state but can be received by other agents within their
own observation, if they are within a certain range of the sender. Unreliable
communication can be modelled by specifying a probability with which mes-
sages are lost (i.e. not received by an agent even if within sending range), or
by randomly changing parts of the message. One may even specify combined
actions ai = (ai,s , ai,c ) where ai,s affects the state s (such as moving around in a
physical world) while ai,c is a communication message which may be received
by other agents j within their observation oj .
In Chapter 11 (Section 11.3), we list several multi-agent environments with
partial observability used in MARL research.
belief state of the agent is given by the distribution of initial states, i.e. b0i = µ.
After taking action ati and observing ot+1 t t+1
i , the belief state bi is updated to bi by
computing a Bayesian posterior distribution
X
0
bt+1
i (s ) ∝ bti (s)T (s0 | s, ati )Oi (ot+1 t 0
i | ai , s ). (3.5)
s∈S
The resulting belief state is exact in the sense that it retains all relevant infor-
mation from the observation history. This belief state is known as a “sufficient
statistic”, because it carries enough information needed to choose optimal
actions and to make predictions about the future. The process of updating belief
states based on observations is also known as (belief state) filtering.
Unfortunately, the space complexity of storing such exact belief states and the
time complexity of updating them using the Bayesian update from Equation 3.5
are each exponential in the number of variables that define the state, making
it intractable for complex environments. Hence, developing algorithms for
efficient approximate filtering has been the subject of much research; see, for
example, Albrecht and Ramamoorthy (2016) and discussions therein.
In a POSG with more than one agent, the definition of belief states and
how to update them becomes significantly more complex. In particular, since
agents may not observe the chosen actions of other agents and their resulting
observations, the agents now also have to infer probabilities over the possible
observations and actions of other agents, which in turn requires knowledge of
their observation functions and policies (e.g. Gmytrasiewicz and Doshi 2005;
Oliehoek and Amato 2016). However, as we will discuss in Section 3.6, in
MARL we typically assume that agents do not possess complete knowledge
about the elements of the POSG, such as S, T , and Oi (including their own
observation function), all of which are required in Equation 3.5. Thus, we will
not dwell further on the intricacies of defining exact belief states in multi-agent
contexts.
To achieve some form of filtering without knowledge of these elements, deep
RL algorithms often use recurrent neural networks (discussed in Section 7.5.2)
to process observations sequentially. The output vector of the recurrent network
learns to encode information about the current state of the environment, and
can be used to condition other functions of the agent such as value and policy
networks. Part II of this book will discuss how MARL algorithms can integrate
such deep learning models in order to learn and operate in POSGs. See also
Section 8.3 for a further discussion on this topic.
some partial information about the state st , and wt–1 j is the communication
message mt–1 j sent by agent j at time t – 1. To model noisy communication, Oi
could specify wt–1 t–1
j = f (mj ) where f can modify the message in some way. For
example, if mj is continuous-valued, then we could specify f (mt–1
t–1 t–1
j ) = mj + η
where η is a random noise component sampled from a Gaussian distribution.
Oi can also model message loss, by specifying a probability with which wt–1 j
is set to ∅ to represent the case that the message from agent j was not received.
Additional communication constraints can be modelled via Oi , such as limited
communication ranges by setting wt–1 j to ∅ if agents i and j are not within
a communication range. Again, since agents have access to the history of
observations, they can in principle remember past received messages.
The agents may use messages to communicate a variety of information,
such as sharing their own observations about the state of the environment, or
communicating their intended actions and plans. In the level-based foraging
example, the set Mi may contain a message corresponding to each possible
location in the grid world, and these messages could be used by the agents
to communicate their intended goal locations to other agents, or to tell other
agents about item locations they have discovered. However, in reinforcement
learning, the standard assumption is that agents do not know the meaning of
the actions in Ai , including the communication actions. (See Section 3.6 for
further discussion of knowledge assumptions.) Therefore, each communication
action in Mi is an abstract action for agent i just like any environment action in
Xi , and the agents have to learn the meaning of actions via repeated trials and
observations. For communication actions, there is also the added difficulty that
agents need to learn how to interpret the messages sent by other agents. This
opens the possibility that the agents can learn and evolve a shared language
or communication protocol (Foerster et al. 2016; Guo et al. 2022; Sukhbaatar,
Szlam, and Fergus 2016; Wang, He, et al. 2020).
The learning algorithms we will present in the later chapters are general
in that they learn policies over any given action sets Ai , which may or may
not include communication actions. While communication actions may be
modelled as part of Ai (such as discussed above), in the remainder of this book
we will keep our descriptions general and will not give special consideration to
communication actions.
In game theory, the standard assumption is that all agents have knowledge
of all components that define the game; this is referred to as a “complete
knowledge game” (Owen 2013). For normal-form games, this means that the
agents know the action spaces and reward functions of all agents (including
their own). For stochastic games and POSGs, agents also know the state space
and state transition function, as well as the observation functions of all agents.
Knowledge of these components can be utilised in different ways. For example,
if agent i knows the reward function Rj of another agent j, then agent i may be
able to estimate the best-response action (Section 4.2) of agent j, which in turn
could inform the optimal action for agent i. If an agent knows the transition
function T , it may predict the effects of its actions on the state and plan its
actions several steps into the future.
However, in most real-world applications of interest (such as those mentioned
in Chapter 1), it is infeasible to obtain accurate and complete specifications
of these components. For such applications, often the best we can obtain is
a simulator of these components that can generate samples of states, (joint)
rewards, and (joint) observations. For example, we may have access to a
simulator Tb that, given an input state s and joint action a, can produce samples
of joint rewards and successor states (r = (r1 , ..., rn ), s0 ) ∼ Tb (s, a), such that
n o Y
Pr Tb (s, a) = (r, s0 ) ≈ T (s0 | s, a) [Ri (s, a, s0 ) = ri ]1 (3.8)
i∈I
optimal decision making have been the subject of much research in game
theory (Perea 2012; Shoham and Leyton-Brown 2008) and agent modelling
(Albrecht and Stone 2018), in MARL they have played a relatively lesser role
since the standard assumption is that agents have no knowledge of most game
components. Exceptions can be found in MARL research for zero-sum or
common-reward games, where the specific reward structure is exploited in some
way in the algorithm design. Chapters 6 and 9 will describe several MARL
algorithms of this latter category.
Lastly, it is usually assumed that the number of agents in a game is fixed, and
that this fact is known by all agents. While outside the scope of this book, it
is interesting to note that recent research in MARL has begun to tackle open
multi-agent environments in which agents may dynamically enter and leave
the environment (Jiang et al. 2020; Rahman, Carlucho, et al. 2023; Rahman
et al. 2021).
3.8 Summary
This chapter introduced a hierarchy of increasingly complex game models, in
which multiple agents interact in a shared environment. In summary, the main
concepts are the following:
• The normal-form game is the most basic type of game model, defining a
single interaction between two or more agents. Each agent uses a policy that
assigns probabilities to the actions available to the agent. Given a joint action
that specifies an action for each agent, each agent receives a reward and the
interaction ends. Normal-form games with two agents are also called matrix
games, since the agent’s reward functions can be represented as matrices.
• Repeated normal-form games repeat the same normal-form game over multi-
ple time steps. The policies of agents can be conditioned on the history of
past joint actions chosen by the agents.
• Stochastic games additionally define an environment state that can change in
response to the agents’ actions. Agents can see the full environment state and
Figure 3.5: Synonymous terms in reinforcement learning (RL) and game theory
(GT). This book uses RL terminology.
60 Chapter 3
the previous actions of all other agents. The game terminates once it reaches
a terminal state or a maximum number of time steps; or it may continue for an
infinite number of time steps. Stochastic games include repeated normal-form
games as a special case in which there is only a single environment state, and
they include MDPs as a special case in which there is only a single agent.
• In partially observable stochastic games (POSGs), the agents make decisions
based on potentially incomplete and noisy observations about their environ-
ment. Instead of observing the complete environment state and the past
actions of other agents, POSGs define observation functions which generate
individual observations for each agent that depend on the state and past joint
action. The POSG model includes all of the other game models covered in
this book as special cases.
• Games can be categorised based on the relationship between the agents’
rewards. In zero-sum games, the agents’ rewards always sum up to zero. In
common-reward games, each agent receives the same reward. General-sum
games are the most general games and do not define any restrictions on the
relationship of reward functions.
• Communication can be modelled in stochastic games and POSGs by includ-
ing special communication actions. These actions can be observed by other
agents but do not affect the environment state. POSGs can also model noisy
and unreliable communication channels with limited range.
• Different assumptions can be made about what knowledge the agents have
about the different elements that define a game, such as the state transition
probabilities and reward functions. In MARL, the standard assumption is
that the agents do not know these elements.
Many of the dimensions of MARL listed in Figure 1.4 (page 9) are determined
in the specification of a game. Similarly to single-agent RL, a learning problem
in MARL is given by the combination of a game model and a learning objective
for the agents. In Chapter 4, we will introduce a range of solution concepts that
can be used as learning objectives in games.
use analogous definitions (e.g. by using densities and integrals) but may have
different existence properties for solutions.1
Game theory has produced a wealth of knowledge on foundational questions
about solution concepts, including: For a given type of game model and solution
concept, is a solution guaranteed to exist in the game? Is the solution unique
(i.e. only one solution exists) or are there potentially many solutions, even
infinitely many? And is a given learning and decision rule, when used by all
agents, guaranteed to converge to a solution? For a more in-depth treatment
of such questions, we recommend the books of Fudenberg and Levine (1998),
Owen (2013), Shoham and Leyton-Brown (2008), and Young (2004).
1. For example, every two-agent zero-sum normal-form game with finite action spaces has a unique
minimax game value, but zero-sum games with continuous actions exist that do not have a minimax
game value (Sion and Wolfe 1957). Sufficient conditions for equilibrium existence in continuous
games have been studied in works such as (Dasgupta and Maskin 1986; Debreu 1952; Fan 1952;
Glicksberg 1952).
returns the history of joint observations from the full history ĥt , and we some-
times abbreviate σ(ĥt ) to ht if it is clear from context. We define the probability
of a joint observation, O(ot | at–1 , st ), as the product i∈I Oi (oti | at–1 , st ).
Q
To make our definitions valid for both finite and infinite time steps, we use
discounted returns2 and assume the standard convention of absorbing states,
which we defined in Section 2.3. That is, once an absorbing state has been
reached, the game will subsequently always transition into that same state with
probability 1 and give a reward of 0 to all agents. To simplify our definitions
below, we also assume that the observation functions and policies of all agents
become deterministic (i.e. assign probability 1 to a certain observation and
action) once an absorbing state as been reached. Thus, our definitions encom-
pass infinite episodes as well as episodes that terminate after a finite number of
time steps, including non-repeated normal-form games which always terminate
after a single time step.3
In the following, we provide two equivalent definitions of expected returns.
The first definition is based on enumerating all full histories in the game, while
the second definition is based on a Bellman-style recursion of value computa-
tions. These definitions are equivalent, but can provide different perspectives
and have been used in different ways. In particular, the first definition resembles
a linear sum and may be easier to interpret, while the second definition uses
a recursion which can be operationalised such as in value iteration for games
(introduced in Section 6.1).
History-based expected return: Given a joint policy π, we can define the
expected return for agent i under π by enumerating all possible full histories
and summing the returns for agent i in each history, weighted by the probability
of generating the history under the POSG and joint policy π. Formally, define
the set Ĥ to contain all full histories ĥt for t → ∞.4 Then, the expected return
for agent i under joint policy π is given by
h i
Ui (π) = lim Eĥt ∼(µ,T ,O,π) ui (ĥt ) (4.1)
t→∞
X
= Pr(ĥt | π) ui (ĥt ) (4.2)
ĥt ∈Ĥ
2. Alternative definitions based on average rewards instead of discounted returns are also possible
for games (Shoham and Leyton-Brown 2008).
3. In the context of non-repeated normal-form games, the expected return for an agent simply
becomes the expected reward of that agent, so we use the term “expected return” in all game models
including non-repeated normal-form games.
4. Recall from Chapter 3 that we assume finite game models, and that at–1 before time t = 0 is the
empty set.
where hπi , π–i i denotes the complete joint policy consisting of πi and π–i . For
convenience and to keep our notation lean, we will sometimes drop the hi (e.g.
Ui (πi , π–i )).
Note that the best response to a given π–i may not be unique, meaning that
BRi (π–i ) may contain more than one best-response policy. For example, in a
non-repeated normal-form game, there may be multiple actions for agent i that
achieve equal maximum expected return against a given π–i , in which case any
probability assignment to these actions is also a best response. (We will see a
concrete example in the Rock-Paper-Scissors game in Section 4.3.)
Besides being useful for compact definitions of solution concepts, best
response operators have also been used in game theory and MARL to iteratively
compute solutions. Two example methods we will see include fictitious play
(Section 6.3.1) and joint action learning with agent modelling (Section 6.3.2),
among others.
4.3 Minimax
Minimax is a solution concept defined for two-agent zero-sum games, in which
one agent’s reward is the negative of the other agent’s reward (Section 3.1).
A classical example of such games is the matrix game Rock-Paper-Scissors
(see Figure 3.2a, page 46). More complex examples with sequential moves
include games such as chess and Go. The existence of minimax solutions for
normal-form games was first proven in the foundational game theory work of
von Neumann (1928) (see also von Neumann and Morgenstern (1944)).
= min
0
max
0
Ui (πi0 , πj0 ) (4.11)
πj πi
= –Uj (π).
Every two-agent zero-sum normal-form game has a minimax solution (von
Neumann and Morgenstern 1944). Minimax solutions also exist in every two-
agent zero-sum stochastic game with finite episode length, and two-agent zero-
sum stochastic games with infinite episode length using discounted returns
(Shapley 1953). Moreover, while more than one minimax solution may exist
in a game, all minimax solutions yield the same unique value Ui (π) for agent i
(and, thus, agent j). This value is also referred to as the (minimax) value of the
game.5 Minimax values can also be defined for zero-sum games with more than
two agents (see, for example, Equation 4.17 in Section 4.4).
In a minimax solution, each agent uses a policy which is optimised against
a worst-case opponent that seeks to minimise the agent’s return. Formally,
there are two parts in the definition of minimax. Equation 4.10 is the minimum
expected return that agent i can guarantee against any opponent. Here, πi is
agent i’s maxmin policy and Ui (π) is i’s maxmin value. Conversely, Equa-
tion 4.11 is the minimum expected return that agent j can force on agent i.
Here, we refer to j’s policy πj as the minmax policy against agent i, and Ui (π)
is agent i’s minmax value. In a minimax solution, agent i’s maxmin value is
equal to its minmax value. Another way to interpret this is that the order of the
min/max operators does not matter: agent i first announcing its policy followed
by agent j selecting its policy, is equivalent to agent j first announcing its policy
followed by agent i selecting its policy. Neither agent gains from such policy
announcements.
A minimax solution can be understood more intuitively as each agent using
a best-response policy against the other agent’s policy. That is, (πi , πj ) is a
minimax solution if πi ∈ BRi (πj ) and πj ∈ BRj (πi ). In the non-repeated Rock-
Paper-Scissors game, there exists a unique minimax solution which is for both
agents to choose actions uniformly randomly (i.e. assign equal probability to
all actions). This solution gives an expected return of 0 to both agents. In
5. Recall that we assume finite game models. For zero-sum games with continuous actions, examples
exist that do not have a minimax game value (Sion and Wolfe 1957).
fact, it can be verified6 in this game that if any agent i uses a uniform policy
πi , then any policy πj for the other agent j is a best-response policy to πi , and
all best-response policies πj ∈ BRj (πi ) yield an expected reward of 0 to agent j
(and to agent i). This example shows that best responses need not be unique,
and there can be many (even infinitely many) possible best-response policies.
However, the joint policy (πi , πj ) in which both policies are uniform policies
is the only joint policy in which both policies are best responses to each other,
making it a minimax solution. In Section 4.4, we will see that this best-response
relation can also be applied to the more general class of general-sum games.
x ai ≥ 0 ∀ai ∈ Ai (4.14)
X
xai = 1 (4.15)
ai ∈Ai
In this linear program, the constraints in Equation 4.13 mean that no single
action of agent j will achieve an expected return greater than Uj∗ against the
policy specified by πi (ai ) = xai . This implies that no probability distribution over
agent j’s actions will achieve a higher expected return than Uj∗ . Finally, the
constraints in Equations 4.14 and 4.15 ensure that the values of xai form a valid
6. Assume agent 1’s policy assigns probability 13 to each action R, P, S. The expected
reward to agent 2 for any policy π2 is: π2 (R)( 13 R2 (R, R) + 13 R2 (P, R) + 13 R2 (S, R)) +
π2 (P)( 13 R2 (R, P) + 13 R2 (P, P) + 13 R2 (S, P)) + π2 (S)( 13 R2 (R, S) + 13 R2 (P, S) + 13 R2 (S, S)) =
π2 (R)(– 13 + 13 ) + π2 (P)(+ 13 – 13 ) + π2 (S)(– 13 + 13 ) = 0. The same can be shown when switching the
roles of agents 1 and 2.
probability distribution. Note that in a minimax solution, we will have Ui∗ = Uj∗
(see Definition 5).
Linear programs can be solved using well-known algorithms, such as the
simplex algorithm which runs in exponential time in the worst case but often
is very efficient in practice; or interior-point algorithms which are provably
polynomial-time.
(Think: other agents –i minimise the maximum achievable return for agent i.)
Under these two conditions, we can construct an equilibrium solution which
uses π to achieve Û, and if at any time t any agent i deviates from its policy
7. The name “folk theorem” apparently came about because the general concept was known before
it was formalised.
8. We first encountered minmax in Section 4.3 where it was defined for two agents, while in
Equation 4.17 we define it for n agents (the mm superscript stands for minmax).
πi in π, the other agents will limit i’s return to vi by using their corresponding
mm
minmax policy π–i from Equation 4.17 indefinitely after t. Thus, since vi ≤ Ûi ,
i has no incentive to deviate from πi , making π an equilibrium.
Given a joint policy π, how can one check whether it is a Nash equilibrium?
Equation 4.16 suggests the following procedure, which reduces the multi-agent
problem to n single-agent problems. For each agent i, keep the other policies π–i
fixed and compute an optimal best-response policy πi0 for i. If the optimal policy
πi0 achieves a higher expected return than agent i’s policy πi in the joint policy π,
i.e. Ui (πi0 , π–i ) > Ui (πi , π–i ), then we know that π is not a Nash equilibrium. For
non-repeated normal-form games, πi0 can be computed efficiently using a linear
program (e.g. Albrecht and Ramamoorthy 2012). For games with sequential
moves, πi0 may be computed using a suitable single-agent RL algorithm.
C D
A 100,100 0,0
B 1,2 1,1
Figure 4.2: Matrix game to illustrate that an -Nash equilibrium (B,D) ( = 1)
may not be close to a real Nash equilibrium (A,C).
that an -Nash equilibrium may not be close to any real Nash equilibrium, in
terms of the expected returns produced by the equilibrium. In fact, the expected
returns under an -Nash equilibrium may be arbitrarily far away from those of
any Nash equilibrium, even if the Nash equilibrium is unique.
Consider the example game shown in Figure 4.2. This game has a unique
Nash equilibrium at (A,C).9 It also has an -Nash equilibrium at (B,D) for = 1,
in which agent 2 could deviate to action C to increase its reward by . First, note
that neither agent’s return under the -Nash equilibrium is within of its return
under the Nash equilibrium. Second, we can arbitrarily increase the rewards
for (A,C) in the game without affecting the -Nash equilibrium (B,D). Thus, in
this example, the -Nash equilibrium does not meaningfully approximate the
unique Nash equilibrium of the game.
To check that a joint policy π constitutes an -Nash equilibrium for some
given , we can use essentially the same procedure for checking Nash equilib-
rium described at the end of Section 4.4, except that we check for Ui (πi0 , π–i ) – >
Ui (πi , π–i ) to determine that π is not an -Nash equilibrium.
9. Saying that a joint action a is an X-equilibrium (e.g. Nash) is a shorthand for saying that a
deterministic joint policy π that assigns probability 1 to this joint action is an X-equilibrium.
S L
S 0,0 7,2
L 2,7 6,6
Figure 4.3: Chicken matrix game.
Now, consider the following joint policy πc which uses correlated actions:
• πc (L, L) = πc (S, L) = πc (L, S) = 13
• πc (S, S) = 0
The expected return under πc to both agents is: 7 · 13 + 2 · 13 + 6 · 13 = 5. It can
be verified that no agent has an incentive to unilaterally deviate from the action
recommended to it by πc , assuming knowledge of πc (and without knowing
the actions recommended to other agents). For example, say agent i received
action recommendation L. Then, given πc , i knows that agent j will choose S
with probability 0.5 and L with probability 0.5. Thus, the expected return for i
when choosing L is 2 · 21 + 6 · 12 = 4, which is higher than the expected return for
choosing S (0 · 12 + 7 · 12 = 3.5). Thus, i has no incentive to deviate from L.
The above example also illustrates how a correlated equilibrium can again
be described as mutual best responses between the agents. In correlated
equilibrium, each action recommendation ai given by πc is a best response
(i.e. achieves maximum expected return for i) to the conditional distribution
–i i)
π–i (a–i | ai ) = P 0πc (haπic,a(ha 0 ,a i) over actions a–i recommended to the other agents
–i
ai ∈Ai i
by πc given ai .
We can obtain an even more general class of equilibrium solutions, called
coarse correlated equilibrium (Moulin and Vial 1978),10 by requiring that the
inequality in Equation 4.19 only needs to hold for unconditional action modi-
fiers, which satisfy ∀a0i ∈ Ai : ξi (a0i ) = ai for some action ai . In other words, each
unconditional action modifier is just a constant action, with one ξi correspond-
ing to each action ai ∈ Ai . This solution concept means that each agent has to
decide upfront, before seeing its recommended action, whether to follow the
joint policy πc assuming that the other agents follow it. If no agent can select a
constant action (i.e. unconditional action modifier) to obtain a higher expected
return compared to its expected return under πc , as per Equation 4.19, then πc is
a coarse correlated equilibrium. Coarse correlated equilibria include correlated
equilibria as a special case in which Equation 4.19 must hold for all possible
action modifiers, not just unconditional action modifiers.
10. Moulin and Vial (1978) proposed this solution concept, but did not name it in their work.
xa ≥ 0 ∀a ∈ A (4.22)
X
xa = 1 (4.23)
a∈A
The constraints in Equation 4.21 ensure the property that no agent can gain
by deviating from the action a0i sampled under the joint policy π(a) = xa to a
different action a00i . The constraints in Equations 4.22 and 4.23 ensure that the
Note that these linear programs can contain many more variables and con-
straints than the linear programs for minimax solutions (Section 4.3.1). We now
have variables corresponding to each joint action a ∈ A, which can grow expo-
nentially in the number of agents (see also the discussion in Section 5.4.4). For n
agents with k actions each, Equation 4.22 specifies kn constraints, Equation 4.21
specifies nk2 constraints, and Equation 4.24 specifies nk constraints.
9
Joint return
8 Pareto-optimal
Deterministic NE
7 Probabilistic NE
0
0 1 2 3 4 5 6 7 8 9
Agent 2 expected reward
Figure 4.4: Feasible expected joint rewards and Pareto frontier in the Chicken
matrix game. Each blue dot shows the expected joint reward obtained by a joint
policy. Red squares show joint rewards of Pareto-optimal joint policies. Also
shown are the joint rewards corresponding to the deterministic and probabilistic
Nash equilibria of the non-repeated game.
For infinitely-repeated games and under the average reward,13 there exists
a folk theorem which shows that any expected joint return in this hull that is
equal to or larger than the minmax value of the agents (Equation 4.17) can be
realised by an equilibrium. In the Chicken game, it can be seen from the game’s
reward matrix that the minmax value for both agents is 2, and so any expected
joint return (U1 , U2 ) with U1 , U2 ≥ 2 in the hull can be realised by means of an
equilibrium. This is where Pareto optimality comes in: we can narrow down the
space of desirable equilibria by requiring that the expected joint return achieved
by an equilibrium also be Pareto-optimal. The Pareto-optimal joint returns are
those which reside on the Pareto frontier shown by the red squares. Thus, for
any given joint policy π, we can project its corresponding expected joint return
into the convex hull and detect Pareto optimality if the joint return lies on the
Pareto frontier.
We have presented Pareto optimality as a concept to refine equilibrium solu-
tions. However, note that a joint policy can be Pareto-optimal without being
an equilibrium solution. Yet, Pareto optimality on its own may not be a very
useful solution concept. In particular, all joint policies in zero-sum games are
Pareto-optimal by definition. In general-sum games, many joint policies may
be Pareto-optimal without being a desirable solution, such as joint policies that
are Pareto-optimal but result in large differences between the agents’ expected
returns. We will discuss this latter case further in Section 4.9.
13. Average returns (or average rewards) correspond to discounted returns with γ → 1.
14. This type of fairness is also known as Nash social welfare, defined as the geometric mean
Q 1
i∈I Ui (π)
n (Caragiannis et al. 2019; Fan et al. 2023).
15. The careful reader may have noticed some limitations of the simple definition of fairness in
Definition 11. For example, if Ui (π) = 0 for any agent, then it does not matter what returns the other
agents obtain under π. And if we allow Ui (π) < 0, then the expected joint return (–0.1, 1, 1) would
be less fair than the expected joint return (–0.1, –100, 100), which is counter-intuitive.
Joint return
10 Pareto-optimal
9 Deterministic NE
Probabilistic NE
8 Fairness-optimal
A B 1
A 10,7 2,2 0
0 1 2 3 4 5 6 7 8 9 10
B 0,0 7,10 Agent 2 expected reward
(a) Battle of the Sexes matrix game. (b) Joint rewards and fairness-optimal outcomes.
Figure 4.5: Feasible joint rewards and fairness-optimal outcomes in the Battle
of the Sexes matrix game. This game models a situation where two people
(classically, a man and a woman, hence the name of the game) want to meet
in one of two places (A or B) but have different preferences about the best
place. Agent 1 prefers place A and agent 2 prefers place B. Joint action (A,B) is
preferred over (B,A), since in the latter case both agents end up at their respective
least-preferred place. The two deterministic joint policies corresponding to
joint actions (A,A) and (B,B) are the only joint policies (in the non-repeated
game) that are both Pareto-optimal and fairness-optimal. The figure also shows
the only probabilistic Nash equilibrium, which is neither Pareto-optimal nor
fairness-optimal.
π have the same unique value Ui (π) = –Uj (π) for agents i, j. Therefore, all
minimax solutions achieve equal welfare and fairness, respectively (besides, the
welfare as defined here will always be zero).
It is easy to prove that welfare optimality implies Pareto optimality. To see
this, suppose a joint policy π is welfare-optimal but not Pareto-optimal. Since
π is not Pareto-optimal, there exists a joint policy π 0 such that ∀i : Ui (π 0 ) ≥
Ui (π) and ∃i : Ui (π 0 ) > Ui (π). However, it follows that i Ui (π 0 ) > i U(π)
P P
4.10 No-Regret
The equilibrium solution concepts discussed in the previous sections are based
on mutual best responses between agents, and are thus a function of the agents’
policies. Another category of solution concepts is based on the notion of regret,
which measures the difference between the rewards an agent received and the
rewards it could have received if it had chosen a different action (or policy) in
past episodes against the observed actions (or policies) of the other agents in
these episodes. An agent is said to have no-regret if, in the limit of infinitely
many episodes, the agent’s average regret across the episodes is at most zero.
Therefore, no-regret considers the performance of learning agents across mul-
tiple episodes, which is in contrast to the other solution concepts introduced
in this chapter that only consider a single joint policy (and not how this joint
policy was learned). In this sense, no-regret can be viewed as an example of
the prescriptive agenda discussed in Section 1.5, which is concerned with the
performance of agents during learning.
There are multiple ways in which regret can be defined. We will first give
a standard definition of regret for non-repeated normal-form games, which
is based on comparing the rewards of different actions in the episodes. This
definition will then be extended to sequential-move games. Let ae denote the
joint action from episodes e = 1, ..., z. Agent i’s regret for not having chosen the
best single action across these episodes is defined as
z
X
Regretiz = max Ri (hai , ae–i i) – Ri (ae ) .
(4.28)
ai ∈Ai
e=1
An agent is said to have no-regret if its average regret in the limit of z → ∞
is at most zero. As a solution concept, no-regret requires that all agents in the
game have no-regret.
Definition 12 (No-regret) In a general-sum game with n agents, the agents
have no-regret if
1
∀i : lim Regretiz ≤ 0. (4.29)
z→∞ z
Episode e 1 2 3 4 5 6 7 8 9 10
Action ae1 C C D C D D C D D D
Action ae2 C D C D D D C C D C
Reward R1 (ae ) -1 -5 0 -5 -3 -3 -1 0 -3 0
Reward R1 (hC, ae2 i) -1 -5 -1 -5 -5 -5 -1 -1 -5 -1
Reward R1 (hD, ae2 i) 0 -3 0 -3 -3 -3 0 0 -3 0
Figure 4.6: Ten episodes between two agents in the non-repeated Prisoner’s
Dilemma matrix game. The bottom two rows show agent 1’s rewards for always
choosing actions C/D against agent 2’s observed actions in the episodes.
action D was the “best” action against the observed actions of agent 2, and so
agent 1’s regret is Regret110 = –15 + 21 = 6 with an average regret (dividing by
10) of 0.6. Indeed, in Prisoner’s Dilemma, D is a dominant action in that it
is a best response against both D and C (see the reward matrix in Figure 3.2c
(page 46) to verify this). In order for agent 1 to achieve no-regret, the episodes
would need to continue in such a way that agent 1’s average regret across the
episodes goes to zero.
We can generalise the definition of no-regret to stochastic games and POSGs,
by redefining the regret over policies rather than actions. For each agent i ∈ I,
let Πi be a finite space of policies that agent i can select from. Let π e denote the
joint policy from episodes e = 1, ..., z, with πie ∈ Πi for all i ∈ I. Then, agent i’s
regret for not having chosen the best policy across these episodes is defined as
z
X
Regretiz = max e
i) – Ui (π e ) .
Ui (hπi , π–i (4.30)
πi ∈Πi
e=1
With this definition of regret, Definition 12 applies to all of the game models
introduced in Chapter 3. (For non-repeated normal-form games, Equation 4.30
is equivalent to Equation 4.28 if we define each Πi to be the set of deterministic
policies corresponding to each action ai ∈ Ai .)
Our example above in Prisoner’s Dilemma illustrates an important conceptual
limitation of regret, which is that it assumes that the actions or policies of other
agents –i remain fixed in the episodes. This assumption is sensible if the other
agents use constant policies that do not change between episodes. However, if
the other agents adapt their policies based on past episodes, then this assumption
is of course violated. Therefore, regret does not actually quantify what would
have happened under counterfactual situations. The second limitation, which
is a result of the first limitation, is that minimising regret is not necessarily
Most computer scientists will have some familiarity with the complexity
classes P and NP. P includes all decision problems whose solutions (if they
exist) can be computed efficiently in polynomial time in the size of the problem
instance. NP includes all decision problems whose solutions (if they exist) can
be computed in polynomial time by a non-deterministic Turing machine, and
in the worst case require exponential time by a deterministic Turing machine.
Unfortunately, these familiar complexity classes are not a good fit for the prob-
lem of computing an equilibrium, since P/NP characterise decision problems
which may or may not have solutions, while we know that games always have
at least one equilibrium solution. On the other hand, computing an equilibrium
which satisfies additional properties is a decision problem, since such solutions
may or may not exist. Such problems include computing equilibria which:
• are Pareto optimal
• achieve a certain minimum expected return for each agent
• achieve a certain minimum social welfare (sum of returns)
• assign zero or positive probability to certain actions of certain agents.
All of these problems are known to be NP-hard (Conitzer and Sandholm 2008;
Gilboa and Zemel 1989).
Some types of games and equilibria do admit polynomial-time algorithms.
Computing a minimax solution in a two-agent zero-sum non-repeated normal-
form game can be formulated via linear programming (Section 4.3.1), which can
be solved in polynomial time. Similarly, computing a correlated equilibrium in
a general-sum non-repeated normal-form game can be done in polynomial time
via a linear program (Section 4.6.1). However, computing a Nash equilibrium
in general-sum non-repeated normal-form games – henceforth simply called
NASH – cannot be solved via linear programming, due to the independence
assumption between policies in a Nash equilibrium.
Problems which always have solutions, such as NASH, are known as total
search problems. Section 4.11.1 will present one subclass of total search prob-
lems, called PPAD. It turns out that NASH is a complete problem in PPAD,
meaning that any other problem in PPAD can be reduced to NASH. We will
discuss the implications for MARL in Section 4.11.2.
Figure 4.7: Any instance of E ND - OF -L INE consists of a set of paths and cycles,
such as the instance shown here. Given the Parent / Child functions and a source
node (a node with no parent node; here shaded in grey), is there an efficient
polynomial-time algorithm to find a sink node (a node with no child node) or a
different source node?
complete problems to which all other problems in PPAD can be reduced.16 This
PPAD-complete problem is called E ND - OF -L INE, and is defined as follows:
Definition 13 (E ND - OF -L INE) Let G(k) = (V, E) be a directed graph consist-
ing of
• a finite set V containing 2k nodes (each node is represented as a bit-string of
length k)
• a finite set E = {(a, b) | a, b ∈ V} of directed edges (from node a to node b, for
a, b ∈ V) such that:
if (a, b) ∈ E then 6 ∃a0 6= a : (a0 , b) ∈ E and 6 ∃b0 6= b : (a, b0 ) ∈ E
Assume functions Parent(v) and Child(v) which, respectively, return the parent
node (if any) and child node (if any) of node v ∈ V. These functions are repre-
sented as boolean circuits with k input bits and k output bits, and run in time
polynomial in k. Given access to functions Parent and Child (but not E), and a
node s ∈ V such that Parent(s) = ∅, find a node e 6= s such that either Child(e) = ∅
or Parent(e) = ∅.17
Figure 4.7 shows an illustration of a E ND - OF -L INE problem instance. The
restriction on E in Definition 13 means that any node in the graph can have at
most one parent node and at most one child node. A node with no parent is
16. Problem A can be “reduced” to problem B if there exist polynomial-time algorithms to transform
any instance of A into an equivalent instance of B, and any solution of the instance of B to a solution
of the original instance of A.
17. To indicate that a node has no parent/child, the respective circuit function simply outputs the
same input node.
also called a source node, while a node with no child is called a sink node. The
“parity argument” in PPAD refers to the fact that any source node in this graph
always has a corresponding sink node. Therefore, if a source node s is given,
then we know that a node e must exist. The node e can, in principle, be found by
following the directed path starting at the given source node s. However, since
we are only given the functions Parent and Child (and not E), the only obvious
way to follow this path is by repeatedly calling the function Child starting at
the source node s. Therefore, finding a sink node may require exponential time
in the worst case, since there are 2k nodes.
Why should we care about PPAD? Just as it is unknown whether efficient
polynomial-time algorithms exist to solve NP-complete problems (the big
“P = NP?” question), it is also unknown whether efficient algorithms exist for
PPAD-complete problems (“P = PPAD?”). Indeed, PPAD contains problems for
which researchers have tried for decades to find efficient algorithms, including
the classic Brouwer fixed-point problem and finding Arrow-Debreu equilibria
in markets (see Papadimitriou (1994) for a more complete list). PPAD has also
been shown to be hard under cryptographic assumptions (Bitansky, Paneth,
and Rosen 2015; Choudhuri et al. 2019; Garg, Pandey, and Srinivasan 2016).
There are currently no known algorithms that can efficiently solve E ND - OF -
L INE (i.e. in time polynomial in k) and, thus, any other problem in PPAD.
Therefore, establishing that a problem is PPAD-complete is a good indicator
that no efficient algorithms exist to solve the problem.
exact equilibria (when = 0) in games with two agents.18 The use of -Nash
equilibrium is to account for the fact that a Nash equilibrium may involve
irrational-valued probabilities in games with more than two agents. Therefore,
the PPAD-completeness of NASH also includes approximation schemes for
computing Nash equilibria, such as MARL algorithms which may only learn
approximate solutions given a finite number of interactions in the game.
The implication for us is that MARL is unlikely to be a magic bullet for solv-
ing games: since no efficient algorithms are known to exist for PPAD-complete
problems, it is unlikely that efficient MARL algorithms exist to compute Nash
equilibria in polynomial time. Much of the research in MARL has focused on
identifying and exploiting structures (or assumptions) in certain game types
which lead to improved performance. However, the PPAD-completeness of
NASH tells us that, in general and without such assumed structures, it is likely
that any MARL algorithm still requires exponential time in the worst case.
4.12 Summary
This chapter introduced a range of solution concepts for games, to define optimal
policies for agents in the game. The main concepts are the following:
• A solution for a game is a joint policy (typically including one policy for
each agent) that satisfies certain conditions which are expressed in terms of
the expected returns for agents and the relationship between the returns.
• Many solution concepts can be compactly represented based on the notion
of best responses. A best-response policy maximises the expected returns of
one agent against a set of given policies for the other agents.
• There exists a series of increasingly general equilibrium solution concepts,
including minimax for two-agent zero-sum games, and Nash equilibrium and
(coarse) correlated equilibrium for general-sum games with two or more
agents. These equilibrium solutions are all based on the idea that every agent
is best-responding to all other agents under the equilibrium, and hence no
agent can unilaterally deviate from the equilibrium to increase its returns.
• Games may have a unique equilibrium solution, or they may have multiple
(even infinitely many) equilibria that can yield different expected returns for
18. As discussed in Section 4.5, an -Nash equilibrium may not be close to any actual Nash
equilibrium, even if the Nash equilibrium is unique. The problem of approximating an actual Nash
equilibrium within a desired distance, as measured in the policy space via norms such as L1 and L2,
is in fact much harder than computing an -Nash equilibrium. This latter problem, for general-sum
normal-form games with three or more agents, is complete for a different complexity class called
FIXP (Etessami and Yannakakis 2010).
s1
Joint policy
Histories
s2 s3
1. Another commonly used term is “training”. We use the terms learning and training
interchangeably, e.g. as in “learning/training a policy”.
a terminal state of the game, and different histories may have different
lengths te . Often, Dz contains the history so far from the current ongoing
episode z and the histories from previous episodes e < z.
Learning algorithm: The learning algorithm L takes the collected data Dz
and current joint policy π z , and produces a new joint policy,
π z+1 = L(Dz , π z ). (5.2)
0
The initial joint policy π is typically random.
Learning goal: The goal of learning is a joint policy π ∗ which satisfies the
properties of a chosen solution concept. Chapter 4 introduced a range of
possible solution concepts, such as Nash equilibrium.
We note several nuances in the above elements:
The chosen game model determines the conditioning of the learned joint
policy. In a non-repeated normal-form game (where episodes terminate after
one time step), policies πi are not conditioned on histories, i.e. they are simple
probability distributions over actions. In a repeated normal-form game, policies
are conditioned on action histories ht = (a0 , ..., at–1 ). In a stochastic game,
policies are conditioned on state-action histories ht = (s0 , a0 , s1 , a1 , ..., st ). In a
POSG, policies are conditioned on observation histories hti = (o0i , ..., oti ). These
conditionings are general and may be constrained depending on the desired
form of policies. For example, in a stochastic game we may condition policies
only on the current state of the game; and in a POSG we may condition policies
using only the most recent k observations.
The histories in Dz may in general be full histories (i.e. contain all states and
joint observations/actions; see Section 4.1) or one of the other types of histories
listed above. Therefore, the histories in Dz may contain more information than
the histories used to condition the agents’ policies. For example, this can be the
case in centralised training with decentralised execution regimes (discussed in
Section 9.1), in which a learning algorithm has access to the observations of all
agents during learning, while the agents’ policies only have access to the local
observations of agents.
The learning algorithm L may itself consist of multiple learning algorithms
that learn individual agent policies, such as one algorithm Li for each agent i.
Each of these algorithms may use different parts of the data in Dz , or use its
own data Diz such as in independent learning (Section 5.3.2). Furthermore, an
important characteristic of RL is that the learning algorithm is actively involved
in the generation of the data by exploring actions, rather than just passively
consuming the data. Thus, the policies produced by the learning algorithm may
actively randomise over actions to generate useful data for learning.
As was discussed in Chapter 4, a game may have more than one solution π ∗
and this depends on the specific solution concept. When we say “convergence
to a solution π ∗ ” the emphasis is on “a”, meaning that π ∗ is some valid solution
under the relevant solution concept.
When we make statements about the theoretical convergence of MARL algo-
rithms, unless specified otherwise, we mean convergence as per Equation 5.3.2
Of course, in practice we cannot collect infinite data, and learning typically
stops after a predefined budget is reached (such as a total allowed number of
episodes or time steps), or once the changes in policies are below some prede-
termined threshold. Whether a learned joint policy π z is in fact a solution can
be tested using procedures such as described in Sections 4.4 and 4.5.
Several other theoretical evaluation criteria have been studied in the literature,
including both weaker and stronger types of convergence. Weaker types of
convergence include:
• Convergence of expected return:
lim Ui (π z ) = Ui (π ∗ ), ∀i ∈ I (5.4)
z→∞
This convergence type means that, in the limit of infinite data z → ∞, the
expected joint return under the learned joint policy π z will converge to the
expected joint return of a solution π ∗ .
2. To be fully precise, some convergence statements in the literature add the additional qualifier
that convergence will happen “with probability 1” or “almost surely”. This qualifier means that,
theoretically, there may be conditions (i.e. trajectories of (π z )) under which convergence might not
happen, but that the probability of these conditions occurring is 0 under the relevant probability
measure. We omit this technicality in favour of simplicity.
episode e, let (aτ , hτ )tτ =0 be the sequence of joint actions aτ sampled from
the joint policy π e conditioned on history hτ , i.e. aτ ∼ π e (· | hτ ).3 (Note that e
denotes the episode number while τ denotes the time step within an episode.)
Let H z be the set containing all of the (aτ , hτ )-pairs from episodes e = 1, ..., z.
Then, the empirical distribution is given by
1 X
π̄ z (a | h) = z [a = aτ ]1 (5.6)
|H (h)| τ τ z
(a ,h )∈H (h)
3. For example, in a non-repeated normal-form game, aτ is the joint action selected by the agents
and hτ = ∅ since the policies are not conditioned on histories. In this book, we mainly refer to
empirical distributions in the context of non-repeated normal-form games.
These weaker convergence types are often used when a particular learning
algorithm is not technically able to achieve the convergence in Equation 5.3.
For example, the fictitious play algorithm presented in Section 6.3.1 learns
deterministic policies, meaning that it cannot represent probabilistic Nash equi-
libria, such as the uniform-random Nash equilibrium in Rock-Paper-Scissors.
However, in certain cases, it can be proven that the empirical action distribution
of fictitious play converges to a Nash equilibrium, as per Equation 5.5 (Fuden-
berg and Levine 1998). The infinitesimal gradient ascent algorithm presented
in Section 6.4.1 can learn probabilistic policies, but may still not converge
to a probabilistic Nash equilibrium, while it can be shown in non-repeated
normal-form games that the average rewards produced by the algorithm do
converge to the average rewards of a Nash equilibrium, as per Equation 5.8
(Singh, Kearns, and Mansour 2000). Lastly, the regret-matching algorithms pre-
sented in Section 6.5 can produce abrupt changes in the joint policy π z , which,
theoretically, may cause it to not converge to any one solution π ∗ . However, it
can be shown for normal-form games that the empirical distributions produced
by the regret-matching algorithms do converge to the set of (coarse) correlated
equilibria, as per Equation 5.7 (Hart and Mas-Colell 2000).
Note that Equation 5.3 implies all of the above weaker types of convergence.
However, it makes no claims about the performance of any individual joint
policy π z for a finite z. In other words, the above convergence types leave open
how the agents perform during learning. To address this, a stronger evaluation
criterion could require additional bounds, such as on the difference between π z
and π ∗ for finite z. (See also the discussion and references in Section 5.5.2.)
In complex games, it may not be computationally practical to check for these
convergence properties. Instead, a common approach is to monitor the expected
returns Ui (π z ) achieved by the joint policy as z increases, usually by visualising
learning curves that show the progress of expected returns during learning, as
shown in Figure 2.4 (page 37). A number of such learning curves will be shown
for various MARL algorithms presented in this book. However, this evaluation
approach may not establish any relationship to the solution π ∗ of the game. For
instance, even if the expected returns Ui (π z ) for all i ∈ I converge after some
finite z, the joint policy π z might not satisfy any of the convergence properties
of Equations 5.3 to 5.8.
To reduce notation in the remainder of this chapter (and book), we will omit
the explicit z-superscript (e.g. π z , Dz ) and we usually omit Dz altogether.
4. This exponential growth assumes that each additional agent comes with additional decision
variables. For example, in level-based foraging (Section 1.1), each agent comes with its own action
set. The reverse direction is when we have a fixed set of actions which are partitioned and assigned
to agents. In the latter case, the total number of actions remains constant, regardless of the number
of agents. We discuss this further in Section 5.4.4.
5. Recall that we use –i in the subscript to mean “all agents other than agent i” (e.g. A–i = ×j6=i Aj ).
where T is the game’s original state transition function defined over the joint
action space (Section 3.3).
As each agent j continues to learn and update its policy πj , the action proba-
bilities of πj in each state s may change. Thus, from the perspective of agent i,
the transition function Ti appears to be non-stationary, when really the only
parts that change over time are the policies πj of the other agents. As a result,
independent learning approaches may produce unstable learning and may not
converge to any solution of the game. Section 5.4.1 discusses non-stationarity
in MARL in further detail.
The learning dynamics of independent learning have been studied in various
idealised models (Barfuss, Donges, and Kurths 2019; Hu, Leung, and Leung
2019; Kianercy and Galstyan 2012; Leonardos and Piliouras 2022; Rodrigues
Gomes and Kowalczyk 2009; Wunder, Littman, and Babes 2010). For example,
Wunder, Littman, and Babes (2010) study an idealised model of IQL with
epsilon-greedy exploration (as shown in Algorithm 5) that uses infinitely small
learning steps α → 0, which makes it possible to apply methods from linear
dynamical systems theory to analyse the dynamics of the model. Based on this
idealised model, predictions about the learning outcomes of IQL can be made
for different classes of general-sum normal-form games with two agents and
two actions, which are summarised in Figure 5.2. These classes are primarily
characterised by the number of deterministic and probabilistic Nash equilibria
the games possess. As can be seen, this idealised version of IQL is predicted to
converge to a Nash equilibrium in some of the game classes, while in others
it may not converge at all or only under certain conditions. An interesting
finding of this analysis is that in games such as Prisoner’s Dilemma, which is a
member of class 3b, IQL can have a chaotic non-convergent behaviour which
Subclass 1a 1b 2a 2b 3a 3b
# deterministic NE 0 0 2 2 1 1
# probabilistic NE 1 1 1 1 0 0
Dominant action? No No No No Yes Yes
Det. joint act. > NE? No Yes No Yes No Yes
IQL converges? Yes No Yes Y/N Yes Y/N
results in rewards that average above the expected reward under the unique
Nash equilibrium of the game.
Despite their relative simplicity, independent learning algorithms still serve
as important baselines in MARL research. In fact, they can often produce
results which are competitive with state-of-the-art MARL algorithms, as shown
by the study of Papoudakis et al. (2021). In Chapter 9, we will see some of the
independent learning algorithms used in current MARL research.
1 1
Figure 5.3: Level-based foraging task used to compare central Q-learning and
independent Q-learning. This example task consists of a 11 by 11 grid world
with two agents and two items. Both agents have level 1, while one item has
level 1 and another item has level 2. All episodes start in this shown state with
the shown positions and levels.
next to the item, they simultaneously attempt to collect, and the sum of their
levels is equal or higher than the item’s level. Every episode begins with the
same start state shown in Figure 5.3, in which both agents have level 1 and are
initially located in the top corners of the grid, and there are two items located in
the centre with levels 1 and 2, respectively. Therefore, while the level-1 item
can be collected by any individual agent, the level-2 item requires that the two
agents collaborate in order to collect the item.
We model this learning problem as a stochastic game in which the agents
observe the full environment state, using a discounted return objective with
discount factor γ = 0.99. An agent receives a reward of 13 when collecting the
level-1 item, and both agents receive a reward of 13 when collecting the level-2
item; thus, the total (undiscounted) reward between the agents after collecting
all items is 1.6 Due to the reward discounting, the agents will need to collect
the items as quickly as possible in order to maximise their returns. Episodes
terminate when all items have been collected or after a maximum of 50 time
steps. In this example, both algorithms use a constant learning rate α = 0.01,
and an exploration rate which is linearly decayed from = 1 to = 0.05 over
the first 80,000 training time steps (recall from Section 2.7 that training time
steps are across the episodes). For CQL, we obtain a scalar reward (Line 8 in
Algorithm 4) by summing the individual rewards, i.e. rt = r1t + r2t .
6. See Section 11.3.1 for a more general definition of the reward function in level-based foraging.
1.0
IQL
0.8
CQL
Evaluation returns
0.6
0.4
0.2
0.0
0.0 0.2 0.4 0.6 0.8 1.0
Environment time steps 1e6
Figure 5.4 shows the evaluation returns7 achieved by CQL and IQL in the
level-based foraging task. Note that Figure 5.4 shows discounted returns with
discount factor γ = 0.99, hence the maximum achievable return is also less than
1. IQL learns to solve this task faster than CQL, which is a result of the fact
that the IQL agents only explore 6 actions in each state while the CQL agent
has to explore 62 = 36 actions in each state. This allows the IQL agents to learn
more quickly to collect the level-1 item, which can be seen in the early jump
in evaluation returns of IQL. Eventually, both IQL and CQL converge to the
optimal joint policy in this task, in which the agent in the right corner goes
directly to the level-2 item and waits for the other agent to arrive; meanwhile,
the agent in the left corner first goes to the level-1 item and collects it, and
then goes to the level-2 item and collects it together with the other agent. This
optimal joint policy requires 13 time steps to solve the task.
7. For a reminder of the term “evaluation return” and how these learning curves are produced, see
Section 2.7.
5.4.1 Non-Stationarity
A central problem in MARL is the non-stationarity resulting from the continual
co-adaptation of multiple agents as they learn from interactions with one another.
This non-stationarity can lead to cyclic dynamics, whereby a learning agent
adapts to the changing policies of other agents, which in turn adapt their policies
to the learning agent’s policy, and so on. An illustrative example of such learning
dynamics can be seen in Figure 5.5, in which two agents learn in the Rock-
Paper-Scissors matrix game using the WoLF-PHC algorithm (which will be
introduced in Section 6.4.4) to update their policies. The lines show how the
agents’ policies co-adapt over time until converging to the Nash equilibrium of
the game, in which both agents choose each action with equal probability.
While non-stationarity caused by the learning of multiple agents is a defining
characteristic of MARL, it is important to note that non-stationarity already
exists in the single-agent RL setting. To gain clarity on the problem, it is useful
to define the notion of stationarity and see how stationarity and non-stationarity
appear in single-agent RL, before moving to the multi-agent RL setting.
A stochastic process {X t }t∈N0 is said to be stationary if the probability dis-
tribution of X t+τ does not depend on τ ∈ N0 , where t and t + τ are time indices.
Intuitively, this means that the dynamics of the process do not change over time.
Now, consider a stochastic process X t which samples the state st at each time
step t. In a Markov decision process, the process X t is completely defined by the
state transition function, T (st | st–1 , at–1 ), and the agent’s policy, π, which selects
the actions a ∼ π(· | s). If this policy does not change over time (meaning that no
learning takes place), then it can be seen that the process X t is indeed stationary,
since st depends only on the state st–1 and action at–1 from the previous time
step (known as the Markov property), and at–1 depends only on st–1 via π(· | st–1 ).
Therefore, the process dynamics are independent of the time t.
However, in reinforcement learning, the policy π changes over time as a
result of the learning process. Using the definition of a learning process from
Section 5.1, the policy π z at time t is updated via π z+1 = L(Dz , π z ), where Dz
contains all data collected up to time t in the current episode z as well as data
0.6 0.4
π1(Paper)
0.4 0.6
0.2 0.8
Figure 5.5: Evolving policies of two agents, each using the WoLF-PHC
algorithm (Section 6.4.4) to update policies, in the non-repeated Rock-Paper-
Scissors matrix game. The diagonal dashed line divides the plot into two
probability simplexes, one for each of the two agents. Each point in the sim-
plex for an agent corresponds to a probability distribution over the agent’s
three available actions. Each line shows the agent’s current policy at episodes
0, 5, 10, 15, ..., 145 (marked by the dots), as well as the converged policy at
episode 100, 000 (marked by a star).
S L S H
S 0,0 7,2 S 4,4 0,3
L 2,7 6,6 H 3,0 2,2
(a) Chicken (b) Stag Hunt
maximum returns for all agents, there can be forces which make the agents
prone to converging to a sub-optimal equilibrium.
Consider the Stag Hunt matrix game shown in Figure 5.6b. This game
models a situation in which two hunters can either hunt a stag (S) or a hare (H).
Hunting a stag requires cooperation and yields higher rewards, while hunting
a hare can be done alone but gives lower rewards. It can be seen that the joint
actions (S,S) and (H,H) are two Nash equilibria in the game (there is also a third
probabilistic equilibrium, which we omit for clarity). While (S,S) yields the
maximum reward to both agents and is Pareto-optimal, (H,H) has a relatively
lower risk in that each agent can guarantee a reward of at least 2 by choosing H.
Thus, (S,S) is the reward-dominant equilibrium meaning that it yields higher
rewards than the other equilibria, and (H,H) is the risk-dominant equilibrium in
that it has lower risk (agents can guarantee higher minimum reward) than the
other equilibria. Algorithms such as independent Q-learning can be prone to
converging to a risk-dominant equilibrium if they are uncertain about the actions
of other agents. In the Stag Hunt example, in the early stages of learning when
agents choose actions more randomly, each agent will quickly learn that action
S can give a reward of 0 while action H gives a reward of 2 or higher. This
can steer the agents to assign greater probability to action H, which reinforces
the risk-dominant equilibrium in a feedback loop since deviating from H is
penalised if the other agent chooses H.
Various approaches can be considered to tackle equilibrium selection. One
approach is to refine the space of solutions by requiring additional criteria,
such as Pareto optimality and welfare/fairness optimality. As we saw in the
example discussed in Section 4.9, in some cases this may reduce an infinite
space of solutions to a single unique solution. In some types of games, the
structure of the game can be exploited for equilibrium selection. For instance,
minimax Q-learning (Section 6.2.1) benefits from the uniqueness of equilibrium
values in zero-sum games, meaning that all minimax solutions give the same
expected joint returns to agents. In no-conflict games such as Stag Hunt, in
which there always exists a Pareto-optimal reward-dominant equilibrium, the
Pareto Actor-Critic algorithm (Chapter 9) uses the fact that all agents know that
the reward-dominant equilibrium is also preferred by all the other agents. Agent
modelling (Section 6.3) can also help with equilibrium selection by making
predictions about the actions of other agents. In the Stag Hunt game, if agent 1
expects that agent 2 is likely to choose action S if agent 1 chose S in past time
steps, then agent 1 may learn to choose S more frequently and the process could
converge to both agents choosing (S,S). However, whether such an outcome
occurs depends on several factors, such as the details of the agents’ exploration
strategies and learning rates, and how the agent model is used by the agent.
If the agents are able to communicate with each other, then they may send
messages about their preferred outcomes or future actions to each other, which
could be used to agree on a particular equilibrium. However, communication
can bring its own challenges if the agents are not bound to the information they
communicate (such as deviating from their announced actions) and other agents
cannot verify the received information. Moreover, if different equilibria yield
different returns for the agents, then there may still be inherent conflicts about
which equilibrium is most preferred.
Note that the above example considers multi-agent credit assignment in one
specific time instance. Over time, this is further compounded by temporal credit
assignment. Agents must also learn to give appropriate credit to their own past
actions and those of other agents. For example, not only did the collect action of
the agent on the left not contribute to the +1 reward, but neither did its previous
move actions that brought it to its current location.
To resolve multi-agent credit assignment, an agent needs to understand the
effect of its own actions on the received reward versus the effect of the other
agents’ actions. Ascribing values to joint actions, as is done in central learning,
is a useful way to disentangle each agent’s contribution to a received reward.
As a simple example, consider the Rock-Paper-Scissors matrix game from
Section 3.1. During training, suppose the two agents choose actions (a1 , a2 ) =
(R, S) and agent 1 receives a reward of +1 (because rock beats scissors), then
the agents choose actions (a1 , a2 ) = (R, P) and agent 1 receives a reward of
–1 (because paper beats rock). If agent 1 uses an action value model Q(s, a1 )
which ascribes values to its own actions (as in independent Q-learning, see
Algorithm 5), then the average value of taking action R may appear to be 0
since Q does not explicitly model the impact of agent 2’s action. In contrast, a
joint action value model Q1 (s, a1 , a2 ) (as in central Q-learning, see Algorithm 4)
can correctly represent the impact of agent 2’s action, by ascribing different
values to joint actions (R, S) and (R, P). Besides central learning algorithms, the
class of joint action learning algorithms (introduced in Chapter 6) uses such
joint action values to learn solutions for games.
tractable, we could factor the action vector into n smaller vectors and use n
agents, one for each of the smaller action vectors. Each agent now deals with a
1000
smaller action space, for example |Ai | = k n if the factored action vectors have
the same length. However, note that the total number of joint actions between
agents, |A| = |A1 | · ... · |An | = k1000 , is independent of the number of agents n.
Indeed, while exponential growth due to the number of agents is an important
challenge in MARL, it is not unique to MARL. Single-agent reductions of the
problem, such as central learning, are still faced with an exponential growth
in the number of actions. Moreover, other approaches to optimal decision
making in multi-agent systems, such as model-based multi-agent planning (e.g.
Oliehoek and Amato 2016), also have to handle the exponential growth. Still,
scaling efficiently with the number of agents is an important goal in MARL
research. Part II of this book will introduce deep learning techniques as one
way to improve the scalability of MARL algorithms.
5.5.1 Self-Play
The term self-play has been used to describe two related but distinct modes
of operation in MARL. The first definition of self-play simply refers to the
assumption that all agents use the same learning algorithm (Banerjee and Peng
2004; Bowling and Veloso 2002; Chakraborty and Stone 2014; Conitzer and
Sandholm 2007; Powers and Shoham 2005; Shoham, Powers, and Grenager
2007; Wunder, Littman, and Babes 2010). Developing algorithms that converge
to some type of equilibrium solution in self-play is at the core of much of the
literature in MARL. Essentially all of the MARL algorithms introduced in
this book operate in this way. For independent learning, self-play is usually
implicitly assumed (such as in independent Q-learning), but note that this is not
strictly a requirement since the agents may use different algorithms within the
independent learning approach.
This definition of self-play is rooted in game theory, specifically the literature
on “interactive learning” (Fudenberg and Levine 1998; Young 2004), which
studies basic learning rules for players (i.e. agents) and their ability to converge
5.5.2 Mixed-Play
Mixed-play describes the case in which the agents use different learning algo-
rithms. One example of mixed-play can be seen in trading markets, in which the
agents may use different learning algorithms developed by the different users or
organisations which control the agents. Ad hoc teamwork (Mirsky et al. 2022;
Stone et al. 2010) is another example, in which agents must collaborate with
previously unknown other agents whose behaviours may be initially unknown.
The study of Albrecht and Ramamoorthy (2012) considered such mixed-
play settings and empirically compared various learning algorithms, including
Nash-Q (Section 6.2), JAL-AM (Section 6.3.2), WoLF-PHC (Section 6.4.4), and
a variant of regret matching (Section 6.5), in many different normal-form games
using a range of metrics, including several of the solution concepts discussed
in Chapter 4. The study concluded that there was no clear winner among the
tested algorithms, each having relative strengths and limitations in mixed-play
settings. While Papoudakis et al. (2021) provide a benchmark and comparison of
contemporary deep learning-based MARL algorithms (such as those discussed
in Chapter 9) for self-play in common-reward games, there is currently no such
study for deep learning-based MARL algorithms in mixed-play settings.
MARL research has also developed algorithms which aim to bridge (algo-
rithm) self-play and mixed-play. For example, the agenda of converging to
an equilibrium solution in self-play was extended with an additional agenda
to converge to a best-response policy if other agents use a stationary policy
(Banerjee and Peng 2004; Bowling and Veloso 2002; Conitzer and Sandholm
2007). Algorithms based on targeted optimality and safety (Powers and Shoham
2004) assume that other agents are from a particular class of agents and aim to
achieve best-response returns if the other agents are indeed from that class, and
otherwise achieve at least maxmin (“security”) returns that can be guaranteed
against any other agents. For example, we could assume that other agents use a
particular policy representation, such as finite state automata or decision trees,
or that their policies are conditioned on the previous x observations from the
history (Chakraborty and Stone 2014; Powers and Shoham 2005; Vu, Powers,
and Shoham 2006). In self-play, these algorithms aim to produce Pareto-optimal
outcomes (Shoham and Leyton-Brown 2008).
5.6 Summary
In this chapter, we made a first foray into using reinforcement learning to learn
solutions for games, and we discussed some of the key challenges in the learning
process. The main concepts from this chapter are the following:
• MARL algorithms are designed to learn a joint policy that satisfies the prop-
erties of a specific solution concept (e.g. Nash equilibrium). The data used
for learning consists of a set of histories (containing the observations, actions,
and rewards of one or more agents) from multiple episodes in the game.
• Different types of convergence criteria exist to analyse and evaluate the
learning performance of MARL algorithms. The standard theoretical criterion
is convergence of the learned joint policy to a solution of the game. Weaker
criteria include convergence of the empirical distribution of joint actions
across episodes to a solution joint policy; and convergence of the averaged
returns across episodes to the expected returns under a solution joint policy.
• Two basic approaches to applying RL in games reduce the multi-agent learn-
ing problem to a single-agent learning problem. Central learning applies
single-agent RL to the space of joint actions to learn a central policy that
chooses actions for each agent. Independent learning uses single-agent RL
independently for each agent to learn independent policies, without explicitly
representing the presence of other agents.
• A key challenge in MARL is the environment non-stationarity caused by
the concurrent learning of multiple agents. Essentially, from the perspective
of each agent, the environment appears non-stationary as the policies of the
other agents are changing over time, which breaks the Markov assumption
in game models. This can lead to cyclic dynamics where each agent ties to
adapt to the changing policies of the other agents, also known as the moving
target problem.
• Equilibrium selection is another key challenge, which can occur whenever
the game has multiple equilibria and where different equilibria yield different
expected returns for the agents. Thus, the agents face the problem of what
equilibrium they should agree to converge to, and how such agreement can
be achieved during learning.
• Additional challenges include the multi-agent credit assignment problem,
in which agents have to determine during learning whose actions among
multiple agents contributed to the received rewards; as well as scaling to
many agents and dealing with an exponential growth of the joint action space.
• Self-play and mixed-play are two basic modes of operation in MARL. We
describe two types of self-play: in algorithm self-play, every agent uses
the same learning algorithm; while in policy self-play, an agent’s policy is
directly trained against itself. Mixed-play describes a scenario in which the
agents use different learning algorithms.
In Chapter 6, we will go beyond the basic central and independent learning
approaches introduced in this chapter, by describing several families of more
specialised MARL algorithms that explicitly model and use certain aspects of
the multi-agent interaction. These algorithms use different approaches to deal
with one or more of the above challenges, which can enable them to successfully
learn different types of game solutions under certain conditions.
1. Technically, Mi,s is a hypermatrix if the game has more than two agents.
X
∗
T (s0 | s, a) Ri (s, a, s0 ) + γVi∗ (s0 ) .
Mi,s = (6.2)
s0 ∈S
Given value functions Vi∗
for each i ∈ I, the corresponding minimax policies
of the stochastic game for each agent, πi (ai | s), are obtained by computing the
∗ ∗
minimax solution in the non-repeated normal-form game given by M1,s , ..., Mn,s ,
for each state s ∈ S. Notice that, analogous to optimal policies in MDPs, the
minimax policies in the stochastic game are conditioned only on the state rather
than state-action histories.
Notice that Equation 6.4 resembles the value update in MDP value iteration
defined in Equation 2.47 and repeated below,
X
T (s0 | s, a) R(s, a, s0 ) + γV(s0 )
V(s) ← max (6.5)
a∈A
s0 ∈S
but replaces the maxa -operator with the Valuei -operator. In fact, in stochastic
games with a single agent (i.e. MDPs), the value iteration algorithm discussed
here reduces to MDP value iteration.2 This can be seen as follows: First,
in stochastic games with a single agent i, the value update becomes Vi (s) ←
Valuei (Mi,s ) where Mi,s is a vector of action values for agent i in state s. Next,
recall the definition of the maxmin value from Equation 4.10 (page 66), which
simply becomes maxπi Ui (πi ) in the single-agent case. In the single-agent case
(i.e. MDPs), we know that there is always a deterministic optimal policy which
in each state chooses an optimal action with probability 1. Using all of the
2. Hence, despite the focus on stochastic games, Shapley (1953) can also be regarded as an early
inventor of value iteration for MDPs.
Specifically, the JAL algorithms we present in this section are all off-policy
∗
algorithms which aim to learn equilibrium Q-values, Qπi , where π ∗ is an
equilibrium joint policy for the stochastic game. To reduce notation, we will
drop the π from Qπi unless required.
In contrast to using single-agent Q-values Q(s, a), using joint action values
Qi (s, a1 , ..., an ) alone is no longer enough to select the best action for agent i in
a given state, i.e. finding maxai Qi (s, a1 , ..., an ), since it depends on the actions
of the other agents in that state. Moreover, since a game can have multiple
equilibria which may yield different expected returns for the agents, learning
∗
Qπi requires some way to agree on a particular equilibrium (we discussed
this equilibrium selection problem in Section 5.4.2). Therefore, using joint
action values requires additional information or assumptions about the actions
of other agents in order to select optimal actions and to compute target values
for temporal-difference learning. In this section, we will discuss a class of JAL
algorithms that use solution concepts from game theory to derive this additional
information; hence, we will refer to them as JAL-GT.
The underlying idea in JAL-GT algorithms is that the set of joint action
values Q1 (s, ·), ..., Qn (s, ·) can be viewed as a non-repeated normal-form game
Γs for state s, in which the reward function for agent i is given by
Ri (a1 , ..., an ) = Qi (s, a1 , ..., an ). (6.10)
As convenient notation, we may also write this as Γs,i (a) for joint actions a.
Visually, for a stochastic game with two agents (i and j) and three possible
actions for each agent, the normal-form game Γs = {Γs,i , Γs,j } in state s can be
written as
with an analogous matrix Γs,j for agent j using Qj . The notation ai,k means that
agent i uses the k-th action, and similarly for aj,k and agent j.
We can solve the normal-form game Γs using existing game-theoretic solution
concepts, such as minimax or Nash equilibrium, to obtain an equilibrium joint
policy πs∗ . Note that πs∗ is a joint policy for a non-repeated normal-form game
(Γs ), while π ∗ is the equilibrium joint policy we seek to learn for the stochastic
game. Given πs∗ , action selection in state s of the stochastic game is simply done
by sampling a ∼ πs∗ with some random exploration (e.g. -greedy). Update
∗
targets for temporal-difference learning of Qπi are derived by using agent i’s
expected return (or value) of the solution to the game Γs0 for the next state s0 ,
given by X
Valuei (Γs0 ) = Γs0 ,i (a)πs∗0 (a) (6.11)
a∈A
where πs∗0 is the equilibrium joint policy for the normal-form game Γs0 .
The pseudocode for a general JAL-GT algorithm for stochastic games is
shown in Algorithm 7. The algorithm observes the actions and rewards of
all agents in each time step, and maintains joint action value models Qj for
every agent j ∈ I, in order to produce the games Γs . Several well-known MARL
A
Goal Goal
B
Figure 6.1: Simplified grid-world soccer game with two agents (A and B). The
circle marks ball possession. Based on Littman (1994).
minimax Q independent Q
% won ep. len. % won ep. len.
vs. random 99.3 13.89 99.5 11.63
vs. hand-built 53.7 18.87 76.3 30.30
vs. optimal 37.5 22.73 0 83.33
Figure 6.2: Percentage of won episodes and average episode length (time steps)
in the simplified soccer game, when policies learned by minimax Q-learning
and independent Q-learning are tested against random, hand-built, and optimal
opponents. Based on Littman (1994).
Figure 6.2 shows the percentage of won episodes and average episode length3
for policies learned by minimax Q-learning and independent Q-learning when
tested against the random, hand-built, and optimal opponents. Against the
random opponent, the policies of both algorithms won in almost all games and
achieved similar episode lengths. Against the hand-built opponent, minimax
Q-learning achieved a win rate of 53.7% which is close to the theoretical 50%
of the exact minimax solution. Independent Q-learning achieved a higher win
rate of 76.3% because its learned policy was able to exploit certain weaknesses
in the hand-built opponent. On the other hand, against the optimal opponent,
minimax Q-learning achieved a win rate of 37.5%. The fact that this win rate
was a little further from the theoretical 50% suggests that the algorithm had not
fully converged during learning, resulting in weaknesses in the policy which
3. The original work (Littman 1994) showed the number of completed episodes within the 100,000
time steps in testing. We find the average episode length (100,000 divided by number of completed
games) more intuitive to interpret.
the optimal opponent was able to exploit in some situations. The policy learned
by independent Q-learning lost in all episodes against its optimal opponent.
This is because any fully-deterministic policy can be exploited by an optimal
opponent. These results show that the policies learned by minimax Q-learning
do not necessarily exploit weaknesses in opponents when they exist, but at the
same time these policies are robust to exploitation by assuming a worst-case
opponent during training.
here is that all agents consistently choose either global optima or saddle points
when computing target values.
send
r = (0,3)
keep keep
r = (1,0) r = (3,1)
s1 s2
send
r = (0,0)
choosing keep. In fact, this game has a unique probabilistic stationary equilib-
rium which is π1 (send | s1) = 23 , π2 (send | s2) = 12
5
. Zinkevich, Greenwald, and
Littman (2005) refer to this class of games as NoSDE games (which stands for
“No Stationary Deterministic Equilibrium”) and proved the following theorem:
This result establishes that the joint action value models learned by JAL-GT
algorithms may not carry sufficient information to derive the correct equilibrium
joint policy in a game. Intuitively, the left-hand equality in Equation 6.12 means
that a JAL-GT algorithm using Qπ,Γ i would derive the same actions for agent i
0 0
as when using Qπi ,Γ , for every state s ∈ S in both stochastic games Γ and Γ0 .
However, the right-hand inequality in Equation 6.12 means that the joint policies
π and π 0 actually yield different expected returns for some agent i between the
two stochastic games. In essence, this discrepancy is due to the fact that the
unique equilibrium π 0 of Γ0 requires specific action probabilities in some states,
such as in the example given above, which cannot be computed using the
Q-functions alone.
4. The term “opponent modelling” was originally used because much of the early research in
this area focused on competitive games, such as chess. We prefer the more neutral term “agent
modelling” since other agents may not necessarily be opponents.
5. The survey of Albrecht and Stone (2018) was followed by a dedicated special journal issue on
agent modelling which contains further articles on the topic (Albrecht, Stone, and Wellman 2020).
Observations Prediction
Agent model
(past states, actions, ...) (action, goal, beliefs, ...)
Figure 6.4: An agent model makes predictions about another agent (such as the
action probabilities, goals, and beliefs of the agent) based on past observations
about the agent.
The most common type of agent modelling used in MARL is called policy
reconstruction. Policy reconstruction aims to learn models π̂j of the policies
πj of other agents based on their observed past actions. In general, learning
the parameters of π̂j can be framed as a supervised learning problem using the
past observed state-action pairs {(sτ , aτj )}t–1
τ =1 of the modelled agent. Thus, we
may choose a parameterised representation for π̂j and fit its parameters using
the (sτ , aτj )-data. Various possible representations for π̂j could be used, such as
look-up tables, finite state automata, and neural networks. The chosen model
representation should ideally allow for iterative updating to be compatible with
the iterative nature of RL algorithms.
Given a set of models for the other agents’ policies, π̂–i = {π̂j }j6=i , the mod-
elling agent i can select a best-response policy (Section 4.2) with respect to
these models,
πi ∈ BRi (π̂–i ). (6.13)
Thus, while in Chapter 4 we used the concept of best-response policies as
a compact characterisation of solution concepts, in this section we will see
how the best-response concept is operationalised in reinforcement learning.
In the following subsections, we will see different methods that use policy
reconstruction and best responses to learn optimal policies.
At the start of the first episode, before any actions have been observed from
agent j (i.e. C(aj ) = 0 for all aj ∈ Aj ), the initial model π̂j can be the uniform
distribution π̂j (aj ) = |A1j | .
In each episode, each agent i chooses a best-response action against the
models π̂–i = {π̂j }j6=i , given by
X Y
BRi (π̂–i ) = arg max Ri (hai , a–i i) π̂j (aj ) (6.15)
ai ∈Ai
a–i ∈A–i j 6= i
6. The agents could also start with different initial actions and then follow non-identical trajectories;
but they would still follow a similar spiral pattern and converge to the Nash equilibrium.
0.6 0.4
π1(Paper)
0.4 0.6
0.2 0.8
empirical action distributions change and co-adapt over time, converging to the
only Nash equilibrium of the game which is for both agents to choose actions
uniform-randomly.
A number of modifications of fictitious play have been proposed in the
literature to obtain improved convergence properties and generalise the method
(e.g. Fudenberg and Levine 1995; Heinrich, Lanctot, and Silver 2015; Hofbauer
and Sandholm 2002; Leslie and Collins 2006; Young 2004). Note that in
repeated normal-form games, the best response defined in Equation 6.15 is
myopic, in the sense that this best response only considers a single interaction
(recall that fictitious play is defined for non-repeated normal-form games).
However, in repeated normal-form games, the future actions of agents can
depend on the history of past actions of other agents. A more complex definition
of best responses in sequential game models, as defined in Equation 4.9, also
Episode e Joint action (ae1 , ae2 ) Agent model π̂2 Agent 1 action values
1 R,R (0.33, 0.33, 0.33) (0.00, 0.00, 0.00)
2 P,P (1.00, 0.00, 0.00) (0.00, 1.00, –1.00)
3 P,P (0.50, 0.50, 0.00) (–0.50, 0.50, 0.00)
4 P,P (0.33, 0.67, 0.00) (–0.67, 0.33, 0.33)
5 S,S (0.25, 0.75, 0.00) (–0.75, 0.25, 0.50)
6 S,S (0.20, 0.60, 0.20) (–0.40, 0.00, 0.40)
7 S,S (0.17, 0.50, 0.33) (–0.17, –0.17, 0.33)
8 S,S (0.14, 0.43, 0.43) (0.00, –0.29, 0.29)
9 S,S (0.13, 0.38, 0.50) (0.12, –0.38, 0.25)
10 R,R (0.11, 0.33, 0.56) (0.22, –0.44, 0.22)
accounts for the long-term effects of actions. In the following sections, we will
see how MARL can combine a best-response operator with temporal-difference
learning to learn non-myopic best-response policies.
Note that the action counts C(s, aj ) include actions observed in the current
episode and previous episodes. If a state s is visited for the first time, then π̂j
can assign uniform probabilities to actions. Conditioning the model π̂j on the
state s is a sensible choice, since we know that the optimal equilibrium joint
policy in a stochastic game requires only the current state rather than the history
of states and actions.
If the true policy πj is fixed or converges, and is indeed conditioned only on the
state s, then in the limit of observing each (sτ , aτj )-pair infinitely often, the model
π̂j will converge to πj . Of course, during learning, πj will not be fixed. To track
changes in πj more quickly, one option is to modify Equation 6.16 to give greater
weight to more recent observed actions in state s. For example, the empirical
probability π̂j (aj | s) may only use up to the most recent 10 observed actions of
agent j in state s. Another option is to maintain a probability distribution over a
space of possible agent models for agent j, and to update this distribution based
on new observations via Bayesian learning (we will discuss this approach in
Section 6.3.3). In general, many different approaches can be considered for
constructing an agent model π̂j from observations (Albrecht and Stone 2018).
JAL-AM algorithms use the learned agent models to select actions for the
learning agent and to derive update targets for the joint action values. Given
agent models {π̂j }j6=i , the value (expected return) to agent i for taking action ai
1.0
0.8
Evaluation returns
0.6
0.4
0.2
IQL
CQL
JAL-AM
0.0
0.0 0.2 0.4 0.6 0.8 1.0
Environment time steps 1e6
Figure 6.7: Joint action learning with agent modelling (JAL-AM) compared
to central Q-learning (CQL) and independent Q-learning (IQL) in the level-
based foraging task from Figure 5.3. Results are averaged over 50 independent
training runs. The shaded area shows the standard deviation over the averaged
returns from each training run. All algorithms use a constant learning rate
α = 0.01 and an exploration rate which is linearly decayed from = 1 to = 0.05
over the first 80,000 time steps.
in state s is given by
X Y
AVi (s, ai ) = Qi (s, hai , a–i i) π̂j (aj | s) (6.17)
a–i ∈A–i j6=i
where Qi is the joint action value model for agent i learned by the algorithm.
Using the action values AVi , the best-response action for agent i in state s is
given by arg maxai AVi (s, ai ); and the update target in the next state s0 uses
maxai AVi (s0 , ai ).
Algorithm 8 provides pseudocode for a general JAL-AM algorithm for
stochastic games. Similar to fictitious play, this algorithm learns best-response
actions rather than best-response policies. Both JAL-AM and JAL-GT are
off-policy temporal-difference learning algorithms. However, unlike JAL-GT,
which requires observing the rewards of other agents and maintaining joint
action value models Qj for every agent, JAL-AM does not require observing
the rewards of other agents and only maintains a single joint action value model
Qi for the learning agent.
Figure 6.7 shows the evaluation returns of JAL-AM using agent models
defined in Equation 6.16, compared to CQL and IQL in the level-based foraging
task from Figure 5.3 (page 100). The learning curves for CQL and IQL are
copied over from Figure 5.4. In this specific task, JAL-AM converged to the
optimal joint policy faster than IQL and CQL. The agent models learned by the
JAL-AM agents reduced the variance in their update targets, as is reflected by
the smaller standard deviation in evaluation returns compared to IQL and CQL.
This allowed JAL-AM to converge (on average) to the optimal joint policy after
about 500,000 training time steps, while IQL required about 600,000 training
time steps to converge to the optimal joint policy.
7. The Bayesian learning approach discussed in this section is also known as type-based reasoning
(Albrecht, Crandall, and Ramamoorthy 2016; Albrecht and Stone 2018).
C,D C C,D
C D
C -1,-1 -5,0 D
C C D
D 0,-5 -3,-3
(a) Prisoner’s Dilemma (b) Agent model “Coop” (c) Agent model “Grim”
Figure 6.8: Two agent models for Prisoner’s Dilemma. The models are shown
as simple finite state automata, where the state (circle) specifies the model’s
action (C for cooperating, D for defecting) and the state transition arrows are
labelled by the previous action of the other agent. Starting states are shown
underlined. The model Coop always cooperates. The model Grim cooperates
initially until the other agent defects, after which Grim defects indefinitely.
policy, how should agent 1 choose its first action? Consider the following cases
(summarised in Figure 6.9):
• If agent 1 cooperates initially and agent 2 uses Coop, then agent 1 has
not learned anything new about agent 2’s model (meaning the probabilities
assigned by agent 1’s belief to the models will not change), since both Coop
and Grim cooperate in response to agent 1 cooperating. If agent 1 cooperates
throughout the episode, it will obtain a return of 10 · (–1) = –10. Therefore,
agent 1 will get a lower return than it could have achieved by using the defect
action against Coop.
• On the other hand, if agent 1 defects initially and agent 2 uses Coop, then
agent 2 will continue to cooperate and agent 1 will learn that agent 2 uses
Coop (meaning that agent 1’s belief will assign probability 1 to Coop), since
Grim would not cooperate in this case. With this new knowledge about
agent 2, agent 1 can achieve maximum returns by defecting throughout the
episode, resulting in a return of 10 · 0 = 0.
• However, if agent 1 defects initially but agent 2 uses Grim, then agent 2 will
defect subsequently and agent 1 will learn that agent 2 uses Grim, which
will lead agent 1 to continue defecting as its best response. Therefore, this
new knowledge about agent 2 comes at the cost of a lower return, which is
0 + 9 · (–3) = –27 (one reward of 0, followed by –3 rewards until the end of
the episode).
This example illustrates that some actions may reveal information about the
policies of other agents, and the resulting more accurate beliefs can be used
to maximise the returns; however, the same actions may carry a risk in that
they may inadvertently change the other agents’ behaviours and result in lower
achievable returns.
In the above example, if agent 1 uses randomised exploration methods such
as -greedy exploration, then eventually it will defect, which will cause agent 2
to go into the “defect mode” if it uses Grim. VI instead evaluates the different
constellations of agent 1’s models and actions, and the impact on agent 1’s
future beliefs and actions under the different constellations, essentially as we
did in the above example. Depending on how far VI looks into the future
when evaluating an action, it may decide that exploring the defect action is
not worthwhile considering the potential cost (i.e. lower achievable return) if
agent 2 uses Grim. In this case, if both agents maintain beliefs over Coop/Grim
and use VI, they may learn a joint policy in which both agents cooperate, as we
will show later in this section.
We now describe these ideas more formally, starting with beliefs over agent
models. Assume we control agent i. Let Π̂j be the space of possible agent
models for agent j, where each model π̂j ∈ Π̂j can choose actions based on
the interaction history ht . (Recall that we assume stochastic games, in which
agents fully observe the state and actions.) In our example above, Π̂j contains
two models π̂jCoop and π̂jGrim . Agent i starts with a prior belief Pr(π̂j | h0 ) which
assigns probabilities to each model π̂j ∈ Π̂j before observing any actions from
agent j. If no prior information is available that makes one agent model more
PDF f(x; )
PDF f(x; )
PDF f(x; )
3.8 8.0
0.6 2.0 0.6 3.1 0.6 6.4
x1
x3
x1
x3
x1
x3
0.4 0.4 2.5 0.4
1.9 4.8
0.8 0.2 0.8 0.2 1.3 0.8 0.2 3.2
0.6 1.6
1.0 0.0 2.0 1.0 0.0 0.0 1.0 0.0 0.0
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0
x2 x2 x2
Figure 6.10: Agent 1’s beliefs about agent 2’s model (policy) after vary-
ing episodes from the Rock-Paper-Scissors non-repeated game example in
Figure 6.6 (page 131): before episode 1 (left), after episode 1 (middle), and
after episode 9 (right). Beliefs are represented as Dirichlet distributions over
probability distributions x = (x1 , x2 , x3 ) for actions R,P,S. Each point in the sim-
plex triangle shows the Dirichlet probability density (defined in Footnote 8) for
a distribution x that is mapped into the simplex coordinate space via barycentric
coordinates. Pseudocount parameters (βk ) are shown next to each simplex.
Means are shown as white stars.
likely than others, then the prior belief could simply be a uniform probability
distribution, as used in our example. After observing agent j’s action atj in state
st , agent i updates its belief by computing a Bayesian posterior distribution:
π̂j (atj | ht ) Pr(π̂j | ht )
Pr(π̂j | ht+1 ) = P 0 t t 0 t
(6.18)
π̂ 0 ∈Π̂j π̂j (aj | h ) Pr(π̂j | h )
j
Note that this Bayesian belief update is computed across episodes, meaning
that the posterior Pr(π̂j | hT ) at the end of episode e is used as the prior Pr(π̂j | h0 )
at the start of episode e + 1.
The above definition of beliefs assumes that Π̂j contains a finite number of
models. We can also define Π̂j to be a continuous space of agent models, in
which case the beliefs are defined as probability densities over Π̂j rather than
discrete probability distributions. For example, we may define Π̂j to contain all
possible policies π̂j (aj | s) that are conditioned on states, as we did in JAL-AM in
Equation 6.16. We can then define the belief over Π̂j based on a set of Dirichlet
distributions,8 with one Dirichlet δs corresponding to each state s ∈ S to represent
a belief over possible policies π̂j (· | s) in state s, where δs is parameterised by
8. A Dirichlet distribution of order K ≥ 2 with parameters β = (β1 , ..., βK ) has a probability density
βk –1
function given by f (x1 , ..., xK ; β) = B(β)–1 K
Q
k=1 xk , where {xk }k=1,...,K belong to the standard
K-1 simplex (i.e. the xk define a probability distribution), and B(β) is the multivariate beta function.
A Dirichlet distribution can be interpreted as specifying a likelihood over different probability
values for {xk }. It is the conjugate prior of the categorical and multinomial distributions.
pseudocounts (β1 , ..., β|Aj | ) (one for each action). An initial setting of βk = 1
for all k produces a uniform prior belief. After observing action atj in state st ,
the belief update is simply done by incrementing (by 1) the pseudocount βk
in δst that is associated with atj . The resulting posterior distribution is again a
Dirichlet distribution. Note that the mean of a Dirichlet distribution δs is given
by the probability distribution π̂j (aj,k | s) = Pβkβl where βk is the pseudocount
l
associated with action aj,k . Therefore, the mean of the Dirichlet corresponds to
the definition of the empirical distribution used in fictitious play (Equation 6.14)
and JAL-AM (Equation 6.16).9 Example Dirichlet distributions after varying
episodes from the Rock-Paper-Scissors non-repeated game (i.e. stochastic game
with a single state) example in Figure 6.6 (page 131) can be seen in Figure 6.10.
Given finite model spaces Π̂j and beliefs Pr(π̂j | ht ) for each other agent j 6= i,
Q
define Π̂–i = ×j6=i Π̂j and Pr(π̂–i | h) = j6=i Pr(π̂j | h) for π̂–i ∈ Π̂–i . Let s(h) denote
the last state in history h (i.e. s(ht ) = st ), and hi denote the concatenation
operation. The VI of action ai ∈ Ai after history h is defined via a recursive
combination of two functions:
X X Y
VIi (ai | h) = Pr(π̂–i | h) Qi (h, hai , a–i i) π̂j (aj | h) (6.19)
π̂–i ∈Π̂–i a–i ∈A–i j6=i
X
Qi (h, a) = T (s0 | s(h), a) Ri (s(h), a, s0 ) + γ max
0
VI (a
i i
0
| hh, a, s0
i) (6.20)
ai ∈Ai
s0 ∈S
9. However, note the difference that the Dirichlet prior is initialised with pseudocounts βk = 1
for all actions (prior to observing any actions), while the empirical distribution used in fictitious
play and JAL-AM is defined over the observed history of actions. Therefore, after one observed
action, the empirical distribution assigns probability 1 to that action while the Dirichlet mean
assigns probability less than 1. However, in practice, this difference is “washed away” as the
number of observed actions increases, and in the limit of observing infinitely many actions (or
state-action pairs), the Dirichlet mean and the empirical distribution used by fictitious play and
JAL-AM converge to the same probability distribution. Moreover, by using initial action counts of
C(aj ) = 1 (or C(s, aj ) = 1) for all actions, the empirical distribution is identical to the Dirichlet mean.
of the recursion10 between Qi and VIi determines how far VI evaluates the
impact of an action into the future of the interaction. Using this definition of
VI, at time t after history ht , agent i selects an action ati with maximum VI, i.e.
ati ∈ arg maxai VIi (ai | ht ).
When applied in the Prisoner’s Dilemma example given above, using a
uniform prior belief over the two models Pr(π̂jCoop | h0 ) = Pr(π̂jGrim | h0 ) = 0.5, no
discounting (i.e. γ = 1), and a recursion depth equal to the length of the episode
(i.e. 10), we obtain the following VI values for agent 1’s actions in the initial
time step t = 0:
VI1 (C) = –9 VI1 (D) = –13.5 (6.21)
Therefore, in this example when using a uniform prior belief, cooperating has
a higher value of information than defecting. Essentially, while VI realises
that the defect action can give absolute certainty about the model used by the
other agent, if that agent uses the Grim model then the cost of causing the
agent to defect until the end of the episode is not worth the absolute certainty in
beliefs. If both agents use this VI approach to select actions, then they will both
cooperate initially, and will continue to cooperate until the last time step. In the
last time step t = 9, the agents obtain the following VI values (for i ∈ {1, 2}):
VIi (C) = –1 VIi (D) = 0 (6.22)
Thus, both agents will defect in the last time step. In this case, as this is the
last time step, there is no more risk in defecting since both Coop and Grim
are expected to cooperate (hence, in either case, defecting will give reward 0)
and the episode will finish. This is an example of the end-game effects first
mentioned in Section 3.2.
The idea of maintaining Bayesian beliefs over a space of agent models and
computing best-responses with respect to the beliefs has been studied in game
theory under the name “rational learning” (Foster and Young 2001; Jordan
1991; Kalai and Lehrer 1993; Nachbar 1997, 2005). A central result in rational
learning is that, under certain strict assumptions, the agents’ predictions of
future play will converge to the true distribution of play induced by the agents’
actual policies, and the agents’ policies will converge to a Nash equilibrium
(Kalai and Lehrer 1993). One important condition of the convergence result is
10. To make the recursion anchor explicit in the definition of VI, we can define VIid and Qdi for
recursion depth d ≥ 0, where VIid calls Qdi and Qdi calls VIid–1 , and we define VIid (ai | h) = 0 if d = 0
or if s(h) is a terminal state (i.e. end of episode). For the case of d = 0, instead of assigning value
VIid (ai | h) = 0, Chalkiadakis and Boutilier (2003) describe an alternative “myopic” approach by
fixing the belief Pr(π̂–i | h) at the end of the recursion, sampling agent models π̂–i from the fixed
belief, solving the corresponding MDPs in which agents –i use those sampled models, and averaging
over the resulting Q-values from the MDPs.
that any history that has positive probability under the agents’ actual policies
must have positive probability under the agents’ beliefs (also known as “absolute
continuity”). This assumption can be violated if the best-responses of agent j
are not predicted by any model in Π̂j . In our Prisoner’s Dilemma example, if
the agents start with a prior belief that assigns probability 0.8 to Coop and 0.2
to Grim, then the agents will obtain VI values of VIi (C) = –5.8, VIi (D) = –5.4
and both agents will initially defect, which is not predicted by the Coop nor
Grim model.11 On the other hand, when using the Dirichlet beliefs discussed
earlier, then any finite history will always have non-zero probability under
the agents’ beliefs. Note that an important difference between the Bayesian
learning presented here and rational learning is that the latter does not use value
of information as defined here. Instead, rational learning assumes that agents
compute best-responses with respect to their current belief over models, without
considering how different actions may affect their beliefs in the future.
11. In fact, the choice of prior belief can have a substantial effect on the achievable returns and
convergence to equilibrium (Albrecht, Crandall, and Ramamoorthy 2015; Dekel, Fudenberg, and
Levine 2004; Nyarko 1998).
F denotes the off-diagonal matrix which contains the terms u and u0 . This
learning algorithm using an infinitesimal step size is referred to as infinitesimal
gradient ascent, or IGA for short.
The centre point (α∗ , β ∗ ) which has a zero-gradient can be found by setting
the left-hand side of Equation 6.33 to zero and solving to obtain:
c – c
2,1 r2,2 – r1,2
2,2
(α∗ , β ∗ ) = , (6.34)
u0 u
(a) F not invertible (b) F has purely real eigen- (c) F has purely imaginary
values eigenvalues
Figure 6.11: The joint policy (α, β) learned by Infinitesimal Gradient Ascent
(IGA) in the unconstrained space (i.e. (α, β) may be outside the unit square)
will follow one of three possible types of trajectories, depending on properties
of the matrix F from Equation 6.33. Shown here are the general schematic
forms of the trajectory types; the exact trajectory will depend on the values of
u and u0 . The plot axes correspond to values of α, β and arrows indicate the
direction of the (α, β)-trajectories. The centre point which has zero-gradient (if
one exists) is marked by the star in the centre. For non-invertible F, (a) shows
one possible set of trajectories, but others are possible depending on u and u0 .
• If (α, β) does not converge, then the average rewards received during learning
converge to the expected rewards of some Nash equilibrium.
• If (α, β) converges, then the converged joint policy is a Nash equilibrium.
While if F is non-invertible or has real eigenvalues it can be shown that (α, β)
converges to a Nash equilibrium in the constrained system, this does not hold
if F has imaginary eigenvalues where ellipses can be wholly contained inside
the unit square, in which case (α, β) will cycle indefinitely if it follows such an
ellipse. Notably, however, Singh, Kearns, and Mansour (2000) showed that the
average rewards obtained by following such an ellipse converge to the expected
rewards of a Nash equilibrium of the game. Thus, this is an example of the
convergence type defined in Equation 5.8 (Section 5.2). An implication of this
result is that when (α, β) converges, then (α, β) must be a Nash equilibrium.
This can also be seen by the fact that (α, β) converges if and only if reaching a
point where the (projected) gradient is zero, and such points can be shown to be
Nash equilibria in the constrained system (since otherwise the gradient cannot
be zero). Finally, Singh, Kearns, and Mansour (2000) showed that these results
also hold for finite step sizes κ if κ is appropriately reduced during learning
1
(e.g. κk = k2/3 ).
This modified IGA learning rule using a variable learning rate is called WoLF-
IGA. Note that αe and β e need not be from the same equilibrium, meaning that
(αe , β e ) may not form a Nash equilibrium.
Figure 6.12: General form of joint policy (α, β) trajectory when using WoLF-
IGA, for the case when F(t) has purely imaginary eigenvalues and the unique
centre point is contained in the unit square. The trajectory (solid line) tightens
in each quadrant and converges to the centre point (grey star), which is a Nash
equilibrium. The plot axes correspond to values for α, β.
Using this variable learning rate, it can be proven that WoLF-IGA is guar-
anteed to converge to a Nash equilibrium in general-sum games with two
agents and two actions. The analysis of WoLF-IGA is near-identical to IGA:
using infinitesimal step sizes, the joint policy will follow a continuous trajec-
tory (α(t), β(t)) in continuous time t which evolves according to the following
differential equation:
∂α
∂t 0 li (t)u α li (t)(r1,2 – r2,2 )
∂β = + . (6.39)
∂t
lj (t)u0 0 β lj (t)(c2,1 – c2,2 )
| {z }
F(t)
As was the case for IGA, we again examine three qualitatively distinct
trajectory types based on properties of the matrix F(t), which now depends on t
due to the variable learning rates. These cases are when 1) F(t) is non-invertible,
2) F(t) is invertible and has purely real eigenvalues, and 3) F(t) is invertible
and has purely imaginary eigenvalues. The crucial part in the analysis is case
3), and in particular the sub-case where the centre point is contained within
the unit square. This is the problematic case where IGA will not converge to
a Nash equilibrium. The WoLF variable learning rate has an important effect
in this sub-case: Bowling and Veloso (2002) showed that in WoLF-IGA, the
trajectories of (α, β) are in fact piecewise elliptical, with four pieces given by
the four quadrants around the centre point (as shown in Figure 6.12), such
that the trajectories spiral toward the centre point. Note that in this case, there
exists only one centre point and this is by definition aqNash equilibrium. In
each quadrant, the ellipse will “tighten” by a factor of llmax
min
< 1, and thus the
trajectory will converge to the centre point.
where
δ
δs,ai = min πi (ai | s) , (6.45)
|Ai | – 1
(
if a0 πi (ai | s)Q(s, ai ) > a0 π̄i (a0i | s)Q(s, a0i )
0 0
P P
lw
δ= i i . (6.46)
ll otherwise
The role of ∆(s, ai ) is to move the policy πi closer to the current greedy
policy with respect to Q, which is done by removing probability mass δs,ai
from all non-greedy actions, and adding the sum of these masses to the greedy
12. Equation 6.44 assumes that there is only one greedy action in state s. If there are multiple greedy
actions that have maximum value in Q(s, ·), then ∆(s, ai ) can be defined to spread the probability
mass uniformly across these greedy actions.
0.6 0.4
π1(Paper)
0.4 0.6
0.2 0.8
Figure 6.13: Evolving policies of two agents, each using the WoLF-PHC algo-
rithm to update policies, in the non-repeated Rock-Paper-Scissors matrix game.
The diagonal dashed line divides the plot into two probability simplexes, one
for each of the two agents. Each point in the simplex for an agent corresponds
to a probability distribution over the agent’s three available actions. Each line
shows the agent’s current policy at episodes 0, 5, 10, 15, ..., 145 (marked by the
dots), as well as the converged policy at episode 100, 000 (marked by a star).
where κk is the step size. Step (1) updates the policy in the direction of the
unconstrained gradient ∇πik Ui (πik , akj ), while Step (2) projects the result back
into a valid probability space via the projection operator
P(x) = arg0 min ||x – x0 || (6.51)
x ∈∆(Ai )
where || · || is the standard L2-norm.13 P(x) projects a vector x back into the
space of probability distributions ∆(Ai ) defined over the action set Ai .
Zinkevich (2003) showed that if all agents use GIGA to learn policies with
a step size of κk = √1k , then in the limit of k → ∞ the policies will achieve
no-regret. Specifically, recall the definition of regret in Equation 4.28 (page 81),
√
13. The L2-norm is defined as ||x|| = x · x, where · is the dot product.
where [x]+ = max[x, 0]. If the denominator in Equation 6.55 is zero, then
πiz+1 may use any probability distribution over actions.
Conditional regret matching computes action probabilities which are pro-
portional to the (positive) average conditional regrets with respect to the
most recent selected action. Let ae denote the joint action from episodes
e = 1, ..., z. Agent i’s conditional regret for not having chosen action ai in
each episode in which it chose action a0i is defined as
X
Regretiz (a0i , ai ) = Ri (hai , ae–i i) – Ri (ae ) .
(6.56)
e:aei =a0i
The average conditional regret for agent i and actions a0i , ai is given by
1
R̄zi (a0i , ai ) = Regretiz (a0i , ai ). (6.57)
z
Again, each agent i starts with an initial policy πi1 that can use any proba-
bility distribution over actions. Then, given the above definition of average
conditional regret, the policy πiz is updated to
( 1 z z
η [R̄i (ai , ai )]+ if ai 6= azi
z+1
πi (ai ) = (6.58)
1 – a0 6=az πiz+1 (a0i ) otherwise
P
i i
where azi is the action chosen by agent i in the last episode z, and η >
2 · maxa∈A |Ri (a)| · (|Ai | – 1) is a parameter that governs how far the action
probabilities will be biased toward actions with high conditional regret (the
higher η, the lower the bias). The lower bound on η ensures that the sum
of the probabilities πiz+1 (ai ) assigned to actions ai 6= azi is at most 1.
Before discussing the asymptotic behaviours of these regret matching algo-
rithms, we can examine their behaviours in the example from Figure 4.6
(page 82) in the Prisoner’s Dilemma matrix game. In Prisoner’s Dilemma,
D is a dominant action since it is a best response against both D and C by the
other agent. Therefore, the unconditional regret Regretiz (C) for action C can
never be positive, which means that [R̄zi (C)]+ = 0 in all episodes. In our example,
this results in unconditional regret matching always assigning probability 0 to
action C and probability 1 to action D in all episodes after the first episode (the
policies πi1 in the first episode can use any probability distribution). Further-
more, note that since Prisoner’s Dilemma is a normal-form game with only two
actions for each agent, the definition of conditional regret is equivalent to the
definition of unconditional regret. Therefore, conditional regret matching has
identical behaviour to unconditional regret matching in Prisoner’s Dilemma.
Now, consider the empirical distribution over joint actions {ae }ze=1 given by
z
1X
π̄ z (a) = [a = ae ]1 (6.63)
z
e=1
with [x]1 = 1 if x is true, and [x]1 = 0 otherwise. The two averaged rewards in
the left-hand and right-hand sides of Equation 6.62 can be equivalently written
as expected rewards under the empirical distribution π̄ z as
X X
π̄ z (a)Ri (ha00i , a–i i) ≤ π̄ z (a)Ri (a) (6.64)
a∈A:ai =a0i a∈A:ai =a0i
1.0 0.8 0.6 0.4 0.2 0.0 1.0 0.8 0.6 0.4 0.2 0.0
1.0 0.0 1.0 0.0
Agent 1 policy Agent 1 empirical
Agent 2 policy Agent 2 empirical
Converged policy
0.8 0.2 0.8 0.2
π1(Paper)
0.4 0.6 0.4 0.6
0.00.0 0.2 0.4 0.6 0.8 1.01.0 0.00.0 0.2 0.4 0.6 0.8 1.01.0
π1(Rock) π1(Rock)
Figure 6.14: Evolving policies of two agents, each using unconditional regret
matching to update policies, in the non-repeated Rock-Paper-Scissors matrix
game. The diagonal dashed line divides the plots into two probability simplexes,
one for each of the two agents. Each point in the simplex for an agent corre-
sponds to a probability distribution over the agent’s three available actions. (a)
Trajectories of the agents’ actual policies πiz over 10, 000 episodes, with policies
from every 10th episode is marked by a dot. (b) Trajectories of the agents’
empirical action distributions π̄iz for the first 500 episodes, with the converged
distribution marked by a star.
behaviour. Yet, despite this apparent chaotic behaviour (or, rather, because
of it), Figure 6.14b shows that the agents’ empirical distributions π̄iz converge
steadily to the unique Nash equilibrium of the game, in which both agents
choose actions uniformly randomly.14 Looking at the average unconditional
regrets of both agents, which are shown in Figure 6.15, we see that they steadily
reduce over the first 1,000 episodes and then enter oscillatory swings around
the zero line. The oscillation is caused by the mutual adaptation of the agents:
agent 1 repeatedly chooses actions with high average regret, which over time
will cause agent 2’s average regrets to increase for the respective best-response
actions against agent 1’s actions, which in turn will increase agent 1’s regrets for
its corresponding best-response actions, and so on. The empirical distribution
of these oscillatory action changes converges to the Nash equilibrium.
14. The empirical distributions π̄iz for each agent i are P given by the respective marginal distributions
of the joint empirical distribution π̄ z , i.e. π̄iz (ai ) = a–i π̄ z (hai , a–i i).
0.20 0.20
Average unconditional regret
Figure 6.15: Average unconditional regrets of both agents for actions Rock (R),
Paper (P), Scissors (S) over 10,000 episodes in the non-repeated Rock-Paper-
Scissors game. The y-axis is limited to the range [–0.2, 0.2] (the average regrets
in the first few episodes are larger than these limits).
When comparing these learning plots to the corresponding plot for WoLF-
PHC shown in Figure 6.13 (page 149), notice the important difference that
WoLF-PHC achieves a much stronger convergence than regret matching; in
WoLF-PHC the policies πiz converge to Nash equilibrium, but in regret matching
only the empirical distribution π̄ z converges to the Nash equilibrium while the
policies πiz shown in Figure 6.14a do not converge.
6.6 Summary
This chapter described several families of foundational MARL algorithms which
are designed to learn different types of game solutions under certain conditions.
We summarise the main ideas as follows:
• Analogous to the value iteration algorithm for MDPs, a value iteration
algorithm exists for zero-sum stochastic games to learn optimal state values
for each agent. The main idea of the algorithm is to compute matrices for
each agent and state, which estimate the expected return (i.e. value) to an
agent when selecting a specific joint action in the state and following the
optimal joint policy afterward. For a given state, a normal-form game can
be constructed using the corresponding matrices of the agents. This normal-
form game is then solved via a minimax solver, to obtain a target value to
update the agents’ value estimate of the state. This value iteration algorithm
provably converges to a minimax joint policy of the stochastic game.
• The value iteration algorithm laid the foundation for a family of MARL
algorithms known as joint action learning. These algorithms use temporal-
difference learning in combination with game-theoretic solution concepts, to
estimate the values of joint actions and learn solutions for games. Several
instances of these algorithms can be defined which use different solution
concepts. Minimax Q-learning is a temporal-difference version of the value
iteration algorithm and converges to minimax solutions under certain condi-
tions in zero-sum stochastic games. Nash Q-learning is based on the Nash
equilibrium concept and can be applied to general-sum stochastic games
with two or more agent, but converges to Nash equilibrium requires very
restrictive assumptions, in part due to equilibrium selection problems.
• Agent modelling is the task of constructing models of other agents that can
make useful predictions about their behaviours. For example, a model can be
learned to predict the probabilities of another agent’s next actions, based on
observations about the past chosen actions of that agent. Given such a model,
an agent can compute best-response actions against the model. Fictitious
play is one of the earliest learning algorithms for normal-form games that
operates in this way, and is able to converge to Nash equilibria in several
types of normal-form games. Joint action learning algorithms can also learn
such agent models and combine them with temporal-difference learning to
learn best-response policies for the agents.
• Bayesian learning approaches for agent modelling compute probabilities
over multiple possible models based on past observations. Using the concept
of value of information, such methods can compute best-response actions that
can trade-off between maximising an agent’s expected returns and discovering
the true model of an agent.
• Policy-based learning methods directly optimise the parameters of some
probabilistic policy functions. In contrast to methods that estimate joint
action values, policy-based methods can directly adjust the probabilities in
policies using techniques based on gradient ascent. By considering infinitely
small update steps, the learning behaviours of these methods can be analysed
using dynamical systems theory. For example, basic infinitesimal gradient
ascent can converge to Nash equilibria in normal-form games, or alternatively
converge to the expected rewards under a Nash equilibrium.
• No-regret learning algorithms learn policies with the aim to minimise notions
of regrets for not taking certain actions in past episodes. Unconditional regret
matching is a no-regret learning algorithms that achieves zero unconditional
regret in the long-run, and the empirical distribution of joint actions converges
to the set of coarse correlated equilibria in normal-form games. Similarly,
Part II of this book will build upon the foundations introduced in Part I and
present MARL algorithms which use deep learning to represent value functions
and agent policies. As we will see, deep learning is a powerful tool which
enables MARL to scale to more complex problems than is possible with tabular
methods. The chapters in this part will introduce the basic concepts of deep
learning, and show how deep learning techniques can be integrated into RL and
MARL to produce powerful learning algorithms.
Chapter 7 provides an introduction to deep learning, including the building
blocks of neural networks, foundational architectures, and the components
of gradient-based optimisation used to train neural networks. This chapter
primarily serves as a concise introduction to deep learning for readers unfamiliar
with the field, and explains all foundational concepts required to understand the
following chapters. Chapter 8 then introduces deep RL, explaining how to use
neural networks to learn value functions and policies for RL algorithms.
Chapter 9 builds on the previous chapters and introduces multi-agent deep RL
algorithms. The chapter begins by discussing different modes of training and
execution, which determine the information available to agents during and after
learning. The chapter then revisits the class of independent learning algorithms
with deep learning, before presenting advanced topics including multi-agent
policy gradient algorithms, value decomposition, and agent modelling. The
chapter also shows how agents can share parameters and experiences in multi-
agent deep RL algorithms to further improve the learning efficiency. Finally, the
chapter presents self-play and population-based training in MARL, which have
led to breakthroughs in tackling very complex multi-agent games. Chapters 10
and 11 conclude this part of the book by discussing practical considerations for
the implementation of MARL algorithms, and presenting environments which
can serve as benchmarks and playground to study these algorithms.
7 Deep Learning
Figure 7.1: A maze environment in which a single agent must reach a goal
location (G). The agent has seen several states, including s1 and s3 during its
trajectories, and tries to estimate values for the circled states s2 and s4 . An agent
with a tabular value function cannot generalise in this manner and would need to
explicitly experience trajectories with s2 and s4 to be able to learn accurate value
estimates. In contrast, a value function using function approximation techniques
(such as linear value functions or deep learning) may be able to generalise and
estimate the values of s2 and s4 to be similar to s1 and s3 , respectively.
video games, and most real-world applications. For example, the state space
of the board game of Go is estimated to contain approximately 10170 possible
states. Storing, managing and updating a table with this number of entries is
completely infeasible for modern computers. Second, tabular value functions
update each value estimate in isolation. For tabular MARL, an agent will update
its value estimate for the state s after visiting this state, and leave its value
function for any other state unchanged. These isolated updates for visited states
and actions come with theoretical guarantees for tabular MARL algorithms,
but mean that an agent has to encounter states before being able to learn their
values. This can render tabular MARL algorithms inefficient in tasks with large
state or action spaces. For learning in complex tasks, it is therefore essential
that agents have the ability to generalise to novel states. Encountering a specific
state should ideally allow an agent to update not just its value estimate for this
particular state, but also update its estimates for different states which are in
some sense similar to the encountered state.
To illustrate the benefits of generalisation, consider the following example
depicted in Figure 7.1. In this single-agent environment, the agent needs to
navigate a maze and reach the goal location (G) to receive positive reward. Given
this reward function, the expected discounted returns of an optimal policy in a
state depend on the path length of the state to the goal. Hence, two states which
are similar in their path length to the goal should have similar value estimates.
Considering the circled states in Figure 7.1, we would expect the value estimate
for state s2 to be similar to the value of s1 . Likewise, the value of s4 should be
similar to the value of s3 . Tabular value functions are trained on each of these
states in isolation, but function approximation offers value functions which can
generalise. This generalisation can enable approximate value functions to pick-
up on the relationships between states and their respective values, and provide
reasonable value estimates for s2 and s4 before encountering these states.
Figure 7.2: Illustration of a feedforward neural network with three layers. The
input x ∈ R3 is processed by two hidden layers with parameters θ1 and θ2 ,
respectively, and a single scalar output is computed in the final output layer
with parameters θ3 .
Figure 7.3: Illustration of a single neural unit computing a scalar output given
an input x ∈ R3 . First, a weighted sum over the input features is computed using
the vector multiplication of w ∈ R3 and the input x and a scalar bias b ∈ R is
added. Finally, a non-linear activation function g : R → R is applied to obtain
the scalar output.
sequentially feeding the output of the previous layer into the next layer. For
example, a feedforward neural network with three layers (or two hidden layers),
visualised in Figure 7.2, can be written as
f (x; θ) = f3 ( f2 ( f1 (x; θ1 ); θ2 ); θ3 ) (7.4)
where θk corresponds to the parameters of layer k and θ = k θk denotes the
S
parameters of the entire network. We denote the number of units in the kth
layer by dk . The number of layers is referred to as the depth of the neural
network and the number of units in a layer is referred to as the width or hidden
dimension of the layer. The generality and representational capacity, i.e. the
complexity of functions a neural network can represent, largely depend on the
depth of the network and the dimensionality of each of its layers. The universal
approximation theorem states that feedforward neural networks with as few
as a single hidden layer are able to approximate any continuous function on a
closed and bounded subset of real-valued vectors given sufficient hidden units
(Cybenko 1989; Hornik 1991; Hornik, Stinchcombe, and White 1989; Leshno
et al. 1993). However, training deeper neural networks with multiple layers
can result in better performance and generalisation at an equivalent number of
total parameters (Goodfellow, Bengio, and Courville 2016). To understand how
these networks approximate the overall function f (x; θ), we will first explain
individual units within layers before explaining how to compose all these pieces
together to form the entire network.
x ∈ Rdx is used instead. First, a unit computes a weighted sum over its input
features using a weight vector w ∈ Rdk–1 and adding a scalar bias b ∈ R to the
weighted sum. This first computation represents a linear transformation and is
followed by a non-linear activation function gk . The entire computation of a
neural unit is visualised in Figure 7.3 and can be formalised as
fk,u (x; θuk ) = gk w> x + b
(7.5)
with parameters θuk ∈ Rdk–1 +1 containing the weight vector w as well as the scalar
bias b. Applying non-linear activation functions to the output of each unit is
essential, because the composition of two linear functions, f and g, can only
represent a linear function itself. Therefore, composing an arbitrary number
of neural units without non-linear activation functions would only result in a
linear function. During the optimisation of a feedforward neural network, the
parameters θuk of each unit in the network are optimised.
4
ReLU 4
ELU ( = 1.0)
2 2
g(x)
0 0
4
Leaky ReLU (c = 0.01) 4
Leaky ReLU (c = 0.1)
2 2
g(x)
0 0
4
Tanh 4
Sigmoid
2 2
g(x)
0 0
4 3 2 1 0 1 2 3 4 4 3 2 1 0 1 2 3 4
x x
Figure 7.5: Common non-linear activation functions defined in Figure 7.4.
functions are most commonly applied to restrict the output of a neural network
to be within the ranges (–1, 1) or (0, 1), respectively. If no such constraints are
required, ReLU and variations of the ReLU function are commonly applied as
default activation functions.
1. Inputs 1. Targets
Update parameters
Figure 7.6: The training loop for gradient-based optimisation of a neural network
parameterised by θ: 1. From a dataset D of input-output pairs (x, y), a subset of
pairs is sampled as a batch. 2. The prediction f (x; θ) is computed for each input
x in the batch. 3. The loss function L is computed between the predictions
f (x; θ) and the corresponding target values y in the batch. 4. The gradients
of the loss function with respect to the network parameters θ are computed
using backpropagation. 5. The parameters θ are updated using a gradient-based
optimiser. Then, the loop starts again with the updated parameters θ.
1. Note that the parameters of feedforward neural networks include all weight matrices and bias
vectors across all layers.
ground truth outputs, here states with their true state values, are available for
training. This setting of machine learning, in which models are trained from
data containing ground truth output labels, is referred to as supervised learning.
However, note that in RL we typically do not know the true state-value function
a priori. Fortunately, temporal-difference learning gives us a framework with
which we can formulate a loss for our value function to approximate discounted
state-value estimates using bootstrapped value estimates (Section 2.6):
B
1X 2
L(θ) = ri + γ V̂(s0i ; θ) – V̂(si ; θ) (7.9)
B
i=1
In this loss, we make use of a batch of experience of the agent consisting of state
s, reward r, and next state s0 , respectively. Minimising this loss will optimise
our network parameters θ, such that V̂ will gradually provide accurate state-
value estimates. Later in Chapter 8, we will discuss details of RL using neural
networks with concrete algorithms, examples, and performance comparisons.
2. Vanilla gradient descent is also sometimes referred to as batch gradient descent since it computes
a single gradient using the entire training data as a batch.
Vanilla
0.7 0.7 Vanilla + default momentum = 0.9
Vanilla + Nesterov momentum = 0.9
0.6 0.6 Optimal
0.5 0.5
0.4
0.4
b
b
0.3
0.3
Vanilla
0.2 Stochastic
Mini-batch B =145.00 0.2
15.00
0.1
30.0
Mini-batch B = 32
45 0 Mini-batch B = 128 0.1
60
.00
.00
45 30.0
0.0 Optimal 0
60
.00
.00
0.0
0.5 0.0 0.5 1.0 1.5 2.0 0.5 0.0 0.5 1.0 1.5 2.0
a a
(a) Gradient descent batch (b) Gradient descent momentum
visualises the optimisation for the same polynomial function as discussed before
using vanilla gradient descent with and without two types of momentum. We can
see significant improvements in the efficiency of gradient descent when using
momentum, with significantly fewer updates needed to obtain parameters close
to the ground truth parameters. As seen by the default momentum, the speed-up
of momentum is effective as long as gradients continue in a similar direction, but
also increases the risk of “overshooting” minima of the loss function. Nesterov
momentum (Nesterov 1983) further improves upon this and exhibits higher
stability than traditional momentum. Several more recent approaches follow
the idea of dynamically adapting the learning rate during optimisation, which
can be considered similar to momentum. Learning rate adaptation has the main
benefits of simplifying the process of choosing the initial learning rate as a
hyperparameter, making the optimisation less sensitive to the choice of the (ini-
tial) learning rate, and to speed-up the optimisation process. All gradient-based
optimisers, which are commonly used to optimise the parameters of neural
networks, apply different learning rate adaptation schemes (Duchi, Hazan, and
Singer 2011; Hinton, Srivastava, and Swersky 2012; Kingma and Ba 2015;
Zeiler 2012). None of these optimisers is consistently observed to perform better
than the others, but the Adam optimiser (Kingma and Ba 2015) has emerged as
a common choice and is often applied as the default optimiser in deep learning.3
7.4.3 Backpropagation
Gradient-based optimisers require the gradients ∇θ L(θ) of the loss with respect
to all parameters of the network. These gradients are needed to understand
how we should change each parameter to minimise the loss. To compute
gradients of the loss function with respect to all parameters in the network, i.e.
including the parameters of every layer within the network, the backpropagation
algorithm (Rumelhart, Hinton, and Williams 1986) is used. This algorithm
considers the fact that a neural network computes a compositional function for
which the chain rule of derivatives can be applied. As a reminder, the chain rule
states for y = g(x) and z = f (y) = f (g(x))
>
∂y
∇x z = ∇y z (7.14)
∂x
∂y
with ∂x being the Jacobian matrix of function g. In words, the chain rule allows
us to compute the gradients of a compositional function with respect to the
3. For a more detailed overview of gradient-based optimisation techniques, we refer the interested
reader to Ruder (2016).
colour channels corresponding to the colours red, blue and green), height, and
width of the image. The dimensionality of this flattened image vector x̃ ∈ Rc·h·w
corresponds to the number of pixels in the image, multiplied by the number of
colour channels. Processing such large input vectors with feedforward neural
networks would require the first layer to contain many parameters, which makes
the optimisation difficult and is hence undesirable. For example, consider a
feedforward neural network which processes images of 128 × 128 pixels with
RGB colours for three colour channels. Representing images of this size, which
is significantly smaller than photos taken with any modern smartphone camera,
corresponds to vectors of dimensionality 128 · 128 · 3 = 49,152. With a small
number of neural units of 128 in the first layer, the network would have a total of
6,291,584 parameters4 which need to be learned. While six million parameters
is not a large neural network in comparison to large language models, it appears
excessive and overly computationally expensive to require so many parameters
to process even small images.5
Secondly, images contain spatial relationships with pixels close to each other,
often corresponding to similar objects shown in the image. Feedforward neural
networks do not consider such spatial relationships and process every input
value individually.
Convolutional neural networks (CNNs) (Fukushima and Miyake 1982;
LeCun et al. 1989) directly make use of the spatial relationships within input
data such as images by processing patches of nearby pixels at a time. Small
groups of parameters called filters or kernels are “slid” over the image in a
convolution operation. For the convolution, each filter moves through the pixels
in each row of the image and encodes all pixel values it covers at a time. For
each patch of pixels the filter moves over, its parameters are multiplied with
the corresponding patch of pixels of the same size in a matrix multiplication
to obtain a single output value. The patch of input values involved in a sin-
gle convolution are also referred to as the receptive field of the corresponding
output value or neuron. Formally, the convolution of a filter with parameters
W ∈ Rdw ×dw over input x ∈ Rdx ×dx for output yi,j ∈ R is defined as follows:
dw X
X dw
yi,j = Wa,b xi+a–1,j+b–1 (7.15)
a=1 b=1
4. The weight matrix would be a 49,152 × 128 dimensional matrix with a total of 6,291,456
parameters and the bias vector would contain 128 parameters.
5. To process 4K RGB images with a resolution of 3840 × 2160 instead of 128 × 128 pixels, the
weight matrix in the first layer of a feedforward neural network alone would contain more than 3
billion parameters.
Filter
Input
Padding
Output Output
Convolution Pooling
Activation Function
Stride
This operation is repeated until the filter has been moved over the entire
image. Note that each filter has one parameter for each cell, which are reused
in every convolution operation. This sharing of parameters across multiple
computations leads to significantly fewer parameters that need to be optimised,
in contrast to a fully-connected network layer, and makes use of the spatial
relationship of images with patches of nearby pixels being highly correlated.
Filters move across the input depending on the stride and padding. The stride
determines the number of pixels the filter moves at each slide, and padding
refers to an increase in the image size, typically by adding zero values around
its borders. These hyperparameters can be used to influence the dimensionality
of the convolution output. Following the convolution, a non-linear activation
function is applied element-wise to each cell of the output matrix.
To highlight the efficiency of CNNs, consider our previous example of pro-
cessing a 128 × 128 RGB image. A CNN with 16 filters of 5 × 5 dimensions
has a total of 1, 216 parameters6 to process an image with 3 input channels. In
contrast, a single-layered feedforward neural network with 128 hidden dimen-
sions has over six million parameters to process the same input. This is because
the number of parameters of the CNN, unlike for the feedforward neural net-
work, is independent of the width and height of the input image since the
6. Each kernel has 5 · 5 · 3 = 75 parameters. Across all 16 filters, this results in 1, 200 parameters in
addition to the 16 bias parameters.
same filters are applied across the entire image. Instead, the output dimension
of a CNN depends on the dimensions of the input as well as the stride and
padding. For example, the CNN with 16 filters of 5 × 5 dimensions applied to
a 128 × 128 × 3 image with stride 2 and padding 0 results in a 63 × 63 × 16
output.
CNNs often apply multiple such convolution operations in a sequence using
varying sizes of filters. In between each convolution, it is common to addi-
tionally apply pooling operations. In these operations, patches of pixels are
aggregated together using operations such as taking the maximum value within
the patch, referred to as max-pooling, or taking the average. These pooling
operations in between convolutions further reduce the output dimensionality
throughout the convolutional neural networks and make the output of the net-
work insensitive to small local changes in the image. This effect is desirable
because the learned locally-insensitive features might generalise better than
features which are associated with a particular location in the image.
Figure 7.8 shows a convolutional neural network with a single kernel pro-
cessing an input and applying pooling for aggregation. After processing spatial
inputs, such as images, with several layers of CNNs, it is common to fur-
ther process the obtained compact representations using feedforward neural
networks.
on such a history requires the entire sequence as input. Similar to image inputs,
processing a long sequence of observations with a feedforward neural network
can require many parameters. Additionally, observations within a history will
likely be correlated, thus sharing parameters to process each observation appears
desirable, similarly to filters being used many times in a convolutional neural
network.
Recurrent neural networks (RNNs) (Rumelhart, Hinton, and Williams 1986)
are neural networks specifically designed to process sequential data. Instead
of using long concatenated sequences of inputs, recurrent neural networks
sequentially process inputs and additionally condition the computation at each
step on a compact representation of the history of previous inputs. This compact
representation, also called the hidden state of the recurrent neural network, is
continually updated to encode more information as the sequence is processed
and serves as a form of memory. In this way, the same neural network and
computation can be applied at each time step, but its function continually adapts
as the sequence is processed and the hidden state changes. Formally, the hidden
state is given by the output of the network
ht = f (xt , ht–1 ; θ) (7.16)
with the initial hidden state h0 usually being initialised as a zero-valued vector.
In line with common notation in deep learning, we denote the hidden state of
a recurrent neural network with h, but highlight that elsewhere in the book h
refers to the history of observations. Figure 7.9 illustrates the computational
graph for a recurrent neural network processing a sequence of inputs. Some
recurrent neural network architectures provide separate outputs for the updated
hidden state and the main output of the network.
Optimising recurrent neural networks over long sequences is difficult, with
gradients often vanishing (becoming close to zero) or exploding (becoming very
large) due to the backpropagation repeatedly multiplying gradients and Jaco-
bians of the network parameters throughout the sequence. Various approaches
have been proposed to address this challenge, including skip connections (Lin
et al. 1996), which add connections between computations across multiple time
steps, and leaky units (El Hihi and Bengio 1995; Mozer 1991), which allow
tuning the accumulation of previous time steps through linear self-connections.
However, the most commonly used and effective recurrent neural network
architectures are long short-term memory cells (LSTMs) (Hochreiter and
Schmidhuber 1997) and gated recurrent units (GRUs) (Cho et al. 2014). Both
approaches are based on the idea of allowing the recurrent neural network to
decide when to try to remember or discard accumulated information within the
hidden state.
7.6 Summary
In this chapter, we introduced deep learning and neural networks as a general
approach to learn approximate functions. The main concepts from this chapter
are the following:
• For tabular value functions and policies, updates only occur for visited states
and actions. However, it is infeasible to visit all states and actions many
times to learn accurate estimates in complex environments with large state
or action spaces. This necessitates function approximation to learn value
functions and policies which generalise across states and actions. Linear
function approximation approximates functions using a linear combination
of features. This approach is simple but also limited in the functions it can
represent and requires careful selection of features. Instead, deep learning
can approximate any function with neural networks given sufficient capacity.
• Feedforward neural network – also known as fully connected neural networks
or multilayer perceptrons (MLPs) – are the foundational building block of
most neural networks. They are organised in layers with each layer represent-
ing a parameterised linear transformation followed by a non-linear activation
function. An input is sequentially passed through the layers in the network
with the input to each layer being the output of the previous layer. The output
of the final layer is the output of the network.
• Neural networks are typically initialised with random parameters. To approx-
imate a desirable function, their parameters are iteratively updated using
gradient-based optimisation. The objective of this optimisation is to min-
imise a differentiable loss function. For each optimisation step, the loss for a
batch of inputs is computed. Then, the gradients of the loss with respect to
the parameters of the neural network are computed using backpropagation.
The backpropagation algorithm efficiently computes the necessary gradients
by iteratively applying the chain rule of derivation. Lastly, the parameters of
the neural network are updated with a gradient-based optimisers based on
the idea of gradient descent.
• Convolutional neural networks are an architecture of neural networks
designed to process high-dimensional inputs with spatial structure like images.
Parameterised kernels are “slid” over the input in a convolutional operation
to process groups of neighbouring inputs together. The same learned kernels
can process different parts of the input, thus sharing parameters and reducing
the number of parameters in the network. Additionally, convolutional neural
networks typically use pooling operations to reduce the dimensionality of the
input and make the network more robust to small translations of the input.
These operations are a form of inductive bias that allow convolutional neural
networks to effectively learn representations of high-dimensional inputs.
• Recurrent neural networks are a family of neural networks designed to pro-
cess sequences of inputs. A sequence is iteratively processed by the same
neural network which maintains a hidden state as a compact representation of
the history of previous inputs. Before processing each sequence, the hidden
state is initialised. At each step, the current hidden state and input of the
sequence is fed into the network to obtain the output for the current step
and a new hidden state. This process is repeated to compute the outputs
for each step of the sequence. The most common recurrent neural network
architectures are long short-term memory cells (LSTMs) and gated recurrent
units (GRUs), which allow the network to decide when to try to remember or
discard accumulated information within the hidden state.
In Chapter 8, we will build on the foundations of RL, introduced in Chapter 2,
and the newly introduced ideas of deep learning. We will introduce deep RL
algorithms that use neural networks to approximate value functions and policies,
as well as key challenges that need to be considered when applying deep learning
to RL. Chapter 9 will then extend these ideas to multi-agent RL.
Figure 8.1: Neural network architecture for action-value functions: the network
receives a state as its input, and outputs action-value estimates for every possible
(discrete) action.
1. In practice, this is typically achieved by masking out the maximum action value in the next state
with a value which takes on 0 if st+1 is terminal and 1 otherwise.
since computing the gradients of the loss with respect to the network param-
eters θ using backpropagation will compute backward passes through both
instances of the value function. However, in this case we want to update the
value estimate for the current state-action pair (st , at ), and the bootstrapped
value estimate should only serve as a target value to optimise the main value
estimate Q(st , at ; θ) towards. To avoid computing gradients through the boot-
strapped target value, we can stop any gradient flow backwards through the
target.2 This detail has to be considered in all deep RL algorithms which use
bootstrapping to estimate target values. Pseudocode for this algorithm with
an -greedy exploration policy, which we will refer to as deep Q-learning, is
shown in Algorithm 10. In the following sections and chapters, we will denote
the parameters of neural networks which represent value functions with θ.
This simple extension of Q-learning suffers from two important issues. First,
the moving target problem is further exacerbated by representing the value
function with function approximation, since value estimates for all states and
actions continually change as the value function is updated. Second, the strong
correlation of consecutive samples used to update the value function leads to
undesirable overfitting of the network to the most recent experiences. The
2. Deep learning libraries such as PyTorch, Tensorflow, and Jax all provide functionality to restrict
the gradient computation to specific components within the loss. This is an implementation detail,
even though an important one, and not essential to understand the core components within the
following algorithms.
following two sections will focus on these challenges and how to address them
for deep reinforcement learning.
3. As a reminder, an algorithm is called off-policy when the experience used to update its policy is
gathered by following a behaviour policy different from the policy that is being learned.
It is important to see that the deadly triad requires all three components. With-
out function approximation, the target value estimates would never be updated
and overestimated without actively visiting these states. Without bootstrapped
target values, value estimates could only be diverging for state-action pairs
that are never visited (due to the absence of correction), and such divergence
would be without consequence for other value estimates. If we used on-policy
experience samples instead of off-policy samples, the bootstrapped target values
would be computed using states and actions that are also visited by the current
policy and, thus, overestimation of target values can be corrected once these
states and actions are visited and their value estimates are updated.
To reduce the instability of training caused by the moving target problem and
reduce the risk of diverging value estimates, we can equip the agent with an
additional network. This so-called target network with parameters θ uses the
same architecture as the main value function and is initialised with the same
parameters. The target network can then be used instead of the main value
function to compute bootstrapped target values:
t
r
if st+1 is terminal
t t+1 0
y = r + γ maxa0 Q(s , a ; θ) otherwise
t (8.4)
| {z }
target network
These target values can be used to compute the same loss of deep Q-learning
(Equation 8.2).
Instead of optimising the target network with gradient descent, the parameters
of the target network are periodically updated by copying the current parameters
of the main value function network to the target network, i.e. θ ← θ . This
process ensures that the bootstrapped target values are not too far from the
estimates of the main value function, but remain unchanged for a fixed number
of updates and thereby increase the stability of target values. Furthermore,
we reduce the risk of diverging value estimates caused by the deadly triad by
decoupling the network used to compute bootstrapped value estimates for the
target from the main value function. Pseudocode for deep Q-learning with target
networks is given in Algorithm 11.
the training data, and secondly, that all data points are sampled from the same
training distribution. Both of these components of the i.i.d. assumption are typi-
cally violated in RL. For the former, individual samples of experiences, given
by tuples of state, action, reward, and next state, are clearly not independent
and highly correlated. This correlation is obvious given the formalisation of an
environment as an MDP. The experience at time step t is directly dependent
on and, therefore, not independent from the experience at time step t – 1, with
st+1 and rt being determined by the transition function conditioned on st and at .
Regarding the assumption of data points being sampled from the same training
distribution, the distribution of encountered experiences in RL depends on the
currently executed policy. Therefore, changes in the policy also cause changes
in the distribution of experiences.
But why are we concerned about these correlations? What impact do these
correlations have on the training of the agent using deep value functions?
Consider the example illustrated in Figure 8.2, in which an agent controls
a spaceship. For several time steps, the agent receives similar sequences of
experiences in which it approaches the goal location from the right direction
(Section 8.1.3). The agent sequentially updates its value function parameters
using these experiences, which may lead to the agent becoming specialised for
these particular, recent experiences of approaching the goal location from the
···
right side. Suppose that, after several such episodes, the agent has to approach
the goal from the left side (Section 8.1.3), but throughout the previous updates
has learned a value function which is specialised to states where the spaceship
is located to the right of the goal location. This specialised value function may
provide inaccurate value estimates for states where the spaceship is located to
the left of the goal location and, thus, may fail to successfully land from this
side. Moreover, updating the value function with the experience samples from
this most recent episode might lead to the agent forgetting how to approach
the goal from the right side. This phenomenon is referred to as catastrophic
forgetting and is a fundamental challenge of deep learning. This challenge is
further exacerbated in RL since changes in the policy will lead to changes in the
data distribution encountered by the agent, and changes in the data distribution
might change the optimal policy. This dependence can lead to oscillating or
even diverging policies.
To address these issues, we can randomise the experience samples used
to train the agent. Instead of using the sequential experiences as the agent
receives them to update the value function, experience samples are collected
in a so-called replay buffer D. For training of the value function, a batch of
experiences B is sampled uniformly at random from the replay buffer, B ∼ U(D).
This sampling has two additional benefits for the optimisation of the value
function: (1) experiences can be reused multiple times for training, which can
improve sample efficiency, and (2) by computing the value loss over batches
of experiences rather than for an individual experience sample, we can obtain
a more stable gradient for the network optimisation with lower variance. This
effect is similar to the benefits of mini-batch gradient descent over stochastic
gradient descent discussed in Section 7.4.2. During training, the mean squared
error loss is computed over the batch and minimised to update the parameters
with the targets of the kth experience sample yk being computed as given in Equa-
tion 8.3. Formally, the replay buffer can be denoted as a set D = {(st , at , rt , st+1 )}
of experience samples. Typically, the replay buffer is implemented with a fixed
capacity as a first-in-first-out queue, i.e. once the buffer is filled to its capacity
with experience samples, the oldest experiences are continually replaced as
new experience samples are added to the buffer. It is important to note that
experiences within the buffer have been generated using the policy of the agent
at earlier time steps during training. Therefore, these experiences are off-policy
and, thus, a replay buffer can only be used to train off-policy RL algorithms
such as algorithms based on Q-learning.
1.0
Deep Q-Learning
0.8
Deep Q-Learning + target network
Deep Q-Learning + replay buffer
Evaluation returns
DQN
0.6
0.4
0.2
0.0
(a) Single-agent level-based 0 20000 40000 60000 80000 100000
Environment time steps
foraging environment
(b) Learning curves
To see the impact of both the target network and replay buffer on the
learning of the agent, we show the learning curves of four algorithms in the
single-agent level-based foraging environment in Figure 8.3: Deep Q-learning
(Algorithm 10), deep Q-learning with target networks, deep Q-learning with
a replay buffer, and the full DQN algorithm with a replay buffer and target
networks (Algorithm 12). In this environment, visualised in Figure 8.3a, the
agent moves within an 8 × 8 gridworld to collect a single item.4 The agent and
the item are randomly placed at the beginning of each episode. To collect the
item, and receive a reward of +1, the agent has to move next to the item and
select its collect action. For any other action, the agent receives a reward of
0. We see that training the agent with deep Q-learning leads to a slow and
unstable increase in evaluation returns. Adding target networks leads to no
notable improvement in performance. Training the agent with deep Q-learning
and batches sampled from a replay buffer slightly increases evaluation returns
in some runs, but performance remains noisy across runs. Finally, training
the agent with the full DQN algorithm, i.e. using target networks and a replay
buffer, leads to a stable and quick increase in performance and convergence to
near-optimal discounted returns.
This experiment demonstrates that, in isolation, neither the addition of target
networks nor of a replay buffer are sufficient to train the agent with deep
Q-learning in this environment. Adding a target network reduces stability
issues caused by the moving target problem, but the agent still receives highly
correlated samples and is unable to train its value function to generalise over all
initial positions of the agent and item, which are randomised at the beginning
of each episode. Training the agent with a replay buffer addresses the issue
of correlated experiences, but without target networks suffers from unstable
optimisation due to the moving target problem. Only the combination of both
of these ideas in the DQN algorithm leads to a stable learning process.
4. In contrast to the multi-agent level-based foraging environment introduced in Figure 1.2, the
agent and item have no levels and collection will always be successful as long as the agent is located
next to the item.
maxa0 Q(st+1 , a0 ; θ). This maximum value estimate is likely to select an action-
value estimate that is overestimated. By decoupling the selection of the action
from the estimation of its value, the overestimation can be reduced. For DQN,
this can be achieved by computing the target value with the greedy action in the
next state being determined by the main value function, arg maxa0 Q(st+1 , a0 ; θ),
and using the target network, Q(st+1 , ·; θ), to evaluate its value:
(
t rt if st+1 is terminal
y = t (8.6)
r + γQ(st+1 , arg maxa0 Q(st+1 , a0 ; θ); θ) otherwise
Decoupling the greedy action selection and the value estimation in this way
reduces the risk of overestimation, since the main value network and target
network are unlikely to overestimate the action value of the same actions. For
example, even if the main value network overestimates the value for action a0
and, thus, identifies this action as the greedy action, the target value will not
be overestimated unless the target network overestimates the action value for
action a0 as well. Likewise, if the target network overestimates the value of
action a0 , this overestimation does not affect the target value unless the main
value network also identifies action a0 as the greedy action. By substituting
the previous target computation from Equation 8.4 in Algorithm 12 with this
new target, we obtain the double DQN (DDQN) algorithm (van Hasselt, Guez,
and Silver 2016). Given its simplicity and effectiveness, the DDQN target
computation is commonly found in many deep RL and MARL algorithms that
are based on DQN.
Besides DDQN, many extensions to the original DQN algorithm have been
proposed to improve its performance and stability. Schaul et al. (2016) argue
that not all experiences in the replay buffer are equally important. Therefore,
instead of sampling experiences uniform-randomly from the replay buffer, they
propose to prioritise experiences with larger temporal-difference errors during
the sampling process. Fortunato et al. (2018) propose to add parameterised and
learned noise to the weights of the neural network to encourage exploration.
Wang et al. (2016) show that decoupling the action-value function into a state-
value function and an advantage function can simplify the learning process
and improve generalisation of action-value estimates. Instead of learning a
single point estimate for each action value, Bellemare, Dabney, and Munos
(2017) propose to learn a distribution over possible values for each action. By
combining many of these extensions, we obtain a more advanced version of
the DQN algorithm, called Rainbow, which has shown to exhibit significantly
higher performance than DQN across Atari games (Hessel et al. 2018).
5. We note that there are other policy representations for value-based RL algorithms than -greedy
policies. For example, agents can follow a Boltzmann policy which is similar to a softmax policy
over the action-value estimates with a decaying temperature parameter, or agents can follow
a deterministic exploration policy given with upper confidence bounds (UCB) from the bandit
literature. However, all these policies of value-based algorithms are applied for exploration and
converge to a deterministic policy for evaluation.
1.0 1.0
Greedy ( = 0) Probabilistic [1/3, 1/3, 1/3]
0.8 -greedy ( = 0.1) 0.8 Probabilistic [0.8, 0.1, 0.1]
-greedy ( = 0.5) Probabilistic [0.1, 0.0, 0.9]
-greedy ( = 1.0) Probabilistic [0.4, 0.1, 0.5]
0.6 0.6
π(· | s)
π(· | s)
0.4 0.4
0.2 0.2
0.0 0.0
a1 a2 a3 a1 a2 a3
Actions Actions
Figure 8.4: Illustration of the varying flexibility of (a) -greedy policies and (b)
softmax policies (Equation 8.7) for a task with three discrete actions.
(Figure 8.1). The policy network receives a state as input and outputs a scalar
output for every action, l(s, a), to represent the preference of the policy to select
action a in state s. These preferences are transformed into a probability distribu-
tion using the softmax function, defined as the exponential of the preferences
divided by the sum of the exponentials of all preferences:
el(s,a;φ)
π(a | s; φ) = P l(s,a0 ;φ)
(8.7)
a0 ∈A e
In the remainder of this book, we will always denote the parameters of a param-
eterised value function with θ, and denote the parameters of a parameterised
policy with φ.
in the environment6 , and Qπ represents the value for a given action and state
under the policy. The function J is similar to a loss function (Section 7.4.1) but
with the key difference that we aim to maximise rather than minimise it. As we
can see, the policy gradient does not depend on unknown information about the
environment, such as the transition function and reward function, so we can use
it in RL where we typically assume these functions to be unknown.7
Note that the policy gradient theorem assumes the state-visitation distribution
and action values to be given under the currently optimised policy π. This
assumption becomes more apparent by rewriting the policy gradient as an
expectation under the current policy. For this derivation, we make use of the
fact that the policy gradient theorem defines a sum over states of the environment,
weighted by their respective probability of occurring under the current policy,
Pr(s | π):
X X
∇φ J(φ) ∝ Pr(s | π) Qπ (s, a)∇φ π(a | s; φ) (8.9)
s∈S a∈A
" #
X
= Ea∼π(·|s;φ) Qπ (s, a)∇φ π(a | s; φ) (8.10)
a∈A
" #
X
π ∇φ π(a | s; φ)
= Ea∼π(·|s;φ) π(a | s; φ)Q (s, a) (8.11)
π(a | s; φ)
a∈A
∇φ π(a | s; φ)
= Ea∼π(·|s;φ) Qπ (s, a) (8.12)
π(a | s; φ)
π
= Ea∼π(·|s;φ) Q (s, a)∇φ log π(a | s; φ) (8.13)
We denote the natural logarithm with log unless stated otherwise. We can
see that the state distribution and action selection are both induced by the
policy and, thus, fall within the expectation in Equation 8.10 and Equation 8.12,
respectively. In the end, we obtain a simple expression (Equation 8.13) within
the expectation under the current policy to express the gradients of the policy
parameters towards policy improvement. This expectation also clearly illustrates
the restriction that follows from the policy gradient theorem: the optimisation
6. Similar to the definition of the distribution over full histories under π in Equation 4.3, we can
define the state-visitation distribution by omitting the observation function needed to account for
partial observability.
7. In this section we provide the policy gradient for episodic environments. For continual environ-
ments, in which the experience of the agent is not split into finite episodes, we need to substitute
the expected returns as a measure of quality of the policy with a suitable metric for the continual
setting. One candidate for a metric of the quality of the policy for continual environments would be
average rewards which capture the expected reward received by the agent under the state-visitation
distribution and action selection of the current policy at any time step.
of the parameterised policy is limited to on-policy data, i.e. the data used to
optimise π is generated by the policy π itself (Section 2.6). Therefore, data
collected by interacting with the environment using any different policy π 0 ,
in particular also including previous policies obtained during the training of
π, can not be used to update π following the policy gradient theorem. The
application of a replay buffer, as seen in Section 8.1.3, would constitute such a
violation of the assumption, because it contains experiences generated by “older
and outdated” versions of our currently optimised policy. Therefore, a replay
buffer cannot be used to update a policy following the policy gradient theorem.
Furthermore, we have to approximate the expected returns under our current
policy denoted with Qπ . Most algorithms which train an action-value function,
such as DQN and other algorithms based on the classical Q-learning algorithm,
do not satisfy this requirement. Instead, they directly approximate the optimal
value function, i.e. the expected returns under the optimal policy, by following
the Bellman optimality equation (Section 2.6).
Looking at Equation 8.12 of the policy gradient theorem further provides an
intuitive interpretation for policy improvement:
∇φ π(a | s; φ)
π
∇φ J(φ) = Ea∼π(·|s;φ) Q (s, a)
π(a | s; φ)
The numerator of the fraction within the expression, ∇φ π(a | s; φ), represents
the gradient of the policy pointing in the direction in parameter space that most
increases the probability of repeating action a on future visits to state s. This
gradient is weighted by the quality of the action a in state s as given by the
value or expected returns of the policy Qπ (s, a). This weighting ensures that the
policy optimises its parameters such that actions with higher expected returns
become more probable than actions with lower expected returns. Lastly, the
denominator of the fraction, π(a | s; φ), can be thought of as a normalising factor
to correct for the data distribution induced by the policy. The policy π might
take some actions with significantly higher probabilities than others and, hence,
more updates might be done to the policy parameters to increase the probability
of more likely actions. To account for this factor, the policy gradient needs to
be normalised by the inverse of the action probability under the policy.
Algorithm 13 REINFORCE
1: Initialise policy network π with random parameters φ
2: Repeat for every episode:
3: for time step t = 0, 1, 2, . . . , T – 1 do
4: Observe current state st
5: Sample action at ∼ π(· | st ; φ)
6: Apply action at ; observe reward rt and next state st+1
PT–1 PT–1 τ –t τ
7: Loss L(φ) ← – T1 t=0 τ =t γ r log π(at | st ; φ)
8: Update parameters φ by minimising the loss L(φ)
it. Monte Carlo estimation is one possible sampling method, which uses on-
policy samples of episodic returns to approximate the expected returns of the
policy. Using the policy gradient theorem with Monte Carlo samples gives rise
to REINFORCE (Williams 1992), which minimises the following loss for the
episodic history h = {s0 , a0 , r0 , …, sT–1 , aT–1 , rT–1 , sT }:
T–1 T–1
!
1 X X τ –t τ
L(φ) = – γ r log π(at | st ; φ) (8.14)
T τ =t
t=0
T–1 T–1
!
1 X X τ –t τ τ τ +1
=– γ R(s , a , s ) log π(at | st ; φ) (8.15)
T τ =t
t=0
Given the policy gradient theorem provides a gradient in the direction of policies
with higher expected returns and we want to define a to-be-minimised loss, this
loss corresponds to the negative policy gradient (Equation 8.13) with Monte
Carlo estimates of the expected returns under the current policy π. During
training, the REINFORCE algorithm first collects a full episodic trajectory with
history h by using its current policy π. After an episode has terminated, the
return estimate is computed to estimate the policy gradient that minimises the
loss given in Equation 8.15. Pseudocode for the REINFORCE algorithm is
given in Algorithm 13.
Unfortunately, Monte Carlo return estimates have high variance, which leads
to high variance of gradients and unstable training in REINFORCE. This high
variance arises due to the returns of each episode depending on all states and
actions encountered within the episode. Both states and actions are samples of a
probabilistic transition function and policy, respectively. To reduce the variance
of return estimates, we can subtract a baseline from the return estimates. For
any baseline b(s) defined over the state s, the gradient derived from the policy
gradient theorem (Equation 8.13) remains unchanged in expectation. Therefore,
even when substracting a baseline, we still optimise the parameters of the policy
to maximise its expected returns but we reduce the variance of the computed
gradients. To see that the policy gradient remains unchanged, we can rewrite
the policy gradient theorem as follows:
X X
∇φ J(φ) ∝ Pr(s | π) (Qπ (s, a) – b(s)) ∇φ π(a | s; φ) (8.16)
s∈S a∈A
" #
X
π
= Eπ (Q (s, a) – b(s)) ∇φ π(a | s; φ) (8.17)
a∈A
" #
X ∇ φ π(a | s; φ)
= Eπ π(a | s; φ) (Qπ (s, a) – b(s)) (8.18)
π(a | s; φ)
a∈A
∇φ π(a | s; φ)
= Eπ (Qπ (s, a) – b(s)) (8.19)
π(a | s; φ)
= Eπ (Qπ (s, a) – b(s)) ∇φ log π(a | s; φ)
(8.20)
π
= Eπ Q (s, a)∇φ log π(a | s; φ) – Eπ b(s)∇φ log π(a | s; φ) (8.21)
X X
= Eπ Qπ (s, a)∇φ log π(a | s; φ) –
Pr(s | π) b(s)∇φ π(a | s; φ)
s∈S a∈A
(8.22)
π X X
= Eπ Q (s, a)∇φ log π(a | s; φ) – Pr(s | π)b(s)∇φ π(a | s; φ)
s∈S a∈A
(8.23)
X
= Eπ Qπ (s, a)∇φ log π(a | s; φ) –
Pr(s | π)b(s)∇φ 1 (8.24)
s∈S
X
= Eπ Qπ (s, a)∇φ log π(a | s; φ) –
Pr(s | π)b(s)0 (8.25)
s∈S
st , aτ ∼ π(· | sτ )
(8.31)
For N = T, with T being the episode length, the computed return estimate
corresponds to the Monte Carlo episodic returns with no bootstrapped value
estimates, as used in REINFORCE. These return estimates have high variance
but are unbiased. For N = 1, we obtain one-step bootstrapped return estimates,
as given in Equation 8.30, with low variance and high bias. Using the hyperpa-
rameter of N allows us to choose between bias and variance of return estimates.
Figure 8.5 illustrates this trade-off of N-step return estimates at the example
of an actor-critic algorithm. We train a policy and state-value function using
the A2C algorithm with N-step returns (detailed in Section 8.2.5) in a single-
agent level-based foraging environment (Figure 8.3a, page 192). After training
with N = 5, we collect 10,000 episodes with the trained policy and compute the
bias and variance of N-step return estimates with N ∈ [1, 10] and Monte Carlo
return estimates using the trained critic and dataset of episodes. As expected,
the variance of the N-step return estimates increases with increasing N and
Monte Carlo returns exhibit the highest variance. In contrast, the bias gradually
decreases with increasing N close to the level of Monte Carlo returns, which are
unbiased. In practice, N-step returns are commonly applied for small N, such
as N = 5 or N = 10, to obtain return estimates with fairly low bias and variance.
For notational brevity, we will write pseudocode and equations using one-
step bootstrapped return estimates, but note that N-step return estimates can be
applied to substitute any of these value estimates. In the following subsections,
we will introduce two actor-critic algorithms: advantage actor-critic (A2C) and
proximal policy optimisation (PPO).
8. Mnih et al. (2016) originally proposed the Asynchronous Advantage Actor Critic (A3C) algorithm,
which uses asynchronous threads to collect experience from the environments. For simplicity, and
because in practice it often does not make a difference, we avoid the asynchronous aspect of this
algorithm and introduce a simplified version of its synchronous implementation (A2C). We will
further discuss asynchronous and synchronous parallelisation of training in Section 8.2.8.
1e 3 1e 2
0.8
Bias
0.6
0.6
0.4
0.4
0.2 0.2
0.0 1-step 2-step 3-step 4-step 5-step 6-step 7-step 8-step 9-step 10-step MC 0.0
Figure 8.5: Variance and bias of N-step return estimates for N ∈ {1, …, 10} and
Monte Carlo returns for a state-value function trained with A2C for 100,000
time steps in a single-agent level-based foraging environment visualised in
Figure 8.3a. We use N-step return estimates with N = 5 during training.
is given by
Advπ (s, a) = Qπ (s, a) – V π (s) (8.32)
π π
with Q and V representing action-value and state-value functions with respect
to a policy π, respectively. For the policy π, the action value Qπ (s, a) represents
the expected returns of π when first applying action a in state s and afterwards
following π. In contrast, the state value V π (s) represents the expected return
when following the policy π already in state s rather than taking a specific
predetermined action. The advantage can therefore be understood as quantifying
how much higher the expected returns are when applying the specific action a
compared to following the policy π in state s. The advantage takes on positive
values whenever the chosen action a achieves higher expected return than the
current policy π. Similarly, the advantage is negative when the chosen action a
achieves lower expected return than the current policy π. This interpretation of
the advantage can be used to guide the optimisation of the policy. For a positive
advantage, we should increase the probability of the policy π to select action a
in state s; and we should decrease the probability of the policy π to select action
a in state s whenever the advantage is negative. In the following, we will omit
the superscript π for brevity and assume that the advantage and value functions
are computed with respect to the current policy π.
As defined in Equation 8.32, estimating the advantage requires both an action-
value function and a state-value function. Fortunately, we can estimate an
action-value function using the immediate rewards and the state-value estimate
of the following state, while accounting for the fact that the value of terminal
states must be zero as explained in Section 8.1.1:
(
t t rt if st+1 is terminal
Q(s , a ) = t (8.33)
r + γV(st+1 ) otherwise
With this estimation of the action-value function, we only rely on a state-value
function to approximate the advantage:
(
t t t t t rt – V(st ) if st+1 is terminal
Adv(s , a ) = Q(s , a ) – V(s ) = t
r + γV(st+1 ) – V(st ) otherwise
(8.34)
Similar to return estimates (Equation 8.31), we can estimate the advantage using
N-step returns to reduce the variance of the obtained estimate:
N–1
(
X –V(st ) if st+N is terminal
Adv(st , at ) = γ τ R(st+τ , at+τ , st+τ +1 )
τ =0 +γ N V(st+N ) – V(st ) otherwise
(8.35)
In A2C, we optimise the parameters φ of the actor to maximise the advantage
by minimising the following loss:
L(φ) = –Adv(st , at ) log π(at | st ; φ) (8.36)
For the optimisation of the parameters of the critic θ, we compute the squared
error of the value estimate of the current state and the bootstrapped target
estimate yt (here one-step bootstrapped target estimate)
(
t rt if st+1 is terminal
y = t t+1
(8.37)
r + γV(s ; θ) otherwise
leading to the following loss:
2
L(θ) = yt – V(st ; θ) (8.38)
Commonly, multi-step target estimates are used to reduce the variance of the
critic’s loss. In this case, the target can be computed as shown in Equation 8.31.
Full pseudocode for the A2C algorithm is given in Algorithm 14. We denote
this algorithm as “Simplified A2C” because the algorithm was originally pro-
posed with multi-step estimates as defined in Equations 8.31 and 8.35 and two
further techniques: asynchronous or synchronous parallelisation of training, and
entropy regularisation to incentivise exploration. We will discuss parallelisation
of training in Section 8.2.8. Entropy regularisation adds an additional term to
the actor loss given by the negative entropy of the policy in the current state:
X
– H (π(· | s; φ)) = π(a | s; φ) log π(a | s; φ) (8.39)
a∈A
The entropy of the policy π is a measure of the policy’s uncertainty. The entropy
is maximised for a uniform distribution, i.e. when the policy selects all actions
with equal probability. Minimising the negative entropy, i.e. maximising the
entropy, as part of the actor loss penalises the policy for assigning a very high
probability to any action. This regularisation term discourages premature con-
vergence to a suboptimal close-to-deterministic policy and, thereby, incentivises
exploration.
the expected performance of the policy. The risk of such significant changes
of the policy as a consequence of a single gradient optimisation step can be
reduced using trust regions. Intuitively, trust regions define an area within the
space of policy parameters in which the policy would not change significantly
and, thus, we would “trust” that the resulting policy with such parameters
would not lead to a significant reduction in performance. Trust region policy
optimisation (TRPO) (Schulman et al. 2015) constrains each optimisation step
of the policy in policy gradient RL algorithms to a small trust region. In this
way, TRPO reduces the risk of any degradation in quality of the policy and,
thereby, gradually and safely improves the quality of the policy. However, each
update with TRPO requires either solving a constrained optimisation problem
or computing a penalty term which are both computationally expensive.
Proximal policy optimisation (PPO)9 (Schulman et al. 2017) builds on the
idea of trust regions for policy optimisation and computes a computationally
efficient surrogate objective to avoid large jumps in policy in a single optimisa-
tion step. This surrogate objective makes use of importance sampling weights
ρ(s, a) which are defined as the fraction of probabilities of selecting a given
action a in state s for two policies:
π(a | s; φ)
ρ(s, a) = (8.40)
πβ (a | s)
For the importance sampling weight ρ, the policy π parameterised by φ rep-
resents the policy we want to optimise, and πβ represents a behaviour policy
which was used to select the action a in state s. The importance sampling weight
can be thought of as a factor to shift from the distribution of data encountered
under policy πβ to the data distribution of policy π. This factor adjusts the data
distribution to make the data generated by πβ “appear” on-policy for π.
Using these weights, PPO is able to update the policy multiple times using
the same data. Typical policy gradient algorithms rely on the policy gradient
theorem and, therefore, assume data to be on-policy. However, after a single
update of the policy, the policy changes and any previously collected data is
not on-policy anymore. Additionally, the importance sampling weight can be
seen as a measure of divergence of the policies, with an importance weight of 1
corresponding to both policies having equal probabilities of selecting an action
a in state s. PPO makes use of these properties to update the policy multiple
times using the same data, and restrict the change of the policy by restricting the
9. Schulman et al. (2017) propose two versions of the PPO algorithm in their work. In this section,
we describe the PPO algorithm with a clipped surrogate objective. This algorithm is simpler and
more common than the alternative PPO algorithm with a KL divergence penalty term, and often
just referred to as PPO.
importance sampling weight. This is achieved using a actor loss with clipped
importance sampling weights
ρ(st , at )Adv(st , at ),
L(φ) = – min (8.41)
clip ρ(st , at ), 1 – , 1 + Adv(st , at )
where ρ represents the importance sampling weight as defined in Equation 8.40,
the advantage Adv(st , at ) is computed using a state-value function as given in
Equation 8.34, and represents a hyperparameter which determines how much
the policy is allowed to deviate from the previous policy πβ .
Pseudocode for PPO is given in Algorithm 15, with Ne denoting the num-
ber of epochs, i.e. number of updates, for a given batch of data. Similar to
A2C, we denote the presented algorithm as “Simplified PPO” because it is
typically applied in combination with parallel training (Section 8.2.8), entropy
regularisation (Section 8.2.5), and N-step returns (Section 8.2.4) to obtain larger
1.0
0.8
Evaluation returns
0.6
0.4
0.2
REINFORCE
A2C
PPO
0.0
0 20000 40000 60000 80000 100000
Environment time steps
Figure 8.6: Learning curves for REINFORCE, A2C, and PPO in the single-
agent level-based foraging environment shown in Figure 8.3a. We train all
algorithms for 100,000 time steps. Visualised learning curves and shading
correspond to the mean and standard deviation across discounted episodic
returns across five runs with different random seeds. Across all algorithms, we
use a discount factor γ = 0.99 during training, small critic and actor networks
with two layers of 32 hidden units, ReLU activation function, and conduct a
small grid search to identify suitable hyperparameters. REINFORCE is trained
without a baseline and with a learning rate of α = 10–3 . For A2C and PPO we
use N-step returns with N = 5 and a learning rate of 3 · 10–4 . Lastly, PPO uses a
clipping parameter = 0.2 and optimises its networks for Ne = 4 epochs using
the same batch of experience.
batches of data for more stable optimisation and improve the exploration of the
algorithm.
Agent
Policy Training
Vectorisation
Thread 1 Thread n
Environment 1
... Environment n
more stable than REINFORCE due to less variant return estimates, and the
agent robustly obtains optimal returns across all episodes thereafter. Lastly, we
see that PPO is able to learn slightly faster than A2C, which can be explained
by its optimisation being able to use each batch of experiences multiple times.
1.0 1.0
0.8 0.8
Evaluation returns
Evaluation returns
0.6 0.6
0.4 0.4
A2C 1 sychronous environment A2C 1 sychronous environment
0.2
A2C 4 sychronous environments 0.2
A2C 4 sychronous environments
A2C 16 sychronous environments A2C 16 sychronous environments
A2C 64 sychronous environments A2C 64 sychronous environments
0.0 0.0
0.0 0.2 0.4 0.6 0.8 1.0 0 50 100 150 200 250 300
Environment time steps 1e6
Training time (in seconds)
(a) Performance per time steps trained (b) Performance per training time
10. We commonly use N-step returns to obtain value estimates with reduced bias. In this case, the
batch of K · N experiences across N time steps and K environments can be used to optimise the
networks.
experiences within the same wall-clock time11 for larger values of K but also
uses more experience for each optimisation of its networks as it makes use of
the experience across all K environments.
To illustrate the impact of synchronous parallel environments on the training
of the agent, we train the simplified A2C algorithm with synchronous envi-
ronments (Algorithm 16) with varying numbers of synchronous environments
in a single-agent level-based foraging environment. The environment has a
larger 12 × 12 grid and the agent has to collect two items to receive all possible
episodic rewards. We train the agent with K ∈ {1, 4, 16, 64} for five minutes
and present (discounted) evaluation returns across time steps trained and wall-
clock training time in Figure 8.8. On one hand, the experiment illustrates that
training for smaller values of K can be comparably sample-efficient because of
the frequent optimisations of the agent’s networks (Figure 8.8a). On the other
hand, the optimisation is less stable because each optimisation is computed over
a smaller batch of experiences, so the agent trained with K = 1 environments
does not converge to the optimal policy. Inspecting the wall-clock efficiency in
Figure 8.8b, we see that training with larger values of K can be considerably
more efficient (whilst making use of larger amounts of computational resources).
However, these benefits diminish with growing number of synchronous environ-
ments. For large values of K, individual threads have to wait for other threads to
finish their transition to receive the next action from the agent and continue their
interaction, so idle time of threads increases the more parallel environments are
deployed.
Asynchronous training, visualised in Figure 8.9, instead parallelises the opti-
misation of the agent. In addition to keeping an instance of the environment,
each thread keeps a copy of the agent to interact with its environment instance.
Each thread separately computes the loss and gradients to optimise the parame-
ters of the agent’s networks based only on data collected within the environment
instance of the thread. Once gradients are computed, the networks of the central
agent are updated and the newly obtained parameters are sent to all threads
to update their agent copies. Therefore, the agent’s networks are separately
updated by all threads, with each thread’s optimisation only using data collected
within that particular thread. Implementing memory-safe asynchronous paral-
lelisation and optimisation is more involved and requires careful engineering
considerations, but has the major benefit of not relying on information of all
11. Wall-clock time is the elapsed time that a clock or stopwatch would measure between the start
and end of the training, irrespective of the required resources and number of parallel threads or
processes.
Agent
Policy
Thread 1 Thread n
Policy Training Policy Training
...
Environment 1 Environment n
We further note that these techniques are most commonly applied with on-
policy policy gradient because these algorithms cannot use replay buffers, but
parallel training and data collection are also applicable to off-policy algorithms.
Also, in this section we focused on two conceptually simple ideas to improve
the efficiency of policy gradient algorithms. More complex ideas have been
proposed in the literature, which largely focus on parallelisation across large
computing infrastructures (e.g. Espeholt et al. (2020) and Espeholt et al. (2018)).
12. This approach is not applicable to tasks with potentially infinite episodes.
as its input and continually updates its hidden state to represent information
about the full episodic history of observations. At the beginning of each episode,
this hidden state is initialised to a zero-valued vector. By using recurrent neural
networks within the architecture of policies and value functions in partially
observable environments, RL algorithms can receive one observation at a time to
internally represent the full episodic history. Using this architectural change for
all networks, the deep RL algorithms introduced in this chapter can be applied
to partially observable environments. Among recurrent network architectures,
gated recurrent units (GRU) and long short-term memory (LSTM) networks are
commonly used in deep RL (Hausknecht and Stone 2015; Jaderberg et al. 2019;
Morad et al. 2023; Rashid et al. 2018).
8.4 Summary
In this chapter, we introduced deep single-agent RL algorithms which leverage
neural networks to represent the value function and policy of the agent. In the
following, we summarise the key concepts introduced in this chapter:
• With tabular value functions, updating the value of a state only changes the
value estimate for that particular state. In contrast, with function approxima-
tion such as neural networks, updating the value of a state may change the
value estimates of all states. This generalisation property is a key advantage
of function approximation over tabular representations but also introduces
new challenges.
• The moving target problem arises whenever function approximation is used
to compute bootstrapped target values. These target values depend on the
value of the next state, and thus change with every parameter update. To
address the resulting unstable training, target networks are introduced. These
networks are initialised as copies of the main value functions and used to
compute target values. Target networks are updated less frequently than the
value network and thus provide a more stable target value.
• Correlations of consecutive experiences represent a second challenge when
training an RL with function approximators. The experience of the agent
at any time strongly depends on the previous experience, and the overall
data distribution shifts as the policy changes. To address these problems, an
experience replay buffer is introduced. The replay buffer stores experiences
of transitions. During training, random batches of experiences can then be
sampled to update the value function. This approach breaks the temporal
correlations between consecutive transitions and allows to reuse experiences
has to be transformed into a single reward for training, which might be difficult
or impossible in general-sum games, (2) the central policy has to learn over
the joint action space which typically1 grows exponentially in the number of
agents, and (3) agents might be physically or virtually distributed entities which
might not allow for communication from and to a central policy for centralised
control. For example, for autonomous vehicles, it may not be realistic to expect
to transmit and receive the sensor and camera information of all surrounding
vehicles in real time. Furthermore, even if information sharing across vehicles
was possible and instantaneous, learning a centralised control policy to control
all vehicles would be very difficult due to the scale and complexity of the
problem. In this case, decentralised control is a more reasonable approach
to implement individual agents for each vehicle and to decompose the larger
single-agent problem into multiple smaller multi-agent problems.
1. In Section 5.4.4 we discussed an example in which the growth is not considered exponential.
function. Effectively, with changes in the policies of other agents, the transition,
observation and reward functions, as perceived by each individual agent, change
as well. These changes can lead to unstable learning and poor convergence
of independent learning. Despite these challenges, independent learning often
performs well in practice and can serve as a first step to develop more complex
algorithms. Building on independent learning algorithms with tabular value
functions (Section 5.3.2), we will introduce deep independent learning algo-
rithms in Section 9.3. We note that independent learning is not the only way
decentralised training can be implemented. Agent modelling (Section 6.3), for
example, covers a variety of methods that can be used to model the changing
behaviour of other agents in the environment.
will denote the policy, value function, and action-value function of agent i
with π(·; φi ), V(·; θi ), and Q(·; θi ), respectively. To keep notation slim, we do
not explicitly denote the policy and value functions with a subscript of the
respective agent when it is clear from the parameterisation which agent is
considered. For example, we will write π(·; φi ) instead of πi (·; φi ). We note
that this notation simplifies potential distinction between agent networks; the
networks might differ beyond the parameterisation, e.g. due to varying input
and output dimensions for different observation and action spaces across agents.
In partially observable multi-agent games, agents only receive local obser-
vations about the environment, which might be different between agents
(Section 3.4). During centralised training, agents may use joint information
across all agents during training but condition their policies only on their local
observation history. To represent this discrepancy of information available dur-
ing training and execution, we will introduce all following MARL algorithms
using notation for partially observable environments. For this purpose, we
will use h to denote histories of observations. However, we note that some
centralised training algorithms make use of the full state s of the environment
during training. In these cases, we will specifically highlight where the local
observation history hti = (o0i , o1i , …, oti ) of agent i, the joint observation history
ht = (o0 , o1 , …, ot ), or the state st at time step t should be used. In environments
where the full state is not available and only partial observations are accessi-
ble, the state of the environment can be approximated by the joint observation
history st ≈ ht . In fully observable environments, agents use the state s of the
environment instead of individual or joint observation histories to make use of
the full information available to agents.
Section 8.3 discussed the application of recurrent neural networks to effi-
ciently condition deep value functions and policies on the history of observations.
These networks can receive one observation at a time and internally represent
the observation history as a hidden state. Due to this practice and for nota-
tional brevity, many publications define the policy and value functions of deep
RL algorithms as a function conditioned only on the most recent observation.
Instead, we will explicitly condition the policy and value function networks at
time step t on the history of local or joint observations, denoted with hti and ht ,
respectively.
with θi denoting the parameters of agent i’s target network. The value function
parameters are optimised simultaneously by minimising the aggregate loss
across all agents L(θ1 ) + L(θ2 ) + … + L(θN ). We also present pseudocode of the
IDQN algorithm in Algorithm 17.
It is worth noting that the replay buffer can lead to problems in IDQN that do
not occur in single-agent RL. In multi-agent environments, the behaviour of
an agent is not only determined by its own actions but also influenced by the
actions of other agents in the environment. Therefore, an agent might receive
identical observation and select identical action but still receive significantly
different returns depending on the policies of other agents. This creates a
challenge when using a replay buffer, as it assumes that the stored experiences
will remain relevant over time. However, in MARL, the policies of other agents
are changing as they learn, and this can make the experiences stored in the
replay buffer quickly become outdated.
To understand the problem that can occur with off-policy algorithms like
IDQN that use a replay buffer to store experiences in multi-agent settings,
consider an example of two agents learning to play chess. Suppose agent 1 is
using a specific opening move that is initially effective against agent 2, but is
actually a weak strategy in the long run. Agent 2 has not yet learned to counter
this opening and so is not penalizing agent 1 for using it. As agent 2 learns to
counter the opening, the old opening sequences where agent 1 had success will
still be stored in the replay buffer. Agent 1 will keep learning from these old
examples, even though they are no longer relevant to the current state of the
learning process, since the improved policy of agent 2 can counter those actions.
This can lead to a situation in which agent 1 continues to use the weak opening
even after it has been countered by the agent 2.
One way to address the issue of non-stationarity when using a replay buffer in
MARL is to use smaller replay buffers. As a result, the buffer will more quickly
reach its maximum capacity and older experiences will be removed. This
reduces the risk of stored experiences becoming outdated, and allows agents
to learn from recent data. However, there are also more elaborate methods.
Foerster et al. (2017) use a replay buffer that also stores importance sampling
weights for each experience. These importance sampling weights contain the
2. We note that agents do not observe the full state of the environment, but only their own obser-
vations. However, agents receive information about the termination of episodes, and can use this
information to compute the respective targets. For more details on how agents receive termination
information in practice, see Section 10.1.
action probabilities of all agents to select the action they chose, and serve as a
snapshot of the policies of other agents at the time of collecting the experience.
Using the importance sampling weights, agents can re-weight the experiences
in the replay buffer to account for the changing policies of other agents and,
thereby, correct the non-stationarity of the data distribution. Hysteretic Q-
learning (Matignon, Laurent, and Le Fort-Piat 2007) uses a smaller learning
rate for updates which would decrease action-value estimates. This approach
is motivated by the observation that these decreasing estimates might be the
result of the stochasticity of other agents’ policies. Similarly, the concept of
leniency (Panait, Tuyls, and Luke 2008) ignores decreasing updates of action-
value estimates with a given probability that decreases throughout training, to
account for the stochasticity of agent policies early in training. Both concepts
of hysteretic and lenient learning have been applied to deep multi-agent RL
algorithms (Omidshafiei et al. 2017; Palmer et al. 2018), and extended by
distinguishing between negative updates as a consequence of miscoordination
or stochasticity (Palmer, Savani, and Tuyls 2019).
8: for agent i = 1, . . . , n do
PT–1 PT–1 τ –t τ
9: Loss L(φi ) ← – T1 t=0 τ =t γ ri log π(ati | hti ; φi )
10: Update parameters φi by minimising the loss L(φi )
generated by the agents’ current policies. As the policies of the agents evolve
over time, the experiences collected by each agent reflect the most up-to-date
policies of the other agents in the environment. This feature of on-policy
algorithms is important in multi-agent settings because the policies of the
agents are continually evolving. Learning from the most up-to-date policies of
other agents enables each agent to adapt to changes in the environment or the
policies of the other agents, and, thus, can lead to more stable learning.
Consider again the chess example from Section 9.3.1, which discussed how
algorithms that use replay buffers may be unable to learn that an opening has
been countered by the other agent. On-policy algorithms like REINFORCE
are less susceptible to this problem because they always learn from the most
up-to-date policies of the other agents. In the chess example, as agent 2 learns
to counter the opening, the trajectories collected by agent 1 will immediately
reflect that change. In this way, on-policy algorithms can adapt more quickly to
changes in the policies of the other agents.
The A2C (Section 8.2.5) and PPO (Section 8.2.6) algorithms can also be
extended similarly to REINFORCE and be applied independently in multi-agent
settings. We now describe the independent learning algorithm for A2C with
multiple environments (Section 8.2.8). In independent A2C with parallel envi-
ronments, each agent receives experiences from multiple parallel environments.
Therefore, the collected experience across all agents and parallel environments
form batches of higher dimensionality. For example, the observations collected
Similar matrices can be formed for the actions and rewards. Calculating the
A2C loss requires iterating and summing over the individual losses. The policy
loss of a single agent over the data collected from an environment k becomes
L(φi | k) = – rit,k + γV(ht+1,k
i ; θ i ) – V(ht,k
i ; θ i ) log π(at,k t,k
i | hi ; φi ) (9.4)
| {z }
Advantage Adv(ht,k t,k
i , ai )
and the final policy loss sums and averages over the batch
K
1 XX
L(φ) = L(φi | k) (9.5)
K i∈I
k=1
with much larger state spaces. To demonstrate this, we train IA2C on level-
based foraging with a grid size of 15 × 15, and random initial locations and
levels for the agents and items, which adheres to the open-sourced level-based
foraging environment presented in Chapter 11. This learning problem has a
state space that is many orders of magnitude larger than the tasks we explored
in Part I. Indicatively, two agents and two items in a 15 × 15 grid result in
approximately 5 billion (5 × 109 ) combinations.
Our experiments with IA2C on the larger level-based foraging environment
demonstrate the power of deep RL algorithms in tackling tasks with larger state
spaces. As shown in Figure 9.1a, IA2C learned a joint policy that collects
all the available items, which is indicated by the evaluation returns reaching
values close to 1 (as detailed in Section 11.3.1). This result was reached within
1.0 1.0
0.8 0.8
Evaluation returns
Evaluation returns
0.6 0.6
0.4 0.4
0.2 0.2
Independent A2C Independent A2C
0.0 0.0
0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5
Environment time steps 1e7
Environment time steps 1e7
(a) Task with two agents and two items. (b) Task with three agents and three items.
3. We define a general variant of the multi-agent policy gradient theorem using expected returns
for partially observable environments. Related work first defined the multi-agent policy gradient
theorem using centralised critics conditioned on the joint observation of all agents (Lowe et
al. 2017) or the environment state (Foerster, Farquhar, et al. 2018). However, as we will discuss
in Section 9.4.2, using critics conditioned on agent observations or environment states rather
than observation histories with potential additional centralised information can result in bias and
increased variance of policy gradients (Kuba et al. 2021; Lyu et al. 2023).
4. The definitions for the distribution of full histories, Pr(ĥ | π), the action-value function, Qπ
i , and
the extraction of the individual agent’s history using σi (ĥ) can be found in Section 4.1.
Figure 9.2: Architecture of a centralised critic for agent i. The critic is condi-
tioned on the individual agent observation history and centralised information,
and outputs a single scalar to represent the approximated value.
The value loss of this centralised critic, also shown in Figure 9.2, becomes:
2
L(θi ) = yi – V(hti , zt ; θi ) with yi = rit + γV(hti , zt ; θi ). (9.10)
This modification offers an important benefit: the network gains access to
more information with which to make predictions. Access to such centralised
information can prove beneficial in some environments, by having the critic be
more accurate in its estimation of the returns of a policy. Moreover, by having
access to information about all other agents, a centralised critic may adapt faster
to the non-stationary policies of other agents.
Lyu et al. (2023) studied centralised critics and what their inputs should con-
tain. At a minimum, the critic should be conditioned on the agent’s observation
history hti , which is the policy’s input. Without it, the critic may be biased
since it has less information than the actor itself, and may be unable infer the
policy it is evaluating. To illustrate this concept intuitively, we may envision a
situation involving a policy denoted as π(a | ht ), a value function conditioned
on the current observation, V(ot ), and a partially observable environment. In
this environment, we have a sequence of observations, o1 , o2 , o3 , and another
sequence in which the first two observations are different, ō1 , ō2 , o3 . Let us
assume that, in this environment, these two trajectories would lead to signifi-
cantly different outcomes for the agent in terms of expected returns. Now, if
we were to estimate the value V(o3 ) without considering the historical data,
it would inherently be biased. This bias emerges because V(o3 ) would have
to account for both potential outcomes. Conversely, the policy, conditioned
on the observation history ht , has the ability to recognise the difference in the
past observations and select the correct actions. In their work, Lyu et al. (2023)
even show that any additional information may introduce higher variance in the
policy gradient during training, without improving the theoretical convergence
guarantees. This is because the additional information may simply add noise to
the estimation.
Nevertheless, when we examine empirical performance, particularly in the
context of deep RL, we sometimes find a beneficial trade-off when incorporating
additional information zt , which may be attributed to the fact that the theoretical
assumptions behind the study of Lyu et al. (2023) do not universally hold true in
every domain. For instance, an underlying assumption is the convergence of the
critics to the true value functions under the current policies of all agents. How-
ever, in the deep learning setting, this assumption might not hold, as the critic’s
20
30
Evaluation returns
Listener
40
50
60 Centralised-Critic A2C
Speaker
Independent A2C
0 1 2 3 4 5 6 7 8
Environment time steps 1e7
Figure 9.3: Using a centralised critic with A2C for the speaker-listener game
allows the agents to learn the task by addressing the partial observability of
the environment. With the centralised critic, the algorithm converges to higher
returns than IA2C.
only observe the shape of the landmark that will maximise the common reward
and can transmit an integer from 1 to 3 to the listener. The goal of the game is for
the two agents to learn to cooperate such that the listener always moves to the
goal landmark. Given the partial observability, this game is quite challenging:
the speaker has to learn to recognise the different shapes and transmit a distinct
message long enough for the listener to learn to move to the correct landmark.
Conditioning the critic of each agent on the observations of both agents (the
landmark goal and the positions of the agent and landmarks) allows them to
learn more precise value estimates despite the partial observability, and leads to
the performance advantages seen in Figure 9.3b.
use these techniques to train the action-value critic for multi-agent actor-critic
algorithms, recall that the multi-agent policy gradient theorem requires the
estimation of the expected returns under the current policies of all agents. To
train the critic to estimate the expected returns under the current policies of
all agents, we have to use on-policy data. In contrast, a replay buffer contains
off-policy data which may not represent the distribution of experiences under the
current policies. Likewise, DQN updates directly train the critic to approximate
the optimal returns instead of the expected returns under the current policies.
To represent a centralised action-value critic as a neural network, the network
might take the individual observation history and centralised information as
inputs and output one action-value for each joint action. However, this network
architecture would suffer from large dimensionality of the output due to the
exponential growth of the joint action space with the number of agents. To
avoid such a large output dimension, we can instead model the action-value
critic of agent i to receive the actions a–i of all other agents as additional inputs.
Then, the network only computes a single output for each action of agent i
which corresponds to the action-value for the given individual observation
history hti , additional centralised information zt , and the joint action composed
by concatenating the particular action of agent i, ai , with the joint action of all
other agents, a–i . This architecture is illustrated in Figure 9.4.
an action-value function is its ability to directly estimate the impact of the action
selection on the expected returns. However, a value function without action
inputs, V(hi , z; θi ), can still be used to approximate the advantage (Equation 8.32)
which also provides preferences over particular actions. Additionally, training
an action-value function with one output for every possible action may be
more difficult than training a function with only a single output. Given these
considerations, why would we want to train action-value critics for multi-agent
actor-critic algorithms, instead of learning simpler critics only conditioned on
the agent observation history hi and potential centralised information z?
One motivation for training action-value critics is to use their action inputs to
address the multi-agent credit assignment problem, discussed in Section 5.4.3,
based on the concept of difference rewards (Tumer and Agogino 2007; Wolpert
and Tumer 2002). Difference rewards approximate the difference between the
received reward and the reward agent i would have received if it had chosen a
different action ãi :
di = Ri (s, hai , a–i i) – Ri (s, hãi , a–i i) (9.14)
The action ãi is also referred to as the default action. Difference rewards aim
to consider the counterfactual question of “Which reward would agent i have
received if it instead had selected its default action?”. Answering this question
is valuable in settings where all agents receive a common reward, because it
provides information about the concrete contribution of agent i to the received
reward. However, computing difference rewards in practice is often difficult
because (1) it is not clear how to select the default action for agent i, and (2)
computing Ri (s, hãi , a–i i) requires access to the reward function. The need to
determine a default action for each agent can be avoided by using the definition
of the aristocrat utility. Instead of the reward agent i would have received if
they had selected their default action, we subtract the expected rewards agent i
would have received if they had followed their current policy:
di = Ri (s, hai , a–i i) – Ea0i ∼πi Ri (s, ha0i , a–i i)
(9.15)
In this way, the aristocrat utility can be seen as the expected difference reward
where the default action is sampled from the current policy. Intuitively, the
aristocrat utility provides an indication of whether action ai is expected to lead
to better or worse rewards in expectation than sampling an action from the
current policy.
Given access to the reward function, Castellini et al. (2021) derive return
estimates over difference rewards determined by the aristocrat utility, and incor-
porate these return estimates into the REINFORCE algorithm for MARL. If the
reward function is not available, they propose to learn a model of the reward
function from experience in the environment, and use this model to estimate
the difference rewards.
Counterfactual multi-agent policy gradient (COMA) (Foerster, Farquhar,
et al. 2018) uses the same concepts to derive a centralised action-value critic5 to
compute a counterfactual baseline, which marginalises out the action of agent
i to estimate the advantage for selecting action ai over following the current
policy:
X
Advi (hi , z, a) = Q(hi , z, a; θ) – π(a0i | hi ; φi )Q(hi , z, ha0i , a–i i; θ) (9.16)
a0i ∈Ai
| {z }
counterfactual baseline
This advantage estimate looks similar to the one defined in Equation 8.32
(page 204), but the baseline is computed based on the aristocrat utility instead
of a value function V. The counterfactual baseline computes the expected
centralised value estimate for agent i for following its own policy, π(a0i , hi ; φi ),
and with the actions of other actions, a–i , being fixed. This baseline is shown to
not change the multi-agent policy gradient (Equation 9.9) in expectation, and
can efficiently be computed using the previously introduced architecture for
centralised action-value critics (Figure 9.4). To train the policy of agent i in
COMA, the action-value estimate in Equation 9.12 is replaced by the advantage
estimate in Equation 9.16. Despite its clear motivation, COMA empirically
suffers from high variance in its baseline (Kuba et al. 2021) and inconsistent
value estimates (Vasilev et al. 2021), which result in unstable training that can
lead to poor performance (Papoudakis et al. 2021).
5. In their original work, Foerster, Farquhar, et al. (2018) condition the critic only on the full state of
the environment and the joint action of all agents. As discussed in Section 9.4.2, the critic should be
additionally conditioned on the individual observation history to obtain an unbiased policy gradient.
A B C
A B
A 11‡ -30 0
A 4,4‡ 0,3
B -30 7† 0
B 3,0 2,2†
C 0 6 5
(a) Stag Hunt game
(b) Climbing game
Figure 9.5: (a) The Stag Hunt matrix game, also seen in Section 5.4.2. (b) The
Climbing matrix game. The Climbing game is a common-reward (which is
always no-conflict) game that shares similar characteristics to the Stag Hunt
game: an optimal Nash equilibrium and joint actions which are less rewarding
but are easier to reach. The Pareto-dominated equilibrium is denoted with † and
the Pareto-optimal one with ‡.
game, both agents prefer the outcome (A, A), making it no-conflict, although
they disagree on the second-best outcome: (B, A) for agent 1 and (A, B) for
agent 2. These two joint actions do not represent Nash equilibria, however, since
agents can unilaterally change their actions to improve their rewards. Finally,
action (B,B) is a Nash equilibrium since an agent changing its action (with the
other agent not doing so) decreases its reward.
When such games have multiple Nash equilibria, learning agents tend to
converge to less risky equilibria by preferring less risky actions (Papoudakis
et al. 2021). In the Stag Hunt example, agents selecting action B are guaranteed
a reward of at least 2, while action A might lead to a reward of 0. Therefore,
even if the highest reward can only be achieved by choosing action A, the
agents tend to learn the suboptimal (B, B) solution. This preference is easy to
illustrate for agents that do not model the actions of the other agents. Suppose
the two agents are initialised with a uniform random policy, π(A) = π(B) = 0.5,
before learning starts. At that moment, the expected reward of choosing action
A for agent 1 is 0.5 · 4 + 0.5 · 0 = 2.0 and for action B the expected reward is
0.5 · 3 + 0.5 · 2 = 2.5 (where 0.5 is the probability of the other agent selecting
A or B). Consequently, when applying the policy gradient, agent 1 learns to
assign a greater probability to action B. This reinforcement further strengthens
the risk-averse (B, B) equilibrium through a positive feedback loop, making
action A even less appealing for agent 2.
Pareto actor-critic (Pareto-AC) (Christianos, Papoudakis, and Albrecht 2023)
is a method of addressing this problem that builds on the multi-agent policy gra-
dient methods discussed in Sections 9.3.2, 9.4.2 and 9.4.4. Pareto-AC addresses
the equilibrium selection problem in no-conflict games by incorporating into the
policy gradient the fact that other agents have the same most preferred outcome.
In no-conflict games, a joint policy that satisfies the most preferred outcome for
an agent is Pareto-optimal (Equation 4.25), as that joint policy by definition has
the highest expected returns for all agents. For the training of agent i following
+
policy πi , the algorithm assumes that all other agents follow a policy π–i which
is in the set of policies that maximise the returns of agent i, i.e.:
+
π–i ∈ arg max Ui (πi , π–i ) (9.18)
π–i
+
Using π–i , the original Nash equilibrium objective (Equation 4.16) can be
changed to
+
πi ∈ arg max Ui (πi , π–i ), (9.19)
πi
which, following the actor-critic methodology, can be optimised by minimising
+
the following loss where the actions at–i of other agents are drawn from π–i :
However, as with any algorithm that uses a joint action-value function, this
calculation has unattractive scaling properties as the number of agents increases.
This issue is even more pronounced in the context of Pareto-AC, where the
explicit iteration over the joint actions of other agents necessitates tractable
approximations, a challenge that remains unresolved.
With Pareto-AC, agents tend to converge to more rewarding equilibria in
no-conflict games, even if the equilibria are riskier. We show experiments in
two environments to illustrate the difference between Pareto-AC and centralised
A2C. First, in Figure 9.6a, Pareto-AC is shown to converge to the Pareto-
optimal equilibrium (A, A) in the Climbing game (Figure 9.5b) while the A2C
algorithm with a centralised value function only converges to action (B,B),
which is a suboptimal solution for both agents. However, matrix games are
not the only situation in which this problem appears. Take the example of
level-based foraging (Section 5.4.3) with two agents and an item that always
needs both agents to cooperate to collect it. In addition, apply a penalty (–0.6
in this example) if an agent attempts to collect an item alone and fails. Such
a game has a suboptimal equilibrium of never trying to collect items in order
1.0
Pareto-AC
20
0.8
Centralised A2C
15
Evaluation returns
Evaluation returns
10 0.6
5
0.4
0
5 0.2
Pareto-AC
10 Centralised A2C 0.0
0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4
0 100000 200000 300000 400000 500000 Environment time steps 1e8
Environment time steps
(b) Learning curves on the level-based
(a) Learning curves on the Climbing game.
foraging game (with penalty).
Figure 9.6: Learning curves comparing centralised A2C with Pareto-AC in the
Climbing and level-based foraging games.
to avoid getting penalised. In Figure 9.6b, we can see the results of Pareto-AC
and a centralised A2C algorithm in level-based foraging with two agents that
always need to cooperate to collect an item that has been placed on a 5 × 5
grid. Centralised A2C quickly learns to avoid collecting the item so as to
not receive any penalties. In contrast, Pareto-AC is optimistic by using the
centralised action-value critic and the modified policy objective (Equation 9.20)
and eventually learns to solve the task.
Pareto-AC is an example of how a centralised action-value function can be
used to improve learning in multi-agent problems. The centralised action-value
function is not only used to learn a joint-action value but also to guide the policy
gradient to the most promising equilibria. With its decentralised actors, which
are only conditioned on the observations, the algorithm adheres to the CTDE
paradigm, enabling the individual agents to execute their actions independently
during execution.
6. The IGM property was first defined by Son et al. (2019). Previously, Rashid et al. (2018)
introduced the term consistency following an almost identical definition. However, prior work
typically defines the IGM property as the equivalence of the greedy joint action with respect to
Q(h, z, a; θ) and the joint action of the individually greedy actions. These definitions disregard the
possibility that there could be multiple greedy actions and, thus, we define the IGM property as
the equivalence of the greedy joint action with respect to Q(h, z, a; θ) and the joint action of the
individually greedy actions that maximise the joint action-value function.
7. Recall from Section 4.1 that a full history ĥ contains the history of states, joint observations, and
joint rewards; σ(ĥ) returns the history of joint observations within full history ĥ; z could for example
include the last state in the history denoted with s(ĥ). We use σi (ĥ) to denote the observation history
of agent i.
Any such linear decomposition satisfies the IGM property (Equation 9.25),
as we will show below:
Proof. Let ĥ be a full history, h = σ(ĥ) be the joint observation history, hi = σi (ĥ)
be the observation history of agent i, and z denote potential additional centralised
information. We will first prove that for any greedy joint action a∗ ∈ A∗ (h, z; θ),
the individual actions in a∗ of all agents are also greedy with respect to their
This work is subject to a Creative Commons CC-BY-NC-ND license. c MIT Press
Multi-Agent Deep Reinforcement Learning 247
individual utility functions. Then, we will show that any greedy individual
actions of all agents together represent a greedy joint action with respect to
Q(h, z, a; θ).
"⇒" Let a∗ = (a∗1 , …, a∗n ) ∈ A∗ (h, z; θ) be a greedy joint action with respect to
Q(h, z, ·; θ). To prove that the individual actions of all agents a∗1 , …, a∗n are
also greedy with respect to agents’ individual utility functions, i.e.
∀i ∈ I : a∗i ∈ A∗i (hi ; θi ), (9.31)
we need to show that for any agent i and any action ai ∈ Ai , we have
∀i ∈ I : Q(hi , a∗i ; θi ) ≥ Q(hi , ai ; θi ). (9.32)
We will prove this by contradiction. Assume that there exists an agent i and
an action ai ∈ Ai such that
Q(hi , a∗i ; θi ) < Q(hi , ai ; θi ). (9.33)
Given the linear decomposition of Q(h, z, a; θ), we have
X
Q(h, z, a∗ ; θ) = Q(hi , a∗i ; θi ) (9.34)
i∈I
X
= Q(hi , a∗i ; θi ) + Q(hj , a∗j ; θj ) (9.35)
j6=i
X
< Q(hi , ai ; θi ) + Q(hj , a∗j ; θj ) (9.36)
j6=i
"⇐" Let a∗1 ∈ A∗1 (h1 ; θ1 ), …, a∗n ∈ A∗n (hn ; θn ) be greedy individual actions of
all agents with respect to their individual utility functions. Let a∗ =
(a∗1 , …, a∗n ) be the joint action composed of such greedy individual actions
of all agents. To prove that a∗ is a greedy joint action with respect to
Q(h, z, a; θ), we need to show that for any joint action a0 ∈ A(h, z; θ), we
have
Q(h, z, a∗ ; θ) ≥ Q(h, z, a0 ; θ). (9.38)
= Q(h, z, a0 ; θ) (9.42)
where a = (a01 , …, a0n ) is any
0
joint action composed of arbitrary individual
actions a01 ∈ A1 , …, a0n ∈ An . Therefore, a∗ is a greedy joint action with
respect to Q(h, z, a; θ) (Equation 9.38).
This concludes the proof.
with
X
Q(ht , at ; θ) = Q(hti , ati ; θi ) and (9.44)
i∈I
X
max Q(ht+1 , a; θ̄) = max Q(ht+1
i , ai ; θ̄i ). (9.45)
a∈A ai ∈Ai
i∈I
Using this optimisation objective, all agents implicitly learn their individual
utility functions. These functions are computationally tractable and naturally
gives rise to policies with agent i choosing the greedy action with respect to its
individual utility function for any given history, i.e. ati = arg maxai ∈Ai Q(hti , ai ; θi ).
Note that no constraints are imposed on the individual utility functions and only
the shared rewards are used during the optimisation.
8. An observant reader may have noticed that this decomposition is simply a disconnected
coordination graph (Figure 9.7).
Figure 9.8: Network architecture of VDN and QMIX. Given the observation
histories, the individual utility networks output utility values for all actions of
the respective agents. For VDN, the centralised action-value function is then
approximated by the sum of the individual utilities for the chosen actions. For
QMIX, the centralised action-value function is approximated by a monotonic
aggregation computed by the mixing network, which is parameterised by the
output of a trained hypernetwork.
Proof. Let ĥ be a full history, h = σ(ĥ) be the joint observation history, hi = σi (ĥ)
be the observation history of agent i, and z denote potential additional centralised
information. We will first prove that for any greedy joint action a∗ ∈ A∗ (h, z; θ),
the individual actions in a∗ of all agents are also greedy with respect to their
individual utility functions. Then, we will show that any greedy individual
actions of all agents together represent a greedy joint action with respect to
Q(h, z, a; θ).
"⇒" Let a∗ = (a∗1 , …, a∗n ) ∈ A∗ (h, z; θ) be a greedy joint action with respect to
Q(h, z, ·; θ). To prove that the individual actions of all agents a∗1 , …, a∗n are
also greedy with respect to agents’ individual utility functions, i.e.
∀i ∈ I : a∗i ∈ A∗i (hi ; θi ), (9.48)
we need to show that for any agent i and any action ai ∈ Ai , we have
∀i ∈ I : Q(hi , a∗i ; θi ) ≥ Q(hi , ai ; θi ). (9.49)
We will prove this by contradiction. Assume that there exists an agent i and
an action ai ∈ Ai such that
Q(hi , a∗i ; θi ) < Q(hi , ai ; θi ). (9.50)
Given the monotonic decomposition of Q(h, z, a; θ), we have
Q(h, z, a∗ ; θ) (9.51)
= fmix Q(h1 , a∗1 ; θ1 ), …, Q(hi , a∗i ; θi ), …, Q(hn , a∗n ; θn ); θmix
(9.52)
< fmix Q(h1 , a∗1 ; θ1 ), …, Q(hi , ai ; θi ), …, Q(hn , a∗n ; θn ); θmix
(9.53)
= Q(h, z, ha∗–i , ai i; θ) (9.54)
where the inequality follows from Equation 9.50 and the monotonicity
of fmix with respect to its inputs (Equation 9.46). This contradicts the
assumption that a∗ is a greedy joint action with respect to Q(h, z, a; θ).
Therefore, Equation 9.50 cannot hold, and we have Equation 9.49, so the
individual actions of all agents within the greedy joint action a∗ are also
greedy with respect to their individual utility functions.
"⇐" Let a∗1 ∈ A∗1 (h1 ; θ1 ), …, a∗n ∈ A∗n (hn ; θn ) be greedy individual actions of
all agents with respect to their individual utility functions. Let a∗ =
(a∗1 , …, a∗n ) be the joint action composed of such greedy individual actions
of all agents. To prove that a∗ is a greedy joint action with respect to
Q(h, z, a; θ), we need to show that for any joint action a0 ∈ A(h, z; θ), we
have
Q(h, z, a∗ ; θ) ≥ Q(h, z, a0 ; θ). (9.55)
9. We note that in their original paper, Rashid et al. (2018) do not assume strict monotonicity of the
mixing network fmix with respect to its inputs. Instead, they assume that fmix is monotonic with
respect to its inputs, i.e.
∂Q(h, z, a; θ)
∀i ∈ I, ∀a ∈ A : ≥0. (9.59)
∂Q(hi , ai ; θi )
In this case, the proof of the implication "⇒" that the joint action composed of greedy individual
actions with respect to all agents’ utility functions is a greedy joint action with respect to Q(h, z, a; θ)
still holds. However, the proof of the implication "⇐" does not. To see why the strict monotonicity
assumption is necessary, consider the following counterexample for two agents with actions A1 =
{a1,1 , a1,2 } and A2 = {a2,1 , a2,2 }, respectively. Let a1,1 and a2,1 be the greedy actions with respect to
the agents’ individual utility functions
Q(h1 , a1,1 ; θ1 ) > Q(h1 , a1,2 ; θ1 ) and Q(h2 , a2,1 ; θ2 ) > Q(h2 , a2,2 ; θ2 ), (9.60)
and let the decomposed centralised action-value function be
Q(h, z, (a1 , a2 ); θ) = fmix (Q(h1 , a1 ; θ1 ), Q(h2 , a2 ; θ2 ); θmix ) = Q(h1 , a1 ; θ1 ) (9.61)
for some actions a1 ∈ A1 and a2 ∈ A2 , meaning that the mixing function only considers the first
agent’s utility and ignores the second agent’s utility. We can see that
∂Q(h, z, a; θ) ∂Q(h, z, a; θ)
=1≥0 and =0≥0 (9.62)
∂Q(h1 , a1 ; θ1 ) ∂Q(h2 , a2 ; θ2 )
so the monotonicity assumption (Equation 9.59) holds. However, the joint action a = (a1,1 , a2,2 ) is a
greedy joint action with respect to Q(h, z, a; θ), since a1,1 maximises the individual utility of agent
1 and the mixing function is maximised whenever the individual utility of agent 1 is maximised, but
a2,2 is not a greedy action with respect to the individual utility of agent 2. Therefore, the implication
"⇐" does not hold without the strict monotonicity assumption (Equation 9.46).
z as its input10 and outputs the parameters θmix of the mixing network (hence
the name "hyper"). To ensure positive weights, the hypernetwork fhyper applies
an absolute value function as activation function to the outputs corresponding
to the weight matrix of the mixing network fmix and, thus, ensures monotonic-
ity.11 Whenever Q(h, z, a; θ) is needed for optimisation, the individual utilities
Q(h1 , a1 ; θ1 ), …, Q(hn , an ; θn ) are computed and the mixing network parameters
θmix are obtained by feeding centralised information into the hypernetwork. The
utilities are then aggregated to Q(h, z, a; θ) using the mixing network with the
parameters received by the hypernetwork.
The entire architecture of QMIX is illustrated in Figure 9.8b. During optimi-
sation, all parameters θ of the decomposed centralised action-value function,
including the parameters of individual utility networks θ1 , …, θn and parameters
of the hypernetwork θhyper , are jointly optimised by minimising the value loss
given by
2
1 X
L(θ) = rt + γ max Q(ht+1 , zt+1 , a; θ̄) – Q(ht , zt , at ; θ)
B t t t t t+1 t+1 a∈A
(h ,z ,a ,r ,h ,z )∈B
(9.63)
over a batch B sampled from the replay buffer D with the centralised value
function and its target network being given by Equation 9.47 with parameters θ
and θ̄, respectively. The parameters of the mixing network are not optimised by
gradient-based optimisation, but instead are always obtained as an output of the
optimised hypernetwork. Algorithm 22 shows pseudocode for QMIX. It is also
worth noting that the replay buffer in QMIX needs to store centralised informa-
tion zt in addition to individual agent observations, since the hypernetwork is
conditioned on this information.
It is straightforward to see that any linear decomposition of the centralised
action-value function, as seen in Equation 9.30, also upholds the monotonicity
property, but there exist monotonic decompositions of the centralised action-
value function which are not linear. A simple example is the following linear
decomposition X
Q(h, z, a; θ) = αi (h)Q(hi , ai ; θi ) (9.64)
i∈I
where αi (h) ≥ 0 are positive weights. The weights can be interpreted as the
relative importance of each agent’s contribution to the centralised action-value
function given a particular joint observation history h. Any such weighting
10. In their original work, Rashid et al. (2018) condition the hypernetwork on the state, i.e. z = s.
11. We note that the absolute value function could allow zero weights which violate the necessary
strict monotonicity assumption. However, for the strict monotonicity to be violated, all weights
corresponding to a single agent’s utility would need to be zero, which does not occur in practise.
Algorithm 22 QMIX
1: Initialise n utility networks with random parameters θ1 , . . . , θn
2: Initialise n target networks with parameters θ1 = θ1 , . . . , θn = θn
3: Initialise hypernetwork with random parameters θhyper
4: Initialise a shared replay buffer D
5: for time step t = 0, 1, 2, . . . do
6: Collect current centralised information zt and observations ot1 , . . . , otn
7: for agent i = 1, . . . , n do
8: With probability : choose random action ati
9: Otherwise: choose ati ∈ arg maxai Q(hti , ai ; θi )
10: Apply actions; collect shared reward rt , next centralised information zt+1
and observations ot+1
1 , …, on
t+1
Q2 (A) Q2 (B)
Q1 (A) Q(A, A) Q(A, B)
Q1 (B) Q(B, A) Q(B, B)
Figure 9.9: Visualisation of the format in which the value decomposition of
algorithms will be shown.
utility functions. To see this, we can write the centralised action-value function
as
Q(a1 , a2 ) = Q1 (a1 ) + Q2 (a2 ) (9.65)
and assign concrete values to the individual utility functions:
Q1 (A) = 1, Q1 (B) = 5, Q2 (A) = 0, Q2 (B) = 4 (9.66)
It is easy to verify that these utilities indeed lead to the desired centralised action-
value function, but many similar assignments are possible. Figure 9.10 shows
this linear game as well as the individual utility functions and decomposed
centralised action-value functions learned by VDN and QMIX. As expected,
both VDN and QMIX are able to learn an accurate centralised action-value
function.12 The learned individual utility functions of VDN are different from
the ones given in Equation 9.66, but also accurately represent the centralised
action-value function. However, these results also emphasise the fact that these
algorithms are optimised to learn accurate centralised action-value functions,
which may result in individual action-value functions that are difficult to inter-
pret. In particular for QMIX, which aggregates individual utilities through
its monotonic mixing network, the individual utilities are difficult to inter-
pret besides their ability to represent the centralised action-value function and
policies of both agents.
The second matrix game is monotonic but not linear, i.e. the centralised
action-value function can be represented by a monotonic decomposition of the
individual action-value functions.
12. For all experiments, we train a feedforward neural network with two layers of 64 hidden units
and ReLU activation function for each agent’s utility function in both VDN and QMIX. All
networks are optimised with the Adam optimiser using a learning rate of α = 3 · 10–4 and batches of
128 experiences sampled from the replay buffer. Target networks are updated every 200 time steps.
To ensure sufficient exploration of all joint actions, agents follow a uniform policy throughout
training. For QMIX, the hypernetwork is represented by a two-layer feedforward neural network
with 32 hidden units each and the ReLU activation function between layers. For the mixing network,
we use a two-layer feedforward neural network with 64 units and the ELU activation function in
between layers. Both algorithms were trained until convergence (at most 20,000 time steps).
20.0
17.5
Q(a1 , a2 )
15.0
10 12.5
10.0
7.5
5.0
0 2.5
0.0
0
2
1
0 −1
0
)
−5 −2
−4
a2
−3 −3
2(
−2
Q1 (a−1
Q
−4
0
−5
1) 1
2
Figure 9.12: Visualisation of the QMIX mixing function fmix in the monotonic
matrix game. The mixing function is shown after convergence over the indi-
vidual utilities of both agents, with shading corresponding to the estimated
centralised action-value function. The mixed utilities of all four joint actions
are highlighted together with their centralised action-value estimate, with the
optimal joint action shown as a star.
Figure 9.11 shows the monotonic game as well as the learned individual
utilities and centralised action-value function for VDN and QMIX. As expected,
VDN is unable to learn an accurate centralised action-value function due to its
inability to represent non-linear value functions. In contrast, using its monotonic
mixing function, QMIX learns the true centralised action-value function. To
further illustrate the monotonic aggregation of individual utilities of both agents
through the mixing network fmix in QMIX, we visualise the mixing function
over the individual agents’ utilities in Figure 9.12. Due to the monotonicity
constraint, the estimated joint action values increase with increasing individual
utilities. The optimal joint action has a value of +10, which is significantly larger
A B
A 1 5
A B B 5 9
A 0 0
B 0 0 A B
A 0 0
B 0 10
Figure 9.13: Two-step common-reward stochastic game with three states. The
transition function across the three states and the reward function within each
state is shown here. In the initial state the agents receive no rewards for any
action and agent 1 decides which matrix game to choose in the next time step:
either the linearly decomposable game, as represented by the state in the top
right, or the monotonically decomposable game, as represented by the state in
the bottom right.
Figure 9.14: Learned value decomposition of (a) VDN and (b) QMIX in the
two-step common-reward stochastic game. Both algorithms were trained to
estimate undiscounted returns (γ = 1).
than the value of all the other joint actions, which is 0. QMIX’s monotonic
aggregation is able to accurately represent all these values by learning a mixing
function with a steep incline from the value estimates of suboptimal actions to
the optimal joint action value, as seen by the sharp change in shading in the
top-right corner of the visualisation.
However, it is worth noting that despite inaccurate value estimates, VDN
learns the optimal policy of (B,B) by greedily following its individual value
function. We often evaluate MARL algorithms based on the evaluation returns
(or rewards in non-repeated matrix games) and here VDN and QMIX would both
obtain the optimal reward of +10 by greedily following the learned individual
utility functions. To see that the inaccurate value estimates of VDN can be
A B C
A 11 -30 0
B -30 7 0
C 0 6 5
(a) Climbing game
–4.56 –4.15 3.28 –16.60 –0.24 –4.68
–4.28 –8.84 –8.43 –1.00 –7.44 –11.16 –11.16 –11.16
–6.10 –10.66 –10.25 –2.82 7.65 –11.15 2.34 –1.37
5.31 0.75 1.16 8.59 11.27 –4.95 8.72 5.01
(b) VDN – Climbing game (c) QMIX – Climbing game
Figure 9.15: (a) Reward table of the Climbing matrix game with the learned
value decomposition of (b) VDN and (c) QMIX.
13. The value and utility functions learned by VDN and QMIX at both steps at t = 1 in the two-step
game are different from the ones seen in Figures 9.10 and 9.11 due to different initialisation of the
networks and stochasticity introduced by the -greedy exploration policy. We train both algorithms
with γ = 1.0.
1.0
IDQN
0.8
VDN
QMIX
Evaluation returns
0.6
0.4
0.2
0.0
0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.00
Environment time steps 1e6
(a) Environment
(b) Learning curve
and the centralised action-value function Q over the joint history h, centralised
information z, and joint action a is parameterised by θq . These conditions use
three components: (1) individual utility functions {Q(hi , ai ; θi )}i∈I which are
combined to a linear decomposition akin to VDN, (2) a non-decomposed and
unrestricted centralised action-value function Q(h, z, a; θq ), and (3) a utility
function V(h, z; θv ). The unrestricted centralised action-value function does not
enable decentralised execution but is used as a supervision signal for the linear
decomposition through the sum of individual utility functions. However, in
partially observable environments, the utility functions might lack information
QTRAN optimises for the property stated in Equation 9.67 for the greedy joint
action. The second regularisation term is computed as follows
X
m= Q(hti , ati ; θi ) – Q(ht , zt , at ; θq ) + V(ht , zt ; θv ) (9.72)
i∈I
1 X
Lnopt ({θi }i∈I , θv ) = min(0, m)2 (9.73)
B
(ht ,zt ,at ,rt ,ht+1 ,zt+1 )∈B
and optimises for the property stated in Equation 9.67 for non-greedy joint
actions a.14 We note that the optimisation of QTRAN does not directly enforce
the formulated properties defined in Equations 9.67 and 9.68, but minimises the
additional loss terms (Equations 9.71 and 9.73), which reach their minimum
when the properties from Equations 9.67 and 9.68 are satisfied. Therefore,
these properties are only satisfied asymptotically and not throughout the entire
training process.
To get a sense of the centralised action-value functions QTRAN is able to
represent, we train the algorithm in the linear game, the monotonic game, and
the Climbing game seen in Section 9.5.4. We can see that the unconstrained cen-
tralised action-value function Q(h, z, a; θq ) of QTRAN almost exactly converges
to the true reward table for all three games (Figures 9.17a, 9.18a and 9.19a). The
learned linear decomposition does not converge to the correct centralised action
values (Figures 9.17b, 9.18b and 9.19b), but together with the learned utilities is
converging to values which satisfy the formulated constraints in Equation 9.67
for the linear game and Climbing game. For the monotonically decomposable
14. The QTRAN algorithm presented in this section is the QTRAN-base algorithm proposed by
Son et al. (2019). In their work, they also propose an alternative algorithm QTRAN-alt which uses
a counterfactual centralised action-value function, similar to COMA (Section 9.4.4), to compute an
alternative condition for non-greedy actions.
A B 2.01 4.13
A 0.99 5.00 1.82 3.83 5.95 V(h, z; θv ) =
B 5.00 9.00 3.77 5.78 7.9 1.10
(a) Q(h, z, a; θq ) (b) Linear decomposition (c) Utility
A B 0.75 3.83
A 0.01 0.00 0.72 1.47 4.55 V(h, z; θv ) =
B 0.00 10.00 3.75 4.50 7.58 2.45
(a) Q(h, z, a; θq ) (b) Linear decomposition (c) Utility
matrix game, the constraint for the optimal action a∗ = (B, B) is not satisfied
exactly with
X
Q(hi , a∗i ; θi ) – Q(h, z, a∗ ; θq ) + V(h, z; θv ) = 7.58 – 10.00 + 2.45 (9.74)
i∈I
= 0.03 (9.75)
6= 0 (9.76)
but is still close to zero. Given the constraints of Equation 9.67 are a sufficient
and necessary condition to satisfy the IGM property, we can see that QTRAN
is able to learn a decomposition of the centralised action-value function that
satisfies the IGM property for the linear and Climbing game, and is close to
satisfying it for the monotonic game. For all three games, the learned decompo-
sition of QTRAN gives rise to the optimal policy, indicating its ability to express
more complex decompositions than VDN and QMIX, which were unable to
learn the optimal policy in the Climbing game.
However, it is worth noting that QTRAN suffers from several issues in
complex environments. First, training the non-decomposed centralised action-
value function becomes challenging in tasks with a large number of agents or
actions, since the joint action space typically grows exponentially to the number
of agents. Second, the formal conditions underlying the QTRAN algorithm are
sufficient and necessary to ensure the IGM property, but the practical algorithm
relaxes these conditions for a tractable optimisation objective as detailed above.
This relaxation means that the IGM property is not ensured throughout training,
which can lead to degrading performance.
There exist several other value decomposition algorithms which, similar to
QTRAN, aim to expand on QMIX by either increasing the capacity of the
trained value functions or relaxing the monotonicity constraint. For example,
Rashid et al. (2020) observe that VDN and QMIX put equal importance on
learning value estimates for each possible joint action. However, to learn the
optimal policy and ensure the IGM property, greedy or potentially optimal
actions are more important. Based on this intuition, they propose the weighted
QMIX algorithm which trains a unconstrained mixing network in addition to the
typical utility functions and mixing network of QMIX. Using this unconstrained
centralised action-value function, the algorithm can learn a weighted value loss
which puts more emphasis on learning the value estimates of joint actions which
are considered potentially optimal. Similar to QTRAN, weighted QMIX is
not guaranteed to ensure the IGM property all throughout training, but the
algorithm is shown to improve on the performance of QMIX in many complex
environments. Instead, Wang et al. (2021) propose a duplex decomposition
which represents the centralised action-value function and action-conditioned
utility functions with value and advantage functions:
Q(h, z, a) = V(h, z) + A(h, z, a) with V(h, z) = max
0
Q(h, z, a0 ) (9.77)
a ∈A
of agent i. Then, the agent model π̂ji of agent i for the policy of agent j can be
updated by minimising the cross-entropy loss between the action probabilities
predicted by π̂ji and the true action of agent j:
L(φij ) = – log π̂ji (atj | hti ; φij ) (9.79)
Minimising this cross-entropy loss is equivalent to maximising the likelihood
of the agent model to select the action atj of agent j given the observation history
hti of agent i. We can train the agent models of all agents using mini-batches of
experiences sampled from a replay buffer.
To compute Equation 9.82, we need to sum over all possible joint actions
a–i of all other agents. This is intractable in environments with many agents
or large action spaces. To address this issue, we can approximate the expected
target value by sampling a fixed number K of joint actions from the models of
all other agents and computing the average target values over these sampled
joint actions. Let ĥ be the full history with hi = σi (ĥ) denoting the individual
observation history of agent i, then we can approximate the expected action
value for agent i with action ai and history hi as follows:
K
1X
AV(hi , ai ; θi ) = Q(hi , hai , ak–i i; θi ) (9.83)
K akj ∼π̂ji (·|hi )
k=1
Algorithm 23 shows pseudocode for the joint action learning algorithm with
neural networks. The computation of action values can either be instantiated
using the exact computation of Equation 9.82 or the approximate sampling-
version of Equation 9.83. The latter is more efficient but might introduce
additional variance in the learning process. To see how these two approaches
compare in practice, we train agents using both approaches in a small level-
based foraging environment in which agents have to cooperate to collect any
items (Figure 9.20). IDQN fails to learn policies which reliably collect items
whereas JAL-AM learns to collect items. Interestingly, the sampling-version of
JAL-AM learns faster and more reliably than the exact variant. This is likely due
1.0
IDQN
0.8
JAL-AM (exact)
JAL-AM (sampled K = 10)
Evaluation returns
0.6
0.4
0.2
0.0
0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.00
Environment time steps 1e6
to sampling variant prioritising action values over the most common actions of
the other agent for which the value function has been trained more and, thereby,
might provide more precise value estimates. Additionally, the sampling process
might introduce additional noise which might help to explore the environment.
Figure 9.21: The encoder network f e outputs a representation mti of the policies
of other agents. During training, a decoder network f d is trained to reconstruct
the action probabilities of all other agents’ policies from the representation.
During execution, the representation is used to condition the policy of the agent.
would like to condition the policies and value functions of agents on the policies
of all other agents. However, this may not be feasible for multiple reasons. First,
the policies of other agents might be available during centralised training but
are not known to the agent during decentralised execution. Second, the policies
of other agents might be too complex to be represented explicitly. For example,
the policies of other agents might be implemented using neural networks. In
this case, we could represent their policies by the parameters, φ–i = {φj }j6=i , of
their policy networks, but the number of parameters is typically too large to
effectively condition agent policies on it. Third, the policies of agents change
during learning. Given these challenges, we would like to obtain compact
representations of the policies of other agents that can be inferred from the
individual observations of an agent, and adapt as the policies of other agents
change. In the following, we will discuss how such representations of agent
policies can be learned.
Encoder-decoder neural network architectures are commonly used to learn
a compact representation of some data which is indicative of specific features.
As visualised in Figure 9.21, these types of architectures consist of two neural
networks: an encoder network f e and a decoder network f d . To learn represen-
tations of the policies of other agents from the individual observation history of
agent i, the encoder network f e takes as input the observation history and outputs
a representation, mti = f e (hti ; ψie ). The decoder network f d takes as input the rep-
resentation mti of the corresponding encoder and predicts the action probabilities
i,t
of all other agents at the current time step, π̂–i = f d (mti ; ψid ). π̂ji,t represents a
|Aj |-dimensional vector corresponding to the action probabilities for each action
of agent j at time step t according to agent i’s agent model. We denote the
probability of the model for action aj with π̂ji,t (aj ), and denote the parameters
of the encoder and decoder of agent i with ψie and ψid , respectively. Given
these reconstructions, the encoder and decoder networks are jointly trained to
minimise the cross-entropy loss for the predicted action probabilities and true
actions of all other agents at the current time step:
X
L(ψie , ψid ) = – log π̂ji,t (atj ) with π̂ji,t = f d f e (hti ; ψie ); ψid
(9.84)
j6=i
Using this loss, the encoder network e is trained to represent information within
mti which is indicative of the action selection of all other agents.
The policies and value functions of agents can then be additionally con-
ditioned on the representations outputted by the respective encoder of the
agent.15 One benefit of this approach of agent modelling is that it is agnos-
tic to the MARL algorithm used to learn the policies of agents. Any MARL
algorithm can be extended to condition the policies and value functions of
agents on the representations outputted by the encoder networks. For example,
to extend centralised A2C (Algorithm 20) with this approach, we can condition
the policy, π(· | hi , mi ; φi ), and centralised critic, V(hi , z, mi ; θi ), on the learned
representations.
To illustrate the benefits of this type of agent modelling, we train agents using
the centralised A2C algorithm with and without representation-based agent
modelling in a fully observable level-based foraging environment. Figure 9.22
visualises the environment and learning curves for all agents. We observe that
agents with agent models learn to collect items more quickly than agents without
agent models. Agents with agent models seemingly learn more efficiently, and
converge to higher average returns. Convergence to higher average returns and
notably lower variance across runs indicate that the policies learned with agent
models are more robustly collecting all food items for maximum returns.
Similar approaches of agent modelling with encoder-decoder architectures
have been applied in a slightly different setting in which a single main agent
is trained to interact with other fixed agents (Papoudakis, Christianos, and
Albrecht 2021; Rabinowitz et al. 2018; Zintgraf et al. 2021). In this setting,
the other agents do not learn during the training of the main agent, but follow
fixed policies sampled from a predetermined set of policies. In addition to
reconstructing the policies of other agents, these approaches propose to predict
the observations or current “mental state” of other agents in the decoder as an
indicator for the information other agents might have.
So far, we answered the question of how agents can gain knowledge about the
policies of other agents and leverage this knowledge during learning and acting.
15. During training, gradients from the MARL algorithm used to train policies and value functions
would naturally flow backwards through the representations mti and update the parameters ψie of
the encoder. To train the encoder purely from the encoder-decoder reconstruction loss defined in
Equation 9.84, we stop gradients of the MARL training from flowing into the encoder.
1.0
Centralised A2C
0.8
Centralised A2C + AM
Evaluation returns
0.6
0.4
0.2
0.0
0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6
Environment time steps 1e8
Beyond this question, agents can further consider how other agents might react
to their decision making, or how their actions might affect the learning of
other agents. Humans naturally consider how their actions might affect others,
and what others might believe about their own knowledge. This is known as
theory of mind (Premack and Woodruff 1978) or recursive reasoning (Albrecht
and Stone 2018; Doshi, Gmytrasiewicz, and Durfee 2020). The probabilistic
recursive reasoning framework of Wen et al. (2019) applies this concept to
MARL. In this framework, agents recursively reason about the belief of other
agents’ policies and how their actions might affect other agents. The framework
is implemented using Bayesian variational inference to model the believes of
agents about the policies of other agents as well as the uncertainty of these
beliefs. These believes are updated and recursively propagated, and integrated
into fully decentralised actor-critic and value-based MARL algorithms.
Opponent shaping is a similar idea, which tries to answer the following
question: how can agents leverage the fact that other agents are learning by
shaping their behaviour to their own advantage?16 If the main agent knows
the optimisation objective or learning rule by which other agents are updating
their policies, then we can compute higher-order gradients through the learning
objective of other agents to account for the changes in their policy caused by
actions taking by the main agent. This idea can be integrated into deep RL
algorithms (Foerster, Chen, et al. 2018; Letcher et al. 2019) to shape the policies
of other agents. More broadly, this approach can be viewed as a meta learning
problem in which multiple agents learn policies within the “inner” learning
process, whereas the “outer” learning process optimises the policy of the main
agent considering the learning of all other agents in the “inner” process (Kim
et al. 2021; Lu et al. 2022).
16. A related idea is the concept of value of information, discussed in Section 6.3.3, as well as
ideas based on “manipulation learning” (Fudenberg and Levine 1998) and “teaching” (Shoham and
Leyton-Brown 2008).
(a) Agents have to move to two (b) Both agents have to move to the
different locations. same location.
actions, and receive the same rewards for performing similar actions to gather
the items. We define such a game as an environment with weakly homogeneous
agents.
Definition 14 (Weakly homogeneous agents) An environment has weakly
homogeneous agents if for any joint policy π = (π1 , π2 , . . . , πn ) and permutation
between agents σ : I → I, the following holds:
Ui (π) = Uσ(i) hπσ(1) , πσ(2) , . . . , πσ(n) i , ∀i ∈ I (9.85)
At a first glance, one could assume that in such an environment, the optimal
policy of the agents would be to behave identically. This is not necessarily
true, as we will discuss in the rest of the section, especially given the numerous
solution concepts in MARL (Chapter 4). However, to capture environments
in which all agents act identically under the optimal policy, we also define an
environment with strongly homogeneous agents.
Definition 15 (Strongly homogeneous agents) An environment has strongly
homogeneous agents if the optimal joint policy consists of identical individual
policies, formally:
• The environment has weakly homogeneous agents.
• The optimal joint policy π ∗ = (π1 , π2 , . . . , πn ) consists of identical
policies π1 = π2 = · · · = πn .
1.0
0.8
Evaluation returns
0.6
0.4
Fully Shared Parameters
0.2 Shared Parameters (Critic-only)
Shared Parameters (Actor-only)
0.0 Not Sharing Parameters
0.0 0.5 1.0 1.5 2.0 2.5 3.0
Environment time steps 1e8
Figure 9.24: Learning curves for the Independent Actor-Critic algorithm with
parameter sharing across both critic and actor, parameter sharing only for the
critic or actor, and without any parameter sharing on the level-based foraging
environment.
of agents. Secondly, the shared parameters are updated using the experiences
generated by all agents, resulting in a more diverse and larger set of trajectories
for training.
The caveat is that strongly homogeneous agents is a strong assumption, and
one that might be difficult to verify. If agents share parameters, the policies
of the agents are identical. An environment with weakly homogeneous agents
will not see the same benefit from parameter sharing. An example was already
shown in Figure 9.23a, in which the agents are weakly homogeneous but the
optimal policies of the agents are different and cannot be learned if the constraint
of Equation 9.86 is enforced. So, when approaching a problem one should
consider the properties of the environment and decide whether techniques such
as parameter sharing could be useful.
Theoretically, we might be able to train diverse policies and still retain the
benefits of shared parameters in environments with weakly homogeneous agents.
To achieve that, the observations of the agents can include their index i to create a
new observation ōti . Since each agent will always receive an observation which
contains their own index, they will theoretically be able to develop distinct
behaviours. But in practice, learning distinct policies by relying on the index
may not be sufficient, as the representational capacity of neural networks might
not be enough to represent several distinct strategies. The work of Christianos et
al. (2021) discusses the limitations of parameter sharing and parameter sharing
with an observation that contains the agent index.
To further illustrate the advantages of parameter sharing in an environment
with strongly homogeneous agents, we have executed four variations of the
independent A2C algorithm (Algorithm 19) in a level-based foraging environ-
ment. The four variations are i) using parameter sharing in both actor and critic,
ii) parameter sharing only on the critic, iii) parameter sharing only on the actor,
and iv) no parameter sharing. Figure 9.24 presents the average returns during
training for the four variations in a level-based foraging environment with a
grid-world of size 6 × 6, two level-1 agents, and one level-2 item. In the figure,
we observe that sharing parameters leads to the algorithm converging in fewer
time steps. However, it does not necessarily increase the final converged returns,
as all algorithms eventually reach a similar policy in this example.
to correct for transitions collected by other algorithms and apply the idea of
experience sharing to on-policy settings.
In on-policy based algorithms, such as A2C or PPO, each agent generates
one on-policy trajectory in every episode. The algorithms we have seen so
far, such as independent A2C or A2C with centralised state-value critics, used
the experience of each agent’s own sampled trajectory to update the agent’s
networks with respect to their policy loss (e.g. Equation 9.4). The SEAC
algorithm reuses trajectories of other agents while considering that they have
been collected as off-policy data, i.e. the trajectories were generated by agents
executing different policies than the one being optimised. Correcting for off-
policy samples can be achieved with importance sampling. The loss for such
off-policy policy gradient optimisation from a behavioural policy πβ can be
written as
π(at | ht ; φ) t
r + γV(ht+1 ; θ) – V(ht ; θ) log π(at | ht ; φ),
L(φ) = – t t
πβ (a | h )
in a similar fashion to Equation 8.40 in Section 8.2.6. In the independent A2C
framework of Section 9.3.2, we can extend the policy loss to use the agent’s own
trajectories (denoted with i) along with the experience of other agents (denoted
with k), shown below:
L(φi ) = – rit + γV(ht+1 t t t
i ; θi ) – V(hi ; θi ) log π(ai | hi ; φi )
X π(at | ht ; φi )
k k
rt + γV(ht+1 t t t
–λ k ; θi ) – V(hk ; θi ) log π(ak | hk ; φi )
π(atk | htk ; φk ) k
k6=i
(9.87)
Using this loss function, each agent is trained on both on-policy data while
also using the off-policy data collected by all other agents at each training step.
The hyperparameter λ controls how much the experience of others is weighted,
with λ = 1 indicating that the experience of others has the same weight to the
agent and λ = 0 collapsing to no-sharing. The value function can also make
use of experience sharing. The final algorithm is identical to Independent A2C
(Algorithm 19), but uses Equation 9.87 for learning the parameters.
Why would one consider using experience sharing over parameter sharing?
Experience sharing is certainly more expensive in terms of computational power
per environment step, as it increases the size of the batch and has a number
of neural network parameters that scale with the number of agents. However,
it has been shown (Christianos, Schäfer, and Albrecht 2020) that initialising
agent-specific neural networks can lead to learning better policies: not making
the assumption that an environment contains strongly homogeneous agents and
not restricting policies to be identical can lead to higher converged returns.
17. Other terms used to refer to simulations in MCTS include “rollouts” and “playouts”. These
terms come from the original research focus of using MCTS in competitive board games. We prefer
the neutral term “simulation” since MCTS can be applied in a broad range of decision problems.
18. This version of UCB was originally called “UCB1” in the work of Auer, Cesa-Bianchi, and
Fischer (2002).
u
ExploreAction
u
ExploreAction
u
ExploreAction
u
InitialiseNode
Figure 9.25: Tree expansion and backpropagation in MCTS. Each node in the
tree corresponds to a state. The tree is expanded by sampling actions (edges)
until a previously unvisited state ŝl is reached, and a new node for the state is
initialised and added to the tree (dashed circle). The new state ŝl is evaluated
via u = f (ŝl ), and u is propagated back through the predecessor nodes in the tree
until reaching the root node (e.g. using Equation 9.89).
Once the new state ŝl has been initialised in the search tree and a value
estimate u = f (ŝl ) is obtained, MCTS backpropagates u and the rewards r̂τ
through the nodes that have been visited in the simulation, starting with the
predecessor node ŝl–1 and following its predecessors until reaching the root node.
For each node ŝτ to be updated, the function Update(Q, ŝτ , âτ ) increases the
counter N(ŝτ , âτ ) = N(ŝτ , âτ ) + 1, and the action-value function Q is updated. In
general, Q may be updated using any of the known temporal-difference learning
rules used in RL (Section 2.6). For example, we may update Q(ŝτ , âτ ) using
the off-policy Q-learning update rule with learning target r̂τ + γu if τ = l – 1,
and learning target r̂τ + γ maxa0 ∈A Q(ŝτ +1 , a0 ) if τ < l – 1. However, if the MDP
is terminating (i.e. each episode will terminate at some point) and all rewards
are zero until a terminal state is reached, such as in the zero-sum board games
considered in this section, then u can be backpropagated directly via19
1
Q(ŝτ , âτ ) ← Q(ŝτ , âτ ) + u – Q(ŝτ , âτ )
τ τ
(9.89)
N(ŝ , â )
which computes the average value of u for state-action pair ŝτ , âτ . Note that, in
this case, if ŝ is a terminal state giving reward r̂ (such as +1 for win, –1 for lose),
then the evaluation u = f (ŝ) should be equal or close to the reward r̂. Figure 9.25
depicts the process of expanding the search tree and backpropagating u through
the predecessor nodes.
After the k simulations have been completed, MCTS uses BestAction(st ) to
select an action at for the current state st , based on the information stored in the
node corresponding to st . In general, BestAction(st ) returns a probability distri-
bution π t over actions a ∈ A. A common choice is to select a most-tried action,
at ∈ arg maxa N(st , a), or an action with the highest value, at ∈ arg maxa Q(st , a).
After executing at and observing the new state st+1 , the MCTS procedure repeats
and k simulations are performed in st+1 , and so forth.
20. The self-play approach we describe in this section can also work even if the agents’ roles are
not completely symmetric. For example, the game of chess is not perfectly symmetric because
the white player makes the first move, which gives it a slight advantage; and the relative starting
positions of the king and queen differ between the two players. What matters is that we can define
ego-centric observations and agents have access to the same actions, and that there are generalisable
characteristics over the agents’ roles. For example, in chess, many play strategies can be used by
both white and black players.
for agent 2, we can transform the state vector s = (2, x, y) (in which x stores the
locations of the black pieces, and y for white pieces) by changing the agent
number and swapping the order of the x/y vectors, ψ(s) = (1, y, x). If we visualise
the state s, as shown in Figure 9.26, where white pieces are located at the top
of the board and black pieces are at the bottom, then the transformation ψ(s)
corresponds to rotating the board by 180 degrees and swapping the colours of the
pieces from black to white and vice versa. The transformed state ψ(s) can now
be used by agent 1’s policy to choose an action for agent 2, i.e. a2 ∼ π1 (· | ψ(s)).
Given such a state transformation method, we can adapt the general MCTS
algorithm shown in Algorithm 25 to implement a policy self-play approach
for zero-sum games in which the agents take turns. Essentially, from the
perspective of agent 1, in self-play the MCTS simulations become a Markov
decision process in which agent 1 chooses the action in each time step. Using
ŝτi to denote the state and agent i whose turn it is to choose an action in that
state, the self-play simulation process for agent 1 becomes:
π1 (ŝτ1 ) → âτ → π1 (ψ(ŝτ2 +1 )) → âτ +1 → π1 (ŝτ1 +2 ) → âτ +2 → π1 (ψ(ŝτ2 +3 )) → âτ +3 ...
(9.90)
To implement this process in the MCTS simulations, we modify the algorithm
to replace ŝτ with ŝτi to keep track of which agent is choosing the action in
the state, and in the function calls of ExploreAction (Line 8), InitialiseNode
(Line 13), and Update (Line 16) we input ŝτi if i = 1, and ψ(ŝτi ) if i 6= 1. Addi-
tionally, since the evaluation u = f (ŝl ) is always from the perspective of agent 1
(e.g. +1 reward if agent 1 wins), in function Update when using Equation 9.89
to update Q(ŝτi , âτ ), we have to flip the sign of u if i 6= 1. Thus, these functions
always operate on states as if it was agent 1’s turn in each state. Note that
MCTS controls agent 1, hence the time steps t in Line 2 include only the times
in which it is agent 1’s turn.
The above version of self-play MCTS uses the current version of the policy
to select actions for each agent. This algorithm can be further modified by
training the policy not just against the current version of itself, but also against
past versions of the policy. For example, we may maintain a set Π which
contains copies of agent 1’s policies (or the Q functions from which they are
derived) from different points of the training process. Using Π, the self-play
MCTS algorithm may sample any policy from this set when selecting actions
for agent 2. In this way, we may obtain a potentially more robust policy for
agent 1 by forcing it to perform well against not just one specific policy (itself)
but also against past versions of the policy. Thus, Π serves a similar purpose
to the replay buffer D (Section 8.1.3), which is to reduce overfitting. We will
explore this idea in more depth in Section 9.9.
Figure 9.27: AlphaZero match results reported by Silver et al. (2018). Results
show percentage of won/drawn/lost games for AlphaZero when playing white.
when computing the loss in stochastic gradient descent. Thus, f (s) learns value
estimates u to predict the outcome z, and action probabilities p to predict the
MCTS action probabilities π, for any given state s.
The action probability vector p is used in AlphaZero to guide the action
selection in the MCTS simulations. When expanding the search tree for new
node ŝl (Line 13), the function InitialiseNode(ŝl ) computes (u, p) = f (ŝl ; θ)
with the current parameters θ and, in addition to setting N(ŝl , â) = 0 and
Q(ŝl , â) = 0, initialises action prior probabilities P(ŝl , â) = pâ for all â ∈ A. The
function ExploreAction (Line 8) uses N, Q, P in a formula similar to the UCB
(Equation 9.88) method
â if N(ŝτ , â) = 0
τ √
â =
N(ŝτ )
arg maxâ∈A Q(ŝτ , â) + C(ŝτ )P(ŝτ , â) 1+N(ŝτ ,â) otherwise
(9.93)
where C(ŝ) in an additional exploration rate. Therefore, AlphaZero explores
actions similarly to UCB, but biases the action selection based on the predicted
probabilities p. After completing the MCTS simulations from the root node
st , AlphaZero uses a function BestAction(st ) that selects an action at either
proportionally (for exploration/training) with respect to the root visit counts
t
,·)
N(st , ·), i.e. at ∼ P N(sN(s t t
t ,a) , or greedily a ∈ arg maxa N(s , a).
a∈A
The complete specification of AlphaZero includes various additional imple-
mentation details about the network representation of f , the exploration rate C,
the addition of exploration noise in action probabilities, the sampling of data
batches to compute the loss, action masking, and so on. These details can be
found in the supplementary material of (Silver et al. 2018). However, the core
idea behind AlphaZero is to learn an effective evaluation function f purely from
self-play MCTS, and to use this function to guide the tree search in a vast search
space in order to find optimal actions.
Silver et al. (2018) reported results for AlphaZero in the games of chess,
shogi, and Go, in which it was matched against several strong game-playing
programs: “Stockfish” for chess, “Elmo” for shogi, and AlphaGo Zero (Silver
et al. 2017) for Go trained for three days. In each game, AlphaZero performed
k = 800 MCTS simulations in each state st . Separate instances of AlphaZero
were trained for 9 hours (44 million games) in chess, 12 hours (24 million
games) in shogi, and 13 days (140 million games) in Go. Based on Elo rating
(Elo 1960), AlphaZero first outperformed Stockfish after 4 hours, Elmo after
2 hours, and AlphaGo Zero after 30 hours. Figure 9.27 shows the reported
match results for the final trained instances of AlphaZero (when playing white)
in the three games. AlphaZero was able to achieve this performance using
the same general self-play MCTS method in each game, without requiring
heuristic evaluation functions based on expert knowledge.21 While Stockfish
and Elmo evaluated approximately 60 million and 25 million states per second,
respectively, AlphaZero evaluated only about 60 thousand states per second in
chess and shogi. AlphaZero compensated for the lower number of evaluations
by using its deep neural network to focus more selectively on promising actions.
21. However, AlphaZero did use domain knowledge in the construction of the input and output
features of f (e.g. legality of castling, repetition count for current position, underpromotions for
pawn moves, piece drops in shogi, etc).
? ? ? ? ?
Figure 9.28: Steps in PSRO for a two-agent game, at generation k. (a) Construct
a meta-game M k based on the current policy populations {Πki }i∈I . The cells
show the estimated average returns for each agent (we show example values
for illustration purposes) when they use the respective policies from the row
and column. (b) Use meta-solver on M k to obtain distributions δik for each
population (e.g. Nash equilibrium). The shown blue bars represent probabilities
assigned by δik to the respective policies in the population. (c) Use the oracle
to compute a new policy (e.g. best-response policy) πi0 for each agent i with
respect to the other populations Πk–i and distributions δ–ik
, and add πi0 to the
k
population Πi . PSRO then repeats these steps, where step (a) estimates values
for the missing entries for the new policies πi0 (marked by “?”)
i’s returns from these episodes. Thus, in the limit of running infinitely many
episodes, the reward will converge to the expected return of the agent, i.e.
Ri (π1 , ..., πn ) = Ui (π1 , ..., πn ). The construction of the meta-game M k completes
the population evaluation step.
Given the meta-game M k , PSRO uses a so-called “meta-solver” to compute
probability distributions δik for each population Πki , where δik (πi ) is the probabil-
ity assigned to policy πi ∈ Πki . For example, the meta-solver may compute any
of the solution concepts discussed in Chapter 4, such as Nash equilibrium.22 To
avoid situations in which δik concentrate their probability masses on a small num-
ber of policies, which could potentially lead to overfitting of policies learned by
PSRO, the original PSRO method enforces a lower bound δik (πi ) > for each
policy πi ∈ Πki , where > 0 is a parameter (Lanctot et al. 2017). Other types of
distributions may be computed based on the requirements of the game, such as
in AlphaStar discussed in Section 9.9.3.
Based on M k and the population distributions δik , PSRO uses an oracle to
compute a new policy πi0 to add to each agent’s population. The standard oracle
used in PSRO computes a best-response policy in the game G with respect to
k
(π–i ) = j6=i δjk (πj ):
Q
the distribution δ–i
πi0 ∈ arg max Eπ–i ∼δ–ik Ui (hπi , π–i i)
(9.94)
πi
22. With some small modifications, it is also possible to use a correlated equilibrium meta-solver to
compute a joint distribution δ k over the joint policy space Πk = Πk1 × ... × Πkn , where δ k (π) is the
probability assigned to join policy π ∈ Πk (Marris et al. 2021).
their respective populations Πki for each i, in which case the process has reached
a fixed point.
The basic version of PSRO presented here can be computationally expensive
and difficult to scale to many agents. Constructing a meta-game M k involves
sampling potentially many episodes in the underlying game for each joint policy
π ∈ Πk1 × ... × Πkn , and this space of joint policies grows geometrically with the
size of the populations. Solving M k via Nash equilibrium also has exponential
complexity (see Section 4.11) and the added problem of non-uniqueness (see
Sections 4.7 and 5.4.2). Finally, using RL to compute best-response policies
can also be costly depending on the complexity of the underlying game. To
address these computational bottlenecks, a number of improvements for PSRO
have been developed which make the method more computation-efficient and
scalable (e.g. Balduzzi et al. 2019; Lanctot et al. 2017; McAleer et al. 2020;
Muller et al. 2020; Smith, Anthony, and Wellman 2021). In Section 9.9.3, we
will see how the algorithm AlphaStar builds on PSRO ideas to learn strong
policies in a very complex, high-dimensional multi-agent game.
All of the above means that there is a finite set of deterministic best-response
policies πi0 ∈ Π0i for each agent i in G that the oracle can select from in each
generation k. In the worst case, PSRO will need to add all best-response poli-
cies from Π0i to the population Πki for each agent i before it terminates (recall
that PSRO terminates in generation k if the oracle’s selected policies πi0 are
already contained in the populations Πki for each agent i, respectively). There-
fore, returning to the first question above, we know that PSRO will eventually
converge. In other games it may be possible for PSRO to converge much earlier,
by reaching a point where the new best-response policies πi0 selected by the
oracles are already contained in the respective agent populations Πki . At that
point, by the definition of the Nash equilibrium (Section 4.4), we know that no
agent can unilaterally select a best-response policy πi0 ∈ Π0i that is not already
contained in Πki in order to improve its expected returns against the distribu-
k
tion δ–i . Therefore, returning to the second question above, we know that if
PSRO converges then the outcome {δik }i∈I must be a Nash equilibrium in the
underlying game G.
To illustrate these points, we give two examples in non-repeated matrix games
with two agents (the reward matrices are shown in Figure 3.2, page 46). As a
convenient shorthand, we will say “action X” to refer to the deterministic policy
that assigns probability 1 to action X.
First, consider the non-repeated Rock-Paper-Scissors game. Suppose PSRO
initialises the populations so that agent 1’s initial population Π11 contains action
R, and agent 2’s initial population Π12 contains action P. Figure 9.29 shows
the populations Πki , distributions δik , and best responses πi0 for both agents over
generations k = 1, ..., 5. It can be seen that PSRO converges to the unique Nash
equilibrium of the game, in which both agents randomise uniformly. However,
since this equilibrium assigns positive probability to all joint actions, PSRO will
need to add all deterministic policies (i.e. actions) in both populations before it
converges. At generation k = 5, in which both populations contain all actions,
any best responses π10 , π20 will already be contained in the populations Π51 , Π52 ,
respectively, and hence PSRO terminates.
Next, consider the non-repeated Prisoner’s Dilemma game. Suppose PSRO
initialises the populations Π1i of both agents to contain the defect action. Since
the populations contain only single policies, the distributions δi1 will assign
probability 1 to these policies. The oracle will then return the defect action as
the only best-response πi0 for each agent, which is already contained in both
populations. Hence, in this example, PSRO converges to the unique Nash
equilibrium of the game (in which both agents defect) without enumerating all
possible deterministic policies.
1 R P 1 1 S P
2 R,S P (0, 1) 1 S R
3 R,S R,P ( 23 , 31 ) ( 23 , 13 ) P R/P
4 R,P,S R,P (0, 23 , 31 ) ( 13 , 23 ) R S
5 R,P,S R,P,S ( 13 , 13 , 13 ) ( 13 , 13 , 13 ) R/P/S R/P/S
Note that the above assumes exact meta-games M k , in which the meta rewards
are equal to the expected returns of the agents, i.e. Ri (π1 , ..., πn ) = Ui (π1 , ..., πn )
for all i ∈ I and all joint policies (π1 , ..., πn ) from the populations Πk1 , ..., Πkn .
It also assumes exact best-responses πi0 for each agent. Both assumptions
will likely not hold in practice, since M k is constructed using a finite number
of sampled game episodes under the different joint policies,23 and since the
best-response policies πi0 are learned using a RL algorithm.
For other combinations of meta-solvers and oracles, PSRO will evolve differ-
ent policy populations and may converge to different solution types (e.g. Marris
et al. 2021; Muller et al. 2020).
23. Theoretical bounds exist on the number of sampled episodes required such that a Nash equilib-
rium in the estimated meta-game is also a (approximate) Nash equilibrium in the exact meta-game
(Tuyls et al. 2020).
three different races available in StarCraft II (Terran, Protos, Zerg) offer different
units and require different play strategies. These aspects make StarCraft II a
highly complex game, and its popularity has led to a large and active community
of professional human players that compete in international electronic sports
events. In the year 2019, AlphaStar (Vinyals et al. 2019) was the first artificial
agent to reach Grandmaster level in the full game of StarCraft II,24 placing
it above 99.8% of officially ranked human players according to the ranking
metric used in StarCraft II. AlphaStar achieved this performance by using a
combination of RL and population-based training.
At its core, AlphaStar trains a policy π(ati | hti , z; θi ) for each race parametrised
by θi , which assigns probabilities to actions ati at time t, given the history of
observations and actions hti = (o0i , a0i , o1i , a1i , ..., oti ), and a vector z which sum-
marises a strategy based on human data (detailed below). An observation oti
contains an overview map of the environment (akin to the minimap observed by
human players) and a list of visible friendly and enemy units with associated
attributes (e.g. remaining health points). An action ati specifies the action type
(e.g. move, build, attack), what unit to issue the action to, the target of the
action (e.g. where to move), and when the agent wants to select its next action.
AlphaStar receives a reward of +1 for winning, –1 for losing, and 0 for a draw
outcome; and 0 rewards at all other times. No discounting is used in the learning
objective, to reflect the true goal of winning the game.
As human players are limited by physical constraints when playing the game,
various constraints were put in place for AlphaStar in order to ensure a fair
match against humans. For example, agents were limited to executing at most
22 non-duplicate actions in a 5-second time window.
The action representation used in AlphaStar results in approximately 1026
possible actions an agent can choose from in a time step. Exploring such a
vast search space from scratch is intractable, especially since the only non-
zero rewards are obtained in terminal states after many thousands of sequential
actions. Therefore, AlphaStar uses human play data to initialise the policies.
Prior to any RL training, each policy is trained via supervised learning to imitate
human actions based on recorded matches played by human players. From
each match replay, a statistic z is extracted which encodes information about the
human player’s strategy, such as the order of constructed buildings and statistics
about units during the match. The policy is then trained to predict the human
action ati given only the history ht from the match or also z. By extracting a
24. The StarCraft Multi-Agent Challenge (SMAC) (Samvelyan et al. 2019) provides a smaller
version of StarCraft II and has been used widely in MARL research; see Section 11.3.3.
diverse set of statistics z from the recorded matches and conditioning the policy
on z, the policy is able to produce diverse strategies.
Following policy initialisation, AlphaStar uses a range of deep learning
and RL techniques based on A2C (Section 8.2.4) to train a policy against a
distribution of opponent agents (i.e. the oracle in PSRO). The policy is either
conditioned on a statistic z and the agent receives a reward for following the
strategy corresponding to z; or it is only conditioned on the history ht and free
to choose its own actions. In both cases, the agent is penalised when the action
probabilities deviate from the initial (supervised) policy. While the details of
the RL techniques are important for the training performance of AlphaStar,
for the purpose of this section we will focus on the population-based training
component used in AlphaStar. We refer to the original publication (Vinyals
et al. 2019) for details on the used deep learning architectures and RL methods.
AlphaStar uses a type of population-based training called “League training”,
which follows a similar approach to PSRO. AlphaStar maintains a single league
(population) Πk of policies corresponding to three different types of agents for
each race: main agents, main exploiter agents, and league exploiter agents. The
three types of agents differ in the distributions δik of opponents from the league
against which they are trained, at which points in training their current policies
(i.e. parameters) are added to the league, and when their policies are reset to
the initial parameters obtained from the supervised learning stage.
Let Πk be the current league at generation k, which contains past policy
copies of the three agent types from different points in the training. For a given
policy πi0 ∈ Πk , AlphaStar uses a method called prioritised fictitious self-play
(PFSP) to compute a distribution δik over policies πi in a subset of Πk against
which to train πi0 , defined as
δik (πi ) ∝ f Pr[πi0 wins against πi ]
(9.96)
where f : [0, 1] → [0, ∞) is a weighting function. AlphaStar uses two types of
weighting functions:
• fhard (x) = (1 – x)p , where p ∈ R+ is a parameter, leads PFSP to focus on the
most difficult opponent policies for πi0 . This is the default choice for f .
• fvar (x) = x(1 – x) leads PFSP to focus on opponent policies that are at a similar
level of performance as πi0 .
The probability Pr[πi0 wins against πi ] is estimated empirically by running
multiple matches of πi0 versus πi and recording the average win rate for πi0 .
Based on the above, the details of the three agent types used in the league are
as follows:
• Main agents (one per race) are trained with a proportion of: 35% in self-play,
in which the currently learning policy of the main agent is matched against
itself (as described in Section 9.8.2); 50% over the past policies of all agent
types in Πk using probabilities given by PFSP; and another 15% against past
policies of main exploiters with probabilities given by PFSP. The policies
of main agents are frozen and added to the league every 2x109 training time
steps. Main agents are never reset to the initial parameters.
• Main exploiter agents (one per race) are trained against the main agents
to exploit their weaknesses. With a proportion of 50%, or if their current
estimated probability of winning is less than 0.20, exploiter agents are trained
against past main agent policies in the league with probabilities given by
PFSP using weighting function fvar . The other 50% of training is done against
the currently learning policies of the main agents. Main exploiter agents are
added to the league when they manage to defeat all three main agents in at
least 70% of the played matches, or after 4x109 training time steps. Their
parameters are then reset to the initial parameters.
• League exploiter agents (two per race) are trained against all of the policies
contained in the league, to identify strategies that no policy in the league is
effective against. They are trained against policies in Πk using probabilities
given by PFSP, and they are added to the league if they manage to defeat all
policies in the league in at least 70% of the played matches, or after 2x109
training time steps. At that point, with a probability of 0.25 their parameters
are reset to the initial parameters.
For the matches against top-level human players reported by Vinyals et
al. (2019), AlphaStar initialised the policies via supervised learning based on a
publicly available dataset of 971,000 anonymised matches with human players
that ranked in the top 22% on the StarCraft II metric “Match Making Rating”
(MMR), which is similar to the Elo ranking used in chess (see Section 9.8.3).
After 44 days of League training using 32 third-generation tensor processing
units (TPUs) (Jouppi et al. 2017), the final trained main agents corresponding
to the three races were evaluated in matches against human players (under
an anonymous account) via the official online match-making system used
in StarCraft II. The main agents were evaluated without conditioning on the
statistics z in the policy. The final main agents achieved MMR ratings that placed
them above 99.8% of officially ranked human players, and at “Grandmaster”
level for all three races. Interestingly, the initial policies after supervised learning
already achieved a fairly strong performance, ranking them above 84% of human
players. Vinyals et al. (2019) showed that without using the human play data to
initialise policies, the performance of AlphaStar degrades very substantially as
the search space becomes too difficult to explore from scratch. It is worth noting
that, similarly to AlphaZero (Section 9.8.3), AlphaStar was able to achieve
this superior performance using general deep learning and RL methods. These
methods could similarly be applied to other multi-agent decision problems that
require complex planning and interactions over long timescales.
9.10 Summary
This chapter has presented a range of “deep” MARL algorithms that build on
deep learning methods to learn policies in complex environments. The main
concepts are summarised as follows:
• In deep MARL, different training and execution paradigms have played a
significant role in how algorithms are developed and used. The concept
of centralised information includes information that may be shared across
agents, such as parameters, gradients, observations, actions, or anything
else that is not typically included in an agent’s observation space. Three
paradigms are discussed: In centralised training and execution, agents learn
and operate by sharing centralised information throughout both training and
execution phases. In decentralised training and execution, agents are isolated,
neither sharing information nor communicating with each other, ensuring
fully independent learning based solely on their local observations. Lastly,
centralised training and decentralised execution (CTDE) represents a hybrid
and popular approach. Agents utilise centralised information to facilitate
learning during the training phase. However, agents learn policies that are
only conditioned on the local observations, thus enabling a fully independent
execution of their policies.
• Independent learning is a class of MARL algorithms in which agents indepen-
dently use single-agent RL to to learn their policies. This class of algorithms
in deep MARL can sometimes reach similar performance to more compli-
cated methods despite its relative simplicity. Independent learning algorithms
can be utilised with the decentralised training and execution paradigm.
• The first extension of independent learning we discuss are MARL algorithms
based on the multi-agent policy gradient. These algorithms typically learn a
centralised value function to consider that the returns an agent receives are
also dependent on the policies of other agents. We introduced centralised
critics V(hti , zt ; θi ) which are not only conditioned on an agent’s observa-
tion history but also centralised information zt such as the observations of
In this chapter, we will dive into the implementation of MARL algorithms. The
book comes with a codebase,1 which is built using Python and the PyTorch
framework, and it implements some of the key concepts and algorithms we have
discussed in earlier chapters. The codebase provides a practical and easy-to-use
platform for experimenting with MARL algorithms. The goal of this chapter
is to provide an overview of how these algorithms are implemented in the
codebase, and provide the reader with an understanding of coding patterns that
are repeatedly found in MARL algorithm implementations.
This chapter assumes knowledge of the Python programming language and
a basic understanding of the PyTorch framework. The code examples shown
in this chapter are meant to be educational and describe some of the ideas
that this book explored in earlier chapters, and are not necessarily to be used
as-is. For MARL, many codebases exist that efficiently implement many of the
algorithms discussed in the deep MARL chapters, including EPyMARL2 for
common-reward games, Mava,3 MARLLib,4 and more.
1 import lbforaging
2 import gym
3
4 env = gym.make("Foraging-8x8-2p-1f-v2")
The observation and action spaces can be retrieved from the environment
using the code in Code Block 10.2. In the case of level-based foraging, the
observation space is a 15-dimensional vector for each agent (noted as Box
in the Gym library when it contains float values) containing information on
the placement of agents and items, along with their levels. The action space,
which is Discrete(6) for each agent, specifies that the expected actions are
discrete numbers (0, 1, …, 5) corresponding to the four movement actions, the
collecting action to collect an item, and the noop action which does nothing
(see Section 11.3.1 for further details on this environment). Notably, both the
observation and action spaces are tuples and their n elements correspond to
each agent. In Code Block 10.2, the tuples are of size two, which means that
two agents exist in the environment.
1 env.observation_space
2 >> Tuple(Box(..., 15), Box(..., 15))
3
4 env.action_space
5 >> Tuple(Discrete(6), Discrete(6))
Code Block 10.2: Observation and action spaces. The Box action space also
includes the upper and lower limits of these values (abbreviated as dots).
The crucial step of interacting with the environment is shown in Code Block
10.3. The reset() function initialises the environment and returns the initial
observation for each agent. This can be used to select the actions a0i for each
agent i, which are then passed to the step function. The step() function
simulates the transition from a state to the next state given the joint action, and
returns the n observations (of the next state), a list of n rewards, whether the
episode has terminated, and an optional info dictionary (which we ignore in
the Code Block using Python’s _ notation).
1 observations = env.reset()
2 next_observations, rewards, terminal_signal, _ = env.step(
actions)
and type of non-linearity applied in each unit, which can be changed to best
fit the specific problem being modelled. The input to the module’s constructor
allows for different networks to be created, by changing the size of the inputs
and outputs for each of the agents. In this implementation, the networks defined
in Code Block 10.4 below are independent of each other.
1 import torch
2 from torch import nn
3 from typing import List
4
5 class MultiAgentFCNetwork(nn.Module):
6 def __init__(
7 self,
8 in_sizes: List[int],
9 out_sizes: List[int]
10 ):
11 super().__init__()
12
13 # We use the ReLU activation function:
14 activ = nn.ReLU
15 # We use two hidden layers of 64 units each:
16 hidden_dims = (64, 64)
17
18 n_agents = len(in_sizes)
19 # The number of agents is the length of the
20 # input and output vector
21 assert n_agents == len(out_sizes)
22
23 # We will create ‘n_agents’ (independent) networks
24 self.networks = nn.ModuleList()
25
26 # For each agent:
27 for in_size, out_size in zip(in_sizes, out_sizes):
28 network = [
29 nn.Linear(in_size, hidden_dims[0]),
30 activ(),
31 nn.Linear(hidden_dims[0], hidden_dims[1]),
32 activ(),
33 nn.Linear(hidden_dims[1], out_size),
34 ]
35 self.networks.append(nn.Sequential(*network))
36
37 def forward(self, inputs: List[torch.Tensor]):
38
39 # The networks can run in parallel:
40 futures = [
41 torch.jit.fork(model, inputs[i])
42 for i, model in enumerate(self.networks)
43 ]
44 results = [torch.jit.wait(fut) for fut in futures]
45 return results
In Code Block 10.4, Lines 27 to 35 iterate over the input and output sizes
of each agent and extend the list of the separate neural networks (Line 35).
While all modules are included in the same model, the actual networks are
independent and can even be run in parallel. The loop in Line 40 begins a
parallel forward pass through these networks, which makes the computation
begin in the background. The program can continue while PyTorch computes
the forward pass. In Line 44, the program waits until the computation finishes
and the result is ready (similarly to the async/await pattern found in many
programming languages).
1 class MultiAgentFCNetwork_SharedParameters(nn.Module):
2
3 def __init__(
4 self,
5 in_sizes: List[int],
6 out_sizes: List[int]
7 ):
8
9 # ... same as MultiAgentFCNetwork
10
11 # We will create one (shared) network
12 # This assumes that input and output size of the
13 # networks is identical across agents. If not, one
14 # could first pad the inputs and outputs
15
16 network = [
17 # ... same as MultiAgentFCNetwork
18 ]
19 self.network = nn.Sequential(*network)
20
21 def forward(self, inputs: List[torch.Tensor]):
22
23 # A forward pass of the same network in parallel
24 futures = [
25 torch.jit.fork(self.network, inp)
26 for inp in inputs)
27 ]
28 results = [torch.jit.wait(fut) for fut in futures]
29 return results
output layer
(Q values)
64 fully
connected
units
64 fully
connected
units
input layer
(observation)
Figure 10.1: The resulting MultiAgentFCNetwork of Code Block 10.6 for the
IDQN algorithm with two agents, an observation of size 5, and three possible
actions for each agent.
The difference between the networks created in Code Block 10.4 and 10.5 is
that in the latter there is no loop that creates many sub-networks (Code Block
10.4, Line 27), and instead, the definition of a single sequential module (Code
Block 10.5, Line 19). The parallel execution of the forward operation in Code
Block 10.5’s Line 24 calls the same network each time instead of iterating over
a list.
The resulting networks produced by the Code Block 10.6 above can be seen
in Figure 10.1. The IDQN algorithm can be built on top of these networks.
To predict (and later learn) state-action value for each agent for the IDQN
algorithm, one can use Code Block 10.7, below.
8 _, a_prime = q_values_stacked.max(-1)
Line 4 creates a categorical distribution for each agent, from the outputs of
the network. The categorical distribution can be sampled to produce the actions
that the agents will perform.
past is required (e.g. navigating a long maze), then a recurrent network might
prove useful.
Notably, reward standardisation is an empirical tip but can distort the under-
lying assumptions and goals of an algorithm. For example, consider an
environment with only negative rewards: a one-state MDP with two actions.
The first action rewards the agent with –1 and ends the episode, while the second
action rewards the agent with –0.1 but only ends the episode with a 1% chance.
Standardisation would lead to the second action being considered to provide
positive rewards instead. While the overall preference between actions would
remain the same (the first action leads to lower rewards than the second action),
the nature of the problem has changed. With negative rewards, the agent aims
for the episode to end as early as possible, but now, with potentially positive
rewards, the agent could instead prefer staying in the environment for longer.
This difference in an agent’s goals is subtle, but should be well understood
before standardisation of rewards is applied in practice.
1.0 1.00
0.75
0.8
0.50
Evaluation returns
Evaluation returns
0.6 0.25
0.00
0.4 0.25
0.50
0.2
0.75
Learning Curve (Agent 1)
Single-Agent Learning Curve Learning Curve (Agent 2)
0.0 1.00
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0
Environment time steps 1e6
Environment time steps 1e6
or when there is a gap between the abilities of the heuristic and learned agents
(e.g. a heuristic chess agent with high Elo rating will not offer informative
signals before the learned agent also approaches that rating). Another way to
monitor the performance in zero-sum games is by saving previous instances
of the trainable parameters, creating a pool of opponent agents, and ensuring
that newer agents can typically win against them (as is done in AlphaStar, see
Section 9.9.3). However, this does not necessarily make comparisons easy,
unless the pool of agents is common between different algorithms.
Finally, the information in the learning curves could potentially be condensed
into a single number, either by presenting the maximum value of the curve or
its average. These values offer different functions, the maximum can be used
to signify whether an algorithm at any point solved the environment, and the
average signifies how fast the algorithm learned in the environment as well as its
stability. The maximum can be considered more important since it can inform us
whether a desired behaviour has been achieved. However, multiple algorithms
might reach the same maximum performance. In this case the average can help
to further distinguish algorithms based on their learning speed and stability.
1 for s in {1..5}
2 for i in $(seq 0.01 0.01 0.1}
3 do
4 python algorithm.py --lr=$i --seed=$s &
5 done
6 done
1. The matrix games can be downloaded for use with this book’s codebase at: https://github.com
/uoe-agents/matrix-games
2. Game no. 23 in the original listing provided by Rapoport and Guyer (1966) has a typo: the
reward a2,1 should be 1.
21 (63)
4, 4 1, 2
2, 1 3, 3
77 (77) 78 (78)
2, 2 4, 1 2, 2 3, 1
3, 3 1, 4 4, 3 1, 4
3. Many more environments exist which are not covered in this chapter. For some additional
environments, see: https://agents.inf.ed.ac.uk/blog/multiagent-learning-environments
Figure 11.1 provides a summary of the environments along with their core
properties. Download URLs can be found in the references for each envi-
ronment, given in the respective subsections below. Many environments use
parameters to control the complexity or difficulty of the learning problem (such
as by setting the number of agents, word size, number of items, etc.). We use
the term “task” to refer to a specific parameter setting of an environment.
if: they are positioned adjacent to the item, they all select the collect action,
and the sum of the agents’ levels is equal or greater than the item’s level. Thus,
some items require cooperation between (subsets of) agents. Agents receive
a normalised reward for the collection of an item, which depends on the level
of the item as well as the agents’ contribution to the collection of the item.
Specifically, agent i receives the following reward when collecting item f
lf · li
ri = P P (11.1)
f 0 ∈F lf j∈I(f ) lj
0
where lf is the level of item f , li is the level of agent i, F is the set of all items,
and I(f ) is the set of agents involved in collecting the item f . Hence, the rewards
of agent i are normalised with respect to the level of all items which can be
collected in the environment as well as the relative level of agent i compared to
the level of all agents contributing to the collection of item f .
An LBF task is specified by the size of the grid-world, the number of agents
and items, and their skill levels. Agents and items can be randomly assigned a
level at the beginning of each episode and placed in a random location within
the grid-world. See Figure 11.2 for an illustration of two LBF tasks.
The environment is flexible with tasks requiring varying degrees of coopera-
tion, observability, and scale. Challenging exploration problems can be defined
in LBF by “enforcing cooperation”. In tasks with forced cooperation, item
levels are assigned such that all agents in the environment need to cooperate
to collect any of the items in the grid-world. See Figure 11.2b for an example
environment with forced cooperation. It is also worth noting that many LBF
Speaker
Predators
Landmarks Landmarks
Figure 11.3: Three MPE tasks. (a) Predator-prey: a set of predator agents
must catch a faster prey agent and avoid obstacles (large black circles). (b)
Coordinated navigation: three agents need to scatter to cover three landmarks in
the environment while avoiding collisions with each other. (c) Speaker-listener:
a “listener” agent is randomly assigned one of three landmarks and needs to
navigate to its assigned landmark (highlighted with the same colour). The
listener does not observe its own colour and relies on a “speaker” agent which
can see the listener’s colour and needs to learn to communicate the colour
through binary communication actions.
tasks have mixed objectives across agents, with agents competing to receive
individual reward for items they can collect by themselves and needing to
cooperate with other agents to collect items of higher levels. Such conflicting
objectives make for interesting multi-agent environments.
environment is extendable, more variations and tasks have been proposed (e.g.
Iqbal and Sha (2019)). Three common MPE tasks are shown in Figure 11.3.
To further extend MPE, Bettini et al. (2022) propose the vectorized multi-
agent simulator (VMAS). It also supports 2D multi-agent tasks with continuous
or discrete action spaces. In contrast to the original MPE environment, VMAS
can directly be simulated on GPUs to speed up training, offers more tasks, and
supports multiple interfaces for compatibility across multiple RL frameworks.
Figure 11.4: A SMAC task in which two teams fight against each other. One
team is controlled by the agents (one agent per unit) while the other team is
controlled by a built-in AI. (a) A symmetric task in which each team consists
of the same number and type of units (3 “marines” on each team). (b) An
asymmetric task in which the teams consist of different unit types.
One downside of both SMAC and SMACv2 is that they are based on StarCraft
II, which is a commercial game that still continues to receive updates. This
makes it difficult to reproduce results across different versions of the game,
and running the environment depends on the StarCraft II game which comes
with considerable computational cost. Motivated by this downside, Michalski,
Christianos, and Albrecht (2023) propose the SMAClite environment which
tries to closely mimic the SMAC tasks but does not rely on the StarCraft II
game. Thus, SMAClite is easier to setup and run, and is computationally cheap
in comparison. Michalski, Christianos, and Albrecht (2023) show that agents
trained on SMAClite tasks can be transferred to their SMAC counterparts with
some degradation in performance, indicating that SMAC is accurately but not
perfectly modelled by SMAClite.
(a) Tiny, two agents (b) Small, two agents (c) Medium, four agents
Figure 11.5: Three RWARE tasks with varying sizes and number of agents
(from Christianos, Schäfer, and Albrecht (2020)).
+1 and –1 for scoring and conceding a goal, respectively, and (2) additionally
providing positive rewards for maintaining the ball and moving forward towards
the opponent’s goal region. Similar to reward functions, there are three modes
of observations including (1) a pixel-based observation showing a rendered
screen with a view of the pitch following the ball, a small global map as well
as a scoreboard, (2) a significantly smaller compressed global map showing
information about both teams, ball location, and highlighting the location of the
active player, and (3) a feature vector of 115 values encoding player locations,
ball possession, direction, game mode, active player, and more. Motivated by
the available settings and variations across prior MARL research in GRF, Song
et al. (2023) propose a unified settings for evaluating MARL algorithms in GRF
and benchmark MARL algorithms in several variants of the GRF environment.
The different modes of rewards, observations, and progressively harder sce-
narios provided by the environment make it a suitable benchmark to evaluate
complex multi-agent interactions for cooperative games. The environment can
also be used for two-player competitive games, with each agent controlling one
team, to investigate competitive self-play algorithms.
11.3.6 Hanabi
Hanabi is a cooperative turn-based card game for two to five players in which
each player (agent) holds a set of cards with numerical rank (1 to 5) and colours
(red, green, blue, yellow, white) drawn from a set of 50 cards in total. Players
need to build up ordered stacks of cards, with each stack containing cards of the
same colour with increasing rank. The twist of Hanabi is that each player does
not see their own card but only sees the cards of all other players. Players take
actions in turns with the acting player having three possible actions: giving a
hint, playing a card from their hand, or discarding a card from their hand. When
giving a hint, the acting player selects another player and is allowed to point out
all cards of that player’s hand which match a chosen rank or colour. Each given
hint consumes an information token, with the entire team of players starting the
game with eight information tokens. Once all information tokens have been
used, players can no longer give hints. Information tokens can be recovered by
discarding a card (which also makes the active player draw a new card from
the deck) and by completing a stack of a colour by placing the fifth and final
card on it. When a card is played, it is either successfully continuing one of the
five stacks or is unsuccessful, in which case the team loses a life. The game
ends when the team loses all their three lives, all five stacks were successfully
completed with cards of all five ranks in increasing order, or when the last card
of the deck has been drawn and each player took a final turn. The team receives
a reward for each successfully placed card, for final scores between 0 and 25.
Based on the partial observability of Hanabi, in which agents lack significant
information to guide their action selection, Bard et al. (2020) proposed Hanabi as
a challenging multi-agent environment with agents needing to adopt conventions
to successfully use their limited communication to cooperate. Agents need
to establish implicit communication beyond the limited resource of hints to
succeed. These properties make Hanabi an interesting challenge for cooperative
self-play algorithms, ad hoc teamwork (Mirsky et al. 2022), and communicating
and acting under imperfect information.
11.3.7 Overcooked
Overcooked is a video game in which players control chefs in a kitchen and
need to cooperate to prepare and serve dishes to customers. The game is played
on a grid-based map with a top-down view, with each player controlling one
chef. The chefs can move around the map, pick up ingredients, and interact
with tools such as chopping boards, pans, pots, and plates to prepare and deliver
dishes following given recipes.
There exist multiple implementations which adopt this popular video game
as an environment for cooperative MARL. Rother, Weisswange, and Peters
(2023) propose the Cooking Zoo environment (based on work by Wang, Wu,
et al. (2020)). The environment supports customisation of tasks with varying
number of agents, level layouts, recipes, rewards, and observation spaces. At
each step, agents observe normalised relative positions of all agents, ingredients
and interactive tools such as pots and pans. Agents can decide to move in each
of the four cardinal directions or interact with the object in front of them, such as
placing a carried item on a counter, putting ingredients on a chopping board, or
chopping ingredients. The rewards received for each action can be customised
by setting rewards for correct or incorrect completion of recipes, running out
of time, and progress towards the completion of the current recipe. Given its
customisability, Cooking Zoo allows the generation a diverse range of tasks
which vary in their complexity. This feature makes the environment particularly
well suited to study the ability of MARL algorithms to generalise to new tasks.
Carroll et al. (2019) propose an alternative Overcooked environment with
similar observation and action spaces to Cooking Zoo. By default, only the
completion of recipes is rewarded but the environment also offers an alternative
reward function including rewards for progress towards the completion of the
recipe. The main limitation of this environment, in contrast to Cooking Zoo, is
its support for only a limited set of recipes, five different layouts of maps, and a
lack of customisation. However, it offers a publicly available dataset of human
gameplay which can be used for evaluation or training purposes.
11.4.2 OpenSpiel
OpenSpiel4 (Lanctot et al. 2019) is a collection of environments and MARL
algorithms, as well as other planning/search algorithms (such as MCTS, see
Section 9.8.1), with a focus on turn-based games, also known as “extensive-
form” games. A large diversity of classical turn-based games are provided
in OpenSpiel, which includes games such as Backgammon, Bridge, Chess,
Go, Poker, Hanabi (Section 11.3.6), and many more. The agent-environment
interface used in OpenSpiel is designed with a focus on turn-based games,
though it also supports simultaneous-move games such as the game models
used in this book. The environments in OpenSpiel use a mix of full or partial
observability, and all environments specify discrete actions, observations, and
Figure 11.8: Four Melting Pot environments. (a) Collaborative Cooking: Agents
need to cook a meal together. There exist several versions of the task which vary
in the layout of the kitchen and, thus, required cooperation and specialisation of
agents. (b) Clean-up: Seven agents can collect apples in the environment to gain
reward. The rate of apples spawning is dependent on agents cleaning the nearby
river which leads to a social dilemma with competition between agents for
apples, and the need to clean-up for long-term rewards. (c) Chemistry: Agents
can carry molecules in the environment. When two molecules are brought
close to each other, they may react to synthesise new molecules and generate
reward according to a task-specific reaction graph. (d) Territory: Agents need
to capture resources and eliminate opponent agents by shooting a beam.
Figure 11.9: Three Petting Zoo environments. (a) Pong: Two agents play against
each other in the classic Atari Pong game. (b) Multiwalker: Three agents control
bipedal robots and need to learn to walk together without dropping the package
placed on their heads. (c) Pistonball: Agents control pistons on the ground and
move the ball from the right side of the screen to the left.
states. A challenge in many of the games is that long interaction sequences are
often required by the agents before they receive any rewards.
At the time of writing this book, MARL is a highly active and fast-moving field
of research, as evidenced by the many survey articles which have been published
in this area in recent years. This book focuses on the foundations of MARL
and does not include many algorithms which are at the forefront of MARL
research. To complement the book, this chapter provides a list of survey articles
of MARL research in reverse chronological order, dating back to the first survey
of the field (to our knowledge) published in 1999. Additional survey articles
about particular application domains of MARL exist but have been omitted here.
Bowling, Michael, and Manuela Veloso. 2002. “Multiagent learning using a variable
learning rate.” Artificial Intelligence 136 (2): 215–250.
Brockman, Greg, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman,
Jie Tang, and Wojciech Zaremba. 2016. OpenAI Gym.
Brown, George W. 1951. “Iterative solution of games by fictitious play.” In Proceedings
of the Conference on Activity Analysis of Production and Allocation, Cowles Commission
Monograph 13, 374–376.
Brown, Tom B., Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan,
Prafulla Dhariwal, Arvind Neelakantan, et al. 2020. “Language models are few-shot
learners.” In Advances in Neural Information Processing Systems, 33:1877–1901.
Bruns, Bryan Randolph. 2015. “Names for games: locating 2×2 games.” Games 6 (4):
495–520.
Camerer, Colin F. 2011. Behavioral Game Theory: Experiments in Strategic Interaction.
Princeton University Press.
Campbell, Murray, A. Joseph Hoane Jr, and Feng-Hsiung Hsu. 2002. “Deep Blue.”
Artificial Intelligence 134 (1-2): 57–83.
Caragiannis, Ioannis, David Kurokawa, Hervé Moulin, Ariel D. Procaccia, Nisarg Shah,
and Junxing Wang. 2019. “The unreasonable fairness of maximum Nash welfare.” ACM
Transactions on Economics and Computation (TEAC) 7 (3): 1–32.
Carroll, Micah, Rohin Shah, Mark K. Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and
Anca Dragan. 2019. “On the utility of learning about humans for human-ai coordination.”
In Advances in Neural Information Processing Systems.
Castellini, Jacopo, Sam Devlin, Frans A. Oliehoek, and Rahul Savani. 2021. “Dif-
ference rewards policy gradients.” In Proceedings of the International Conference on
Autonomous Agents and Multiagent Systems.
Cesa-Bianchi, Nicolo, and Gábor Lugosi. 2003. “Potential-based algorithms in on-line
prediction and game theory.” Machine Learning 51:239–261.
Chakraborty, Doran, and Peter Stone. 2014. “Multiagent learning in the presence of
memory-bounded agents.” Autonomous Agents and Multi-Agent Systems 28:182–213.
Chalkiadakis, Georgios, and Craig Boutilier. 2003. “Coordination in multiagent
reinforcement learning: A Bayesian approach.” In Proceedings of the International
Conference on Autonomous Agents and Multiagent Systems, 709–716.
Chang, Yu-Han. 2007. “No regrets about no-regret.” Artificial Intelligence 171 (7):
434–439.
Chen, Xi, and Xiaotie Deng. 2006. “Settling the complexity of two-player Nash
equilibrium.” In 47th Annual IEEE Symposium on Foundations of Computer Science,
261–272. IEEE.
Cho, Kyunghyun, Bart Van Merriënboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi
Bougares, Holger Schwenk, and Yoshua Bengio. 2014. “Learning phrase representations
using RNN encoder-decoder for statistical machine translation.” In Proceedings of
Conference on Empirical Methods in Natural Language Processing, 1724–1734.
Choudhuri, Arka Rai, Pavel Hubáček, Chethan Kamath, Krzysztof Pietrzak, Alon Rosen,
and Guy N. Rothblum. 2019. “Finding a Nash equilibrium is no easier than breaking
Fiat-Shamir.” In Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of
Computing, 1103–1114.
Christianos, Filippos, Georgios Papoudakis, and Stefano V. Albrecht. 2023. “Pareto
actor-critic for equilibrium selection in multi-agent reinforcement learning.” Transactions
on Machine Learning Research.
Christianos, Filippos, Georgios Papoudakis, Arrasy Rahman, and Stefano V. Albrecht.
2021. “Scaling multi-agent reinforcement learning with selective parameter sharing.” In
Proceedings of the International Conference on Machine Learning.
Christianos, Filippos, Lukas Schäfer, and Stefano V. Albrecht. 2020. “Shared Expe-
rience Actor-Critic for Multi-Agent Reinforcement Learning.” In Advances in Neural
Information Processing Systems.
Claus, Caroline, and Craig Boutilier. 1998. “The dynamics of reinforcement learning in
cooperative multiagent systems.” In Proceedings of the 15th National Conference on
Artificial Intelligence, 746–752.
Conitzer, Vincent, and Tuomas Sandholm. 2007. “AWESOME: A general multiagent
learning algorithm that converges in self-play and learns a best response against stationary
opponents.” Machine Learning 67 (1-2): 23–43.
Conitzer, Vincent, and Tuomas Sandholm. 2008. “New complexity results about Nash
equilibria.” Games and Economic Behavior 63 (2): 621–641.
Crandall, Jacob W. 2014. “Towards minimizing disappointment in repeated games.”
Journal of Artificial Intelligence Research 49:111–142.
Crites, Robert H., and Andrew G. Barto. 1998. “Elevator Group Control Using Multiple
Reinforcement Learning Agents.” Machine Learning 33 (2-3): 235–262.
Cybenko, George. 1989. “Approximation by superpositions of a sigmoidal function.”
Mathematics of Control, Signals and Systems 2 (4): 303–314.
Dasgupta, Partha, and Eric Maskin. 1986. “The existence of equilibrium in discontinuous
economic games, I: Theory.” The Review of Economic Studies 53 (1): 1–26.
Daskalakis, Constantinos, Dylan J. Foster, and Noah Golowich. 2020. “Independent
policy gradient methods for competitive reinforcement learning.” In Advances in Neural
Information Processing Systems, 33:5527–5540.
Daskalakis, Constantinos, Paul W. Goldberg, and Christos H. Papadimitriou. 2006. “The
complexity of computing a Nash equilibrium.” In Symposium on Theory of Computing,
71–78.
Daskalakis, Constantinos, Paul W. Goldberg, and Christos H. Papadimitriou. 2009. “The
complexity of computing a Nash equilibrium.” SIAM Journal on Computing 39 (1):
195–259.
de Farias, Daniela, and Nimrod Megiddo. 2003. “How to combine expert (and novice)
advice when actions impact the environment.” In Advances in Neural Information
Processing Systems.
Farina, Gabriele, Tommaso Bianchi, and Tuomas Sandholm. 2020. “Coarse correla-
tion in extensive-form games.” In Proceedings of the AAAI Conference on Artificial
Intelligence, 34:1934–1941. 2.
Filar, Jerzy, and Koos Vrieze. 2012. Competitive Markov decision processes. Springer
Science & Business Media.
Fink, A.M. 1964. “Equilibrium in a stochastic n-person game.” Journal of Science of
the Hiroshima University 28 (1): 89–93.
Fleurbaey, Marc, and François Maniquet. 2011. A Theory of Fairness and Social Welfare.
Cambridge University Press.
Fleuret, François. 2023. The Little Book of Deep Learning.
Foerster, Jakob, Ioannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson.
2016. “Learning to communicate with deep multi-agent reinforcement learning.” In
Advances in Neural Information Processing Systems, vol. 29.
Foerster, Jakob, Richard Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel,
and Igor Mordatch. 2018. “Learning with opponent-learning awareness.” In Proceedings
of the International Conference on Autonomous Agents and Multiagent Systems.
Foerster, Jakob, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon
Whiteson. 2018. “Counterfactual multi-agent policy gradients.” In AAAI conference on
artificial intelligence, vol. 32. 1.
Foerster, Jakob, Nantas Nardelli, Gregory Farquhar, Triantafyllos Afouras, Philip H. S.
Torr, Pushmeet Kohli, and Shimon Whiteson. 2017. “Stabilising experience replay for
deep multi-agent reinforcement learning.” In Proceedings of the International Conference
on Machine Learning, 1146–1155. PMLR.
Forges, Francoise. 1986. “An approach to communication equilibria.” Econometrica:
Journal of the Econometric Society: 1375–1385.
Fortunato, Meire, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband,
Alex Graves, Volodymyr Mnih, et al. 2018. “Noisy networks for exploration.” In
Conference on Learning Representations.
Foster, Dean P., and H. Peyton Young. 2001. “On the impossibility of predicting the
behavior of rational agents.” Proceedings of the National Academy of Sciences 98 (22):
12848–12853.
Fudenberg, Drew, and David K. Levine. 1995. “Consistency and cautious fictitious play.”
Journal of Economic Dynamics and Control 19 (5-7): 1065–1089.
Fudenberg, Drew, and David K. Levine. 1998. The Theory of Learning in Games. MIT
Press.
Fukushima, Kunihiko, and Sei Miyake. 1982. “Neocognitron: A new algorithm for
pattern recognition tolerant of deformations and shifts in position.” Pattern recognition
15 (6): 455–469.
Garg, Sanjam, Omkant Pandey, and Akshayaram Srinivasan. 2016. “Revisiting the cryp-
tographic hardness of finding a Nash equilibrium.” In Annual International Cryptology
Conference, 579–604. Springer.
Gilboa, Itzhak, and Eitan Zemel. 1989. “Nash and correlated equilibria: Some
complexity considerations.” Games and Economic Behavior 1 (1): 80–93.
Glicksberg, Irving L. 1952. “A further generalization of the Kakutani fixed point theorem
with application to Nash points points.” Proceedings of the American Mathematical
Society, no. 38: 170–174.
Gmytrasiewicz, Piotr J., and Prashant Doshi. 2005. “A framework for sequential
planning in multiagent settings.” Journal of Artificial Intelligence Research 24 (1): 49–
79.
Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT
Press.
Greenwald, Amy, and Keith Hall. 2003. “Correlated Q-learning.” In Proceedings of the
International Conference on Machine Learning, 3:242–249.
Greenwald, Amy, and Amir Jafari. 2003. “A general class of no-regret learning algo-
rithms and game-theoretic equilibria.” In Learning Theory and Kernel Machines, 2–12.
Springer.
Guestrin, Carlos, Daphne Koller, and Ronald Parr. 2001. “Multiagent Planning with
Factored MDPs.” In Advances in Neural Information Processing Systems, 1523–1530.
MIT Press.
Guestrin, Carlos, Michail G. Lagoudakis, and Ronald Parr. 2002. “Coordinated Rein-
forcement Learning.” In Proceedings of the International Conference on Machine
Learning, 227–234.
Guo, Shangmin, Yi Ren, Kory Mathewson, Simon Kirby, Stefano V. Albrecht, and
Kenny Smith. 2022. “Expressivity of emergent languages is a trade-off between
contextual complexity and unpredictability.” In International Conference on Learning
Representations.
Hansen, Eric A., Daniel S. Bernstein, and Shlomo Zilberstein. 2004. “Dynamic program-
ming for partially observable stochastic games.” In Proceedings of the AAAI Conference
on Artificial Intelligence, 4:709–715.
Harsanyi, John C. 1967. “Games with Incomplete Information Played by “Bayesian”
Players. Part I. The Basic Model.” Management Science 14 (3): 159–182.
Harsanyi, John C., and Reinhard Selten. 1988. A General Theory of Equilibrium
Selection in Games. MIT Press.
Hart, Sergiu, and Andreu Mas-Colell. 2000. “A simple adaptive procedure leading to
correlated equilibrium.” Econometrica 68 (5): 1127–1150.
Hart, Sergiu, and Andreu Mas-Colell. 2001. “A general class of adaptive strategies.”
Journal of Economic Theory 98 (1): 26–54.
Hausknecht, Matthew J., and Peter Stone. 2015. “Deep recurrent Q-learning for partially
observable MDPs.” In AAAI Fall Symposium Series, 29–37. AAAI Press.
Heinrich, Johannes, Marc Lanctot, and David Silver. 2015. “Fictitious self-play in
extensive-form games.” In Proceedings of the International Conference on Machine
Learning, 805–813.
Hessel, Matteo, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski,
Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. 2018.
“Rainbow: Combining improvements in deep reinforcement learning.” In Thirty-second
AAAI conference on artificial intelligence.
Hinton, Geoffrey, Nitish Srivastava, and Kevin Swersky. 2012. Neural networks for
machine learning lecture 6A overview of mini-batch gradient descent.
Hochreiter, Sepp, and Jürgen Schmidhuber. 1997. “Long short-term memory.” Neural
Computation 9 (8): 1735–1780.
Hofbauer, Josef, and William H. Sandholm. 2002. “On the global convergence of
stochastic fictitious play.” Econometrica 70 (6): 2265–2294.
Hornik, Kurt. 1991. “Approximation capabilities of multilayer feedforward networks.”
Neural Networks 4 (2): 251–257.
Hornik, Kurt, Maxwell Stinchcombe, and Halbert White. 1989. “Multilayer feedforward
networks are universal approximators.” Neural Networks 2 (5): 359–366.
Howard, Ronald A. 1960. Dynamic programming and Markov processes. John Wiley.
Hu, Junling, and Michael P. Wellman. 2003. “Nash Q-learning for general-sum stochastic
games.” Journal of Machine Learning Research 4:1039–1069.
Hu, Shuyue, Chin-wing Leung, and Ho-fung Leung. 2019. “Modelling the dynamics of
multiagent Q-learning in repeated symmetric games: A mean field theoretic approach.”
In Advances in Neural Information Processing Systems, vol. 32.
Iqbal, Shariq, and Fei Sha. 2019. “Actor-Attention-Critic for Multi-Agent Reinforcement
Learning.” In Proceedings of the International Conference on Machine Learning. PMLR.
Jaderberg, Max, Wojciech M. Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Anto-
nio García Castañeda, Charles Beattie, et al. 2019. “Human-level performance in 3D
multiplayer games with population-based reinforcement learning.” Science 364 (6443):
859–865.
Jaderberg, Max, Valentin Dalibard, Simon Osindero, Wojciech M. Czarnecki, Jeff
Donahue, Ali Razavi, Oriol Vinyals, et al. 2017. “Population Based Training of Neural
Networks.” arXiv preprint:1711.09846.
Jafferjee, Taher, Juliusz Ziomek, Tianpei Yang, Zipeng Dai, Jianhong Wang, Matthew
E. Taylor, Kun Shao, Jun Wang, and David H. Mguni. 2023. “Taming multi-agent
reinforcement learning with estimator variance reduction.” arXiv preprint:2209.01054.
Jarrett, Kevin, Koray Kavukcuoglu, Marc’Aurelio Ranzato, and Yann LeCun. 2009.
“What is the best multi-stage architecture for object recognition?” In 2009 IEEE 12th
international conference on computer vision, 2146–2153. IEEE.
Jiang, Jiechuan, Chen Dun, Tiejun Huang, and Zongqing Lu. 2020. “Graph
Convolutional Reinforcement Learning.” In International Conference on Learning
Representations.
Jordan, James S. 1991. “Bayesian Learning in Normal Form Games.” Games and
Economic Behavior 3 (1): 60–81.
Jouppi, Norman P., Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal,
Raminder Bajwa, Sarah Bates, et al. 2017. “In-datacenter performance analysis of
a tensor processing unit.” In Proceedings of the 44th Annual International Symposium
on Computer Architecture, 1–12.
Kaelbling, Leslie Pack, Michael L. Littman, and Anthony R. Cassandra. 1998. “Planning
and acting in partially observable stochastic domains.” Artificial Intelligence 101 (1-2):
99–134.
Kalai, Ehud, and Ehud Lehrer. 1993. “Rational learning leads to Nash equilibrium.”
Econometrica 61 (5): 1019–1045.
Kianercy, Ardeshir, and Aram Galstyan. 2012. “Dynamics of Boltzmann Q learning in
two-player two-action games.” Physical Review E 85 (4): 041145.
Kilgour, D. Marc, and Niall M. Fraser. 1988. “A taxonomy of all ordinal 2×2 games.”
Theory and Decision 24 (2): 99–117.
Kim, Dong Ki, Miao Liu, Matthew D. Riemer, Chuangchuang Sun, Marwa Abdulhai,
Golnaz Habibi, Sebastian Lopez-Cot, Gerald Tesauro, and Jonathan P. How. 2021. “A
policy gradient algorithm for learning to learn in multiagent reinforcement learning.” In
Proceedings of the International Conference on Machine Learning.
Kingma, Diederik P., and Jimmy Ba. 2015. “Adam: A method for stochastic
optimization.” In International Conference on Learning Representations.
Kocsis, Levente, and Csaba Szepesvári. 2006. “Bandit based Monte-Carlo planning.” In
European Conference on Machine Learning, 282–293. Springer.
Kok, Jelle R., and Nikos Vlassis. 2005. “Using the Max-Plus Algorithm for Multiagent
Decision Making in Coordination Graphs.” In Proceedings of the Seventeenth Belgium-
Netherlands Conference on Artificial Intelligence, 359–360.
Krnjaic, Aleksandar, Raul D. Steleac, Jonathan D. Thomas, Georgios Papoudakis, Lukas
Schäfer, Andrew Wing Keung To, Kuan-Ho Lao, et al. 2023. “Scalable multi-agent
reinforcement learning for warehouse logistics with robotic and human co-workers.”
arXiv preprint:2212.11498.
Kuba, Jakub Grudzien, Muning Wen, Linghui Meng, Shangding Gu, Haifeng Zhang,
David H. Mguni, Jun Wang, and Yaodong Yang. 2021. “Settling the variance of
multi-agent policy gradients.” In Advances in Neural Information Processing Systems,
34:13458–13470.
Kurach, Karol, Anton Raichuk, Piotr Stanczyk, Michal Zajac, Olivier Bachem, Lasse
Espeholt, Carlos Riquelme, et al. 2020. “Google research football: A novel reinforce-
ment learning environment.” In AAAI Conference on Artificial Intelligence, 34:4501–
4510. 04.
Lanctot, Marc, Edward Lockhart, Jean-Baptiste Lespiau, Vinícius Flores Zambaldi,
Satyaki Upadhyay, Julien Pérolat, Sriram Srinivasan, et al. 2019. “OpenSpiel: A
Framework for Reinforcement Learning in Games.” arXiv preprint:1908.09453.
Lanctot, Marc, Vinícius Flores Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl
Tuyls, Julien Pérolat, David Silver, and Thore Graepel. 2017. “A unified game-theoretic
Liu, Siqi, Guy Lever, Josh Merel, Saran Tunyasuvunakool, Nicolas Heess, and
Thore Graepel. 2019. “Emergent coordination through competition.” In International
Conference on Learning Representations.
Lowe, Ryan, Yi I. Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. 2017.
“Multi-agent actor-critic for mixed cooperative-competitive environments.” In Advances
in Neural Information Processing Systems, vol. 30.
Lu, Christopher, Timon Willi, Christian A. Schroeder de Witt, and Jakob Foerster.
2022. “Model-free opponent shaping.” In Proceedings of the International Conference
on Machine Learning.
Lyu, Xueguang, Andrea Baisero, Yuchen Xiao, Brett Daley, and Christopher Amato.
2023. “On centralized critics in multi-agent reinforcement learning.” Journal of Artificial
Intelligence Research 77:295–354.
Marris, Luke, Ian Gemp, and Georgios Piliouras. 2023. “Equilibrium-invariant
embedding, metric space, and fundamental set of 2 × 2 normal-form games.” arXiv
preprint:2304.09978.
Marris, Luke, Paul Muller, Marc Lanctot, Karl Tuyls, and Thore Graepel. 2021.
“Multi-agent training beyond zero-sum with correlated equilibrium meta-solvers.” In
Proceedings of the International Conference on Machine Learning, 7480–7491. PMLR.
Matignon, Laëtitia, Guillaume J. Laurent, and Nadine Le Fort-Piat. 2007. “Hysteretic
Q-learning: An algorithm for decentralized reinforcement learning in cooperative multi-
agent teams.” In IEEE/RSJ International Conference on Intelligent Robots and Systems,
64–69. IEEE.
McAleer, Stephen, John B. Lanier, Roy Fox, and Pierre Baldi. 2020. “Pipeline PSRO: A
scalable approach for finding approximate nash equilibria in large games.” In Advances
in Neural Information Processing Systems, 33:20238–20248.
McMahan, H. Brendan, Geoffrey J. Gordon, and Avrim Blum. 2003. “Planning
in the presence of cost functions controlled by an adversary.” In Proceedings of the
International Conference on Machine Learning, 536–543.
Meta Fundamental AI Research Diplomacy Team, Anton Bakhtin, Noam Brown, Emily
Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, et al. 2022. “Human-level play
in the game of Diplomacy by combining language models with strategic reasoning.”
Science 378 (6624): 1067–1074.
Michalski, Adam, Filippos Christianos, and Stefano V. Albrecht. 2023. “SMAClite: A
lightweight environment for multi-agent reinforcement learning.” In Workshop on Multi-
agent Sequential Decision Making Under Uncertainty at the International Conference
on Autonomous Agents and Multiagent Systems.
Mihatsch, Oliver, and Ralph Neuneier. 2002. “Risk-sensitive reinforcement learning.”
Machine Learning 49:267–290.
Mirsky, Reuth, Ignacio Carlucho, Arrasy Rahman, Elliot Fosong, William Macke,
Mohan Sridharan, Peter Stone, and Stefano V. Albrecht. 2022. “A Survey of ad hoc
teamwork research.” In European Conference on Multi-Agent Systems.
Mnih, Volodymyr, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy
P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. “Asyn-
chronous methods for deep reinforcement learning.” In Proceedings of the International
Conference on Machine Learning, 1928–1937. PMLR.
Mnih, Volodymyr, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness,
Marc G. Bellemare, Alex Graves, et al. 2015. “Human-level control through deep
reinforcement learning.” Nature 518 (7540): 529–533.
Morad, Steven, Ryan Kortvelesy, Matteo Bettini, Stephan Liwicki, and Amanda Pro-
rok. 2023. “POPGym: Benchmarking partially observable reinforcement learning.” In
Conference on Learning Representations.
Mordatch, Igor, and Pieter Abbeel. 2018. “Emergence of grounded compositional
language in multi-agent populations.” In AAAI Conference on Artificial Intelligence,
vol. 32. 1.
Moulin, Hervé. 2004. Fair Division and Collective Welfare. MIT Press.
Moulin, Hervé, and J.-P. Vial. 1978. “Strategically zero-sum games: the class of games
whose completely mixed equilibria cannot be improved upon.” International Journal of
Game Theory 7:201–221.
Mozer, Michael C. 1991. “Induction of multiscale temporal structure.” In Advances in
Neural Information Processing Systems, vol. 4.
Mullainathan, Sendhil, and Richard H. Thaler. 2000. Behavioral Economics. National
Bureau of Economic Research, Working Paper 7948.
Muller, Paul, Shayegan Omidshafiei, Mark Rowland, Karl Tuyls, Julien Pérolat, Siqi Liu,
Daniel Hennes, et al. 2020. “A generalized training approach for multiagent learning.”
In International Conference on Learning Representations.
Nachbar, John H. 1997. “Prediction, optimization, and learning in repeated games.”
Econometrica 65 (2): 275–309.
Nachbar, John H. 2005. “Beliefs in Repeated Games.” Econometrica 73 (2): 459–480.
Nair, Vinod, and Geoffrey E. Hinton. 2010. “Rectified linear units improve restricted
boltzmann machines.” In Proceedings of the International Conference on Machine
Learning.
Nash, John F. 1950. “Equilibrium points in n-person games.” Proceedings of the
National Academy of Sciences 36 (1): 48–49.
Nesterov, Yurii E. 1983. “A method for solving the convex programming problem with
convergence rate O(1/k2 ).” In Dokl. Akad. Nauk SSSR, 269:543–547.
Nisan, Noam, Tim Roughgarden, Eva Tardos, and Vijay V. Vazirani. 2007. Algorithmic
Game Theory. Cambridge University Press.
Nyarko, Yaw. 1998. “Bayesian learning and convergence to Nash equilibria without
common priors.” Economic Theory 11 (3): 643–655.
Oliehoek, Frans A. 2010. “Value-Based Planning for Teams of Agents in Stochastic
Partially Observable Environments.” PhD diss.
Shalev-Shwartz, Shai, Shaked Shammah, and Amnon Shashua. 2016. “Safe, multi-agent,
reinforcement learning for autonomous driving.” arXiv preprint:1610.03295.
Shapley, Lloyd S. 1953. “Stochastic Games.” Proceedings of the National Academy of
Sciences of the United States of America 39 (10): 1095.
Shavandi, Ali, and Majid Khedmati. 2022. “A multi-agent deep reinforcement learn-
ing framework for algorithmic trading in financial markets.” Expert Systems with
Applications 208:118124.
Shoham, Yoav, and Kevin Leyton-Brown. 2008. Multiagent systems: Algorithmic,
game-theoretic, and logical foundations. Cambridge University Press.
Shoham, Yoav, Rob Powers, and Trond Grenager. 2007. “If multi-agent learning is the
answer, what is the question?” Artificial Intelligence 171 (7): 365–377.
Silver, David, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van
den Driessche, Julian Schrittwieser, et al. 2016. “Mastering the game of Go with deep
neural networks and tree search.” Nature 529 (7587): 484–489.
Silver, David, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai,
Arthur Guez, Marc Lanctot, et al. 2018. “A general reinforcement learning algorithm
that masters chess, shogi, and Go through self-play.” Science 362 (6419): 1140–1144.
Silver, David, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang,
Arthur Guez, Thomas Hubert, et al. 2017. “Mastering the game of Go without human
knowledge.” Nature 550 (7676): 354–359.
Singh, Satinder, Michael Kearns, and Yishay Mansour. 2000. “Nash Convergence of
Gradient Dynamics in General-Sum Games.” In Proceedings of the 16th Conference on
Uncertainty in Artificial Intelligence, 541–548.
Sion, Maurice, and Philip Wolfe. 1957. “On a game without a value.” Contributions to
the Theory of Games 3:299–306.
Smith, Max Olan, Thomas Anthony, and Michael P. Wellman. 2021. “Iterative empirical
game solving via single policy best response.” In International Conference on Learning
Representations.
Solan, Eilon, and Nicolas Vieille. 2002. “Correlated equilibrium in stochastic games.”
Games and Economic Behavior 38 (2): 362–399.
Son, Kyunghwan, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi.
2019. “QTRAN: Learning to factorize with transformation for cooperative multi-agent
reinforcement learning.” In Proceedings of the International Conference on Machine
Learning, 5887–5896.
Song, Yan, He Jiang, Haifeng Zhang, Zheng Tian, Weinan Zhang, and Jun Wang. 2023.
“Boosting studies of multi-agent reinforcement learning on Google research football
environment: The past, present, and future.” arXiv preprint:2309.12951.
Stone, Peter, Gal A. Kaminka, Sarit Kraus, and Jeffrey S. Rosenschein. 2010. “Ad hoc
autonomous agent teams: Collaboration without pre-coordination.” In Twenty-Fourth
AAAI Conference on Artificial Intelligence.
Sukhbaatar, Sainbayar, Arthur Szlam, and Rob Fergus. 2016. “Learning multiagent
communication with backpropagation.” In Advances in Neural Information Processing
Systems, 29:2244–2252.
Sunehag, Peter, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinícius
Flores Zambaldi, Max Jaderberg, Marc Lanctot, et al. 2018. “Value-decomposition
networks for cooperative multi-agent learning.” In Proceedings of the International
Conference on Autonomous Agents and Multiagent Systems, 2085–2087.
Sutton, Richard S., and Andrew G. Barto. 2018. Reinforcement learning: An introduction
(2nd edition). MIT Press.
Tan, Ming. 1993. “Multi-agent reinforcement learning: Independent vs. cooperative
agents.” In Proceedings of the International Conference on Machine Learning, 330–337.
Terry, Jordan K., Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari,
Ryan Sullivan, Luis S. Santos, et al. 2021. “PettingZoo: Gym for multi-agent rein-
forcement learning.” In Advances in Neural Information Processing Systems, 34:15032–
15043.
Tesauro, Gerald. 1994. “TD-Gammon, a self-teaching backgammon program, achieves
master-level play.” Neural Computation 6 (2): 215–219.
Thrun, Sebastian, and Anton Schwartz. 1993. “Issues in using function approxima-
tion for reinforcement learning.” In Connectionist Models Summer School, 255–263.
Psychology Press.
Tumer, Kagan, and Adrian K. Agogino. 2007. “Distributed agent-based air traffic flow
management.” In International joint conference on Autonomous agents and multiagent
systems, 1–8.
Tuyls, Karl, Julien Pérolat, Marc Lanctot, Edward Hughes, Richard Everett, Joel Z.
Leibo, Csaba Szepesvári, and Thore Graepel. 2020. “Bounds and dynamics for empirical
game theoretic analysis.” Autonomous Agents and Multi-Agent Systems 34:1–30.
van der Pol, Elise. 2016. “Deep Reinforcement Learning for Coordination in Traffic
Light Control.” PhD diss.
van Hasselt, Hado. 2010. “Double Q-learning.” In Advances in Neural Information
Processing Systems, vol. 23.
van Hasselt, Hado, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and
Joseph Modayil. 2018. “Deep reinforcement learning and the deadly triad.” arXiv
preprint:1812.02648.
van Hasselt, Hado, Arthur Guez, and David Silver. 2016. “Deep reinforcement learning
with double q-learning.” In AAAI Conference on Artificial Intelligence, vol. 30.
Vasilev, Bozhidar, Tarun Gupta, Bei Peng, and Shimon Whiteson. 2021. “Semi-
on-policy training for sample efficient multi-agent policy gradients.” In Adaptive and
Learning Agents Workshop at the International Conference on Autonomous Agents and
Multiagent Systems.
Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N.
Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. “Attention is all you need.” In
Advances in Neural Information Processing Systems, vol. 30.