Professional Documents
Culture Documents
An Introduction To Counterfactual Regret Minimization: 1 Motivation
An Introduction To Counterfactual Regret Minimization: 1 Motivation
1 Motivation
In 2000, Hart and Mas-Colell introduced the important game-theoretic algorithm of regret matching.
Players reach equilibrium play by tracking regrets for past plays, making future plays proportional to
positive regrets. The technique is not only simple and intuitive; it has sparked a revolution in computer
game play of some of the most difficult bluffing games, including clear domination of annual computer
poker competitions.
Since the algorithm is relatively recent, there are few curricular materials available to introduce
regret-based algorithms to the next generation of researchers and practitioners in this area. These
materials represent a modest first step towards making recent innovations more accessible to advanced
Computer Science undergraduates, graduate students, interested researchers, and ambitious practition-
ers.
In Section 2, we introduce the concept of player regret, describe the regret-matching algorithm,
present a rock-paper-scissors worked example in the literate programming style, and suggest related
exercises. Counterfactual Regret Minimization (CFR) is introduced in Section 3 with a worked example
solving Kuhn Poker. Supporting code is provided for a substantive CFR exercise computing optimal
play for 1-die-versus-1-die Dudo. In Section 4, we briefly mention means of “cleaning” approximately
optimal computed policies, which can in many cases improve results. Section 5 covers an advanced
application of CFR to games with repeated states (e.g. through imperfect recall abstraction) that can
reduce computational complexity of a CFR training iteration from exponential to linear. Here, we use
our independently devised game of Liar Die to demonstrate application of the algorithm. We then
suggest that the reader apply the technique to 1-die-versus-1-die Dudo with a memory of 3 claims. In
Section 6, we briefly discuss an open research problem: Among possible equilibrium strategies, how
do we compute one that optimally exploits opponent errors? The reader is invited to modify our Liar
Die example code to so as to gain insight to this interesting problem. Finally, in Section 7, we suggest
further challenge problems and paths for continued learning.
2 Regret in Games
In this section, we describe a means by which computers may, through self-simulated play, use regrets
of past game choices to inform future choices. We begin by introducing the familiar game of Rock-
Paper-Scissors (RPS), a.k.a. Roshambo. After defining foundational terms of game theory, we discuss
regret matching and present an algorithm computing strategy that minimizes expected regret. Using
this algorithm, we present a worked example for learning RPS strategy and associated exercises.
∗ [email protected], Gettysburg College, Department of Computer Science, Campus Box 402, Gettysburg, PA
17325-1486
† [email protected], Department of Knowledge Engineering, Maastricht University
1
July 9, 2013 2
2.1 Rock-Paper-Scissors
Rock-Scissors-Paper (RPS) is a two-player game where players each simultaneously make one of three
gestures: rock (a closed fist), paper (an open face-down palm), or scissors (exactly two fingers extended).
With each gesture, there is an opportunity to win, lose, or draw against the other player. Players
showing the same gesture draw. A rock wins against scissors, because “rock breaks scissors”. Scissors
wins against paper, because “scissors cuts paper”. Paper wins against rock, because “paper covers
rock”.
Players will commonly synchronize play by calling out a four-beat chant, “Rock! Paper! Scissors!
Shoot!”, bobbing an outstretched fist on the first three beats, and committing simultaneously to one
of the three gestures on the fourth beat.
A normal-form game is commonly also called a “one-shot game” since each player only makes a
single choice. One can represent such games as an n-dimensional table, where each dimension has
rows/columns corresponding to a single player’s actions, each table entry corresponds to a single action
profile (the intersection of a single action from each player), and the table entry contains a vector of
utilities (a.k.a. payoffs or rewards) for each player. The payoff table for RPS is as follows:
R P S
R 0, 0 −1, 1 1, −1
P 1, −1 0, 0 −1, 1
S −1, 1 1, −1 0, 0
where each entry has the form (u1 , u2 ). By convention, the row player is player 1 and the column player
is player 2. For example, in RPS, A = {(R, R), (R, P ), · · · , (S, P ), (S, S)}.
A normal-form game is zero-sum if the values of each utility vector sum to 0. Constant-sum games,
where the values of each utility vector sum to a constant, may be reformulated as zero-sum games by
adding a dummy player with a single dummy action that always receives the negated constant as a
payoff.
A player plays with a pure strategy if the player chooses a single action with probability 1. A
player plays with a mixed strategy if the player has at least two actions that are played with positive
probability.
We use σ to refer to a mixed strategy, and define σi (s) to be the probability of player i chooses
action s ∈ Si . By convention, −i generally refers to player i’s opponents, so in a two-player game
S−i = S3−i . To compute the expected utility of the game for an agent, sum over each action profile
July 9, 2013 3
the product of each player’s probability of playing their action in the action profile, times the player’s
utility for the action profile:
X X
ui (σi , σ−i ) = σi (s)σ−i (s0 )ui (s, s0 ),
s∈Si s0 ∈S−i
in the two-player case. A best response strategy for player i is one that, given all other player strategies,
maximizes expected utility for player i. When every player is playing with a best response strategy
to each of the other player’s strategies, the combination of strategies is called a Nash equilibrium. No
player can expect to improve play by changing strategy alone.
Consider the Battle of the Sexes game:
Gary
M G
M 2, 1 0, 0
Monica
G 0, 0 1, 2
Monica is the row player, and Gary is the column player. Suppose Monica and Gary are going out
on a date and need to choose an activity (e.g. movie, restaurant, etc.). Gary would like to go to a
football game (G) and Monica wants to see a movie (M ). They both prefer going together to the same
activity, yet each feels less rewarded for choosing the other’s preference.
Suppose Monica always chooses M . Gary is better off choosing M and has no incentive to unilat-
erally deviate from that pure strategy. Likewise, if Gary always chooses G, Monica has no incentive to
unilaterally deviate from her pure strategy. The utility is always (2, 1) or (1, 2). So, (M, M ) and (G, G)
are two pure Nash equilibria profiles. However, there is a mixed strategy Nash equilibrium as well.
An equilibrium can be reached when each player, seeing other strategies, is indifferent to the choice of
action, i.e. all are equally good.
What would have to be the case for Monica to be indifferent to the Gary’s choice? Let σGary (M ) = x
be the probability of Gary choosing the movie. Then the utility that Monica expects is 2x + 0(1 − x)
and 0x + 1(1 − x) respectively. For Monica to be indifferent between G and M , these two expected
utilities would need to be equal. Solving 2x + 0(1 − x) = 0x + 1(1 − x) for x, we get x = 31 .
With symmetric reasoning, Gary is indifferent when Monica chooses the football game with probabil-
ity σM onica (G) = 13 . Thus, Monica and Gary can use mixed strategies of ( 32 , 13 ) and ( 13 , 23 ), respectively.
This pair of mixed strategies forms a Nash equilibrium as neither player can hope to improve their
expected utilities through unilateral strategy change. Both players are indifferent to change. Note that
these Nash equilibrium strategies yield different expected utility for the players. (What is each player’s
expected utility for each of the three equilibria?)
The Nash equilibrium is one solution concept. Another more general solution concept is that of the
correlated equilibrium. Now, imagine that both players have access to some type of random signal from
a third-party. Players receive information about the signal, but not information about what the signal
indicates to other players. If players correlate play with the signals, i.e. each signal corresponds to an
action profile, i.e. an action for each player, and each player expects no utility gain from unilaterally
changing the player mapping of signals to actions, then the players have reached a correlated equilibrium.
Each Nash equilibrium is a correlated equilibrium, but the concept of correlated equilibrium is more
general, and permits important solutions. Consider again the Battle of the Sexes. As a simple signal
example, imagine a fair coin toss. Players could arrive at a cooperative behavior whereby, for instance,
a coin flip of head and tail correspond to both players choosing M and G, respectively. Having reached
this equilibrium, neither player has incentive to unilaterally change this mapping of signals to strategies,
and both players receive an average utility of 1.5.
July 9, 2013 4
• Return the average strategy profile, i.e. the strategy profile sum divided by the number of
iterations.
Over time, this process converges to a correlated equilibrium [3]. In the next section, we provide a
worked example of this algorithm applied to RPS.
Some readers may be unfamiliar with the selection operator (i.e. condition ? true expression : false
expression). It is the expression analogue of an if-else statement. First, the condition is evaluated. If
the result is true/false, the true/false expression is evaluated and the overall expression takes on this
value. The selection operator is found in languages such as C, C++, and Java, and behaves as the “if”
in functional languages such as LISP and Scheme.
Note that the normalizing sum could be non-positive. In such cases, we make the strategy uniform,
giving each action an equal probability (1.0 / NUM ACTIONS).
Once each probability of this mixed strategy is computed, we accumulate that probability to a sum
of all probabilities computed for that action across all training iterations.
The strategy is then returned. Given any such strategy, one can then select an action according to
such probabilities. Suppose we have a mixed strategy (.2, .5, .3). If one divided the number line from
0 to 1 in these proportions, the divisions would fall at .2 and .2 + .5 = .7. The generation of a random
number in the range [0, 1) would then fall proportionally into one of the three ranges [0, .2), [.2, .7), or
[.7, 1), indicating the probabilistic selection of the corresponding action index.
In general, suppose oneP has actions a0 , . . . , ai , . . . , an with probabilities p0 , . . . , pi , . . . , pn . Let cu-
i
mulative probability ci = j=0 pj . (Note that cn = 1 because all probabilities must sum to 1.) A
random number r uniformly generated in the range (0, 1] will select action i if and only if for all j < i,
r ≥ cj and r < ci .
The action is easily computed as follows. First, one generates a random floating-point number in
the range (0, 1], initializes the action index a to 0, and initializes the cumulative probability to 0. If we
were to reach the last action index (NUM ACTIONS - 1), that would necessarily be the action selected,
so as long as the action index is not our last, we add the new probability to our cumulative probability,
break out of the loop if r is found to be less than the cumulative probability, and otherwise increment
the action index.
hGet random action according to mixed-strategy distributioni≡
public int getAction(double[] strategy) {
double r = random.nextDouble();
int a = 0;
double cumulativeProbability = 0;
while (a < NUM_ACTIONS - 1) {
cumulativeProbability += strategy[a];
if (r < cumulativeProbability)
break;
a++;
}
return a;
}
With these building blocks in place, we can now construct our training algorithm:
hTraini≡
public void train(int iterations) {
double[] actionUtility = new double[NUM_ACTIONS];
for (int i = 0; i < iterations; i++) {
hGet regret-matched mixed-strategy actionsi
hCompute action utilitiesi
hAccumulate action regretsi
}
}
July 9, 2013 7
For a given number of iterations, we compute our regret-matched, mixed-strategy actions, compute
the respective action utilities, and accumulate regrets with respect to the player action chosen.
To select the actions chosen by the players, we compute the current, regret-matched strategy, and
use it to select actions for each player. Because strategies can be mixed, using the same strategy does
not imply selecting the same action.
hGet regret-matched mixed-strategy actionsi≡
double[] strategy = getStrategy();
int myAction = getAction(strategy);
int otherAction = getAction(oppStrategy);
Next, we compute the utility of each possible action from the perspective of the player playing
myAction:
hCompute action utilitiesi≡
actionUtility[otherAction] = 0;
actionUtility[otherAction == NUM_ACTIONS - 1 ? 0 : otherAction + 1] = 1;
actionUtility[otherAction == 0 ? NUM_ACTIONS - 1 : otherAction - 1] = -1;
Finally, for each action, we compute the regret, i.e. the difference between the action’s expected
utility and the utility of the action chosen, and we add it to our cumulative regrets.
hAccumulate action regretsi≡
for (int a = 0; a < NUM_ACTIONS; a++)
regretSum[a] += actionUtility[a] - actionUtility[myAction];
For each individual iteration of our training, the regrets may be temporarily skewed in such a
way that an important strategy in the mix has a negative regret sum and would never be chosen.
Regret sums and thus individual iteration strategies are highly erratic1 . What converges to a minimal
regret strategy is the average strategy across all iterations. This is computed in a manner similar to
getStrategy above, but without the need to be concerned with negative values.
hGet average mixed strategy across all training iterationsi≡
public double[] getAverageStrategy() {
double[] avgStrategy = new double[NUM_ACTIONS];
double normalizingSum = 0;
for (int a = 0; a < NUM_ACTIONS; a++)
normalizingSum += strategySum[a];
for (int a = 0; a < NUM_ACTIONS; a++)
if (normalizingSum > 0)
avgStrategy[a] = strategySum[a] / normalizingSum;
else
avgStrategy[a] = 1.0 / NUM_ACTIONS;
return avgStrategy;
}
The total computation consists of constructing a trainer object, performing training for a given
number of iterations (in this case, 1,000,000), and printing the resulting average strategy.
hMain method initializing computationi≡
public static void main(String[] args) {
RPSTrainer trainer = new RPSTrainer();
trainer.train(1000000);
System.out.println(Arrays.toString(trainer.getAverageStrategy()));
}
1 Add print statements to this code to print the regret sums each iteration.
July 9, 2013 8
Putting all of these elements together, we create a Rock Paper Scissors trainer that utilizes regret
matching in order to approximately minimize expected regret over time:
hRPSTrainer.javai≡
import java.util.Arrays;
import java.util.Random;
hDefinitionsi
hGet current mixed strategy through regret-matchingi
hGet random action according to mixed-strategy distributioni
hTraini
hGet average mixed strategy across all training iterationsi
hMain method initializing computationi
}
The average strategy that is computed by regret matching is the strategy that minimizes regret
against the opponent’s fixed strategy. In other words, it is a best response to their strategy. In this
case, the opponent used a strategy of (0.4, 0.3, 0.3). It might not be obvious, but there is always a
pure best response strategy to any mixed strategy. In this case, what pure strategy would be the best
response? Does this correspond to the output of RPSTrainer?
This being a zero-sum game of chips, the losing player loses the number of chips that the winner
gains.
cannot know which game state is the actual game state, and this uncertainty arises from having this
game being partially observable with private card knowledge, and not knowing the opponent’s strategy.
So for Kuhn Poker, there is an information set for each combination of card a player can be holding
with each possible non-terminal sequence of actions in the game. Kuhn Poker has 12 information sets.
Can you list them? How many possible game states are there in each information set?
Let rit (I, a) refer to the regret when players use σ t of not taking action a at information set I
belonging to player i. The cumulative counterfactual regret is defined as:
T
X
RiT (I, a) = rit (I, a) (4)
t=1
July 9, 2013 11
The difference between the value of always choosing action a and the expected value when the
players use σ is an action’s regret, which is then weighted by the probability that other player(s)
(including chance) will play to reach the node. If we define the nonnegative counterfactual regret
RiT,+ (I, a) = max(RiT (I, a), 0), then we apply Hart and Mas-Colell’s regret-matching from Section 2.3
to the cumulative regrets to obtain the new strategy:
T ,+
P Ri (I,a) T ,+ if a∈A(I) RiT,+ (I, a) > 0
P
T +1 R (I,a)
σi (I, a) = a∈A(I) i (5)
1 otherwise.
|A(I)|
For each information set, this equation is used to compute action probabilities in proportion to
the positive cumulative regrets. For each action, CFR then produces the next state in the game, and
computes utilities of each actions through recursively. Regrets are computed from the returned values,
and the value of playing to the current node is finally computed and returned.
The CFR algorithm with chance-sampling is presented in detail in Algorithm 1. The parameters
to CFR are the history of actions, the learning player, the time step, and the reach probabilities
for players 1 and 2, respectively. Variables beginning with v are for local computation and are not
computed according to the previous equations for counterfactual value. In line 9, σc (h, a) refers to the
probability distribution of the outcomes at the chance node h. In lines 16, 18, and 23, P (h) is the
active player after history h. In lines 10, 17, and 19, ha denotes history h with appended action a.
In line 25, π−i refers to the counterfactual reach probability of the node, which in the case of players
{1, 2} is the same as reach probability π3−i . In line 35, ∅ refers to the empty history.
The average strategy profile at information set I, σ̄ T , approaches an equilibrium as T → ∞. The
average strategy at information set I, σ̄ T (I), is obtained by normalizing sI over all actions a ∈ A(I).
What is most often misunderstood about CFR is that this average strategy profile, and not the final
strategy profile, is what converges to a Nash equilibrium [18].
Each information set is represented by an inner class Node. Each node has fields corresponding to
the regret and strategy variable definitions of RPSTrainer with an additional field infoSet containing
the string representation of the information set:
hKuhn node definitionsi≡
String infoSet;
double[] regretSum = new double[NUM_ACTIONS],
strategy = new double[NUM_ACTIONS],
strategySum = new double[NUM_ACTIONS];
2 (This is not the most efficient means of storage and retrieval of information sets, of course. The purpose here,
however, is to clarify the core algorithm rather than optimize its application.)
July 9, 2013 12
Each node also has getStrategy and getAverageStrategy method just like those of RPSTrainer.
The following function corresponds to line 28 in Algorithm 1:
hGet current information set mixed strategy through regret-matchingi≡
private double[] getStrategy(double realizationWeight) {
double normalizingSum = 0;
for (int a = 0; a < NUM_ACTIONS; a++) {
strategy[a] = regretSum[a] > 0 ? regretSum[a] : 0;
normalizingSum += strategy[a];
}
for (int a = 0; a < NUM_ACTIONS; a++) {
if (normalizingSum > 0)
strategy[a] /= normalizingSum;
else
strategy[a] = 1.0 / NUM_ACTIONS;
strategySum[a] += realizationWeight * strategy[a];
}
return strategy;
}
hGet average information set mixed strategy across all training iterationsi≡
public double[] getAverageStrategy() {
double[] avgStrategy = new double[NUM_ACTIONS];
double normalizingSum = 0;
for (int a = 0; a < NUM_ACTIONS; a++)
normalizingSum += strategySum[a];
for (int a = 0; a < NUM_ACTIONS; a++)
if (normalizingSum > 0)
avgStrategy[a] = strategySum[a] / normalizingSum;
else
avgStrategy[a] = 1.0 / NUM_ACTIONS;
return avgStrategy;
}
Finally, we define the String representation of the information set node as the String representation
of the information set followed by the current average node strategy:
hGet information set string representationi≡
public String toString() {
return String.format("%4s: %s", infoSet, Arrays.toString(getAverageStrategy()));
}
July 9, 2013 14
Putting these together, we thus define the inner Node class of our CFR training code whose objects
refer to the information sets I from Algorithm 1:
hInformation set node class definitioni≡
class Node {
hKuhn node definitionsi
hGet current information set mixed strategy through regret-matchingi
hGet average information set mixed strategy across all training iterationsi
hGet information set string representationi
}
To train an equilibrium for Kuhn Poker, we first create an integer array containing the cards. We
implicitly treat the card at index 0 and 1 as the cards dealt to players 1 and 2, respectively. So at the
beginning of each of a given number of training iterations, we simply shuffle these values, which are
implicitly “dealt” or not to the players according to their array positions.
After shuffling, we make the initial call to the recursive CFR algorithm with the shuffled cards,
an empty action history, and a probability of 1 for each player. (These probabilities are probabilities
of player actions, rather than the probability of the chance event of receiving the cards dealt.) This
function effectively implements the Solve() procedure defined from line 32 in Algorithm 1, with one
notable exception below:
hTrain Kuhn poker i≡
public void train(int iterations) {
int[] cards = {1, 2, 3};
double util = 0;
for (int i = 0; i < iterations; i++) {
hShuffle cardsi
util += cfr(cards, "", 1, 1);
}
System.out.println("Average game value: " + util / iterations);
for (Node n : nodeMap.values())
System.out.println(n);
}
Note in particular that cards are shuffled before the call to cfr. Instead of handling chance events
during the recursive calls to CFR, the chance node outcomes can be pre-sampled. Often this is easier
and more straight forward, so the shuffling of the cards replaces the if condition on lines 8 to 10.
This form of Monte Carlo style sampling is called “chance-sampling”, though it is interesting to note
that CFR can be implemented without sampling at all (“Vanilla CFR”) or with many different forms
of sampling schemes [9]. We will assume for the rest of this document that when we use CFR, we
specifically refer to chance-sampled CFR. Cards are shuffled according to the Durstenfeld version of
the Fisher-Yates shuffle3 as popularized by Donald Knuth:
hShuffle cardsi≡
for (int c1 = cards.length - 1; c1 > 0; c1--) {
int c2 = random.nextInt(c1 + 1);
int tmp = cards[c1];
cards[c1] = cards[c2];
cards[c2] = tmp;
}
3 See URL http://en.wikipedia.org/wiki/Fisher-Yates_shuffle#The_modern_algorithm.
July 9, 2013 15
The recursive CFR method begins by computing the player and opponent numbers from the history
length. As previously mentioned, the zero-based card array holds cards for player 1 and 2 at index 0
and 1, respectively, so we internally represent these players as player 0 and player 1.
We next check if the current state is a terminal state (where the game has ended), as on line 6 of
Algorithm 1, and return the appropriate utility for the current player. If it is not a terminal state,
execution continues, computing the information set string representation by concatenating the current
player card with the history of player actions, a string of “p” and “b” characters for “pass” and “bet”,
respectively. This String representation is used to retrieve the information set node, or create it if it is
nonexistant.
The node strategy is computed through regret-matching as before. For each action, cfr is recursively
called with additional history and updated probabilities (according the to node strategy), returning
utilities for each action. From the utilities, counterfactual regrets are computed and used to update
cumulative counterfactual regrets. Finally, the expected node utility is returned.
hCounterfactual regret minimization iterationi≡
private double cfr(int[] cards, String history, double p0, double p1) {
int plays = history.length();
int player = plays % 2;
int opponent = 1 - player;
hReturn payoff for terminal statesi
String infoSet = cards[player] + history;
hGet information set node or create it if nonexistanti
hFor each action, recursively call cfr with additional history and probabilityi
hFor each action, compute and accumulate counterfactual regreti
return nodeUtil;
}
In discerning a terminal state, we first check to see of both players have had at least one action.
Given that, we check for the two conditions for a terminal state: a terminal pass after the first action,
or a double bet. If there’s a terminal pass, then a double terminal pass awards a chip to the player
with the higher card. Otherwise, it’s a single pass after a bet and the player betting wins a chip. If
it’s not a terminal pass, but a two consecutive bets have occurred, the player with the higher card gets
two chips. Otherwise, the state isn’t terminal and computation continues:
hReturn payoff for terminal statesi≡
if (plays > 1) {
boolean terminalPass = history.charAt(plays - 1) == ’p’;
boolean doubleBet = history.substring(plays - 2, plays).equals("bb");
boolean isPlayerCardHigher = cards[player] > cards[opponent];
if (terminalPass)
if (history.equals("pp"))
return isPlayerCardHigher ? 1 : -1;
else
return 1;
else if (doubleBet)
return isPlayerCardHigher ? 2 : -2;
}
July 9, 2013 16
Not being in a terminate state, we retrieve the node associated with the information set, or create
such a node if nonexistant, corresponding to line 12 of Algorithm 1:
hGet information set node or create it if nonexistanti≡
Node node = nodeMap.get(infoSet);
if (node == null) {
node = new Node();
node.infoSet = infoSet;
nodeMap.put(infoSet, node);
}
Next, we compute the node strategy and prepare space for recursively-computed action utilities.
For each action, we append the symbol (“p” or “b”) for the action to the action history, and make a
recursive call with this augmented history and an update to the current player’s probability of playing
to that information set in the current training iteration. Each action probability multiplied by the
corresponding returned action utility is accumulated to the utility for playing to this node for the
current player.
hFor each action, recursively call cfr with additional history and probabilityi≡
double[] strategy = node.getStrategy(player == 0 ? p0 : p1);
double[] util = new double[NUM_ACTIONS];
double nodeUtil = 0;
for (int a = 0; a < NUM_ACTIONS; a++) {
String nextHistory = history + (a == 0 ? "p" : "b");
util[a] = player == 0
? - cfr(cards, nextHistory, p0 * strategy[a], p1)
: - cfr(cards, nextHistory, p0, p1 * strategy[a]);
nodeUtil += strategy[a] * util[a];
}
Finally, the recursive CFR call concludes with computation of regrets. However, these are not simply
accumulated. Cumulative regrets are cumulative counterfactual regrets, weighted by the probability
that the opponent plays to the current information set, as in line 25 of Algorithm 1:
hFor each action, compute and accumulate counterfactual regreti≡
for (int a = 0; a < NUM_ACTIONS; a++) {
double regret = util[a] - nodeUtil;
node.regretSum[a] += (player == 0 ? p1 : p0) * regret;
}
CFR training is initialized by creating a new trainer object and initiating training for a given number
of iterations. Bear in mind that, as in all applications of Monte Carlo, more iterations lead to closer
convergence.
hKuhnTrainer main method i≡
public static void main(String[] args) {
int iterations = 1000000;
new KuhnTrainer().train(iterations);
}
July 9, 2013 17
Putting all of these elements together, we thus create a counterfactual regret minimization (CFR)
trainer for Kuhn Poker:
hKuhnTrainer.javai≡
import java.util.Arrays;
import java.util.Random;
import java.util.TreeMap;
Food for thought: What values are printed when this program is run? What do they mean? Do
you see an opportunity to prune sub-trees for which traversal is provably wasteful? (Hint: What is/are
the important operation(s) applied at each information set and under what conditions would these be
rendered useless?)
More food (seconds?) for thought: if a subtree would never be visited by an optimal player, is there
any reason to compute play for it?
3.5.1 Rules:
Each player is seated around a table and begins with five standard six-sided dice and a dice cup. Dice
are lost with the play of each round, and the object of the game is to be the last player remaining
with dice. At the beginning of each round, all players simultaneously roll their dice once, and carefully
view their rolled dice while keeping them concealed from other players. The starting player makes a
claim about what the players have collectively rolled, and players clockwise in turn each either make
4 In some cases, e.g. Liar’s Dice and Cacho, there are different games of the same name.
5 http://www.boardgamegeek.com
6 as of August 17th , 2011
July 9, 2013 19
a stronger claim or challenge the previous claim, declaring “Dudo” (Spanish for “I doubt it.”). A
challenge ends the round, players lift their cups, and one of the two players involved in the challenge
loses dice. Lost dice are placed in full view of players.
Claims consist of a positive number of dice and a rank of those dice, e.g. two 5’s, seven 3’s, or two
1’s. In Dudo, the rank of 1 is wild, meaning that dice rolls of rank 1 are counted in totals for other
ranks as well. We will denote a claim of n dice of rank r as n × r. In general, one claim is stronger
than another claim if there is an increase in rank and/or number of dice. That is, a claim of 2 × 4 may,
for example, be followed by 2 × 6 (increase in rank) or 4 × 3 (increase in number). The exception to
this general rule concerns claims of wild rank 1. Since 1’s count for other ranks and other ranks do not
count for 1’s, 1’s as a rank occur with half frequency in counts and are thus considered doubly strong
in claims. So in the claim ordering, 1 × 1, 2 × 1, and 3 × 1 immediately precede 2 × 2, 4 × 2, and 6 × 2,
respectively.
Mathematically, one may enumerate the claims in order of strength by defining s(n, r), the strength
of claim n × r, as follows:
n
5n − b 2 c − r − 7 if r 6= 1
s(n, r) = 11n − 6 if r = 1 and r ≤ b dtotal
2 c (6)
5dtotal + n − 1 if r = 1 and r > b dtotal c
2
where dtotal is the total number of dice in play. Thus for 2 players with 1 die each, the claims would
be numbered:
Strength s(n, r) 0 1 2 3 4 5 6 7 8 9 10 11
Claim n × r 1×2 1×3 1×4 1×5 1×6 1×1 2×2 2×3 2×4 2×5 2×6 2×1
Play proceeds clockwise from the round-starting player with claims of strictly increasing strength
until one player challenges the previous claimant with “Dudo”. At this point, all cups are lifted, dice of
the claimed rank (including wilds) are counted and compared against the claim. For example, suppose
that Ann, Bob and Cal are playing Dudo, and Cal challenges Bob’s claim of 7 × 6. There are three
possible outcomes:
• The actual rank count exceeds the challenged claim. In this case, the challenger loses
a number of dice equal to the difference between the actual rank count and the claim count.
Example: Counting 6’s and 1’s, the actual count is 10. Thus, as an incorrect challenger, Cal loses
10 − 7 = 3 dice.
• The actual rank count is less than the challenged claim. In this case, the challenged
player loses a number of dice equal to the difference between the claim count and the actual rank
count. Example: Counting 6’s and 1’s, the actual count is 5. Thus, as a correctly challenged
claimant, Bob loses 7 − 5 = 2 dice.
• The actual rank count is equal to the challenged claim. In this case, every player except
the challenged player loses a single die. Example: Counting 6’s and 1’s, the actual count is indeed
7 as Bob claimed. In this special case, Ann and Cal lose 1 die each to reward Bob’s exact claim.
In the first round, an arbitrary player makes the first claim. The winner of a challenge makes the
first claim of the subsequent round. When a player loses all remaining dice, the player loses and exits
the game. The last remaining player is the winner. The following table provides a transcript of an
example 2-player game with “1:” and “2:” indicating information relevant to each player:
July 9, 2013 20
Exercise
Create a counterfactual regret minimization program DudoTrainer, similar to KuhnTrainer, that per-
forms counterfactual regret minimization for 1-die-versus-1-die endgame play of Dudo, computing and
printing (1) the average utility for player 1 across all training iterations, and (2) all information sets
and associated strategies (with actions ordered by increasing claim strength).
As a test of your code, compute the average player 1 utility across all training runs. Optimal play
7
yields a game value of − 258 ≈ −0.027.
We supply the following optional code to aid in simplifying representational choice. Let us begin
with the assumption that we are limited to two 6-sided dice, yielding 13 possible actions: 12 claims of
1 or 2 of 6 different ranks, plus the doubting “dudo” action. Let us index claims in increasing order of
strength starting at 0, and let the “dudo” action have index 12.
We begin with the definition of helpful relevant constants. Note that for any claim number, one can
look up the corresponding number and rank of the claim using integer arrays claimNum and claimRank,
respectively.
hDudo definitionsi≡
public static final int NUM_SIDES = 6, NUM_ACTIONS = (2 * NUM_SIDES) + 1,
DUDO = NUM_ACTIONS - 1;
public static final int[] claimNum = {1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2};
public static final int[] claimRank = {2, 3, 4, 5, 6, 1, 2, 3, 4, 5, 6, 1};
Next, we suggest that, for the purposes of this exercise, one simply represents the claim history as a
boolean array of length NUM ACTIONS with a value at index i being true if and only if the corresponding
claim/dudo action has been taken. One can convert this to a readable string using the following code:
hConvert Dudo claim history to a Stringi≡
public static String claimHistoryToString(boolean[] isClaimed) {
StringBuilder sb = new StringBuilder();
for (int a = 0; a < NUM_ACTIONS; a++)
if (isClaimed[a]) {
if (sb.length() > 0)
sb.append(’,’);
sb.append(claimNum[a]);
sb.append(’*’);
sb.append(claimRank[a]);
}
return sb.toString();
}
July 9, 2013 21
Further, if one also supplies the roll of the current player, one can conveniently convert the infor-
mation set to a unique integer using the following code:
hConvert Dudo information set to an integer i≡
public static int infoSetToInteger(int playerRoll, boolean[] isClaimed) {
int infoSetNum = playerRoll;
for (int a = NUM_ACTIONS - 2; a >= 0; a--)
infoSetNum = 2 * infoSetNum + (isClaimed[a] ? 1 : 0);
return infoSetNum;
}
For simplicity, we recommend retrieving nodes from a TreeMap that maps such information set
numbers to the information set nodes. The claim history strings are better used in defining the node
string representation.
Putting these all together, we offer DudoUtilities.java for your use:
hDudoUtilities.javai≡
public class DudoUtilities {
hDudo definitionsi
hConvert Dudo claim history to a Stringi
hConvert Dudo information set to an integer i
}
Figure 1: Number of abstracted information sets varying recall limit and number of dice in round.
5.1 Motivation
5.1.1 Imperfect Recall of Actions
The full 5-dice-versus-5-dice, 2-player game of Dudo consists of over 2.9 quintillion information sets.
For common modern machines, 2.9 × 1020 information sets is too large to iterate over for convergence
of mixed strategies. As with successful computational approaches to Texas Hold’em Poker, we can
abstract information sets in order to reduce the problem size. We can then solve the abstraction and
apply the abstracted policy to the original game. For Dudo, the growth rate of possible claim sequences
is most responsible for the overall growth of the extensive game-tree. We also note that the later claims
of a round are less-easily supported and thus more often contain reliable information.
Since more recent claims tend to be more important to the decision at hand, our chosen means
of abstraction is to form abstract information sets that recall up to m previous claims. For example,
consider this 5-vs.-5 round claim sequence: 1 × 5, 2 × 5, 4 × 2, 5 × 4, 3 × 1, 6 × 4, 7 × 2. For a
claim memory limit of m = 3, the 5-vs.-5 round information set for each of these decision would be
enumerated according to the current player dice roll enumeration and the enumeration of the up-to-3
most recent claims: {}, {1 × 5}, {1 × 5; 2 × 5}, {1 × 5; 2 × 5; 4 × 2}, . . . , {3 × 1; 6 × 4; 7 × 2}.
Imperfect recall of actions allows us to trade off fine distinction of judgment for computational space
and time requirements. Figure 1 shows how the number of abstract information sets varies according
to the memory limit m and the number of dice for each player. There are safe abstraction techniques
to transform a game in this way while conserving the guarantee of convergence to an equilibrium in
CFR [10], but unfortunately the abstraction above does not satisfy the required conditions. Therefore,
convergence to a Nash equilibrium may no longer be possible. However, experiments with Dudo, Poker,
and other games have shown that one can apply imperfect recall of actions with only a minor penalty
(compared to the memory savings) in convergence rate and play performance [10, 16, 15].
We conclude this section by highlighting an important and interesting consequence of abstracting
with imperfect recall of actions: For any given training example, one can have different paths to the
same abstracted information set. Thus, recursive CFR calls revisit the same information set within the
same training iteration. For example, suppose we play two-player Dudo with a claim memory limit of
m = 3. The following two claim sequences
s1 : 1 × 2, 1 × 3, 1 × 5, 1 × 6, 2 × 2
s2 : 1 × 5, 1 × 6, 2 × 2
will be part of the same information set since the 1 × 2, 1 × 3 is forgotten. Furthermore, assuming the
same roll outcomes in both cases, the expected utilities from this point on given a strategy profile used
by both players will be the the same in both cases, since the only thing that matters for determining
the payoff is the most recent claim of 2 × 1. CFR would enumerate both sequences. FSICFR avoids the
redundant computation by restructuring the tree traversal and computation of counterfactual regrets.
July 9, 2013 23
5.2 Algorithm
With such games, it is indeed possible to perform counterfactual regret minimization with a single
forward and backward visit to each node in a topologically-sorted directed acyclic graph. Recursive
CFR is restructured as a dynamic programming algorithm where cumulative counterfactual regrets
and thus strategies are updated once per training iteration. Since the node strategies remain fixed
throughout the forward propagation of player reach/realization probability information, we refer to
the algorithm as Fixed-Strategy Iteration Counterfactual Regret Minimization (FSICFR). Like CFR,
it relies on the regret-matching equation (5) for convergence to a Nash equilibrium.
Essentially, CFR traverses extensive game subtrees, recursing forward with reach probabilities that
each player will play to each node (i.e. information set) while maintaining history, and backpropagating
utilities used to update parent node action regrets and thus future strategy.
Fixed-Strategy Iteration CFR (FSICFR) [15] divides the recursive CFR algorithm into two iterative
passes, one forward and one backward, through a DAG of nodes. On the forward pass, visit counts and
reach probabilities of each player are accumulated, yet all strategies remain fixed. (By contrast, in CFR,
the strategy at a node is updated with each CFR visit.) After all visits are counted and probabilities
are accumulated, a backward pass computes utilities and updates regrets. FSICFR computational
time complexity is proportional to the number of nodes times the average node outdegree, whereas
CFR complexity is proportional to the number of node visits times the average node outdegree. Since
the number of node visits grows exponentially in such abstracted problems, FSICFR has exponential
savings in computational time per training iteration relative to CFR.
We now present the FSICFR algorithm for two-player, zero-sum games in detail as Algorithm 2.
Each player node n consists of a number of fields:
visits - the sum of possible paths to this node during a single training iteration.
pSum i - the sum of the probabilities that player i would play to this node along each possible path
during a single training iteration.
r - a mapping from each possible action to the average counterfactual regret for not choosing that
action.
σ - a node strategy, i.e. a probability distribution function mapping each possible action to the prob-
ability of choosing that action.
σSum - the sum of the strategies used over each training iteration the node is visited.
Algorithm 2 FSICFR(L)
Require: A topologically sorted list L of extensive game DAG nodes.
Ensure: The normalized strategy sum n.σSum, i.e. the average strategy for each player node n, ap-
proximates a Nash equilibrium.
1: for each training iteration do
2: Predetermine the chance outcomes at all reachable chance nodes in L.
3: for each node n in order of topologically-sorted, reachable subset L0 ⊂ L do
4: if n is the initial node then
5: n.visits ← n.pSum 1 ← n.pSum 2 ← 1
6: end if
7: if n is a player node then
8: Compute strategy n.σ according to Equation 5.
9: if n.r has no positive components then
10: Let n.σ be the uniform distribution.
11: end if
12: n.σSum ← n.σSum + (n.σ · n.pSum 1 if n.player = 1 else n.σ · n.pSum 2 )
13: for each action a do
14: Let c be the associated child of taking action a in n with probability n.σ(a).
15: c.visits ← c.visits + n.visits
16: c.pSum 1 ← c.pSum 1 + (n.σ(a) · n.pSum 1 if n.player = 1 else n.pSum 1 )
17: c.pSum 2 ← c.pSum 2 + (n.pSum 2 if n.player = 1 else n.σ(a) · n.pSum 2 )
18: end for
19: else if n is a chance node then
20: Let c be the associated child of the predetermined chance outcome for n.
21: c.visits ← c.visits + n.visits
22: c.pSum 1 ← c.pSum 1 + n.pSum 1
23: c.pSum 2 ← c.pSum 2 + n.pSum 2
24: end if
25: end for
26: for each node n in reverse order of topologically-sorted, reachable subset L0 ⊂ L do
27: if n is a player node then
28: n.v ← 0
29: for each action a do
30: Let c be the associated child of taking action a in n with probability n.σ(a).
31: n.v(a) ← (c.v if n.player = c.player else −c.v)
32: n.v ← n.v + n.σ(a) · n.v(a)
33: end for
34: Counterfactual probability cfp ← (n.pSum 2 if n.player = 1 else n.pSum 1 )
35: for each action a do
1
36: n.r(a) ← n.T +n.visits (n.T · n.r(a) + n.visits · cfp · (n.v(a) − n.v))
37: end for
38: n.T ← n.T + n.visits
39: else if n is a chance node then
40: Let [n.player , n.v] ← [c.player , c.v], where c is the predetermined child of n.
41: else if n is a terminal node then
42: n.v ← the utility of n for current player n.player .
43: end if
44: n.visits ← n.pSum 1 ← n.pSum 2 ← 0
45: end for
46: end for
July 9, 2013 26
A significant, domain-specific assumption is required for this simple, zero-sum form of FSICFR:
There is a one-to-one correspondence between player nodes and abstracted information sets, and our
visits to nodes must contain enough state information (e.g. predetermined public and private informa-
tion for both players) such that the appropriate successor nodes may be chosen. In the case of Dudo,
this means that the algorithm, having predetermined player rolls, knows which player information sets
are legal successors.
We note that the predetermination of chance node outcomes may present difficulties for some games
or game abstractions where constraints on a chance node are dependent on the path by which it is
reached (e.g. drawing the next card without knowing how many and thus which cards have been
previously drawn). This does not pose a problem for Dudo or non-draw forms of Poker. Observe that
such predetermination should proceed in topological order, as predetermination of chance nodes affects
reachability of later chance nodes.
Also, we note that if, as in the case of Dudo, the number of visits to a node will be constant across
all training iterations, then one can eliminate the n.visits variable and replace it with the value 1 in
equations.
In summary, consider FSICFR as being similar to the case where we hold the strategy before a CFR
training iteration fixed and execute an entire iteration of regret updates without changing strategy
until after the iteration. In such a case, all operations at a node are the same, and we transform the
exponential tree-recursion of CFR into a linear dynamic-programming graph-traversal for FSICFR. As
a consequence, node update frequency is equalized rather than exponentially proportional to depth.
8 Dice may be easily obtained with a variety of numbers of sides. Role-playing games commonly use a set of dice with
Each Liar Die information set node contains a number of familiar fields, but must also contain
additional information to allow the dynamic programming computation:
hLiar Die node definitionsi≡
public double[] regretSum, strategy, strategySum;
public double u, pPlayer, pOpponent;
In addition to the familiar regretSum, strategy, and strategySum fields, we now have 3 additional
fields. The field u allows each node to hold its utility value in order to allow backpropagation of the
utility to all predecessor nodes in the dynamic programming algorithm. In forward propagation of
reach probabilities (i.e. realization weights), these are accumulated in the pPlayer, and pOpponent
fields. These are not probabilities. Rather, they are a sum of probabilities. (They represent average
probabilities if we divide them by the number of node visits.)
The node constructor takes the number of possible actions as a parameter and allocates field arrays
accordingly:
hLiar Die node constructor i≡
public Node(int numActions) {
regretSum = new double[numActions];
strategy = new double[numActions];
strategySum = new double[numActions];
}
The main task of the LiarDieTrainer constructor is to allocate these nodes. After storing the
number of die sides in the trainer field, the response and claim node arrays are allocated and the nodes
are constructed.
hConstruct trainer and allocate player decision nodesi≡
public LiarDieTrainer(int sides) {
this.sides = sides;
responseNodes = new Node[sides][sides + 1];
for (int myClaim = 0; myClaim <= sides; myClaim++)
for (int oppClaim = myClaim + 1; oppClaim <= sides; oppClaim++)
responseNodes[myClaim][oppClaim]
= new Node((oppClaim == 0 || oppClaim == sides) ? 1 : 2);
claimNodes = new Node[sides][sides + 1];
for (int oppClaim = 0; oppClaim < sides; oppClaim++)
for (int roll = 1; roll <= sides; roll++)
claimNodes[oppClaim][roll] = new Node(sides - oppClaim);
}
July 9, 2013 29
When allocating response nodes, note that there is only one possible action when there is no prior
claim (i.e. ACCEPT) and only one possible action when the opponent claims the highest rank equal to
the number of sides (i.e. DOUBT). When allocating claim nodes, the number of legal claims remaining
are the number of sides minus the previous opponent claim.
The training procedure itself will be presented top-down. A regret vector is allocated with a capacity
equal to the largest number of actions for any node. In FSICFR, we topologically sort the information
set nodes that can be visited in a training iteration, so we precompute our chance events. In the context
of Liar Die, we precompute each roll that is made after the acceptance of each possible non-terminal
claim number, so an array is allocated to store these rolls. Then, for the given number of iterations,
we (1) initialize these rolls and starting probabilities, (2) accumulate realization weights forward, and
(3) backpropagate resulting utilities, adjusting regrets (and thus strategies) once per node. For better
results, we add an optional step that, when training is half done, resets all node strategy sums.
hTrain with FSICFRi≡
public void train(int iterations) {
double[] regret = new double[sides];
int[] rollAfterAcceptingClaim = new int[sides];
for (int iter = 0; iter < iterations; iter++) {
hInitialize rolls and starting probabilitiesi
hAccumulate realization weights forward i
hBackpropagate utilities, adjusting regrets and strategiesi
hReset strategy sums after half of trainingi
}
hPrint resulting strategyi
}
First, we precompute the possible rolls, and set the initial reach probabilities (i.e. realization weights)
to 1:
hInitialize rolls and starting probabilitiesi≡
for (int i = 0; i < rollAfterAcceptingClaim.length; i++)
rollAfterAcceptingClaim[i] = random.nextInt(sides) + 1;
claimNodes[0][rollAfterAcceptingClaim[0]].pPlayer = 1;
claimNodes[0][rollAfterAcceptingClaim[0]].pOpponent = 1;
Next, we need to visit each possible node ordered according to the forward order of possible visits.
Our array structure allows us to do this easily, iterating through opponent claims, first visiting response
nodes and then visiting claim nodes:
hAccumulate realization weights forward i≡
for (int oppClaim = 0; oppClaim <= sides; oppClaim++) {
hVisit response nodes forward i
hVisit claim nodes forward i
}
July 9, 2013 30
Visiting response nodes forward, we iterator through the previous possible claims the player could
make prior to the opponent claim. The corresponding node is accessed and the strategy for that node
is computed with counterfactual regret matching. If the opponent claim allows the player to increase
the claim, i.e. the opponent claim is less than the number of sides, we use our precomputed roll to look
up the claim node that would come next after an ACCEPT action. To this node, we accumulate our
reach/realization probabiltiies. Note that, for FSICFR, we will have visited all possible predecessor
nodes and accumulated all such probabilities before visiting a node. It is as if we are performing the
recursive CFR algorithm, but holding all strategies fixed and later performing a single final regret
computation and update per node.
hVisit response nodes forward i≡
if (oppClaim > 0)
for (int myClaim = 0; myClaim < oppClaim; myClaim++) {
Node node = responseNodes[myClaim][oppClaim];
double[] actionProb = node.getStrategy();
if (oppClaim < sides) {
Node nextNode = claimNodes[oppClaim][rollAfterAcceptingClaim[oppClaim]];
nextNode.pPlayer += actionProb[1] * node.pPlayer;
nextNode.pOpponent += node.pOpponent;
}
}
Similarly, we next visit each possible claim nodes to which we have just propagated probabilities. For
each of these, for each legal claim, we similarly accumulate probabilities forward to successor response
nodes. In this way, we ensure proper visiting order. In general, FSICFR requires that forward visiting
occur in topologically sorted order according to the subgraph formed by precomputed chance events.
hVisit claim nodes forward i≡
if (oppClaim < sides) {
Node node = claimNodes[oppClaim][rollAfterAcceptingClaim[oppClaim]];
double[] actionProb = node.getStrategy();
for (int myClaim = oppClaim + 1; myClaim <= sides; myClaim++) {
double nextClaimProb = actionProb[myClaim - oppClaim - 1];
if (nextClaimProb > 0) {
Node nextNode = responseNodes[oppClaim][myClaim];
nextNode.pPlayer += node.pOpponent;
nextNode.pOpponent += nextClaimProb * node.pPlayer;
}
}
}
Having completed this computation we know the relative frequencies for reaching each reachable
node. We next visit these nodes a second time in reverse order, propagating the resulting utilities
backwards from game end to game beginning, computing and accumulating counterfactual regrets.
Note that, whereas we visited response nodes before claim nodes in forward propagation, here we
reverse the order in backward propagation:
hBackpropagate utilities, adjusting regrets and strategiesi≡
for (int oppClaim = sides; oppClaim >= 0; oppClaim--) {
hVisit claim nodes backward i
hVisit response nodes backward i
}
July 9, 2013 31
After retrieve each node and its strategy, we reset the node utility from its prior computation, and
iterate through each legal claim action. For each legal claim action greater than the opponent’s claim,
we retrieve the corresponding node and, given that this is a zero-sum game, the node player’s utility
is the negation of the next node’s utility for the opponent playing it. We initialize the regret values to
each of these negated successor node utilities, and add the utility to the node utility, weighted by the
probability of choosing that action.
Once we have completed computation of the node utility, we can then subtract these values from
the individual action utilities to compute action regrets. Multiplying these by the opponent reach
probability sums, we thus accumulate counterfactual regret for the node.
Finally, we reset these reach probabilities.
hVisit claim nodes backward i≡
if (oppClaim < sides) {
Node node = claimNodes[oppClaim][rollAfterAcceptingClaim[oppClaim]];
double[] actionProb = node.strategy;
node.u = 0.0;
for (int myClaim = oppClaim + 1; myClaim <= sides; myClaim++) {
int actionIndex = myClaim - oppClaim - 1;
Node nextNode = responseNodes[oppClaim][myClaim];
double childUtil = - nextNode.u;
regret[actionIndex] = childUtil;
node.u += actionProb[actionIndex] * childUtil;
}
for (int a = 0; a < actionProb.length; a++) {
regret[a] -= node.u;
node.regretSum[a] += node.pOpponent * regret[a];
}
node.pPlayer = node.pOpponent = 0;
}
July 9, 2013 32
The computation for visiting response nodes is very similar, except that these are the only nodes
leading to terminal game situations. As such, regrets for the DOUBT action compute utility as 1 or -1
depending on whether or not the opponent’s claim was unsupported or supported, respectively.
hVisit response nodes backward i≡
if (oppClaim > 0)
for (int myClaim = 0; myClaim < oppClaim; myClaim++) {
Node node = responseNodes[myClaim][oppClaim];
double[] actionProb = node.strategy;
node.u = 0.0;
double doubtUtil = (oppClaim > rollAfterAcceptingClaim[myClaim]) ? 1 : -1;
regret[DOUBT] = doubtUtil;
node.u += actionProb[DOUBT] * doubtUtil;
if (oppClaim < sides) {
Node nextNode = claimNodes[oppClaim][rollAfterAcceptingClaim[oppClaim]];
regret[ACCEPT] = nextNode.u;
node.u += actionProb[ACCEPT] * nextNode.u;
}
for (int a = 0; a < actionProb.length; a++) {
regret[a] -= node.u;
node.regretSum[a] += node.pOpponent * regret[a];
}
node.pPlayer = node.pOpponent = 0;
}
At the end of each training iteration, we check if training is halfway complete. If so, we iterate
through all nodes, resetting all strategy sums to 0. As mentioned in the previous section, this is one
means of eliminating the strategy errors accumulated in the early iterations of training.
hReset strategy sums after half of trainingi≡
if (iter == iterations / 2) {
for (Node[] nodes : responseNodes)
for (Node node : nodes)
if (node != null)
for (int a = 0; a < node.strategySum.length; a++)
node.strategySum[a] = 0;
for (Node[] nodes : claimNodes)
for (Node node : nodes)
if (node != null)
for (int a = 0; a < node.strategySum.length; a++)
node.strategySum[a] = 0;
}
July 9, 2013 33
After training is complete, we print a summary of initial claim actions given initial rolls. This is
followed by a complete output of the entire computed strategy for each node.
hPrint resulting strategyi≡
for (int initialRoll = 1; initialRoll <= sides; initialRoll++) {
System.out.printf("Initial claim policy with roll %d: ", initialRoll);
for (double prob : claimNodes[0][initialRoll].getAverageStrategy())
System.out.printf("%.2f ", prob);
System.out.println();
}
System.out.println("\nOld Claim\tNew Claim\tAction Probabilities");
for (int myClaim = 0; myClaim <= sides; myClaim++)
for (int oppClaim = myClaim + 1; oppClaim <= sides; oppClaim++)
System.out.printf("\t%d\t%d\t%s\n", myClaim, oppClaim,
Arrays.toString(responseNodes[myClaim][oppClaim].getAverageStrategy()));
System.out.println("\nOld Claim\tRoll\tAction Probabilities");
for (int oppClaim = 0; oppClaim < sides; oppClaim++)
for (int roll = 1; roll <= sides; roll++)
System.out.printf("%d\t%d\t%s\n", oppClaim, roll,
Arrays.toString(claimNodes[oppClaim][roll].getAverageStrategy()));
The trainer main method specifies the number of die sides, creates the trainer, and executes training
for a given number of iterations.
hLiarDieTrainer main method i≡
public static void main(String[] args) {
LiarDieTrainer trainer = new LiarDieTrainer(6);
trainer.train(1000000);
}
Combining these definitions, we form the complete LiarDieTrainer:
hLiarDieTrainer.javai≡
import java.util.Arrays;
import java.util.Random;
6 Exploiting Mistakes
In such computation of optimal play for Liar Die, we may sometimes observe odd play behavior. For
example, suppose you claim 4 against the computer’s strategy, and the computer accepts. The computer
may reply with a claim of 6, forcing you to doubt and reveal a computer roll of 4 or less. Why does
this appear to be an error in play?
We reason that the computer should have claimed 5 with a roll of 4 or less because, had we accepted
the claim, the computer would be better off. By claiming 6, the computer denies us the ability to trust
a 5 claim in error. Is this rational?
Yes, it is rational: The computer has determined that the rational player will doubt any claim that
follows a 4 claim. A claim of 5 or 6 after a 4 makes no difference to a rational player, because a rational
response is to always doubt. However, this does not maximally take account of and take advantage of
potential player mistakes. This point is related to a joke about two economists walking down a street.
One economist notices a twenty-dollar bill on the sidewalk and leans to pick it up. The other economist
lays a hand the other’s shoulder and explains “It can’t be a twenty-dollar bill. Otherwise, someone
would have picked it up by now.” In a rational world, no one would leave $20 on the ground, so why
consider the possibility? A rational player in the Nash-equilibrium sense would not stoop to pick up
$20, because it’s inconceivable that a rational opponent would leave it there. Such a player would also
not consider playing in such a way as to hope for a future dropped $20.
There are a variety of solution concepts related to this point, including perturbed equilibria, sequen-
tial equilibria, and (quasi-)perfect equilibria. For these exercises, we will not go into depth theoretically,
but rather encourage the learner to experiment empirically with means of consideration of possible er-
rors.
The problem is that the number of terms grows exponentially with the number of actions remaining
from h to a leaf. To see this more clearly, imagine a fixed-depth horizon version of the recursive
definition above which falls back back to the old utilities after some depth d. Then, assuming a fixed
branching factor of |A|: when d = 1, there would be 2|A| child terms. When d = 2, there would be
4|A|2 terms. At depth d, there would be (2|A|)d terms. This places a significant computational burden
July 9, 2013 35
for algebraic representation since it accounts for every possible perturbation in the future. Not only are
utilities potentially polynomials of degree d, but we need to store more regrets for these newly defined
utilities as well, which in practice is prohibitive. One possible means of addressing this problem (which
to the author’s knowledge has not been tried, i.e. would be interesting research) would be to truncate
such representations to a small number of terms as a form of approximate analysis. For example, we
could ignore terms for d > 3.
A much simpler approach that requires very little in the way of modification to the algorithms above
is to not treat the errors algebraically, but rather introduce them with some probability. For example,
one could have an epsilon variable set to a small value (e.g. 0.001), that is directly factored into the
computation of strategies. Similarly to the algorithms of Johanson et. al. in [6, 5], each learning agent
can train against a restricted opponent that is forced to play a random move once every thousand moves
but otherwise learns in the usual way.
Miltersen & Sørensen applied the -error modeling approach by constructing a perturbed sequence-
form representation of the game [13]. In their work, they show that optimal solutions can be obtained
for arbitrarily small > 0 in polynomial time, and hence result in a quasi-perfect equilibrium (which
are sequentially rational). In CFR, one can consider what happens in the limit as → 0; one might
gradually reduce the value of the epsilon variable over time, e.g. by multiplying it by a decay factor (e.g.
.9999) for each training iteration. In this way, one would build an initial regret bias towards behaviors
exploiting error, but an open question is how much such bias would remain through continued chance
sampling with low/zero epsilon. In other words, how could such an epsilon variable be best put to
use in order to create policies that closely approximate a Nash equilibrium while maximally exploiting
past and potential future player mistakes?
7 Further Challenges
Of course, these techniques are applicable to the many hundreds of poker variations that exist. Rules to
over 200 Poker variations may be found in Ernest, Foglio, and Selinker’s Dealer’s Choice: the complete
handbook of Saturday night poker [1].
However, it is often advisable to start with simpler games and work towards greater complexity.
Many simpler bluffing dice games offer good opportunities for increasing your skill in applying these
techniques. Reiner Knizia’s Dice Games Properly Explained [7] has an excellent chapter surveying a
variety of bluffing dice games, many of which have not yet had equilibrium play computed at this time.
and bubblegum.
In retrospect, these methods of choice were overly complex, unfair, and not particularly fun. What
he has passed on to his own children is the Minimum Unique Fingers game, which is his practical
specialization of the lowest unique positive integer game [17].
Players synchronize action by chant the four-beat “One, two, three, go!”. On calling “go!”, each
player simultaneously extends one hand, and with that hand extends some number of fingers (up to 5
and including the possibility of extending 0 fingers, i.e. a fist). The player with the lowest number of
fingers extended, whose number is not duplicated by any other player, wins. In other words, the player
showing the minimum unique number of fingers wins. If there is no player with a unique number of
fingers, there is no winner.
For example, if Ann, Bob, and Cal extend 0, 0, and 5 fingers, respectively, Cal wins. If they extend
0, 1, and 2 fingers, respectively, Ann wins. If they extend 1, 1, and 1 fingers, respectively, no one wins.
Let p be the number of players. To formulate this as a zero-sum game, all payoffs/utilities are 0
when no one wins, and when one wins, the winner gets a payoff of p − 1 while the losers get a payoff
of −1.
As it turns out, regret matching is more generally capable of computing correlated equilibria for
non-zero-sum games and games with more than two players. Study the correlated equilibria solution
concept, starting with [3].
Create a regret minimization program MUFTrainer that is parameterized for the number of players
p and a number of training iterations. Compute and print a correlated equilibrium strategy for 5
children. For this to work, the way the average strategy is computed has to change. Instead, the
algorithm must compute the average joint strategy by normalizing the frequency of observed tuples in
the joint action space (0, 1, 2, 3, 4, 5)5 . Is the correlated equilibrium computed by your program also a
Nash equilibrium? How does one show this?
8 Acknowledgements
This work is partially funded by the Netherlands Organisation for Scientific Research (NWO) in the
framework of the project Go4Nature, grant number 612.000.938.
July 9, 2013 37
References
[1] James Ernest, Phil Foglio, and Mike Selinker. Dealer’s Choice: the complete handbook of Saturday
night poker. Overlook Duckworth, Peter Mayer Publishers, Inc., New York, 2005.
[2] Sam Ganzfried, Tuomas Sandholm, and Kevin Waugh. Strategy purification and thresholding:
Effective non-equilibrium approaches for playing large games. In Proceedings of the International
Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2012.
[3] Sergiu Hart and Andreu Mas-Colell. A simple adaptive procedure leading to correlated equilibrium.
Econometrica, 68(5):1127–1150, September 2000.
[4] Gil Jacobs. The World’s Best Dice Games, new edition. John N. Hansen Co., Inc., Milbrae,
California, 1993.
[5] Michael Johanson and Michael Bowling. Data biased robust counter strategies. In Proceedings of
the Twelfth International Conference on Artificial Intelligence and Statistics (AISTATS), pages
264–271, 2009.
[6] Michael Johanson, Martin Zinkevich, and Michael Bowling. Computing robust counter-strategies.
In J.C. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Pro-
cessing Systems 20 (NIPS), pages 721–728, Cambridge, MA, 2008. MIT Press.
[7] Reiner Knizia. Dice Games Properly Explained. Elliot Right-Way Books, Brighton Road, Lower
Kingswood, Tadworth, Surrey, KT20 6TD U.K., 1999.
[8] Harold W. Kuhn. Simplified two-person poker. In Harold W. Kuhn and Albert W. Tucker, editors,
Contributions to the Theory of Games, volume 1, pages 97–103. Princeton University Press, 1950.
[9] Marc Lanctot. Monte Carlo Sampling and Regret Minimization for Equilibrium Computation and
Decision-Making in Large Extensive Form Games. PhD thesis, University of Alberta, University
of Alberta, Computing Science, 116 St. and 85 Ave., Edmonton, Alberta T6G 2R3, January 2013.
[10] Marc Lanctot, Richard Gibson, Neil Burch, and Michael Bowling. No-regret learning in extensive-
form games with imperfect recall. In Proceedings of the Twenty-Ninth International Conference
on Machine Learning (ICML 2012), 2012.
[11] Marc Lanctot, Kevin Waugh, Martin Zinkevich, and Michael Bowling. Monte carlo sampling
for regret minimization in extensive games. In Y. Bengio, D. Schuurmans, J. Lafferty, C. K. I.
Williams, and A. Culotta, editors, Advances in Neural Information Processing Systems 22, pages
1078–1086. MIT Press, 2009.
[12] Kevin Layton-Brown and Yoav Shoham. Essentials of Game Theory: A Concise, Multidisciplinary
Introduction. Morgan and Claypool Publishers, 2008.
[13] Peter Bro Miltersen and Troels Bjerre Sørensen. Computing a quasi-perfect equilibrium of a
two-player game. Economic Theory, 42(1):175–192, 2010.
[14] Merilyn Simonds Mohr. The New Games Treasury. Houghton Mifflin, Boston, Massachusetts,
1993.
[15] Todd W. Neller and Steven Hnath. Approximating optimal Dudo play with fixed-strategy iteration
counterfactual regret minimization. In H. Jaap van den Herik and Aske Plaat, editors, LNCS
7168: Advances in Computer Games, 13th International Conference, ACG 2011, Tilburg, The
Netherlands, November 20-22, 2011, Revised Selected Papers, pages 169–182. Springer, 2012.
July 9, 2013 38
[16] Kevin Waugh, Martin Zinkevich, Michael Johanson, Morgan Kan, David Schnizlein, and
Michael H. Bowling. A practical use of imperfect recall. In Vadim Bulitko and J. Christopher
Beck, editors, SARA. AAAI, 2009.
[17] Qi Zeng, Bruce R. Davis, and Derek Abbott. Reverse auction: The lowest unique positive integer
game. Fluctuation and Noise Letters, 7(4):L439–L447, 2007.
[18] Martin Zinkevich, Michael Johanson, Michael Bowling, and Carmelo Piccione. Regret minimization
in games with incomplete information. In J.C. Platt, D. Koller, Y. Singer, and S. Roweis, editors,
Advances in Neural Information Processing Systems 20, pages 1729–1736. MIT Press, Cambridge,
MA, 2008.