Routines Chi2016
Routines Chi2016
Nikola Banovic, Tofi Buzali, Fanny Chevalier, Jennifer Mankoff, Anind Dey
Such comparisons can be performed between routines of an Model of Human Routine Behavior
individual, but also between routines of different people or We model demonstrated routine behavior using a Markov
populations. For example, to create interventions that help Decision Processes (MDP) framework [32]. MDP is
aggressive drivers improve their driving style requires an particularly well suited for modeling human routine
understanding of the differences between aggressive and behavior because it explicitly models the user’s context, the
non-aggressive driving routines [18]. Based on how close actions that can be performed in that context, and the
drivers are to aggressive or non-aggressive routines, they preferences people have for different actions in different
can be classified into aggressive and non-aggressive groups. contexts. A Markov decision process is a tuple:
Researchers can then identify routine behaviors that need to
𝑀𝑀𝐷𝑃 = 𝑆, 𝐴, 𝑃 𝑠′|𝑠, 𝑎 , 𝑅 𝑠, 𝑎 (1)
change to make aggressive drivers less aggressive overall.
It consists of a set of states 𝑆 (𝑠 ∈ 𝑆) representing context, leaving the current location. As is common for IRL
and actions 𝐴 (𝑎 ∈ 𝐴) that a person can take. In addition, algorithms [29, 45], we assume a parametric reward
the model includes an action-dependent probability
function that is linear in ➚𝑆,Æ, given unknown weight
distribution for each state transition 𝑃 𝑠′|𝑠, 𝑎 , which
parameters 𝜃:
specifies the probability of the next state 𝑠′ when the person
performs action 𝑎 in state 𝑠. This state transition probability 𝑅 𝑠, 𝑎 = 𝜃𝑇 · ➚𝑠t,𝑎t 2
distribution 𝑃 𝑠′|𝑠, 𝑎 models how the environment
We begin the process of recovering the expected state
responds to the actions that people perform in different
states. When modeling human behavior, the transitions are frequencies (𝐷𝑠) and probability distribution of actions
often stochastic (each pair 𝑠, 𝑎 can transition to many given states (𝑃 𝑎|𝑠 ) by trying to learn the person’s reward
transition states 𝑠′ with different probabilities). However, if functions 𝑅 𝑠, 𝑎 from demonstrated behavior. This
the person has full control over the environment, they can problem reduces to matching the model feature function
also be deterministic (i.e., for each pair 𝑠, 𝑎 there is expectations (𝐸P(𝑆,Æ) ➚(𝑆, 𝐴) ) with demonstrated feature
exactly one transition state 𝑠′ with probability 1). Finally, expectations (𝐸∼ ) ➚(𝑆, 𝐴) ) [1]. To match the expected
𝑃(
there is a reward function 𝑅 𝑠, 𝑎 → that the person counts of different features, we use MaxCausalEnt IRL
incurs when performing action 𝑎 in state 𝑠, which [45], which learns the parameters of the MDP model to
represents the utility that people get from performing match the actual behavior of the person. Unlike other
different actions in different contexts. approaches described earlier, MaxCausalEnt explicitly
models the causal relationships between context and
People’s behavior is then defined by sequences of actions actions, and keeps track of the probability distribution of
they perform as they go from state to state until reaching different actions that people can perform in those contexts.
some goal state. In an MDP framework, such behavior is
defined by a deterministic policy (𝜋: 𝑆 → 𝐴), which To compute the unknown parameters 𝜃, MaxCausalEnt
specifies actions people take in different states. considers the causal relationships between all the different
Traditionally, the MDP is “solved” using algorithms, such features of the states and the actions. The Markovian
as value iteration [6], to find an optimal policy (with the property of MDP, which assumes that the actions a person
highest expected cumulative reward). However, our goal is performs only depend on the information encoded by the
to find the expected frequencies of different states and the previous state, makes computing the causal relationships
between the states and actions computationally feasible.
probability distribution of actions given states 𝑃 𝑎|𝑠
MaxCausalEnt extends the Principle of Maximum Entropy
instead—information necessary to identify people’s [19] to cases where information about probability
routines and variations. distribution is sequentially revealed, as is the case with
Learning Routine Patterns from Demonstrated Behavior behavior logs. This principle ensures that the estimated
In this section we explore how the MaxCausalEnt algorithm probability distribution of actions given states 𝑃 𝑎|𝑠 is
[46], an algorithm typically used to predict human behavior the one that best fits the state and action combinations from
[43, 44], can be applied in a novel way to extract routine the sequences in the behavior logs.
behavior patterns from observed data. MaxCausalEnt
algorithm makes its predictions by computing a policy MaxCausalEnt IRL maximizes the causal entropy
(𝜋: 𝑆 → 𝐴) that best predicts the action people take in (𝐻(𝐴𝑇 " 𝑆𝑇)) of the probability distribution of actions
given states 𝑃 𝐴 |𝑆 :
different states. Our main contribution is our insight that in 𝑡 𝑡