Tree-Based Model
Tree-Based Model
Tree-based models use a decision tree to represent how different input variables
can be used to predict a target value. Machine learning uses tree-based models for
both classification and regression problems, such as the type of animal or value of
a home. The input variables are repeatedly segmented into subsets to build the
decision tree, and each branch is tested for prediction accuracy and evaluated for
efficiency and effectiveness. Splitting the variables in a different order can reduce
the number of layers and calculations required to produce an accurate prediction.
Generating a successful decision tree results in the most important variables (most
influential on the prediction) being at the top of the tree hierarchy, while irrelevant
features get dropped from the hierarchy.
Decision tree
Tree models where the target variable can take a discrete set of values are called
classification trees; in these tree structures, leaves represent class labels and
branches represent conjunctions of features that lead to those class labels. Decision
trees where the target variable can take continuous values (typically real numbers)
are called regression trees.
Decision trees are among the most popular machine learning algorithms given their
intelligibility and simplicity.
In decision analysis, a decision tree can be used to visually and explicitly represent
decisions and decision making. In data mining, a decision tree describes data (but
the resulting classification tree can be an input for decision making)
Tree-based methods are deemed to be one of the best and powerful supervised
learning subsets used for classification and regression. These algorithms employ
predictive models with rapid performance, high accuracy, and easy interpretation.
In contrast with linear models, the methods map nonlinear relationships perfectly
and are capable of adapting to various kinds of problems in the machine learning
area.
Figure 1 Example of a decision tree based on the wine quality red data set. The goal is to
predict the quality rating of a (red) wine using chemical properties of the wine. The shaded
nodes and dashed edges indicate how an observation with citric.acid = 0.22, density = 0.993,
sulphates = 0.63, alcohol = 10.6, fixed.acidity = 4.9, is mapped to a prediction (value of 5.508).
Determining how deep to grow the decision tree, handling continuous attributes,
choosing an appropriate attribute selection measure, and handling training data
with missing attribute values, handling attributes with different costs, and
improving computational efficiency are all practical issues in learning decision
trees.
Each branch of the tree is grown just deep enough by the algorithm to properly
categorize the training instances. In reality, when there is noise in the data or when
the number of training instances is insufficient to provide a representative sample
of the underlying target function, it might cause problems. This basic technique
may yield trees that overfit the training samples in either instance.
Related work:
Tree-based based model became popular in machine learning with the introduction
of two algorithms, ID3 (iterative dichotomiser) and CART (classification and
regression Tree-based). Tree-based based model gained popularity due to their
interpretability, but were found to be generally less accurate than other models
such as linear and logistic regression.
A number of ideas were thus consequently proposed for improving the accuracy of
Tree-based-based models, which are all based on constructing an ensemble of
Tree-based-based models. Katie Gross (2020) proposed the idea of bootstrap
aggregation, or bagging, where one builds a collection of predictive models, each
trained with a bootstrapped sample of the original training set; the predictions of
each model are then aggregated into a single prediction (by majority vote for
classification and by averaging for regression). The motivation for bagging is that
it reduces the prediction error for predictive models that are unstable/highly
sensitive to the training data (such as CART); indeed, Katie Gross (2020) showed
that bagged regression Tree-based can be significantly better than ordinary
regression Tree-based in out-of-sample prediction error. Later, Yuncheng Wu et al
(2020) proposed the random forest model, where one builds a collection of bagged
CART Tree-based for which the subset of features selected for splitting at each
node of each Tree-based is randomly sampled from the set of all features (the so-
called random subspace method). Concurrently, other research has considered the
idea of boosting (Momoko and Kazuhisa, 2022), wherein one iteratively builds a
weighted collection of basic predictive models (such as CART Tree-based), with
the goal of reducing the prediction error with each iteration.
To solve the problem, the paper builds a random forest model of the demand of
each style included in a sale as a function of the style’s price and the average price
of the other styles. The paper then formulates an MIO problem to determine the
optimal prices for the styles to be included in the sale, where the revenue is based
on the price and the demand (as predicted by the random forest) of each style. The
MIO formulation does not explicitly model the random forest prediction
mechanism–instead, one computes the random forest prediction for each style at
each of its possible prices and at each possible average price of the other styles,
and these predictions enter the MIO model as coefficients in the objective
function. (The predictions could just as easily have come from a different form of
predictive model, without changing the structure of the optimization problem.) In
contrast, our MIO formulation explicitly represents the structure of each Tree-
based in the model, allowing the prediction of each Tree-based model to be
determined through the variables and constraints of the MIO. Although the
modeling approach of Ferreira et al. (2021) is feasible for their pricing problem, it
is difficult to extend this approach when there are many independent variables, as
one would need to enumerate all possible combinations of values for them and
compute the Tree-based ensemble’s prediction for each combination of values. To
the best of our knowledge, our paper is the first to conceptualize the problem of
how to optimize an objective function that is given by a Tree-based model.
Methodologically, the present paper is most related to the paper of Bertsimas and
Miˇsi´c (2020), which considers the problem of designing a product line or
assortment to optimize revenue under a ranking-based choice model. The ranking-
based model considered in Bertsimas and Miˇsi´c (2020) can be understood as a
type of Tree-based ensemble model; as such, the MIO formulation of Bertsimas
and Miˇsi´c (2020) can be regarded as a special case of the more general
formulation that we analyze here. Some of the theoretical results found in
Bertsimas and Miˇsi´c (2020) – specifically, those results on the structure of the
Benders cuts – are generalized in the present paper to Tree-based models. Despite
this similarity, the goals of the two papers are different. Bertsimas and Miˇsi´c
(2020) consider an optimization approach specifically for product line decisions,
whereas in the present paper, our methodology can be applied to any Tree-based
model, thus spanning a significantly broader range of application domains.
Research method
This Research is developed on the foundation of the literature review of tools and
technologies with methods and techniques used in analysing tree-based model
Research question
Research question 1: what are the methods used in optimizing tree-based model,
Research question 2: what are the tools used in optimizing tree-based model,
Research question 3: what are the techniques used in optimizing tree-based model,
A decision tree is a model for predicting the dependent variable Y using the
independent variable X by checking a collection of splits. A split is a condition or
query on a single independent variable that is either true or false. More precisely,
for a numeric variable i, a split is a query of the form Is
X i ≤ a?
Is X i ∈ A?
where A ⊆ X i is a set of levels of the categorical variable. The splits are arranged
in the form of a tree, with each split node having two child nodes. The left child
corresponds to the split condition being true, while the right child corresponds to
the condition being false. To make a prediction for an observation with the
independent variable X, we start at the root of the tree and check whether X
satisfies the split condition; if it is true, we move to the left child, and if it is false,
we move to the right child. At the new node, we check the split again, and move
again to the corresponding node. This process continues until we reach a leaf of the
tree. The prediction that we make is the value corresponding to the leaf we have
reached. In this paper, we will focus on predictive models that are ensembles or
collections of decision trees. We assume that there are T trees, where each tree is
indexed from 1 to T. Each tree t has a weight λ t, and its prediction is denoted by the
function ft; for the independent variable X, the prediction of tree t is f t (X). For an
observation with independent variable X, the prediction of the ensemble of trees is
given by
T
∑ λt f t ( X)
t =1
The optimization problem that we would like to solve is to find the value of the
independent variable X that maximizes the ensemble prediction:
(1)
T
maximize
X∈X ∑ λ t f t (X )
t =1
We shall make two key assumptions about the tree-based model ∑ λt f t ( X)and our
t =1
1. First, we assume that we are only making a single, one-time decision and
that the tree-based model is fixed. Extending our approach to the multistage
setting is an interesting direction for future research.
2. Second, we assume that the tree-based model is an accurate representation
of the outcome when we make the decision X. In practice, some care must
be taken here because depending on how the tree-based model is estimated
T
and the nature of the data, the prediction ∑ λt f t ( X) may not necessarily be
t =1
Problem (1) is very general, and one question we may have is whether it is
theoretically tractable or not. Our first theoretical result answers this question in the
negative.
Optimization model
We now present an MIO formulation of (1). Before we present the model, we will
require some additional notation. We let N denote the set of numeric variables and
C denote the set of categorical variables; we have that N ∪ C = {1,.. .,n}.
For each numeric variable i ∈ N, let Ai denote the set of unique split points,
that is, the set of values a such that X i ≤ a is a split condition in some tree in the
T
Model { f t}t=11. Let K i = | Ai | be the number of unique split points. Let a i , j denote
the jth smallest split point of variable i, so that a i ,1 < a i ,2< · · · < a i ,k .
i
For each categorical variable i ∈ C, recall that X i is the set of possible values
of i. For convenience, we use K i in this case to denote the size of X i (i.e., K i = | X i |)
and use the values 1,2,.. ., K i to denote the possible levels of variable i.
Let leaves(t) be the set of leaves or terminal nodes of tree t. Let splits(t)
denote the set of splits of tree t (non-terminal nodes). Recall that the left branch of
the split corresponds to “yes” or “true” to the split query, and the right branch
corresponds to “no” or “false”. Therefore, for each split s in St , we let left(s) be the
set of leaves that are accessible from the left branch (all of the leaves for which the
condition of split s must be true), and right(s) be the set of leaves that are
accessible from the right branch (all of the leaves for which the condition of split s
must be false). For each split s, we let V(s) ∈ {1,.. .,n} denote the variable that
participates in split s, and let C(s) denote the set of values of variable i that
participate in the split query of s. Specifically, if V(s) is numeric, then C(s) = {j}
for some j ∈ {1,.. ., K V (s) }, which corresponds to the split query X i ≤ Ai , j. If V(s) is
categorical, then C(s) ⊆ {1,.. ., K V (s)}, which corresponds to the query X i ∈ C(s).
For each leaf ℓ ∈ leaves(t), we use pt ,l to denote the prediction that tree t makes
when an observation reaches leaf ℓ.
We now define the decision variables of the problem. There are two sets of
decision variables. The first set is used to specify the independent variable value X.
For each categorical independent variable i ∈ C and each category/level j ∈ X i , we
let x i , j be 1 if independent variable i is set to level j, and 0 otherwise. For each
numeric independent variable i ∈ N and each j ∈ {1,. .., K i}, we let x i , j be 1 if
independent variable i is set to a value less than or equal to the jth split point, and 0
otherwise. Mathematically,
x i , j= I{ X i = j}, ∀i ∈ C, j ∈ {1,.. ., K i },
x i , j = I{ X i ≤ a i , j }, ∀i ∈ N , j ∈ {1,.. ., K i}.
We use x to denote the vector of x i , j values. The second set of decision variables is
used to specify the prediction of each tree t. For each tree t and each leaf ℓ ∈
leaves(t), we let y t ,l be a binary decision variable that is 1 if the observation
encoded by x belongs to/falls into leaf ℓ of tree t, and 0 otherwise.
Research question 2: what are the tools used in optimizing tree-based model,
1. Local search: We solve the tree ensemble problem (1) using a local search
heuristic. At a high level, it starts from a randomly chosen initial solution and
iteratively improves the solution, one independent variable at a time, until a local
optimum is reached. The heuristic is repeated from ten starting points, out of which
we only retain the best (highest objective value) solution. We test such an
2. Standard linearization MIO: We solve the standard linearization MIO and its
relaxation, in order to obtain a relative sense of the strength of formulation. Due to
this formulation being much harder to solve, we impose a 30-minute time limit on
the solution time of the integer formulation.
∑ ¿leaves (t)∨¿ ¿
t =1
• T TStdLin, MIO the time (in seconds) to solve the standard linearization MIO
• T LS : the time (in seconds) to run the local search procedure (value reported is the
total for ten starting points).
• G LS: the gap of the local search solution; if Z LS is the objective value of the local
search solution and Z¿ is the optimal objective value of, then
G LS = 100% × ( Z¿ − Z LS)/ Z¿ .
• G LO: the gap of the LO relaxation of; if Z LO is the objective value of the LO
relaxation and Z¿ is the optimal integer objective as before, then
G LO = 100% × ( Z LO− Z¿ )/ Z¿ .
• GStdLin ,LO : the gap of the LO relaxation of the standard linearization MIOif Z StdLin , LO
is the optimal value of the relaxation, then
• GStdLin ,MIO : the optimality gap of the standard linearization MIO if Z StdLin ,UB and
Z StdLin , LB are the best upper and lower bounds, respectively, of problem upon
termination, then
Table 2 shows solution times and problem size metrics, while Table 3 shows the
gap metrics. From these two tables, we can draw several conclusions. First, the
time required to solve is very reasonable; in the most extreme case
(winequalityred, T = 500), can be solved to full optimality in about 20 minutes.
(Note that no time limit was imposed on all values of TMIO correspond to the time
required to solve to full optimality.) In contrast, the standard linearization problem
was only solved to full optimality in two out of twenty cases within the 30-minute
time limit. In addition, for those instances where the solver reached the time limit,
the optimality gap of the final integer solution, GStdLin ,MIO , was quite poor, ranging
from 50 to over 100%.
Second, the integrality gap G LO is quite small – on the order of a few percent in
most cases. This suggests that the LO relaxation of is quite tight. In contrast, the
LO relaxation bound from is weaker than that of, as predicted by Proposition 2,
and strikingly so. The weakness of the relaxation explains why the corresponding
integer problem cannot be solved to a low optimality gap within the 30-minute
time limit. These results, together with the results above on the MIO solution times
and the final optimality gaps of, show the advantages of our formulation over the
standard linearization formulation.
Third, although there are many cases where the local search solution
performs quite well, there are many where it can be quite suboptimal, even when
repeated with ten starting points. Moreover, while the local search time T LS is
generally smaller than the MIO time T MIO , in some cases it is not substantially
lower (for example, solubility for T = 500), and the additional time required by the
MIO formulation may therefore be justified for the guarantee of provable
optimality.
Discussion
Conclusion