Binning
Binning
Guoping Zeng
Elevate/Think Finance
4150 International Plaza
Fort Worth, TX 76109, USA
Copyright © 2014 Guoping Zeng. This is an open access article distributed under the Creative
Commons Attribution License, which permits unrestricted use, distribution, and reproduction in
any medium, provided the original work is properly cited.
Abstract
Binning is a categorization process to transform a continuous variable into a small set of
groups or bins. Binning is widely used in credit scoring. In particular, it can be used to
define the Weight of Evidence (WOE) transformation. In this paper, we first derive an
explicit solution to a logistic regression model with one independent variable that has
undergone a WOE transformation. We then use this explicit solution as a necessary
condition for a good binning algorithm, thus providing a simple way to catch binning
errors.
1 Introduction
Binning is a categorization process to transform a continuous variable into a small
set of groups or bins. Binning is widely used in credit scoring. While binning can
be used to find Kolmogorov-Smirnov (KS) and lift chart from scores, binning is
more frequently used at an early stage to select variables in credit scoring.
Similar attributes (values) of an independent variable are grouped into the
same bin to enhance the predictive power. After binning is done, Information
Value (IV) and other types of metric divergence measures can be calculated [8].
In particular, binning can be used to introduce Weight of Evidence (WOE)
transformations for continuous variables.
3230 Guoping Zeng
The rest of the paper is organized as follows. In Section 2, the basic of logistic
regression and maximum likelihood estimate are reviewed. In Section 3, we derive
an explicit solution to a logistic regression model with one continuous variable
that has undergone the WOE transformation. Section 4 states the necessary
condition for good binning and presents a numerical example to catch binning
errors. The paper is concluded in Section 5.
(2.1)
[ ] (2.2)
∏ ( ) [ ( )] (2.3)
Note that if is known, either 0 or 1, the 2 terms in the product of (2.3) reduces
to only one term as the other term will have value of 1.
[ ]
∑ { [ ( )] [ ( )]} (2.4)
∑ ∑ ( ) (2.5)
3232 Guoping Zeng
∑ ∑ ( ) (2.6)
Since binning is done for each independent variable, from now on we shall focus on
a single independent variable . In this case, Equations (2.5) and (2.6) become
∑ ∑ (2.7)
∑ ∑ (2.8)
Remark 2.1. If x has missing values, they will be simply ignored in logistic
regression. Specifically, missing values will be taken out from (2.7) and (2.8). Thus,
the 4 summations in (2.7) and (2.8) are taken for all non-missing values of .
Let’s study the existence and uniqueness of the maximum likeliness estimate. Since
the log likelihood is globally concave, the maximum likeliness estimate can have at
most one solution [1]. Therefore, if the maximum likeliness estimate has a solution,
it must be unique. However, there are situations in which the log likelihood
function has no maximum and hence the maximum likelihood estimate does not
exist. For instance, in case of complete separation or quasi-complete separation [2]
the maximum likelihood estimate does not exist. Consider a dataset with 20
observations, where y = 0 if x is and 1 if x is
The left hand side of (2.8) is 55, and the absolute value of right hand side of (2.8) is
|∑ [ ]| ∑ | |
On the other hand, if the simultaneous equations (2.7) and (2.8) have a solution, a
natural question arises: can it be explicitly solved. For some models and data, the
answer is yes. For instance, when the system has only one independent variable and
this independent variable is dichotomous [5]. For most models, they cannot be
explicitly solved and must be solved by numerical methods.
Good binning algorithm in credit scoring 3233
… … … … …
Denote the number of good accounts and number of bad accounts at group by
and respectively. Then, is equal to the sum of in group , that is,
Since the observations with missing will be ignored, we may assume y has no
missing values without loss of generality. Hence, represents the
number of accounts in bin 1, and the number of accounts at
group j for . For convenience, we define as 0.
Note that all the values of in each group have the same WOE. In this case, the
values of are transformed into grouped WOE’s, thus reducing complexities to
the modeling.
Theorem 3.1. When a logistic regression model is fitted with one independent
variable that has undergone a WOE transformation, the maximum likelihood
estimate has an explicit solution ( ) and where and are
the number of bad accounts and number of good accounts, respectively.
Case I: No groups have 0 good accounts or 0 bad accounts, that is, and
are all positive for
We first verify (2.7). Note that the new independent variable has the same
value inside each group. Substituting and into the left hand side
of (2.7), we obtain
Good binning algorithm in credit scoring 3235
∑ ∑
∑ ∑
∑ ∑( ∑ ) ∑
∑ ∑
∑ ∑
∑ ∑
∑ (∑ ) ∑ ∑
∑ ∑
∑ ∑ ∑
∑ ∑ ∑
Note that if has no missing values, bin does not exist. The above proof
still holds after the last row of table 1 is removed.
Case II: Some groups have 0 bad accounts or 0 good accounts, that is,
or for some i.
In this case, WOEs are not defined and so the new independent variable
will have missing values in these groups. Assume among the k + 1 groups only
groups do not have 0 bad account or 0 good account, where
Then, the left hand side of (2.7) becomes
Good binning algorithm in credit scoring 3237
∑ ( ) ∑ ( )
( ) ( )
( ) ( )
∑ ∑
∑ ( ) ∑ ( )
( ) ( )
( ∑ ) ( ∑ )
∑ ∑
3238 Guoping Zeng
Remark 3.1. Refaat [6] uses an example to find an explicit solution to the
maximum likelihood estimate for an independent categorical variable. We have
extended to an independent continuous variable, considered missing values and
bins with 0 good or bad accounts, and analytically verified the solution.
4 A Necessary Condition
Theorems 3.1 can serve as a necessary condition for a good binning algorithm.
data age1;
input age y @@;
datalines;
10 0 10 0 10 1 10 0 10 1 10 0 10 0 10 0 10 0 10 0
10 1 10 0 10 0 10 0 10 1 10 1 10 0 10 0 10 0 10 0
10 0 10 0 10 0 10 0 10 0 10 0 10 0 10 1 10 0 10 0
10 0 10 0 10 0 10 1 10 0 10 0 10 0 10 0 10 0 10 0
10 0 10 0 10 0 10 0 10 0 10 0 10 0 10 1 10 0 10 1
20 0 20 0 20 0 20 1 20 0 20 0 20 1 20 0 20 0 20 0
20 1 20 0 20 0 20 0 20 0 20 0 20 1 20 0 20 1 20 0
20 0 20 0 20 0 20 1 20 0 20 0 20 0 20 0 20 0 20 0
30 1 30 0 30 0 30 1 30 1 30 0 30 0 30 0 30 0 30 0
40 0 40 0 40 0 40 0 40 0 40 0 40 0 40 0 40 0 48 0
.0.0.1.0.0.0.1.0.0.0.1
;
run;
The data after the datalines statement in the above SAS program represent the
values of age and in turn. Note that age has some missing values. Using
equal-size binning with SAS PROC RANK, we obtain the first 4 columns. We
then calculate WOE for each bin and put it in column 5.
Good binning algorithm in credit scoring 3239
Note that the total number of good accounts and total number of bad accounts
The following SAS data step will perform the WOE transform to transform values
of into WOE.
data age2;
set age1;
if age = . then M_age = 0.4744579796;
if age = 10 then M_age = -0.061060257;
else if age = 20 then M_age = 0.0689928715;
else if age = 30 then M_age = 0.6079893722;
else if age > 30 then M_age = . ;
run;
Standard Wald
Parameter DF Estimate Error Chi-Square Pr > ChiSq
Intercept 1 -1.4553 0.2794 27.1382 <.0001
M_age 1 1 0.9874 1.0258 0.3112
Table 4.2: Maximum Likelihood Estimate Output 1
3240 Guoping Zeng
Next, let’s merge the bin with missing values to bin 3 and run logistic regression
again to obtain results Table 4.3.
data age4;
set age2;
if age > 30 then M_age = 0.6079893722;
run;
Obviously, it does not follow Theorem 3.1. Therefore, we have caught binning
errors. Indeed, when bin 4 is merged into bin 3, the number of good accounts and
number of bad accounts in the new bin should be adjusted to 10 + 7 = 17 and 0 +
3 = 3, respectively. Hence, WOE in the new bin is
( )
Let’s change WOE and run one more time logistic regression.
data age5;
set age2;
if age >= 30 then M_age = ;
run;
proc logistic data=age5 descending;
model y = M_age;
run;
Good binning algorithm in credit scoring 3241
As shown in Table 4.4, this time it follows Theorem 3.1. Note that the slope is not
exactly 1 and the intercept is not exactly -1.4553 due to inevitable computational
errors. Hence, we have caught and corrected the binning error.
5 Conclusions
References
[5] Hosmer D. W. and Lemeshow, S. (2000). Applied Logistic Regression. 2nd Ed.,
John Wiley and Sons, New York.
[9] Zeng, G. (2014). A Rule of thumb for reject inference in credit scoring.
Mathematical Finance Letters, Article ID 2.