1

Final Project Report

Logistic Regression and Linear Discriminant Analysis –

Holiday Package Analysis

PREDICTIVE MODELLING

Nabeel Ahmed Khan

August ‘21

Date: 27/08/2021
 2

Table of Contents

Table of Contents..................................................................................................................................2
Table of Figures.....................................................................................................................................3
1. Executive Summary.......................................................................................................................4
2. Introduction...................................................................................................................................4
3. Data Details...................................................................................................................................5
4. Data Ingestion and EDA - Descriptive Statistics, Duplicate/Null value Check, Uni-/Bi-Variate
Analysis, Outlier Check..........................................................................................................................5
4.1 Data Ingestion........................................................................................................................5
Sample of the Dataset...................................................................................................................5
Data Info........................................................................................................................................6
Data Shape & Data Types..............................................................................................................6
Data Description............................................................................................................................7
Duplicate Value Check...................................................................................................................7
NULL Value Check..........................................................................................................................7
Inference.......................................................................................................................................8
4.2 Univariate Analysis................................................................................................................8
Distribution Plots...........................................................................................................................8
Count Plots..................................................................................................................................10
Inference.....................................................................................................................................11
4.3 Transforming Categorical variables to Numeric Variables...................................................11
4.4 Bivariate Analysis.................................................................................................................12
Pairplots.......................................................................................................................................12
Correlation Heatmap...................................................................................................................13
4.5 Outlier Check.......................................................................................................................13
5. Logistic Regression.......................................................................................................................17
5.1. Preparing for Model Formulation........................................................................................17
5.2. Formulating a Logistic Regression Model on the Training data...........................................17
6. Logistic Regression: Performance Metrics...................................................................................17
6.1 LR: Model Score...................................................................................................................17
6.2 LR: Confusion Matrix............................................................................................................17
6.3 LR: Classification Report......................................................................................................18
6.4 LR: AUC Score......................................................................................................................18
6.5 LR: ROC Curve......................................................................................................................19
7. Linear Discriminant Analysis........................................................................................................20
 3

7.1 Preparing for Model Formulation........................................................................................20

7.2 Formulating a LDA Model on the Training data...................................................................20
8. LDA: Performance Metrics...........................................................................................................20
8.1 LDA: Model Score................................................................................................................20
8.2 LDA: Confusion Matrix.........................................................................................................20
8.3 LDA: Classification Report....................................................................................................21
8.4 LDA: AUC Score....................................................................................................................21
6.6 LDA: ROC Curve....................................................................................................................22
9. Programming Files.......................................................................................................................23
10. Inferences: Insights & Recommendations...............................................................................23
10.1 Insights.................................................................................................................................23
10.2 Recommendations...............................................................................................................24

Table of Figure

Figure 1: Holiday Package Data Info......................................................................................................6

Figure 2: Data Shape & Data types........................................................................................................6
Figure 3: Data Description of Integer Type Variables............................................................................7
Figure 4: Data Description of Object Type Variables.............................................................................7
Figure 5: Null Value Check.....................................................................................................................7
Figure 6: Univariate Analysis - Distribution Plots...................................................................................9
Figure 7: Univariate Analysis - Count Plots..........................................................................................10
Figure 8: Holiday Package Dataset Pair Plots.......................................................................................12
Figure 9: Holiday Package Dataset Correlation Heatmap....................................................................13
Figure 10: Box Plots for Continuous Variables.....................................................................................15
Figure 11: Box Plots for Continuous Variables post Outlier Treatment...............................................16
Figure 12: LR ROC Curve for Training Data..........................................................................................19
Figure 13: LR ROC Curve for Test Data.................................................................................................19
Figure 12: LR ROC Curve for Training Data..........................................................................................22
Figure 13: LR ROC Curve for Test Data.................................................................................................22

Table of Tables

Table 1: Dataset Sample........................................................................................................................5

 4

1. Executive Summary

A tour and travel agency which deals with holiday package sales has shared a dataset of 872

employees of a company who were offered a holiday package by the agency. Some of these

employees opted for the package while some did not. The dataset contains a number of

variables and the information on these variables for the various employees is provided, along

with the information on whether an employee opted for a holiday package or not.

The company is trying to predict whether an employee will opt for a holiday package on the

basis of the values of the other attributes/characteristics. Basically, the agency wants a model

which offers a way of predicting the action of an employee to opt (or not opt) for a holiday

package. They also want to understand the factors (from these variables in the dataset) that

are crucial in determining whether a given employee purchases a holiday package, so that the

agency can focus its resources to sell their package to such employees. So, as part of this

project report, I will try to explore the various attributes provided in the dataset and their

contribution to the decision of an employee to purchase the package.

2. Introduction

The intent for this entire exercise is perform analysis on the holiday package dataset. We will

try to explore this dataset by using descriptive statistics, univariate and bivariate analysis,

Exploratory Data Analysis and applying LDA and Logistic Regression. This dataset contains

details on about 872 employee, and I will try to analyse the various attributes provided about

the employees in the dataset, and determine how they contribute in determining whether an

employee opts for a holiday package or not.

 5

3. Data Details

The first column contains an index variable, which is simply the serial number of the entry. I

dropped the index column as it is useless for the model. Following are the data variables:

Holliday_Package : Opted for Holiday Package? (yes/no)

Salary : Employee salary

age : Age of the employee in years

edu : Years of formal education

no_young_children : The number of young children (younger than 7 years)

no_older_children : The number of older children (7 years or more)

foreign : Foreigner (yes/no)

4. Data Ingestion and EDA - Descriptive Statistics, Duplicate/Null value

Check, Uni-/Bi-Variate Analysis, Outlier Check

4.1 Data Ingestion

Sample of the Dataset

Table 1: Dataset Sample

 6

As we can see, the holiday package dataset has 7 variables with each employee having the

same set of characteristics. Here, we can say that Holliday_Package is the dependent or target

feature and the rest of the variables are the independent or predictor variables. Based on the

independent variables, the value (yes/no) of the Holliday_Package variable is defined.

Data Info
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 872 entries, 0 to 871
Data columns (total 8 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 Unnamed: 0 872 non-null int64
1 Holliday_Package 872 non-null object
2 Salary 872 non-null int64
3 age 872 non-null int64
4 educ 872 non-null int64
5 no_young_children 872 non-null int64
6 no_older_children 872 non-null int64
7 foreign 872 non-null object
dtypes: int64(6), object(2)
memory usage: 54.6+ KB

Figure 1: Holiday Package Data Info

Data Shape & Data Types

(872, 7)
Holliday_Package object
Salary int64
age int64
educ int64
no_young_children int64
no_older_children int64
foreign object
dtype: object

Figure 2: Data Shape & Data types

It can be observed that after dropping the serial number column, the dataset has 872

employee records and 7 variables. Of the 7 variables, the target feature Holliday_Package and

foreign are object types while the other are integer type.
 7

Data Description
Salary age educ no_young_children no_older_children

count 872.000000 872.000000 872.000000 872.000000 872.000000

mean 47729.172018 39.955275 9.307339 0.311927 0.982798

std 23418.668531 10.551675 3.036259 0.612870 1.086786

min 1322.000000 20.000000 1.000000 0.000000 0.000000

25% 35324.000000 32.000000 8.000000 0.000000 0.000000

50% 41903.500000 39.000000 9.000000 0.000000 1.000000

75% 53469.500000 48.000000 12.000000 0.000000 2.000000

max 236961.000000 62.000000 21.000000 3.000000 6.000000

Figure 3: Data Description of Integer Type Variables

Holliday_Package foreign

Count 872 872

unique 2 2

top no no

freq 471 656

Figure 4: Data Description of Object Type Variables

Duplicate Value Check

When I checked for any duplicate entries, I got the following output:
Duplicate Entries: 0

NULL Value Check

Holliday_Package 0
Salary 0
age 0
educ 0
no_young_children 0
no_older_children 0
foreign 0
dtype: int64

Figure 5: Null Value Check

As we can see, there are no null values in the dataset.

 8

Inference
1. Our Target Feature is the Holliday_Package variable. It has a binary value which

equals 0 for employees not opting to purchase the holiday package and 1 for the

employee opting to purchase the holiday package.

2. All other variables except foreign, like age, educ are continuous numeric variables

3. Since, the target feature Holliday_Package and the variable foreign are object type,

we need to encode them and convert them into categorical variables for creating a

model for the Holiday Package dataset

4. The first column contained an index variable, which is simply the serial number of the

entry. I dropped the index column as it is useless for the model

5. There are no Duplicate entries in the dataset

6. There are no NULL values in the dataset

4.2 Univariate Analysis

Distribution Plots
I plotted the numeric data variables below.
 9

Figure 6: Univariate Analysis - Distribution Plots

 10

Count Plots
I plotted the categorical data variables below.

Figure 7: Univariate Analysis - Count Plots

 11

Inference
1. Nearly 46% of employees have opted for the Holiday Package

2. Majority of the employees are native to the country

3. Majority of the employees do not have any children.

4. Majority of the employees who have children have 1 younger children (kids less than

7 years of age) and/or 1-2 older children (kids more than 7 years of age)

5. The average years of education range from 3-17; while the major chunk of employees

have 8-12 years of education

6. Majority of the employees (more than two-thirds) fall in the $25000 to $55000 range

of salary.

7. Nearly all the employees fall in the 20-60 years of age range.

4.3 Transforming Categorical variables to Numeric Variables

I encoded the object type/categorical variables Holliday_Package, and foreign using Panda’s

.codes function
 12

4.4 Bivariate Analysis

Before proceeding with Bivariate Analysis, I transformed the categorical variables to numeric

variables.

Pairplots

Figure 8: Holiday Package Dataset Pair Plots

 13

Correlation Heatmap

Figure 9: Holiday Package Dataset Correlation Heatmap

From the Pair plots and the Correlation Heatmap, it can be observed that the variables in the

Holiday Package dataset form separated clusters and are not much corelated with one

another. Maximum correlation between two variables is between education is salary and that

too is 0.33 only. Also, whatever limited correlation our target feature Holliday_Package has,

it has that with the foreign variable only.

4.5 Outlier Check

I ran boxplot for the holiday package dataset as follows.

14
 15

Figure 10: Box Plots for Continuous Variables

As we can see from the Box Plots above, mostly the variable Salary has outliers, which may impact

the efficacy of the regression model I will build. I have treated the outliers in the dataset

using the 25th and 75th percentiles. Post that, I re-checked for outliers once more (please see

below).
 16

Figure 11: Box Plots for Continuous Variables post Outlier Treatment
 17

5. Logistic Regression

5.1. Preparing for Model Formulation

I dropped the target feature Holliday_Package and stored it separately. Then I have used the

randomized training and test data splitting function from Sklearn package to split the data

into train and test datasets in the ratio 70:30 (The test data size should be 30% of the total

data).

5.2. Formulating a Logistic Regression Model on the Training data

Then I applied LinearRegression to obtain the bestfit model on training data.

.37)*height + Intercept

6. Logistic Regression: Performance Metrics

6.1 LR: Model Score

1. The Accuracy Score for the Regression Model on Training data is 0.51967

2. The Accuracy Score for the Regression Model on Test data is 0.53053

6.2 LR: Confusion Matrix

1. Confusion Matrix for the Regression Model on Training data

[[294 32]
[261 23]]

2. Confusion Matrix for the Regression Model on Test data

[[129 16]
[107 10]]
 18

6.3 LR: Classification Report

1. Classification Report for the Regression Model on Training data

precision recall f1-score support

0 0.53 0.90 0.67 326

1 0.42 0.08 0.14 284

accuracy 0.52 610

macro avg 0.47 0.49 0.40 610
weighted avg 0.48 0.52 0.42 610

2. Classification Report for the Regression Model on Test data

precision recall f1-score support

0 0.55 0.89 0.68 145

1 0.38 0.09 0.14 117

accuracy 0.53 262

macro avg 0.47 0.49 0.41 262
weighted avg 0.47 0.53 0.44 262

6.4 LR: AUC Score

AUCTrain: 0.567

AUCTest: 0.627
 19

6.5 LR: ROC Curve 

Figure 12: LR ROC Curve for Training Data

Figure 13: LR ROC Curve for Test Data

 20

7. Linear Discriminant Analysis

I used a separate jupyter notebook for my Linear Discriminant Analysis of the Holiday

Package data.

7.1 Preparing for Model Formulation

I converted the categorical variables to dummy variables using Panda’s get_dummies

function.

I captured the target feature (Holliday_Package_yes after converting to dummy variables)

into separate vectors for Training set and Test set. Then I have used the randomized training

and test data splitting function from Sklearn package to split the data into train and test

datasets in the ratio 70:30 (The test data size should be 30% of the total data).

7.2 Formulating a LDA Model on the Training data

Then I applied LDA to obtain the bestfit model on training data.

8. LDA: Performance Metrics

8.1 LDA: Model Score

1. The Accuracy Score for the LDA model on Training data is 0.6721

2. The Accuracy Score for the LDA model on Test data is 0.6412

8.2 LDA: Confusion Matrix

1. Confusion Matrix for the LDA Model on Training data

[[252 74]
[126 158]]

2. Confusion Matrix for the LDA Model on Test data

 21

[[103 42]
[52 65]]

8.3 LDA: Classification Report

1. Classification Report for the Regression Model on Training data

precision recall f1-score support

0 0.67 0.77 0.72 326

1 0.68 0.56 0.61 284

accuracy 0.67 610

macro avg 0.67 0.66 0.66 610
1. weighted avg 0.67 0.67 0.67 610

2. Classification Report for the Regression Model on Test data

precision recall f1-score support

0 0.66 0.71 0.69 145

1 0.61 0.56 0.58 117

accuracy 0.64 262

macro avg 0.64 0.63 0.63 262
weighted avg 0.64 0.64 0.64 262

8.4 LDA: AUC Score

AUCTrain: 0.742

AUCTest: 0.703
 22

8.5 LDA: ROC Curve 

Figure 14: LR ROC Curve for Training Data

Figure 15: LR ROC Curve for Test Data

 23

9. Programming Files

Predictive_Modellin Predictive_Modellin Predictive Predictive_Modellin

g_Nabeel_Khan_Final_Project_Report_LDA.pdf
g_Nabeel_Khan_Final_Project_Report_LDA.ipynb
Modelling_Nabeel Khan_Final
g_Nabeel_Khan_Final_Project_Report_Logistic_Regression.ipynb
Project Report-Logistic Regression.pdf

10. Inferences: Insights & Recommendations

10.1 Insights

1. The model score for the Logistics Regression’s Training dataset is 51.9% and that for

Test dataset is 53.0%.

2. Classification Report of Logistic Regression Model:

PrecisionLDA_Train = 42% | PrecisionLDA_Test = 38%

RecallLDA_Train = 8% | RecallLDA_Test = 9%

F1LDA_Train = 14% | F1LDA_Test = 14%

3. The AUC for Training data is 56.7% and Test data is 62.7%

4. The Logistic Regression model doesn’t seem to be a good fit and may needs improvement

5. The model score for the LDA’s Training dataset is 67.21% and that for Test dataset is

64.12%. As we can see the accuracy score of LDA model is better than that of Logistic

Regression model

6. Classification Report of LDA Model:

PrecisionLDA_Train = 68% | PrecisionLDA_Test = 61%

RecallLDA_Train = 56% | RecallLDA_Test = 56%

F1LDA_Train = 61% | F1LDA_Test = 58%

Clearly the Classification report for the LDA model is better than that of the Logistic

Regression model

7. The AUC for Training data is 74.2% and Test data is 70.3% which quite similar

Again the AUC for the LDA model is better than that of the Logistic Regression model
 24

8. As we can see that the Logistic regression model and the LDA model are able to predict

the behaviour of the employees regarding opting the Holiday Package for around 53%

and 62.7% of the employees with accuracy. Since the accuracy score of LDA model is

better, I will prefer the LDA model.

9. Another thing is that the LDA model can correctly predict the employee behaviour 56%

of the times while the Logistic Regression model can correctly predicts for only 8-9% of

the times.

10. From the above I can say that the LDA model is better for the travel agency to use in an

attempt to improve their bottom line.

10.2 Recommendations

1. The dataset has outliers in the salary variable. We know that Logistic Regression is a

better predictor when outliers are present. Therefore, it is recommended to treat the

outliers before proceeding to use LDA

2. As we saw above, if an employee is a foreigner and the employee does not have any

young children (can be seen using Holliday_Package as hue in bivariate plots), the

probability of an employee to purchase the Holiday Package is higher. Also, many

employees who have older children do not go for the Holiday Package. So, the agency

can devise special promotional programs and discounts to such employees to incentivize

them for opting for the Holiday Package.

3. Moreover, a lot of employees having higher salary are not purchasing the Holiday

Package (again, can be seen using Holliday_Package as hue in bivariate plots). SO the

travel agency can come up with a plan to create more product awareness and introduce

targeted promotions for such employees.

 25

4. The age of the employee is not a material in opting for holiday package, so it can be

ignored.

5. It was observed from the correlation coefficients that the target feature Holliday_Package

has a high negative correlation with no_young_children. So, it would go a long way if the

travel agency can tailor their holiday packages so as to make them more appealing to

employees with infants and young children.