Professional Documents
Culture Documents
Data Mining - Project
Data Mining - Project
Clustering Clean_Ads
1
Table of Contents
Part 1 - Clustering: Read the data and perform basic analysis such as printing a few rows (head and
tail), info, data summary, null values duplicate values, etc.
Part 1 - Clustering: Treat missing values in CPC, CTR and CPM using the formula given.
Part 1 - Clustering: Check if there are any outliers. Do you think treating outliers is necessary for K-
Means clustering? Based on your judgment decide whether to treat outliers and if yes, which method
to employ. (As an analyst your judgment may be different from another analyst).
Part 1 - Clustering: Perform z-score scaling and discuss how it affects the speed of the algorithm.
Part 1 - Clustering: Perform Hierarchical by constructing a Dendrogram using WARD and Euclidean
distance.
Part 1 - Clustering: Make Elbow plot (up to n=10) and identify optimum number of clusters for k-
means algorithm.
Part 1 - Clustering: Print silhouette scores for up to 10 clusters and identify optimum number of
clusters.
Part 1 - Clustering: Profile the ads based on optimum number of clusters using silhouette score and
your domain understanding [Hint: Group the data by clusters and take sum or mean to identify trends
in Clicks, spend, revenue, CPM, CTR, & CPC based on Device Type. Make bar plots].
Part 2 - PCA: Read the data and perform basic checks like checking head, info, summary, nulls, and
duplicates, etc.
Part 2 - PCA: Perform detailed exploratory analysis by creating certain questions like (i) which state
has highest gender ratio and which has the lowest? (ii) Which district has the highest & lowest gender
ratio? (Example Questions). Pick 5 variables out of the given 24 variables below for EDA: No_HH,
TOT_M, TOT_F, M_06, F_06, M_SC, F_SC, M_ST, F_ST, M_LIT, F_LIT, M_ILL, F_ILL,
TOT_WORK_M, TOT_WORK_F, MAINWORK_M, MAINWORK_F, MAIN_CL_M, MAIN_CL_F,
MAIN_AL_M, MAIN_AL_F, MAIN_HH_M, MAIN_HH_F, MAIN_OT_M, MAIN_OT_F
Part 2 - PCA: We choose not to treat outliers for this case. Do you think that treating outliers for this
case is necessary?
Part 2 - PCA: Scale the Data using z-score method. Does scaling have any impact on outliers?
Compare box plots before and after scaling and comment.
Part 2 - PCA: Perform all the required steps for PCA (use sklearn only) Create the covariance Matrix
Get Eigen values and eigen vector.
Part 2 - PCA: Identify the optimum number of PCs (for this project, take at least 90% explained
variance). Show Scree plot.
Part 2 - PCA: Compare PCs with Actual Columns and identify which is explaining most variance.
Write inferences about all the Principal components in terms of actual variables.
2
Problem Statement: 1
Clustering:
The ads24x7 is a Digital Marketing company which has now got seed funding of $10 Million. They are
expanding their wings in Marketing Analytics. They collected data from their Marketing Intelligence
team and now wants you (their newly appointed data analyst) to segment type of ads based on the
features provided. Use Clustering procedure to segment ads into homogeneous groups.
CPM = (Total Campaign Spend / Number of Impressions) * 1,000. Note that the Total Campaign
Spend refers to the 'Spend' Column in the dataset and the Number of Impressions refers to the
'Impressions' Column in the dataset.
CPC = Total Cost (spend) / Number of Clicks. Note that the Total Cost (spend) refers to the 'Spend'
Column in the dataset and the Number of Clicks refers to the 'Clicks' Column in the dataset.
CTR = Total Measured Clicks / Total Measured Ad Impressions x 100. Note that the Total
Measured Clicks refers to the 'Clicks' Column in the dataset and the Total Measured Ad Impressions
refers to the 'Impressions' Column in the dataset.
Q.1) Read the data and perform basic analysis such as printing a few rows (head and tail), info,
data summary, null values duplicate values, etc.
Ans:
Data Frame printing rows with Head (Prints top 5 rows) function as below :
Data Frame printing rows with Tail (Prints last 5 rows) function as below :
3
Data Frame Summary as below :
(23066, 19) There are 23066 rows, and 19 columns into the Data.
There are 4636 Null values in CTR, CPM, and CPC Columns.
4
Q.2) Treat missing values in CPC, CTR and CPM using the formula given.
Ans:
Note that the Total Campaign Spend refers to the 'Spend' Column in the dataset and the Number of
Impressions refers to the 'Impressions' Column in the dataset.
Note that the Total Cost (spend) refers to the 'Spend' Column in the dataset and the Number of Clicks
refers to the 'Clicks' Column in the dataset.
Note that the Total Measured Clicks refers to the 'Clicks' Column in the dataset and the Total
Measured Ad Impressions refers to the 'Impressions' Column in the dataset.
Data Set Before Treating Missing Value Data Set After Treating Missing Value
5
Q.3) Check if there are any outliers. Do you think treating outliers is necessary for K-Means
clustering? Based on your judgement decide whether to treat outliers and if yes, which
method to employ. (As an analyst your judgement may be different from another analyst).
Ans:
I have checked with the data and it seems that there are Outliers. Below is the Boxplot figure of
Features before Treating Outliers.
Yes. Treating Outliers is necessary for K-Means Clustering. We are going to treat outliers by IQR
Method. (IQR : Interquartile Range).
I have created a ‘remove_outlier’ function using IQR formulas’. We can’t perform Outlier Treatment on
Categorical features. Hence, I have created new dataset of Int64 and Float64 data types with the
name of df1. And, applied ‘remove_outlier’ function on it and removed outliers.
6
Q.4) Perform z-score scaling and discuss how it affects the speed of the algorithm.
Ans:
Here, I have applied z-score method and got the below output.
7
Q.5) Perform Hierarchical by constructing a Dendrogram using WARD and Euclidean distance.
Ans:
Please find below Dendrogram performed for Hierarchical using WARD and Euclidean Distance on
the Scaled Data such as “data1_scaled”.
In this Dendrogram, value of P = 10, which means that only the last 10 merged clusters are shown.
Q.6) Make Elbow plot (up to n=10) and identify optimum number of clusters for k-means
algorithm.
8
For checking the Optimal number of clusters we use WSS (Within Sum Of Square).
When we move from K=1 to K=2, We see that there is a significant drop in the value.
Also when we move from k=2 to k-3, k=3 to k=4, k=4 to k=5 there is a significant drop as well.
k=5 to k=6, the drop in values reduces significantly.
Hence In this case, the WSS is not significantly dropping beyond 5, so 5 is optimal number
of clusters.
Q.7) Print silhouette scores for up to 10 clusters and identify optimum number of clusters.
9
Silhouette Score for k=8: 0.590
Silhouette Score for k=9: 0.601
Silhouette Score for k=10: 0.496
I have calculated Silhouette Score for scaled data using the silhouette_score() function.
The Silhouette Score is a measure of how similar an object is to its own cluster compared to other
clusters, and it ranges from -1 to 1, with higher values indicating better clustering.
As per Elbow plot/scree-plot, we concluded that the optimal number of clusters should be 5 because 2
would be very less number of clusters.
Q.8) Profile the ads based on optimum number of clusters using silhouette score and your
domain understanding [Hint: Group the data by clusters and take sum or mean to identify
trends in Clicks, spend, revenue, CPM, CTR, & CPC based on Device Type. Make bar plots].
Ans:
• I have performed KMeans Clustering on scaled data, and then added the predicted cluster labels to
two different data sets: data1_scaled_copy and df1.
• The KMeans function from scikit-learn is used to create a KMeans object with n_clusters=5 (i.e., 5
clusters).
• Created clusters for the Ads based on optimum number of clusters using silhouette score.
• The groupby method from Pandas is used to group the data by the KMeans cluster labels, and the
mean and median methods are used to compute the mean and median values of each feature within
each cluster. The resulting data frames are stored in the variables mean and median.
• Then, the pd.concat method from Pandas is used to concatenate the mean and median data frames
along the vertical axis (axis=0) to create a single data frame df_kmeans. The index attribute of the
resulting data frame is set to a list of strings representing the names of each group followed by either
'Mean' or 'Median'.
10
Q.9) Conclude the project by providing summary of your learning’s.
Ans:
There are 23066 rows, and 19 columns into the Dataset.
There are 4636 Null values in CTR, CPM, and CPC Columns.
I have treated missing values in CPC, CTR, and CPM columns using the given formula
As per Elbow plot/scree-plot, we concluded that the optimal number of clusters should be 5.
11
I have created 5 clusters for the Dataset.
When amount of money spent on specific ad variations within a specific campaign or ad set is
increases then Revenue is also increases.
When impression count of the particular Advertisement increases then Revenue is also
increases
12
Problem Statement: 2
PCA:
PCA FH (FT): Primary census abstract for female headed households excluding institutional
households (India & States/UTs - District Level), Scheduled tribes - 2011 PCA for Female Headed
Household Excluding Institutional Household. The Indian Census has the reputation of being one of
the best in the world. The first Census in India was conducted in the year 1872. This was conducted
at different points of time in different parts of the country. In 1881 a Census was taken for the entire
country simultaneously. Since then, Census has been conducted every ten years, without a break.
Thus, the Census of India 2011 was the fifteenth in this unbroken series since 1872, the seventh after
independence and the second census of the third millennium and twenty first century. The census has
been uninterruptedly continued despite of several adversities like wars, epidemics, natural calamities,
political unrest, etc. The Census of India is conducted under the provisions of the Census Act 1948
and the Census Rules, 1990. The Primary Census Abstract which is important publication of 2011
Census gives basic information on Area, Total Number of Households, Total Population, Scheduled
Castes, Scheduled Tribes Population, Population in the age group 0-6, Literates, Main Workers and
Marginal Workers classified by the four broad industrial categories, namely, (i) Cultivators, (ii)
Agricultural Laborers, (iii) Household Industry Workers, and (iv) Other Workers and also Non-
Workers. The characteristics of the Total Population include Scheduled Castes, Scheduled Tribes,
Institutional and Houseless Population and are presented by sex and rural-urban residence. Census
2011 covered 35 States/Union Territories, 640 districts, 5,924 sub-districts, 7,935 Towns and
6,40,867 Villages. The data collected has so many variables thus making it difficult to find useful
details without using Data Science Techniques. You are tasked to perform detailed EDA and identify
Optimum Principal Components that explains the most variance in data. Use Sklearn only.
Note: The 24 variables given in the Rubric is just for performing EDA. You will have to
consider the entire dataset, including all the variables for performing PCA. Data file -
PCA India Data Census.xlsx
Q.1) Read the data and perform basic checks like checking head, info, summary, nulls, and
duplicates, etc.
Ans: PCA India Data Census.xlsx dataset is loaded into the data frame.
Data Frame printing rows with Head (Prints top 5 rows) function as below:
13
Data Frame printing rows with Tail (Prints last 5 rows) function as below :
(640, 61)
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 640 entries, 0 to 639
Data columns (total 61 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 State Code 640 non-null int64
1 Dist.Code 640 non-null int64
2 State 640 non-null object
3 Area Name 640 non-null object
4 No_HH 640 non-null int64
5 TOT_M 640 non-null int64
6 TOT_F 640 non-null int64
7 M_06 640 non-null int64
8 F_06 640 non-null int64
9 M_SC 640 non-null int64
10 F_SC 640 non-null int64
11 M_ST 640 non-null int64
12 F_ST 640 non-null int64
13 M_LIT 640 non-null int64
14 F_LIT 640 non-null int64
15 M_ILL 640 non-null int64
16 F_ILL 640 non-null int64
17 TOT_WORK_M 640 non-null int64
18 TOT_WORK_F 640 non-null int64
19 MAINWORK_M 640 non-null int64
20 MAINWORK_F 640 non-null int64
21 MAIN_CL_M 640 non-null int64
22 MAIN_CL_F 640 non-null int64
23 MAIN_AL_M 640 non-null int64
24 MAIN_AL_F 640 non-null int64
25 MAIN_HH_M 640 non-null int64
26 MAIN_HH_F 640 non-null int64
27 MAIN_OT_M 640 non-null int64
28 MAIN_OT_F 640 non-null int64
29 MARGWORK_M 640 non-null int64 14
14
30 MARGWORK_F 640 non-null int64
31 MARG_CL_M 640 non-null int64
32 MARG_CL_F 640 non-null int64
33 MARG_AL_M 640 non-null int64
34 MARG_AL_F 640 non-null int64
35 MARG_HH_M 640 non-null int64
36 MARG_HH_F 640 non-null int64
37 MARG_OT_M 640 non-null int64
38 MARG_OT_F 640 non-null int64
39 MARGWORK_3_6_M 640 non-null int64
40 MARGWORK_3_6_F 640 non-null int64
41 MARG_CL_3_6_M 640 non-null int64
42 MARG_CL_3_6_F 640 non-null int64
43 MARG_AL_3_6_M 640 non-null int64
44 MARG_AL_3_6_F 640 non-null int64
45 MARG_HH_3_6_M 640 non-null int64
46 MARG_HH_3_6_F 640 non-null int64
47 MARG_OT_3_6_M 640 non-null int64
48 MARG_OT_3_6_F 640 non-null int64
49 MARGWORK_0_3_M 640 non-null int64
50 MARGWORK_0_3_F 640 non-null int64
51 MARG_CL_0_3_M 640 non-null int64
52 MARG_CL_0_3_F 640 non-null int64
53 MARG_AL_0_3_M 640 non-null int64
54 MARG_AL_0_3_F 640 non-null int64
55 MARG_HH_0_3_M 640 non-null int64
56 MARG_HH_0_3_F 640 non-null int64
57 MARG_OT_0_3_M 640 non-null int64
58 MARG_OT_0_3_F 640 non-null int64
59 NON_WORK_M 640 non-null int64
60 NON_WORK_F 640 non-null int64
dtypes: int64(59), object(2)
memory usage: 305.1+ KB
There are ‘No duplicate values’ and ‘No Null values’ in to the dataset.
15
Q.2) Perform detailed Exploratory analysis by creating certain questions like
(i) Which state has highest gender ratio and which has the lowest?
(ii) Which district has the highest & lowest gender ratio? (Example Questions). Pick 5
variables out of the given 24 variables below for EDA: No_HH, TOT_M, TOT_F,
M_06, F_06, M_SC, F_SC, M_ST, F_ST, M_LIT, F_LIT, M_ILL, F_ILL, TOT_WORK_M,
TOT_WORK_F, MAINWORK_M, MAINWORK_F, MAIN_CL_M, MAIN_CL_F,
MAIN_AL_M, MAIN_AL_F, MAIN_HH_M, MAIN_HH_F, MAIN_OT_M, MAIN_OT_F
Ans:
By Above Barplot (State, TOT_M), We can get following Questions such as:
16
By Above Barplot (State, M_LIT), We can get following Questions such as:
By Above Barplot (State, F_LIT), We can get following Questions such as:
By Above Barplot (State, TOT_WORK_M), We can get following Questions such as:
17
Q.3) We choose not to treat outliers for this case. Do you think that treating outliers for this
case is necessary?
Ans: Yes. Treating Outliers in this is necessary. I have dropped categorical features from Dataset to
verify and treat Outliers. I have verified Outliers with the help of Boxplot.
After applying remove_outlier function on the Data2_Test dataset we get following output.
18
19
Q.4) Scale the Data using z-score method. Does scaling have any impact on outliers? Compare
boxplots before and after scaling and comment.
Ans: I have already treated Outliers in Q.3) only. But still I applied the Z-score for the scaling of the
Dataset. Please find below outputs by Boxplot and Describe function for ‘Before’ and ‘After’
20
After Applying Z-Score
21
Q.5) Perform all the required steps for PCA (use sklearn only) Create the covariance Matrix Get
eigen values and eigen vector.
22
Check the eigen values
Q.6) Identify the optimum number of PCs (for this project, take at least 90% explained
variance). Show Scree plot.
23
Q.7) Compare PCs with Actual Columns and identify which is explaining most variance. Write
inferences about all the Principal components in terms of actual variables.
Ans:
24
Q.8) Write linear equation for first PC.
Ans:
PC1=a1x1*a2x2*……anxn.
Thank You
25