Advanced Statistics
Advanced Statistics
DIPTI PATIL
Problem 1A:
Salary is hypothesized to depend on educational qualification and occupation. To understand
the dependency, the salaries of 40 individuals [SalaryData.csv] are collected and each
person’s educational qualification and occupation are noted. Educational qualification is at
three levels, High school graduate, Bachelor, and Doctorate. Occupation is at four levels,
Administrative and clerical, Sales, Professional or specialty, and Executive or managerial. A
different number of observations are in each level of education – occupation combination.
[Assume that the data follows a normal distribution. In reality, the normality assumption may
not always hold if the sample size is small.]
1. State the null and the alternate hypothesis for conducting one-way ANOVA for both
Education and Occupation individually.
2. Perform a one-way ANOVA on Salary with respect to Education. State whether the
null hypothesis is accepted or rejected based on the ANOVA results.
From the above table, we can see that P value < 0.05, therefore, we reject the Null
Hypothesis.
3. Perform a one-way ANOVA on Salary with respect to Occupation. State whether the
null hypothesis is accepted or rejected based on the ANOVA results.
From the above table, we can see that P value > 0.05, therefore, we fail to reject the Null
Hypothesis.
4. If the null hypothesis is rejected in either (2) or in (3), find out which class means are
significantly different. Interpret the result.
1. What is the interaction between two treatments? Analyze the effects of one variable
on the other (Education and Occupation) with the help of an interaction plot.[hint:
use the ‘pointplot’ function from the ‘seaborn’ function]
2. Perform a two-way ANOVA based on Salary with respect to both Education and
Occupation (along with their interaction Education*Occupation). State the null and
alternative hypotheses and state your results. How will you interpret this result?
Hypothesis for conducting two-way ANOVA for Education & Occupation combined wrt
salary
Ho = For all levels of Education & Occupation, average salary is equal
Ha = For at least one Education & Occupation, average salary is not equal
Assumption: Confidence Level(P value) = 0.05
3. Explain the business implications of performing ANOVA for this particular case study.
• The average salary highly depends on the Level of Education and least dependent on
the Level of Occupation.
• Even though from the Point plot we can see the Occupation does effect the average
salary earned, as Doctorate and Bachelors show more average salary earned than the
HS-grad. We may need to capture more number of observations to get this result
from ANOVA test.
• The Occupation Prof-Speciality have a very high value for Doctorate than Bachelors &
HS-grad. This means, less number of candidates are available this position among
Doctorate or the work requirement is very high, hence highest average salary among
Doctorate.
• For Sales and clerical amongst Occupation, average salary for Bachelors & Doctorate
is almost similar. This means that higher qualification such as Doctorate is not
preferred or required for these jobs.
• The ANOVA test suggests that the combinations of ‘Education & Occupation’ levels
does affect the average salary earned by candidate.
Problem 2:
The dataset Education - Post 12th Standard.csv contains information on various colleges. You
are expected to do a Principal Component Analysis for this case study according to the
instructions given. The data dictionary of the 'Education - Post 12th Standard.csv' can be
found in the following file: Data Dictionary.xlsx.
Q1. Perform Exploratory Data Analysis [both univariate and multivariate analysis to be
performed]. What insight do you draw from the EDA?
• Describe shows most of the variables (Apps, Accept, Enroll, Top10perc, F_undergrad,
P_undergrad, Personal, Expend) are right skewed as Mean value > Median value.
• For other variables Mean value is almost equal to Median value, which shows normal
distribution. We will confirm same by Distplot in Univariate Analysis.
• From Distplot we see that variables like ‘PhD’ & ‘Terminal’ are somewhat left skewed.
• It is seem from Box Plot above that variables like Apps, Accept, Enroll, F_undergrad,
P_undergrad, Personal, Expend have lots of Outliers present in data.
• From the above Correlation Matrix we can see that, there is high corelation between:
o Application and Application Accepted.
o Enrolment and Application Accepted.
o Full time undergraduate and Enrolment.
o Top 10 percentage & Top 25 percentage.
o Top 10 percentage & Expenditure on them.
• Also, from above heatmap we can see the Correlation between variables, which
causes Multicollinearity. Multicollinearity causes problem which undermines the
statistical significance or independent variable.
Q2. Is scaling necessary for PCA in this case? Give justification and perform scaling.
Yes, scaling is absolutely necessary for PCA in this case. The main objective of scaling is to
normalize a data and bring all the variable data on the same scale for further comparison. It is
a step of data pre processing which is applied to independent variables or features of data.
For example, in the given data, some variables are in ten thousands, some in hundreds, and
some in percentages, so we cannot compare the data. As large values will overweight the
smaller values so we perform scaling.
Q3. Comment on the comparison between the covariance and the correlation matrices from this
data [on scaled data].
After Scaling:
Insights:
• The scaled dataset has all similar max values and comparable min values.
• The mean value for each of the variables are comparable to 0 and standard deviation
1.
• The range of each variables hence are now standardized and are all unit-less
quantities.
Q5. Extract the eigenvalues and eigenvectors.[print both]
Kindly refer jyupter file for all the Eigen Vectors value.
• Eigenvalue and Eigenmatrix are mainly used to capture key information that stored in
a large matrix.
• It provides summary of large matrix.
• Performing computation on large matrix is slow and require more memory and CPU,
eigenvectors and eigenvalues can improve the efficiency in computationally intensive
task by reducing dimensions after ensuring of the key information is maintained.
Q6. Perform PCA and export the data of the Principal Component (eigenvectors) into a data
frame with the original features
We load these components into a dataframe along with the list of columns we had earlier
considered in scaled data. Below is the representative screenshot of in which the Portion of
Variable is extracted by PCs.
Q7. Write down the explicit form of the first PC (in terms of the eigenvectors. Use values with
two places of decimals only). [hint: write the linear equation of PC in terms of eigenvectors
and corresponding features]
Explicit form of the first PC
Q8. Consider the cumulative values of the eigenvalues. How does it help you to decide on the
optimum number of principal components? What do the eigenvectors indicate?
• As we see from above picture, the first eigen value captures 33.26% of the
information represented by all the 17 variables
• Similarly, the 1st and 2nd eigen value together captures 62.02% of the information
and so on
• It is observed that by considering up to 8th eigen value, the total variance captured is
up to 90.79%
• Hence in this case,upto 8th eigen value captures 90.79% information present in 17
numeric features in the original dataset.
• Eigen vectors indicate the direction of the principal components, we can multiply the
original data by the eigenvectors to re-orient our data onto the new axes.
Above picture is the Scree Plot:
Yaxis represents variation captured from features/Dimensions/variable
Xaxis represents Principal Components.
Q9. Explain the business implication of using the Principal Component Analysis for this case
study. How may PCs help in the further analysis? [Hint: Write Interpretations of the Principal
Components Obtained]
• There are 17 variables in the given data set. After doing PCA, we deduce that we can
reduce the number of variables from 17 to 8. As 8 PCs capture almost 90.33% of the
information present in the given dataset.
• Also, we can do the further analysis on 5 PCs which captures 79.66% of information of
17 features in the dataset if it is fine with the Business requirement.