Ch1 Data Science
Ch1 Data Science
Understanding Data
When we are ready to learn Data Science, It is obvious that we should know what actually a
data is. We must also know the various sources of data.
Definitions:
First, let’s look at what a few trusted sources consider data to be.
First up, we’ll look at the Cambridge English Dictionary, which states that data is:
In both the definition, we got some value that maybe facts(qualitative) or maybe
numbers(quantitative).
So for more simplicity, Data is raw fact which can be represented in a meaningful way to form
some information. Data mainly comes in the following forms: text, graphics, numerical, voice,
and video.
In real time scenario, the data we found can be broadly categorize in the the basis of its form:
1. Structured data – Structured data is data whose elements are addressable for effective
analysis. It has been organized into a formatted repository that is typically a database. It
concerns all data which can be stored in database SQL in a table with rows and columns.
They have relational keys and can easily be mapped into pre-designed fields. Today, those
data are most processed in the development and simplest way to manage
information. Example: Relational data.
And this brings us to the second quality of big data: velocity. Data is being generated and
collected faster than ever before. In our YouTube example, new data is coming at you every
minute! In a completely different example, say you have a question about shipping times or
routes. Well, most transport trucks have real time GPS data available - you could in real time
analyse the trucks movements… if you have the tools and skills to do so!
The third quality of big data is variety. In the examples I’ve mentioned so far, you have
different types of data available to you. In the YouTube example, you could be analysing video
or audio, which is a very unstructured data set, or you could have a database of video lengths,
views or comments, which is a much more structured dataset to analyse.
A summary of three qualities that characterize big data
Data Science
There are no fixed definition for data Science but we can answer what data science is. Here we
will discuss some of the common understanding of Data science definition. Data science is all
about:
We can form some definition using the important tasks related to Data Science:
Data Science is an interdisciplinary field that allows you to extract knowledge from structured,
semistructured, or unstructured data. Data science enables you to translate a business
problem into a research project and then translate it back into a practical solution.
Data science is a deep study of the massive amount of data, which involves extracting
meaningful insights from structured, semi structured, and unstructured data that is processed
using the scientific method, different technologies, and algorithms.
It is a multidisciplinary field that uses tools and techniques to manipulate the data so that you
can find something new and meaningful.
“who combines the skills of software programmer, statistician and storyteller slash artist to
extract the nuggets of gold hidden under mountains of data”
Data is the oil for today's world. With the right tools, technologies, algorithms, we can
use data and convert it into a distinctive business advantage
Data Science can help you to detect fraud using advanced machine learning algorithms
It helps you to prevent any significant monetary losses
Allows to build intelligence ability in machines
You can perform sentiment analysis to gauge customer brand loyalty
It enables you to take better and faster decisions
Helps you to recommend the right product to the right customer to enhance your
business
Domain Expert
Data
Statistics Engineering
Data
Science
Statistics: Statistics is the most critical unit of Data Science basics. It is the method or science
of collecting and analyzing numerical data in large quantities to get useful insights.
Visualization: Visualization technique helps you to access huge amounts of data in easy to
understand and digestible visuals.
Machine Learning: Machine Learning explores the building and study of algorithms which learn
to make predictions about unforeseen/future data.
Deep Learning: Deep Learning method is new machine learning research where the algorithm
selects the analysis model to follow.
Data Science Process
The Data Science activity starts with the inquisitiveness of the Data Scientist. A Data scientist must form a ques
The next task is to find related data to solve the question in a convenient and significant way. The data may be
Finally, the Model. Using statistical, Mathematics, and computer science, a good model to be prepared to anal
Figure: Data Analysis tasks
1. Discovery:
Discovery step involves acquiring data from all the identified internal & external sources which
helps you to answer the business question.The data can be:
2. Preparation:
Data can have lots of inconsistencies like missing value, blank columns, incorrect data
format which needs to be cleaned. You need to process, explore, and condition data
before modeling. The cleaner your data, the better are your predictions. In this phase, we
need to perform the following tasks:
Data cleaning
Data Reduction
Data integration
Data transformation
3. Model Planning:
In this stage, you need to determine the method and technique to draw the relation between
input variables. Planning for a model is performed by using different statistical formulas
and visualization tools. SQL analysis services, R, Python, and SAS/access are some of the
tools used for this purpose.
4. Model Building:
In this step, the actual model building process starts. Here, Data scientist distributes datasets
for training and testing. Techniques like association, classification, and clustering are applied to
the training data set. The model once prepared is tested against the "testing" dataset.
5. Operationalize:
In this stage, you deliver the final baselined model with reports, code, and technical documents.
Model is deployed into a real-time production environment after thorough testing.
6. Communicate Results
In this stage, the key findings are communicated to all stakeholders. This helps you to decide if
the results of the project are a success or a failure based on the inputs from the model.
1. Data acquisition: The primary step in the life cycle of any data science project is to acquire the
right data from multiple sources. Data acquisition involves acquiring data from different
internal and external sources that can help answer business questions. Data can be extracted
from various sources, such as logs from web servers, social media data, online repositories,
or databases.
2. Data preparation: Often referred to as data cleaning or data wrangling, it is a critical step in
the life cycle. The data collected from different sources is frequently messy and is typically
missing various values. Therefore, it is crucial to clean this data to derive value from it.
3. Data exploration: After cleaning the data, you can perform hypothesis testing and visualize
the data to understand the data better. Data exploration is sometimes called data mining. It is
used to identify patterns in your data set and find important potential features with statistical
analysis.
4. Predictive modeling: To train your machine to make predictions, you need to build predictive
models. For this, you have to choose the right algorithm on which the machine is to be
trained. Historical data is then split into training and validation sets. The model is trained using
the training set. The trained model is validated using the validation dataset, and the model is
then evaluated for accuracy and efficiency.
5. Model interpretation and deployment: After a rigorous evaluation of the model, you can
deploy into a production-like environment for final user acceptance. You'll want to present
your model to a non-technical person and convey the actionable insights derived from the
data.
Data Science Jobs Roles
Most prominent Data Scientist job titles are:
Data Scientist
Data Engineer
Data Analyst
Statistician
Data Architect
Data Admin
Business Analyst
Data/Analytics Manager
Now in this Data Science Tutorial, let's learn what each role entails in detail:
Data Scientist:
Role: A Data Scientist is a professional who manages enormous amounts of data to come up
with compelling business visions by using various tools, techniques, methodologies, algorithms,
etc.
Data Engineer:
Role: The role of data engineer is of working with large amounts of data. He develops,
constructs, tests, and maintains architectures like large scale processing system and
databases.
Languages: SQL, Hive, R, SAS, Matlab, Python, Java, Ruby, C + +, and Perl
Data Analyst:
Role: A data analyst is responsible for mining vast amounts of data. He or she will look for
relationships, patterns, trends in data. Later he or she will deliver compelling reporting and
visualization for analyzing the data to take the most viable business decisions.
Statistician:
Role: The statistician collects, analyses, understand qualitative and quantitative data by using
statistical theories and methods.
Data Administrator:
Role: Data admin should ensure that the database is accessible to all relevant users. He also
makes sure that it is performing correctly and is being kept safe from hacking.
Business Analyst:
Internet Search: Google search use Data science technology to search a specific result within
a fraction of a second
Image & Speech Recognition: Speech recognizes system like Siri, Google assistant, Alexa
runs on the technique of Data science. Moreover, Facebook recognizes your friend when you
upload a photo with them, with the help of Data Science.
Gaming world: EA Sports, Sony, Nintendo, are using Data science technology. This enhances
your gaming experience. Games are now developed using Machine Learning technique. It can
update itself when you move to higher levels.
Online Price Comparison: PriceRunner, Junglee, Shopzilla work on the Data science
mechanism. Here, data is fetched from the relevant websites using APIs.
Here the problem statement (question) is pretty clear “Which Olympic Sport Is Hardest On Its
Goalies?” Now they collected summer Olympics data from 2012, Soccer world cup data from
2014 and 2015 for men and women. Data source: SPORTS-REFERENCE.COM,
BARRIESVIEW.COM.
Women’s soccer goalkeepers have some of the highest save percentages of any Olympic sport:
Soccer and field hockey keepers have the “easiest” jobs, generally saving around 70 percent of
shots. Soccer goalies see far fewer attempts, though, leading to much more variance
throughout a tournament (that low point for men’s soccer is English keeper Joe Hart, who saw
only five shots at the 2014 World Cup but let in four). Water polo goalkeepers let in shots at a
much greater clip, especially on the women’s side, but handball goalies have it worst of all,
letting in around seven in 10 shots. But the saves, when they come, can be spectacular:
Example2:
As like most of the Indian we are a more or less interested about cricket. Here let us know, can
we get answer of the following question applyiing simple Data Science techniques:
Collect the commentary of the last 4 years of the T20 matches played by India. Download a sample dataset
“https://drive.google.com/file/d/1gkQ_IoqJBGIpqdWqBTdEc9umh-Tal8Rh/view”.
It’s time to analyze the commentary and find some appealing insights. Let’s do
it!
Reference
1. https://www.javatpoint.com/data-science
2. https://www.tutorialspoint.com/python_data_science/index.htm
3. https://www.guru99.com/data-science-tutorial.html
4. https://www.analyticsvidhya.com/
5. Python for Data Analysis, by Wes McKinney, O’Reilly 2012, First Edition.