Data Science and Analytics For Smes: Consulting, Tools, Practical Use Cases Afolabi Ibukun Tolulope
Data Science and Analytics For Smes: Consulting, Tools, Practical Use Cases Afolabi Ibukun Tolulope
and Analytics
for SMEs
Consulting, Tools,
Practical Use Cases
―
Afolabi Ibukun Tolulope
Data Science and
Analytics for SMEs
Consulting, Tools,
Practical Use Cases
Afolabi Ibukun Tolulope
Data Science and Analytics for SMEs: Consulting, Tools, Practical
Use Cases
Afolabi Ibukun Tolulope
London, United Kingdom
Acknowledgments�����������������������������������������������������������������������������xiii
Preface�����������������������������������������������������������������������������������������������xv
Chapter 1: Introduction������������������������������������������������������������������������1
1.1 Data Science���������������������������������������������������������������������������������������������������1
1.2 Data Science for Business������������������������������������������������������������������������������2
1.3 Business Analytics Journey����������������������������������������������������������������������������4
Events in Real Life and Description�����������������������������������������������������������������5
Capturing the Data������������������������������������������������������������������������������������������9
Accessible Location and Storage������������������������������������������������������������������10
Extracting Data for Analysis��������������������������������������������������������������������������10
Data Analytics������������������������������������������������������������������������������������������������12
Summarize and Interpret Results������������������������������������������������������������������14
Presentation��������������������������������������������������������������������������������������������������15
Recommendations, Strategies, and Plan�������������������������������������������������������15
Implementation���������������������������������������������������������������������������������������������16
1.4 Small and Medium Enterprises (SME)�����������������������������������������������������������16
1.5 Business Analytics in Small Business�����������������������������������������������������������17
1.6 Types of Analytics Problems in SME�������������������������������������������������������������19
iii
Table of Contents
iv
Table of Contents
v
Table of Contents
vi
Table of Contents
Data Files������������������������������������������������������������������������������������������323
Index�������������������������������������������������������������������������������������������������327
vii
About the Author
Afolabi Ibukun is a Data Scientist and is
currently an Assistant Professor in Computer
Science at Northeastern University London.
She holds a B.Sc in Engineering Physics, an
M.Sc and Ph.D in Computer Science. Afolabi
Ibukun has over 15 years working experience
in Computer Science research, teaching and
mentoring. Her specific areas of interest
are Data & Text Mining, Programming and
Business Analytics. She has supervised several
undergraduate and postgraduate students
and published several articles in international journals and conferences.
Afolabi Ibukun is also a Data Science Nigeria Mentor (https://www.
datasciencenigeria.org/mentors/) and currently runs a Business
Analytics Consulting and Training firm named I&F Networks Solutions.”
08021247616
[email protected]
linkedin.com/in/afolabi-ibukun-051777a6
github.com/ibkAfolabi
ix
About the Technical Reviewer
Hitesh Hinduja is an ardent Artificial Intelligence (AI) and Data Platforms
enthusiast currently working as Senior Manager in Data Platforms (Azure)
and AI at Microsoft. He worked as a Senior Manager in AI at Ola Electric,
where he led a team of 20+ people in the areas of machine learning,
statistics, computer vision, deep learning, natural language processing,
and reinforcement learning. He has filed 14+ patents in India and the
United States and has numerous research publications under his name.
Hitesh had been associated in research roles at India’s top B-schools –
Indian School of Business, Hyderabad, and the Indian Institute of
Management, Ahmedabad. He is also actively involved in training and
mentoring and has been invited as a guest speaker by various corporates
and associations across the globe.
Hitesh is an avid learner and enjoys reading books in his free time.
xi
Acknowledgments
First of all, I would like to acknowledge God almighty for making it
possible for me to write this book. I would also like to thank my husband
Oluwafemi Afolabi for his support and encouragement that has made
this book a reality. I deeply appreciate Prof. Olufunke Oladipupo and Dr.
Joke Badejo who have taught me a lot, both as a data scientist and in other
aspects of life. Lastly, I would like to say thank you to Timileyin Owoseni
and Christabel Uzor, my M.Sc. students who also helped with the book. I
will not forget my wonderful students that I have been fortunate to teach
and advise, Obinna Okorie, Temi Oyedepo, and many others; I learned a
lot from them. I would like to appreciate afrimash.com for the opportunity
to learn practical data science consulting.
xiii
Preface
This book is written from the perspective of offering a Business Analytics
service as a product. It helps to understand how to package your analytics
solution as a product that can be offered as a consulting service. Some
of these products are customer loyalty, market segmentation, sales and
revenue increase, etc. It is also particularly focused on small businesses
and their peculiarity in analytics. Understanding the contents of the book
will help anyone interested in applying data analytics to make a difference
in small businesses achieve such, starting from the beginner’s level. It
uses a do-it-yourself approach to analytics, and the tools used are easily
available online and are nonprogramming based.
The book teaches the tricks and techniques of Business Analytics
consulting for small businesses. In particular, readers will be able to create
and measure the success of their analytics project. The book also provides
a career guide and helps to jump-start the world of Business Analytics
consulting career. The approach in the book is to focus on popular
problems in the small and medium business world that have data science
solutions and then introduce the technique and how to use it to solve the
problems. Readers will not only learn the fundamental techniques used in
solving these problems, but they will also experience how to use them in
practical use cases and problem scenarios. The techniques are taught in a
simple way, but the book is supported with a lot of reference and resource
material that can help build more mastery on the techniques.
The book is divided into four major parts. Part 1 (Chapters 1 and 2)
explains the fundamental concepts explored in this book, such as data
science, data science for business (Business Analytics), and what it takes
to carry out any analytics project both generally and specifically for a
xv
Preface
small business. In this part, we also explore issues around data and how
to manage and prepare it for the analytics project with practical examples.
Part 2 (Chapters 3 and 4) focuses on analytics consulting and explains how
to navigate your way through to becoming successful in the data analytics
consulting space. It also gives a detail of the phases involved in Business
Analytics consulting. Part 3 (Chapters 5–8) is focused on the data mining
techniques common with small businesses, and this is expressed in an
approach that first explains the basic concepts of these techniques in a
simple way and then uses a real small business problem scenario for the
practical application. This part is practical oriented and based on case
study problems experienced by small businesses. In this book, we will
explore five major practical business problem scenarios and several small
business problems for illustration. These are covered in Chapters 5–8. The
techniques used demonstrate how to solve these problems. It is important
to say here that despite using a particular problem as a case study, it is
not only in this situation that the approach can be deployed, but it can
be used in other similar problem scenarios. The techniques selected
are based on their popularity in practice, and they fall under the broad
classification of prediction (predicting numerical outcome), classification
(predicting categorical outcome), and descriptive analytics. Finally, Part 4
brings the consulting principles to life by using an SME case study to
model the already explained consulting phases in Part 2 and adopting the
appropriate techniques among the ones explained in Part 3. Although each
chapter stands alone, we advise that you read Part 1 before proceeding to
Part 3, and Part 2 before proceeding to Part 4.
The tools used for the practical examples are RapidMiner Studio and
Gephi. The book is written such that all the RapidMiner Studio and Gephi
screenshots are included with details of how to run them. GitHub will also
be used to store the practical project.
This book goes beyond explaining the techniques to giving an
experience of applying the recommendations from the modeling to get
results. In particular, we use a sample of business scenarios experienced
xvi
Preface
in the past for the use cases. The book is also supported with a real-life
business group on Telegram (https://t.me/+kSSQjNhhz6plZTk0) where
we harvest business problems and encourage readers to be a part of
solving the problems. We have in this book real-life business case studies
(from the Telegram group) that can be used as a reference. The book
also comes with links to YouTube videos that help to explain some of the
concepts better.
This book, together with the solutions to the exercises and more
application scenarios and data science techniques, is available as an
online course and training; you can visit datasciencenaija.com for more
details.
We appreciate reader feedback. We would like to know what
you think about this book, good or bad. To give us general feedback,
simply send an email to [email protected]. Also, the data
and more information about the book can also be obtained at
www.datasciencenaija.com/book.
xvii
CHAPTER 1
Introduction
In this chapter, we introduce data science generally and narrow it down to
data science for business which is also referred to as Business Analytics.
We then give a detailed explanation of the process involved in Business
Analytics in the form of the Business Analytics journey. In this journey,
we explain what it takes from start to finish to carry out an analytics
project in the business world, focusing on small business consulting, even
though the process is generic to all types of business, small or large. We
also give a description of what small business refers to in this book and
the peculiarities of navigating an analytics project in such a terrain. To
conclude the chapter, we talk about the types of analytics problems that
are common to small businesses and the tools available to solve these
problems given the budget situation of small businesses when it comes to
analytics projects.
1.1 Data Science
In simple terms, data science refers to the ability to take data, generate
an understanding from the data, process the data, extract value from the
data, visualize the data, and present it in such a way that decisions can be
made from this presentation. Data science is described as the process of
extracting knowledge from huge amounts of data. It is an intersection of
Mathematics, Statistics, Visualization, and Artificial Intelligence. Artificial
Intelligence is a superset of machine learning, which is a superset of deep
learning.
© Afolabi Ibukun Tolulope 2022 1
A. I. Tolulope, Data Science and Analytics for SMEs,
https://doi.org/10.1007/978-1-4842-8670-8_1
Chapter 1 Introduction
2
Chapter 1 Introduction
3
Chapter 1 Introduction
4
Chapter 1 Introduction
5
Chapter 1 Introduction
domain understanding of the business terrain and also the typical success
strategies of the business since the analytics recommendation will be used
to improve this purpose. Depending on the kind of business in question,
there are several means to discover their particular analytics needs. Some
of these means include using interviews, social media (text mining), focus
groups, and so on. It is also important to note that these needs should be
focused on the goal of specific business. Generally, Business Analytics
helps to boost business processes, reduce business cost, drive business
strategy, and monitor and improve business financial performance. Uber,
for example, enhanced its Customer Obsession Ticket Assistant (COTA) in
early 2018 to improve the speed and accuracy while responding to support
tickets using predictive analytics. This is a great example of a company
that has implemented Business Analytics. Many companies now employ
predictive analytics to anticipate maintenance and operational concerns
before they become major problems; this is according to a KPMG analysis
on emerging infrastructure trends.12
In this first stage of Business Analytics journey, we develop an
understanding of the purpose of the data mining project or Business
Analytics project particularly to know if it is a one-shot effort to answer
a question or questions or if it is an ongoing procedure. After this, we
then highlight the decisions we want to make and determine or suggest
the analysis output that will help to make the decision. It will help to
have a design that is a conceptual design of the analysis that will create
the output.
The conceptual business model is used to understand data in the
business domain and how the business actually works. It particularly helps
to understand the importance of context in solving analytical problems.
It also helps us to understand how important elements relate together.
Since Business Analytics solves problems, that is, it answers questions, it is
therefore important to know at this stage what questions are worth asking
in the course of the analytics project. This conceptual business model will
help us to do that.
6
Chapter 1 Introduction
7
Chapter 1 Introduction
prospects and many more. In the second step of the conceptual model which
deals with how Alegria contacts the prospects in step one, you want to store
information on the time of contact, location, etc. For each of the remaining
steps, such as negotiating for collecting of waste, collecting the waste,
recycling the waste, and making revenue from the waste, you will have to
bring out the data attributes that could be captured. This is how you are
able to use the conceptual business model to have a complete understanding
of the data in the business domain.
8
Chapter 1 Introduction
Capturing the Data
After successfully identifying what will form the basis of your data and
the type of data using the conceptual business models, the next thing is
to capture the data and store it in a form that can be used for analytics.
Capturing the data deals with the mechanism that captures the physical
or digital representation of that real-world phenomenon and puts it
somewhere. In this stage, there is a need to identify what is to be stored
or retrieved (i.e., if the data has been stored before). Sometimes, it might
involve feature selection and representation (which will be explained
in Section 2.1). The peculiarity of small businesses when it comes to
capturing the data is that there is a lot of data literacy needed to be done
due to the fact that most of these businesses are characterized with lack
of an enterprise management system that manages their data; also, they
particularly don’t know what the data needs of their business are. There
is therefore a need for data integrity evangelism so as to get the best out of
the analytics project. For this reason, the task of capturing data for analysis
is a little challenging; most of the data is in the form of excel sheets that
might not be particularly analytics ready. There are several data cleaning
and preprocessing functions in Microsoft Excel which can be used. Also,
there might be a need to use more interactive data visualization tools like
Tableau (www.tableau.com/) and Power BI (www.microsoft.com/en-us/
download/details.aspx?id=58494) for this purpose of capturing and
preparing the data for analysis. There are other tools available for free or
for a token on the Internet that can help capture small business processes
which will help make available data at the back end that can jump-
start the analytics process no matter how small the business is. Google
Analytics, for example, can help small businesses, particularly ecommerce
businesses, to create actionable strategies from automatically generated
data from the ecommerce platform. This data can be used to measure
marketing campaign performance, return on investment, and many
more (analytics.google.com). For businesses that have been able to put
9
Chapter 1 Introduction
the correct data gathering and governance procedures in place, the data
capturing stage is much more easy to navigate as all that is left is to check
the appropriateness of the data for the particular analytics goal.
10
Chapter 1 Introduction
11
Chapter 1 Introduction
earned into a range between N50,000 and N100,000. There are also situations
where you need to create new variables from what you have, for example,
creating a variable true or false capturing if a customer showed up or not
from the purchase data on customers. It is also important for us to know
what each variable means and whether to include it in the analysis or not.
Data Analytics
In this stage, we determine the data mining task to be carried out. This
ranges from descriptive analytics, diagnostic analytics, predictive analytics,
prescriptive analytics, and cognitive analytics. Sometimes, you might
need to combine these tasks or use them as a prerequisite to each other.
It is important to note here that one method isn’t more important than
another; it all depends on the end goal or the purpose of the Business
Analytics project.
Descriptive analytics helps to describe what things look like now or what
happened in the past. In descriptive analytics, we use available information
to better understand the business environment and apply the knowledge
along with business acumen to make better decisions. Descriptive analytics
makes use of simple aggregations, cross-tabulations, and simple statistics
like means, medians, standard deviations, and distributions, for example,
histograms. Some advanced descriptive analytics includes associations or
clustering algorithms. An example of the question that descriptive analytics
helps to answer is: What type of customers are renting our equipment?
Sometimes, we use descriptive analytics to identify the link between
two variables; this field of study is known as association rule mining. A
supermarket, for example, might collect information on customer purchase
behavior. The supermarket can use this information to discover which
products are usually purchased together and sell them accordingly. This
is known as market basket analysis. In addition, clustering can be used to
identify groupings and structures in data that are “similar” in some sense
without relying on existing data structures.
12
Chapter 1 Introduction
13
Chapter 1 Introduction
14
Chapter 1 Introduction
Presentation
In presenting the results, you need a clear understanding of how the
business works; then with the help of charts, graphs, or tables, you will
be able to present what is going on. You will also need to supplement the
figure with a short narrative explaining what they mean. Simplicity is the
word here, regardless of how complex the analytics procedure is; at this
point, you will need to reduce the details and sieve out all your analysis
into a few key points. At this point, you want to check if you have been
able to successfully answer all the questions you set out to answer at the
beginning of the analytics project. It’s interesting to know that sometimes
you might even discover that you were asking or pursuing the wrong goal
or question to begin with. Do not forget that as a business analyst, solid
presentation skills are required to be able to communicate your analytics
results efficiently. Depending on the tools you are using to make your
presentation (Tableau dashboard, Microsoft PowerPoint, etc.), you will
need to develop mastery in its use and ability to use it to communicate
effectively.
15
Chapter 1 Introduction
Implementation
In the implementation stage, the result of the analysis is put to test live.
The nature of this implementation will be determined by your strategy
in the previous step. In this stage, you will have to monitor the success
metrics you highlighted related to each goal in the first step of the
analytics journey. Depending on whether the results are successful or
not, you might need to iterate the process based on lessons learned in
the analytics journey. This entire process is iterative in nature. Also, after
implementation, there is a need to use feedback from the users to iterate
the analytics process for better analytics results and interpretation of
results. The stage to be repeated in this iteration is determined by the
nature of the feedback. There are situations where there will be a need to
revisit the system data capture stage or even the data analysis stage.
16
Chapter 1 Introduction
other things. According to the World Bank, MSMEs account for over
90% of all enterprises and more than 50% of all jobs worldwide. In
emerging economies, formal SMEs can account for up to 40% of national
income (GDP).
According to the Bank of Industry (BOI) in a report by PwC3, SMEs can
be defined using Table 1-1.
In the same report, the most pressing problem currently faced by SMEs
is obtaining finance, followed by finding customers. For SMEs, the task of
finding customers is mostly captured by marketing analytics, analytics in
customer relationship management, and so on, which takes the lion share
of Business Analytics.
17
Chapter 1 Introduction
enterprises, the variety and velocity are often the same. The good thing
is that, so far, they are able to use the data quickly and efficiently and can
compete with larger competitors in the same space. Interestingly, one of
the edges that small businesses have over larger ones when it comes to
maximizing data science is that they do not need a large data science team
to get value out of data.
The major problems of small and medium businesses when it comes
to applying data science to improve revenue include the following:
The amount of data collected: Due to technology advancement, there
is an enormous amount of data available in small businesses, but these
businesses lack a data system that efficiently collects and organizes
information.
Data integration: For analysis to be complete and accurate, there is a
need to bring together data across multiple, disjointed sources; currently,
this is done manually and can be time-consuming and cumbersome.
Data that lacks quality and integrity: Garbage in, garbage out. There
are no accurate insights that we can get from data that is full of errors
and does not reflect the whole problem scenario, sometimes referred to
as asymmetrical data. For more details of how SMEs can benefit from
Business Analytics, the challenges, and solution, this can be found in
Coleman et al. (2008).6
The major solution to some of the problems stated earlier is to
implement a data governance policy for the business no matter how
small. Data governance is a set of procedures, responsibilities, policies,
standards, and measurements that ensure that information is used
effectively and efficiently to help an organization achieve its objectives.7
Small businesses need to realize that the earlier they have this in place, the
better they are prepared for the future that is here already.
18
Chapter 1 Introduction
19
Chapter 1 Introduction
20
Chapter 1 Introduction
21
Chapter 1 Introduction
the fundamental concepts explored in this book, such as data science, data
science for business (Business Analytics), and what it takes to carry out any
analytics project both generally and specifically for a small business. In this
part, we also explore issues around data and how to manage and prepare it
for the analytics project with practical examples. Part 2 (Chapters 3 and 4)
focuses on analytics consulting and explains how to navigate your way
through to becoming successful in the data analytics consulting space. It
also gives a detail of the phases involved in Business Analytics consulting.
Part 3 (Chapters 5–8) is focused on the data mining techniques common
with small businesses, and this is expressed in an approach that first
explains the basic concepts of these techniques in a simple way and
then uses a real business problem scenario for the practical application.
This part is practical oriented and based on case study problems
experienced by small businesses. In this book, we will cover five different
practical business problems. These business problems are covered in
Chapters 5–8. The techniques used in the book demonstrate how to solve
these problems. It is important to say here that despite using a particular
problem as a case study, it is not only in this situation that the approach
can be deployed, but it can be used in other similar problem scenarios.
The techniques selected are based on their popularity in practice, and
they fall under the broad category of prediction (predicting numerical
outcome), classification (predicting categorical outcome), and descriptive
analytics. Finally, Part 4 brings the consulting principles into practice
by using an SME case study to model the already explained consulting
phases in Part 2 and adopting the appropriate techniques among the ones
explained in Part 3. Although each chapter stands alone, we advise that
you read Part 1 before proceeding to Part 3, and Part 2 before proceeding
to Part 4.
22
Chapter 1 Introduction
23
Chapter 1 Introduction
Using Gephi
Gephi is an open source and free software for visualization and exploration
of all kinds of graphs and networks. To download Gephi, visit https://
gephi.org/. Some installation guide can be found at https://gephi.
org/users/install/. If you have any issues installing Gephi, visit www.
youtube.com/watch?v=-JU-S5dMDVo. After installing Gephi successfully,
24
Chapter 1 Introduction
you will see the screen in Figure 1-6. A quick introduction to Gephi can
be found at https://gephi.org/tutorials/gephi-tutorial-quick_
start.pdf.
1.9 Problems
1. Design a conceptual business model for the
following business scenarios:
25
Chapter 1 Introduction
1.10 References
1. Galit Shmueli, Nitin R. Patel, & Peter C. Bruce,
Data Mining for Business Intelligence, Concepts,
Techniques and Applications in Microsoft Office
Excel with XLMiner, published by John Wiley &
Sons, Inc., Hoboken, New Jersey, 2010.
26
Chapter 1 Introduction
8. www.gartner.com/en/information-technology/
glossary/customer-analytics
9. www.techopedia.com/definition/29495/
operational-analytics
27
Chapter 1 Introduction
28
CHAPTER 2
Data for Analysis
in Small Business
In this chapter, we will look at the various sources of data generally and
in small business. This is important because the major challenge of
consulting for small business is the lack of data or quality data for analysis.
This chapter will therefore detail the sources of data for analysis explaining
first the type or form that data exists and some general ideas of how to
collect such data. It gives an overview on data quality and integrity issues
and touches on data literacy. In addition, we will explain typical data
preparation procedures for data analytics techniques. To conclude, we
look at data visualization, particularly toward preparing data for various
analytics tasks as explained in Section 1.3.
2.1 Source of Data
Data can be retrieved in different forms; the most basic category of data is
the structured, semistructured, and unstructured data.1
Structured data: This type of data has a predefined order to it, and
it is formatted to a particular structure before it is stored. This structure
is referred to as schema-on-write. One good example is data stored in a
relational database management system.
Unstructured data: This is when data is stored in its native format and
not processed until it is used, which is known as schema-on-read. It has
no form or order. It can be stored in a variety of file formats; examples
are emails, social media posts, presentations, chats, IoT sensor data, and
satellite imagery.
Semistructured data: This form of data includes metadata that
outlines specific features of the data3 and allows it to be manipulated
more efficiently than unstructured data. Data stored in XML format is an
example of this kind of data.
In business, the types of systems that capture data include the
following:
30
Chapter 2 Data for Analysis in Small Business
31
Chapter 2 Data for Analysis in Small Business
you a lot of information about what your customers are looking at either
on your own website or even the competitors’ website. For ecommerce
websites, for example, we can collect customer reviews, web transaction
data, customers’ demographic data, and weblog data. An example of
companies that help collect web data is Comscore (www.comscore.com/).
These companies can help to collect data which can give insights to how a
particular company is trending on social media, see what is happening on
the company’s competitor’s website, and so on.
A real business illustration I: Consulting for small business requires
creativity in sourcing for analytics data; for example, a small business that
organizes training on different subjects aims at using information on its
current training attendee to build a model to predict which of its prospects
will be interested in training in the future. Their intention is to approach the
prospects predicted to be interested with their new training package. Since
it’s a small business, there is minimal budget to carry out this task, and
therefore they need to know the exact prospects to target with the available
resources. In order to get external data for this task, the data analyst had
to put a targeted advert on Facebook, accompanied by a Google Form link
which is used to collect the details of interested prospects. This is one way to
source external data creatively.
A real business illustration II: Hadetulah eye clinic is a small business
of about 20 staff in total and about five branches in Nigeria. They offer
several services relating to eye treatment; from diagnosing to sales of eye
treatment equipment. Even though they do not have a core enterprise
application running, they capture financial operation’s data in the form of
excel sheets. They also store information on the referral hospitals, equipment
details, rental or sales information, and invoices. For Hadetulah, the
customer and people systems are captured in excel sheets in the form of
name of customer, phone, age, sex, occupation, address, and so on. They
are also able to capture information such as the amount spent on types of
lenses, staff who attended to the customer, and so on. Figure 2-1 gives an
illustration of a feature representation example for Hadetulah opticals.
32
Chapter 2 Data for Analysis in Small Business
From this figure, we can see the customer’s picture and the features or
attributes that represent the customer. The figure further gives an example
of the values that each attribute might have and illustrates how this (an
historical collection of features of different customers) will be an input to
the prediction algorithm given that the attribute amount is the label. After
developing the model, we can now use it to predict for a new customer.
Data Privacy
It is very important as a data analyst to familiarize yourself with the data
privacy guidelines, and ignorance is not an excuse even when it comes
to data privacy issues. To this end, there are standards in place to help
enforce data privacy. Professional or commercial organizations establish
certain ethical standards in order to put in place a set of nonlegally binding
information treatment procedures. To enforce the treatment of certain
sets of data, legal standards have been established by law and order. A
company or an institution can also establish its own policy guidelines
or standards. There is also the good judgment standard, which states
that even if something is technically correct by more formal rules, one
should constantly consider “Is this a good idea?” and “What might the
repercussions be?”
33
Chapter 2 Data for Analysis in Small Business
34
Chapter 2 Data for Analysis in Small Business
data can be used for its intended purpose and the degree to which data
accurately represents the real world. There is a need to measure the quality
of data to be used for analysis, and this can be done by checking the data
for the following:
Completeness: This is used to check if we have all of the information
we should have. We respond to questions such as, “Have all events been
captured?” Are all reference data values taken into account?
Uniqueness: This is when we check if single events are captured
only once.
Accuracy: This is where we determine if the data is a true
representation of the concept it is attempting to convey. Ensuring
numerical and string values are correct, as well as timestamps and other
properties, is among the other activities.
Consistency: In checking for consistency, we want to ensure that the
data format is as expected. There is a need to ensure that the same data is
captured the same way every time.
Conformance/validity: Does the data in the database comply with the
syntax, encoding, or other model requirements? Are data formats correct?
Are codes as expected? Are naming conventions adhered to?
Timeliness: Here, we want to know if data is available by the time it is
needed. This is also referred to as “data latency.”
Provenance: Here, we want to know if we have visibility into the origins
of the data. And how much confidence do we have that the data is real and
accurate?
From a survey in 2017 on 7376 respondents by Kaggle5 on the data
science competition community, it was discovered after asking them to
rate the barriers they faced at work that the highest in the rank was the
dirty data problem which carried a percentage of 49.4% out of the whole
population surveyed.
Particularly common in the SME business environment is the dirty
data problem. These are situations where data is full of errors such as
misspelled data, typos, and duplicate data. This can be fixed systematically
35
Chapter 2 Data for Analysis in Small Business
when identified. There are also situations where data violates business
rules; to solve this, there might be a need to involve a cross-functional
effort between departments of the organization. There are situations where
data is collected using the wrong method or the wrong population. This
data often comes from asking the wrong business questions; all these can
be overcome by putting a proper data governance procedure in place for
the business at the start or carry out corrective measure when discovered.
Currently, there are algorithms that can be used to analyze data for
duplicates and other mistakes. It is important to emphasize here that
prevention is the best cure, that is, improved procedures of data collection,
forms, and records with more standardized fields are preferable. In
addition, make origins and history of data visible and transparent in order
to trace back every mistake. In conclusion, the completeness, validity, and
consistency of your data depend on the methods, and each business will
have developed and customized such methods for its particular needs.
2.3 Data Governance
Data governance is simply defined as the process put in place to make
sure that data is available, usable, secured, and has integrity. In the past,
only large corporations cared about data governance, but because small
businesses are also onboard making data-informed decisions, they are
also embracing data governance. The data governance process is built on
a collection of rules and regulations that secure data and ensure that it is
handled in accordance with all applicable external laws. The data steward
is responsible for developing and enforcing these uniform norms and
regulations. For small businesses, the way to go about data governance
is to tackle it from these three directions. First, assign someone (or more
if you can afford it) to the task of data governance. A good rule of thumb
is that as soon as you have a full-time analyst in your team, they should
explicitly own data governance.9 Second, the person in charge comes up
36
Chapter 2 Data for Analysis in Small Business
2.4 Data Preparation
After the necessary data has been retrieved for analytics, the very first thing
to do is to clean the data. This is necessary because analysis done on faulty
data can be misleading, and there can be serious economic consequences
of using uncleaned data for analysis. In the past, many major organizations
have lost a lot of revenue due to this, and a popular example is the
JPMorgan Chase trading loss.11
First of all, let’s lay some foundation about the types of variables so that
we can understand the data preparation procedures. The data obtained is
in the form of variables. Categorical variables have a fixed range of values
and can be unordered (called nominal variables) with categories such as
North America, Europe, and Asia; or they could be ordered (called ordinal
variables) with categories such as high value and low value. Numerical
variables are never ending. They are continuous (able to assume any
real numerical value, usually in a given range). Numerical variables can
be integers which take only integer values. Another kind of variable is
the date, and we have other ones depending on the analytics platform
being used.
37
Chapter 2 Data for Analysis in Small Business
When cleaning the data, we are examining the data for bad formatting,
trailing spaces, duplicates, empty rows, synonyms, and abbreviations. In
cleaning the data, you ask questions like: Do character variables have valid
values? Are numerical values within range? Are there duplicate values/
rows? Are values unique for some variables? Are dates valid? Does the
dataset even contain attributes indicating what you want to predict? In
which case there is no need for prediction. Are there missing values? Is there
inconsistency in description?
In preparing and examining the data you want to use, summary
statistics can be used to discover the complete set of variables in the data,
learn the characteristics of the variables and the range of values in the
variables, and select variables for the analysis. Mean, median, standard
deviation, minimum, maximum, and so on are all examples of summary
statistics. The minimum and maximum can be used to detect extreme
values that might be errors. The mean and median give a sense of the
central values of that variable, and a large deviation between the two also
indicates skew. The standard deviation gives a sense of how dispersed the
data are (relative to the mean).
Summary Statistics
Example 2.1
The data OnlineQuestion1.xlsx needed to go through a data cleaning
process. First, understand and familiarize yourself with the data using
summary statistics, handle missing data and outliers, and finally normalize
the data. We will deal with each of these tasks as they are treated.
38
Chapter 2 Data for Analysis in Small Business
39
Chapter 2 Data for Analysis in Small Business
40
Chapter 2 Data for Analysis in Small Business
41
Chapter 2 Data for Analysis in Small Business
42
Chapter 2 Data for Analysis in Small Business
Missing Data
How do we handle missing data? You cannot just simply throw away data
in many cases because you need to have enough data for meaningful
analysis. Also, there might be more than one attribute with different
missing values in which case too many rows will be removed. Sometimes,
43
Chapter 2 Data for Analysis in Small Business
45
Chapter 2 Data for Analysis in Small Business
46
Chapter 2 Data for Analysis in Small Business
47
Chapter 2 Data for Analysis in Small Business
48
Chapter 2 Data for Analysis in Small Business
49
Chapter 2 Data for Analysis in Small Business
50
Chapter 2 Data for Analysis in Small Business
51
Chapter 2 Data for Analysis in Small Business
52
Chapter 2 Data for Analysis in Small Business
2.5 Data Visualization
Data can be visualized for several purposes in an analytics project.
Data can be visualized for prediction, classification, and unsupervised
learning, also known as advanced descriptive analytics. For a detailed
understanding of data visualization, check out the following references:
Galit et al. (2019), chapter 3 or Thomas Rahlf (2017).
When visualizing data for prediction, we explore the data with the
following guide:
53
Chapter 2 Data for Analysis in Small Business
54
Chapter 2 Data for Analysis in Small Business
2.6 Problems
1. Using the data named OnlineQuestionAfterOutlier.
csv, create dummies of all categorical attributes and
normalize all numerical attributes.
2.7 References
1. Talend, (2022, March) Structured vs. Unstructured
Data: A Complete Guide, www.talend.com/
resources/structured-vs-unstructured-data/
55
Chapter 2 Data for Analysis in Small Business
5. www.kaggle.com/kaggle/kaggle-survey-2017
13. https://docs.rapidminer.com/
56
Chapter 2 Data for Analysis in Small Business
57
CHAPTER 3
Business Analytics
Consulting
In this chapter, we will look at Business Analytics consulting, particularly
what the concept implies and how to build such a career path. We will
explain the types of Business Analytics consulting that exist and then
narrow them down to how to navigate the world of Business Analytics
consulting for small businesses. In this chapter, we will look at how to
manage a typical analytics project and measure the success of analytics
projects. In conclusion, we will discuss issues revolving around how to bill
analytics projects, particularly as a consultant.
60
Chapter 3 Business Analytics Consulting
Some of these companies also take data science interns. There are also
independent data science consultants who offer data science consulting
services without working for the company or organization whether it’s a
small or large business. Some are referred to as data science freelancers,
and to get jobs, they use platforms like www.freelancer.com/, www.
toptal.com/, www.upwork.com/, and so on, while some just position
themselves as Business Analytics consultants or consulting firm and work
independent of any organization or platform. In practice, there are several
issues that an independent data scientist might face; on the top of this list
is the resistance of the business in question to release all the data needed
for analysis. The advice to independent data science consultants is to
not scare the business by asking for all you need at once. However, you
have to do your initial analysis and know what you will need to solve the
problem. The approach is to retrieve some amount of data to start with
and show some results that can encourage the clients to open up more in
terms of data. You can simulate what they stand to gain from the data if
made available. Always carry your clients and stakeholders along at every
stage of the analytics project. Business Analytics consulting should be seen
as offering products, in which case the product is in the form of solving
analytical problems. Business Analytics consultants can package analytics
solutions to specific problems as products offered. This book looks at
Business Analytics consulting from this perspective. Some of the problems
termed as products that we look at include customer loyalty (Chapter 7),
customer and product segmentation (Chapter 8), customer profiling
(Chapter 7), and so on. It is important to note that these problems, though
common in retail business, also find expressions in other types of business.
It is important for Business Analytics consultants to know that data
science techniques are embedded in business understanding. If you don’t
know how a business operates and how to improve a business normally,
you cannot know how the data science technique will be used or where it
fits in, to improve the sales of the business, for example. A good example is
in using data science to improve ecommerce business sales. For example,
61
Chapter 3 Business Analytics Consulting
to sell more of products not selling, you have to know how this can be
done in typical marketing, for example, through cross-selling. This will
not inform us of the kind of analytics to carry out. In this situation, you
might want to use association rule analysis to discover products that have
occurred together in the past and then use it for cross-selling.
A Business Analytics consultant might perform three types of roles
when working with clients: expert (outside looking in, you’re working
with a client who has a problem and wants you to fix it), pair-of-hands
(providing a service that the client company might be able to perform
themselves,) and collaborator (working side by side, the outside consultant
joins forces with members of the client organization to work on the project
or solve the problem together). Each of these represents a different kind of
interaction and a different source of satisfaction for the consultant.3 When
you take up consulting opportunities, it is important to state clearly which
mode you want to operate with.
62
Chapter 3 Business Analytics Consulting
The next and most important thing to put in place for managing an
analytics project effectively is procedures for effective communication with
all parties involved in the analytics project, from your team members to the
stakeholders; they need to be kept in the know of what is going on whether
good or bad, progressive or not. If need be, you need to also establish a
line of escalation in case you need something resolved, for example, you
are having issues retrieving data updates from the appropriate quarters.
Closely related to this is the fact that you need to be able to monitor your
team using whatever kind of tracker app you are comfortable with; this
will enable you to get regular updates and identify and fix issues before
it causes more damage to the analytics process. It will also help the
communication of the analytics team. Finally, since the whole essence of
consulting is to make money by solving business problems with analytics,
then, there is a need to track hours and finances invested in the project so
you know how to bill accurately, though this also depends on the billing
approach you have chosen.
In addition to the above, for you to be able to successfully manage an
analytics project you need to be able to conceptualize analytical solutions
for real-world problems. This means that when the problem is presented
to you in the form of a real-world or real business problem, you need to
be able to translate it to an analytics problem (note that not all problems
are solvable with analytics) and then break the solution down into smaller
components and link them together to give the overall results as revealed
in Figure 3-2.
63
Chapter 3 Business Analytics Consulting
64
Chapter 3 Business Analytics Consulting
65
Chapter 3 Business Analytics Consulting
66
Chapter 3 Business Analytics Consulting
67
Chapter 3 Business Analytics Consulting
68
Chapter 3 Business Analytics Consulting
3.5 References
1. Isabelle Flückiger (April 5, 2021) Discover 9
Consultancy Segments to Start an Exciting
Data Science Journey for Any Experience Level.
https://towardsdatascience.com/discover-9-
consultancy-segments-to-start-an-exciting-
data-science-journey-for-any-experience-
level-a972cb6b63e4
69
Chapter 3 Business Analytics Consulting
70
CHAPTER 4
Business Analytics
Consulting Phases
This chapter will look at the stages involved in Business Analytics
consulting, mainly when the analytics service is offered as a product
from either within or outside the business. We will look at the proposal
and initial stages of analysis that gives direction to the analytics project.
Then we look at the details involved in the pre-engagement, engagement,
and post-engagement phases. It is important to know that the stages are
presented in a typical or generic way, but when implemented, there might
be a reason to modify or customize them for the application scenario.
to invest in such a project at the moment given the fact that they are still
making some income based on their present methodology. The onus
now lies on the consultant to convince them on the transformation and
achievement that data analytics could bring about. In situations where
this stage is not needed (i.e., the stakeholders are already onboard), you
can just have a brief introduction meeting which will detail the analytics
procedure and elicit the kind of problems that are relevant to the business.
It will also help at this point to get feedback from the stakeholders and
know what is on their mind and what they hope to achieve with the
analytics project. Though there are situations where the stakeholders
might not know what constitutes analytics problems, they will definitely
know the needs of the organization. At this stage, very good presentation
skills will be needed to get your proposal accepted. At the proposal stage, it
will help to present problems that have been customized for the business
in particular. The consultant needs to be careful at this stage not to give
false hope or to propose to solve problems for which data resources are not
available, all in a bid to get the proposal accepted.
In the initial analysis stage, you want to find out what is available
in terms of the data, data quality, and integrity. This is the stage where
you explore the proposed problems with respect to the data to be used
in solving each of the problems. At the end of this phase, you should be
able to give a detailed report of what is achievable given the available
data. Also, in the initial analytics stage, you need to understand the
organization’s operational procedure (conceptual business model) as
explained under the Business Analytics journey in Section 1.3 of Chapter 1.
This will enable you to be able to understand the optimization approaches
for the proposed analytics models and how the result of your analytics
will fit into the organization or business process. You will also be able
to discover when and who will be responsible for implementing your
recommendation and so on. With this, you are able to know who is to be
involved in the communication stages of the analytics project. After the
72
Chapter 4 Business Analytics Consulting Phases
initial analysis, you should also be able to give an idea of what are the
expected outcomes of the analytics project at a high level of abstraction
and advice on best data practices for the organization. The tasks involved
in the initial analysis stage can be summarized as follows:
73
Chapter 4 Business Analytics Consulting Phases
At this stage, you need to make it clear that the analytics project
is an iterative process, which means that you might need to make
recommendations which will be implemented and evaluated and, if
necessary, repeated. This initial analysis will help to
74
Chapter 4 Business Analytics Consulting Phases
4.2 Pre-engagement Phase
In the pre-engagement phase, there is a need to carry out an exhaustive
exploratory analysis using descriptive analytics to expose or discover
the situation of things in relation to the intended goals. At the end of this
phase, there might be a need to refine the goals based on their feasibility
assessment. For each goal, you need to highlight the present situation
using descriptive analytics, the conclusions that can be deduced, and the
recommendations for immediate implementation or future analytics.
There is a need to define success criteria, data readiness, and assumptions
75
Chapter 4 Business Analytics Consulting Phases
for each of the goals stated. This is termed the kickoff and readiness phase.
It is where you try to map the journey for your clients and see where
analytics can play an important role in this journey.
Technically, the pre-engagement phase consists of the following:
Clients' analytics journey mapping: After developing the conceptual
business model, you then try to identify the stage of the conceptual
business model where the analytics solution will be used and what it will
be used for. This will help you to know the data needs for the analytics
project, particularly specific attributes.
Scoping out immediate and future needs: To do this, you will take the
goals of the analytics project and detail what you need to solve that is
actually available and that needs to be available in the future. For example,
if there is a need to clone the external market, what external data will be
needed, and is it possible to get access to it?
Timelines: There is a need to state the time that each of the phases will
take so that all parties will be carried along. This has to be done by stating
clearly the milestones and time of their accomplishments.
Engagement model: This model will help to give all parties involved
an idea of where they will be featuring in the project and what their role
will be. It can also include a solution architecture that gives the clients an
idea of how the analytics project will progress. Figure 4-2 is an example
of a simple solution architecture together with the engagement model for
consulting for an ecommerce business.2
76
Chapter 4 Business Analytics Consulting Phases
77
Chapter 4 Business Analytics Consulting Phases
are to give a detail of the values of the metrics before and after the goal
is accomplished. The stakeholders should be communicated on these
success criteria, so that they can give their contributions based on their
domain knowledge and organizational goals.
Data readiness and assumptions: Finally, the pre-engagement phase
should make it clearly known what is obtainable from the available data.
This will clear any false expectation from the analytics project. In some
situations, it could either redirect the project or cause some data decisions
to be made such as stalling the project to curate data and so on. By
stating the assumptions for the successful completion of the project, the
stakeholders are able to know why some things do not go as planned.
At the end of the pre-engagement phase, there is a need to come
up with the documents containing the details of the pre-engagement
phase, which is presented to the stakeholders. This is important because
oftentimes there would have been a further modification and refinement
of the conclusions from the initial analysis stage.
4.3 Engagement Phase
Depending on the kind of analytics project, meaning, for example,
we are expected to come up with software or recommendations for
implementation or both, the engagement phase is when you build the
solution to the problem or goal identified.
The engagement phase typically includes modeling and analysis,
regular feedback, and communicating insights effectively to clients. The
components of the engagement phase stated might increase depending
on the project at hand but not reduced as each of these components
determine the success of the project.
Modeling and analysis: First, determine the data mining task. Will it
be descriptive analytics, predictive analytics, or prescriptive analytics?
Will it be a combination of all these or will the task be used to complement
78
Chapter 4 Business Analytics Consulting Phases
79
Chapter 4 Business Analytics Consulting Phases
4.4 Post-Engagement Phase
After communicating the recommendations of your analytics project, the
next phase is the post-engagement phase, also known as the measurement
tracking phase. This phase consists of the go-live preparation, go-
live, measurement tracking against success metrics, and feedback
incorporation. The steps in the post-engagement phase are iterative.
Go-live preparation: This is when you organize workshops and training
to educate those that will implement the recommendations and how they
will implement them such that the overall goal of the analytics project can
be achieved. Sometimes, there might be a need to use demos, fliers, and
videos just to pass the message across. You also need to factor in change
management, that is, how the people, processes, and systems that drive
the organization will respond to the change that the recommendation
will bring about to achieve the overall goal. If the solution involves a
technology product, for example, you also need to tidy up loose ends as to
how the solution will fit into the existing system.
80
Chapter 4 Business Analytics Consulting Phases
4.5 Problems
Given the following scenarios, explain the details of the activities for the
pre-engagement, engagement, and post-engagement phases of a freelance
business analyst that was given a contract based on the following:
81
Chapter 4 Business Analytics Consulting Phases
4.6 References
1. Ramesh Sharda, Dursun Delen, Efraim Turban
(2020) Analytics, Data Science & Artificial
Intelligence Systems for Decision Support. Eleventh
edition, published by Pearson.
2. Vishranth Chandrashekar (July 2020) Journey Into
Data Science Consulting | How To Become A Data
Science Consultant | Great Learning, www.youtube.
com/watch?v=Rnhv_gayPHs
82
CHAPTER 5
Descriptive Analytics
Tools
This chapter is focused on the mostly common descriptive analytics
tools used in business generally and specifically in small businesses. The
chapter will help to use descriptive analytics tools to understand your
business and make recommendations that can improve your business
profits. For small businesses, descriptive analytics helps SMEs to make
sense of available data to monitor business indicators at a glance, help
SME owners observe sales trends and patterns on an overall basis, and
deep dive into product categories and customer groups. It also helps SMEs
plan product strategy, pricing policies that will maximize their projected
revenues and derive valuable insights for getting more customers.
5.1 Introduction
In Business Analytics, descriptive analytics helps to link the market to the
firm through information and provide information needed for actionable
decisions. Therefore, descriptive analytics can be seen as the principles
for systematically collecting and interpreting data that can aid decision
makers. There are three types of descriptive analytics:
5.2 Bar Chart
The bar chart can be used to examine categorical data for unexpected
patterns of results, for example, there may be many more of some
categories than others. Some categories may be missing completely or
there might even be uneven distributions in the categories. There might
even be extra categories, for example, if there was a mistake and the
gender was recorded as “M” and “F,” but also as “m” and “f,” “male” and
“female.” There may also be unbalanced experiments such that some data
are missing or unusable; this can be discovered using bar charts. Bar charts
84
Chapter 5 Descriptive Analytics Tools
can also help to discover when we have too many categories and make
decisions as to whether we are to combine them and how. Finally, it can
reveal errors and missing values.
Creating a bar chart in RapidMiner
85
Chapter 5 Descriptive Analytics Tools
86
Chapter 5 Descriptive Analytics Tools
5.3 Histogram
A histogram is used to group data into intervals, and drawing a bar for each
interval shows the empirical distribution. Histograms help to understand
the center of the data. Histograms are an excellent tool for identifying the
shape of your data distribution. There are situations where data is either
skewed to the right or left, and you have to perform data transformation.
When using a histogram to examine your data for prediction, for example,
87
Chapter 5 Descriptive Analytics Tools
you want to make sure the data is not skewed to the right or to the left in
such a way that it could affect the result of your analysis. That is, the data
needs to be symmetric or almost symmetric.
The log method is good for transforming right-skewed data and bad for
zero and negative values. The square root method is good for transforming
right-skewed data and bad for negative values. The square is good for
transforming data that is left-skewed but bad for negative values. The cube
root is good for transforming data that is right-skewed and also negative
values but not effective for log transformation.
Creating a histogram in RapidMiner
88
Chapter 5 Descriptive Analytics Tools
89
Chapter 5 Descriptive Analytics Tools
5.4 Line Graphs
Line graphs are used to track changes over short and long periods. When
smaller changes exist, line graphs are better to use than bar graphs. Line
graphs can also be used to compare changes over the same period of time
for more than one group.
Creating line graphs in RapidMiner
90
Chapter 5 Descriptive Analytics Tools
5.5 Boxplots
A boxplot is a standardized way of displaying the distribution of data based
on a five-number summary: minimum, first quartile (Q1), median, third
quartile (Q3), and maximum. It can tell you about your outliers and what
their values are. It can also tell you if your data is symmetrical, how tightly
your data is grouped, and if and how your data is skewed. It can also be
used for comparisons of distributions across subgroups.
Creating boxplots in RapidMiner
91
Chapter 5 Descriptive Analytics Tools
92
Chapter 5 Descriptive Analytics Tools
5.6 Scatter Plots
The scatter plots help to discover features such as
93
Chapter 5 Descriptive Analytics Tools
94
Chapter 5 Descriptive Analytics Tools
95
Chapter 5 Descriptive Analytics Tools
96
Chapter 5 Descriptive Analytics Tools
5.8 Treemaps
Treemaps display data in nested rectangles. It can be used to visualize the
size of categorical attributes. Each category of the categorical attribute
represents a rectangle, whereas the number in each category represents
the size of those rectangles.
Creating treemaps in RapidMiner
5.9 Heat Maps
A heat map is a graphical display of numerical data where color is used
to denote values. They are especially useful for the purpose of visualizing
correlation tables and visualizing missing values in the data. To do this,
the information is conveyed in a two-dimensional table. A correlation
98
Chapter 5 Descriptive Analytics Tools
table for a set of attributes has the same number of rows and columns.
If the number of attributes is a lot, then a subset of the attributes can be
used. Heat maps make it easy and faster to scan correlation values with the
color codes.
Creating heat maps in RapidMiner
5.10 Geographical Maps
Maps are basically the geographical representation of your data. There are
certain types of inferences that are best achieved only when the data is
visualized geographically.
Creating geographical maps in RapidMiner
101
Chapter 5 Descriptive Analytics Tools
Solution
102
Chapter 5 Descriptive Analytics Tools
103
Chapter 5 Descriptive Analytics Tools
104
Chapter 5 Descriptive Analytics Tools
105
Chapter 5 Descriptive Analytics Tools
106
Chapter 5 Descriptive Analytics Tools
107
Chapter 5 Descriptive Analytics Tools
108
Chapter 5 Descriptive Analytics Tools
5.12 Problems
1. This exercise is the continuation of the
practical business problem I (Section 5.11). The
intention is to discover more insights, using the
SuperstoreNigeria.xls to answer the following
questions. (Note: Some questions might require
more interactive visualization tools like Tableau.)
a. What is the sum of sales for each subcategory and how does
this relate with the profit, that is, which subcategory has the
highest and lowest profit. Use color to indicate this.
b. What is the sum of sales for each segment and how does
this relate with the quantity, that is, which segment has
the highest and lowest quantity produced. Use color to
indicate this.
109
Chapter 5 Descriptive Analytics Tools
i. What is the total quantity sold and total sales made for office
supplies using standard class ship mode.
110
Chapter 5 Descriptive Analytics Tools
5.13 References
1. Graphical Data Analysis with R by Antony Unwin,
published by CRC Press, 2015.
2. Jim Frost (March 2022) Using Histograms to
Understand Your Data, https://statisticsbyjim.
com/basics/histograms
111
CHAPTER 6
Predicting Numerical
Outcomes
In this chapter, we will explore the popular techniques used for prediction,
particularly in the retail business. The approach used in explaining these
techniques is to use them in solving a business problem. The business
problem to be addressed is the sales prediction problem which is
common in the retail business. The chapter first explains the fundamental
concept of prediction techniques; next, we look at how such techniques
are evaluated. After this, we describe the business problem we intend
to solve. We then pick each of the selected techniques one by one and
explain the algorithms involved and how they can be used to solve the
problem described. The prediction techniques used are the multiple linear
regression, the regression trees, and the neural network. To conclude the
chapter, we compare the results of the three algorithms and conclude on
the problem in question. In this chapter, therefore, the analytics product
offered is the sales prediction problem for small retail businesses.
6.1 Introduction
In Section 1.3 under the analytics journey, it was mentioned that there are
three major types of analytics: descriptive, predictive, and prescriptive.
In this chapter, the predictive analytics will be used to solve the sales
114
Chapter 6 Predicting Numerical Outcomes
115
Chapter 6 Predicting Numerical Outcomes
1 n
∑ ei
n i =1
(Eq. 6.2)
1 n
∑ei
n i =1
(Eq. 6.3)
1 n ei
100 × ∑
n i =1 yi
(Eq. 6.4)
116
Chapter 6 Predicting Numerical Outcomes
1 n ei
100 × ∑
n i =1 yi
(Eq. 6.5)
∑e
i −1
2
i
(Eq. 6.6)
117
Chapter 6 Predicting Numerical Outcomes
Variable Description
Id Unique identification
Price Offer price
Age Age in months
KM Kilometers
Fuel_Type Fuel type (Petrol, Diesel, CNG)
HP Horsepower
Met_Color Metallic color
Automatic Automatic (Yes = 1, No = 0)
CC Cylinder volume in cubic
centimeters
Doors Number of doors
Quarterly_Tax Quarterly tax
Weight Weight in kilograms
118
Chapter 6 Predicting Numerical Outcomes
Regardless of the prediction algorithm that will be used, the very first
thing is to perform basic data preprocessing and explore the data using the
necessary visualization tools for the purpose of prediction. The following
preprocessing task will be carried out:
2. Outlier detection
3. Visualization
4. Data transformation
5. Correlation analysis
All these steps have been handled one way or the other in numbers
1–10 under the practical demonstration.
Practical demonstration
119
Chapter 6 Predicting Numerical Outcomes
120
Chapter 6 Predicting Numerical Outcomes
121
Chapter 6 Predicting Numerical Outcomes
122
Chapter 6 Predicting Numerical Outcomes
123
Chapter 6 Predicting Numerical Outcomes
124
Chapter 6 Predicting Numerical Outcomes
125
Chapter 6 Predicting Numerical Outcomes
126
Chapter 6 Predicting Numerical Outcomes
n −1
Radj
2
=1−
n − p −1
(1 − R 2 ) (Eq. 6.8)
127
Chapter 6 Predicting Numerical Outcomes
2
Unlike R2, which ignores the number of predictors used, Radj
penalizes the number of predictors used. For a more detailed discussion
of the multiple linear regression model, see Galit et al. (2018).3
The validation/test dataset is used to determine MLR’s prediction
performance. These are dataset partitions that are not meant to be used to
pick predictors or calculate model parameters.
Variable selection in linear regression
Choosing the right form to use is determined by domain knowledge,
data availability, and the required predictive capability. It’s important
to be cautious before employing all of the available variables to build
your model because collecting a full complement of predictors for future
forecasts can be costly or impossible. Furthermore, we may be able to
measure fewer predictors more reliably (e.g., in surveys), and the more
predictors we have, the more likely we are to have missing values in the
data. To limit the number of predictors, there are a few options. To reduce
the number of predictors, one technique is to employ domain knowledge.
The utilization of computational power and statistical performance
measures is another option. Some of these approaches are as follows:
128
Chapter 6 Predicting Numerical Outcomes
129
Chapter 6 Predicting Numerical Outcomes
130
Chapter 6 Predicting Numerical Outcomes
131
Chapter 6 Predicting Numerical Outcomes
132
Chapter 6 Predicting Numerical Outcomes
133
Chapter 6 Predicting Numerical Outcomes
134
Chapter 6 Predicting Numerical Outcomes
Note that we can also predict for the new record manually by using the
regression equation obtained from the model in Figure 6-11.
6.5 Regression Trees
The family of decision tree classifiers includes regression trees. The
data-driven method of decision tree classifiers necessitates a significant
amount of data. ID3, C4.5, CART (Classification and Regression Trees),
135
Chapter 6 Predicting Numerical Outcomes
and others are examples. This group of techniques uses a greedy (i.e.,
nonbacktracking) approach in which decision trees are built in a recursive
divide-and-conquer approach from the top down. As the tree is created,
the training set is recursively partitioned into smaller subsets. The way
characteristics are chosen in the creation of the tree and the pruning
processes utilized in decision tree algorithms differ.
Attribute selection measures in decision tree classifiers
A heuristic for picking the splitting criterion that “best” splits a given
data partition, D, of class-labeled training tuples into individual classes
is an attribute selection measure. If we split D into smaller partitions, the
optimum splitting requirement is for each partition to be pure. Because
they determine how tuples at a given node are split, attribute selection
measures are also known as splitting rules. The attribute selection measure
assigns a score to each attribute that describes the training tuples in
question. For the given tuples, the attribute with the highest measure score
is chosen as the splitting attribute. If the splitting attribute is continuous-
valued or we are limited to binary trees, we must determine a split point or
a splitting subset as part of the splitting criterion, respectively.2 The three
popular attribute selection measures are
• Information gain
• Gain ratio
• Gini index
136
Chapter 6 Predicting Numerical Outcomes
138
Chapter 6 Predicting Numerical Outcomes
139
Chapter 6 Predicting Numerical Outcomes
AND MET_Color>0.500
140
Chapter 6 Predicting Numerical Outcomes
141
Chapter 6 Predicting Numerical Outcomes
142
Chapter 6 Predicting Numerical Outcomes
143
Chapter 6 Predicting Numerical Outcomes
144
Chapter 6 Predicting Numerical Outcomes
145
Chapter 6 Predicting Numerical Outcomes
X −a
X norm = (Eq. 6.9)
b−a
146
Chapter 6 Predicting Numerical Outcomes
147
Chapter 6 Predicting Numerical Outcomes
148
Chapter 6 Predicting Numerical Outcomes
149
Chapter 6 Predicting Numerical Outcomes
150
Chapter 6 Predicting Numerical Outcomes
151
Chapter 6 Predicting Numerical Outcomes
6.8 Problems
1. Using the MLR model in Figure 6-11, write the
regression equation to predict for the transaction
with the following attribute values:
a. Neural network
c. Regression trees
Present the result obtained from each algorithm in detail for the
following stages:
a. Data visualization
b. Feature selection
c. Model development
d. Model evaluation
152
Chapter 6 Predicting Numerical Outcomes
6.9 References
1. Selva Prabhakaran (March 2017) Complete
Introduction to Linear Regression in R, www.
machinelearningplus.com/machine-learning/
complete-introduction-linear-regression-r/
153
CHAPTER 7
Classification
Techniques
In this chapter, even though there are several classification techniques,
we will explore the popular ones used for classification in the business
domain. In doing this, we will use the third business problem centered on
customer loyalty using neural networks, classification trees, and random
forest algorithms. In solving this problem, we are particular about how
to get and retain more customers for our small business. We will also
introduce some other classification-based techniques such as K-nearest
neighbor and logistic regression. In using these techniques to solve the
problem, we explain the fundamental concepts in the chosen algorithms
and use them to demonstrate how these problem solving processes can be
adopted in real business scenarios.
TN + TN
(Eq. 7.1)
P+N
FN + FN
(Eq. 7.2)
P+N
156
Chapter 7 Classification Techniques
TP
(Eq. 7.3)
P
TN
(Eq. 7.4)
N
Precision
TP
(Eq. 7.5)
TP + FP
2 x precision x recall
(Eq. 7.6)
precision + recall
The terms true positive, true negative, false positive, positive, and
negative samples are abbreviated as TP, TN, FP, P, and N, respectively.
• True positives (TP): These are the positive tuples that
the classifier properly labels.
• True negatives (TN): These are the negative tuples that
the classifier correctly categorized.
• False positives (FP): These are negative tuples that have
been mistakenly categorized as positive.
• False negatives (FN): Positive tuples are classified as
negative.
157
Chapter 7 Classification Techniques
Yes no total
Yes tp fn p
no fp tn n
total p’ n’ p+n
158
Chapter 7 Classification Techniques
determined cut-off value when there are many records and a single class is
of interest. If the probability of inclusion in the desired class is greater than
the cut-off, the record is moved to the desired class. There’s a default cut-
off value of 0.5 in two-class classifiers.3
159
Chapter 7 Classification Techniques
that though we used a bank as a case study here, any other business can be
used with appropriate data attributes.
The data for the business problem is named Churn_Modelling.csv,
and it contains data on the customers of a bank. The data description is as
follows:
160
Chapter 7 Classification Techniques
161
Chapter 7 Classification Techniques
163
Chapter 7 Classification Techniques
7.3 Neural Network
Please refer to Section 6.5 for details on the neural network algorithm.
Customer loyalty with neural network
Using the customer loyalty problem introduced in Section 7.2, the
following demonstrates how to use a neural network to solve the problem:
164
Chapter 7 Classification Techniques
165
Chapter 7 Classification Techniques
166
Chapter 7 Classification Techniques
167
Chapter 7 Classification Techniques
168
Chapter 7 Classification Techniques
7.4 Classification Tree
Please refer to Section 6.4 for details on decision tree algorithms.
Decision tree algorithms can be used to predict both numerical
and categorical outcomes. When the predicted outcome is categorical,
it is referred to as classification trees. Classification trees use predictor
information to “drop” a record down the tree until it reaches a node at the
bottom (terminal node). We can then classify it by gathering a “vote” of all
the training data that belongs to the terminal node during the formation of
the tree. The class with the most votes gets the new record.
169
Chapter 7 Classification Techniques
170
Chapter 7 Classification Techniques
171
Chapter 7 Classification Techniques
172
Chapter 7 Classification Techniques
173
Chapter 7 Classification Techniques
174
Chapter 7 Classification Techniques
175
Chapter 7 Classification Techniques
176
Chapter 7 Classification Techniques
The list of the trees is displayed in Figure 7-15a. If you click each tree,
you will see the model of the tree as shown in Figure 7-15a. There are a
total of 100 different trees.
177
Chapter 7 Classification Techniques
The list of the trees is displayed in Figure 7-16a. If you click each tree,
you will see the model of the tree as shown in Figure 7-16a. There are a
total of 100 different trees.
Having used the different algorithms to solve this problem, we can now
compare their accuracy and select the best for this particular data. Just as
for the prediction, for the classification too, there might be a need to use
different data and algorithm setting to experiment for better accuracy.
178
Chapter 7 Classification Techniques
Comparing Figures 7-7b, 7-11b, 7-15b, and 7-16b, we can see that the
best of the algorithms is the neural network (Figure 7-7b), which gave an
accuracy of 85.79%, just slightly higher than random forest.
For this problem, the objective is to predict which customers will be
loyal to prospective customers. With the result of the prediction for the
new customers, we can identify this, particularly if we select those that
have a high propensity. Also for the existing customers, we can predict
the loyal ones and the ones about to churn and take necessary actions.
If, for example, the goal is to target prospective customers, we can rank
the prospective customer records from highest to lowest propensity (in
the newly predicted dataset) in order to act on those with the highest
propensity for target marketing.
7.6 K-Nearest Neighbor
In this section, we’ll look at the K-nearest neighbor (KNN) algorithm
and how it can help us obtain more clients using a practical example.
K-nearest neighbor can be used to predict both numerical and categorical
output. In this example, it will be used to predict the categorical output.
This is particularly because the task of getting more customers is a
classification task.
KNN searches the training dataset for records that are “similar” for the
purpose of classifying or predicting a new record. By using averaging for
prediction as well as voting for classification, these “neighbors” are used to
produce a classification or prediction for the new record. The method is to
search the training data for records that are comparable to (or have values
similar to) the record we want to classify. We classify the one we wish to
classify based on the classes that these related records fall under.
179
Chapter 7 Classification Techniques
( x1 − u1 ) + ( x2 − u2 ) + …………….. + ( x p − u p )
2 2 2
(Eq. 7.7)
180
Chapter 7 Classification Techniques
181
Chapter 7 Classification Techniques
The evaluation of the KNN model can be done based on the methods
for evaluating prediction or classification as the case demand since it can
be used for both.
Classification example using KNN
A picture frame designer would like to find a way of classifying their
customers in Lagos into those likely to purchase a family size frame and
those not likely to buy one. We will use a random sample of 12 families
who have purchased the frames and 12 who have not, living in Ibadan (a
similar city).
182
Chapter 7 Classification Techniques
183
Chapter 7 Classification Techniques
Figure 7-18. Color-coded scatter plot for Income and House Area
184
Chapter 7 Classification Techniques
185
Chapter 7 Classification Techniques
186
Chapter 7 Classification Techniques
7.7 Logistic Regression
Logistic regression is based on a model that links the predictors to the
result. A small set of data can be used to train logistic regression classifiers,
which can then be applied to large groups of new records. When the result
variable, Y, is categorical, logistic regression extends the concepts of linear
regression.
Based on the values of its predictor variables, logistic regression may
be used to classify a new record, whose class is not known, into one of
the classes (called classification). It can also be used to discover features
distinguishing between records in various classes in terms of their
predictor profile or predicting variables in data where the class is known
(called profiling).
When we need to classify customers as returning or nonreturning
(classification) in business, find features that discriminate between
customers (profiling), and so on, we use logistic regression. We can use
logistic regression to calculate a new record’s propensity to fall into one
187
Chapter 7 Classification Techniques
of several classes, and records are then ranked from highest to lowest
propensity so that we can focus our efforts on those with the highest
propensity.
The predictor variables x1, x2,......xp could be categorical, continuous, or
a combination of both.
Steps of logistic regression
1. Calculate the propensities of belonging to
each class.
Just like in KNN, the default cut-off value in two-class classifiers is 0.5.
The logistic regression model
From the linear regression equation in Equation 6.7, we can express p
as a linear function predictor (x1,x2……xp):
1
p= (Eq. 7.9)
(
− β 0 + β1 x1 + β 2 β x2 +…+ β q β xq )
1+ e
188
Chapter 7 Classification Techniques
p
Odds (Y = 1) = (Eq. 7.10)
1− p
odds
p= (Eq. 7.11)
1 + odds
We may describe the relationship between the odds and the predictors
by substituting, for example, Equation 7.9 into Equation 7.11 as
Odds (Y = 1) = e
β 0 + β1 x1 + β 2 x2 +…+ β q xq
(Eq. 7.12)
189
Chapter 7 Classification Techniques
Among these 5000 customers, only 480 (= 9.6%) accepted the personal
loan offered to them in a previous campaign. The goal is to build a model
that identifies customers who are most likely to accept the loan offer in
future mailings.
The bank application scenario is just a selected example; other
application scenarios is in a sales campaign for example, where the goal
is to build a model that identifies customers who are most likely to make
a purchase. In a service campaign, that is, a business that offers one form
of service or another to clients, the goal is to build a model that identifies
clients who will take the service offer in future proposals or marketing.
The outcome variable is personal loan, with yes defined as the success
(this is equivalent to setting the outcome variable to 1 for an acceptor and
0 for a non-acceptor).
190
Chapter 7 Classification Techniques
191
Chapter 7 Classification Techniques
192
Chapter 7 Classification Techniques
193
Chapter 7 Classification Techniques
194
Chapter 7 Classification Techniques
The odds that a customer with income zero will accept the loan are
estimated by e -6:127+(0.037)(0) = 0.00218.
The odds of accepting the loan with an income of $100K are
e-6:127+(0.037)(100)
= 0.0883.
7.8 Problems
1. Write the logistic regression equation from the
model in Figure 7-24a.
195
Chapter 7 Classification Techniques
7.9 References
1. Jiawei Han, Micheline Kamber and Jian Pei (2012)
Data Mining Concepts and Techniques, Third
Edition, published by Elsevier.
196
Chapter 7 Classification Techniques
6. www.researchgate.net/publication/331749857_
Generalized_Linear_Models
197
CHAPTER 8
Advanced Descriptive
Analytics
This chapter is focused mainly on advanced descriptive analytics techniques.
In this chapter, we will first explain the concept of clustering which is a type of
unsupervised learning approach. We will then pick one clustering technique
which is the k-means clustering. Using the fourth practical business problem,
we will explain how we can use the k-means clustering technique to solve a
real business problem. Next, we will explain the association rule example and
finally network analysis. We will focus the explanation of these techniques
on solving business-related problems, particularly for small businesses, and
conclude with the fifth business problem which is focused on using network
analytics for employee efficiency.
8.1 Clustering
Clustering is an unsupervised machine learning approach that aims to
segment data into similar groups of records to gain insight. Cluster analysis
is a method that forms clusters of records by calculating the distance
between these records. Clustering is applied in business2 in various ways
which include categorizing clients based on demographic and transaction
history information and then tailoring a marketing plan for each category.
It can also be used to create segments for customers and give them
200
Chapter 8 Advanced Descriptive Analytics
The measures used for the distance between two records (numerical
data) include the following:
• Euclidean distance: This type of measure requires the
normalization of continuous measurements before
computing. A different distance (such as the statistical
distance) is also recommended if the measurements
are highly correlated. Outliers can make Euclidean
distance skewed. For Euclidean distance, outliers are
either deleted or more robust distances like Manhattan
distances are used.
• Correlation-based similarity.
For more information on the preceding list, check Han et al. (2012).
Similarity measures are more intuitively appealing than distance
measures for measuring the distance between two records having
categorical data with binary values. In this case, the matching coefficient
and Jaccard’s coefficient are the most helpful similarity measurements.
For measuring the distance between two records, categorical data and
mixed data (some continuous and some binary), a similarity coefficient
suggested by Gower is very useful.5
201
Chapter 8 Advanced Descriptive Analytics
ď
Ă
202
Chapter 8 Advanced Descriptive Analytics
8.2 K-Means
The k-means algorithm divides the sample into a predefined number k of
well-separated (no overlaps) clusters with the purpose of making these
clusters similar in terms of the measures adopted (e.g., Figure 8-1). The
total distance (or sum of squared Euclidean distances) between records
and their cluster centroid is a commonly used metric of within-cluster
dispersion. The k-means algorithm is expressed in Figure 8-2.
How to determine k
203
Chapter 8 Advanced Descriptive Analytics
Validating clusters
Finding meaningful clusters is a key goal of cluster analysis. The
following steps can assist you in accomplishing this:
1. Cluster interpretability: Is the interpretation of
the clusters that resulted reasonable? Explore the
properties of each cluster to interpret the clusters.
You can do this by doing the following:
a. Analyze the results of the cluster analysis to
derive summary statistics (such as averages,
minimums, and maximums) for each cluster’s
measurement.
b. Examine the clusters for separation along with
some common feature (variable) that was
omitted from the cluster analysis.
204
Chapter 8 Advanced Descriptive Analytics
• Davies-Bouldin index
205
Chapter 8 Advanced Descriptive Analytics
• Jaccard index8
• Fowlkes-Mallows index8
206
Chapter 8 Advanced Descriptive Analytics
207
Chapter 8 Advanced Descriptive Analytics
Data description
The noodle retail data consist of the following attributes:
Brand
Country
Customer Id
Gender
208
Chapter 8 Advanced Descriptive Analytics
Income Class
List Price
Number of Records
Online Order
Owns Car
Product Class
Product Id
Product Size
Purchases
Past 3 Years
State
Transaction Date
Transaction Id
Deceased Indicator
Tenure
1. (a) Data preprocessing: Create a repository
(Clustering); create two subfolders in the repository
and name them data and process. Import the data
named NoodleRetail.xlsx into the data subfolder.
When importing, make sure to exclude the following
attributes based on the reasons stated:
• Country (same values all through)
209
Chapter 8 Advanced Descriptive Analytics
The following missing values are obvious from these statistics and will
be treated by replacing them with the average value of the attributes:
• Brand-93
• Online order-170
• Product class-93
• Product size-93
• Tenure-214
210
Chapter 8 Advanced Descriptive Analytics
Run the process and visualize the resulting data (only the numerical
attributes) for outliers using the settings in Figure 8-6.
211
Chapter 8 Advanced Descriptive Analytics
212
Chapter 8 Advanced Descriptive Analytics
213
Chapter 8 Advanced Descriptive Analytics
214
Chapter 8 Advanced Descriptive Analytics
From Figure 8-10, we see that there are no strongly correlated pair (the
highest pairwise correlation is 0.331), and for this reason, all the attributes
will be used for this demonstration. (Some attributes on top of the list
might be removed later to see if there will be better cluster separation
when evaluated.) This decision is concluded based on the Pearson
correlation formula, which is used by the Correlation Matrix operator.
215
Chapter 8 Advanced Descriptive Analytics
n xy x y
r
n x2 x
2
n y y
2 2
The preceding formula is used to get the value for R, and the value for
R must fall in the following range:
• +1 to 0.7: Means that the strength of the variables is
perfectly related in a positive way
• 0.6 to 0.5: Means that the strength of variables is
fairly related
• 0.4 to 0.2: Means that the strength of variables is
poorly related
• -1 to -0.7: Means that the strength of variables is
strongly related in a negative way
• -0.6 to -0.5: Means that the strength of variables is
fairly related
• -0.4 to -0.2: Means that the strength of variables is
poorly related
• 0: Means that there is no strength or relation
(b) We then move to the second part of preprocessing which deals with
importing the afresh (because we need the customer ID for clustering) and
selecting only the attributes that are not strongly correlated. The result of
this process will be stored in a csv file which will then be imported again
for k-means clustering.
216
Chapter 8 Advanced Descriptive Analytics
217
Chapter 8 Advanced Descriptive Analytics
For the Replace Missing Value operator, select Brand, Online order,
Product class, Product size, and Tenure; for the Nominal to Numerical
operator, select all available categorical data; and for the Normalize
operator, select all apart from customer Id. For the csv file parameter of the
Write CSV operator, select NoodleRetailNormalize.csv (which is an empty
csv file that you would have created by yourself and saved somewhere
on your system). After running the process, the data will be stored in
NoodleRetailNormalize.csv.
1. Determine k: To determine k, there are several
approaches to do this, but we shall be supplying
the number of k (which is 2) to use for this
demonstration, representing the domain expert.
2. Perform k-means clustering and validate the clusters:
Numerical measures were selected because we
have converted all the attributes to numerical.
Import the NoodleRetailNormalize.csv, and create
a new process (call it kmeans); use the settings in
Figure 8-13 for this process (the k-means clustering).
For the k parameter of the k means operator, set it as
2 and leave all other parameter settings as default.
218
Chapter 8 Advanced Descriptive Analytics
219
Chapter 8 Advanced Descriptive Analytics
Figure 8-14b is the output of the data that has been clustered,
indicating (using the cluster Id) the cluster to which each customer
belongs. This can be further visualized to see how well the customers are
separated. (The Cluster Model Visualizer operator can help, but it does
not come with the free edition of RapidMiner.)
220
Chapter 8 Advanced Descriptive Analytics
The plot tab selected in Figure 8-14c is the plot of the centroid table
which helps to see which attributes describe different clusters.
1. Interpret your results: To characterize the resulting
clusters, we examine the cluster centroids, using
this line chart (also known as the profile plot). We
can see from Figure 8-14c that the clusters are not
too separated from each other for some attributes,
but nonetheless there are some attributes that give
distinct separation such as the Brand = Dangote,
Brand = Chikki, Product class = medium, Product
class = low, Product class = high, Product size = large,
Product size = small, and list price. These are the main
distinguishing attributes for this k-means model, and
they can be used to characterize each of the clusters.
To characterize the different clusters, compare the cluster centroid
(centroid table in Figure 8-14c) and try to give each cluster a label.
This label is obtained from any or all of the variables in the dataset.
221
Chapter 8 Advanced Descriptive Analytics
The question here then is: Which clusters would you target for offers, and
what types of offers would you target to customers in that cluster?
8.4 Association Analysis
Association analysis is a technique used to discover groups of related items
(clusters) in databases that store transactions.
It aims to figure out what kinds of things people tend to buy together.
When this discovery is made, these items can be shown together among
the other items, offered in post-transaction coupons, or recommended
for online shopping. In order to select a subset of rules, the stages of
establishing association rules include formulating rules and evaluating the
strength of those rules.
Algorithms for mining frequent itemsets are used to construct
association rules. The classic one selected is the Apriori algorithm of Agrawal
et al. (1993). The important steps in the algorithm are shown in Figure 8-15.
The algorithm can generate a lot of rules, but we only need to make
a decision using the strongest based on the dependence between the
antecedent and consequent. If we have the following rule:
If diaper and biscuits are purchased, THEN wipes are pur-
chased on the same trip.
Antecedent: Diaper and biscuits are purchased.
Consequent: Wipes are purchased on the same trip.
The strength of association implied by a rule is measured using
223
Chapter 8 Advanced Descriptive Analytics
confidence
lift ratio = (Eq. 8.4)
benchmark confidence
224
Chapter 8 Advanced Descriptive Analytics
Table 8-1 is the example of the format for the data used for
association analysis using the Apriori algorithm. Each role represents the
transactions, and each column represents an item, and the total column
is the total items available. An example of the source of transactional
data for association analysis is data collected using barcode scanners in
supermarkets where each record lists all items bought by a customer on a
single-purchase transaction.
1 0 1 1 1 1 0 1
2 0 0 1 0 1 0 1
3 0 1 0 0 1 1 1
4 0 0 1 1 1 0 1
5 0 1 0 0 1 0 1
6 0 0 0 0 1 0 0
7 0 1 1 1 1 0 1
8 0 0 1 1 0 0 1
9 0 0 0 0 1 0 0
10 1 1 1 1 0 0 0
11 0 0 1 0 0 0 1
225
Chapter 8 Advanced Descriptive Analytics
226
Chapter 8 Advanced Descriptive Analytics
227
Chapter 8 Advanced Descriptive Analytics
228
Chapter 8 Advanced Descriptive Analytics
229
Chapter 8 Advanced Descriptive Analytics
If Fila, llbean, and Adidas are purchased together, then, with confidence
of 94.5%, Nike and Bata will also be purchased. This rule has a lift ratio
of 1.379.
The preceding rule can be used in several ways, for example, if the
customer is requesting for a particular brand of shoe, you can introduce
a new brand to the customer. This new brand is recommended based on
solving the brand association problem which makes it certain that the
customer will also want to buy the new brand. This type of inference can
be implemented in several ways by SMEs.
Examples include through email marketing, recommender engines,
promotions, and so on.
230
Chapter 8 Advanced Descriptive Analytics
8.5 Network Analysis
A network is a set of objects (nodes) with interconnections (edges). An
example of a communication network obtained from tweets is shown in
Figure 8-21.
231
Chapter 8 Advanced Descriptive Analytics
Network metrics
232
Chapter 8 Advanced Descriptive Analytics
233
Chapter 8 Advanced Descriptive Analytics
&
,
'
& ,
' &
234
Chapter 8 Advanced Descriptive Analytics
235
Chapter 8 Advanced Descriptive Analytics
237
Chapter 8 Advanced Descriptive Analytics
238
Chapter 8 Advanced Descriptive Analytics
239
Chapter 8 Advanced Descriptive Analytics
240
Chapter 8 Advanced Descriptive Analytics
241
Chapter 8 Advanced Descriptive Analytics
242
Chapter 8 Advanced Descriptive Analytics
243
Chapter 8 Advanced Descriptive Analytics
244
Chapter 8 Advanced Descriptive Analytics
245
Chapter 8 Advanced Descriptive Analytics
246
Chapter 8 Advanced Descriptive Analytics
For this example, we will stop here, but there are a lot of other settings
that can be used to visualize this network for various types of insights.
Also, the preview tab can be used for better visualization of the network.
From Figure 8-35, we can see the node ecobank_nigeria has the highest
eigenvector centrality, signifying that it is the most central because its
neighbors are also central.
Network analysis – example 2
For the second example, we will be creating a product network
of a food and agricultural business. The edge list is created from the
associations of the products as they are purchased together. The data file
to be used for creating this network is named EdgeListProducts.csv and
NodeListProducts.csv, respectively.
1. Open a new project in Gephi. Load the data into
Gephi as an undirected graph (File ➤ Open), and
load the EdgeListProducts.csv first. Figure 8-36 is
the interface after successfully loading the edge list
into Gephi.
247
Chapter 8 Advanced Descriptive Analytics
248
Chapter 8 Advanced Descriptive Analytics
249
Chapter 8 Advanced Descriptive Analytics
250
Chapter 8 Advanced Descriptive Analytics
251
Chapter 8 Advanced Descriptive Analytics
252
Chapter 8 Advanced Descriptive Analytics
253
Chapter 8 Advanced Descriptive Analytics
In this problem scenario, the network data was collected via a survey
and converted to a node (nodeslabelStaff.csv) and edge list (edgeslistStaff.
csv), respectively.
In this problem scenario, we are looking at the possibility of generating
any or some of the following insights. Is the network good or bad, are
people really doing what they are supposed to be doing in terms of
collaboration, and so on? To do this, we will answer questions like: How
do collaboration patterns vary? How do collaboration patterns matter for
important outcomes? This can be explored with respect to Individuals,
groups, or organizations.
To answer the question, “How do collaboration patterns vary?”, we can
use simple descriptive statistics; we can pick a network size, for example, and
compare across individuals and compare changes over time. The result of
this can be used for managing employees in the following areas: performance
assessment, roles and responsibilities, pay and promotions, training and
mentoring, job rotations and career development, and retention.
To answer the question, “How do collaboration patterns matter for
important outcomes we care about for our employees?”, first we should
realize that there are different types of outcomes for individuals such as
performance (sales per quarter, cost savings, self-reported 1–3 ratings,
manager-reported 1–3 ratings, bonus), satisfaction, commitment, burnout,
turnover, etc.
In this practical example, we have chosen to use network size to
examine performance. To do this, we will check the correlation between
size and performance. For our case study, the performance data is
captured in sales per quarter (measured in millions).
Practical demonstration
1. Load the edgeslistStaff.csv into Gephi, and
set the Graph Type to undirected. Import the
second csv file (nodeslabelStaff.csv) using Import
spreadsheet under the File menu. When importing
254
Chapter 8 Advanced Descriptive Analytics
255
Chapter 8 Advanced Descriptive Analytics
256
Chapter 8 Advanced Descriptive Analytics
257
Chapter 8 Advanced Descriptive Analytics
258
Chapter 8 Advanced Descriptive Analytics
259
Chapter 8 Advanced Descriptive Analytics
260
Chapter 8 Advanced Descriptive Analytics
8.7 Problems
1. Repeat the practical business problem V, using
the same data (nodeslabelStaff.csv); this time
around, correlate the centrality measures with the
performance.
2. Using the dataset VegetableTransactions.csv,
perform an association rule analysis, and comment
on the results. Select the most important rules
using the necessary criteria and give appropriate
interpretation to the selected rules.
3. Using the data named Alegra.xlsx, perform
clustering analysis using the k-means clustering
algorithm. Validate, visualize, and interpret the
resulting clusters.
8.8 References
1. Agrawal, R., Imielinski, T. and Swami, A. (1993)
Mining Association Rules between Sets of Items
in Large Databases. Proceedings of the 1993 ACM
SIGMOD International Conference on Management
of Data, Washington DC, 26-28 May 1993, 207-216.
2. Maksim Tsvetovat, Alexander Kouznetsov (2011)
Finding Connections on the Social Web, Social
Network Analysis for Startups.
3. Anil K. Maheshwari (2015) Business Intelligence
and Data Mining, published by Business Expert
Press, LLC.
261
Chapter 8 Advanced Descriptive Analytics
262
Chapter 8 Advanced Descriptive Analytics
https://en.wikipedia.org/wiki/Eigenvector_
centrality
https://en.wikipedia.org/wiki/Closeness_
centrality
https://en.wikipedia.org/wiki/Betweenness_
centrality
263
CHAPTER 9
9.1 SME Ecommerce
Ecommerce is the buying and selling of goods and services online. It
involves trading of products or services directly or indirectly, by any means
of venture, for the purpose of wealth creation, thereby adding value to the
end users, and to improve means of livelihood.
Ecommerce small and medium enterprises (SMEs) are start-up
ventures that employ digital online services as a platform for creating
awareness and launching products and services to target customers. These
involve promotional efforts and marketing strategies to initiate online
purchases and drive sales and for maintaining repeated purchases.
The following are some of the analytics goals that one might want to
achieve in an ecommerce analytics project:
266
Chapter 9 Case Study Part I
The types of data that one might analyze in such domain include
i. Transaction data
268
Chapter 9 Case Study Part I
purchase them. They take orders both offline and online, but even the
offline orders are eventually entered into the online system so as to be
able to have a holistic view to running the business operation. They have
a staff strength of about 20 employees in different capacity; some are into
the management of the offline operations such as logistics and supply,
stocking and procurement, and so on, while others are into managing the
online operations which include digital marketing and sales management.
FarmCo is being managed by an ecommerce software which is the main
data source for this analytics project.
The conceptual business model for FarmCo is given by Figure 9-1. It
helps to understand the major parts and layout of the business process and
to particularly know where the data for analytics will be coming from.
269
Chapter 9 Case Study Part I
Typically, the whole analytics process takes between three and six
months, but every project comes with its peculiarities, and, therefore, this
time might change from project to project.
The initial analysis phase will be covered in Section 9.3.
In the pre-engagement phase, you want to carry out some level of
analysis to know what is obtainable from the data you have been able to
retrieve in the initial analysis phase. Based on this result, you are able to
present a detailed analytics journey mapping for the clients so that there
will be a clear understanding of the roles of all parties in the project. At
this point, you want to make it plain what is obtainable and what is to be
moved into the future analytics project (recommendations to improve
the analytics in the future). Timelines have to be specified with margins,
and, most importantly, the stated goals each have to be clearly attached
to the success criteria and how they will be measured. Finally, the data
readiness (which translates to the feasibility of making the intended
recommendations from the data) should be stated with the assumptions
upon which the analytics project is to be carried.
270
Chapter 9 Case Study Part I
271
Chapter 9 Case Study Part I
9.3 Initial Analysis
The initial analysis has to do with the following steps:
272
Chapter 9 Case Study Part I
During the initial analysis for FarmCo, the goals that the stakeholders
intended to solve (or they proposed) include the following:
273
Chapter 9 Case Study Part I
9.4 Analytics Approach
For each of the goals to be explored in this analytics project, we will use
the analytics approach described in Section 1.3. Just as mentioned earlier,
depending on the application situation, we might first use descriptive
analytics as a form of exploratory research to discover questions to be
answered in the diagnosis analysis stage and then follow this up with
predictive and prescriptive. Other times, we might just go straight into any
of them. All these decisions are determined by the nature of the analytics
project.
Goal 1: Increase website traffic
Using descriptive analytics, we need to find out the current state of the
website traffic. We want to answer questions like the following:
274
Chapter 9 Case Study Part I
275
Chapter 9 Case Study Part I
9.5 Pre-engagement
In this section, we will give examples of some of the descriptive analytics
outputs for answering some of the questions asked in the previous section.
Also, in the pre-engagement phase, there is a need to detail the metrics
to be used in assessing the success of these goals. Also, at this stage after
getting the results from the descriptive analytics, we need to decide if
recommendations will be made for implementation (i.e., we have been
able to discover some insights that the business can use to make some
decisions) or for further analysis (i.e., the results will form the basis for the
modeling in the engagement phase) or for future analytics (i.e., there is a
need to provide data advice that could help the analytics in the future).
Goal 1: Increase website traffic
For this goal, the success criterion is the website traffic, and the current
situation of such is depicted in Figure 9-4.
276
Chapter 9 Case Study Part I
From Figure 9-4, we can see that the traffic goes up and down, with a
steady decline from June. At this point, we want to find out if there is any
reason for this from the organization’s point of view, for example, change
in marketing, weather, government issues, and so on. Ideally, the traffic is
supposed to be increasing steadily.
There are several angles to these descriptive analytics that could be
presented based on the available data, for example, we can also try to see
the top ten most visited pages and the time of these visits. In addition,
for each marketing channel, SEO, email campaign, coupon code, social
media, etc., report the new visitors, returning visitors, orders, sales,
conversions, new online accounts, and bounce that has come from them in
total (i.e., not monthly). The result of this will help to know which channel
is bringing in the visits, for example, and the diagnosis can start from
there. We will also be able to know if the investments in these channels are
proportional to the visit’s results.
277
Chapter 9 Case Study Part I
278
Chapter 9 Case Study Part I
279
Chapter 9 Case Study Part I
280
Chapter 9 Case Study Part I
9.6 References
1. Devin Pickell (February 28, 2019) 4 Types of Data
Analytics Your Business Can Benefit. From www.
g2.com/articles/types-of-data-analytics
281
CHAPTER 10
Analytics result
284
Chapter 10 Case Study Part II
285
Chapter 10 Case Study Part II
286
Chapter 10 Case Study Part II
288
Chapter 10 Case Study Part II
289
Chapter 10 Case Study Part II
290
Chapter 10 Case Study Part II
291
Chapter 10 Case Study Part II
292
Chapter 10 Case Study Part II
293
Chapter 10 Case Study Part II
294
Chapter 10 Case Study Part II
RFM segmentation
RFM segmentation1 is used to target certain clusters of clients with
communications that are more tailored to their specific behaviors. RFM
segmentation is a useful tool for identifying groups of clients who should
be given extra attention. The letters RFM stand for recency, frequency,
and monetary value. RFM uses numerical scales to produce a high-level
portrayal of clients that is both brief and instructive. It is straightforward to
comprehend and interpret.
295
Chapter 10 Case Study Part II
296
Chapter 10 Case Study Part II
297
Chapter 10 Case Study Part II
298
Chapter 10 Case Study Part II
This resulted into the data named FinalRFM.xlsx (though this data has
about five tiers instead of four as in Table 10-1).
It is possible to use the results of RFM segmentation to target certain
groups of clients depending on the RFM segments in which they appear.
Groups like best customers are examples of this type of consumer
(customers who are found in R-Tier-1, F-Tier-1, and M-Tier-1, meaning
that they transacted recently, do so often, and spend more than other
customers).
High-spending new customers: Customers in 1-4-1 and 1-4-2. These
are consumers that have only transacted once, but very recently, and have
spent a significant amount of money.
Lowest-spending active loyal customers: Customers in segments 1-1-3
and 1-1-4 make up this group. They transacted recently and do so often,
but spend the least.
Churned best customers: Customers in groups 4-1-1, 4-1-2, 4-2-1, and
4-2-2. They used to transact frequently and spend a lot of money, but they
haven't done so in a long time, and so on.
299
Chapter 10 Case Study Part II
301
Chapter 10 Case Study Part II
302
Chapter 10 Case Study Part II
303
Chapter 10 Case Study Part II
304
Chapter 10 Case Study Part II
305
Chapter 10 Case Study Part II
Figure 10-19 is the result of the correlation matrix for the internal
data; highly correlated variables are a disadvantage when using Euclidian
distance. The pairwise table is given in Figure 10-20 (sorted in descending
order). Based on this result, we will eliminate Item Count, No of C, Email
Status = P, and Quantity as they have high correlation values.
306
Chapter 10 Case Study Part II
307
Chapter 10 Case Study Part II
its cluster. The settings in Figure 10-21 will give the visualized cluster in
Figures 10-22a (K=2), 10-22b (performance of K=2), 10-23a (K=3), 10-23b
(performance of K=3), 10-24a (K=4), and 10-24b (performance of K=4).
308
Chapter 10 Case Study Part II
309
Chapter 10 Case Study Part II
310
Chapter 10 Case Study Part II
From the preceding results, we select K=3 due to the following reasons:
Having decided on K=3 for the internal data, to choose the best
segmentation result for the problem at hand (internal data), we can use
the profit made as a metric and then for each segment calculate the mean
profit for all members in the cluster. There are other metrics that can be
311
Chapter 10 Case Study Part II
used such as cancel rates, etc., depending on the application scenario. The
output of the cluster (Figure 10-25) can be linked with the original FarmCo.
Internal.xlsx to know the profit of each cluster member and then calculate
each cluster’s mean profit made. This is revealed in Table 10-2. Notice that
the profit made is not part of the attributes for clustering as it will not be in
the external data.
Cluster_0 172,000
Cluster_1 745,555
Cluster_2 175,000
312
Chapter 10 Case Study Part II
313
Chapter 10 Case Study Part II
314
Chapter 10 Case Study Part II
Import the external data again and, this time, set the role of Customer
ID to Id. Exclude the following while importing: Customer ID, Shipping
State, Quantity, No of C, Item Count. The settings in Figure 10-29 will
give the visualized cluster in Figures 10-30a (K=2), 10-30b (performance
of K=2), 10-31a (K=3), 10-31b (performance of K=3), 10-32a (K=4), and
10-32b (performance of K=4). Note that when selecting attributes, select
the ones used for internal clustering.
315
Chapter 10 Case Study Part II
316
Chapter 10 Case Study Part II
317
Chapter 10 Case Study Part II
318
Chapter 10 Case Study Part II
From the preceding results, we select K=2 due to its smallest davies_
bouldin value.
The characteristics of the clusters are given as follows:
Cluster_0
Billing State=Ogun is low.
Billing State=Others is slightly high.
Billing State=Lagos is slightly high.
Email Status=A is slightly high.
DiscountedPrice is low.
Cluster_1
Billing State=Ogun is very high.
Billing State=Others is low.
Billing State=Lagos is slightly low.
Email Status=A is low.
DiscountedPrice is slightly high.
(Note that other k values that give better separation and distinct
characteristics can be tried out.)
319
Chapter 10 Case Study Part II
10.3 Problems
1. Using the data named PoultryNigeria.gephi, use
Gephi to arrive at Figure 10-5.
320
Chapter 10 Case Study Part II
10.4 References
1. RFM Segmentation, www.optimove.com/resources/
learning-center/rfm-segmentation
2. Anthony Capetola (February 28, 2021) This is
How You Can Use Predictive Analytics to Sell
Smarter Through Email, www.coredna.com/blogs/
predictive-analytics
3. Jennifer Xue, How to Use Predictive Analytics for
Better Marketing Performance, www.singlegrain.
com/digital-marketing-analytics/how-to-use-
predictive-analytics-for-better-marketing-
performance/
4. Divya Chandana (May 2021) Exploring Customers
Segmentation With RFM Analysis and K-Means
Clustering, https://medium.com/web-mining-
is688-spring-2021/exploring-customers-
segmentation-with-rfm-analysis-and-k-means-
clustering-118f9ffcd9f0
321
Data Files
Chapter 2
• FarmCo.External.xlsx
• OnlineQuestion1.xlsx
• OnlineQuestionAfterMissing.csv
• OnlineQuestionAfterOutlier.csv
Chapter 5
• OnlineQuestion1.xlsx
• customersNigeria.xlsx
• chapter5Assign.xlsx
Chapter 6
• ToyotaCorollaNewData.xlsx
• FoodAgricGroup.xlsx
Chapter 7
• Churn_Modelling.csv
• FramesData.xlsx
• InternalData.xlsx
• newBank.xlsx
• NoodleRetailLRExternal.xlsx
• NoodleRetailLRInternal.xlsx
• Prospects.xlsx
• TrainingDS3.csv
• TrainingDS3NewData.csv
Chapter 8
• Alegra.xlsx
• dynamic.gexf
• EdgeListProducts.csv
• edgeslistStaff.csv
• NodeListProducts.csv
• nodeslabelStaff.csv
• NoodleRetail.xlsx
• NoodleRetailNormalize.csv
324
DATA FILES
• shoes.xlsx
• StatisticStaff.csv
• VegetableTransactions.csv
Chapter 10
• PoultryNigeria.gexf
• FoodAgricGroup.xlsx
• FoodAgricTransAsso.xlsx
• SampleHistorical.xlsx
• FinalRFM.xlsx
• RFMFoodAgric.xlsx
• FarmCo.External.xlsx
• FarmCo.Internal.xlsx
325
Index
A, B recommendations/
strategies/plan, 16, 17
Boosted trees
results, 14, 15
classification, 174
small/medium
customer loyalty, 177
businesses, 18, 19
performance, 178
storage/location, 10
random (see Random forest/
visualization, 11
boosted trees)
resulting model/
performance, 177
Business Analytics
C
application types, 2 Classification techniques, 155
conceptual business algorithms, 158
model, 8–11 classifier, 155
customer analytics, 3 confusion matrix, 158
data analytics, 12–14 customer loyalty
data capturing model, 9, 10 bar chart, 162
data integration, 18 boxplot visualization, 164
definition, 2 company’s products/
events, 6–9 services, 159
extracting data, 11–13 data description, 160
financial analytics, 2 data importing, 161
implementation stage, 16 data preprocessing, 160, 161
operational analytics, 4 practical business
quality/integrity, 18 problem, 159
people analytics, 3 scatter plot, 163
presentation, 15 decision tree
real business problem, 7–10 algorithms, 169
328
INDEX
329
INDEX
330
INDEX
G, H
E, F Global positioning satellite
Ecommerce SMEs (GPS), 10, 31
analytics approach, 274–276
analytics project, 266, 267
case study Business I, J, K
Analytics road map, 270 Key performance indicators
conceptual business (KPIs), 2
model, 269 K-means algorithm
engagement model, 271 clustering algorithm, 203
FarmCo, 268 determination, 203, 204
initial analysis phase, 270 gap statistic methods, 203
modeling/measurement interpreting process, 206
phases, 271, 272 separation, 205
offline/online operation, 269 target marketing
post/pre-engagement purpose, 206
phases, 270, 271 validation, 204–206
software, 269 K-means clustering, 199, 203,
clickstream data, 268 208, 216–218
definition, 265 K-nearest neighbor (KNN)
domain types, 267 boxplot, 183
initial analysis, 272, 273 correlation matrix, 184, 185
pre-engagement phase data partition, 182
conversion rate, 280 definition, 179
definition, 276 Euclidean distance, 180
descriptive analytics, 277 evaluation, 182
recommendation, 278 frame designer, 182
revenue phase, 279 inner process, 186
331
INDEX
332
INDEX
R
P, Q Random forest/boosted trees
Prediction techniques, 113 cross validation operator, 175
classification, 115 customer loyalty, 175
evaluation, 115–117 inner process, 176
historical data, 114 main process, 175
machine learning, 114 objective, 179
MLR (see Multiple linear performance, 176, 177
regression (MLR)) steps, 174
neural network, 143–151 Regression trees
numerical measurements, 116 attribute selection, 136
practical business problem benefits, 137
consulting approach, 117 data-driven method, 135
correlation matrix process, datasets resulting, 141, 143
125, 126 decision tree structure, 136, 137
data description, 118 evaluation, 140
demonstration, 119, 120 inner process, 142
histograms, 121, 122 logical rules, 140
load data, 120 main process, 142
333
INDEX
334
INDEX
335