CS8091 BIGDATA ANALYTICS QUESTION BANK - Watermark
CS8091 BIGDATA ANALYTICS QUESTION BANK - Watermark
• Big Data platform is IT solution which combines several Big Data tools and
utilities into one packaged solution for managing and analyzing Big Data.
• Big data platform is a type of IT solution that combines the features and
capabilities of several big data application and utilities within a single solution.
• It is an enterprise class IT platform that enables organization in developing,
deploying, operating and managing a big data infrastructure /environment.
Big data is a term for data sets that are so large or complex that traditional data
processing applications are inadequate.
Challenges include
• Analysis,
• Capture,
• Data Curation,
• Search,
• Sharing,
• Storage,
• Transfer,
• Visualization,
• Querying,
• Updating
Information Privacy.
• The term often refers simply to the use of predictive analytics or certain other
advancedmethods to extract value from data, and seldom to a particular size of
data set.
• ACCURACY in big data may lead to more confident decision making, and
better decisions can result in greater operational efficiency, cost reduction and
reduced risk.
• Big data usually includes data sets with sizes beyond the ability of commonly
used
• software tools to capture, curate, manage, and process data within a tolerable
elapsed
• time. Big data "size" is a constantly moving target.
• Big data requires a set of techniques and technologies with new forms of
integration to
• reveal insights from datasets that are diverse, complex, and of a massive scale
a) Hadoop
• Hadoop is open-source, Java based programming framework and server software
which is used to save and analyze data with the help of 100s or even 1000s of
commodity servers in a clustered environment.
• Hadoop is designed to storage and process large datasets extremely fast and in
fault tolerant way.
• Hadoop uses HDFS (Hadoop File System) for storing data on cluster of
commodity computers. If any server goes down it know how to replicate the data
and there is no loss of data even in hardware failure.
• Hadoop is Apache sponsored project and it consists of many software packages
which runs on the top of the Apache Hadoop system.
• Top Hadoop based Commercial Big Data Analytics Platform
• Hadoop provides set of tools and software for making the backbone of the Big
Data analytics system.
• Hadoop ecosystem provides necessary tools and software for handling and
analyzing Big Data.
• On the top of the Hadoop system many applications can be developed and
plugged-in to provide ideal solution for Big Data needs.
b) Cloudera
• Cloudra is one of the first commercial Hadoop based Big Data Analytics
Platform offering Big Data solution.
• Its product range includes Cloudera Analytic DB, Cloudera Operational DB,
Cloudera Data Science & Engineering and Cloudera Essentials.
• All these products are based on the Apache Hadoop and provides real-time
processing and analytics of massive data sets.
e) MapR
• MapR is another Big Data platform which us using the Unix file system for
handling data.
• It is not using HDFS and this system is easy to learn anyone familiar with the
Unix system.
• This solution integrates Hadoop, Spark, and Apache Drill with a real-time data
processing feature.
(iii) Velocity – The term 'velocity' refers to the speed of generation of data. How fast
the data is generated and processed to meet the demands, determines real potential in
the data.
Big Data Velocity deals with the speed at which data flows in from sources like business
processes, application logs, networks, and social media sites, sensors, Mobile devices,
etc. The flow of data is massive and continuous.
(iv) Variability – This refers to the inconsistency which can be shown by the data at
times, thus hampering the process of being able to handle and manage the data
effectively.
Benefits of Big Data Processing
Ability to process Big Data brings in multiple benefits, such as-
Businesses can utilize outside intelligence while taking decisions
Access to social data from search engines and sites like Facebook, twitter are enabling
organizations to fine tune their business strategies.
Improved customer service
Traditional customer feedback systems are getting replaced by new systems designed
with Big Data technologies. In these new systems, Big Data and natural language
processing technologies are being used to read and evaluate consumer responses.
Early identification of risk to the product/services, if any
Better operational efficiency
Big Data technologies can be used for creating a staging area or landing zone for new
data before identifying what data should be moved to the data warehouse. In addition,
such integration of Big Data technologies and data warehouse helps an organization to
offload infrequently accessed data.
3. Explain in detail about Nature of Data and its applications.
Data
• Data is a set of values of qualitative or quantitative variables; restated, pieces
of data are individual pieces of information.
• Data is measured, collected and reported, and analyzed, whereupon it can be
visualized using graphs or images.
Properties of Data
For examining the properties of data, reference to the various definitions of data.
Reference to these definitions reveals that following are the properties of data:
a) Amenability of use
b) Clarity
c) Accuracy
d) Essence
e) Aggregation
f) Compression
g) Refinement
.Amenability of use: From the dictionary meaning of data it is learnt that data are
facts used in deciding something. In short, data are meant to be used as a base for
arriving at definitive conclusions.
TYPES OF DATA
• In order to understand the nature of data it is necessary to categorize them into
various types.
• Different categorizations of data are possible.
• The first such categorization may be on the basis of disciplines, e.g., Sciences,
Social Sciences, etc. in which they are generated.
• Within each of these fields, there may be several ways in which data can be
categorized into types.
The distinction between the four types of scales center on three different characteristics:
1. The order of responses – whether it matters or not
2. The distance between observations – whether it matters or is interpretable
3. The presence or inclusion of a true zero
Nominal Scales
Nominal scales measure categories and have the following characteristics:
• Order: The order of the responses or observations does not matter.
• Distance: Nominal scales do not hold distance. The distance between a 1 and a
2 is not the same as a 2 and 3.
• True Zero: There is no true or real zero. In a nominal scale, zero is
uninterruptable.
Appropriate statistics for nominal scales: mode, count, frequencies
Displays: histograms or bar charts
Ordinal Scales
At the risk of providing a tautological definition, ordinal scales measure, well, order.
So, our characteristics for ordinal scales are:
• Order: The order of the responses or observations matters.
• Distance: Ordinal scales do not hold distance. The distance between first and
second is unknown as is the distance between first and third along with all
observations.
• True Zero: There is no true or real zero. An item, observation, or category
cannot finish zero.
Appropriate statistics for ordinal scales: count, frequencies, mode
Displays: histograms or bar charts
Interval Scales
Interval scales provide insight into the variability of the observations or data.
Classic interval scales are Likert scales (e.g., 1 - strongly agree and 9 - strongly
disagree) and
Semantic Differential scales (e.g., 1 - dark and 9 - light).
In an interval scale, users could respond to “I enjoy opening links to thwebsite from a
company email” with a response ranging on a scale of values.
Appropriate statistics for interval scales: count, frequencies, mode, median, mean,
standard deviation (and variance), skewness, and kurtosis.
Displays: histograms or bar charts, line charts, and scatter plots.
Ratio Scales
Ratio scales appear as nominal scales with a true zero.
They have the following characteristics:
• Order: The order of the responses or observations matters.
• Distance: Ratio scales do do have an interpretable distance.
• True Zero: There is a true zero.
Income is a classic example of a ratio scale:
• Order is established. We would all prefer $100 to $1!
• Zero dollars means we have no income (or, in accounting terms, our revenue
exactly equals our expenses!)
• Distance is interpretable, in that $20 appears as twice $10 and $50 is half of a
$100.
For the web analyst, the statistics for ratio scales are the same as for interval scales.
Appropriate statistics for ratio scales: count, frequencies, mode, median, mean,
standard deviation (and variance), skewness, and kurtosis.
Displays: histograms or bar charts, line charts, and scatter plots.
HDFS is a distributed file system that handles large data sets running on commodity
hardware. It is used to scale a single Apache Hadoop cluster to hundreds (and even
thousands) of nodes. HDFS is one of the major components of Apache Hadoop, the
others being MapReduce and YARN. HDFS should not be confused with or replaced
by Apache HBase, which is a column-oriented non-relational database management
system that sits on top of HDFS and can better support real-time data needs with its in-
memory processing engine.
Fast recovery from hardware failures
Because one HDFS instance may consist of thousands of servers, failure of at least one
server is inevitable. HDFS has been built to detect faults and automatically recover
quickly.
Access to streaming data
HDFS is intended more for batch processing versus interactive use, so the emphasis in
the design is for high data throughput rates, which accommodate streaming access to
data sets.
Accommodation of large data sets
HDFS accommodates applications that have data sets typically gigabytes to terabytes
in size. HDFS provides high aggregate data bandwidth and can scale to hundreds of
nodes in a single cluster.
Portability
To facilitate adoption, HDFS is designed to be portable across multiple hardware
platforms and to be compatible with a variety of underlying operating systems.
A decision tree (also called prediction tree) uses a tree structure to specify sequences of
decisions and consequences. Given input , the goal is to predict a response or output
variable . Each member of the set is called an input variable. The prediction can be
achieved by constructing a decision tree with test points and branches. At each test
point, a decision is made to pick a specific branch and traverse down the tree.
Eventually, a final point is reached, and a prediction can be made. Each test point in a
decision tree involves testing a particular input variable (or attribute), and each branch
represents the decision being made. Due to its flexibility and easy visualization,
decision trees are commonly deployed in data mining applications for classification
purposes.
The input values of a decision tree can be categorical or continuous. A decision tree
employs a structure of test points (called nodes) and branches, which represent the
decision being made. A node without further branches is called a leaf node. The leaf
nodes return class labels and, in some implementations, they return the probability
scores. A decision tree can be converted into a set of decision rules. In the following
example rule, income and mortgage_amount are input variables, and the response is the
output variable default with a probability score.
Decision trees have two varieties: classification trees and regression trees.
Classification trees usually apply to output variables that are categorical—often
binary—in nature, such as yes or no, purchase or not purchase, and so on. Regression
trees, on the other hand, can apply to output variables that are numeric or continuous,
such as the predicted price of a consumer good or the likelihood a subscription will be
purchased.
Decision trees can be applied to a variety of situations. They can be easily represented
in a visual way, and the corresponding decision rules are quite straightforward.
Additionally, because the result is a series of logical if-then statements, there is no
underlying assumption of a linear (or nonlinear) relationship between the input
variables and the response variable.
The term branch refers to the outcome of a decision and is visualized as a line
connecting two nodes. If a decision is numerical, the “greater than” branch is usually
placed on the right, and the “less than” branch is placed on the left. Depending on the
nature of the variable, one of the branches may need to include an “equal to” component.
Internal nodes are the decision or test points. Each internal node refers to an input
variable or an attribute. The top internal node is called the root. The decision tree in
Figure is a binary tree in that each internal node has no more than two branches. The
branching of a node is referred to as a split.
Sometimes decision trees may have more than two branches stemming from a node. For
example, if an input variable Weather is categorical and has three choices—Sunny,
Rainy, and Snowy—the corresponding node Weather in the decision tree may have
three branches labelled as Sunny, Rainy, and Snowy, respectively.
The depth of a node is the minimum number of steps required to reach the node from
the root. In Figure for example, nodes Income and Age have a depth of one, and the
four nodes on the bottom of the tree have a depth of two.
Leaf nodes are at the end of the last branches on the tree. They represent class labels—
the outcome of all the prior decisions. The path from the root to a leaf node contains a
series of decisions made at various internal nodes.
In Figure the root node splits into two branches with a Gender test. The right branch
contains all those records with the variable Gender equal to Male, and the left branch
contains all those records with the variable Gender equal to Female to create the depth
1 internal nodes. Each internal node effectively acts as the root of a subtree, and a best
test for each node is determined independently of the other internal nodes. The left-hand
side (LHS) internal node splits on a question based on the Income variable to create leaf
nodes at depth 2, whereas the right-hand side (RHS) splits on a question on the Age
variable.
The decision tree shows that females with income less than or equal to $45,000 and
males 40 years old or younger are classified as people who would purchase the product.
In traversing this tree, age does not matter for females, and income does not matter for
males.
Decision trees are widely used in practice. For example, to classify animals, questions
(like cold-blooded or warm-blooded, mammal or not mammal) are answered to arrive
at a certain classification. Another example is a checklist of symptoms during a doctor’s
evaluation of a patient. The artificial intelligence engine of a video game commonly
uses decision trees to control the autonomous actions of a character in response to
various scenarios. Retailers can use decision trees to segment customers or predict
response rates to marketing and promotions. Financial institutions can use decision trees
to help decide if a loan application should be approved or denied. In the case of loan
approval, computers can use the logical if-then statements to predict whether the
customer will default on the loan. For customers with a clear (strong) outcome, no
human interaction is required; for observations that may not generate a clear response,
a human is needed for the decision.
By limiting the number of splits, a short tree can be created. Short trees are often used
as components (also called weak learners or base learners) in ensemble methods.
Ensemble methods use multiple predictive models to vote, and decisions can be made
based on the combination of the votes. Some popular ensemble methods include
random forest [4], bagging, and boosting [5]. Section 7.4 discusses these ensemble
methods more.
The simplest short tree is called a decision stump, which is a decision tree with the
root immediately connected to the leaf nodes. A decision stump makes a prediction
based on the value of just a single input variable.
3. Explain in detail about Naïve Bayes Classification.
A naïve Bayes classifier assumes that the presence or absence of a particular feature of
a class is unrelated to the presence or absence of other features. For example, an object
can be classified based on its attributes such as shape, color, and weight. A reasonable
classification for an object that is spherical, yellow, and less than 60 grams in weight
may be a tennis ball. Even if these features depend on each other or upon the existence
of the other features, a naïve Bayes classifier considers all these properties to contribute
independently to the probability that the object is a tennis ball.
The input variables are generally categorical, but variations of the algorithm can accept
continuous variables. There are also ways to convert continuous variables into
categorical ones. This process is often referred to as the discretization of continuous
variables. In the tennis ball example, a continuous variable such as weight can be
grouped into intervals to be converted into a categorical variable. For an attribute such
as income, the attribute can be converted into categorical values as shown below.
The conditional probability of event C occurring, given that event A has already
occurred, is denoted as , which can be found using the formula in Equation
Equation can be obtained with some minor algebra and substitution of the conditional
probability:
John checked in at least two hours early only 40% of the time, or . Therefore,
The probability that John received an upgrade given that he checked in early is 0.75, or
.
The probability that John received an upgrade given that he did not arrive two hours
early is 0.35, or. Therefore, the probability that John received an upgrade can be
computed as shown in Equation
Thus, the probability that John did not receive an upgrade. Using Bayes’ theorem, the
probability that John did not arrive two hours early given that he did not receive his
upgrade is shown in Equation
Another example involves computing the probability that a patient carries a disease
based on the result of a lab test. Assume that a patient named Mary took a lab test for a
certain disease and the result came back positive. The test returns a positive result in
95% of the cases in which the disease is actually present, and it returns a positive result
in 6% of the cases in which the disease is not present. Furthermore, 1% of the entire
population has this disease. What is the probability that Mary actually has the disease,
given that the test is positive?
Let C = {having the disease} and A = {testing positive}. The goal is to solve the
probability of having the disease, given that Mary has a positive test result, . From the
problem description, and. Bayes’ theorem defines. The probability of testing positive,
that is, needs to be computed first. That computation is shown in Equation
According to Bayes’ theorem, the probability of having the disease, given that Mary
has a positive test result, is shown in Equation.
That means that the probability of Mary actually having the disease given a positive test
result is only 13.79%. This result indicates that the lab test may not be a good one. The
likelihood of having the disease was 1% when the patient walked in the door and only
13.79% when the patient walked out, which would suggest further tests.
A more general form of Bayes’ theorem assigns a classified label to an object with
multiple attributes such that the label corresponds to the largest value of. he probability
that a set of attribute values (composed of variables) should be labelled with a
classification label equals the probability that the set of variables given is true, times
the probability of divided by the probability of. Mathematically, this is shown in
Equation.
Decision trees use greedy algorithms, in that they always choose the option that seems
the best available at that moment. At each step, the algorithm selects which attribute to
use for splitting the remaining records. This selection may not be the best overall, but it
is guaranteed to be the best at that step. This characteristic reinforces the efficiency of
decision trees. However, once a bad split is taken, it is propagated through the rest of
the tree. To address this problem, an ensemble technique (such as random forest) may
randomize the splitting or even randomize data and come up with a multiple tree
structure. These trees then vote for each class, and the class with the most votes is
chosen as the predicted class
There are a few ways to evaluate a decision tree. First, evaluate whether the splits of the
tree make sense. Conduct sanity checks by validating the decision rules with domain
experts, and determine if the decision rules are sound.
Next, look at the depth and nodes of the tree. Having too many layers and obtaining
nodes with few members might be signs of overfitting. In overfitting, the model fits the
training set well, but it performs poorly on the new samples in the testing set. Figure
7.7 illustrates the performance of an overfit model. The x-axis represents the amount of
data, and the yaxis represents the errors. The blue curve is the training set, and the red
curve is the testing set. The left side of the gray vertical line shows that the model
predicts well on the testing set. But on the right side of the gray line, the model performs
worse and worse on the testing set as more and more unseen data is introduced.
For decision tree learning, overfitting can be caused by either the lack of training data
or the biased data in the training set. Two approaches [10] can help avoid overfitting in
decision tree learning.
Stop growing the tree early before it reaches the point where all the training data is
perfectly classified. Grow the full tree, and then post-prune the tree with methods such
as reduced-error pruning and rule-based post pruning.
Last, many standard diagnostics tools that apply to classifiers can help evaluate
overfitting. These tools are further discussed in Section 7.3.
Decision trees are computationally inexpensive, and it is easy to classify the data. The
outputs are easy to interpret as a fixed sequence of simple tests. Many decision tree
algorithms are able to show the importance of each input variable. Basic measures, such
as information gain, are provided by most statistical software packages.
Decision trees are able to handle both numerical and categorical attributes and are robust
with redundant or correlated variables. Decision trees can handle categorical attributes
with many distinct values, such as country codes for telephone numbers. Decision trees
can also handle variables that have a nonlinear effect on the outcome, so they work
better than linear models (for example, linear regression and logistic regression) for
highly nonlinear problems. Decision trees naturally handle variable interactions. Every
node in the tree depends on the preceding nodes in the tree.
In a decision tree, the decision regions are rectangular surfaces. Figure 7.8 shows an
example of five rectangular decision surfaces (A, B, C, D, and E) defined by four
values— —of two attributes (and). The corresponding decision tree is on the right side
of the figure. A decision surface corresponds to a leaf node of the tree, and it can be
reached by traversing from the root of the tree following by a series of decisions
according to the value of an attribute. The decision surface can only be axis-aligned for
the decision tree.
The structure of a decision tree is sensitive to small variations in the training data.
Although the dataset is the same, constructing two decision trees based on two different
subsets may result in very different trees. If a tree is too deep, overfitting may occur,
because each split reduces the training data for subsequent splits.
Decision trees are not a good choice if the dataset contains many irrelevant variables.
This is different from the notion that they are robust with redundant variables and
correlated variables. If the dataset contains redundant variables, the resulting decision
tree ignores all but one of these variables because the algorithm cannot detect
information gain by including more redundant variables. On the other hand, if the
dataset contains irrelevant variables and if these variables are accidentally chosen as
splits in the tree, the tree may grow too large and may end up with less data at every
split, where overfitting is likely to occur. To address this problem, feature selection can
be introduced in the data pre-processing phase to eliminate the irrelevant variables.
Although decision trees are able to handle correlated variables, decision trees are not
well suited when most of the variables in the training set are correlated, since overfitting
is likely to occur. To overcome the issue of instability and potential overfitting of deep
trees, one can combine the decisions of several randomized shallow decision trees—the
basic idea of another classifier called random forest [4]—or use ensemble methods to
combine several weak learners for better classification. These methods have been
shown to improve predictive power compared to a single decision tree.
For binary decisions, a decision tree works better if the training dataset consists of
records with an even probability of each result. In other words, the root of the tree has
a 50% chance of either classification. This occurs by randomly selecting training
records from each possible classification in equal numbers. It counteracts the likelihood
that a tree will stump out early by-passing purity tests because of bias in the training
data.
When using methods such as logistic regression on a dataset with many variables,
decision trees can help determine which variables are the most useful to select based on
information gain. Then these variables can be selected for the logistic regression.
Decision trees can also be used to prune redundant variables.
CS8091 BIG DATA ANALYTICS
UNIT III ASSOCIATION AND RECOMMENDATION SYSTEM
QUESTION BANK
PART-A
1. What are Recommenders?
• Recommenders are instances of personalization software.
• Personalization concerns adapting to the individual needs, interests, and preferences of
each user.
Includes:
– Recommending
– Filtering
– Predicting (e.g. form or calendar appt. completion)
From a business perspective, it is viewed as part of Customer Relationship Management
(CRM).
2. What is Dimensionality Reduction?
Dimension Reduction refers to:
– The process of converting a set of data having vast dimensions into data with lesser
dimensions ensuring that it conveys similar information concisely.
– These techniques are typically used while solving machine learning problems to
obtain better features for a classification or regression task.
3. List out the problems on using Recommendation systems
• Inconclusive user feedback forms
• Finding users to take the feedback surveys
• Weak Algorithms
• Poor results
• Poor Data
• Lack of Data
• Privacy Control (May NOT explicitly collaborate with recipients)
4. List out the types of Recommender Systems.
– Content
– Collaborative
– Knowledge
5. What is Association Mining?
Finding frequent patterns, associations, correlations, or causal structures among sets
of items or objects in transaction databases, relational databases, and other information
repositories.
6. What is the Purpose of Apriori Algorithm?
Apriori algorithm is an influential algorithm for mining frequent item sets for
Boolean association rules. The name of the algorithm is based on the fact that the algorithm
uses prior knowledge of frequent item set properties.
7. List out the applications of Association rules.
– Basket data analysis,
– cross-marketing,
– catalogue design,
– loss-leader analysis,
– clustering,
– classification
8. Define support and confidence in Association rule mining.
Support S is the percentage of transactions in D that contain AUB.
Confidence c is the percentage of transactions in D containing A that also contain B.
Support (A=>B) = P(AUB)
Confidence (A=>B) =P(B/A)
9. What is Association rule?
Association rule finds interesting association or correlation relationships among a large set
of data items, which is used for decision-making processes. Association rules analyses
buying patterns that are frequently associated or purchased together.
10. Describe the method of generating frequent item sets without candidate generation.
Frequent-pattern growth(or FP Growth) adopts divide-and-conquer strategy.
Steps:
– Compress the database representing frequent items into a frequent pattern tree or FP
tree
– Divide the compressed database into a set of conditional databases
– Mine each conditional database separately
PART B & C
1. Explain about the basics of Recommendation Systems
A Common Challenge:
– Assume you’re a company selling items of some sort: movies, songs, products, etc.
– Company collects millions of ratings from users of their items
– To maximize profit / user happiness, you want to recommend items that users are
likely to want
Recommender systems
• Systems for recommending items (e.g. books, movies, CD’s, web pages, newsgroup
messages) to users based on examples of their preferences.
• Many websites provide recommendations (e.g. Amazon, NetFlix, Pandora).
• Recommenders have been shown to substantially increase sales at on-line stores.
• Recommender systems are a technological proxy for a social process.
• Recommender systems are a way of suggesting like or similar items and ideas to a
users specific way of thinking.
• Recommender systems try to automate aspects of a completely different information
discovery model where people try to find other people with similar tastes and then
ask them to suggest new things.
Motivation for Recommender Systems
Automates quotes like:
–"I like this book; you might be interested in it"
–"I saw this movie, you’ll like it“
–"Don’t go see that movie!"
Usage
• Massive E-commerce sites use this tool to suggest other items a consumer may
want to purchase
• Web personalization
Ways its used
• Survey’s filled out by past users for the use of new users
• Search-style Algorithms
• Genre matching
• Past purchase querying
Problems on using Recommendation System
• Inconclusive user feedback forms
• Finding users to take the feedback surveys
• Weak Algorithms
• Poor results
• Poor Data
• Lack of Data
• Privacy Control (May NOT explicitly collaborate with recipients)
Maintenance
• Costly
• Information becomes outdated
• Information quantity (large, disk space expansion)
The Future of Recommender Systems
• Extract implicit negative ratings through the analysis of returned item.
• How to integrate community with recommendations
• Recommender systems will be used in the future to predict demand for products,
enabling earlier communication back the supply chain.
2. Explain content-based filtering in detail.
CONTENT-BASED RECOMMENDATIONS
• Main idea: Recommend items to customer x similar to previous items rated highly by x
Example:
•Movie recommendations
–Recommend movies with same actor(s), director, genre, …
•Websites, blogs, news
–Recommend other sites with “similar” content
Item Profiles
• For each item, create an item profile
• Profile is a set (vector) of features
o Movies: author, title, actor, director,…
o Text: Set of “important” words in document
• How to pick important features?
o Usual heuristic from text mining is TF-IDF
(Term frequency * Inverse Doc Frequency)
• Term … Feature
• Document … Item
• fij = frequency of term (feature) i in doc (item) j
• ni = number of docs that mention term i
• N = total number of docs
• TF-IDF score: wij = TFij × IDFi
• Doc profile = set of words with highest TF-IDF scores, together with their scores
User Profiles and Prediction
Discovering Features of Documents
• There are many kinds of documents for which a recommendation system can be
useful. For example, there are many news articles published each day, and we cannot
read all of them.
• A recommendation system can suggest articles on topics a user is interested in, but
• how can we distinguish among topics?
• Web pages are also a collection of documents. Can we suggest pages a user might
want to see?
• Likewise, blogs could be recommended to interested users, if we could classify blogs
by topics.
• Unfortunately, these classes of documents do not tend to have readily available
information giving features.
• A substitute that has been useful in practice is the identification of words that
characterize the topic of a document.
• First, eliminate stop words – the several hundred most common words, which tend
to say little about the topic of a document.
• For the remaining words, compute the TF.IDF score for each word in the document.
The ones with the highest scores are the words that characterize the document.
Content-Based Recommenders
• Find me things that I liked in the past.
• Machine learns preferences through user feedback and builds a user profile
• Explicit feedback – user rates items
• Implicit feedback – system records user activity
– Clicksteam data classified according to page category and activity, e.g. browsing a
product page
– Time spent on an activity such as browsing a page
• Recommendation is viewed as a search process, with the user profile acting as the
query and the set of items acting as the documents to match.
Recommending Items to Users Based on Content
• With profile vectors for both users and items, we can estimate the degree to which a
user would prefer an item by computing the cosine distance between the user’s and
item’s vectors.
• As in Example 9.2, we may wish to scale var- ious components whose values
are not boolean.
• The random-hyperplane and locality-sensitive-hashing techniques can be used to
place (just) item profiles in buckets.
• In that way, given a user to whom we want to recommend some items, we can apply
the same two techniques – random hyperplanes and LSH – to determine in which
buckets we must look for items that might have a small cosine distance from the user.
Pros: Content-based Approach
• +: No need for data on other users
– No cold-start or sparsity problems
• +: Able to recommend to users with unique tastes
• +: Able to recommend new & unpopular items
– No first-rater problem
• +: Able to provide explanations
– Can provide explanations of recommended items by listing content-features that
caused an item to be recommended
Cons: Content-based Approach
• –: Finding the appropriate features is hard
– E.g., images, movies, music
• –: Recommendations for new users
– How to build a user profile?
• –: Overspecialization
– Never recommends items outside user’s content profile
– People might have multiple interests
– Unable to exploit quality judgments of other users
3. Explain about Collaborative Filtering in detail.
Introduction u to Collaborative Filtering
• Collaborative filtering leverages product transactions to give recommendations.
• In this type of model, for a specific customer,
– we find similar customers based on transaction history and recommend items that
the customer in question hasn’t purchased yet and which the similar customers tended
to like.
Collaborative Filtering
• Match people with similar interests as a basis for recommendation.
1) Many people must participate to make it likely that a person with similar interests
will be found.
2) There must be a simple way for people to express their interests.
3) There must be an efficient algorithm to match people with similar interests.
Designing Collaborative Filtering
APRIORI
k=1
o Find frequent set Lk from Ck of all candidate itemsets
o Form Ck+1 from Lk; k = k + 1
o Repeat 2-3 until Ck is empty
• Details about steps 2 and 3
o Step 2: scan D and count each itemset in Ck , if it’s greater than minSup, it is frequent
o Step 3: next slide
1
5. What is Sampling data in a stream?
• Sampling from a finite stream is a special case of sampling from a station- ary window
in which the window boundaries correspond to the first and last stream elements.
• The foregoing schemes fall into the category of equal-probability sampling because each
window element is equally likely to be included in the sample.
Frequency Moments
• Consider a stream S = {a1, a2, ..., am} with elements from a domain D = {v1, v2, ...,
vn}.
• Let mi denote the frequency (also sometimes called multiplicity) of value vi ∈ D; i.e.,
the number of times vi appears in S.
• The k th frequency moment of the stream is defined as:
Fk = Xn i=1 mk i (1)
• We will develop algorithms that can approximate Fk by making one pass of the stream
and using a small amount of memory o(n + m).
• Frequency moments have a number of applications.
• F0 represents the number of distinct elements in the streams (which the FM-sketch
from last class estimates using O(log n) space.
• F1 is the number of elements in the stream m.
2
8. What does Real-Time Analytics Platform mean?
• A real-time analytics platform enables organizations to make the most out of real-time
data by helping them to extract the valuable information and trends from it.
• Such platforms help in measuring data from the business point of view in real time,
further making the best use of data.
PART – B
Data Stream
• Large data volume ,likely structured, arriving a very high rate.
• DS is not only you see in YouTube.
Definition
• “A data stream is
– a real-time, continuous, ordered (implicitly by arrival time or explicitly by
timestamp) sequence of items.
– It is impossible to control the order in which items arrive, nor is it feasible to
locally store a stream in its entirety.”
3
The data stream is not necessarily Structured records and Unstructured such as audio or
video data. Data streams are massive volumes of data, records arrive at a high rate
Classificat
Frau ion model
d?
Labeli
Frau
d
4
For example, storing one day’s worth of IP network traffic flows in the AT&T IP backbone
alone may require as much as 2.5TB.
In addition to the above characteristics of data streams, a large number of streaming
applications require near real-time sophisticated analyses.
Stream Classification
• Stream Classification
– Construct a classification model based on past records
– Use the model to predict labels for new data
– Help decision making
Stream Management
• In many data mining situations, we do not know the entire data set in advance
• Stream Management is important when the input rate is controlled externally:
– Google queries
– Twitter or Facebook status updates
– We can think of the data as infinite and non-stationary (the distribution
changes over time)
Data Streaming
5
c. Filtering a data stream
– Select elements with property x from the stream
d. Counting distinct elements
– Number of distinct elements in the last k elements
of the stream
e. Estimating moments
– Estimate avg./std. dev. of last k elements
f. Finding frequent elements
• Most stream data are at pretty low-level or multi-dimensional in nature, needs multi-
level and multi-dimensional processing .
• In addition to the above characteristics of data streams, a large number of streaming
applications require near real-time sophisticated analyses.
• The data stream characteristics are clearly very different from those assumed for data
stored in traditional DBMSs and thus make traditional DBMSs ill-suited for efficient
implementation of many data stream processing applications.
• New data processing techniques are required to monitor and analyze massive volumes
of data streams in real-time.
2. Explain with a neat diagram about Stream data model and its Architecture. (13)
Answer :
6
• Q: How do you make critical calculations about the stream using a
limited amount of (secondary) memory?
Ad-Hoc
Queries
Standing . . . 1, 5, 2, 7, 0, 9, 3
Processor Output
. . . a, r, v, t, y, h, b
. . . 0, 0, 1, 0, 1, 1, 0
time
Limited
Streams Entering.
Archival
Each is stream is
Storage
J. Leskovec, A. Rajaraman, J.
composed of
elements/tuples
Computational Model
A data stream is a (massive) sequence of elements : e1, e2, … en.
Stream processing requirements
Single pass : Each record is examined at most once.
Bounded storage : Limited Memory (M) for storing synopsis.
Real-time: per record processing time (to maintain synopsis) must be low.
7
A general model for data stream mining
8
Stream data Architecture
9
• Stochastic Gradient Descent (SGD) is an
example of a stream algorithm
• In Machine Learning we call this: Online Learning
– Allows for modeling problems where we have a continuous stream of data
– We want an algorithm to learn from it and
slowly adapt to the changes in data
• Idea: Do slow updates to the model
– SGD (SVM, Perceptron) makes small updates
– So: First train the classifier on training data.
– Then: For every example from the stream, we slightly update the model (using
small learning rate)
10
• The industry is moving from painstaking integration of technologies like Kafka and
Storm,
– towards full stack solutions that provide an end-to-end streaming data
architecture.
A Motivating Example
• Again let us start with a running example that illustrates the problem and what we can
do about it.
• Suppose we have a set S of one billion allowed email addresses – those that we will
allow through because we believe them not to be spam. The stream consists of pairs:
an email address and the email itself.
• Since the typical email address is 20 bytes or more, it is not reasonable to store S in
main memory.
• Thus, we can either use disk accesses to determine whether or not to let through any
given stream element, or we can devise a method that requires no more main memory
than we have available, and yet will filter most of the undesired stream elements.
• Suppose for argument’s sake that we have one gigabyte of available main memory. In
the technique known as Bloom filtering, we use that main memory as a bit array. In
this case, we have room for eight billion bits, since one byte equals eight bits.
• Devise a hash function h from email addresses to eight billion buckets. Hash each
member of S to a bit, and set that bit to 1. All other bits of the array remain 0.
11
• Since there are one billion members of S, approximately 1/8th of the bits will be 1.
• The exact fraction of bits set to 1 will be slightly less than 1/8th, because it is possible
that two members of S hash to the same bit.
• The purpose of the Bloom filter is to allow through all stream elements whose keys are
in S, while rejecting most of the stream elements whose keys are not in S.
• To initialize the bit array, begin with all bits 0. Take each key value in S and hash it
using each of the k hash functions. Set to 1 each bit that is hi(K) for some hash
function hi and some key value K in S.
• To test a key K that arrives in the stream, check that all of h1(K), h2(K), . . . , hk(K)
• are 1’s in the bit-array. If all are 1’s, then let the stream element through. If one or
more of these bits are 0, then K could not be in S, so reject the stream element.
12
Analysis of Bloom Filtering
• If a key value is in S, then the element will surely pass through the Bloom filter.
However, if the key value is not in S, it might still pass.
• We need to understand how to calculate the probability of a false positive, as a
function of n, the bit-array length, m the number of members of S, and k, the number
of hash functions.
• The model to use is throwing darts at targets. Suppose we have x targets and y darts.
Any dart is equally likely to hit any target. After throwing the darts, how many targets
can we expect to be hit at least once?
• The analysis is similar to the analysis and goes as follows:
• The probability that a given dart will not hit a given target is (x − 1)/x.
• The probability that none of the y darts will hit a given target is x−1
x _y
.
We can w rite this expression as (1 – 1 x )x( y x ).
• Using the approximation (1−ǫ)1/ǫ = 1/e for small ǫ (recall Section 1.3.5),
we conclude that the probability that none of the y darts hit a given
target is e−y/x.
FILTERING STREAMS
Example:
• Consider the running example of Section 4.3.1. We can use the above calculation to
get the true expected number of 1’s in the bit array.
• Think of each bit as a target, and each member of S as a dart. Then the probability that
a given bit will be 1 is the probability that the corresponding target will be hit by one
or more darts. Since there are one billion members of S, we have y = 109 darts. As
there are eight billion bits, there are x = 8 × 109
• targets. Thus, the probability that a given target is not hit is e−y/x = e−1/8 and the
probability that it is hit is 1 − e−1/8. That quantity is about 0.1175.
• we suggested that 1/8 = 0.125 is a good approximation, which it is, but now we have
the exact calculation. 2
• We can apply the rule to the more general situation, where set S has m members, the
array has n bits, and there are k hash functions. The number of targets is x = n, and the
number of darts is y = km. Thus, the probability that a bit remains 0 is e−km/n. We
want the fraction of 0 bits to be fairly large, or else the probability that a nonmember
of S will hash at least once to a 0 becomes too small, and there are too many false
13
positives. For example, we might choose k, the number of hash functions to be n/m or
less.
• Then the probability of a 0 is at least e−1 or 37%. In general, the probability of a false
• positive is the probability of a 1 bit, which is 1 − e−km/n, raised to the kth power, i.e.,
(1 − e−km/n)k.
Example : we found that the fraction of 1’s in the array of our running example is
0.1175, and this fraction is also the probability of a false positive. That is, a
nonmember of S will pass through the filter if it hashes to a 1, and the probability of it
doing so is 0.1175.
• Suppose we used the same S and the same array, but used two different hash functions.
This situation corresponds to throwing two billion darts at eight billion targets, and the
probability that a bit remains 0 is e−1/4. In order to be a false positive, a nonmember
of S must hash twice to bits that are 1, and this probability is (1 − e−1/4)2, or
approximately 0.0493. Thus, adding a second hash function for our running example is
an improvement, reducing the false-positive rate from 0.1175 to 0.0493. 2
4. Explain the concept of Estimating Moments (13)
• The generalization of the problem of counting distinct elements in a stream is vital role
on streaming Management.
• The problem, called computing “moments,” involves the distribution of frequencies of
different elements in the stream.
• The streaming Management includes how well we are managing moments of all orders
and how best we concentrate on computing second moments, from which the general
algorithm for all moments is a simple extension.
Definition of Moments
Suppose a stream consists of elements chosen from a universal set. Assume the universal set
is ordered so we can speak of the ith element for any i.
Then the kth-order moment (or just kth moment) of the stream is the sum over all i of (mi) k .
Example:
• The 0th moment is the sum of 1 for each mi that is greater than 0.4
• That is, the 0th moment is a count of the number of distinct elements in the stream.
14
• We can use the method of Section 4.4 to estimate the 0th moment of a stream.
• Technically, since mi could be 0 for some elements in the universal set, we need to
make explicit in the definition of “moment” that 00 is taken to be 0.
• For moments 1 and above, the contribution of mi’s that are 0 is surely 0.
• The 1st moment is the sum of the mi ’s, which must be the length of the stream.
• Thus, first moments are especially easy to compute; just count the length of the stream
seen so far.
• The second moment is the sum of the squares of the mi ’s.
• It is sometimes called the surprise number, since it measures how uneven the
distribution of elements in the stream is.
• To see the distinction, suppose we have a stream of length 100, in which eleven
different elements appear.
• The most even distribution of these eleven elements would have one appearing 10
times and the other ten appearing 9 times each.
• In this case, the surprise number is 102 + 10 × 9 2 = 910.
• At the other extreme, one of the eleven elements could appear 90 times and the other
ten appear 1 time each.
• Then, the surprise number would be 902 + 10 × 1 2 = 8110.
Frequency Moments
• Consider a stream S = {a1, a2, ..., am} with elements from a domain D = {v1, v2, ...,
vn}.
• Let mi denote the frequency (also sometimes called multiplicity) of value vi ∈ D; i.e.,
the number of times vi appears in S.
• The k th frequency moment of the stream is defined as:
Fk = Xn i=1 mk i (1)
• We will develop algorithms that can approximate Fk by making one pass of the stream
and using a small amount of memory o(n + m).
• Frequency moments have a number of applications.
• F0 represents the number of distinct elements in the streams (which the FM-sketch
from last class estimates using O(log n) space.
• F1 is the number of elements in the stream m.
There is no problem computing moments of any order if we can afford to keep in main
memory a count for each element that appears in the stream.
However, also as in that section, if we cannot afford to use that much memory, then we need
to estimate the kth moment by keeping a limited number of values in main memory and
computing an estimate from these values.
15
For the case of distinct elements, each of these values were counts of the longest tail produced
by a single hash function.
Example :
Suppose the stream is a, b, c, b, d, a, c, d, a, b, d, c, a, a, b.
• The length of the stream is n = 15.
• Since a appears 5 times, b appears 4 times, and c and d appear three times each, the
second moment for the stream is 5 2 + 42 + 32 + 32 = 59.
• Suppose we keep three variables, X1, X2, and X3 assume that at “random” we pick the
3rd, 8th, and 13th positions to define these three variables.
• When we reach position 3, we find element c, so we set X1.element = c and X1.value
= 1.
• Position 4 holds b, so we do not change X1. Likewise, nothing happens at positions 5
or 6. At position 7, we see c again, so we set X1.value = 2.
• At position 8 we find d, and so set X2.element = d and X2.value = 1.
• Positions 9 and 10 hold a and b, so they do not affect X1 or X2.
• Position 11 holds d so we set X2.value = 2, and
• position 12 holds c so we set X1.value = 3.
• At position 13, we find element a, and so set X3.element = a and X3.value = 1.
• Then, at position 14 we see another a and so set X3.value = 2.
• Position 15, with element b does not affect any of the variables, so we are done, with
final values X1.value = 3 and X2.value = X3.value = 2.
16
• ✷ We can derive an estimate of the second moment from any variable X. This
estimate is n(2X.value − 1).
Example:
We can prove that the expected value of any variable constructed as in Section 4.5.2 is the
second moment of the stream from which it is constructed.
Let e(i) be the stream element that appears at position i in the stream, and let c(i) be the
number of times element e(i) appears in the stream among positions i, i + 1, . . . , n.
Example:
Consider the stream of Example 4.7. e(6) = a, since the 6th position holds a.
Also, c(6) = 4, since a appears at positions 9, 13, and 14, as well as at position 6.
Note that a also appears at position 1, but that fact does not contribute to c(6).
The expected value of n(2X.value − 1) is the average over all positions i between 1 and n of
n(2c(i) − 1), that is E n(2X.value − 1) = 1 n Xn i=1 n(2c(i) − 1)
We can simplify the above by canceling factors 1/n and n, to get E n(2X.value − 1) = Xn i=1
2c(i).
However, to make sense of the formula, we need to change the order of summation by
grouping all those positions that have the same element.
For instance, concentrate on some element a that appears ma times in the stream.
17
The positions with a before that yield terms 5, 7, and so on, up to 2ma − 1, which is the term
for the first position in which a appears.
That is, the formula for the expected value of 2X.value − 1 can be written: E n(2X.value − 1)
= X a 1 + 3 + 5 + · · · + (2ma − 1)
The proof is an easy induction on the number of terms in the sum. Thus, E n(2X.value − 1) =
P a (ma) 2 , which is the definition of the second moment.
✷ Suppose the data representing the contents of the window is whatever sequence of bits
represents both w and x.
• Ask the query “how many 1’s are in the last k bits?” The query-answering algorithm
will produce the same answer, whether the window contains w or x, because the
algorithm can only see their representation.
• But the correct answers are surely different for these two bit-strings.
• Thus, we have proved that we must use at least N bits to answer queries about the last
k bits for any k. In fact, we need N bits, even if the only query we can ask is “how
many 1’s are in the entire window of length N?”
• The argument is similar to that used above. Suppose we use fewer than N bits to
represent the window, and therefore we can find w, x, and k as above.
• It might be that w and x have the same number of 1’s, as they did in both cases of
Example 4.10.
18
• However, if we follow the current window by any N − k bits, we will have a situation
where the true window contents resulting from w and x are identical except for the
leftmost bit, and therefore, their counts of 1’s are unequal.
• However, since the representations of w and x are the same, the representation of the
window must still be the same if we feed the same bit sequence to these
representations. Thus, we can force the answer to the query “how many 1’s in the
window?” to be incorrect for one of the two possible window contents.
Example 4.11 : Figure 4.2 shows a bit stream divided into buckets in a way
that satisfies the DGIM rules.
At the right (most recent) end we see two buckets of size 1. To its left we see one bucket of
size 2. Note that this bucket covers four positions, but only two of them are 1. Proceeding left,
we see two buckets of size 4, and we suggest that a bucket of size 8exists further left.
Notice that it is OK for some 0’s to lie between buckets. Also, observe from Fig. 4.2 that the
buckets do not overlap; there are one or two of each size up to the largest size, and sizes only
increase moving left. 2
19
Query Answering in the DGIM Algorithm
Suppose we are asked how many 1’s there are in the last k bits of the window, for some 1 ≤ k
≤ N. Find the bucket b with the earliest timestamp that includes at least some of the k most
recent bits. Estimate the number of 1’s to be the sum of the sizes of all the buckets to the right
(more recent) than bucket b, plus half the size of b itself.
Example
• Suppose the stream is that of Fig. 4.2, and k = 10.
• Then the query asks for the number of 1’s in the ten rightmost bits, which happen to be
0110010110. Let the current timestamp (time of the rightmost bit) be t.
• Then the two buckets with one 1, having timestamps t − 1 and t − 2 are completely
included in the answer.
• The bucket of size 2, with timestamp t − 4, is also completely included.
• However, the rightmost bucket of size 4, with timestamp t−8 is only partly included.
• We know it is the last bucket to contribute to the answer, because the next bucket to its
left has timestamp less than t − 9 .
******************
Decaying Windows
• We have assumed that a sliding window held a certain tail of the stream, either the
most recent N elements for fixed N, or all the elements that arrived after some time in
the past.
• Sometimes we do not want to make a sharp distinction between recent elements and
those in the distant past, but want to weight the recent elements more heavily.
20
Definition of the Decaying Window
• An alternative approach is to redefine the question so that we are not asking for a count
of 1’s in a window.
• Rather, let us compute a smooth aggregation of all the 1’s ever seen in the stream, with
decaying weights, so the further back in the stream, the less weight is given.
• Formally, let a stream currently consist of the elements a1, a2, . . . , at, where a1 is the
first element to arrive and at is the current element. Let c be a small constant, such as
10−6 or 10−9.
Define the exponentially decaying window for this stream to be the sum
t−1
X i=0
at−i(1 − c)i
• The effect of this definition is to spread out the weights of the stream elements as far
back in time as the stream goes.
• In contrast, a fixed window with the same sum of the weights, 1/c, would put equal
weight 1 on each of the most recent 1/c elements to arrive and weight 0 on all previous
elements.
Figure 4.4:
• A decaying window and a fixed-length window of equal weight It is much easier to
adjust the sum in an exponentially decaying window than in a sliding window of fixed
length. In the sliding window, we have to worry about the element that falls out of the
window each time a new element arrives.
• That forces us to keep the exact elements along with the sum, or to use an
approximation scheme such as DGIM.
21
However, when a new element at+1 arrives at the stream input, all we need to do is:
1. Multiply the current sum by 1 − c.
2. Add at+1.
• The reason this method works is that each of the previous elements has now moved
one position further from the current element, so its weight is multiplied by 1 − c.
• Further, the weight on the current element is (1 − c)0 = 1, so adding at+1 is the correct
way to include the new element’s contribution.
• We imagine that the number of possible movies in the stream is huge, so we do not
want to record values for the unpopular movies.
• Therefore, we establish a threshold, say 1/2, so that if the popularity score for a movie
goes below this number, its score is dropped from the counting.
• For reasons that will become obvious, the threshold must be less than 1, although it
can be any number less than 1.
• This example should be taken with a grain of salt, because, as we pointed out, there
aren’t enough different movies for this technique to be essential.
22
• Imagine, if you will, that the number of movies is extremely large, so counting ticket
sales of each one separately is not feasible.
Why RTAP?
• Decision systems that go beyond visual analytics have an intrinsic need to analyze data
and respond to situations.
• Depending on the sophistication, such systems may have to act rapidly on
incoming information, grapple with heterogeneous knowledge bases, work across
multiple domains, and often in a distributed manner.
• Big Data platforms offer programming and software infrastructure to help perform
• analytics to support the performance and scalability needs of such decision
support systems for IoT domains.
• There has been significant focus on Big Data analytics platforms on the volume
dimension of Big Data.
• In such platforms, such as MapReduce, data is staged and aggregated over time, and
analytics are performed in a batch mode on these large data corpus.
• These platforms weakly scale with the size of the input data, as more distributed
compute resources are made available.
23
• However, as we have motivated before, IoT applications place an emphasis on online
analytics, where data that arrives rapidly needs to be processed and analyzed with low
latency to drive autonomic decision making.
Apache Flink
• Flink is an open-source platform that handles distributed stream and batch data
processing.
• At its core is a streaming data engine that provides for data distribution, fault tolerance,
and communication, for undertaking distributed computations over the data streams.
• In the last year, the Apache Flink community saw three major version releases for the
platform and the community event Flink Forward in San Francisco.
• Apache Flink contains several APIs to enable creating applications that use the Flink
engine. Some of the most popular APIs on the platform are-
• DataStream API for unbounded streams, DataSet API for static data embedded in
Python, Java, and Scala, and the Table API with a SQL-like language.
Spark Streaming
• Apache Spark is used to build scalable and fault-tolerant streaming applications.
• With Spark Streaming, you get to use Apache Spark's language-integrated API which
lets you write streaming jobs in the similar way as you write batch jobs.
• Spark Streaming supports the three languages- Java, Scala, Python. Apache Spark is
being used in various leading industries today, such as- Healthcare, Finance, e-
commerce, Media and Entertainment, Travel industry, etc.
• The popularity of Apache Spark adds the glitter to the platform Spark Streaming.
IBM Streams
• This streaming analytics platform from IBM enables the applications developed by
users to gather, analyze, and correlate information that comes to them from a variety of
sources.
• The solution is known to handle high throughput rates and up to millions of events and
messages per second, making it a leading proprietary streaming analytics solution for
real-time applications.
• IBM Stream computing helps analyze large streams of data in the form of unstructured
texts, audio, video, and geospatial, and allows for organizations to spot risks and
opportunities and make efficient decisions.
24
Software AG's Apama Streaming Analytics
• Apama Streaming analytics platform is built for streaming analytics and automated
action on fast-moving data on the basis of intelligent decisions.
• The software bundles up other aspects like messaging, event processing, in-memory
data management and visualization and is ideal for fast-moving Big Data Analytics
Solutions. Sensors that bring in loads of data from different sources can be churned
using this solution in real-time.
• With Apama, you can act on high-volume business operations in real-time.
Thus the real-time Analytics Platform is useful for Real time application development and its
successful implementations.
PART – C
1. CASE STUDY - STOCK MARKET PREDICTION
You are required to make a case study on STOCK MARKET PREDICTION with
following requirements:
i) Briefly introduce about Stock market and its prediction
ii) The Solution Path of the stock Market Perdition.
iii) Do the Empirical Study of the Stock Market Prediction.
25
• Predicting stock performance is a very large and profitable area of study
• Many companies have developed stock predictors based on neural networks
• This technique has proven successful in aiding the decisions of investors
• Can give an edge to beginning investors who don’t have a lifetime of experience
26
Using all words (in naïve bayes) works well for some tasks
Finding subsets of words may help in other tasks
Hand-built polarity lexicons
Use seeds and semi-supervised learning to induce lexicons
The most important for predicting stock market prices are neural networks because they are
able to learn nonlinear-mappings between inputs and outputs.
It may be possible to perform better than traditional analysis and other computer-based
methods with the neural-networks ability to learn-nonlinear, chaotic-systems.
27
28
Time Series Analysis Vs NN
TS - Instructions and rules are central
TS – A Mathematical formula define the dynamics
NN - Don't perform according to preset rules. Learns from regularities and sets its own rules
NN – Not described explicitly in mathematical terms
Benifits with NN
• Generalization ability and robustness
29
• Mapping of input/output
• No assumptions of model has to be made
• Flexilbilty
Drawbacks with NN
• Black-box property
• Overfitting
• Expertise for choice of input
• Training takes a lot of time
30
Stock data is highly complex and hard to model, therefore a non-linear model is benecial
A large set of interacting input series is often required to explain a specfic stock, which suites
neural network
The idea is to use the previous model error as additional information to the system
Recurrent system is described as
st = f (st-1 , ut ) state transition
yt = g (st) output equation
Developing ECNN
31
Functions f and g are not specied, yt is the computed output and st describes the state.
st = f (st-1 , ut , yt-1 - ytd )
yt = g (st )
ytd is observed data
f and g can be stated as
Adding NN Role
We implement a NN
st = N(st-1 , ut , yt-1-yt-1d ; v )
yt = N(st ; w)
Now optimisation problem is
We apply an activation function
st = tanh(Ast-1 + But + D(Cst-1-yt-1d ))
yt = C (st )
Final ECNN
32
Learning Algorithm
• A good prediction algorithm should outperform the naive algorithm, i.e. predicted
value of stock in next time step is same as the present value.
• Naive algorithm is a direct consequence of Efficient Market Hypothesis which states
that the current market price is an assimilation of all information available therefore
no changes of future changes can be made.
Terms
• Rkt is the k-step return at time t.
Performance measures
Hit Rate - accounts the number of times direction of the stock is same as predicted
Return of investment – takes into account the sign and the quantity of actual return
Realised Potential – shows:
• how much of the total
• movement algorithm
• successfully identifies.
33
EMPIRICAL STUDY OF THE STOCK MARKET ANALYSIS
Empirical Study
• Well traded stocks with a reasonable spread are considered.
• Certain time invariant structures are identified and learnt quickly so in latter part of
the training some weights are frozen.
• Occurrences of invariant structures were more evident in weekly rather than daily
structures.
Data series
Closing price y
Highest price during the day yH .
Lowest price during the day yL .
Volume V , the total amount of stocks traded during the day.
Training procedure
Data was divided into 3 subsets Training set, Validation set, Generalization set.
Weights were initialized uniformly in the range [-1,1].
After training waights associated with the best performance in the Validation set were
selected and applied to Generalization set to get final results.
34
One day forecast of Swedish stock Exchange
weekly forecast
Daily Prediction
35
Thus the Stock market prediction is depends on the human information processing
capabilities and the nature and efficiency of the technique used for the same.
***************
36
• Politics: what do people think about this candidate or issue?
• Prediction: predict election outcomes or market trends from sentiment
Detection of Friendliness
• Friendly speakers use collaborative conversational style
– Laughter
– Less use of negative emotional words
– More sympathy
• That’s too bad I’m sorry to hear that
– More agreement
• I think so too
– Less hedges
• kind of sort of a little …
37
– Rank the attitude of this text from 1 to 5
• Advanced:
– Detect the target, source, or complex attitude types
38
Brendan O'Connor, Ramnath Balasubramanyan, Bryan R. Routledge, and Noah A. Smith. 2010.
From Tweets to Polls: Linking Text Sentiment to Public Opinion Time Series. In ICWSM-2010
A Baseline Algorithm
• Tokenization
• Feature Extraction
• Classification using different classifiers
– Naïve Bayes
– MaxEnt
– SVM
39
• Twitter mark-up (names, hash tags)
• Capitalization (preserve for
words in all caps)
• Phone numbers, dates
• Emoticons
• Useful code:
– Christopher Potts sentiment tokenizer
– Brendan O’Connor twitter tokenizer
Potts emoticons
Negation
Add NOT_ to every word between negation and following punctuation:
didn’t like this movie , but I
didn’t NOT_like NOT_this NOT_movie but I
40
Binarized (Boolean feature) Multinomial Naïve Bayes
• Intuition:
– For sentiment (and probably for other text classification domains)
– Word occurrence may matter more than word frequency
• The occurrence of the word fantastic tells us a lot
• The fact that it occurs 5 times may not tell us much more.
– Boolean Multinomial Naïve Bayes
• Clips all the word counts in each document at 1
|docsj |
P(cj )¬
|total # documents|
Calculate P(wk | cj) terms
Textj single doc containing all docsj
For each word wk in Vocabulary
nk # of occurrences of wk in Textj
nk + a
P(wk | c j ) ¬
n + a | Vocabulary |
41
Cross-Validation
• Break up data into 10 folds
– (Equal positive and negative inside each fold?)
• For each fold
– Choose the fold as a temporary test set
– Train on 9 folds, compute performance on the test fold
• Report average performance of the 10 runs
Iteration
1 Test Training
2 Test Training
4 Training Test
5 Training Test
Thus the sentiment analysis requires the much needed attention to get good recommendation for the
given scenario.
42
CS8091 BIG DATA ANALYTICS
UNIT V NO SQL DATA MANAGEMENT FOR BIG DATA AND VISUALIZATION
QUESTION BANK
PART A
1. Define NO SQL Database.
A NoSQL (originally referring to "non SQL" or "non relational") database provides a
mechanism for storage and retrieval of data that is modelled in means other than the
tabular relations used in relational databases. Such databases have existed since the late
1960s, but did not obtain the "NoSQL" moniker until a surge of popularity, triggered
by the needs of Web 2.0 companies. NoSQL databases are increasingly used in big data
and real-time web applications.[5] NoSQL systems are also sometimes called "Not only
SQL" to emphasize that they may support SQL-like query languages, or sit alongside
SQL databases in polyglot persistent architectures.
2. What is Schema-less Model?
schema less means the database don't have fixed data structure, such as MongoDB, it
has JSON-style data store, you can change the data structure as you wish.
3. Define Key Value Stores.
A key-value store is a database which uses an array of keys where each key is associated
with only one value in a collection. It is quite similar to a dictionary or a map data
structure. Key-value stores can be considered as the most primary and the simplest
version of all the databases.
4. Define Document Stores.
A document-oriented database, or document store, is a computer program designed for
storing, retrieving and managing document-oriented information, also known as semi-
structured data
Document-oriented databases are one of the main categories of NoSQL databases, and
the popularity of the term "document-oriented database" has grown[2] with the use of
the term NoSQL itself. XML databases are a subclass of document-oriented databases
that are optimized to work with XML documents. Graph databases are similar, but add
another layer, the relationship, which allows them to link documents for rapid traversal.
5. Define Tabular Stores.
A tabular database, as the name implies is a database that is structured in a tabular form.
It arranges data elements in vertical columns and horizontal rows. Each cell is formed
by the intersection of a column and row. Each row and column is uniquely numbered
to make it orderly and efficient.
6. Define Object Data Stores.
Object storage (also known as object-based storage) is a computer data storage architecture
that manages data as objects, as opposed to other storage architectures like file systems which
manages data as a file hierarchy, and block storage which manages data as blocks within sectors
and tracks.
7. What is Hive?
•Hive is a data ware house system for Hadoop.
•It runs SQL like queries called HQL (Hive query language) which gets internally converted t
map reduce jobs.
•Hive was developed by Facebook.
•Hive supports Data definition Language(DDL), Data Manipulation Language(DML) and user
defined functions
8. What is Sharding?
Sharding is a database architecture pattern related to horizontal partitioning — the
practice of separating one table's rows into multiple different tables, known as
partitions. Each partition has the same schema and columns, but also entirely different
rows.
9. What is Hbase?
HBase is an open-source non-relational distributed database modeled after Google's
Bigtable and written in Java. It is developed as part of Apache Software Foundation's
Apache Hadoop project and runs on top of HDFS or Alluxio, providing Bigtable-like
capabilities for Hadoop.
10. Difference Between Hbase and Hive.
HBase Hive
• Despite providing SQL functionality • Apache HBase is a NoSQL
key/value store which runs on top
of HDFS.
• Unlike Hive, HBase operations run in real- • Hive does not fully runs in real-
time on its database rather than MapReduce time
jobs.
• HBase provides Interactive query. • , Hive does not provide interactive
querying yet - it only runs batch
processes on Hadoop.
PART B
1. Explain in detail about HIVE.
INTRODUCTION TO HIVE
• Apache Hive is a data ware house system for Hadoop that runs SQL like queries called HQL
(Hive query language) which gets internally converted to map reduce jobs.
• Hive was developed by Facebook.
• It supports Data definition Language, Data Manipulation Language and user defined
functions.
• Before learning Hive, you must have the knowledge of Hadoop and Java.
What is HIVE?
• Hive is a data ware house system for Hadoop.
• It runs SQL like queries called HQL (Hive query language) which gets internally converted
to map reduce jobs.
• Hive was developed by Facebook.
• Hive supports Data definition Language(DDL), Data Manipulation Language(DML) and user
defined functions.
DDL: create table, create index, create views.
DML: Select, Where, group by, Join, Order By
Pluggable Functions:
UDF: User Defined Function
UDAF: User Defined Aggregate Function
UDTF: User Defined Table Function
Points to remember
Hive's metastore is used to persist schema i.e. table definition(table name, columns, types),
location of table files, row format of table files, storage format of files.
Hive Query Language is similar to SQL and gets reduced to map reduce jobs in backend.
Hive's default database is derby.
Hive Components
• Shell: allows interactive queries
• Driver: session handles, fetch, execute
• Compiler: parse, plan, optimize
• Execution engine: DAG of stages (MR, HDFS, metadata)
• Metastore: schema, location in HDFS, SerDe
• Hadoop is great for large-data processing!
– But writing Java programs for everything is verbose and slow
– Not everyone wants to (or can) write Java code
• Solution: develop higher-level data processing languages
– Hive: HQL is like SQL
– Pig: Pig Latin is a bit like Perl
Data Model
• Tables
– Typed columns (int, float, string, boolean)
– Also, list: map (for JSON-like data)
• Partitions
– For example, range-partition tables by date
• Buckets
– Hash partitions within ranges (useful for sampling, join optimization)
Metastore
• Database: namespace containing a set of tables
• Holds table definitions (column types, physical layout)
• Holds partitioning information
• Can be stored in Derby, MySQL, and many other relational databases
Physical Layout
• Warehouse directory in HDFS
– E.g., /user/hive/warehouse
• Tables stored in subdirectories of warehouse
– Partitions form subdirectories of tables
• Actual data stored in flat files
– Control char-delimited text, or SequenceFiles
– With custom SerDe, can use arbitrary format
Hive: Example
• Hive looks similar to an SQL database
• Relational join on two tables:
– Table of word counts from Shakespeare collection
– Table of word counts from the bible
Load Load
Join
url = url
Group by user
Filter
System-Level Dataflow
Pages
load
... ...
canonicalize
join by url
...
group by user
...
filter
Thus the HIVE data flow mechanism is used for data storage and retrieval make easy.
2. Give a Survey on Analyzing Big data with Twitter.
• Twitter (www.twitter.com) is an online social networking and micro blogging
platform that enables a user to send and read short 140 character text messages, called
“tweets”.
Twitter has now become a global platform for mass communication. The data it
generates is finding many uses:-
• in disaster management,
• brand management (understanding brand strengths, sentiments, visibility),
• politics (campaign management),
• personal (giving updates to family/friends) etc.
Features of Twitter
• It is ranked amongst the top 10 most visited websites by Alexa’s web traffic
analysis for the last 2 years.
• It has ~250+ million monthly active users who send about ~500+ million
tweets per day (ref. Wikipedia). 78% of the twitter users access it on a mobile device
and 77% are living outside US.
• It supports 35+ languages. Its vast global reach and usage has made Twitter
into a remarkable platform for analysing and understanding trends and events shaping
the world either on a global or a local scale.
• It is a great medium to understand ground realities of global events (like
natural disasters, terror-attacks) as users tweet their impressions even before news
crews are able to reach the areas.
• Given the reach, businesses, politicians, and celebrities are using it to market
their messages / views / opinions to their followers. The volume, velocity and variety
of data that gets generated on twitter fits the description of big data. We now explore
some approaches on analysing the same.
• Map Reduce - is a programming model for processing large data sets with a
parallel, distributed algorithm on a cluster.
• It allows developers to write programs that process massive amounts of
unstructured data in parallel across a distributed cluster or even standalone computers.
• Originally, it was developed at Google for indexing Web pages but it has now
become a generic term. It has two main functions – Map and Reduce.
b) Sentiment Analysis – looking for keywords about brands and analysing them to
compute a score of sentiment for that brand.
Graph Theory is the study of graphs, which are mathematical structures used to model
pairwise relations between objects.
A "graph" is made up of "vertices" or "nodes" and lines called edges that connect
them.
A graph may be undirected, meaning that there is no distinction between the two
vertices associated with each edge, or its edges may be directed from one vertex to
another (ref. Wikipedia).
Relationships in a social network are treated like graphs.
For example, User (A) is a friend of User (B),User (B) is a friend of User (A) and a
follower of User (C). We can depict this relationship like a graph such as the one
shown in figure1.
Unlike Facebook, where a connection is bi-directional (if A is a friend of B then B is a
friend of A – both are friends and follow each other),
Twitter’s relationships are asymmetric.
On Twitter, if A is following B then A is a friend of B, but B is not friend of A.
A will get tweets of B reporting on his home page but B will not see tweet of A until
he/she explicitly starts to follow A.
We can use concepts of Graph Theory to analyze twitter data.
d) Shortest Path / Most Network Connections – Allow us to figure out who are
most influences. And we can use it for other analysis as well – depth of network etc.
a) Twitter Counter:
• Sstarted as a self-funded start up in Amsterdam in 2008.
• It is a third party application that provides analytics of twitter usage.
• It allows some basic features as free use and other features tagged as premium
which comes with subscription.
b) Twitonomy: -
• It provides a good basic statistics free and for other features user needs to sign
up for a pro account
c) Twtrland:-
• It provides some niche features like best time to tweet free along with
additional statistics, but keeps things like export, prediction, comparison in its paid
version.
d) SocialBro: –
• It provides free access to all its features but only for 2 weeks, after which user
needs to pay subscription to continue using the services.
Key Features of Twitter Analytics Tools
We have categorized features being provided by various players into following
buckets and then mapped the tools for their availability / non availability in both free /
paid versions.
Big Data Ecommerce Case Studies to understand use of Big Data in Ecommerce sector
• Here are some of the most successful case studies in ecommerce and retail industry
which will inspire you to use data even more correctly.
• These companies are the market leaders in their niche and have reported some huge
profit in business after using big data.
• Let’s start with the Big Data case ecommerce case studies and few big data retail case
studies where big data is providing ROI.
The below successful big data ecommerce case studies show how Ecommerce industry are
using Big Data, Hadoop, Machine Learning, and Analytics to scale the business.
Alibaba Big Data Case Study
• This China-based billion dollar Ecommerce Company and world’s largest retailer has
revealed how big data has helped them to increase the revenue.
• It has also helped them to use the data in the offline sector and another retail sector to
grow the business.
• According to Danfeng Li (Alibaba’s director of big data and technologies), Alibaba’s
various properties hold about 80% of China’s PC, internet and app data, which it can
integrate with first-party data to create singularly powerful data models.
• According to the report by Reuters, the number of mobile-first users have been
increased to 42% and has reached to 410 Million. This accounts for the total of 73% GMV.
Check this WARC report for more details.