Professional Documents
Culture Documents
Big Data Analytics Using Hadoop
Big Data Analytics Using Hadoop
ABSTRACT iii. Velocity: Increasing use of the online space meant that the
This paper is an effort to present the basic understanding of data that was available was rapidly changing and therefore
BIG DATA is and it’s usefulness to an organization from the had to be made available and used at the right time to be
performance perspective. Along-with the introduction of BIG effective [4].
DATA, the important parameters and attributes that make this
emerging concept attractive to organizations has been
1.1 The Challenges For Big Firms
highlighted. The paper also evaluates the difference in the Big data may be new for startups and for online firms, but
challenges faced by a small organization as compared to a many large firms view it as something they have been
medium or large scale operation and therefore the differences wrestling with for a while. Some managers appreciate the
in their approach and treatment of BIG DATA. A number of innovative nature of big data, but more find it “business as
application examples of implementation of BIG DATA across usual” or part of a continuing evolution toward more data.
industries varying in strategy, product and processes have They have been adding new forms of data to their systems and
been presented. The second part of the paper deals with the models for many years, and don’t see anything revolutionary
technology aspects of BIG DATA for it’s implementation in about big data [5]. Put another way, many were pursuing big
organizations. Since HADOOP has emerged as a popular tool data before big data was big.
for BIG DATA implementation, the paper deals with the When these managers in large firms are impressed by big
overall architecture of HADOOP alongwith the details of it’s data, it’s not the “bigness” that impresses them. Instead it’s
various components. Further each of the components of the one of three other aspects of big data: the lack of structure, the
architecture has been taken up and described in detail. opportunities presented, and low cost of the technologies
involved. This is consistent with the results from a survey of
Keywords more than fifty large companies by New Vantage Partners in
Big data, hadoop, analytic databases, analytic applications. 2012 [6]. It found, according to the survey summary:
1. INTRODUCTION 1.2 It’s About Variety, Not Volume
Companies across the world have been using data since a long The survey indicates companies are focused on the variety of
time to help them take better decisions in order to enhance data, not its volume, both today and in three years. The most
their performances. It is the first decade of the 21st century important goal and potential reward of Big Data initiatives is
that actually showcased a rapid shift in the availability of data the ability to analyze diverse data sources.
and it’s applicability for improving the overall effectiveness
of the organization. This change that was to revolutionize the Application areas and implementation examples:
use of data brought into advent the concept that became 1. Big Data for cost reduction: Some organizations that are
popular as Big Data [1]. pursuing Big data believe strongly that for the storage of large
What is Big Data: Big data is the availability of a large data that is structured, Big data technologies like Hadoop
amount of data which becomes difficult to store, process and clusters are very cost effective solutions that can be efficiently
mine using a traditional database primarily because of the data utilized for cost reduction [7].
available is large, complex, unstructured and rapidly changing One company’s cost comparison, for example, estimated that
[2]. This is probably one of the important reasons why the the cost of storing one terabyte for a year was $37,000 for a
concept of Big data was first embraced by online firms like traditional relational database, $5,000 for a database
Google, eBay, Facebook, Linkedin etc. appliance, and only $2,000 for a Hadoop cluster.1 Of course,
Big Data in small v/s big companies: There is a specific these figures are not directly comparable, in that the more
reason as to why Big data was first appreciated by the online traditional technologies may be somewhat more reliable and
firms and start-ups as mentioned above. These companies easily managed. Data security approaches, for example, are
were built around the concept of using rapidly changing data not yet fully developed in the Hadoop cluster environment [8].
and did not probably face the challenge of integrating the new
and unstructured data with the already available ones [3]. If 1.3 Big Data At Ups
we look at the challenges regarding Big data being faced by UPS is no stranger to big data, having begun to capture and
the online firms and the start-ups we can highlight the track a variety of package movements and transactions as
following: early as the 1980s. The company now tracks data on 16.3
million packages per day for 8.8 million customers, with an
i. Volume: The largeness of the data available made it a average of 39.5 million tracking requests from customers per
challenge as it was neither possible nor efficient to handle day. The company stores over 16 peta-bytes of data [9].
such a large volume of data using traditional databases.
Much of its recently acquired big data, however, comes from
ii. Variety: As compared to the earlier versions, where data telematic sensors in over 46,000 vehicles. The data on UPS
was available in one or two forms (possibly text and tables), package cars (trucks), for example, includes their speed,
the current versions would mean data being available direction, braking, and drive train performance [10]. The data
additionally in the form of pictures, videos, tweets etc. is not only used to monitor daily performance, but to drive a
major redesign of UPS drivers’ route structures. This
1
International Journal of Computer Applications (0975 – 8887)
Volume 108 – No 12, December 2014
initiative, called ORION (On-Road Integrated Optimization Groups You May Like, Jobs You May Be Interested In,
and Navigation), is arguably the world’s largest operations Who’s Viewed My Profile, and several others. These
research project. It also relies heavily on online map data, and offerings have brought millions of new customers to
will eventually reconfigure a driver’s pickups and drop-offs in LinkedIn.
real time. The project has already led to savings in 2011 of
more than 8.4 million gallons of fuel by cutting 85 million Another strong contender for the best at developing products
miles off of daily routes [11]. UPS estimates that saving only and services based on big data is Google. This company, of
one daily mile driven per driver saves the company $30 course, uses big data to refine its core search and ad-serving
million, so the overall dollar savings are substantial. The algorithms. Google is constantly developing new products and
company is also attempting to use data and analytics to services that have big data algorithms for search or ad
optimize the efficiency of its 2000 aircraft flights per day placement at the core, including Gmail, Google Plus, Google
[12]. Apps, etc. Google even describes the self-driving car as a big
data applications. Some of these product developments pay
1.4 Big Data For Time Reduction off, and some are discontinued, but there is no more prolific
The second common objective of big data technologies and creator of such offerings than Google.
solutions is time reduction. Macy’s merchandise pricing 1.5.1 Example Of Big Data For New Offerings
optimization application provides a classic example of
Caesars (formerly Harrah’s) Entertainment has long been a
reducing the cycle time for complex and large-scale analytical
leader in the use of analytics, particularly in the area of
calculations from hours or even days to minutes or seconds
customer loyalty, marketing, and service. Today, Caesars is
[13]. The department store chain has been able to reduce the
augmenting these traditional analytics capabilities with some
time to optimize pricing of its 73 million items for sale from
big data technologies and skills. The primary objective of
over 27 hours to just over 1 hour. Described by some as “big
exploring and implementing big data tools is to respond in
data analytics,” this capability set obviously makes it possible
real time for customer marketing and service.
for Macy’s to re-price items much more frequently to adapt to
changing conditions in the retail marketplace. This big data For example, the company has data about its customers from
analytics application takes data out of a Hadoop cluster and its Total Rewards loyalty program, web clickstreams, and
puts it into other parallel computing and in-memory software from real-time play in slot machines. It has traditionally used
architectures [14]. Macy’s also says it achieved 70% hardware all those data sources to understand customers, but it has been
cost reductions. Kerem Tomak, VP of Analytics at difficult to integrate and act on them in real time, while the
Macys.com, is using similar approaches to time reduction for customer is still playing at a slot machine or in the resort.
marketing offers to Macy’s customers. He notes that the
company can run a lot more models with this time savings. In order to pursue this objective, Caesars has acquired both
Hadoop clusters and open-source and commercial analytics
1.5 Big Data For New Offerings software. It has also added some data scientists to its analytics
An organization can use big data to develop new products and group.
offerings to it’s customers. This is particularly effective for There are other goals for the big data capabilities as well.
organizations that use the online space for it’s products and Caesars pays fanatical attention—typically through human
services. With access to a large amount of data and the real observation—to ensuring that its most loyal customers don’t
time need of the customers, an organization can not only add wait in lines. With video analytics on big data tools, it may be
value to it’s existing offerings but also develop new offerings able to employ more automated means for spotting service
to match the needs of it’s customers. issues involving less frequent customers [15]. Caesars is also
The best example may be LinkedIn, which has used big data beginning to analyze mobile data, and is experimenting with
and data scientists to develop a broad array of product targeted real-time offers to mobile devices.
offerings and features, including People You May Know,
1.6 Big Data For Improving Process
Efficiency
Big data can be used for improving the process efficiency
also. An excellent use of Big data in this regard is cricket
specially with the advent of the Indian Premier League (IPL).
Not only are matches analysed using the data available in
order to formulate future strategies but even minute details
like the performance of a bowler against a particular batsman
and that too on a particular ground under certain conditions
are being made available for the stakeholders to improve their
efficiency.
For example, how will a batsman like Glenn Maxwell perform
against a bowler like Sunil Narine at Eden Gardens or how
different will it be at Mohali in Chandigarh is available to be
used. Not only this but also data like how many balls has a
Fig 1: The figure shows the cluster where data can be particular batsman faced against a particular bowler the
inserted or capped. number of dot balls and the number of runs scored [16].
Another example in this regard is the use of Big data to
predict the probability (at any time during a match) of a team
2
International Journal of Computer Applications (0975 – 8887)
Volume 108 – No 12, December 2014
Pig is another one built by yahoo, it’s a high level data flow Pig Runtime : which competes pig latin and converts it into
language to pull data out of clusters and now Pig and hive are map reduce job to submit to cluster.
under the Hadoop Map Reduce job submitted to cluster. This Hive : is another Data access project extremely popular like
the beauty of open source framework people can built, add pig. Hive is a way to project structures on to data inside a
and community keeps on growing in Hadoop more cluster so it is really a database. Data-warehouse built on top
technologies and projects are added into Hadoop ecosystem of Hadoop and it contains a query language Hive QL like
[19]. SQL to hive query language and it is extremely similar to sql.
Hive is similar thing like pig. It converts these queries into
map reduce jobs that gets submitted to cluster.
3
International Journal of Computer Applications (0975 – 8887)
Volume 108 – No 12, December 2014
Moving Down to Technology stack we have: format that we can either store in our disk or send across the
wires so that another application can unpack it and deserialize
Data storage: Remember out of the box is a batch processing into a format that they can understand. Avro is more generic
system. We put the data into HDFS system; once we read in
many time or what if we needed to get specific data; What if Thrift is more specific for creating flexible schemas that work
we want to do real time processing system on top of Hadoop with hadoop data. Its specific because it is meant for cross-
data and that’s why we have some of the column oriented language compatibility, so we build an application with
database known as Hbase these are just Appache projects but hadoop data in java and if we want to use same object inside
there a buzz term for this NoSQL. Not once SQL that wants it an application that you built on Ruby, Python or C++.
stands for does not mean you can’t use sql like language to get
data out. What it means the underlying structure of the Data Intelligence: We also have data intelligence in the form
database are not strict like they are in relational world very of Drill and Mahout.
loose, very flexible which makes them very scalable : that’s Drill: Drill is actually an incubator project and is designed to
what we need in the world of Big data and Hadoop, infact do interactive analysis on nested data.
those are lot of NoSQL database platform out here. One of the
most popular is Mangodb. Mahout: Mahout is a machine learning library that concurs
the three Cs :
Mongodb is extremely very popular, especially among
programmers because it is really very easy to work with. It is 1. Recommendation (Collaborative Filtering)
document style storage model which means programmers can 2. Clustering (which is a way to group related text and
take data models and clone. We call objects in those documents)
applications and serialize them right into Mongodb and with
the same ease can bring them back into application [20][21]. 3. Classification (which is a way to categorize related text and
documents).
Hbase was based on Google Big table, which is a way we can
create table which contains millions of rows and we can put So Amazon uses all this stuff to a further recommendation
indexes on them and can do serious data analysis and Hbase is like music sites uses to recommend songs you listen and also
data analysis we put indexing on them and go to high to do predictive analysis.
performance which seeks to find data which we are looking
Sqoop: On the left side of Figure 2. We have Sqoop. It is a
for and nice thing about Hbase is pig and hive natively agree
widely popular project because it is easy to integrate hadoop
with Hbase. So you write pig and hive queries against data
with relational systems. For instance, we have result of map
sitting in HBase table [23][24].
reduce. Rather than taking these results and putting them on
Cassandra is designed to handle large amount of data across HDFS, and require Pig and Hive query, we can send those
many commodity servers, providing high availability with no results to relational world so that a data professional can do
single point of failure. Cassandra offers robust support for their own analysis and become the part of process. So, Sqoop
clusters spanning multiple data centres. It has its root in is popular for pushing hadoop data into relational world, but it
Amazon with more data storage table and it is designed for is also popular for pushing data from relational world into
real time interactive transaction processing on top of our hadoop, like archiving [25].
hadoop cluster. So both of them solve different problems but
Flume and Chukwa: are real time log processing tools so
they both require seeking against our hadoop data.
that we can set up our frame-work where our applications,
We also have random collection of projects that span of operating systems, services like web-services that generate
different categories some of these solve specific business mountains of log data. It’s a way we can push real time data
problems and some of them are little more generic like- information right into hadoop and we can also do real time
analysis.
Hcatalog: It is known as metadata table and storage
management system. What does it mean it’s a way to create a Over the right hand side of Figure 2., we have a tool for
shared schema, it’s a way for tools like Pig, Hive for managing, monitoring and orchestrating all the things that go
interoperable also to have consistent view of data across those in our cluster:
tools.
Zoo Keeper: It is a distributed service coordinate. So it’s a
Lucent: Lucent is there for full text searching an API loaded way in which we keep our all services running across all our
with algorithms to do things like standard full text searching, cluster synchronous. So, it handles all synchronization and
wild card searching, phrase searching, range searching kind of serialization. It also gives centralized management for these
stuff. services.
Hama: Hama is there for BSP (Book Synchronous Oozie: It is a work flow library that allows us to play, and to
Processing). Here we need to work with large amount of connect lot of those essential projects for instances, Pig, Hive
scientific data. and Sqoop.
Crunch: Crunch is there for writing and testing and running Ambari: It allows you to provision a cluster which means that
map reduce pipeline. It essentially gives you full control to we can install services, so that we can pick Pig, Hive, Sqoop,
overall four phrases, which is going to be: 1. Map, 2. Reduce, Hbase and install it. It will go across all the nodes in cluster
3. Shuffle, and 4. Combine. It is there to help in joining and and also we can manage our services from one centralized
aggregation that is very hard to do in low level map reduce, so location like starting up, stopping, reconfiguring and we can
Crunch is there to make you little more easier inside map also monitor a lot of these projects from Ambari.
reduce pipeline.
Avro: These are data serialization technology which is a way
in which we can take data from application, package up into a
4
International Journal of Computer Applications (0975 – 8887)
Volume 108 – No 12, December 2014
IJCATM : www.ijcaonline.org 5