Professional Documents
Culture Documents
Hive
Hive
Sqoop: It is used to import and export data to and from between HDFS and RDBMS.
Pig: It is a procedural language platform used to develop a script for MapReduce
operations.
Hive: It is a platform used to develop SQL type scripts to do MapReduce operations.
The traditional approach using Java MapReduce program for structured, semi-structured,
and unstructured data.
The scripting approach for MapReduce to process structured and semi structured data
using Pig.
The Hive Query Language (HiveQL or HQL) for MapReduce to process structured data
using Hive.
What is Hive
Hive is a data warehouse infrastructure tool to process structured data in Hadoop. It resides on
top of Hadoop to summarize Big Data, and makes querying and analyzing easy.
Initially Hive was developed by Facebook, later the Apache Software Foundation took it up and
developed it further as an open source under the name Apache Hive. It is used by different
companies. For example, Amazon uses it in Amazon Elastic MapReduce.
Hive is not
A relational database
A design for OnLine Transaction Processing (OLTP)
A language for real-time queries and row-level updates
Features of Hive
It stores schema in a database and processed data into HDFS.
It is designed for OLAP.
It provides SQL type language for querying called HiveQL or HQL.
It is familiar, fast, scalable, and extensible.
Architecture of Hive
The following component diagram depicts the architecture of Hive:
This component diagram contains different units. The following table describes each unit:
The following table defines how Hive interacts with Hadoop framework:
7 Internally, the process of execution job is a MapReduce job. The execution engine
sends the job to JobTracker, which is in Name node and it assigns this job to
TaskTracker, which is in Data node. Here, the query executes MapReduce job.
Metadata Ops
7.1
Meanwhile in execution, the execution engine can execute metadata operations with
Metastore.
Fetch Result
8
The execution engine receives the results from Data nodes.
Send Results
9
The execution engine sends those resultant values to the driver.
Send Results
10
The driver sends the results to Hive Interfaces.
1. Hive Clients
2. Hive Services
3. Hive Storage and Computing
Hive Clients:
Hive provides different drivers for communication with a different type of applications. For
Thrift based applications, it will provide Thrift client for communication.
For Java related applications, it provides JDBC Drivers. Other than any type of applications
provided ODBC drivers. These Clients and drivers in turn again communicate with Hive server
in the Hive services.
Hive Services:
Client interactions with Hive can be performed through Hive Services. If the client wants to
perform any query related operations in Hive, it has to communicate through Hive Services.
CLI is the command line interface acts as Hive service for DDL (Data definition Language)
operations. All drivers communicate with Hive server and to the main driver in Hive services as
shown in above architecture diagram.
Driver present in the Hive services represents the main driver, and it communicates all type of
JDBC, ODBC, and other client specific applications. Driver will process those requests from
different applications to meta store and field systems for further processing.
Hive services such as Meta store, File system, and Job Client in turn communicates with Hive
storage and performs the following actions
Metadata information of tables created in Hive is stored in Hive "Meta storage database".
Query results and data loaded in the tables are going to be stored in Hadoop cluster on
HDFS.
From the above screenshot we can understand the Job execution flow in Hive with Hadoop
EE should first contacts Name Node and then to Data nodes to get the values stored in
tables.
EE is going to fetch desired records from Data Nodes. The actual data of tables resides in
data node only. While from Name Node it only fetches the metadata information for the
query.
It collects actual data from data nodes related to mentioned query
Execution Engine (EE) communicates bi-directionally with Meta store present in Hive to
perform DDL (Data Definition Language) operations. Here DDL operations like
CREATE, DROP and ALTERING tables and databases are done. Meta store will store
information about database name, table names and column names only. It will fetch data
related to query mentioned.
Execution Engine (EE) in turn communicates with Hadoop daemons such as Name node,
Data nodes, and job tracker to execute the query on top of Hadoop file system
Hive Continuously in contact with Hadoop file system and its daemons via Execution engine.
The dotted arrow in the Job flow diagram shows the Execution engine communication with
Hadoop daemons.
Local mode
Map reduce mode
If the Hadoop installed under pseudo mode with having one data node we use Hive in this
mode
If the data size is smaller in term of limited to single local machine, we can use this mode
Processing will be very fast on smaller data sets present in the local machine
If Hadoop is having multiple data nodes and data is distributed across different node we
use Hive in this mode
It will perform on large amount of data sets and query going to execute in parallel way
Processing of large data sets with better performance can be achieved through this mode
Below are the key features of Hive that differ from RDBMS.
Hive resembles a traditional database by supporting SQL interface but it is not a full
database. Hive can be better called as data warehouse instead of database.
Hive enforces schema on read time whereas RDBMS enforces schema on write time.
In RDBMS, a table’s schema is enforced at data load time, If the data being
loaded doesn’t conform to the schema, then it is rejected. This design is called schema on write.
But Hive doesn’t verify the data when it is loaded, but rather when a
it is retrieved. This is called schema on read.
Schema on read makes for a very fast initial load, since the data does not have to be read,
parsed, and serialized to disk in the database’s internal format. The load operation is just a file
copy or move.
Schema on write makes query time performance faster, since the database can index columns
and perform compression on the data but it takes longer to load data into the database.
Hive is based on the notion of Write once, Read many times but RDBMS is designed
for Read and Write many times.
In RDBMS, record level updates, insertions and deletes, transactions and indexes are
possible. Whereas these are not allowed in Hive because Hive was built to operate over
HDFS data using MapReduce, where full-table scans are the norm and a table update is
achieved by transforming the data into a new table.
In RDBMS, maximum data size allowed will be in 10’s of Terabytes but whereas Hive
can 100’s Petabytes very easily.
RDBMS is best suited for dynamic data analysis and where fast responses are expected
but Hive is suited for data warehouse applications, where relatively static data is
analyzed, fast response times are not required, and when the data is not changing rapidly.
To overcome the limitations of Hive, HBase is being integrated with Hive to support
record level operations and OLAP.
Hive is very easily scalable at low cost but RDBMS is not that much scalable that too it
is very costly scale up.
Data Units
In the order of granularity - Hive data is organized into:
Databases: Namespaces function to avoid naming conflicts for tables, views, partitions,
columns, and so on. Databases can also be used to enforce security for a user or group of
users.
Tables: Homogeneous units of data which have the same schema. An example of a table
could be page_views table, where each row could comprise of the following columns
(schema):
o timestamp—which is of INT type that corresponds to a UNIX timestamp of
when the page was viewed.
o userid —which is of BIGINT type that identifies the user who viewed the page.
o page_url—which is of STRING type that captures the location of the page.
o referer_url—which is of STRING that captures the location of the page from
where the user arrived at the current page.
o IP—which is of STRING type that captures the IP address from where the page
request was made.
Partitions: Each Table can have one or more partition Keys which determines how the
data is stored. Partitions—apart from being storage units—also allow the user to
efficiently identify the rows that satisfy a specified criteria; for example, a date_partition
of type STRING and country_partition of type STRING. Each unique value of the
partition keys defines a partition of the Table. For example, all "US" data from "2009-12-
23" is a partition of the page_views table. Therefore, if you run analysis on only the "US"
data for 2009-12-23, you can run that query only on the relevant partition of the table,
thereby speeding up the analysis significantly. Note however, that just because a partition
is named 2009-12-23 does not mean that it contains all or only data from that date;
partitions are named after dates for convenience; it is the user's job to guarantee the
relationship between partition name and data content! Partition columns are virtual
columns, they are not part of the data itself but are derived on load.
Buckets (or Clusters): Data in each partition may in turn be divided into Buckets based
on the value of a hash function of some column of the Table. For example the
page_views table may be bucketed by userid, which is one of the columns, other than the
partitions columns, of the page_view table. These can be used to efficiently sample the
data.
In Hive, the database is considered as a catalog or namespace of tables. So, we can maintain
multiple tables within a database where a unique name is assigned to each table. Hive also
provides a default database with a name default.
Initially, we check the default database provided by Hive. So, to check the list of existing
databases, follow the below command: -
Each database must contain a unique name. If we create two databases with the same
name, the following error generates: -
If we want to suppress the warning generated by Hive on creating the database with the
same name, follow the below command: -
Hive also allows assigning properties with the database in the form of key-value pair.
import java.sql.SQLException;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.Statement;
import java.sql.DriverManager;
Class.forName(driverName);
// get connection
Connection con =
DriverManager.getConnection("jdbc:hive://localhost:10000/default", "", "");
Statement stmt = con.createStatement();
con.close();
}
}
In this section, we will see various ways to drop the existing database.
Let's check the list of existing databases by using the following command: -
In Hive, it is not allowed to drop the database that contains the tables directly. In such a
case, we can drop the database either by dropping tables first or use Cascade keyword
with the command.
Let's see the cascade command used to drop the database:-
In Hive, we can create a table by using the conventions similar to the SQL. It supports a wide
range of flexibility where the data files for tables are stored. It provides two types of table: -
Internal table
External table
Internal Table
The internal tables are also called managed tables as the lifecycle of their data is controlled by
the Hive. By default, these tables are stored in a subdirectory under the directory defined by
hive.metastore.warehouse.dir (i.e. /user/hive/warehouse). The internal tables are not flexible
enough to share with other tools like Pig. If we try to drop the internal table, Hive deletes both
table schema and data.
1. hive> create table demo.employee (Id int, Name string , Salary float)
2. row format delimited
3. fields terminated by ',' ;
Let's see the metadata of the created table by using the following command:-
Let's see the result when we try to create the existing table again.
In such a case, the exception occurs. If we want to ignore this type of exception, we can use if
not exists command while creating the table.
1. hive> create table if not exists demo.employee (Id int, Name string , Salary float)
2. row format delimited
3. fields terminated by ',' ;
While creating a table, we can add the comments to the columns and can also define the
table properties.
hive> create table demo.new_employee (Id int comment 'Employee Id', Name string com
ment 'Employee Name', Salary float comment 'Employee Salary')
Let's see the metadata of the created table by using the following command: -
like demo.employee;
Here, we can say that the new table is a copy of an existing table.
External Table
The external table allows us to create and access a table and a data externally. The external
keyword is used to specify the external table, whereas the location keyword is used to determine
the location of loaded data.
As the table is external, the data is not present in the Hive directory. Therefore, if we try to drop
the table, the metadata of the table will be deleted, but the data still exists.
hive> create external table emplist (Id int, Name string , Salary float)
location '/HiveDirectory';
Once the internal table has been created, the next step is to load the data into it. So, in Hive, we
can easily load data from any file to the database.
Let's load the data of the file into the database by using the following command: -
1. load data local inpath '/home/codegyani/hive/emp_details' into table demo.employee;
Now, we can use the following command to retrieve the data from the database.
https://www.javatpoint.com/hive-load-data