HP Vertica 7.1.x AnalyzingData
HP Vertica 7.1.x AnalyzingData
Legal Notices
Warranty
The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be
construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.
The information contained herein is subject to change without notice.
Copyright Notice
Copyright 2006 - 2015 Hewlett-Packard Development Company, L.P.
Trademark Notices
Adobe is a trademark of Adobe Systems Incorporated.
Microsoft and Windows are U.S. registered trademarks of Microsoft Corporation.
UNIX is a registered trademark of The Open Group.
Page 2 of 181
Contents
Contents
Writing Queries
11
11
11
Temporary Tables
13
SQL Queries
13
Simple Queries
13
Joins
14
Cross Joins
14
Subqueries
15
Sorting Queries
16
16
Subqueries
18
19
22
22
23
In Place of an Expression
25
Comparison Operators
25
26
27
Notes
27
ANY Subqueries
27
28
ALL Subqueries
29
Page 3 of 181
Analyzing Data
Contents
30
IN and NOT IN
32
34
35
35
36
37
38
39
Flattening Views
40
Examples
40
41
UPDATE Subqueries
41
DELETE Subqueries
46
Subquery Examples
48
Single-Row Subqueries
48
Multiple-Row Subqueries
48
Multicolumn Subqueries
49
51
Subquery Restrictions
Joins
51
54
54
55
Inner Joins
57
58
Natural Joins
60
Cross Joins
61
Page 4 of 181
Analyzing Data
Contents
Outer Joins
63
63
64
64
Range Joins
65
Key Ranges
65
Slowly-Changing Dimensions
66
67
69
71
72
72
74
77
Window Partitioning
77
Syntax
77
Examples
77
78
78
78
Window Ordering
79
Window Framing
81
Syntax
81
81
82
Examples
Windows with a Logical Offset (RANGE)
82
86
Page 5 of 181
Analyzing Data
Contents
Reporting Aggregates
91
91
92
Naming Windows
97
99
99
99
100
101
102
105
Event-Based Windows
106
107
109
111
112
115
117
Constant Interpolation
118
119
119
119
Linear Interpolation
120
GFI Examples
121
Constant Interpolation
122
Linear Interpolation
123
124
Page 6 of 181
Analyzing Data
Contents
124
Using slice_time
124
125
127
127
129
130
130
130
131
131
132
132
133
133
134
135
135
137
137
141
142
142
142
143
144
144
Page 7 of 181
Analyzing Data
Contents
145
145
145
147
147
149
149
151
153
153
154
156
156
Grouping Functions
156
157
157
158
158
158
159
160
160
161
161
162
163
163
Page 8 of 181
Analyzing Data
Contents
163
163
165
167
167
169
169
169
171
172
174
174
175
176
177
177
178
178
179
180
181
Page 9 of 181
Analyzing Data
Contents
Page 10 of 181
Writing Queries
Queries are database operations that retrieve data from one or more tables or views. In HP Vertica,
the top-level SELECT statement is the query, and a query nested within another SQL statement is
called a subquery.
HP Vertica is designed to run the same SQL standard queries that run on other databases.
However, there are some differences between HP Vertica queries and queries used in other
relational database management systems.
The HP Vertica transaction model is different from the SQL standard in a way that has a profound
effect on query performance. You can:
l
Run a query on a static snapshot of the database from any specific date and time. Doing so
avoids holding locks or blocking other database operations.
Use a subset of the standard SQL isolation levels and access modes (read/write or read-only)
for a user session.
In HP Vertica, the primary structure of a SQL query is its statement. Each statement ends with a
semicolon, and you can write multiple queries separated by semicolons; for example:
=> CREATE TABLE t1( ..., date_col date NOT NULL, ...);
=> CREATE TABLE t2(..., state VARCHAR NOT NULL, ...);
Page 11 of 181
Analyzing Data
Writing Queries
Syntax
HP Vertica can run a query from a snapshot of the database taken at a specific date and time or at a
specific epoch. The syntax is:
AT TIME 'timestamp' select...
AT EPOCH epoch_number select...
AT EPOCH LATEST select...
The command queries all data in the database up to and including the specified epoch or the epoch
representing the specified date and time, without holding a lock or blocking write operations. The
specified TIMESTAMP and epoch_number values must be greater than or equal to the Ancient
History Mark epoch.
Parameters
AT EPOCH LATEST
Queries all committed data in the database up to, but not including, the
current epoch.
AT EPOCH epoch_number
Queries all data in the database up to and including the specified epoch
without holding a lock or blocking write operations.
AT TIME 'timestamp'
Historical queries, also known as snapshot queries, are useful because they access data in past
epochs only. Historical queries do not need to hold table locks or block write operations because
they do not return the absolute latest data. Their content is private to the transaction and valid only
for the length of the transaction.
Historical queries behave in the same manner regardless of transaction isolation level. Historical
queries observe only committed data, even excluding updates made by the current transaction,
unless those updates are to a temporary table.
Be aware that there is only one snapshot of the logical schema. This means that any changes you
make to the schema are reflected across all epochs. If, for example, you add a new column to a
table and you specify a default value for the column, all historical epochs display the new column
and its default value.
Page 12 of 181
Analyzing Data
Writing Queries
See Also
l
Transactions
AboutQueryExecution
Temporary Tables
You can use the CREATE TEMPORARY TABLE statement to implement certain queries using multiple
steps:
1. Create one or more temporary tables.
2. Execute queries and store the result sets in the temporary tables.
3. Execute the main query using the temporary tables as if they were a normal part of the logical
schema.
See CREATE TEMPORARY TABLE in the SQL Reference Manual for details.
SQL Queries
All DML (Data Manipulation Language) statements can contain queries. This section introduces
some of the query types in HP Vertica, with additional details in later sections.
Note: Many of the examples in this chapter use the VMart schema. For information about other
HP Vertica-supplied queries, see the Getting Started Guide.
Simple Queries
Simple queries contain a query against one table. Minimal effort is required to process the following
query, which looks for product keys and SKU numbers in the product table:
=> SELECT product_key, sku_number FROM public.product_dimension;
product_key | sku_number
-------------+----------43
| SKU-#129
87
| SKU-#250
42
| SKU-#125
49
| SKU-#154
37
| SKU-#107
36
| SKU-#106
Page 13 of 181
Analyzing Data
Writing Queries
86
41
88
40
(10 rows)
|
|
|
|
SKU-#248
SKU-#121
SKU-#257
SKU-#120
Joins
Joins use a relational operator that combines information from two or more tables. The query's ON
clause specifies how tables are combined, such as by matching foreign keys to primary keys. In
the following example, the query requests the names of stores with transactions greater than 70 by
joining the store key ID from the store schema's sales fact and sales tables:
=> SELECT store_name, COUNT(*) FROM store.store_sales_fact
JOIN store.store_dimension ON store.store_sales_fact.store_key = store.store_
dimension.store_key
GROUP BY store_name HAVING COUNT(*) > 70 ORDER BY store_name;
store_name | count
------------+------Store49
|
72
Store83
|
78
(2 rows)
For more detailed information, see Joins. See also the Multicolumn subqueries section in Subquery
Examples.
Cross Joins
Also known as the Cartesian product, a cross join is the result of joining every record in one table
with every record in another table. A cross join occurs when there is no join key between tables to
restrict records. The following query, for example, returns all instances of vendor and store names
in the vendor and store tables:
=> SELECT vendor_name, store_name FROM public.vendor_dimension
CROSS JOIN store.store_dimension;
vendor_name
| store_name
--------------------+-----------Deal Warehouse
| Store41
Deal Warehouse
| Store12
Deal Warehouse
| Store46
Deal Warehouse
| Store50
Deal Warehouse
| Store15
Deal Warehouse
| Store48
Deal Warehouse
| Store39
Sundry Wholesale
| Store41
Sundry Wholesale
| Store12
Sundry Wholesale
| Store46
Page 14 of 181
Analyzing Data
Writing Queries
Sundry Wholesale
Sundry Wholesale
Sundry Wholesale
Sundry Wholesale
Market Discounters
Market Discounters
Market Discounters
Market Discounters
Market Discounters
Market Discounters
Market Discounters
Market Suppliers
Market Suppliers
Market Suppliers
Market Suppliers
Market Suppliers
Market Suppliers
Market Suppliers
...
(4000 rows)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Store50
Store15
Store48
Store39
Store41
Store12
Store46
Store50
Store15
Store48
Store39
Store41
Store12
Store46
Store50
Store15
Store48
Store39
...
This example's output is truncated because this particular cross join returned several thousand
rows. See also Cross Joins.
Subqueries
A subquery is a query nested within another query. In the following example, we want a list of all
products containing the highest fat content. The inner query (subquery) returns the product
containing the highest fat content among all food products to the outer query block (containing
query). The outer query then uses that information to return the names of the products containing
the highest fat content.
=> SELECT product_description, fat_content FROM public.product_dimension
WHERE fat_content IN
(SELECT MAX(fat_content) FROM public.product_dimension
WHERE category_description = 'Food' AND department_description = 'Bakery')
LIMIT 10;
product_description
| fat_content
-------------------------------------+------------Brand #59110 hotdog buns
|
90
Brand #58107 english muffins
|
90
Brand #57135 english muffins
|
90
Brand #54870 cinnamon buns
|
90
Brand #53690 english muffins
|
90
Brand #53096 bagels
|
90
Brand #50678 chocolate chip cookies |
90
Brand #49269 wheat bread
|
90
Brand #47156 coffee cake
|
90
Brand #43844 corn muffins
|
90
(10 rows)
Page 15 of 181
Analyzing Data
Writing Queries
Sorting Queries
Use the ORDER BY clause to order the rows that a query returns.
Partitioning on a FLOAT type could return nondeterministic results because of the precision,
especially when the numbers are close to one another, such as results from the RADIANS()
function, which has a very small range of output.
To get deterministic results, use NUMERIC if you must partition by data that is not an INTEGER
type.
If you add MAX(promotion_key) to analytic ORDER BY, the results are the same on any machine:
=> SELECT COUNT(*) FROM (SELECT DISTINCT MAX(store.store_sales_fact.promotion_key),
NTILE(79) OVER(PARTITION BY MAX(store.store_sales_fact.cost_dollar_amount)
ORDER BY store.store_sales_fact.product_version,
MAX(store.store_sales_fact.promotion_key))
Page 16 of 181
Analyzing Data
Writing Queries
FROM store.store_sales_fact
GROUP BY store.store_sales_fact.product_version,
store.store_sales_fact.sales_dollar_amount) AS store;
Page 17 of 181
Analyzing Data
Writing Queries
Subqueries
Subqueries provide a great deal of flexibility to SQL statements by letting you perform in one step
what, otherwise, would require several steps. For example, instead of having to write separate
queries to answer multiple-part questions, you can write a subquery.
A subquery is a SELECT statement within another SELECT statement. The inner statement is the
subquery, and the outer statement is the containing statement (often referred to in HP Vertica as
the outer query block).
Like any query, a subquery returns records from a table that could consist of a single column and
record, a single column with multiple records, or multiple columns and records. Queries can be
noncorrelated or correlated. You can even use them to update or delete records in a table based on
values that are stored in other database tables.
Notes
l
Many examples in this section use the VMart database. See Introducing the VMart Example
Database.
Page 18 of 181
Analyzing Data
Writing Queries
Syntax
< search_condition > {
[ { AND | OR [ NOT ] } { < predicate > | ( < search_condition > ) } ]
} [ ,... ]
< predicate >
{ expression comparison-operator expression
... | string-expression [ NOT ] { LIKE | ILIKE | LIKEB | ILIKEB } string-expression
...
...
...
...
|
|
|
|
expression
expression
expression
expression
IS [ NOT ] NULL
[ NOT ] IN ( subquery | expression [ ,...n ] )
comparison-operator [ ANY | SOME ] ( subquery )
comparison-operator ALL ( subquery )
Parameters
<search-condition>
a query expression
a subquery
Page 19 of 181
Analyzing Data
Writing Queries
{ AND | OR | NOT }
<predicate>
expression
Page 20 of 181
Analyzing Data
Writing Queries
comparison-operator
string_expression
IS [ NOT ] NULL
ALL
Page 21 of 181
Analyzing Data
Writing Queries
ANY | SOME
[ NOT ] EXISTS
[ NOT ] IN
Page 22 of 181
Analyzing Data
Writing Queries
Examples
The following list shows some of the ways you can filter complex conditions in the WHERE clause:
l
Subquery expression:
=> SELECT * FROM t WHERE x=(SELECT x FROM t)+1 OR x<>(SELECT x FROM t)+1;
Page 23 of 181
Analyzing Data
Writing Queries
=> SELECT * FROM t WHERE (SELECT * FROM t)IS NULL OR (SELECT * FROM tt)IS NULL;
OR expression with boolean column and subquery that returns Boolean data type:
=> SELECT * FROM t2 WHERE x = (SELECT x FROM t2) OR x;
Analytic function, NULL-handling function, string function, math function, and so on:
=> SELECT x FROM t WHERE x > (SELECT COALESCE (x,y) FROM t GROUP BY x,y) OR
x < 9;
Multicolumn subqueries:
=> SELECT * FROM t WHERE (x,y) = (SELECT x,y FROM t) OR x > 5;
Page 24 of 181
Analyzing Data
Writing Queries
See Also
l
Subquery Restrictions
In Place of an Expression
Subqueries that return a single value (unlike a list of values returned by IN subqueries) can be used
just about anywhere an expression is allowed in SQL. It can be a column name, a constant, a
function, a scalar subquery, or a combination of column names, constants, and functions
connected by operators or subqueries.
For example:
=> SELECT c1 FROM t1
=> SELECT c1 FROM t1
=> SELECT c1 FROM t1
COALESCE((t1.c1
HP Vertica returns an error on queries where more than one row would be returned by any subquery
used as an expression:
=> SELECT c1 FROM t1 WHERE c1 = (SELECT c1 FROM t2) ORDER BY c1;
ERROR: more than one row returned by a subquery used as an expression
See Also
l
Subquery Restrictions
Comparison Operators
HP Vertica supports Boolean subquery expressions in the WHERE clause with any of the following
operators: (>, <, >=, <=, =, <>, <=>).
WHERE clause subqueries filter results and take the following form:
SELECT <column, ...> FROM <table>
WHERE <condition> (SELECT <column, ...> FROM <table> WHERE <condition>);
Page 25 of 181
Analyzing Data
Writing Queries
These conditions are available for all data types where comparison makes sense. All Comparison
Operators are binary operators that return values of TRUE, FALSE, or UNKNOWN (NULL).
Expressions that correlate to just one outer table in the outer query block are supported, and these
correlated expressions can be comparison operators.
The following subquery scenarios are supported:
SELECT * FROM T1 WHERE T1.x = (SELECT MAX(c1) FROM T2);
SELECT * FROM T1 WHERE T1.x >= (SELECT MAX(c1) FROM T2 WHERE T1.y = T2.c2);
SELECT * FROM T1 WHERE T1.x <= (SELECT MAX(c1) FROM T2 WHERE T1.y = T2.c2);
See Also
l
Subquery Restrictions
The following command searches for customers whose company name starts with "Ev" and returns
the total count:
=> SELECT COUNT(*) FROM customer_dimension WHERE customer_name LIKE
(SELECT 'Ev%' FROM customer_dimension LIMIT 1);
count
------153
(1 row)
HP Vertica also supports single-row subqueries as the pattern argument for LIKEB and ILIKEB
predicates; for example:
=> SELECT * FROM t1 WHERE t1.x LIKEB (SELECT t2.x FROM t2);
LIKE
LIKEB
ILIKE
ILIKEB
NOT LIKE
NOT LIKEB
NOT ILIKE
NOT IILIKEB
Page 26 of 181
Analyzing Data
Writing Queries
Note: The ESCAPE keyword is not valid for the above symbols.
See LIKE-predicate in the SQL Reference Manual for additional examples.
Notes
l
IN is equivalent to = ANY.
ANY Subqueries
Subqueries that use the ANY keyword yield a Boolean result when any value retrieved in the
subquery matches the value of the lefthand expression.
Expression
Returns
> ANY(1,10,100)
Returns TRUE for any value > 1 (greater than at least one value or greater than
the minimum value)
< ANY(1,10,100)
Returns TRUE for any value < 100 (less than at least one value or less than
the maximum value)
= ANY(1,10,100)
Returns TRUE for any value = 1 or 10 or 100 (equals any of the values)
Page 27 of 181
Analyzing Data
Writing Queries
An ANY subquery within an expression. Note that the second statement uses the SOME
keyword:
=> SELECT c1 FROM t1 WHERE COALESCE((t1.c1 > ANY (SELECT c1 FROM t2)), TRUE);
=> SELECT c1 FROM t1 WHERE COALESCE((t1.c1 > SOME (SELECT c1 FROM t2)), TRUE);
Note that omitting the ANY keyword returns an error because more than one row would be
returned by the subquery used as an expression:
=> SELECT c1 FROM t1 WHERE c1 = (SELECT c1 FROM t2) ORDER BY c1;
Page 28 of 181
Analyzing Data
Writing Queries
ALL Subqueries
Subqueries that use the ALL keyword yield a Boolean result when all values retrieved in the
subquery match the specified condition of the lefthand expression.
Expression
Returns
> ALL(1,10,100)
Returns the expression value > 100 (greater than the maximum value)
< ALL(1,10,100)
Returns the expression value < 1 (less than the minimum value)
See Also
l
Subquery Restrictions
Page 29 of 181
Analyzing Data
Writing Queries
The EXISTS condition is considered to be met if the subquery returns at least one row. Since the
result depends only on whether any records are returned, and not on the contents of those records,
the output list of the subquery is normally uninteresting. A common coding convention is to write all
EXISTS tests as follows:
EXISTS (SELECT 1 WHERE ...)
In the above fragment, SELECT 1 returns the value 1 for every record in the query. If the query
returns, for example, five records, it returns 5 ones. The system doesn't care about the real values
in those records; it just wants to know if a row is returned.
Alternatively, a subquerys select list that uses EXISTS might consist of the asterisk (*). You do not
need to specify column names, because the query tests for the existence or nonexistence of
records that meet the conditions specified in the subquery.
EXISTS (SELECT * WHERE ...)
Notes
l
Examples
The following query retrieves the list of all the customers who purchased anything from any of the
stores amounting to more than 550 dollars:
=> SELECT customer_key, customer_name, customer_state
FROM public.customer_dimension WHERE EXISTS
Page 30 of 181
Analyzing Data
Writing Queries
Whether you use EXISTS or IN subqueries depends on which predicates you select in outer and
inner query blocks. For example, to get a list of all the orders placed by all stores on January 2, 2003
for vendors with records in the vendor table:
=> SELECT store_key, order_number, date_ordered
FROM store.store_orders_fact WHERE EXISTS
(SELECT 1 FROM public.vendor_dimension
WHERE public.vendor_dimension.vendor_key = store.store_orders_fact.vendor_key)
AND date_ordered = '2003-01-02';
store_key | order_number | date_ordered
-----------+--------------+-------------37 |
2559 | 2003-01-02
16 |
552 | 2003-01-02
35 |
1156 | 2003-01-02
13 |
3885 | 2003-01-02
25 |
554 | 2003-01-02
21 |
2687 | 2003-01-02
49 |
3251 | 2003-01-02
19 |
2922 | 2003-01-02
26 |
1329 | 2003-01-02
40 |
1183 | 2003-01-02
(10 rows)
The above query looks for existence of the vendor and date ordered. To return a particular value,
rather than simple existence, the query looks for orders placed by the vendor who got the best deal
on January 4, 2004:
=> SELECT store_key, order_number, date_ordered
FROM store.store_orders_fact ord, public.vendor_dimension vd
WHERE ord.vendor_key = vd.vendor_key AND vd.deal_size IN
(SELECT MAX(deal_size) FROM public.vendor_dimension)
AND date_ordered = '2004-01-04';
store_key | order_number | date_ordered
-----------+--------------+-------------166 |
36008 | 2004-01-04
113 |
66017 | 2004-01-04
198 |
75716 | 2004-01-04
27 |
150241 | 2004-01-04
148 |
182207 | 2004-01-04
9 |
188567 | 2004-01-04
Page 31 of 181
Analyzing Data
Writing Queries
45 |
24 |
121 |
(9 rows)
202416 | 2004-01-04
250295 | 2004-01-04
251417 | 2004-01-04
See Also
l
Subquery Restrictions
IN and NOT IN
While you cannot equate a single value to a set of values, you can check to see if a single value is
found within that set of values. Use the IN clause for multiple-record, single-column subqueries.
After the subquery returns results introduced by IN or NOT IN, the outer query uses them to return
the final result.
[NOT] IN subqueries take the following form:
{ expression [ NOT ] IN ( subquery )| expression [ NOT ] IN ( expression ) }
There is no limit to the number of parameters passed to the IN clause of the SELECT statement; for
example:
=> SELECT * FROM tablename WHERE column IN (a, b, c, d, e, ...);
HP Vertica also supports queries where two or more outer expressions refer to different inner
expressions:
=> SELECT * FROM A WHERE (A.x,A.x) IN (SELECT B.x, B.y FROM B);
Examples
The following query uses the VMart schema to illustrate the use of outer expressions referring to
different inner expressions:
=> SELECT product_description, product_price FROM product_dimension
WHERE (product_dimension.product_key, product_dimension.product_key) IN
(SELECT store.store_orders_fact.order_number,
store.store_orders_fact.quantity_ordered
FROM store.store_orders_fact);
product_description
| product_price
-----------------------------+--------------Brand #72 box of candy
|
326
Brand #71 vanilla ice cream |
270
(2 rows)
Page 32 of 181
Analyzing Data
Writing Queries
To find all products supplied by stores in MA, first create the inner query and run it to ensure that it
works as desired. The following query returns all stores located in MA:
=> SELECT store_key FROM store.store_dimension WHERE store_state = 'MA';
store_key
----------13
31
(2 rows)
Then create the outer or main query that specifies all distinct products that were sold in stores
located in MA. This statement combines the inner and outer queries using the IN predicate:
=> SELECT DISTINCT s.product_key, p.product_description
FROM store.store_sales_fact s, public.product_dimension p
WHERE s.product_key = p.product_key
AND s.product_version = p.product_version
AND s.store_key IN
(SELECT store_key
FROM store.store_dimension
WHERE store_state = 'MA')
ORDER BY s.product_key;
product_key |
product_description
-------------+--------------------------------------1 | Brand #1 white bread
1 | Brand #4 vegetable soup
3 | Brand #9 wheelchair
5 | Brand #15 cheddar cheese
5 | Brand #19 bleach
7 | Brand #22 canned green beans
7 | Brand #23 canned tomatoes
8 | Brand #24 champagne
8 | Brand #25 chicken nuggets
11 | Brand #32 sausage
...
...
(281 rows)
When using NOT IN, the subquery returns a list of zero or more values in the outer query where the
comparison column does not match any of the values returned from the subquery. Using the
previous example, NOT IN returns all the products that are not supplied from MA.
Notes
HP Vertica supports multicolumn NOT IN subqueries in which the columns are not marked NOT
NULL. Previously, marking the columns NOT NULL was a requirement; now, if one of the columns
is found to contain a a NULL value during query execution, HP Vertica returns a run-time error.
Similarly, IN subqueries nested within another expression are not supported if any of the column
values are NULL. For example, if in the following statement column x from either table contained a
NULL value, Vertica would return a run-time error:
Page 33 of 181
Analyzing Data
Writing Queries
See Also
l
Subquery Restrictions
IN-predicate
Notes
l
Scalar subqueries in the select-list return a single row/column value. These subqueries use
Boolean comparison operators: =, >, <, <>, <=, >=.
If the query is correlated, it returns NULL if the correlation results in 0 rows. If the query returns
more than one row, the query errors out at run time and HP Vertica displays an error message
that the scalar subquery must only return 1 row.
Subquery expressions such as [NOT] IN, [NOT] EXISTS, ANY/SOME, or ALL always return a
single Boolean value that evaluates to TRUE, FALSE, or UNKNOWN; the subquery itself can
Page 34 of 181
Analyzing Data
Writing Queries
Subqueries in the ORDER BY and GROUP BY clauses are supported; for example, the
following statement says to order by the first column, which is the select-list subquery:
SELECT (SELECT MAX(x) FROM t2 WHERE y=t1.b) FROM t1 ORDER BY 1;
See Also
l
Subquery Restrictions
Page 35 of 181
Analyzing Data
Writing Queries
Inline expansion may work best if the WITH clause is not referenced multiple times in the query, or
if some local optimizations are possible after inline expansion.
Note: You must be a superuser to enable materialization.
To enable the materialization method, use add_vertica_options:
SELECT add_vertica_options('OPT', 'ENABLE_WITH_CLAUSE_MATERIALIZATION');
Page 36 of 181
Analyzing Data
Writing Queries
product_version,
SUM(quantity_ordered*unit_price) AS total_price
FROM store_orders_fact_new
GROUP BY store_key,
product_key,
product_version
ORDER BY total_price;
Page 37 of 181
Analyzing Data
Writing Queries
See Also
l
Subquery Restrictions
WITH Clause
Correlated
The difference between noncorrelated (simple) subqueries and correlated subqueries is that in
simple subqueries, the containing (outer) query only has to take action on the results from the
subquery (inner query). In a correlated subquery, the outer query block provides values for subquery
to use in its evaluation.
Page 38 of 181
Analyzing Data
Writing Queries
Notes
l
You can use an outer join to obtain the same effect as a correlated subquery.
Arbitrary uncorrelated queries are permitted in the WHERE clause as single-row expressions;
for example:
=> SELECT COUNT(*) FROM SubQ1 WHERE SubQ1.a = (SELECT y from SubQ2);
Noncorrelated queries in the HAVING clause as single-row expressions are permitted; for
example:
=> SELECT COUNT(*) FROM SubQ1 GROUP BY SubQ1.a
from SubQ2));
See Also
l
Subquery Restrictions
In an optimization called subquery flattening, some FROM clause subqueries are flattened into the
containing query, improving the performance of subquery-containing queries.
Using the above query, HP Vertica can internally flatten it as follows:
=> SELECT * FROM (SELECT a, MAX(a) FROM t1 GROUP BY a) AS t0;
Both queries return the same results, but the flattened query runs more quickly.
The optimizer will choose the flattening plan if the subquery or view does not contain the following:
l
Aggregates
Analytics
Page 39 of 181
Analyzing Data
Writing Queries
DISTINCT keyword
EXISTS subquery
To see if a FROM clause subquery has been flattened, inspect the query plan, as described in
EXPLAIN in the SQL Reference Manual. Typically, the number of value expression nodes
(ValExpNode) decreases after flattening.
Flattening Views
When you specify a view in a FROM clause query, HP Vertica first replaces the view name with the
view definition query, creating further opportunities for subquery flattening. This process is called
view flattening and works the same way as subquery flattening. See Implementing Views in the
Administrator's Guide for additional details about views.
Examples
If you have a predicate that applies to a view or subquery, the flattening operation can allow for
optimizations by evaluating the predicates before the flattening takes place. In this example,
without flattening, HP Vertica must first evaluate the subquery, and only then can the predicate
WHERE x > 10 be applied. In the flattened subquery, HP Vertica applies the predicate before
evaluating the subquery, thus reducing the amount of work for the optimizer because it returns only
the records WHERE x > 10 to the containing query.
Assume that view v1 is defined as follows:
=> CREATE VIEW v1 AS SELECT * FROM a;
Page 40 of 181
Analyzing Data
Writing Queries
=> SELECT * FROM (SELECT * FROM a) AS t1 JOIN b ON x=y WHERE x > 10;
The following example is how HP Vertica transforms FROM clause subqueries within a WHERE
clause IN subquery:
l
Original query: SELECT * FROM a WHERE b IN (SELECT b FROM (SELECT * FROM t2) AS D
WHERE x=1;
Flattened query: SELECT * FROM a WHERE b IN (SELECT b FROM t2) AS D WHERE x=1;
See Also
l
Subquery Restrictions
UPDATE Subqueries
If you want to update records in a table based on values that are stored in other database tables,
you can nest a subquery within an UPDATE statement. See also UPDATE in the SQL Reference
Manual.
Syntax
UPDATE [[db-name.]schema.]table SET column = { expression | DEFAULT } [, ...]
[ FROM from-list ]
[ WHERE Clause ]
Semantics
UPDATE changes the values of the specified columns in all rows that satisfy the
condition. Only the columns to be modified need to be specified in the SET clause.
Columns that are not explicitly modified retain their previous values.
Outputs
Page 41 of 181
Analyzing Data
Writing Queries
Performance Tip
To be eligible for DELETE optimization, all target table columns referenced in a DELETE or UPDATE
statement's WHERE clause must be in the projection definition.
For example, the following simple schema has two tables and three projections:
CREATE TABLE tb1 (a INT, b INT, c INT, d INT);
CREATE TABLE tb2 (g INT, h INT, i INT, j INT);
The first projection references all columns in tb1 and sorts on column a:
CREATE PROJECTION tb1_p AS SELECT a, b, c, d FROM tb1 ORDER BY a;
Consider the following DML statement, which references tb1.a in its WHERE clause. Since both
projections on tb1 contain column a, both are eligible for the optimized DELETE:
DELETE FROM tb1 WHERE tb1.a IN (SELECT tb2.i FROM tb2);
Restrictions
Optimized DELETEs are not supported under the following conditions:
l
With replicated and pre-join projections if subqueries reference the target table. For example, the
following syntax is not supported:
DELETE FROM tb1 WHERE tb1.a IN (SELECT e FROM tb2, tb2 WHERE tb2.e = tb1.e);
With subqueries that do not return multiple rows. For example, the following syntax is not
Page 42 of 181
Analyzing Data
Writing Queries
supported:
DELETE FROM tb1 WHERE tb1.a = (SELECT k from tb2);
The table specified in the UPDATE list cannot also appear in the FROM list (no self joins); for
example, the following is not allowed:
=> BEGIN;
=> UPDATE result_table
SET address='new' || r2.address
FROM result_table r2
WHERE r2.cust_id = result_table.cust_id + 10;
ERROR: Self joins in UPDATE statements are not allowed
DETAIL: Target relation result_table also appears in the FROM list
If more than one row in a table to be updated matches the WHERE predicate, HP Vertica returns
an error specifying which row had more than one match.
UPDATE Example
The following series of commands illustrate the use of subqueries in UPDATE statements; they all
use the following simple schema:
=> CREATE TABLE result_table(
cust_id INTEGER,
address VARCHAR(2000));
Page 43 of 181
Analyzing Data
Writing Queries
30
30
40
50
(5 rows)
|
|
|
|
Beach Avenue
Booth Hill Road
Mt. Vernon Street
Hillside Avenue
In the following example, a noncorrelated subquery is used to change the address record in
results_table to 'New Address' when the query finds a customer ID match in both tables:
=> UPDATE result_table
SET address='New Address'
WHERE cust_id IN (SELECT new_cust_id FROM new_addresses);
The output returns the expected count indicating that three rows were updated:
Page 44 of 181
Analyzing Data
Writing Queries
OUTPUT
-------3
(1 row)
Now query the result_table table to see the changes for matching customer ID 20 and 30.
Addresses for customer ID 40 and 50 are not updated:
=> SELECT * FROM result_table;
cust_id |
address
---------+-----------------20 | New Address
30 | New Address
30 | New Address
40 | Mt. Vernon Street
50 | Hillside Avenue
(5 rows)
In the following example, a correlated subquery is used to replace all address records in the
results_table with the new_address record from the new_addresses table when the query
finds match on the customer ID in both tables:
=> UPDATE result_table
SET address=new_addresses.new_address
FROM new_addresses
WHERE cust_id = new_addresses.new_cust_id;
Again, the output returns the expected count indicating that three rows were updated:
OUTPUT
-------3
(1 row)
Now query the result_table table to see the changes for customer ID 20 and 30. Addresses for
customer ID 40 and 50 are not updated, and customer ID 60 is omitted because there is no match:
=> SELECT * FROM result_table;
cust_id |
address
---------+-----------------20 | Infinite Loop
30 | Loop Infinite
30 | Loop Infinite
40 | Mt. Vernon Street
Page 45 of 181
Analyzing Data
Writing Queries
50 | Hillside Avenue
(5 rows)
DELETE Subqueries
If you want to delete records in a table based on values that are stored in other database tables, you
can nest a subquery within a DELETE statement. See also DELETE in the SQL Reference Manual.
Syntax
DELETE FROM [[db-name.]schema.]table WHERE Clause
Semantics
The DELETE operation deletes rows that satisfy the WHERE clause from the
specified table. If the WHERE clause is absent, all table rows are deleted. The result
is a valid, even though the statement leaves an empty table.
Outputs
DELETE Example
The following series of commands illustrate the use of subqueries in DELETE statements; they all
use the following simple schema:
=>
=>
=>
=>
=>
=>
Page 46 of 181
Analyzing Data
Writing Queries
--2
(1 row)
To preserve the data for this example, issue the rollback command:
=> ROLLBACK;
Roll back to the previous state and verify that you still have two rows:
=> ROLLBACK;SELECT * FROM t;
a
--1
2
(2 rows)
The following command uses a correlated subquery to delete all rows in table t where t.a matches
a value of t2.a.
=> DELETE FROM t WHERE EXISTS (SELECT * FROM t2 WHERE t.a = t2.a);
OUTPUT
-------1
(1 row)
Page 47 of 181
Analyzing Data
Writing Queries
Roll back to the previous state and query the table again:
=> ROLLBACK;=> SELECT * FROM t;
a
--1
2
(2 rows)
See Also
l
Subquery Restrictions
Subquery Examples
This topic illustrates some of the subqueries you can write. The examples use the VMart example
database.
Single-Row Subqueries
Single-row subqueries are used with single-row comparison operators (=, >=, <=, <>, and <=>) and
return exactly one row.
For example, the following query retrieves the name and hire date of the oldest employee in the
Vmart database:
=> SELECT employee_key, employee_first_name, employee_last_name, hire_date
FROM employee_dimension
WHERE hire_date = (SELECT MIN(hire_date) FROM employee_dimension);
employee_key | employee_first_name | employee_last_name | hire_date
--------------+---------------------+--------------------+-----------2292 | Mary
| Bauer
| 1956-01-11
(1 row)
Multiple-Row Subqueries
Multiple-row subqueries return multiple records.
For example, the following IN clause subquery returns the names of the employees making the
highest salary in each of the six regions:
=> SELECT employee_first_name, employee_last_name, annual_salary, employee_region
FROM employee_dimension WHERE annual_salary IN
(SELECT MAX(annual_salary) FROM employee_dimension GROUP BY employee_region)
ORDER BY annual_salary DESC;
employee_first_name | employee_last_name | annual_salary | employee_region
Page 48 of 181
Analyzing Data
Writing Queries
---------------------+--------------------+---------------+------------------Alexandra
| Sanchez
|
992363 | West
Mark
| Vogel
|
983634 | South
Tiffany
| Vu
|
977716 | SouthWest
Barbara
| Lewis
|
957949 | MidWest
Sally
| Gauthier
|
927335 | East
Wendy
| Nielson
|
777037 | NorthWest
(6 rows)
Multicolumn Subqueries
Multicolumn subqueries return one or more columns. Sometimes a subquery's result set is
evaluated in the containing query in column-to-column and row-to-row comparisons.
Note: Multicolumn subqueries can use the <>, !=, and = operators but not the <, >, <=, >=
operators.
You can substitute some multicolumn subqueries with a join, with the reverse being true as well.
For example, the following two queries ask for the sales transactions of all products sold online to
customers located in Massachusetts and return the same result set. The only difference is the first
query is written as a join and the second is written as a subquery.
Join query:
=> SELECT *
FROM online_sales.online_sales_fact
INNER JOIN public.customer_dimension
USING (customer_key)
WHERE customer_state = 'MA';
Subquery:
=> SELECT *
FROM online_sales.online_sales_fact
WHERE customer_key IN
(SELECT customer_key
FROM public.customer_dimension
WHERE customer_state = 'MA');
The following query returns all employees in each region whose salary is above the average:
=> SELECT e.employee_first_name, e.employee_last_name, e.annual_salary,
e.employee_region, s.average
FROM employee_dimension e,
(SELECT employee_region, AVG(annual_salary) AS average
FROM employee_dimension GROUP BY employee_region) AS s
WHERE e.employee_region = s.employee_region AND e.annual_salary > s.average
ORDER BY annual_salary DESC;
employee_first_name | employee_last_name | annual_salary | employee_region |
average
---------------------+--------------------+---------------+-----------------+----------------Doug
| Overstreet
|
995533 | East
|
61192.786013986
Matt
| Gauthier
|
988807 | South
|
57337.8638902996
Lauren
| Nguyen
|
968625 | West
|
Page 49 of 181
Analyzing Data
Writing Queries
56848.4274914089
Jack
56848.4274914089
William
58928.2276119403
Luigi
59614.9170454545
Sarah
57337.8638902996
Craig
61192.786013986
Sam
59614.9170454545
Luigi
59614.9170454545
Michael
57337.8638902996
Doug
57187.2510548523
Ruth
61192.786013986
Luigi
56848.4274914089
Laura
61192.786013986
...
| Campbell
963914 | West
| Martin
943477 | NorthWest
| Campbell
939255 | MidWest
| Brown
901619 | South
| Goldberg
895836 | East
| Vu
889841 | MidWest
| Sanchez
885078 | MidWest
| Weaver
882685 | South
| Pavlov
881443 | SouthWest
| McNulty
874897 | East
| Dobisz
868213 | West
| Lang
865829 | East
You can also use the EXCEPT, INTERSECT, and UNION [ALL] keywords in FROM, WHERE,
and HAVING clauses.
The following subquery returns information about all Connecticut-based customers who bought
items through either stores or online sales channel and whose purchases amounted to more than
500 dollars:
=> SELECT DISTINCT customer_key, customer_name FROM public.customer_dimension
WHERE customer_key IN (SELECT customer_key FROM store.store_sales_fact
WHERE sales_dollar_amount > 500
UNION ALL
SELECT customer_key FROM online_sales.online_sales_fact
WHERE sales_dollar_amount > 500)
AND customer_state = 'CT';
customer_key | customer_name
--------------+-----------------200 | Carla Y. Kramer
733 | Mary Z. Vogel
931 | Lauren X. Roy
1533 | James C. Vu
2948 | Infocare
4909 | Matt Z. Winkler
5311 | John Z. Goldberg
5520 | Laura M. Martin
5623 | Daniel R. Kramer
6759 | Daniel Q. Nguyen
...
Page 50 of 181
Analyzing Data
Writing Queries
For example, the following statement uses the VMart database and returns the number of
customers who purchased lowfat products. Note that the GROUP BY clause is required because
the query uses an aggregate (COUNT).
=> SELECT s.product_key, COUNT(s.customer_key) FROM store.store_sales_fact s
GROUP BY s.product_key HAVING s.product_key IN
(SELECT product_key FROM product_dimension WHERE diet_type = 'Low Fat');
The subquery first returns the product keys for all low-fat products, and the outer query then counts
the total number of customers who purchased those products.
product_key | count
-------------+------15 |
2
41 |
1
66 |
1
106 |
1
118 |
1
169 |
1
181 |
2
184 |
2
186 |
2
211 |
1
229 |
1
267 |
1
289 |
1
334 |
2
336 |
1
(15 rows)
Subquery Restrictions
The following list summarizes subquery restrictions in HP Vertica.
Page 51 of 181
Analyzing Data
Writing Queries
Subqueries are not allowed in the defining query of a CREATE PROJECTION statement.
Subqueries can be used in the select-list, but GROUP BY or aggregate functions are not allowed
in the query if the subquery is not part of the GROUP BY clause in the containing query; for
example, the following two statement returns an error message:
=> SELECT y, (SELECT MAX(a) FROM t1) FROM t2 GROUP BY y;
ERROR: subqueries in the SELECT or ORDER BY are not supported if the
subquery is not part of the GROUP BY
=> SELECT MAX(y), (SELECT MAX(a) FROM t1) FROM t2;
ERROR: subqueries in the SELECT or ORDER BY are not supported if the
query has aggregates and the subquery is not part of the GROUP BY
Subqueries are supported within UPDATE statements with the following exceptions:
n
The table specified in the UPDATE list cannot also appear in the FROM list (no self joins).
FROM clause subqueries require an alias but tables do not. If the table has no alias, the query
must refer to columns inside it as <table>.<column>; however, if the column names are uniquely
identified among all tables used by the query, then preceding the column with a table name is not
enforced.
If the ORDER BY clause is inside a FROM clause subquery, rather than in the containing query,
the query could return unexpected sort results. This is because HP Vertica data comes from
multiple nodes, so sort order cannot be guaranteed unless an ORDER BY clause is specified in
the outer query block. This behavior is compliant with the SQL standard but it might differ from
other databases.
Multicolumn subqueries cannot use the <, >, <=, >= comparison operators. They can use <>,
!=, and = operators.
WHERE and HAVING clause subqueries must use Boolean comparison operators: =, >, <, <>,
<=, >=. Those subqueries can be noncorrelated and correlated.
n
[NOT] IN and ANY subqueries nested within another expression are not supported if any of
the column values are NULL. In the following statement, for example, if column x from either
table t1 or t2 contains a NULL value, HP Vertica returns a run-time error:
Page 52 of 181
Analyzing Data
Writing Queries
ERROR:
NULL value
HP Vertica returns an error message during subquery run time on scalar subqueries that return
more than one row.
Aggregates and GROUP BY clauses are allowed in subqueries, as long as those subqueries are
not correlated.
Multiple correlations are allowed only for subqueries that are joined with an equality predicate (<,
>, <=, >=, =, <>, <=>) but IN/NOT IN, EXISTS/NOT EXISTS predicates within correlated
subqueries are not allowed:
=> SELECT t2.x, t2.y, t2.z FROM t2 WHERE t2.z NOT IN
(SELECT t1.z FROM t1 WHERE t1.x = t2.x);
ERROR: Correlated subquery with NOT IN is not supported
Up to one level of correlated subqueries is allowed in the WHERE clause if the subquery
references columns in the immediate outer query block. For example, the following query is not
supported because the t2.x = t3.x subquery can only refer to table t1 in the outer query,
making it a correlated expression because t3.x is two levels out:
=> SELECT t3.x, t3.y, t3.z FROM t3 WHERE t3.z IN (
SELECT t1.z FROM t1 WHERE EXISTS (
SELECT 'x' FROM t2 WHERE t2.x = t3.x) AND t1.x = t3.x);
ERROR: More than one level correlated subqueries are not supported
Page 53 of 181
Analyzing Data
Writing Queries
Joins
Queries can combine records from multiple tables, or multiple instances of the same table. A query
that combines records from one or more tables is called a join. Joins are allowed in a SELECT
statement, as well as inside a subquery.
HP Vertica supports the following join types:
l
There are three basic algorithms that perform a join operation: hash, merge, and nested loop:
l
A hash join is used to join large data sets. The smaller joined table is used to build a hash table in
memory on the join column. The HP Vertica optimizer then scans the larger table and probes the
hash table to look for matches. The optimizer chooses a hash join when projections are not
sorted on the join columns.
If both inputs are pre-sorted, the optimizer can skip a sort operation and choose a merge join.
The term sort-merge join refers to the case when one or both inputs must be sorted before the
merge join. In this case, HP Vertica sorts the inner input but only if the outer input is already
sorted on the join keys.
The ANSI SQL-92 standard provided more specific join syntax, with join conditions named in the
ON clause:
=> SELECT * FROM T1
Page 54 of 181
Analyzing Data
Writing Queries
[ INNER | LEFT OUTER | RIGHT OUTER | FULL OUTER | NATURAL | CROSS ] JOIN T2
ON T1.id = T2.id
See SQL-99 ANSI syntax at BNF Grammar for SQL-99 for additional details.
Although some users continue to use the older join syntax, HP encourages you to use the SQL-92
join syntax whenever possible because of its many advantages:
l
SQL-92 outer join syntax is portable across databases; the older syntax was not consistent
between databases. (HP Vertica does not support proprietary outer join syntax such as '+' that
can be used in some databases.)
SQL-92 syntax provides greater control over whether predicates are to be evaluated during or
after outer joins. This was also not consistent between databases when using the older syntax.
See "Join Conditions vs. Filter Conditions" below.
SQL-92 syntax eliminates ambiguity in the order of evaluating the joins, in cases where more
than two tables are joined with outer joins.
Union joins can be expressed using the SQL-92 syntax, but not in the older syntax.
The ON clause contains relational operators (for example, <, <=, >, >=, <>, =, <=>) or other
predicates that specify which records from the left and right input relations to combine, such as
by matching foreign keys to primary keys. ON can be used with inner, left outer, right outer, and
full outer joins. Cross joins and union joins do not use an ON clause.
Inner joins return all pairings of rows from the left and right relations for which the ON clause
evaluates to TRUE. In a left join, all rows from the left relation in the join are present in the result;
any row of the left relation that does not match any rows in the right relation is still present in the
Page 55 of 181
Analyzing Data
Writing Queries
result but with nulls in any columns taken from the right relation. Similarly, a right join preserves
all rows from the right relation, and a full join retains all rows from both relations.
l
The WHERE clause is evaluated after the join is performed. It filters records returned by the
FROM clause, eliminating any records that do not satisfy the WHERE clause condition.
HP Vertica automatically converts outer joins to inner joins in cases where it is correct to do so,
allowing the optimizer to choose among a wider set of query plans and leading to better
performance.
Page 56 of 181
Analyzing Data
Writing Queries
Inner Joins
An inner join combines records from two tables based on a join predicate and requires that each
record in the first table has a matching record in the second table. Inner joins, thus, return only those
records from both joined tables that satisfy the join condition. Records that contain no matches are
not preserved.
Inner joins take the following form:
SELECT <column list>
FROM <left joined table>
[INNER] JOIN <right joined table>
ON <join condition>
Notes
n
Inner joins are commutative and associative, which means you can specify the tables in any
order you want, and the results do not change.
If you omit the INNER keyword, the join is still an inner join, the most commonly used type of
join.
Join conditions that follow the ON keyword generally can contain many predicates connected
with Boolean AND, OR, or NOT predicates.
For best performance, do not join on any LONG VARBINARY and LONG VARCHAR columns.
You can also use inner join syntax to specify joins for pre-join projections. See Pre-Join
Projections and Join Predicates.
Some SQL-related books and online tutorials refer to a left-joined table as the outer table and a
right-joined table as the inner table. The HP documentation often uses the left/right table
concept.
Example
In the following example, an inner join produces only the set of records that matches in both T1 and
T2 when T1 and T2 have the same data type; all other data is excluded.
=> SELECT * FROM T1 INNER JOIN T2 ON (T1.id = T2.id);
If a company, for example, wants to know the dates vendors in Utah sold inventory:
Page 57 of 181
Analyzing Data
Writing Queries
To clarify, if the vendor dimension table contained a third row that has no corresponding date when
a vendor sold inventory, then that row would not be included in the result set. Similarly, if on some
date there was no inventory sold by any vendor, those rows would be left out of the result set. If you
want to include all rows from one table or the other regardless of whether a match exists, you can
specify an outer join.
See Also
l
*
*
*
*
*
FROM
FROM
FROM
FROM
FROM
fact
fact
fact
fact
fact
JOIN
JOIN
JOIN
JOIN
JOIN
dim
dim
dim
dim
dim
ON
ON
ON
ON
ON
fact.x
fact.x
fact.x
fact.x
fact.x
= dim.x;
> dim.y;
<= dim.y;
<> dim.y;
<=> dim.y;
Page 58 of 181
Analyzing Data
Writing Queries
sale_date_key | cc_open_date
---------------+-------------156 | 2005-08-12
(1 row)
The second example uses older join syntax and the WHERE clause to join the same tables to get the
same results:
=> SELECT sale_date_key, cc_open_date
FROM online_sales.online_sales_fact, online_sales.call_center_dimension
WHERE online_sales.online_sales_fact.call_center_key =
online_sales.call_center_dimension.call_center_key
AND sale_date_key = 156;
sale_date_key | cc_open_date
---------------+-------------156 | 2005-08-12
(1 row)
HP Vertica also permits tables with compound (multiple-column) primary and foreign keys. For
example, to create a pair of tables with multi-column keys:
=> CREATE TABLE dimension(pk1 INTEGER NOT NULL, pk2 INTEGER NOT NULL);=> ALTER TABLE
dimension ADD PRIMARY KEY (pk1, pk2);
=> CREATE TABLE fact (fk1 INTEGER NOT NULL, fk2 INTEGER NOT NULL);
=> ALTER TABLE fact ADD FOREIGN KEY (fk1, fk2) REFERENCES dimension (pk1, pk2);
To join tables using compound keys, you must connect two join predicates with a Boolean AND
operator. For example:
=> SELECT * FROM fact f JOIN dimension d ON f.fk1 = d.pk1 AND f.fk2 = d.pk2;
You can write queries with expressions that contain the <=> operator for NULL=NULL joins.
=> SELECT * FROM fact JOIN dim ON fact.x <=> dim.y;
The <=> operator performs an equality comparison like the = operator, but it returns true, instead of
NULL, if both operands are NULL, and false, instead of NULL, if one operand is NULL.
=> SELECT 1 <=> 1, NULL <=> NULL, 1 <=> NULL;
?column? | ?column? | ?column?
----------+----------+---------t
| t
| f
(1 row)
Page 59 of 181
Analyzing Data
Writing Queries
Note: Writing NULL=NULL joins on primary key/foreign key combinations is not an optimal
choice because PK/FK columns are usually defined as NOT NULL.
When composing joins, it helps to know in advance which columns contain null values. An
employee's hire date, for example, would not be a good choice because it is unlikely hire date would
be omitted. An hourly rate column, however, might work if some employees are paid hourly and
some are salaried. If you are unsure about the value of columns in a given table and want to check,
type the command:
=> SELECT COUNT(*) FROM tablename WHERE columnname IS NULL;
Natural Joins
A natural join is just a join with an implicit join predicate. Natural joins can be inner, left outer, right
outer, or full outer joins and take the following form:
SELECT <column list> FROM <left-joined table>
NATURAL [ INNER | LEFT OUTER | RIGHT OUTER | FULL OUTER ] JOIN <right-joined table>
Natural joins are, by default, natural inner joins; however, there can also be natural (left/right) outer
joins. The primary difference between an inner and natural join is that inner joins have an explicit join
condition, whereas the natural joins conditions are formed by matching all pairs of columns in the
tables that have the same name and compatible data types, making natural joins equi-joins
because join condition are equal between common columns. (If the data types are incompatible, HP
Vertica returns an error.)
Note: The data type coercion chart lists the data types that can be cast to other data types. If
one data type can be cast to the other, those two data types are compatible.
The following query is a simple natural join between tables T1 and T2 when the T2 column val is
greater than 5:
=> SELECT * FROM T1 NATURAL JOIN T2 WHERE T2.val > 5;
The following example shows a natural join between the store_sales_fact table and the
product_dimension table with columns of the same name, product_key and product_version:
Page 60 of 181
Analyzing Data
Writing Queries
The following three queries return the same result expressed as a basic query, an inner join, and a
natural join. at the table expressions are equivalent only if the common attribute in the store_
sales_fact table and the store_dimension table is store_key. If both tables have a column
named store_key, then the natural join would also have a store_sales_fact.store_key =
store_dimension.store_key join condition. Since the results are the same in all three instances,
they are shown in the first (basic) query only:
=> SELECT store_name FROM store.store_sales_fact, store.store_dimension
WHERE store.store_sales_fact.store_key = store.store_dimension.store_key
AND store.store_dimension.store_state = 'MA' ORDER BY store_name;
store_name
-----------Store11
Store128
Store178
Store66
Store8
Store90
(6 rows)
In the case of the natural join, the join predicate appears implicitly by comparing all of the columns
in both tables that are joined by the same column name. The result set contains only one column
representing the pair of equally-named columns.
=> SELECT store_name FROM store.store_sales_fact
NATURAL JOIN store.store_dimension
WHERE store.store_dimension.store_state = 'MA' ORDER BY store_name;
Cross Joins
Cross joins are the simplest joins to write, but they are not usually the fastest to run because they
consist of all possible combinations of two tables records. Cross joins contain no join condition and
return what is known as a Cartesian product, where the number of rows in the result set is equal to
the number of rows in the first table multiplied by the number of rows in the second table.
Page 61 of 181
Analyzing Data
Writing Queries
The following query returns all possible combinations from the promotion table and the store sales
table:
=> SELECT * FROM promotion_dimension CROSS JOIN store.store_sales_fact;
Since this example returns over 600 million records, many cross join results can be extremely large
and difficult to manage. Cross joins can be useful, however, such as when you want to return a
single-row result set.
Tip: Tip: Filter out unwanted records in a cross with WHERE clause join predicates:
=> SELECT * FROM promotion_dimension p
CROSS JOIN store.store_sales_fact f
WHERE p.promotion_key LIKE f.promotion_key;
For details on what qualifies as a join predicate, see Pre-Join Projections and Join Predicates.
HP recommends that you do not write implicit cross joins (coma-separated tables in the FROM
clause). These queries could imply accidental omission of a join predicate.
The following statement is an example of an implicit cross join:
=> SELECT * FROM promotion_dimension , store.store_sales_fact;
If you intend is to run a cross join, write an explicit cross join query using CROSS JOIN keywords,
such as in the following statement:
=> SELECT * FROM promotion_dimension CROSS JOIN store.store_sales_fact;
Examples
The following example creates two small tables and their superprojections and then runs a cross
join on the tables:
=>
=>
=>
=>
=>
=>
=>
=>
CREATE
CREATE
INSERT
INSERT
INSERT
INSERT
INSERT
SELECT
In the result set, the cross join retrieves records from the first table and then creates a new row for
every row in the 2nd table. It then does the same for the next record in the first table, and so on.
Page 62 of 181
Analyzing Data
Writing Queries
Outer Joins
Outer joins extend the functionality of inner joins by letting you preserve rows of one or both tables
that do not have matching rows in the non-preserved table. Outer joins take the following form:
SELECT <column list>
FROM <left-joined table>
[ LEFT | RIGHT | FULL ] OUTER JOIN <right-joined table>
ON <join condition>
Note: Omitting the keyword OUTER from your statements does not affect results of left and right
joins. LEFT OUTER JOIN and LEFT JOIN perform the same operation and return the same
results.
To exclude the non-matched values from T2, write the same left outer join, but filter out the records
you don't want from the right side by using a WHERE clause:
=> SELECT * FROM T1 LEFT OUTER JOIN T2
ON T1.x = T2.x WHERE T2.x IS NOT NULL;
The following example uses a left outer join to enrich telephone call detail records with an
incomplete numbers dimension. It then filters out results that are known not to be from
Massachusetts:
=> SELECT COUNT(*) FROM calls LEFT OUTER JOIN numbers
Page 63 of 181
Analyzing Data
Writing Queries
The above query is equivalent to the following query, where T1 RIGHT OUTER JOIN T2 = T2 LEFT
OUTER JOIN T1.
=> SELECT * FROM T2 LEFT OUTER JOIN T1 ON T2.x = T1.x;
The following example identifies customers who have not placed an order:
=> SELECT customers.customer_id FROM orders RIGHT OUTER JOIN customers
ON orders.customer_id = customers.customer_id
GROUP BY customers.customer_id HAVING COUNT(orders.customer_id) = 0;
Notes
HP Vertica also supports joins where the outer (preserved) table or subquery is replicated on more
than one node and the inner (non-preserved) table or subquery is segmented across more than one
node. For example, in the following query, the fact table, which is almost always segmented,
appears on the non-preserved side of the join, and it is allowed:
Page 64 of 181
Analyzing Data
Writing Queries
Range Joins
HP Vertica provides performance optimizations for <, <=, >, >=, and BETWEEN predicates in join ON
clauses. These optimizations are particularly useful when a column from one table is restricted to
be in a range specified by two columns of another table.
Key Ranges
Multiple, consecutive key values can map to the same dimension values. Consider, for example, a
table of IPv4 addresses and their owners. Because large subnets (ranges) of IP addresses could
belong to the same owner, this dimension can be represented as:
=> CREATE TABLE ip_owners(
ip_start INTEGER,
ip_end INTEGER,
owner_id INTEGER);
=> CREATE TABLE clicks(
ip_owners INTEGER,
dest_ip INTEGER);
A query that associates a click stream with its destination can use a join similar to the following,
which takes advantage of the range optimization:
=> SELECT owner_id, COUNT(*) FROM clicks JOIN ip_owners
ON clicks.dest_ip BETWEEN ip_start AND ip_end
GROUP BY owner_id;
Page 65 of 181
Analyzing Data
Writing Queries
Slowly-Changing Dimensions
Sometimes there are multiple dimension ranges, each relevant over a different time period. For
example, stocks might undergo splits (and reverse splits), and the price or volume of two trades
might not be directly comparable without taking this into account. A split factor can be defined,
which accounts for these events through time:
=> CREATE TABLE splits(
symbol VARCHAR(10),
start DATE,
"end" DATE,
split_factor FLOAT);
A join with an optimized range predicate can then be used to match each trade with the effective
split factor:
=> SELECT trades.symbol, SUM(trades.volume * splits.split_factor)
FROM trades JOIN splits
ON trades.symbol = splits.symbol AND trades.tdate between splits.start AND splits.end
GROUP BY trades.symbol;
Notes
l
Operators <, <=, >, >=, or BETWEEN must appear as top-level conjunctive predicates for range
join optimization to be effective, as shown in the following examples:
The following example query is optimized because BETWEEN is the only predicate:
=> SELECT COUNT(*) FROM fact JOIN dim
ON fact.point BETWEEN dim.start AND dim.end;
This next example uses comparison operators as top-level predicates (within AND):
=> SELECT COUNT(*) FROM fact JOIN dim
ON fact.point > dim.start AND fact.point < dim.end;
The following query is not optimized because OR is the top-level predicate (disjunctive):
Page 66 of 181
Analyzing Data
Writing Queries
If range columns can have NULL values indicating that they are open-ended, it is possible to use
range join optimizations by replacing nulls with very large or very small values:
=> SELECT COUNT(*) FROM fact JOIN dim
ON fact.point BETWEEN NVL(dim.start, -1) AND NVL(dim.end, 1000000000000);
If there is more than one set of ranging predicates in the same ON clause, the order in which the
predicates are specified might impact the effectiveness of the optimization:
=> SELECT COUNT(*) FROM fact JOIN dim
ON fact.point1 BETWEEN dim.start1 AND dim.end1
AND fact.point2 BETWEEN dim.start2 AND dim.end2;
The optimizer chooses the first range to optimize, so write your queries so that the range you
most want optimized appears first in the statement.
l
The use of the range join optimization is not directly affected by any characteristics of the
physical schema; no schema tuning is required to benefit from the optimization.
The range join optimization can be applied to joins without any other predicates, and to HASH or
MERGE joins.
To determine if an optimization is in use, search for RANGE in the EXPLAIN plan. For example:
=> EXPLAIN SELECT owner_id, COUNT(*) FROM clicks JOIN ip_owners
ON clicks.dest_ip BETWEEN ip_start AND ip_end GROUP BY owner_id;
Page 67 of 181
Analyzing Data
Writing Queries
projection contains f.id = d.id, the pre-join can be used, even with the presence of f.id2 =
d.id2.
=> SELECT * FROM fact f JOIN dim d ON
When you create a pre-join projection, HP Vertica acquires an S lock on the anchor table. In
addition, it acquires U locks on all the other tables in the pre-join projection. These locks prevent HP
Vertica from adding a column to a dimension table while creating a new pre-join projection that uses
that dimension table.
Note: HP Vertica uses a maximum of one pre-join projection per query. More than one pre-join
projection might appear in a query plan, but at most, one will have been used to replace the join
that would be computed with the precomputed pre-join. Any other pre-join projections are used
as regular projections to supply records from a particular table.
Examples
The following is an example of a pre-join projection schema with a single-column constraint called
customer_key. The first sequence of statements creates a customer table in the public schema
and a store_sales table in the store schema. The dimension table has one primary key, and the
fact table has a foreign key that references the dimension table's primary key.
=> CREATE TABLE public.customer_dimension (
customer_key integer,
annual_income integer,
largest_bill_amount integer);
=> CREATE TABLE store.store_sales_fact (
customer_key integer,
sales_quantity integer,
sales_dollar_amount integer);
=> ALTER TABLE public.customer_dimension
ADD CONSTRAINT pk_customer_dimension PRIMARY KEY (customer_key);
=> ALTER TABLE store.store_sales_fact
ADD CONSTRAINT fk_store_sales_fact FOREIGN KEY (customer_key)
REFERENCES public.customer_dimension (customer_key);
=> CREATE PROJECTION p1 (
customer_key,
annual_income,
largest_bill_amount)
AS SELECT * FROM public.customer_dimension UNSEGMENTED ALL NODES;
=> CREATE PROJECTION p2 (
customer_key,
sales_quantity,
sales_dollar_amount)
AS SELECT * FROM store.store_sales_fact UNSEGMENTED ALL NODES;
Page 68 of 181
Analyzing Data
Writing Queries
The pre-join projection contains columns from both tables and has a join predicate between
customer_dimension and store_sales_fact along the FK-PK (primary key-foreign key)
constraints defined on the tables.
The following query uses a pre-join projection because the join predicates match the pre-join
projection's predicates exactly:
=> SELECT COUNT(*) FROM public.customer_dimension INNER JOIN store.store_sales_fact
ON public.customer_dimension.customer_key = store.store_sales_fact.customer_key;
count
------10000
(1 row)
Inner joins are commutative and associative, which means you can specify the tables in any
order you want, and the results do not change.
If you omit the INNER keyword, the join is still an inner join, the most commonly used type of
join.
Join conditions that follow the ON keyword generally can contain many predicates connected
with Boolean AND, OR, or NOT predicates.
For best performance, do not join on any LONG VARBINARY and LONG VARCHAR columns.
You can also use inner join syntax to specify joins for pre-join projections. See Pre-Join
Projections and Join Predicates.
HP Vertica supports any arbitrary join expression with both matching and non-matching column
values; for example:
Page 69 of 181
Analyzing Data
Writing Queries
=>
=>
=>
=>
=>
SELECT
SELECT
SELECT
SELECT
SELECT
*
*
*
*
*
FROM
FROM
FROM
FROM
FROM
fact
fact
fact
fact
fact
JOIN
JOIN
JOIN
JOIN
JOIN
dim
dim
dim
dim
dim
ON
ON
ON
ON
ON
fact.x
fact.x
fact.x
fact.x
fact.x
= dim.x;
> dim.y;
<= dim.y;
<> dim.y;
<=> dim.y;
HP Vertica permits joins between tables with compound (multiple-column) primary and foreign
keys, as long as you connect the two join predicates with a Boolean AND operator.
You can write queries with expressions that contain the <=> operator for NULL=NULL joins.
=> SELECT * FROM fact JOIN dim ON fact.x <=> dim.y;
The <=> operator performs an equality comparison like the = operator, but it returns true, instead
of NULL, if both operands are NULL, and false, instead of NULL, if one operand is NULL.
l
HP recommends that you do not write implicit cross joins (such as tables named in the FROM
clause separated by commas). Such queries could imply accidental omission of a join predicate.
If your intent is to run a cross join, write explicit CROSS JOIN syntax.
HP Vertica supports joins where the outer (preserved) table or subquery is replicated on more
than one node and the inner (non-preserved) table or subquery is segmented across more than
one node.
HP Vertica uses a maximum of one pre-join projection per query. More than one pre-join
projection might appear in a query plan, but at most, one will have been used to replace the join
that would be computed with the precomputed pre-join. Any other pre-join projections are used
as regular projections to supply records from a particular table.
Page 70 of 181
Find the highest score among all students in the same grade
Compare the current sales bonus each salesperson received against his or her previous bonus
Analytic functions return aggregate results but they do not group the result set. They return the
group value multiple times, once per record.
You can sort these group values, or partitions, using a window ORDER BY clause, but the order
affects only the function result set, not the entire query result set. This ordering concept is
described more fully later.
Notes
HP Vertica supports a full list of analytic functions, including:
l
FIRST_VALUE(arguments): Allows the selection of the first value of a table or partition without
having to use a self-join
NTILE(value): Equally divides the data set into a {value} number of subsets (buckets)
RANK(): Assigns a rank to each row returned from the query with respect to the other ordered
rows
STDDEV(arguments): Computes the statistical sample standard deviation of the current row
with respect to a group of rows
For additional details, see Analytic Functions in the SQL Reference Manual.
Page 71 of 181
Analyzing Data
Using SQL Analytics
Evaluation Order
Analytic functions conform to the following phases of execution:
1. Take the input rows.
Analytic functions are computed after WHERE, GROUP BY, HAVING clause operations, and
joins are performed on the query.
2. Group input rows according to the PARTITION BY clause.
The analytic PARTITION BY clause (called the window_partition_clause) is different from
table partition expressions. See Working with Table Partitions in the Administrator's Guide for
details.
3. Order rows within groups (partitions) according to ORDER BY clause.
Page 72 of 181
Analyzing Data
Using SQL Analytics
The analytic ORDER BY clause (called the window_order_clause) is different from the SQL
ORDER BY clause. If the query has a final ORDER BY clause (outside the OVER() clause),
the final results are ordered according by the SQL ORDER BY clause, not the window_order_
clause. See NULL Placement By Analytic Functions and Designing Tables to Minimize RunTime Sorting of NULL Values in Analytic Functions in the Analyzing Data Guide for additional
information about sort computation.
4. Compute some function for each row.
Notes
Analytic functions:
l
Require the OVER() clause. However, depending on the function, the window_frame_clause
and window_order_clause might not apply. For example, when used with analytic aggregate
functions like SUM(x), you can use the OVER() clause without supplying any of the windowing
clauses; in this case, the aggregate returns the same aggregated value for each row of the result
set.
Can be used in a subquery or in the parent query but cannot be nested; for example, the
following query is not allowed:
=> SELECT MEDIAN(RANK() OVER(ORDER BY sal) OVER()).
WHERE, GROUP BY and HAVING operators are technically not part of the analytic function;
however, they determine on which rows the analytic functions operate.
Note: Several examples throughout this section refer back to this example's allsales table
schema.
Page 73 of 181
Analyzing Data
Using SQL Analytics
The following analytic query returns the median value from the allsales table. Note that the median
value is reported for every row in the result set:
=> SELECT name, sales, MEDIAN(sales) OVER () AS
MEDIAN FROM allsales;
name | sales | median
------+-------+-------G
|
10 |
20
C
|
15 |
20
D
|
20 |
20
B
|
20 |
20
F
|
40 |
20
E
|
50 |
20
A
|
60 |
20
(7 rows)
See Also
l
Analytic Functions
Aggregate functions
GROUP BY clause
Page 74 of 181
Analyzing Data
Using SQL Analytics
Example
This examples illustrate the difference between aggregate functions and their analytic counterpart
using table employees defined in the below sample schema:
CREATE TABLE employees(emp_no INT, dept_no INT);
INSERT INTO employees VALUES(1, 10);
INSERT INTO employees VALUES(2, 30);
INSERT INTO employees VALUES(3, 30);
INSERT INTO employees VALUES(4, 10);
INSERT INTO employees VALUES(5, 30);
INSERT INTO employees VALUES(6, 20);
INSERT INTO employees VALUES(7, 20);
INSERT INTO employees VALUES(8, 20);
INSERT INTO employees VALUES(9, 20);
INSERT INTO employees VALUES(10, 20);
INSERT INTO employees VALUES(11, 20);
COMMIT;
Table employees:
SELECT * FROM employees ORDER BY emp_no;
emp_no | dept_no
--------+--------1 |
10
2 |
30
3 |
30
4 |
10
5 |
30
6 |
20
7 |
20
8 |
20
9 |
20
10 |
20
11 |
20
(11 rows)
Both queries below ask for the number of employees are in each department:
Aggregate query/result
Analytics query/result
OVER(PARTITION BY dept_no
Page 75 of 181
Analyzing Data
Using SQL Analytics
Aggregate query/result
Analytics query/result
dept_no | emp_count
---------+----------10 |
2
20 |
6
30 |
3
(3 rows)
If you wanted to add the employee number to the above aggregate query, you would add the emp_
no column to the GROUP BY clause. For results, you would get emp_count=1 for each row
unless the data contained employees with the same emp_no value. For example:
SELECT dept_no, emp_no, COUNT(*)
AS emp_count
FROM employees
GROUP BY dept_no, emp_no ORDER BY 1, 2;
dept_no | emp_no | emp_count
---------+--------+----------10 |
1 |
1
10 |
4 |
1
20 |
6 |
1
20 |
7 |
1
20 |
8 |
1
20 |
9 |
1
20 |
10 |
1
20 |
11 |
1
30 |
2 |
1
30 |
3 |
1
30 |
5 |
1
(11 rows)
See Also
l
Page 76 of 181
Analyzing Data
Using SQL Analytics
Window Partitioning
Window partitioning is optional. When specified, the window_partition_clause divides the rows in
the input based on user-provided expressions, such as aggregation functions like SUM(x). Window
partitioning is similar to the GROUP BY clause except that it returns only one result row per input
row. If you omit the window_partition_clause, all input rows are treated as a single partition.
The analytic function is computed per partition and starts over again (resets) at the beginning of
each subsequent partition. The window_partition_clause is specified within the OVER() clause.
Syntax
OVER( window_partition_clause
window_order_clause
window_frame_clause )
Examples
The examples in this topic use the allsales schema defined in How Analytic Functions Work.
CREATE
INSERT
INSERT
INSERT
INSERT
Page 77 of 181
Analyzing Data
Using SQL Analytics
Results are grouped into partitions for MA (35) and NY (20) under the median column.
state | name | sales | median
-------+------+-------+-------NY
| C
|
15 |
20
NY
| B
|
20 |
20
NY
| F
|
40 |
20
------------------------------MA
| G
|
10 |
35
MA
| D
|
20 |
35
MA
| E
|
50 |
35
MA
| A
|
60 |
35
(7 rows)
Page 78 of 181
Analyzing Data
Using SQL Analytics
You can work around this by having the 'WHERE sales > m' predicate complete in a subquery:
=> SELECT * FROM
(SELECT name, sales, MEDIAN(sales) OVER () AS m FROM allsales) sq
WHERE sales > m;
name | sales | m
------+-------+---F
|
40 | 20
E
|
50 | 20
A
|
60 | 20
(3 rows)
Window Ordering
Window ordering sorts the rows specified by the OVER() clause and specifies whether data is
sorted in ascending or descending order as well as the placement of null values; for example:
ORDER BY expr_list [ASC | DESC ] [NULLS {FIRST | LAST | AUTO ]. The ordering of the data
affects the results.
Using ORDER BY in an OVER clause changes the default window to RANGE UNBOUNDED
PRECEDING AND CURRENT ROW, which is described in Window Framing.
The following table shows the default null placement, with bold clauses to indicate what is implicit:
Ordering
Null placement
ORDER BY column1
Because the window_order_clause is different from a query's final ORDER BY clause, window
ordering might not guarantee the final result order; it specifies only the order within a window result
set, supplying the ordered set of rows to the window_frame_clause(if present), to the analytic
function, or to both.
To guarantee ordering of the final result set, use the SQL ORDER BY clause. (See also NULL
Placement By Analytic Functions and Designing Tables to Minimize Run-Time Sorting of NULL
Values in Analytic Functions.) You cannot use an integer for an ORDERBYclause inside an
OVER() clause.
Page 79 of 181
Analyzing Data
Using SQL Analytics
Syntax
OVER( window_partition_clause
window_frame_clause )
window_order_clause
Examples
The below examples use the allsales table schema, defined in How Analytic Functions Work.
CREATE TABLE allsales(state VARCHAR(20), name VARCHAR(20), sales INT);
INSERT INTO allsales VALUES('MA', 'A', 60);
INSERT INTO allsales VALUES('NY', 'B', 20);
INSERT INTO allsales VALUES('NY', 'C', 15);
INSERT INTO allsales VALUES('MA', 'D', 20);
INSERT INTO allsales VALUES('MA', 'E', 50);
INSERT INTO allsales VALUES('NY', 'F', 40);
INSERT INTO allsales VALUES('MA', 'G', 10);
COMMIT;
Example 1
Example 2
Page 80 of 181
Analyzing Data
Using SQL Analytics
Window Framing
Window framing represents a unique construct, called a moving window. It defines which values in
the partition are evaluated relative to the current row. You specify a window frame type by using the
RANGE or ROWS keywords, described in the next topics. Both RANGE and ROWS are
respective to the CURRENT ROW, which is the next row for which the analytic function computes
results. As the current row advances, the window boundaries are recomputed (move) along with it,
determining which rows fall into the current window.
An analytic function with a window frame specification is computed for each row based on the rows
that fall into the window relative to that row. If you omit the window_frame_clause, the default
window is RANGE UNBOUNDED PRECEDING AND CURRENT ROW.
Syntax
OVER( window_partition_clause
window_order_clause
window_frame_clause )
See window_frame_clause in the SQL Reference Manual for more detailed syntax.
CREATE
INSERT
INSERT
INSERT
INSERT
Page 81 of 181
Analyzing Data
Using SQL Analytics
INSERT INTO
INSERT INTO
INSERT INTO
INSERT INTO
INSERT INTO
INSERT INTO
INSERT INTO
COMMIT;
emp
emp
emp
emp
emp
emp
emp
VALUES(20,109,6);
VALUES(20,109,8);
VALUES(20,110,10);
VALUES(20,110,9);
VALUES(30,102,2);
VALUES(30,103,3);
VALUES(30,105,5);
Examples
The examples on this page use the emp table schema defined in Window Framing:
CREATE TABLE emp(deptno INT, sal INT, empno INT);
INSERT INTO emp VALUES(10,101,1);
INSERT INTO emp VALUES(10,104,4);
INSERT INTO emp VALUES(20,100,11);
INSERT INTO emp VALUES(20,109,7);
INSERT INTO emp VALUES(20,109,6);
INSERT INTO emp VALUES(20,109,8);
INSERT INTO emp VALUES(20,110,10);
INSERT INTO emp VALUES(20,110,9);
INSERT INTO emp VALUES(30,102,2);
INSERT INTO emp VALUES(30,103,3);
INSERT INTO emp VALUES(30,105,5);
COMMIT;
The green box represents the analytic window relative to the current row.
Page 82 of 181
Analyzing Data
Using SQL Analytics
The following example uses the ROWS-based window for the COUNT() analytic function to return
the department number, salary, and employee number with a count. The window_frame_clause
specifies the rows between the current row and two preceding. Using ROWS in the window_frame_
clause specifies the window as a physical offset and defines the start- and end-point of a window
by the number of rows before and after the current row.
SELECT deptno, sal, empno, COUNT(*) OVER (PARTITION BY deptno ORDER BY sal
ROWS BETWEEN 2 PRECEDING AND CURRENT ROW)
AS count FROM emp;
Notice that the partition includes department 20, and the current row and window are the same
because there are no rows that precede the current row within that partition, even though the query
specifies 2 preceding. The value in the count column (1) represents the number of rows in the
current window.
As the current row moves, the window spans from 1 preceding to the current row, which is as far as
it can go within the constraints of the window_frame_clause. COUNT() returns the number of rows
in the window, even if 2 preceding is specified. In the count column, (2) includes the current row and
the row above, which is the maximum the statement of 2 preceding allows within the current
partition.
Page 83 of 181
Analyzing Data
Using SQL Analytics
The current row moves again, and the window can now span 2 preceding the current row within the
partition. The count (3) includes the number of rows in the partition (2 above + current), which is the
maximum the statement of 2 preceding allows within the current partition.
When the current row moves, the window also moves to maintain 2 preceding and current row. The
count of 3 repeats because it represents the number of rows in the window, which has not changed:
Page 84 of 181
Analyzing Data
Using SQL Analytics
The current row advances again, and the window is defined by the same window, so the count does
not change.
In this example, the current row advances once more. Notice the current row has reached the end
of the deptno partition.
Page 85 of 181
Analyzing Data
Using SQL Analytics
INTERVAL Year to Month can be used in an analytic RANGE window when the ORDER BY
column type is TIMESTAMP/TIMESTAMP WITH TIMEZONE, or DATE. TIME/TIME WITH
TIMEZONE are not supported.
Page 86 of 181
Analyzing Data
Using SQL Analytics
INTERVAL Day to Second can be used when the ORDER BY column type is
TIMESTAMP/TIMESTAMP WITH TIMEZONE, DATE, and TIME/TIME WITH TIMEZONE.
Examples
The examples on this page use the emp table schema defined in Window Framing:
CREATE TABLE emp(deptno INT, sal INT, empno INT);
INSERT INTO emp VALUES(10,101,1);
INSERT INTO emp VALUES(10,104,4);
INSERT INTO emp VALUES(20,100,11);
INSERT INTO emp VALUES(20,109,7);
INSERT INTO emp VALUES(20,109,6);
INSERT INTO emp VALUES(20,109,8);
INSERT INTO emp VALUES(20,110,10);
INSERT INTO emp VALUES(20,110,9);
INSERT INTO emp VALUES(30,102,2);
INSERT INTO emp VALUES(30,103,3);
INSERT INTO emp VALUES(30,105,5);
COMMIT;
The green box represents the analytic window relative to the current row.
In the following query, RANGE specifies the window as a logical offset (value-based). The ORDER
BY column is the column on which the range is applied.
SELECT deptno, sal, empno, COUNT(*) OVER (PARTITION BY deptno ORDER BY sal
RANGE BETWEEN 2 PRECEDING AND CURRENT ROW)
AS COUNT FROM emp;
The partition includes department 20, and the current row and window are the same because there
are no rows that precede the current row within that partition:
Page 87 of 181
Analyzing Data
Using SQL Analytics
In the next example, the ORDER BY column value is 109, so 109 - 2 = 107. The window would
include all rows whose ORDER BY column values are between 107 and 109 inclusively.
Here, the current row advances, and 107-109 are still inclusive.
Page 88 of 181
Analyzing Data
Using SQL Analytics
Though the current row advances again, the window is the same.
The current row advances so that the ORDER BY column value becomes 110 (before it was 109).
Now the window would include all rows whose ORDER BY column values were between 108 and
110, inclusive, because 110 - 2 = 108.
Page 89 of 181
Analyzing Data
Using SQL Analytics
As the current row advances, the window still includes rows for 108-110, inclusive.
Remember that a window frame can also be time based, such as the following query:
SELECT ts, bid, avg(bid) OVER
(ORDER BY ts RANGE BETWEEN '40 SECONDS' PRECEDING AND CURRENT ROW)
FROM ticks WHERE stock = 'VERT'
GROUP BY bid, ts ORDER BY ts;
Page 90 of 181
Analyzing Data
Using SQL Analytics
Reporting Aggregates
Some of the analytic functions that take the window_frame_clause are the reporting aggregates.
These functions let you compare a partition's aggregate values with detail rows, taking the place of
correlated subqueries or joins.
l
AVG()
COUNT()
SUM()
If you use a window aggregate with an empty OVER() clause, the analytic function is used as a
reporting function, where the entire input is treated as a single partition.
Examples
Think of the window for reporting aggregates as a window defined as UNBOUNDED PRECEDING
and UNBOUNDED FOLLOWING. The omission of a window_order_clause makes all rows in the
partition also the window (reporting aggregates).
SELECT deptno, sal, empno, COUNT(sal)
OVER (PARTITION BY deptno) AS COUNT FROM emp;
deptno | sal | empno | count
--------+-----+-------+------10 | 101 |
1 |
2
Page 91 of 181
Analyzing Data
Using SQL Analytics
10 | 104 |
4 |
2
-----------------------------20 | 110 |
10 |
6
20 | 110 |
9 |
6
20 | 109 |
7 |
6
20 | 109 |
6 |
6
20 | 109 |
8 |
6
20 | 100 |
11 |
6
-----------------------------30 | 105 |
5 |
3
30 | 103 |
3 |
3
30 | 102 |
2 |
3
(11 rows)
If the OVER() clause in the above query contained a window_order_clause (for example, ORDER
BY sal), it would become a moving window (window aggregate) query with a default window of
RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW:
SELECT deptno, sal, empno, COUNT(sal)
FROM emp;
deptno | sal | empno | count
--------+-----+-------+------10 | 101 |
1 |
1
10 | 104 |
4 |
2
-----------------------------20 | 100 |
11 |
1
20 | 109 |
7 |
4
20 | 109 |
6 |
4
20 | 109 |
8 |
4
20 | 110 |
10 |
6
20 | 110 |
9 |
6
-----------------------------30 | 102 |
2 |
1
30 | 103 |
3 |
2
30 | 105 |
5 |
3
(11 rows)
Page 92 of 181
Analyzing Data
Using SQL Analytics
Due to default window semantics, LAST_VALUE does not always return the last value of a
partition. If you omit the window_frame_clause from the analytic clause, LAST_VALUE operates
on this default window. Results, therefore, can seem non-intuitive because the function does not
return the bottom of the current partition. It returns the bottom of the window, which continues to
change along with the current input row being processed.
Remember the default window:
OVER (PARTITION BY deptno ORDER BY sal)
Page 93 of 181
Analyzing Data
Using SQL Analytics
Page 94 of 181
Analyzing Data
Using SQL Analytics
If you want to return the last value of a partition, use UNBOUNDED PRECEDING AND
UNBOUNDED FOLLOWING.
SELECT deptno, sal, empno, LAST_VALUE(empno)
OVER (PARTITION BY deptno ORDER BY sal
ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS lv
FROM emp;
Vertica recommends that you use LAST_VALUE with the window_order_clause to produce
deterministic results.
Page 95 of 181
Analyzing Data
Using SQL Analytics
In the following example, empno 6, 7, and 8 have the same salary, so they are in adjacent rows.
empno 8 appears first in this case but the order is not guaranteed.
Notice in the output above, the last value is 7, which is the last row from the partition deptno = 20. If
the rows have a different order, then the function returns a different value:
Now the last value is 6, which is the last row from the partition deptno = 20. The solution is to add a
unique key to the sort order. Even if the order of the query changes, the result will always be the
same, and so deterministic.
SELECT deptno, sal, empno, LAST_VALUE(empno)
OVER (PARTITION BY deptno ORDER BY sal, empno
ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) as lv
FROM emp;
Page 96 of 181
Analyzing Data
Using SQL Analytics
Notice how the rows are now ordered by empno, the last value stays at 8, and it does not matter the
order of the query.
Naming Windows
You can use the WINDOW clause to name one or more windows and avoid typing long OVER()
clause syntax. The WINDOW clause takes the following form:
WINDOW window_name AS ( window_definition_clause );
[ window_partition_clause ] [ window_order_clause ]
Example
In the following example, RANK() and DENSE_RANK() use the partitioning and ordering
specifications in the window definition for a window named w:
=> SELECT RANK() OVER w , DENSE_RANK() OVER w
FROM employee_dimension
WINDOW w AS (PARTITION BY employee_region ORDER by annual_salary);
Though analytic functions can reference a named window to inherit the window_partition_clause,
you can define your own window_order_clause; for example:
=> SELECT RANK() OVER(w ORDER BY annual_salary ASC) ,
annual_salary DESC)
FROM employee_dimension
WINDOW w AS (PARTITION BY employee_region);
Page 97 of 181
Analyzing Data
Using SQL Analytics
Notes:
l
The window_partition_clause is defined in the named window specification, not in the OVER()
clause.
The OVER() clause can specify its own window_order_clause only if the window_
definition_clause did not already define it. For example, if the second example above is
rewritten as follows, the system returns an error:
=> SELECT RANK() OVER(w ORDER BY annual_salary ASC) ,
DENSE_RANK() OVER(w ORDER BY annual_salary DESC)
FROM employee_dimension
WINDOW w AS (PARTITION BY employee_region ORDER BY annual_salary);
ERROR: cannot override ORDER BY clause of window "w"
Page 98 of 181
Analyzing Data
Using SQL Analytics
Table allsales:
=> SELECT * FROM allsales;
state | name | sales
-------+------+------MA
| A
|
60
NY
| B
|
20
NY
| C
|
15
MA
| D
|
20
MA
| E
|
50
NY
| F
|
40
MA
| G
|
10
(7 rows)
The following query calculates the median of sales from the allsales table. Note that when you use
OVER() with no parameters, the query returns the same aggregated value for each row of the result
set:
=> SELECT name, sales, MEDIAN(sales) OVER() AS median FROM allsales;
name | sales | median
------+-------+-------G
|
10 |
20
C
|
15 |
20
D
|
20 |
20
Page 99 of 181
Analyzing Data
Using SQL Analytics
B
|
F
|
E
|
A
|
(7 rows)
20
40
50
60
|
|
|
|
20
20
20
20
Without analytics, you'd have to write an overly complex query to get the median sales, but
performance will suffer, and the query returns only one row:
=> SELECT sales MEDIAN FROM
(
SELECT a1.name, a1.sales, COUNT(a1.sales) Rank
FROM allsales a1, allsales a2
WHERE a1.sales < a2.sales OR
(a1.sales=a2.sales AND a1.name <= a2.name)
GROUP BY a1.name, a1.sales
ORDER BY a1.sales desc
) a3
WHERE Rank =
(SELECT (COUNT(*)+1) / 2 FROM allsales);
MEDIAN
-------20
(1 row)
Schema
DROP TABLE Ticks CASCADE;
CREATE
INSERT
INSERT
INSERT
INSERT
INSERT
INSERT
INSERT
INSERT
INSERT
COMMIT;
Analyzing Data
Using SQL Analytics
ticks Table
SELECT * FROM ticks;
ts
| stock | bid
---------------------+-------+------2011-07-12 10:23:59 | abc
| 10.75
2011-07-12 10:25:22 | xyz
| 45.16
2011-07-12 10:23:58 | abc
| 10.34
2011-07-12 10:25:27 | xyz
| 49.33
2011-07-12 10:23:54 | abc
| 10.12
2011-07-12 10:31:15 | xyz
|
2011-07-12 10:25:15 | abc
| 11.98
2011-07-12 10:25:16 | abc
|
2011-07-12 10:31:12 | xyz
| 65.25
(9 rows)
Query
SELECT ts, stock, bid, last_value(price1 IGNORE NULLS)
OVER(ORDER BY ts) - last_value(price2 IGNORE NULLS)
OVER(ORDER BY ts)
as price_diff
FROM
(SELECT ts, stock, bid,
CASE WHEN stock = 'abc' THEN bid ELSE NULL END AS price1,
CASE WHEN stock = 'xyz' then bid ELSE NULL END AS price2
FROM ticks
WHERE stock IN ('abc','xyz')
) v1
ORDER BY ts;
ts
| stock | bid | price_diff
---------------------+-------+-------+-----------2011-07-12 10:23:54 | abc
| 10.12 |
2011-07-12 10:23:58 | abc
| 10.34 |
2011-07-12 10:23:59 | abc
| 10.75 |
2011-07-12 10:25:15 | abc
| 11.98 |
2011-07-12 10:25:16 | abc
|
|
2011-07-12 10:25:22 | xyz
| 45.16 |
-33.18
2011-07-12 10:25:27 | xyz
| 49.33 |
-37.35
2011-07-12 10:31:12 | xyz
| 65.25 |
-53.27
2011-07-12 10:31:15 | xyz
|
|
-53.27
(9 rows)
Analyzing Data
Using SQL Analytics
Query
SELECT ts, bid, AVG(bid)
OVER(ORDER BY ts
RANGE BETWEEN INTERVAL '40 seconds'
PRECEDING AND CURRENT ROW)
FROM ticks
WHERE stock = 'abc'
GROUP BY bid, ts
ORDER BY ts;
ts
| bid |
?column?
---------------------+-------+-----------------2011-07-12 10:23:54 | 10.12 |
10.12
2011-07-12 10:23:58 | 10.34 |
10.23
2011-07-12 10:23:59 | 10.75 | 10.4033333333333
2011-07-12 10:25:15 | 11.98 |
11.98
2011-07-12 10:25:16 |
|
11.98
(5 rows)
DROP TABLE Ticks CASCADE;
CREATE
INSERT
INSERT
INSERT
INSERT
INSERT
INSERT
INSERT
INSERT
INSERT
COMMIT;
Analyzing Data
Using SQL Analytics
The three conditions described in this section refer to the following schema.
CREATE TABLE sortopt (
a INT NOT NULL,
b INT NOT NULL,
c INT,
d INT
);
CREATE PROJECTION sortopt_p (
a_proj,
b_proj,
c_proj,
d_proj )
AS SELECT * FROM sortopt
ORDER BY a,b,c
UNSEGMENTED ALL NODES;
INSERT INTO sortopt VALUES(5,2,13,84);
INSERT INTO sortopt VALUES(14,22,8,115);
INSERT INTO sortopt VALUES(79,9,401,33);
Condition #1
All columns in the query's GROUP BY clause must be included in the projection's sort columns. If
even one column in the GROUP BY clause is excluded from the projection's ORDER BY clause, HP
Vertica uses GROUPBY HASH instead of GROUPBY PIPELINED:
Given a projection sort order ORDER BY a, b, c:
GROUP BY a
GROUP BY a,b
GROUP BY b,a
GROUP BY a,b,c
GROUP BY c,a,b
GROUP BY a,b,c,d
The query optimizer uses GROUPBY HASH because column d is not part of the
projection sort columns.
Condition #2
If the number of columns in the query's GROUP BY clause is less than the number of columns in the
projection's ORDER BY clause, columns in the query's GROUP BY clause must occur first in the
projection's ORDER BY clause.
Given a projection sort order ORDER BY a, b ,c:
Analyzing Data
Using SQL Analytics
GROUP BY a
GROUP BY a,b
GROUP BY b,a
GROUP BY a,b,c
GROUP BY c,a,b
GROUP BY a,c
The query optimizer uses GROUPBY HASH because columns a and c do not occur
first in the projection sort columns.
Condition #3
If the columns in a query's GROUP BY clause do not appear first in the projection's ORDER BY clause,
then any early-appearing projection sort columns that are missing in the query's GROUP BY clause
must be present as single-column constant equality predicates in the query's WHERE clause.
Given a projection sort order ORDER BY a, b, c:
SELECT a FROM tab WHERE a = 10 GROUP BY b
Analyzing Data
Using SQL Analytics
Schema:
CREATE TABLE bookorders(
vendorid VARCHAR(100),
date TIMESTAMP,
sequenceno INT,
askprice FLOAT,
asksize INT,
bidprice FLOAT,
bidsize INT);
INSERT INTO bookorders VALUES('3325XPK','2011-07-12
INSERT INTO bookorders VALUES('3345XPZ','2011-07-12
INSERT INTO bookorders VALUES('445XPKF','2011-07-12
INSERT INTO bookorders VALUES('445XPKF','2011-07-12
INSERT INTO bookorders VALUES('3425XPY','2011-07-12
INSERT INTO bookorders VALUES('3727XVK','2011-07-12
INSERT INTO bookorders VALUES('5325XYZ','2011-07-12
INSERT INTO bookorders VALUES('3675XVS','2011-07-12
INSERT INTO bookorders VALUES('8972VUG','2011-07-12
COMMIT;
10:23:54',
10:23:55',
10:23:56',
10:23:57',
10:23:58',
10:23:59',
10:24:01',
10:24:05',
10:25:15',
1,
2,
3,
3,
4,
5,
6,
7,
8,
10.12,
10.55,
10.22,
10.22,
11.87,
11.66,
15.05,
15.43,
14.95,
55,
58,
43,
59,
66,
51,
44,
47,
52,
10.23,
10.75,
54);
10.25,
11.90,
11.67,
15.10,
58);
15.11,
59);
57);
61);
66);
62);
59);
57);
Query:
SELECT
sequenceno Seq,
date "Time",
vendorid ID,
LAST_VALUE (bidprice IGNORE NULLS)
OVER (PARTITION BY vendorid ORDER BY sequenceno
ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT
AS "Bid Price",
LAST_VALUE (bidsize IGNORE NULLS)
OVER (PARTITION BY vendorid ORDER BY sequenceno
ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT
AS "Bid Size",
LAST_VALUE (askprice IGNORE NULLS)
OVER (PARTITION BY vendorid ORDER BY sequenceno
ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT
AS "Ask Price",
LAST_VALUE (asksize IGNORE NULLS)
OVER (PARTITION BY vendorid order by sequenceno
ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT
AS "Ask Size"
ROW)
ROW)
ROW)
ROW )
Analyzing Data
Using SQL Analytics
FROM bookorders
ORDER BY sequenceno;
Seq |
Time
|
ID
| Bid Price | Bid Size | Ask Price | Ask Size
-----+---------------------+---------+-----------+----------+-----------+---------1 | 2011-07-12 10:23:54 | 3325XPK |
10.23 |
59 |
10.12 |
55
1 | 2011-07-12 10:23:54 | 3325XPK |
10.23 |
59 |
10.12 |
55
2 | 2011-07-12 10:23:55 | 3345XPZ |
10.75 |
57 |
10.55 |
58
2 | 2011-07-12 10:23:55 | 3345XPZ |
10.75 |
57 |
10.55 |
58
3 | 2011-07-12 10:23:56 | 445XPKF |
54 |
61 |
10.22 |
43
3 | 2011-07-12 10:23:57 | 445XPKF |
10.25 |
61 |
10.22 |
59
3 | 2011-07-12 10:23:56 | 445XPKF |
54 |
61 |
10.22 |
43
3 | 2011-07-12 10:23:57 | 445XPKF |
10.25 |
61 |
10.22 |
59
4 | 2011-07-12 10:23:58 | 3425XPY |
11.9 |
66 |
11.87 |
66
4 | 2011-07-12 10:23:58 | 3425XPY |
11.9 |
66 |
11.87 |
66
5 | 2011-07-12 10:23:59 | 3727XVK |
11.67 |
662 |
11.66 |
51
5 | 2011-07-12 10:23:59 | 3727XVK |
11.67 |
62 |
11.66 |
51
6 | 2011-07-12 10:24:01 | 5325XYZ |
15.1 |
59 |
15.05 |
44
6 | 2011-07-12 10:24:01 | 5325XYZ |
15.1 |
59 |
15.05 |
44
7 | 2011-07-12 10:24:05 | 3675XVS |
58 |
|
15.43 |
47
7 | 2011-07-12 10:24:05 | 3675XVS |
58 |
|
15.43 |
47
8 | 2011-07-12 10:25:15 | 8972VUG |
15.11 |
57 |
14.95 |
52
8 | 2011-07-12 10:25:15 | 8972VUG |
15.11 |
57 |
14.95 |
52
(18 rows)
Event-Based Windows
Event-based windows let you break time series data into windows that border on significant events
within the data. This is especially relevant in financial data where analysis often focuses on specific
events as triggers to other activity.
There are two event-based window functions in HP Vertica. These functions are an HP Vertica
extension and are not part of the SQL-99 standard:
l
Analyzing Data
Using SQL Analytics
'XYZ',
'XYZ',
'XYZ',
'XYZ',
10.0);
11.0);
10.5);
11.0);
ts
| symbol | bid
---------------------+--------+-----2009-01-01 03:00:00 | XYZ
|
10
2009-01-01 03:00:03 | XYZ
|
11
2009-01-01 03:00:06 | XYZ
| 10.5
2009-01-01 03:00:09 | XYZ
|
11
(4 rows)
SELECT CONDITIONAL_CHANGE_EVENT(bid)
OVER(ORDER BY ts)
FROM Tickstore3;
==>
ts
| symbol | bid | cce
---------------------+--------+------+----2009-01-01 03:00:00 | XYZ
|
10 | 0
2009-01-01 03:00:03 | XYZ
|
11 | 1
2009-01-01 03:00:06 | XYZ
| 10.5 | 2
2009-01-01 03:00:09 | XYZ
|
11 | 3
(4 rows)
The following figure is a graphical illustration of the change in the bid price. Each value is different
from its previous one, so the window ID increments for each time slice:
Analyzing Data
Using SQL Analytics
So the window ID starts at 0 and increments at every change in from the previous value.
In this example, the bid price changes from $10 to $11 in the second row, but then stays the same.
The CONDITIONAL_CHANGE_EVENT function increments the event window ID in row 2, but not
subsequently:
SELECT ts, symbol, bidFROM
Tickstore3
ORDER BY ts;
ts
| symbol |
bid
---------------------+--------+----2009-01-01 03:00:00 | XYZ
|
10
2009-01-01 03:00:03 | XYZ
|
11
2009-01-01 03:00:06 | XYZ
|
11
2009-01-01 03:00:09 | XYZ
|
11
SELECT CONDITIONAL_CHANGE_EVENT(bid)
BY ts)
FROM Tickstore3;
=
=>
OVER(ORDER
ts
| symbol | bid | cce
---------------------+--------+------+----2009-01-01 03:00:00 | XYZ
|
10 | 0
2009-01-01 03:00:03 | XYZ
|
11 | 1
2009-01-01 03:00:06 | XYZ
|
11 | 1
2009-01-01 03:00:09 | XYZ
|
11 | 1
The following figure is a graphical illustration of the change in the bid price at 3:00:03 only. The price
stays the same at 3:00:06 and 3:00:09, so the window ID remains at 1 for each time slice after the
change:
Analyzing Data
Using SQL Analytics
ts
| symbol
---+--------+------+----2009-01-01 03:00:00 | XYZ
2009-01-01 03:00:03 | XYZ
2009-01-01 03:00:06 | XYZ
2009-01-01 03:00:09 | XYZ
| bid
| cte------------------
|
10 |
|
11 |
| 10.5 |
|
11 |
0
1
1
2
Analyzing Data
Using SQL Analytics
The following figure is a graphical illustration that shows the bid values and window ID changes.
Because the bid value is greater than $10.6 on only the second and fourth time slices (3:00:03 and
3:00:09), the window ID returns <0,1,1,2>:
In the following example, the first query returns all records from the TickStore3 table, ordered by the
tickstore values (ts). The second query uses the CONDITIONAL_TRUE_EVENT function to
increment the window ID each time the bid value is greater than 10.6. The first time the function
increments the event window ID is on row 2, where the value is 11. The window ID then increments
each time after that, because the expression (bid > 10.6) tests true for each time slice:
SELECT ts, symbol,
bidFROM Tickstore3
ORDER BY ts;
ts
|
symbol | bid
---------------------+-------+-----2009-01-01 03:00:00 |
XYZ
|
10
2009-01-01 03:00:03 |
XYZ
|
11
2009-01-01 03:00:06 |
XYZ
|
11
2009-01-01 03:00:09 |
XYZ
|
11
ts
| symbol
---+--------+------+----2009-01-01 03:00:00 | XYZ
2009-01-01 03:00:03 | XYZ
2009-01-01 03:00:06 | XYZ
2009-01-01 03:00:09 | XYZ
| bid
| cte------------------
|
|
|
|
|
|
|
|
10
11
11
11
0
1
2
3
The following figure is a graphical illustration that shows the bid values and window ID changes.
The bid value is greater than 10.6 on the second time slice (3:00:03) and remains for the remaining
two time slices. The function increments the event window ID each time because the expression
tests true:
Analyzing Data
Using SQL Analytics
LAG(bid))
ts
| symbol | bid
---------------------+--------+-----2009-01-01 03:00:00 | XYZ
|
10
2009-01-01 03:00:03 | XYZ
|
11
2009-01-01 03:00:06 | XYZ
| 10.5
2009-01-01 03:00:09 | XYZ
|
11
ts
| symbol | bid | cte
---------------------+--------+------+----2009-01-01 03:00:00 | XYZ
|
10 | 0
2009-01-01 03:00:03 | XYZ
|
11 | 0
2009-01-01 03:00:06 | XYZ
| 10.5 | 1
2009-01-01 03:00:09 | XYZ
|
11 | 1
The following figure illustrates the second query above. When the bid price is less than the previous
value, the window ID gets incremented, which occurs only in the third time slice (3:00:06):
Analyzing Data
Using SQL Analytics
See Also
l
CONDITIONAL_CHANGE_EVENT [Analytic]
CONDITIONAL_TRUE_EVENT [Analytic]
LAG [Analytic]
Analyzing Data
Using SQL Analytics
TIMESTAMP);
3:00:00 pm');
3:00:25 pm');
3:00:45 pm');
3:01:45 pm');
3:02:45 pm');
3:02:55 pm');
3:03:55 pm');
In the following query, sessionization performs computation on the SELECT list columns, showing
the difference between the current and previous timestamp value using LAG(). It evaluates to true
and increments the window ID when the difference is greater than 30 seconds.
=> SELECT userId, timestamp,
CONDITIONAL_TRUE_EVENT(timestamp - LAG(timestamp) > '30 seconds')
OVER(PARTITION BY userId ORDER BY timestamp) AS session FROM WebClicks;
userId |
timestamp
| session
--------+---------------------+--------1 | 2009-12-08 15:00:00 |
0
1 | 2009-12-08 15:00:25 |
0
1 | 2009-12-08 15:00:45 |
0
1 | 2009-12-08 15:01:45 |
1
2 | 2009-12-08 15:02:45 |
0
2 | 2009-12-08 15:02:55 |
0
2 | 2009-12-08 15:03:55 |
1
(7 rows)
In the output, the session column contains the window ID from the CONDITIONAL_TRUE_
EVENT function. The window ID evaluates to true on row 4 (timestamp 15:01:45), and the ID that
follows row 4 is zero because it is the start of a new partition (for user ID 2), and that row does not
evaluate to true until the last line in the output.
You might want to give users different time-out thresholds. For example, one user might have a
slower network connection or be multi-tasking, while another user might have a faster connection
and be focused on a single Web site, doing a single task.
Analyzing Data
Using SQL Analytics
To compute an adaptive time-out threshold based on the last 2 clicks, use CONDITIONAL_TRUE_
EVENT with LAG to return the average time between the last 2 clicks with a grace period of 3
seconds:
SELECT userId, timestamp, CONDITIONAL_TRUE_EVENT(timestamp - LAG(timestamp) >
(LAG(timestamp, 1) - LAG(timestamp, 3)) / 2 + '3 seconds')
OVER(PARTITION BY userId ORDER BY timestamp) AS session
FROM WebClicks;
userId |
timestamp
| session
--------+---------------------+--------2 | 2009-12-08 15:02:45 |
0
2 | 2009-12-08 15:02:55 |
0
2 | 2009-12-08 15:03:55 |
0
1 | 2009-12-08 15:00:00 |
0
1 | 2009-12-08 15:00:25 |
0
1 | 2009-12-08 15:00:45 |
0
1 | 2009-12-08 15:01:45 |
1
(7 rows)
Note: You cannot define a moving window in time series data. For example, if the query is
evaluating the first row and theres no data, it will be the current row. If you have a lag of 2, no
results are returned until the third row.
See Also
l
Event-Based Windows
CONDITIONAL_TRUE_EVENT [Analytic]
TS_FIRST_VALUE and TS_LAST_VALUE are time series aggregate functions that return the
value at the start or end of a time slice, respectively, which is determined by the interpolation
scheme.
TIME_SLICE is a (SQL extension) date/time function that aggregates data by different fixedtime intervals and returns a rounded-up input TIMESTAMP value to a value that corresponds
with the start or end of the time slice interval.
See Also
l
Event-Based Windows
Analyzing Data
Using Time Series Analytics
Analyzing Data
Using Time Series Analytics
In HP Vertica, time series data is represented by a sequence of rows that conforms to a particular
table schema, where one of the columns stores the time information.
Both time and the state of data within a time series are continuous. This means that evaluating SQL
queries over time can be challenging because input records usually occur at non-uniform intervals
and could contain gaps. Consider, for example, the following table, which contains two input rows
five seconds apart, at 3:00:00 and 3:00:05.
=> SELECT * FROM TickStore;
ts
| symbol | bid
---------------------+--------+-----2009-01-01 03:00:00 | XYZ
|
10
2009-01-01 03:00:05 | XYZ
| 10.5
(2 rows)
Given those two inputs, how would you determine a bid price that fell between the two points, such
as at 3:00:03 PM?
The TIME_SLICE function, which normalizes timestamps into corresponding time slices, might
seem like a logical candidate; however, TIME_SLICE does not solve the problem of missing inputs
(time slices) in the data. Instead, HP Vertica provides gap-filling and interpolation (GFI)
functionality, which fills in missing data points and adds new (missing) data points within a range of
known data points to the output using time series aggregate functions and the SQL TIMESERIES
clause.
But first, we'll illustrate the components that make up gap filling and interpolation in HP Vertica,
starting with Constant Interpolation.
The images in the following topics use the following legend:
l
Analyzing Data
Using Time Series Analytics
The red dots represent the input records in the table, $10.0 and $10.5.
The blue stars represent the output values, including interpolated values.
Constant Interpolation
Given known input timestamps at 03:00:00 and 03:00:05 in the sample TickStore schema, how
might you determine the bid price at 03:00:03?
A common interpolation scheme used on financial data is to set the bid price to the last seen value
so far. This scheme is referred to as constant interpolation, in which HP Vertica computes a new
value based on the previous input records.
Note: Constant is HP Vertica's default interpolation scheme. Another interpolation scheme,
linear, is discussed in an upcoming topic.
Returning to the problem query, here is the table output, which shows a 5-second lag between bids
at 03:00:00 and 03:00:05:
=> SELECT * FROM TickStore;
ts
| symbol | bid
---------------------+--------+-----2009-01-01 03:00:00 | XYZ
|
10
2009-01-01 03:00:05 | XYZ
| 10.5
(2 rows)
Using constant interpolation, the interpolated bid price of XYZ remains at $10.0 at 3:00:03, which
falls between the two known data inputs (3:00:00 PM and 3:00:05). At 3:00:05, the value changes
to $10.5. The known data points are represented by a red dot, and the interpolated value at 3:00:03
is represented by the blue star.
In order to write a query that makes the input rows more uniform, you first need to understand the
TIMESERIES clause and time series aggregate functions.
Analyzing Data
Using Time Series Analytics
Note: The TIMESERIES clause requires an ORDER BY operation on the timestamp column.
The first two rows fall within the first time slice, which runs from 3:00:00 to 3:00:02. These are
the input rows for the TSA function's output for the time slice starting at 3:00:00.
The second two rows fall within the second time slice, which runs from 3:00:03 to 3:00:05.
These are the input rows for the TSA function's output for the time slice starting at 3:00:03.
The result is the start of each time slice.
Analyzing Data
Using Time Series Analytics
Examples
The following examples compare the values returned with and without the TS_FIRST_VALUE TSA
function.
This example shows the TIMESERIESclause without the TS_FIRST_VALUETSA function.
=> SELECT slice_time, bid FROM TickStore TIMESERIES slice_time AS '3 seconds' OVER
(PARTITION by TickStore.bid ORDER BY ts);
This example shows both the TIMESERIES clause and the TS_FIRST_VALUE TSA function. The
query returns the values of the bid column, as determined by the specified constant interpolation
scheme.
=> SELECT slice_time, TS_FIRST_VALUE(bid, 'CONST') bid FROM TickStore
TIMESERIES slice_time AS '3 seconds' OVER(PARTITION by symbol ORDER BY ts);
HP Vertica interpolates the last known value and fills in the missing datapoint, returning 10 at
3:00:03:
First query
slice_time
| bid
---------------------+----2009-01-01 03:00:00 | 10
2009-01-01 03:00:03 | 10.5
(2 rows)
Interpolated value
==>
slice_time
| bid
---------------------+----2009-01-01 03:00:00 | 10
2009-01-01 03:00:03 | 10
(2 rows)
Linear Interpolation
Instead of interpolating data points based on the last seen value (Constant Interpolation), linear
interpolation is where HP Vertica interpolates values in a linear slope based on the specified time
slice.
The query that follows uses linear interpolation to place the input records in 2-second time slices
and return the first bid value for each symbol/time slice combination (the value at the start of the
time slice):
=> SELECT slice_time, TS_FIRST_VALUE(bid, 'LINEAR') bid FROM Tickstore
TIMESERIES slice_time AS '2 seconds' OVER(PARTITION BY symbol ORDER BY ts);
slice_time
| bid
---------------------+-----2009-01-01 03:00:00 |
10
2009-01-01 03:00:02 | 10.2
2009-01-01 03:00:04 | 10.4
Analyzing Data
Using Time Series Analytics
(3 rows)
The following figure illustrates the previous query results, showing the 2-second time gaps (3:00:02
and 3:00:04) in which no input record occurs. Note that the interpolated bid price of XYZ changes to
10.2 at 3:00:02 and 10.3 at 3:00:03 and 10.4 at 3:00:04, all of which fall between the two known
data inputs (3:00:00 and 3:00:05). At 3:00:05, the value would change to 10.5.
Note: The known data points above are represented by a red dot, and the interpolated values
are represented by blue stars.
The following is a side-by-side comparison of constant and linear interpolation schemes.
CONST interpolation
LINEAR interpolation
GFI Examples
This topic illustrates some of the queries you can write using the constant and linear interpolation
schemes.
Analyzing Data
Using Time Series Analytics
Constant Interpolation
The first query uses TS_FIRST_VALUE() and the TIMESERIES clause to place the input records
in 3-second time slices and return the first bid value for each symbol/time slice combination (the
value at the start of the time slice).
Note: The TIMESERIES clause requires an ORDER BY operation on the TIMESTAMP
column.
Because the bid price of stock XYZ is 10.0 at 3:00:03, the first_bid value of the second time
slice, which starts at 3:00:03 is till 10.0 (instead of 10.5) because the input value of 10.5 does not
occur until 3:00:05. In this case, the interpolated value is inferred from the last value seen on stock
XYZ for time 3:00:03:
slice_time
| symbol | first_bid
---------------------+--------+----------2009-01-01 03:00:00 | XYZ
|
10
2009-01-01 03:00:03 | XYZ
|
10
(2 rows)
The next example places the input records in 2-second time slices to return the first bid value for
each symbol/time slice combination:
=> SELECT slice_time, symbol, TS_FIRST_VALUE(bid) AS first_bid FROM TickStore
TIMESERIES slice_time AS '2 seconds' OVER (PARTITION BY symbol ORDER BY ts);
The result now contains three records in 2-second increments, all of which occur between the first
input row at 03:00:00 and the second input row at 3:00:05. Note that the second and third output
record correspond to a time slice where there is no input record:
slice_time
| symbol | first_bid
---------------------+--------+----------2009-01-01 03:00:00 | XYZ
|
10
2009-01-01 03:00:02 | XYZ
|
10
2009-01-01 03:00:04 | XYZ
|
10
(3 rows)
Using the same table schema, the next query uses TS_LAST_VALUE(), with the TIMESERIES
clause to return the last values of each time slice (the values at the end of the time slices).
Analyzing Data
Using Time Series Analytics
Note: Time series aggregate functions process the data that belongs to each time slice. One
output row is produced per time slice or per partition per time slice if a partition expression is
present.
Notice that the last value output row is 10.5 because the value 10.5 at time 3:00:05 was the last
point inside the 2-second time slice that started at 3:00:04:
slice_time
| symbol | last_bid
---------------------+--------+---------2009-01-01 03:00:00 | XYZ
|
10
2009-01-01 03:00:02 | XYZ
|
10
2009-01-01 03:00:04 | XYZ
|
10.5
(3 rows)
Remember that because constant interpolation is the default, the same results are returned if you
write the query using the CONST parameter as follows:
=> SELECT slice_time, symbol, TS_LAST_VALUE(bid, 'CONST') AS last_bid FROM TickStore
TIMESERIES slice_time AS '2 seconds' OVER (PARTITION BY symbol ORDER BY ts);
Linear Interpolation
Based on the same input records described in the constant interpolation examples, which specify 2second time slices, the result of TS_LAST_VALUE with linear interpolation is as follows:
=> SELECT slice_time, symbol, TS_LAST_VALUE(bid, 'linear') AS last_bid
FROM TickStore
TIMESERIES slice_time AS '2 seconds' OVER (PARTITION BY symbol ORDER BY ts);
In the results, no last_bid value is returned for the last row because the query specified TS_LAST_
VALUE, and there is no data point after the 3:00:04 time slice to interpolate.
slice_time
| symbol | last_bid
---------------------+--------+---------2009-01-01 03:00:00 | XYZ
|
10.2
2009-01-01 03:00:02 | XYZ
|
10.4
2009-01-01 03:00:04 | XYZ
|
(3 rows)
Analyzing Data
Using Time Series Analytics
In the following output, the original output is compared to output returned by multiple time series
aggregate functions.
ts
| symbol | bid
----------+--------+-----03:00:00 | XYZ
|
10
03:00:05 | XYZ
| 10.5
(2 rows)
==>
slice_time
| symbol | fv_c | fv_l | lv_c
---------------------+--------+------+------+-----2009-01-01 03:00:00 | XYZ
|
10 |
10 |
10
2009-01-01 03:00:03 | XYZ
|
10 | 10.3 | 10.5
(2 rows)
Using slice_time
In a TIMESERIES query, you cannot use the column slice_time in the WHERE clause because
the WHERE clause is evaluated before the TIMESERIES clause, and the slice_time column is
not generated until the TIMESERIES clause is evaluated. For example, HP Vertica does not
support the following query:
Analyzing Data
Using Time Series Analytics
Instead, you could write a subquery and put the predicate on slice_time in the outer query:
=> SELECT * FROM (
SELECT symbol, slice_time,
TS_FIRST_VALUE(bid IGNORE NULLS) AS fv
FROM TickStore
TIMESERIES slice_time AS '2 seconds'
OVER (PARTITION BY symbol ORDER BY ts) ) sq
WHERE slice_time = '2009-01-01 03:00:00';
symbol |
slice_time
| fv
--------+---------------------+---XYZ
| 2009-01-01 03:00:00 | 10
(1 row)
You can create a simple union between the start and end range of the timeframe of interest in order
to return every time point. This example uses a 1-second time slice:
=> SELECT ts FROM (
SELECT '2009-01-01 03:00:00'::TIMESTAMP AS time FROM TickStore
UNION
SELECT '2009-01-01 03:00:05'::TIMESTAMP FROM TickStore) t
TIMESERIES ts AS '1 seconds' OVER(ORDER BY time);
ts
Analyzing Data
Using Time Series Analytics
--------------------2009-01-01 03:00:00
2009-01-01 03:00:01
2009-01-01 03:00:02
2009-01-01 03:00:03
2009-01-01 03:00:04
2009-01-01 03:00:05
(6 rows)
The next query creates a union between the start and end range of the timeframe using 500millisecond time slices:
=> SELECT ts FROM (
SELECT '2009-01-01 03:00:00'::TIMESTAMP AS time
FROM TickStore
UNION
SELECT '2009-01-01 03:00:05'::TIMESTAMP FROM TickStore) t
TIMESERIES ts AS '500 milliseconds' OVER(ORDER BY time);
ts
----------------------2009-01-01 03:00:00
2009-01-01 03:00:00.5
2009-01-01 03:00:01
2009-01-01 03:00:01.5
2009-01-01 03:00:02
2009-01-01 03:00:02.5
2009-01-01 03:00:03
2009-01-01 03:00:03.5
2009-01-01 03:00:04
2009-01-01 03:00:04.5
2009-01-01 03:00:05
(11 rows)
The following query creates a union between the start- and end-range of the timeframe of interest
using 1-second time slices:
=> SELECT * FROM (
SELECT ts FROM (
SELECT '2009-01-01 03:00:00'::timestamp AS time FROM TickStore
UNION
SELECT '2009-01-01 03:00:05'::timestamp FROM TickStore) t
TIMESERIES ts AS '1 seconds' OVER(ORDER BY time) ) AS outer_table
LEFT OUTER JOIN inner_table ON outer_table.ts = inner_table.ts;
The union returns a complete set of records from the left-joined table with the matched records in
the right-joined table. Where the query found no match, it extends the right side column with null
values:
ts
|
ts
| bid
---------------------+---------------------+----2009-01-01 03:00:00 |
|
Analyzing Data
Using Time Series Analytics
2009-01-01
2009-01-01
2009-01-01
2009-01-01
2009-01-01
(6 rows)
03:00:01
03:00:02
03:00:03
03:00:04
03:00:05
|
|
| 2009-01-01 03:00:02 |
|
|
| 2009-01-01 03:00:04 |
|
|
1
2
Figure 2. CONST-
Figure 3. LINEAR-
NULLs
Analyzing Data
Using Time Series Analytics
time interval denoted by the shaded region. As a result, if TS_FIRST_VALUE(bid) is evaluated with
constant interpolation on the time slice that begins at 3:00:02, its output is non-null. However, TS_
FIRST_VALUE(bid) on the next time slice produces null. If the last value of the 3:00:02 time slice is
null, the first value for the next time slice (3:00:04) is null. However, if you were to use a TSA
function with IGNORE NULLS, then the value at 3:00:04 would be the same value as it was at
3:00:02.
To illustrate, insert a new row into the TickStore table at 03:00:03 with a null bid value, HP Vertica
will output a row for the 03:00:02 record with a null value but no row for the 03:00:03 input:
=> INSERT INTO tickstore VALUES('2009-01-01 03:00:03', 'XYZ', NULL);
=> SELECT slice_time, symbol, TS_LAST_VALUE(bid) AS last_bid FROM TickStore
-> TIMESERIES slice_time AS '2 seconds' OVER (PARTITION BY symbol ORDER BY ts);
slice_time
| symbol | last_bid
---------------------+--------+---------2009-01-01 03:00:00 | XYZ
|
10
2009-01-01 03:00:02 | XYZ
|
2009-01-01 03:00:04 | XYZ
|
10.5
(3 rows)
If you specify IGNORE NULLS, HP Vertica fills in the missing data point using a constant
interpolation scheme. Here, the bid price at 03:00:02 is interpolated to the last known input record
for bid, which was $10 at 03:00:00:
=> SELECT slice_time, symbol, TS_LAST_VALUE(bid IGNORE NULLS) AS last_bid FROM TickStore
-> TIMESERIES slice_time AS '2 seconds' OVER (PARTITION BY symbol ORDER BY ts);
slice_time
| symbol | last_bid
---------------------+--------+---------2009-01-01 03:00:00 | XYZ
|
10
2009-01-01 03:00:02 | XYZ
|
10
2009-01-01 03:00:04 | XYZ
|
10.5
(3 rows)
Now if you were to insert a row where the timestamp column contained a null value, HP Vertica
would filter out that row before gap filling and interpolation occurred.
=> INSERT INTO tickstore VALUES(NULL, 'XYZ', 11.2);
=> SELECT slice_time, symbol, TS_LAST_VALUE(bid) AS last_bid FROM TickStore
-> TIMESERIES slice_time AS '2 seconds' OVER (PARTITION BY symbol ORDER BY ts);
Analyzing Data
Using Time Series Analytics
Similar to regular joins, an event series join has inner and outer join modes, which are described in
the topics that follow.
For full syntax, including notes and restrictions, see INTERPOLATE in the SQL Reference Manual
Analyzing Data
Event Series Joins
Although TIMESTAMP is more commonly used for the event series column, the examples in this
topic use TIME to keep the output simple.
INSERT INTO
INSERT INTO
INSERT INTO
INSERT INTO
INSERT INTO
INSERT INTO
INSERT INTO
INSERT INTO
COMMIT;
hTicks
hTicks
hTicks
hTicks
aTicks
aTicks
aTicks
aTicks
VALUES
VALUES
VALUES
VALUES
VALUES
VALUES
VALUES
VALUES
aTicks
12:04:
12:04:
stock |
time
| price
-------+----------+------HPQ
| 12:00:00 | 50.00
HPQ
| 12:01:00 | 51.00
HPQ
| 12:05:00 | 51.00
HPQ
| 12:06:00 | 52.00
(4 rows)
stock |
time
| price
-------+----------+-------ACME | 12:00:00 | 340.00
ACME | 12:03:00 | 340.10
ACME | 12:05:00 | 340.20
ACME | 12:05:00 | 333.80
(4 rows)
Analyzing Data
Event Series Joins
ask
stock |
time
| price
-------+----------+-------HPQ
| 12:00:00 | 100.10
HPQ
| 12:01:00 | 100.00
ACME | 12:00:00 | 80.00
ACME | 12:03:00 | 79.80
ACME | 12:05:00 | 79.90
(5 rows)
stock |
time
| price
-------+----------+-------HPQ
| 12:01:00 | 101.00
ACME | 12:00:00 | 80.00
ACME | 12:02:00 | 75.00
(3 rows)
Analyzing Data
Event Series Joins
aTicks
To replace the gaps with interpolated values for those missing data points, use the
INTERPOLATE predicate to create an event series join. The join condition is restricted to the ON
clause, which evaluates the equality predicate on the timestamp columns from the two input tables.
In other words, for each row in outer table hTicks, the ON clause predicates are evaluated for each
combination of each row in the inner table aTicks.
Simply rewrite the full outer join query to use the INTERPOLATE predicate with the required
PREVIOUS VALUE keywords. Note that a full outer join on event series data is the most common
scenario for event series data, where you keep all rows from both tables
=> SELECT * FROM hTicks h FULL OUTER JOIN aTicks a
ON (h.time INTERPOLATE PREVIOUS VALUE a.time);
HP Vertica interpolates the missing values (which appear as NULL in the full outer join) using that
table's previous value:
Analyzing Data
Event Series Joins
Note: The output ordering above is different from the regular full outer join because in the event
series join, interpolation occurs independently for each stock (hTicks and aTicks), where the
data is partitioned and sorted based on the equality predicate. This means that interpolation
occurs within, not across, partitions.
If you review the regular full outer join output, you can see that both tables have a match in the time
column at 12:00 and 12:05, but at 12:01, there is no entry record for ACME. So the operation
interpolates a value for ACME (ACME,12:00,340) based on the previous value in the aTicks table.
Here's what the same data looks like using a traditional left outer join:
=> SELECT * FROM hTicks h LEFT OUTER JOIN aTicks a ON h.time = a.time;
stock |
time
| price | stock |
time
| price
-------+----------+-------+-------+----------+-------HPQ
| 12:00:00 | 50.00 | ACME | 12:00:00 | 340.00
HPQ
| 12:01:00 | 51.00 |
|
|
HPQ
| 12:05:00 | 51.00 | ACME | 12:05:00 | 333.80
HPQ
| 12:05:00 | 51.00 | ACME | 12:05:00 | 340.20
Analyzing Data
Event Series Joins
HPQ
| 12:06:00 | 52.00 |
(5 rows)
Note that a right outer join has the same behavior with the preserved table reversed.
In the below output, the first row for stock HPQ shows nulls because there is no entry record for
HPQ before 12:01.
stock |
time
| price | stock |
time
| price
-------+----------+--------+-------+----------+-------ACME | 12:00:00 | 80.00 | ACME | 12:00:00 | 80.00
Analyzing Data
Event Series Joins
ACME |
ACME |
ACME |
HPQ
|
HPQ
|
(6 rows)
12:00:00
12:03:00
12:05:00
12:00:00
12:01:00
| 80.00 |
| 79.80 |
| 79.90 |
| 100.10 |
| 100.00 |
ACME
ACME
ACME
HPQ
|
|
|
|
|
12:02:00 | 75.00
12:02:00 | 75.00
12:02:00 | 75.00
|
12:01:00 | 101.00
Note also that the same row (ACME,12:02,75) from the ask table appears three times. The first
appearance is because no matching rows are present in the bid table for the row in ask, so Vertica
interpolates the missing value using the ACME value at 12:02 (75.00). The second appearance
occurs because the row in bid (ACME,12:05,79.9) has no matches in ask. The row from ask that
contains (ACME,12:02,75) is the closest row; thus, it is used to interpolate the values.
If you write a regular full outer join, you can see where the mismatched timestamps occur:
=> SELECT * FROM bid b FULL OUTER JOIN ask a ON (b.time = a.time);
stock |
time
| price | stock |
time
| price
-------+----------+--------+-------+----------+-------ACME | 12:00:00 | 80.00 | ACME | 12:00:00 | 80.00
ACME | 12:03:00 | 79.80 |
|
|
ACME | 12:05:00 | 79.90 |
|
|
HPQ
| 12:00:00 | 100.10 | ACME | 12:00:00 | 80.00
HPQ
| 12:01:00 | 100.00 | HPQ
| 12:01:00 | 101.00
|
|
| ACME | 12:02:00 | 75.00
(6 rows)
ran a search
made a purchase
Analyzing Data
Event Series Pattern Matching
Example
This example includes the HP Vertica pattern matching functions to analyze users' browsing
history over website2.com. It identifies patterns where the user performed the following tasks:
l
In the following statement, pattern P (Entry Onsite* Purchase) consist of three event types:
Entry, Onsite, and Purchase. When HP Vertica finds a match in the input table, the associated
Analyzing Data
Event Series Pattern Matching
pattern instance must be an event of type Entry followed by 0 or more events of type Onsite, and an
event of type Purchase
SELECT uid,
sid,
ts,
refurl,
pageurl,
action,
event_name(),
pattern_id(),
match_id()
FROM clickstream_log
MATCH
(PARTITION BY uid, sid ORDER BY ts
DEFINE
Entry
AS RefURL NOT ILIKE '%website2.com%' AND PageURL ILIKE '%website2.com%',
Onsite
AS PageURL ILIKE
'%website2.com%' AND Action='V',
Purchase AS PageURL ILIKE
'%website2.com%' AND Action = 'P'
PATTERN
P AS (Entry Onsite* Purchase)
ROWS MATCH FIRST EVENT);
In the output below, the first four rows represent the pattern for user 1's browsing activity, while the
following three rows show user 2's browsing habits.
uid | sid |
ts
|
refurl
|
pageurl
| action | event_name
| pattern_id | match_id
-----+-----+----------+----------------------+----------------------+--------+-----------+------------+---------1 | 100 | 12:00:00 | website1.com
| website2.com/home
| V
| Entry
|
1 |
1
1 | 100 | 12:01:00 | website2.com/home
| website2.com/floby
| V
| Onsite
|
1 |
2
1 | 100 | 12:02:00 | website2.com/floby
| website2.com/shamwow | V
| Onsite
|
1 |
3
1 | 100 | 12:03:00 | website2.com/shamwow | website2.com/buy
| P
| Purchase
|
1 |
4
2 | 100 | 12:10:00 | website1.com
| website2.com/home
| V
| Entry
|
1 |
1
2 | 100 | 12:11:00 | website2.com/home
| website2.com/forks
| V
| Onsite
|
1 |
2
2 | 100 | 12:13:00 | website2.com/forks
| website2.com/buy
| P
| Purchase
|
1 |
3
(7 rows)
See Also
l
MATCH Clause
Analyzing Data
Event Series Pattern Matching
PREDICATE OUTSIDE HISTOGRAM: The optimizer encountered a predicate that was false
for the entire histogram created by ANALYZE_STATISTICS or ANALYZE HISTOGRAM.
NO HISTOGRAM: The optimizer encountered a predicate on a column for which it does not
have a histogram.
Analyzing Data
Optimizing Query Performance
MEMORY LIMIT HIT: The optimizer used all its allocated memory creating the query plan. If you
see this value, simplify your query instead of increasing the memory allocation.
The QUERY_EVENTS table also gives a detailed description of each issue and suggests
solutions. For more information about this system table, see QUERY_EVENTS.
Run the EXPLAIN command. Each step (path) represents a single operation that the optimizer
uses for its execution strategy.
Query the QUERY_PLAN_PROFILES system table. This table provides detailed execution
status for currently running queries. Output from the QUERY_PLAN_PROFILES table shows
the real-time flow of data and the time and resources consumed for each path in each query plan.
Analyzing Data
Optimizing Query Performance
The precision of the NUMERIC column is 18 digits or less. Vertica has finetuned the
performance of NUMERIC data for the common case of 18 digits of precision. Vertica does not
recommend converting FLOAT columns to NUMERIC columns that require a precision of more
than 18 digits.
The precision of the FLOAT values is bounded, and the values will all fall within a specified
precision for a NUMERIC column. One example is monetary values like product prices or
Analyzing Data
Optimizing Query Performance
Analyzing Data
Optimizing Query Performance
To optimize this query, determine whether all of the projections can perform the SELECT operation
in a timely manner. Run SELECT COUNT(*) statement for each projection, specifying the date
range, and note the response time. For example:
=> SELECT COUNT(*) FROM [ projection_name ]
WHERE trade_date BETWEEN '2007-11-01' AND '2007-12-01;
If one or more of the queries is slow, check the uniqueness of the trade_date column and
determine if it needs to be in the projections ORDER BY clause and/or can be encoded using RLE.
RLE replaces sequences of the same data values within a column by a pair that represents the
value and a count. For best results, order the columns in the projection from lowest cardinality to
highest cardinality, and use RLE to encode the data in low-cardinality columns.
Note: For an example of using sorting and RLE, see Choosing Sort Order: Best Practices.
If the number of unique columns is unsorted, or if the average number of repeated rows is less than
10, trade_date is too close to being unique and cannot be encoded using RLE. In this case, add a
new column to minimize the search scope.
The following example adds a new column trade_year:
1. Determine if the new column trade_year returns a manageable result set. The following query
returns the data grouped by trade_year:
=> SELECT DATE_TRUNC('trade_year', trade_date), COUNT(*)
FROM trades
GROUP BY DATE_TRUNC('trade_year',trade_date);
2. Assuming that trade_year = 2007 is near 8k, add a column for trade_year to the trades
table. The SELECT statement then becomes:
=> SELECT * FROM trades
WHERE trade_year = 2007
AND trade_date BETWEEN '2007-11-01' AND '2007-12-01';
As a result, you have a projection that is sorted on trade_year, which can be encoded using
RLE.
Analyzing Data
Optimizing Query Performance
Because the primary key is a high-cardinality column, HP Vertica has to search a large amount of
data.
To optimize the schema for this query, create a new column named buckets and assign it the value
of the primary key divided by 10000. In this example, buckets=(int) pk/10000. Use the buckets
column to limit the search scope as follows:
=> SELECT FROM [table]
WHERE buckets IN (1,...)
AND pk IN (12345, 12346, 12347,...);
Creating a lower cardinality column and adding it to the query limits the search scope and improves
the query performance. In addition, if you create a projection where buckets is first in the sort order,
the query may run even faster.
Analyzing Data
Optimizing Query Performance
HP Vertica can use a local merge join if tables target and source use one of the following
projection designs, where their inputs are pre-sorted through the CREATE PROJECTION ORDER BY
clause:
l
Tip: For best merge performance, the source table should be smaller than the target table.
See Also
l
Analyzing Data
Optimizing Query Performance
Condition #1
All columns in the query's GROUP BY clause must be included in the projection's sort columns. If
even one column in the GROUP BY clause is excluded from the projection's ORDER BY clause, HP
Vertica uses GROUPBY HASH instead of GROUPBY PIPELINED:
Analyzing Data
Optimizing Query Performance
GROUP BY a,b
GROUP BY b,a
GROUP BY a,b,c
GROUP BY c,a,b
GROUP BY a,b,c,d
The query optimizer uses GROUPBY HASH because column d is not part of the
projection sort columns.
Condition #2
If the number of columns in the query's GROUP BY clause is less than the number of columns in the
projection's ORDER BY clause, columns in the query's GROUP BY clause must occur first in the
projection's ORDER BY clause.
Given a projection sort order ORDER BY a, b ,c:
GROUP BY a
GROUP BY a,b
GROUP BY b,a
GROUP BY a,b,c
GROUP BY c,a,b
GROUP BY a,c
The query optimizer uses GROUPBY HASH because columns a and c do not occur
first in the projection sort columns.
Condition #3
If the columns in a query's GROUP BY clause do not appear first in the projection's ORDER BY clause,
then any early-appearing projection sort columns that are missing in the query's GROUP BY clause
must be present as single-column constant equality predicates in the query's WHERE clause.
Given a projection sort order ORDER BY a, b, c:
SELECT a FROM tab WHERE a = 10 GROUP BY b
Analyzing Data
Optimizing Query Performance
Analyzing Data
Optimizing Query Performance
| |
Execute on: All Nodes
| | +---> GROUPBY HASH (SORT OUTPUT) (GLOBAL RESEGMENT GROUPS) (LOCAL RESEGMENT GROUPS)
[Cost: 245, Rows: 10K (NO STATISTICS)] (PATH ID:
4)
| | |
Group By: foo.b
| | |
Execute on: All Nodes
| | | +---> STORAGE ACCESS for foo [Cost: 202, Rows: 10K (NO STATISTICS)] (PATH ID: 5)
| | | |
Projection: public.foo_super
| | | |
Materialize: foo.b
| | | |
Execute on: All Nodes
If the result of the join operation is the input to the GROUP BY clause, HP Vertica performs the join
first, as in the following example. The segmentation of those intermediate results may not be
consistent with the GROUP BY clause in your query, resulted in resegmentation at run time.
=> EXPLAIN SELECT * FROM foo AS F, foo WHERE foo.a = F.b GROUP BY 1,2,3,4;
Access Path:
+-GROUPBY HASH (LOCAL RESEGMENT GROUPS) [Cost: 869, Rows: 10K (NO STATISTICS)] (PATH ID:
1)
| Group By: F.a, F.b, foo.a, foo.b
| Execute on: All Nodes
| +---> JOIN HASH [Cost: 853, Rows: 10K (NO STATISTICS)] (PATH ID: 2) Outer (RESEGMENT)
(LOCAL ROUND ROBIN)
| |
Join Cond: (foo.a = F.b)
| |
Execute on: All Nodes
| | +-- Outer -> STORAGE ACCESS for F [Cost: 403, Rows: 10K (NO STATISTICS)] (PUSHED
GROUPING) (PATH ID: 3)
| | |
Projection: public.foo_super
| | |
Materialize: F.a, F.b
| | |
Execute on: All Nodes
| | +-- Inner -> STORAGE ACCESS for foo [Cost: 403, Rows: 10K (NO STATISTICS)] (PATH ID:
4)
| | |
Projection: public.foo_super
| | |
Materialize: foo.a, foo.b
| | |
Execute on: All Nodes
If your query does not include joins, the GROUP BY clauses are processed using the existing
database projections.
Examples
Assume the following projection:
CREATE PROJECTION SEGMENTED BY HASH(a,b) ALL NODES
The following table explains whether or not resegmentation occurs at run time and why.
GROUP BY a
Requires resegmentation at run time. The query does not contain all the
projection segmentation columns.
Analyzing Data
Optimizing Query Performance
GROUP BY a, b
Does not require resegmentation at run time. The GROUP BY clause contains
all the projection segmentation columns.
GROUP BY a, b, c
Does not require resegmentation at run time. The GROUP BY clause contains
all the projection segmentation columns.
GROUP BY a+1, b
To determine if resegmentation will occurs during your GROUP BY query, look at the EXPLAIN plan.
For example, the following plan uses GROUPBY PIPELINED sort optimization and requires
resegmentation to perform the GROUP BY calculation:
+-GROUPBY PIPELINED (RESEGMENT GROUPS) [Cost: 194, Rows: 10K (NO STATISTICS)]
(PATH ID: 1)
The following plan uses GROUPBY PIPELINED sort optimization, but does not require
resegmentation:
+-GROUPBY PIPELINED [Cost: 459, Rows: 10K (NO STATISTICS)] (PATH ID: 1)
Analyzing Data
Optimizing Query Performance
You can use aggregate functions to get the total expenses per category or per year:
=> SELECT SUM(Amount), Category FROM expenses GROUP BY Category;
SUM
| Category
---------+-----------99.96 | Books
449.96 | Electrical
=> SELECT SUM(Amount), Year FROM expenses GROUP BY Year;
SUM
| Year
--------+-----149.97 | 2005
109.99 | 2006
29.99 | 2008
259.97 | 2007
When only the Year column is NULL, the subtotal is for all the Category values.
When both the Year and Category columns are NULL, the subtotal is for all Amount values for
both columns.
Using the ORDERBYclause orders the results by expense category, the year the expenses took
place, and the GROUPBYlevel that the GROUPING_IDfunction creates:
=> SELECT Category, Year, SUM(Amount) FROM expenses
GROUP BY ROLLUP(Category, Year) ORDER BY Category, Year, GROUPING_ID();
Category
| Year | SUM
------------+------+--------
Analyzing Data
Optimizing Query Performance
Books
Books
Books
Books
Electrical
Electrical
Electrical
Electrical
|
|
|
|
|
|
|
|
|
2005 |
2007 |
2008 |
|
2005 |
2006 |
2007 |
|
|
39.98
29.99
29.99
99.96
109.99
109.99
229.98
449.96
549.92
Similarly, the following query calculates the total sales by year and the overall sales total and then
uses the ORDERBYclause to sort the results:
=> SELECT Category, Year, SUM(Amount) FROM expenses
GROUP BY ROLLUP(Year, Category) ORDER BY 2, 1, GROUPING_ID();
Category | Year | SUM
------------+------+-------Books
| 2005 | 39.98
Electrical | 2005 | 109.99
| 2005 | 149.97
Electrical | 2006 | 109.99
| 2006 | 109.99
Books
| 2007 | 29.99
Electrical | 2007 | 229.98
| 2007 | 259.97
Books
| 2008 | 29.99
| 2008 | 29.99
|
| 549.92
(11 rows)
You can use the CUBEaggregate to perform all possible groupings of the category and year
expenses. The following query returns all possible groupings, ordered by grouping:
=> SELECT Category, Year, SUM(Amount) FROM expenses
GROUP BY CUBE(Category, Year) ORDER BY 1, 2, GROUPING_ID();
Category
| Year | SUM
------------+------+-------Books
| 2005 | 39.98
Books
| 2007 | 29.99
Books
| 2008 | 29.99
Books
|
| 99.96
Electrical | 2005 | 109.99
Electrical | 2006 | 109.99
Electrical | 2007 | 229.98
Electrical |
| 449.96
| 2005 | 149.97
| 2006 | 109.99
| 2007 | 259.97
| 2008 | 29.99
|
| 549.92
The results include subtotals for each category and each year and a total ($549.92) for all
transactions, regardless of year or category.
Analyzing Data
Optimizing Query Performance
CUBEPerforms the aggregation for all permutations of the CUBE expression that you specify.
GROUPING SETSAllows you to specify exactly which groupings of aggregations you need.
Grouping Functions
You can use any of three grouping functions with ROLLUP, CUBE, and GROUPINGSETS:
l
GROUP_IDReturns one or more numbers, starting with zero (0), to uniquely identify duplicate
sets.
Analyzing Data
Optimizing Query Performance
Analyzing Data
Optimizing Query Performance
as:
SELECT a, b, c FROM table1 GROUP BY a, b, c;
For fastest execution, apply the optimization techniques for GROUP BY queries described in
Optimizing GROUP BY Queries.
Analyzing Data
Optimizing Query Performance
One of the sorted projections delivers an order that enables sorted aggregation to be performed.
When an approximate value will suffice or the values need to be rolled up, consider using the
APPROXIMATE_COUNT_DISTINCT*functions.
You have a large data set and you do not require an exact count of distinct values.
You have a large data set and you don't require an exact count of distinct values.
You want to pre-compute the distinct counts and later combine them in different ways.
Analyzing Data
Optimizing Query Performance
as:
SELECT a, b, COUNT(dcnt) FROM
(SELECT a, b, c AS dcnt FROM table1 GROUP BY a, b, c)
GROUP BY a, b;
For fastest execution, apply the optimization techniques for GROUP BY queries.
For a query with multiple DISTINCT aggregates, there is no projection design that can avoid using
GROUPBY HASH and resegmenting the data. To improve performance of this query, make sure that it
has large amounts of memory available. For more information about memory allocation for queries,
see The Resource Manager.
Analyzing Data
Optimizing Query Performance
The following sections provide recommendations for designing your projections and your queries to
reduce query run time and improve the performance of queries that perform joins.
Merge joinIf both inputs are pre-sorted on the join column, the optimizer chooses the faster
merge join, which also uses less memory. Vertica can only perform merge joins on queries that
have INSERT and SELECT operations.
Hash joinUsing the hash join algorithm, HP Vertica uses the smaller (inner) joined table to
build an in-memory hash table on the join column. HP Vertica then scans the outer (larger) table
and probes the hash table for matches. A hash join has no sort requirement, but it consumes
more memory because a hash table is built with the values in the inner table.The cost of
performing a hash join is low if the entire hash table can fit in memory, but the cost rises when
the hash table is written to disk. The optimizer chooses a hash join when projections are not
sorted on the join columns.
The optimizer automatically chooses the most appropriate algorithm given the query and
projections in a system. You can facilitate a merge join by adding a projection that is sorted on the
join key.
Analyzing Data
Optimizing Query Performance
You also get a merge join if your query has an equality predicate in your query and the join key
immediately follows the column used in the equality predicate, as in the following example. After
applying the predicate, the data is sorted by the join_key column:
CREATE TABLE first (
data INT,
join_key INT
);
CREATE TABLE second (
data INT,
join_key INT
);
CREATE PROJECTION first_p (data, join_key)
AS SELECT data, join_key FROM first
ORDER BY data, join_key;
CREATE PROJECTION second_p (data, join_key)
AS SELECT data, join_key FROM second
ORDER BY join_key;
SELECT first.data, second.data FROM first, second
WHERE first.join_key = second.join_key AND first.data = 5;
Analyzing Data
Optimizing Query Performance
Analyzing Data
Optimizing Query Performance
The following conditions must be true for two segmented projections p1 of table t1 and p2 of table
t2 to participate in a join of t1 to t2:
l
The join columns must share the same base data type. For example:
n
If t1.j1 is a CHAR(10), t2.j1 can be any CHAR or VARCHAR (for example, CHAR(10),
VARCHAR(10), VARCHAR(20)), but t2.j1 cannot be an INTEGER.
The assignment of segments to nodes must match. For example, if p1 and p2 use an OFFSET
clause, their offsets must match.
If HP Vertica finds projections for t1 and t2 that are not identically segmented, the data is
redistributed across the network during query run time, as necessary.
Tip: If you are creating custom designs, try to use segmented projections for joins
whenever possible. See the following section "Designing Identically Segmented
Projections for K-Safety".
The following statements create two tables and specify ISP conditions:
Analyzing Data
Optimizing Query Performance
CREATE TABLE t1 (id INT, x1 INT, y1 INT) SEGMENTED BY HASH(id) ALL NODES;
CREATE TABLE t2 (id INT, x2 INT, y2 INT) SEGMENTED BY HASH(id) ALL NODES;
Corresponding to this design, the following syntax shows ISP-supported join conditions:
SELECT
SELECT
SELECT
SELECT
*
*
*
*
FROM
FROM
FROM
FROM
t1
t1
t1
t1
JOIN
JOIN
JOIN
JOIN
t2
t2
t2
t2
ON
ON
ON
ON
t1.id
t1.id
t1.x1
t1.id
=
=
=
=
t2.id;
-- ISP
t2.id AND t1.x1 = t2.x2; -- ISP
t2.x2;
-- NOT ISP
t2.x2;
-- NOT ISP
The following syntax is another example of suboptimal buddy projection design. The projections are
not identically segmented to each other because their segmentation expressions differ, so the
projections do not qualify as buddies:
CREATE PROJECTION t1_b2 (id, x1, y1)
AS SELECT * FROM t1
SEGMENTED BY HASH(id, x1)
ALL NODES OFFSET 1;
Buddy projections can use different sort orders. For details, see Hash Segmentation in the SQL
Reference Manual.
Notes
l
HP Vertica recommends that you use Database Designer to create projections, which uses
HASH and ALL NODES syntax.
Analyzing Data
Optimizing Query Performance
See Also
l
CREATE PROJECTION
Analyzing Data
Optimizing Query Performance
Based on this sort order, if a SELECT * FROM sortopt query contains one of the following ORDER
BY clauses, the query does not have to resort the projection:
l
ORDER BY a
ORDER BY a, b
ORDER BY a, b, c
For example, HP Vertica does not have to resort the projection in the following query because the
sort order includes columns specified in the CREATE PROJECTION..ORDER BY a, b, c clause,
which mirrors the query's ORDER BY a, b, c clause:
=> SELECT * FROM sortopt ORDER BY a, b, c;
a | b | c | d
Analyzing Data
Optimizing Query Performance
----+----+-----+----5 | 2 | 13 | 84
14 | 22 |
8 | 115
79 | 9 | 401 | 33
(3 rows)
If you include column d in the query, HP Vertica must re-sort the projection data because column d
was not defined in the CREATE PROJECTION..ORDER BY clause. Therefore, the ORDER BY d query
won't benefit from any sort optimization.
You cannot specify an ASC or DESC clause in the CREATE PROJECTION statement's ORDER
BY clause. HP Vertica always uses an ascending sort order in physical storage, so if your query
specifies descending order for any of its columns, the query still causes HP Vertica to re-sort the
projection data. For example, the following query requires HP Vertica to sort the results:
=> SELECT * FROM sortopt ORDER BY a DESC, b, c;
a | b | c | d
----+----+-----+----79 | 9 | 401 | 33
14 | 22 |
8 | 115
5 | 2 | 13 | 84
(3 rows)
See Also
l
CREATE PROJECTION
Analyzing Data
Optimizing Query Performance
Analyzing Data
Optimizing Query Performance
OVER(ORDER BY ...)
(SQL) ORDER BY
projection.)
If you do not care about NULL placement in queries that involve analytic computations, or if you
know that columns contain no NULL values, specify NULLS AUTOirrespective of data type. HP
Vertica chooses the placement that gives the fastest performance. Otherwise, specify NULLS
FIRST or NULLS LAST.
You can carefully formulate queries so HP Vertica can avoid sorting the data and process the query
quicker, as illustrated by the following example. HP Vertica sorts inputs from table t on column x,
as specified in the OVER(ORDER BY) clause, and then evaluates RANK():
=> CREATE TABLE t (
x FLOAT,
y FLOAT );
=> CREATE PROJECTION t_p (x, y) AS SELECT * FROM t
ORDER BY x, y UNSEGMENTED ALL NODES;
=> SELECT x, RANK() OVER (ORDER BY x) FROM t;
In the preceding SELECT statement, HP Vertica eliminates the ORDER BY clause and runs the query
quickly because column x is a FLOAT data type. As a result, the projection sort order matches the
analytic default ordering (ASC + NULLS LAST). HP Vertica can also avoid having to sort the data
when the underlying projection is already sorted.
However, if column x is defined as INTEGER, HP Vertica must sort the data because the
projection sort order for INTEGER data types (ASC + NULLS FIRST) does not match the default
Analyzing Data
Optimizing Query Performance
analytic ordering (ASC + NULLS LAST). To help HP Vertica eliminate the sort, specify the
placement of NULLs to match the default ordering:
=> SELECT x, RANK() OVER (ORDER BY x NULLS FIRST) FROM t;
If you omit NULLS LAST in the preceding query, it eliminates the sort because ASC + NULLS LAST is
the default sort specification for both the analytic ORDER BY clause and for string-related columns in
HP Vertica.
If you do not care about NULL placement in queries that involve analytic computations, or if you
know that columns contain no NULL values, specify NULLS AUTO. In the following query, HP
Vertica chooses the placement that gives the fastest performance:
=> SELECT x, RANK() OVER (ORDER BY x NULLS AUTO) FROM t;
See Also
l
Analyzing Data
Optimizing Query Performance
INSERT
INSERT
INSERT
INSERT
INSERT
INSERT
INTO
INTO
INTO
INTO
INTO
INTO
employee
employee
employee
employee
employee
employee
VALUES(101,10,50000);
VALUES(103,10,43000);
VALUES(104,10,45000);
VALUES(105,20,97000);
VALUES(108,20,33000);
VALUES(109,20,51000);
Tip: If you do not care about NULL placement in queries that involve analytic computations, or
if you know that columns contain no NULL values, specify NULLS AUTO in your queries. HP
Vertica attempts to choose the placement that gives the fastest performance. Otherwise,
specify NULLS FIRST or NULLS LAST.
Analyzing Data
Optimizing Query Performance
If table t1 contains millions of rows, it is time consuming to sort all the x values. Instead, HP
Vertica keeps track of the smallest three values in x.
Note: If you omit the ORDER BY clause, when using the LIMIT clause, the results can be
nondeterministic.
Sort operations that precede a SQL analytics computation benefit from Top-K optimization if the
query contains an OVER(ORDER BY) clause and a predicate on the ROW_NUMBER function, as in the
following example:
=> SELECT x FROM
(SELECT *, ROW_NUMBER() OVER (ORDER BY x) AS row
FROM t1) t2 WHERE row <= 3;
The preceding query has the same behavior as the following query, which uses a LIMIT clause:
=> SELECT ROW_NUMBER() OVER (ORDER BY x) AS RANK FROM t1 LIMIT 3;
You can use ROW_NUMBER() with the analytic window_partition_clause, something you cannot do if
you use LIMIT:
=> SELECT x, y FROM
(SELECT *, ROW_NUMBER() OVER (PARTITION BY x ORDER BY y)
AS row FROM t1) t2 WHERE row <= 3;
Note: When the OVER() clause includes the window_partition_clause, Top-K optimization
occurs only when the analytic sort matches the input's sort, for example, if the projection is
sorted on columns x and y in table t1.
If you still want to improve the performance of your query, consider using the optimization
techniques described in Optimizing ORDER BY Queries.
Analyzing Data
Optimizing Query Performance
Analyzing Data
Optimizing Query Performance
The following INSERT does not require a sort because the query result has the column order of the
projection:
=> INSERT /*+direct*/ INTO destination SELECT * FROM source;
The following INSERT requires a sort because the order of the columns in the SELECT statement
does not match the projection order:
=> INSERT /*+direct*/ INTO destination SELECT col1, col3, col2 FROM source;
The following INSERT does not require a sort. The order of the columns doesn't match, but the
explicit ORDER BY causes the output to be sorted by c1, c3, c2 in HP Vertica:
=> INSERT /*+direct*/ INTO destination SELECT col1, col3, col2 FROM source
Analyzing Data
Optimizing Query Performance
Analyzing Data
Optimizing Query Performance
Query performance after large deletesA large number of (unpurged) deleted rows can
negatively affect query performance.
To eliminate rows that have been deleted from the result, a query must do extra processing. If
10% or more of the total rows in a table have been deleted, the performance of a query on the
table degrades. However, your experience may vary depending on the size of the table, the table
definition, and the query. If a table has a large number of deleted rows, consider purging those
rows to improve performance. For more information on purging, see Purging Deleted Data.
Recovery performanceRecovery is the action required for a cluster to restore K-safety after
a crash. Large numbers of deleted records can degrade the performance of a recovery. To
improve recovery performance, purge the deleted rows. For more information on purging, see
Purging Deleted Data.
ConcurrencyDELETE and UPDATE take exclusive locks on the table. Only one DELETE or
UPDATE transaction on a table can be in progress at a time and only when no loads (or INSERTs)
are in progress. DELETEs and UPDATEs on different tables can be run concurrently.
Pre-join projectionsAvoid pre-joining dimension tables that are frequently updated. DELETE
and UPDATE operations on pre-join projections cascade to the fact table, causing large DELETE or
UPDATE operations.
For detailed tips about improving DELETE and UPDATE performance, see Optimizing DELETEs and
UPDATEs for Performance.
Analyzing Data
Optimizing Query Performance
Caution: HP Vertica does not remove deleted data immediately but keeps it as history for the
purposes of historical query. A large amount of history can result in slower query
performance. For information about how to configure the appropriate amount of history to
retain, see Purging Deleted Data.
In the following query, both p1 and p2 are eligible for DELETE and UPDATE optimization because
column a is available:
DELETE from t WHERE a = 1;
In the following example, only projection p1 is eligible for DELETE and UPDATE optimization because
the b column is not available in p2:
DELETE from t WHERE b = 1;
Analyzing Data
Optimizing Query Performance
The first projection references all columns in tb1 and sorts on column a:
CREATE PROJECTION tb1_p AS SELECT a, b, c, d FROM tb1 ORDER BY a;
Consider the following DML statement, which references tb1.a in its WHERE clause. Since both
projections on tb1 contain column a, both are eligible for the optimized DELETE:
DELETE FROM tb1 WHERE tb1.a IN (SELECT tb2.i FROM tb2);
Restrictions
Optimized DELETEs are not supported under the following conditions:
l
With replicated and pre-join projections if subqueries reference the target table. For example, the
following syntax is not supported:
DELETE FROM tb1 WHERE tb1.a IN (SELECT e FROM tb2, tb2 WHERE tb2.e = tb1.e);
With subqueries that do not return multiple rows. For example, the following syntax is not
Analyzing Data
Optimizing Query Performance
supported:
DELETE FROM tb1 WHERE tb1.a = (SELECT k from tb2);
To optimize the DELETEs, make time_key appear in the ORDER BY clause of all your projections.
This schema design results in better performance of the DELETE operation.
In addition, add additional sort columns to the sort order such that each combination of the sort key
values uniquely identifies a row or a small set of rows. For more information, see Choosing Sort
Order: Best Practices. To analyze projections for sort order issues, use the EVALUATE_DELETE_
PERFORMANCE function.